How we rate the evidence
Every question gets two separate ratings. They answer different things: how solid the research is, and how directly it applies to products built on large language models.
Evidence: how solid is the research?
| Strong Strong evidence, safe to rely on | Several controlled experiments or field studies, with hundreds of participants or more, pointing the same way. |
|---|---|
| Moderate Promising, test it with your users | At least one well-powered controlled experiment or large audit, but no independent replication in this set. |
| Limited Early signal, treat it as a hypothesis | Small, qualitative or builder-run studies, or only an abstract we could verify. |
| Principles Expert principles, not yet tested | Expert principles or guidelines with no outcome test. |
LLM fit: what was it tested on?
| High Tested on LLM products | Tested with real LLM products or LLM tools in realistic work. |
|---|---|
| Medium Tested in lab or early LLM studies, likely carries over | Tested with an LLM, but in a lab, with a fictional or simulated system, or on a narrow task; or pre-LLM work plus an LLM study that points the same way. |
| Low Older AI research, check before applying | Pre-LLM systems: simple statistical models, classic classifiers or simulated AI. The mechanism may transfer, but it hasn't been tested with LLMs. |
How we check
- Every number, sample size, venue and version comes from the paper text itself. If we can't verify a detail, we leave it out.
- Quotes are checked word for word against the paper.
- Rules are tagged From the study when a paper tested or stated them, and Our reading when they're our interpretation.
- Ratings are re-checked whenever a paper is added.