Methodology

1. The shape of the problem

Legal research has a failure mode that generic retrieval does not. If a system asked “does market dilution defeat fair use?” returns five passages that all say yes, the answer reads as authoritative and is wrong — because two federal judges reached opposite conclusions on that question within two days of each other. Missing the case that cuts the other way is not a thinner answer; it is a bad one.

That makes AI-and-copyright a good test domain. It is genuinely unsettled, the disagreements are recent and sharp, and the opinions are public domain.

2. Building the corpus

49 published opinions, ~640,000 words: the three decisions that have actually reached the merits on AI training and fair use, the adjacent rulings on pleading and DMCA §1202, the doctrinal lineage they argue over, and — added later, and explained in §5b — seventeen deliberately different AI cases spanning nine other areas of law, from voice cloning to algorithmic hiring, that the retriever is supposed to reject for a fair-use question.

The awkward discovery was that these need two incompatible pipelines. The doctrinal ancestors live in CourtListener's case-law database with clean text, parallel citations, and citation edges attached. The modern AI decisions are not there at all — they come from govinfo's USCOURTS packages or the courts' own sites as PDFs, with no citation metadata. One of them, NYT v. OpenAI, has zero govinfo granules because the Southern District of New York does not feed that docket to GPO.

• Resolving a case is harder than it looks. Searching CourtListener for the citation string “510 U.S. 569” returns opinions that citeCampbell, not Campbell — the first attempt silently ingested an unrelated 2025 case. Searching by name is also insufficient: Campbell's top two hits are the 1993 cert-stage entries, not the 1994 merits decision. The fetcher now requires an exact parallel-citation match and refuses to guess.
• Chunking on structure, not characters. Opinions have parts and subsections; splitting on them means a chunk tends to hold a complete piece of reasoning. An early version of the heading regex made the trailing period optional, which matched ordinary prose — Warhol alone produced 63 fabricated section labels before that was caught.

3. Retrieval, stage by stage

Every stage is a separate function so the evaluation harness can switch it off and price it. BM25 is written out rather than imported, and checked against rank_bm25: they agree to three decimals on rare terms and diverge on common ones, because the reference implementation clamps negative IDF to a floor and collapses “fair” and “use” to identical weights.

• Two lanes. BM25 for exact terms of art; dense vectors (Titan Text Embeddings v2) for meaning.
• Reciprocal Rank Fusion combines them by rank rather than score — BM25 scores and cosine similarities live on incomparable scales.
• A temporal filter drops opinions that did not exist yet.
• A cross-encoder (Cohere Rerank 3.5) rescores each passage against the question, reading both together rather than comparing two independent vectors.
• Per-case diversification caps how many passages any one opinion contributes — and runs after the reranker, not before.

4. What the measurements said

These are the results on the first corpus — 28 fair-use opinions, a 19-question golden set — before it was widened; §5b re-runs the same grid on the full 49-case corpus and some of these findings move. The headline metric is critical recall — the fraction of must-have cases retrieved — because that is the one whose failure produces a confidently wrong answer.

• BM25 alone beat dense retrieval alone (0.895 vs 0.789). Legal writing is thick with exact terms of art, and this corpus is small enough that semantic bridging has little room to help.
• Hybrid was worse than BM25 alone (0.868). Fusion diluted a strong sparse signal. Combining retrievers is not automatically an improvement.
• Diversification was the single biggest win (0.868 → 0.921) and is the only reason the two-days-apart disagreement surfaces at all.
• Stage order mattered as much as stage choice. Reranking selects the top-N passages, which re-concentrates onto a few opinions and silently undoes an earlier diversification pass. Moving diversification after the reranker recovered 0.895 → 0.921.
• Only the temporal filter prevents anachronism. Asked what the law was in 2015, every other configuration confidently retrieved a 2023 decision.

One caution the numbers themselves impose: with 19 questions, differences under about 0.05 are inside the noise. A 0.026 gap is roughly one question changing its mind, which is not a finding.

5. Query transformation, and a result still in dispute

HyDE — drafting a hypothetical judicial passage and searching with that instead of the bare question — improved overall recall, precision, and mean reciprocal rank, but cost critical recall (0.921 → 0.895). On the market-dilution question it retrieved the case on one side and dropped the case on the other.

It was switched off on that basis and then switched back on, because the reasoning was too quick. The delta is one case out of roughly thirty-eight — inside the noise band above. And both this result and BM25's win over dense retrieval have a simpler explanation than “the technique does not work”: seventeen of twenty-eight opinions arethe doctrinal ancestors, so “HyDE drifts toward the ancestors” largely means “drifts toward most of the corpus.” Both techniques earn their keep by bridging across breadth this corpus does not yet have. Widening it and re-running the grid is the real experiment.

5b. A corpus that can grade itself is not a test

The first version of this corpus was all fair use — 28 opinions arguing the same doctrine. That is a problem for evaluation, not just for coverage: when every document is about the same thing, high retrieval scores prove nothing, because everything is plausibly relevant. A retriever that returns “a fair-use case” for a fair-use question looks perfect whether or not it actually understood the question.

So the corpus was widened in two steps. First, six unrelated AI cases as a small control group. Then a full expansion to seventeen non-fair-use opinions spanning nine areas of law — algorithmic insurance denials, AI voice cloning, data scraping and the CFAA, biometric privacy, algorithmic hiring, chatbot product liability, algorithmic price-fixing, patent inventorship, and AI authorship. That took the corpus to 49 opinions and the golden set to 34 questions. Several of the new domains ship with their own pre-AI ancestor, so the citation graph grows real parallel lineages (Midler and Waits behind the voice-cloning case; Van Buren behind the scraping case), not just isolated outliers. Across cases the mean embedding similarity fell from 0.24 to 0.21 while within-case similarity held near 0.49 — more separated from each other, still coherent inside.

The retriever discriminates. Every cross-domain question returns its own domain — hiring pulls the Workday case first, chatbot harm the Character.AI case, price-fixing the RealPage case — and, the subtler test, a copyright bridgestill bridges: asked about ingesting copyrighted works to build a detection system, the retriever surfaces Turnitin's plagiarism-scanner case alongside the AI-training cases rather than rejecting it. A fair-use question now rejects sixteen of the seventeen distractors.

One does leak, and it is worth showing rather than hiding. Asked whether training a language model on books is fair use, the retriever pulls in an AI voice-cloning case at rank six — because that case is itself about training a model on copyrighted material and preemption, so the surface language genuinely overlaps. The evaluation harness caught it only because the golden set names that case forbidden for that question. And the fix is the technique §5 left in dispute: rewriting the query as hypothetical judicial prose (HyDE) pulls it back toward fair-use vocabulary, drops the leak entirely, and posts the best ranking of any configuration. HyDE looked marginal on a corpus that was mostly one doctrine; on a corpus with real distractors, it earns its keep.

Re-running the whole grid also settled the question §5 left open. The gap between BM25 and dense retrieval on critical recall narrowed from 0.106 to 0.030, and dense now ranks the first correct case higher than BM25 does; on the data-scraping question, dense retrieves the controlling cases that exact-term BM25 misses entirely. Fusion, which diluted the sparse signal on the narrow corpus, now matches BM25 on critical recall and beats it on overall recall. None of this dethrones BM25 — it still edges critical recall on a corpus this size — but the case for semantic retrieval is now evidence rather than conjecture: it pays off exactly as the corpus stops being about a single thing.

6. The agentic loop

After retrieving, the model is asked a narrow question: is this enough, or does it turn on a precedent you have not been given? If the latter, it follows an edge in the citation graph — built from the opinions themselves — and retrieves again, bounded at three hops. Asked what authority Ross relied on, it retrieves Ross, judges itself insufficient, follows the citation to Warhol, then stops.

The prompt pushes toward stopping. An answer grounded in a few directly relevant opinions beats one padded with tangential authority, and an agent that always finds a reason to keep going is just an expensive way to add noise.

7. Why it runs where it runs

The pipeline runs in a dedicated Lambda behind a Function URL in RESPONSE_STREAMmode, not in the site's Next.js runtime. That was not a preference — it was measured. AWS Amplify's SSR platform buffers server-sent events rather than streaming them, and its CDN terminates the origin connection at about thirty seconds. A probe confirmed both: 94 SSE events from an existing endpoint arrived simultaneously at 12.4 seconds, and a longer response failed outright at 30.5. A Function URL delivers frames incrementally from 0.4 seconds out past 90.

The corpus and its vectors are ~6 MB, so they ship inside the deployment package. An earlier design put them in S3; at this size that bought a round trip, an IAM policy, and a failure mode, and nothing else.

8. What is deliberately absent

No Westlaw or Lexis headnotes, no Shepard's or KeyCite treatment signals. Those are proprietary editorial layers, and their absence is a real limitation: the citation graph here records that one case cites another, but not whether it followed, distinguished, or criticised it. Inferring that from the surrounding text is an open problem and a natural next step.

The exclusion is also the subject matter. Thomson Reuters v. Ross Intelligence — in this corpus — held that copying 2,243 Westlaw headnotes to build training data was infringement and not fair use.

9. Stack

Python (uv) for the offline corpus and evaluation harness · TypeScript on AWS Lambda with a streaming Function URL · Amazon Bedrock — Claude Sonnet 4.6 for analysis, assessment, and synthesis; Titan Text Embeddings v2; Cohere Rerank 3.5 · hand-written BM25 and Reciprocal Rank Fusion · AWS CDK · Next.js on Amplify · sources: CourtListener (Free Law Project), govinfo, and the courts' own opinion archives.