Country of Origin - Search Agent
A proof of concept for agentic search: instead of one embedding lookup and a hopeful summary, the agent plans its own queries, reads specific PDF pages, checks its own claims against what it actually read — and abstains when the corpus can't support an answer.
![The agent answering a Swedish question about Syrian passport cost and validity — its own search queries listed above, the answer citing sources [S1][S2][S3]](/images/coi-showcase-hero.png)
A proof of concept, not a product. Not affiliated with, endorsed by, or built for any government authority or agency, and nothing it produces is legal advice or a case decision. The sample corpus is public material only — COI reports published openly by the EUAA, the UN, Landinfo (NO), the Danish Immigration Service, the UK Home Office, the Netherlands MFA and similar bodies. No internal, restricted or case-related material was used anywhere in the project.
Why This Exists
The real subject here isn't migration casework — it's how well a model designs. Anthropic's Fable 5 was handed a short problem statement and nothing else, and produced the full architecture and a phased implementation plan (DESIGN.md and PLAN.md). Opus 4.8 then implemented them. Both documents were written first and have never been retrofitted to the code, so they read as what they are: a model's unaided design work, and the result of building it.
The domain was picked because it's unforgiving. Country-of-origin information (COI) is what migration caseworkers consult to assess conditions in an applicant's home country: dense, multilingual reports, thick with footnotes, where a fact's truth value flips depending on the publication date — everything about Syria before and after Assad's fall in December 2024 reads differently. A retrieval system that bluffs in that setting is worse than no system at all.
This is the entire prompt the design step was given:
Everything that isn't in that paragraph — hybrid BM25 + dense retrieval with RRF fusion, the LLM reranker, section-aware chunking, the groundedness judge and the abstain path, the verification badge, the golden eval set — came out of the design step, not the prompt.
What It Looks Like
Questions go in in Swedish, answers always come back in Swedish with per-sentence citations. Each [S#] marker is clickable and opens the source PDF at the exact cited page, so a caseworker verifies rather than trusts.
Click any screenshot to view it full size.
Architecture
TypeScript end to end. A CLI ingests PDFs into a single SQLite file that holds both the keyword index (FTS5) and the vectors (sqlite-vec); a Next.js app serves the agent, the streaming chat UI, and the PDFs themselves. Nothing leaves the machine except the LLM calls.
Ingestion pipeline
Run as pnpm ingest <folder>, idempotent per file hash so re-running only picks up new or changed documents.
Extract
Per-page text items with font size and position via pdfjs. Reading order, paragraphs and headings are reconstructed from font-size outliers and numbering patterns like 2.1.1.
Clean
Strips the furniture: repeated headers and footers (detected by cross-page repetition), page numbers, tables of contents, and boilerplate disclaimer pages.
Separate footnotes
Footnote-zone text is stored per page and linked to it rather than embedded mid-paragraph — but stays searchable, because footnotes carry the source names and dates caseworkers ask about.
Chunk
600–900 tokens with 15% overlap, never crossing a section boundary. Each chunk records its section_path, page range and detected language.
Enrich (contextual retrieval)
A cheap model writes a one-line context prefix per chunk — organisation, report, date, section, topic. Prepended for indexing, never shown in answers. This is what fixes the "orphan chunk" problem in section-heavy reports.
Document metadata
An LLM pass over the first few pages pulls out title, publishing organisation, publication date, reference period, language and report number — injected into every citation.
Embed
Multilingual embeddings, model and dimensions from .env. The store records which model produced each vector, so changing models forces a re-embed instead of silently degrading retrieval.
Ingestion is a quality gate, not a loading screen. Every document gets a report — pages parsed, chunks produced, language mix, extraction warnings. A document that fails extraction is flagged, never silently skipped.
Retrieval — the Quality Core
The hard part is cross-lingual: Swedish questions against a mostly-English corpus. Three mechanisms stack on top of each other.
1 · Query understanding
One cheap LLM call turns the Swedish question into a normalized query, an English translation with domain-term expansion ("värnplikt" → "military service, conscription"), optional sub-queries, and any stated filters.
2 · Hybrid search + RRF
BM25 (FTS5 with a trigram tokenizer, so Scandinavian compounds match) and dense cosine search run for both language variants, then merge with Reciprocal Rank Fusion. Embeddings catch sv→en meaning; BM25 catches exact terminology, names and report numbers.
3 · LLM rerank
The top ~40 fused candidates get listwise-reranked against the original Swedish question; 8–12 survive. The reranker's scores double as the input to the abstention gate.
The agentic loop on top
Retrieval isn't a single stuffed prompt — it's exposed to the answering model as tools, so it can reformulate a query, drill into one document, or check a newer report for contradictions before committing to an answer.
searchFull hybrid + rerank pipeline
read_pagesPull whole pages when a chunk is truncated
list_documentsCorpus overview with orgs and dates
This is what lifts genuinely multi-hop questions — "How do the EUAA's and Landinfo's assessments of the security situation in Latakia differ?" — above what single-shot RAG can do. Every retrieved excerpt carries its document's date, and the system prompt is explicit about it: prefer the newest source on conflict, always state the information date, and flag when the freshest available material is older than the question implies.
Correctness and Abstention
Three independent gates, any one of which can send the request down the abstain path.
Retrieval gate — before generation
If the reranker's best scores fall below a calibrated threshold, generation is skipped entirely. Cheap, and the model never even sees the temptation to improvise.
Self-assessment — during generation
"I cannot answer this reliably" is a first-class output format, not a failure mode. Partial answers are allowed, but must delimit what is and isn't supported.
Groundedness judge — after generation
A second model receives each claim alongside the excerpt it cites and verifies entailment. Unsupported claims trigger one regeneration attempt with the judge's feedback; if they survive that, the claim is dropped or the whole answer degrades to abstention.
The UI shows the verdict rather than hiding it. An answer that is largely supported but contains one claim the judge couldn't tie to a cited extract renders as nedgraderad — downgraded — with the unsupported sentence called out instead of quietly deleted.
When the corpus can't support a reliable answer at all, the agent abstains — and the abstention is informative rather than a dead end. It distinguishes three failure modes (not covered by the corpus, sources conflict too much, information likely outdated) and offers the closest related material instead of guessing.
Knowing What's Actually in There
A corpus page lists every ingested document with its metadata and extraction health — because "the agent didn't find it" and "the document never parsed" are very different problems, and a user deserves to be able to tell them apart.
Evaluation Harness
Built early rather than last. A golden set of ~40 Swedish questions written against the sample corpus, in five classes — and every retrieval or prompt change gets validated against it. This harness is the definition of "high quality" for the project.
| Question class | What it tests |
|---|---|
| Single-document lookup | The answer sits on a known page of a known report |
| Cross-document synthesis | e.g. military service across EUAA + Landinfo + DIS reports |
| Swedish-source questions | Retrieval into the Swedish-language reports in the corpus |
| Unanswerable | Plausible COI questions the corpus doesn't cover — must abstain |
| Temporal | Pre- and post-December-2024 distinctions |
Retrieval recall @ k=12
Answer faithfulness (LLM judge)
Abstention on the unanswerable class
Answers in Swedish
Running It
Models are configured per role in .env and are independently swappable — a strong model for answering and judging, a fast one for query understanding, reranking and enrichment.
The CLI prints the Swedish answer, the verification status (verifierad ✓ / nedgraderad / avstår), the cited sources and the full tool trace — the same information the web UI renders, minus the PDF panel.
Technology Stack
The whole knowledge base is one SQLite file — index, vectors, metadata and eval set together. No vector database, no external infrastructure, nothing to provision. For an evaluation or pilot setting, being able to copy the corpus as a single file is worth more than any ANN index would buy at this scale.