AI ENGINEER · LLM SYSTEMS & EVALS · SYDNEY
Andrew Xia
The AI drafts. The rules decide.
I build AI systems with a deterministic core — auditable, evaluated, calm. My flagship reads air-cargo paperwork with an LLM, then lets a rule engine — never the model — issue every compliance verdict.
96.3% STRICT FIELD ACCURACY511-FIELD GROUND-TRUTH CORPUS57 DETERMINISTIC RULESSHA-256 AUDIT TRAIL
Verdict — shipment pre-check, replayed
An air-cargo compliance workspace I built solo for a Sydney freight forwarder. Pick a case and watch the real pipeline's behaviour: an LLM extracts the paperwork, 57 deterministic rules issue the verdict, and every flag carries a receipt.
SYNTHETIC DATA · real pipeline, mock documents
- Upload
- Classify
- Extract
- Validate
- Score
- Draft
View the eval report
Extraction eval — production extractor vs hand-labelled ground truth
measured 2026-06 · corpus v1
| Corpus | 85 documents · 511 labelled fields · 17 document sets · clean digital English PDFs · single forwarder's document mix |
| Strict exact-match accuracy | 96.3% |
| Semantic accuracy (format/granularity tolerant) | 99.0% |
| Extraction failures | 0 · True misses: 2 (both dissected in the write-up) |
| Method | Field-by-field diff, strict + lenient matchers; identifiers compared separator-stripped; every mismatch printed for human review |
Numbers cover clean digital English PDFs from one forwarder’s document mix. Scanned/photographed documents and other layouts are outside this corpus — measured honesty beats round numbers.
How this demo works
This demo replays pre-recorded outputs of the real pipeline run against synthetic shipping documents — same extraction schemas, same rule engine, same verdict logic. Nothing here calls a network: no LLM API, no tracking, no data leaving your browser. Document values are invented; rule IDs and thresholds are simplified for display. The full system runs privately for the client; this replay is the honest public slice of it.
Demo recording: 2026-06-12 · rule snapshot a3f81c92…
Case study — Verdict
Air-cargo compliance pre-check for a Sydney freight forwarder. Solo build, April 2026 – present.
The problem
Freight operators triage shipment paperwork by hand: airway bills, invoices, packing lists, dangerous-goods declarations — checked against biosecurity windows, prohibited-imports rules and air-transport restrictions, under deadline. Misses are expensive: held cargo, re-lodgement, penalties. The people doing this work don’t need an oracle; they need a tireless pre-checker that shows its working.
The constraint that shaped everything
A compliance verdict you can’t reproduce is a liability. So the system is built on one rule: the model never decides. The LLM extracts fields, classifies documents and drafts follow-ups. Every pass / warning / blocked verdict comes from a deterministic rule engine — pure functions over validated fields, versioned with SHA-256 snapshots, so any verdict can be replayed bit-for-bit later.
Architecture
What decides what
| Concern | Deterministic rules | LLM | Human |
|---|---|---|---|
| Compliance verdicts (pass/warn/block) | ● | — | review queue |
| Field extraction from documents | schema validation | ● | corrections UI |
| Document classification | keyword heuristic first | fallback on low confidence | — |
| Follow-up emails | template engine | drafting assist | always reviewed |
| Risk scoring | ● | — | — |
Built & measured
In plain terms: Verdict reads the shipping paperwork, checks it against the real biosecurity, dangerous-goods and customs rulebooks, and shows its working — with a copyable receipt — on every call. The specifics, for engineers:
9 document types · 57 rules across 3 modules, encoding ISPM-15, DAFF BMSB seasonal measures, IATA DGR (65th ed.) with a ~190-entry UN air whitelist, Prohibited Imports Regulations, WCO HS and Incoterms 2020 · 15 authenticated REST endpoints (JWT, RBAC, IDOR protection, audit logging) · bilingual PDF + OCR ingestion (English/Chinese) · table-driven test suite (~1,000 cases incl. 8 adversarial rounds) gated by CI on test, type-check, build and lint · extraction measured at 96.3% strict / 99.0% semantic on the 511-field corpus above.
Data boundary & provider choice
The LLM layer is a hand-rolled, OpenAI-compatible client behind a provider abstraction — currently DeepSeek (chosen on cost during development; a full 85-document eval run costs cents) with automatic failover to MiniMax. Synthetic and anonymised data only during development; no client PII in prompts. Because the abstraction is the contract, swapping to a sovereignty-compliant provider — Azure OpenAI (Sydney), AWS Bedrock, or a local model — is a configuration change, not a rewrite. The deterministic core narrows the blast radius either way: whatever the provider, it only ever drafts and extracts. It never decides.
Honest limitations
Mock-data demo above; the real system runs privately for one client and is not a public product. Accuracy is measured on clean digital English PDFs from that client’s document mix — not scans or photos. Rules encode my reading of public regulations, verified by tests, not by a licensed broker. No production-traffic claims.
Roadmap (designed, not built)
- designed HS-code classification with tariff retrieval (pgvector on the existing Postgres)
- designed compliance copilot for explain-why queries (read-only)
- designed email-thread intake orchestration
Listed as design intentions — nothing here is implemented, and saying so is the point.
Status verified 2026-06-12
How I build with AI
Verdict was built with an AI pair — governed like an untrusted worker, not trusted like a colleague.
- Spec first.
Five source-of-truth documents and a 17-stage prompt pipeline define every change before the model sees it.
- Guard hooks.
Custom pre-execution hooks block dangerous commands and out-of-contract edits before they happen.
- Read-only reviewers.
Separate subagents audit diffs against spec — they can flag, never write.
- Gates, not vibes.
Pre-commit runs type-check, tests and build; CI re-runs everything. Green is the only accepted evidence.
- Failure-mode dictionary.
Nine documented incident patterns — including the model fabricating build evidence — each mapped to the safeguard that now catches it.