Most RAG tutorials hand you a chunker. RecursiveCharacterTextSplitter, TokenTextSplitter, a LangChain convenience wrapper. You set a size, maybe an overlap, and move on. This post is about what happens when that isn’t enough, and how to build something better in ~150 lines of pure, dependency-free Go.
Consider any long-form audio recording like a podcast, a lecture series, or an interview archive. In multilingual regions, it’s common for a speaker to switch freely between two languages, sometimes mid-sentence. A bilingual lecture transcript might look like this:
"Iska matlab hai, the observer and the observed are not two things."
"Jab tum sach mein dekh rahe ho, toh observer hi observation ban jata hai."
"This is what most mindfulness teachers miss when they talk about awareness."
"Aur yahi baat hai jo Western psychology abhi begin kar rahi hai samajhna."
Now imagine splitting that with chunk_size=512, overlap=50. You will get cuts in the middle of thoughts, leaving one half of a complete argument on each side of a boundary. Instead of coherent ideas, you get syntactically broken, incomplete fragments that degrade embedding quality and confuse the retriever.
The core problem with fixed-size chunking is that it is blind to meaning. It knows characters and tokens. It does not know when the speaker finishes one idea and starts another.
B -->|semantic break| C["Topic: Practice & technique\n(atoms 10–14)"]
Chunks that track ideas, not byte offsets.
What is an “atom”?
An atom is our smallest indivisible unit of text—typically a single complete sentence or a discrete speech utterance. By operating on whole atoms, we ensure a sentence is never chopped in half.
For example, in the bilingual transcript above:
Atom 0:"Iska matlab hai, the observer and the observed are not two things."
Atom 1:"Jab tum sach mein dekh rahe ho, toh observer hi observation ban jata hai."
No matter the chunk size limits, these atoms remain fully intact. They may be grouped together or separated at a semantic boundary, but they will never be split down the middle.
U["usecase.go\n(pure business logic, chunkBlock)"]
R["repository/\n(store, mlclient)"]
T --> H --> U --> R
style U fill:#1e3a5f,color:#fff
style R fill:#1a2e1a,color:#fff
By keeping the pure business logic in the usecase layer, the semantic chunking algorithm can be tested independently of Postgres or the embedding model.
Each atom (a single sentence or distinct utterance) is embedded via BGE-M3, a multilingual 1024-dimensional dense encoder that handles Hindi, English, and Hinglish in the same vector space.
BGE-M3 is called via a Python FastAPI sidecar (the pyworker). Go owns all orchestration and logic; Python is confined strictly to GPU-bound inference. The HTTP contract is minimal: a list of strings in, a list of float32 slices out.
sequenceDiagram
participant Go as Go usecase
participant Worker as pyworker /embed
participant BGE as BGE-M3 (GPU)
Go->>Worker: POST /embed {texts: [...N atoms...], normalize: true}
Raw per-sentence embeddings are noisy. A single noisy or off-topic sentence can spike a distance value falsely.
Why smooth?
When people speak naturally, they often throw in brief, off-topic fillers (e.g., “Hold on, let me take a sip of water”). If we don’t smooth the vectors, this single sentence creates a massive distance “spike,” tricking the algorithm into cutting the chunk in half. By averaging it with its neighbors, we pull that outlier back toward the main topic and prevent a false cut.
We reduce this noise by replacing each atom’s vector with the mean of itself and its ±1 neighbors, then re-normalizing to unit length:
A3["Noisy document\n(many small spikes)"] -->|"too many cuts"| A4["❌ Tiny fragments"]
end
subgraph "Percentile threshold = p92"
B1["Short document"] -->|"top 8% distances"| B2["✅ Adapts to file"]
B3["Noisy document"] -->|"top 8% distances"| B4["✅ Adapts to file"]
end
With valley_percentile: 0.92 (from config.yaml), exactly the top 8% of cosine-distance values in a given block become cut points. A shorter, lower-variance block naturally gets fewer cuts; a longer, denser block gets more cuts automatically, without any per-file tuning.
Valley detection proposes cuts. Size guards correct them. The two-pass precedence rule is: merge undersize first, then split oversize. This ordering matters because you never want to split something you are about to merge.
When a span is too large, we split at the strongest internal valley within a sensible size band. If no valley exists in the band, we fall back to the target word count:
Valley["Strongest valley at atom 2-to-3, distance=0.9"]
Split["Split: [0..2] + [3..5], 150w + 150w"]
Again["Still oversize -> recurse"]
Final["[0..1] + [2..2] + [3..4] + [5..5]"]
Oversize --> Valley --> Split --> Again --> Final
The documented whole-sentence exception: a single atom (one sentence) that exceeds maxWords is returned as-is. It cannot be split further without breaking sentence integrity.
Multiple chunk_config values coexist in the database under the same source text. You can run a parameter sweep (default vs. small) without re-embedding the source, since the per-sentence embeddings are stable; only the chunking groupings change.
Each stage is independently re-runnable. The contract for the chunk stage: delete existing chunks for this (file_id, chunk_config) pair, then regenerate. Source text segments are never touched.
CREATEINDEXchunks_bge_hnswON chunks USING hnsw (bge_vec halfvec_ip_ops);
CREATEINDEXchunks_gemini_hnswON chunks USING hnsw (gemini_vec halfvec_ip_ops);
halfvec (16-bit float) instead of vector (32-bit) halves storage and index size with negligible precision loss for normalized retrieval. halfvec_ip_ops (inner product) is equivalent to cosine similarity when vectors are L2-normalized — which both BGE-M3 and Gemini vectors are enforced to be before write.
graph LR
BGE["BGE-M3\n1024-d float32\nasserted unit-norm"] -->|"UPDATE chunks SET bge_vec"| PG["chunks table\nhalfvec(1024)\nHNSW index"]
Gemini["gemini-embedding-001\n3072-d float32\nnormalized defensively"] -->|"UPDATE chunks SET gemini_vec"| PG2["chunks table\nhalfvec(3072)\nHNSW index"]
By designing your core algorithm (valley detection and size guards) as pure logic with no I/O or database dependencies, testing becomes incredibly easy. You don’t need mocks, test databases, or running API workers. You just pass in arrays of numbers and assert the exact output.
For example, testing the valley detection logic is as simple as:
Percentile thresholding beats absolute thresholds. Your threshold adapts to each file’s variance distribution automatically without per-file tuning.
Smooth before computing distances. A single noisy sentence creates a false spike; ±1 neighbor averaging removes most of them cheaply.
Merge before split in size guards. Merging can create an oversize span — which is then handled. The reverse causes subtler bugs.
Pure-logic separation is not just an architecture virtue. Your most complex algorithm becomes testable in five lines, portable, and auditable without any framework dependency.
halfvec in pgvector is the right default for normalized vectors: halved storage, identical retrieval semantics (inner product equals cosine for unit vectors), HNSW indexable on vectors that would otherwise be too wide.
Design stub mode in from day one. A Python embedding sidecar with a stub fallback (PYWORKER_STUB=1) makes local development as fast as any pure-Go project. Do not bolt it on later.
Premature optimization in vector storage is a silent bug. It doesn’t crash. It just quietly makes your AI dumber, and you won’t notice until a user does.
Choosing not to optimize is a real architectural decision. It’s not laziness. Sometimes it’s the right call. But two things can independently force you off it — dataset size, and your vector DB’s own constraints.
100,000 chunks, roughly a corpus of 500 books, costs 400 MB unoptimized. That fits in a db.t3.medium. Memory alone isn’t a reason to optimize until you’re well past 1M chunks.
When you hit the pgvector HNSW wall with a 3,072-dim model, you actually have two exits depending on whether your model supports MRL:
halfvec (always available): keep all 3,072 dimensions, drop float precision to 16-bit. The vector shape is intact, just less decimal resolution. Works with any model.
Matryoshka truncation to under 2,000 dims (MRL models only): keep full 32-bit floats, but trim dimensions down to e.g. 768. Now you’re back under the HNSW limit with no precision loss at all. Only valid for gemini-embedding-002 or OpenAI’s text-embedding-3 family.
So the HNSW block isn’t a “use halfvec, end of story.” It’s a forcing function that eliminates “do nothing,” but it still leaves you a choice between losing depth (halfvec) or losing width (Matryoshka) — the same tradeoff from sections 4 and 5, just now driven by the DB constraint rather than memory.
Matryoshka Representation Learning (MRL) is a training technique where the model packs its most important, broadest semantic concepts into the earliest dimensions, and pushes fine-grained nuance toward the later dimensions.
Think of it like a Russian nesting doll. The first doll gives you the rough shape. The inner dolls add detail.
Because the dimensions are sorted by importance, you can literally cut off the tail end. Truncating 3,072 to 768 dimensions saves 75% on storage and index size, with acceptable accuracy loss for broad queries.
This only works if the model was trained with MRL. Apply truncation to a model that wasn’t, and you’re randomly deleting 25% of its semantic representation. Accuracy will collapse, and it won’t be obvious why.
flowchart LR
Q["Apply truncation to model X"]
Q -->|"Model trained with MRL"| Safe["Safe\nDimensions are importance-sorted\nTail is genuinely less critical"]
Q -->|"Standard model, no MRL"| Danger["DANGEROUS\nDimensions are not sorted\nYou are randomly deleting 25% of meaning"]
This one is simpler to reason about. Instead of cutting dimensions, we keep all of them and just lower the precision of each number. A 32-bit float like 0.12345678 becomes a 16-bit float like 0.1234. Same shape, less storage.
flowchart LR
F32["32-bit float\n4 bytes per dimension\ne.g. 0.12345678"]
F16["16-bit float halfvec\n2 bytes per dimension\ne.g. 0.1234"]
Here’s something I didn’t know until I hit it: pgvector’s standard vector type has a hard limit on HNSW indexes.
schema.sql
-- Standard vector: HNSW fails at > 2000 dims
CREATE INDEX ON chunks USING hnsw (gemini_vec vector_ip_ops); -- ERROR
-- halfvec: HNSW works at any supported dimension
CREATE INDEX ON chunks USING hnsw (gemini_vec halfvec_ip_ops); -- Works
If you’re using gemini-embedding-002 (3,072 dims) with a plain vector column, you can’t build an HNSW index at all. Every query falls back to a sequential scan. halfvec isn’t optional here. It’s the only path that works.
flowchart LR
DIM["gemini-embedding-002\n3072 dimensions"]
DIM -->|"store as vector"| FAIL["vector(3072)\nHNSW index: BLOCKED\nFallback: sequential scan\nQuery latency: O(n)"]
DIM -->|"store as halfvec"| OK["halfvec(3072)\nHNSW index: WORKS\nQuery latency: O(log n)"]
Knowing which strategy to pick is one thing. Knowing what you’re paying for each option is what makes the argument in a planning doc or a cloud budget review.
The HNSW index roughly doubles your raw vector storage in RAM. It has to — the graph structure that makes approximate nearest-neighbour search fast needs to live in memory too.
How strategy changes your RAM at the same corpus size
For managed PostgreSQL with pgvector (AWS RDS, GCP Cloud SQL), the instance must hold the full HNSW index in RAM. If it can’t, queries fall back to disk. The instance tier is the cost you’re committing to.
To make it concrete: 1M chunks at 3,072-dim with no optimization needs ~24 GB, which lands you on a ~$380/month AWS instance. Switch to halfvec and the same corpus fits on the $190/month tier. Switch to Matryoshka at 768 dims and it fits comfortably within 8 GB — the cheapest tier available.
That’s the real cost of the choice. Not just bytes saved, but the billing tier your database has to live on.
The eval step is in the tree on purpose. You shouldn’t reach “use Matryoshka” without running a validation pass first. That’s the difference between a decision and a guess.
The pgvector HNSW limit catches people off-guard. If you pick gemini-embedding-002 and skip this check, you’ll be doing sequential scans in production and wondering why queries are slow.
This is the variable I see underestimated most often, and it’s the one that actually hurts people in production.
The same optimization that’s perfectly safe for an e-commerce product catalog will quietly destroy retrieval quality for a specialized knowledge corpus. Here’s why:
In a narrow domain like the Vedanta corpus above, Matryoshka groups all paragraphs about “awareness” tightly together in the first 768 dimensions. A query about a specific philosophical distinction will pull back whichever paragraph has the most surface-level overlap with the query text, not the semantically correct one.
This isn’t an accuracy problem. It’s a precision problem. Recall stays high; the system retrieves something that looks relevant. Precision collapses; it’s just the wrong thing.
Most teams don’t catch vector optimization errors before production. Here’s the actual pattern:
sequenceDiagram
participant Eng as Engineer
participant DB as pgvector
participant User as User
Eng->>DB: Truncate 3072 to 768 dims, redeploy
Eng->>Eng: Run 3 manual test queries
Eng->>Eng: "Looks fine to me"
Eng->>DB: Ship to production
User->>DB: "What is the difference between sakshi and turiya?"
DB->>User: Returns paragraph about generic mindfulness awareness
User->>User: "This AI doesn't understand my question"
User->>User: Stops using the product
Note over Eng,User: No error. No log. No alert. Just silent accuracy degradation.
The problem with “run a few queries and check” is that you’re testing with broad queries. Those are exactly the queries where truncation works fine. The nuanced ones that break are the ones you didn’t think to test.
Volume: At least 50 per domain. 200+ if your corpus is specialized.
Coverage: Mix broad queries (“What is mindfulness?”) with precise ones (“What distinguishes turiya from deep sleep in Kashmir Shaivism?”). The precise ones are what expose truncation failures.
Ground truth: Not “probably this chunk.” The exact chunk ID, verified by someone who actually knows the domain.
If truncating drops your top-5 retrieval accuracy from 95% to 60%, the evidence is sitting right in your database. The memory savings aren’t worth the lobotomy. But if halfvec matches raw accuracy within 2%, you have proof the optimization is free. Take it.
The eval loop turns an architectural guess into a decision you can stand behind.
Optimize wisely, measure precisely. Sometimes the best algorithm is no algorithm at all.
In the AI coding era, I spin up a lot of Go projects. Some are experiments, some reach production. Every time I start a new one with a coding agent, I find myself explaining the same structural decisions over and over — in an unstructured, inconsistent way. That friction is what this post is about. The four-layer structure described here is my answer: a lightweight, experience-backed Go project shape that I have not found a better replacement for. This post is the structured version of what I tell every coding agent when starting a Go service — so you can use it the same way.
Go is intentionally permissive about project layout. Unlike Spring Boot, Django, or Rails, there is no framework that nudges you into a shape. The compiler enforces import cycles and nothing else. Everything beyond that is convention.
This is a feature. It is also a foot-gun.
The permissiveness feels great at the start. You write a main.go, add a handler, wire a database connection, ship it. Then you add a second handler. Then a third external dependency. Suddenly you are making a different decision every time you add a file: where does this validation go? Is this a handler responsibility or a usecase responsibility? Why is my business logic importing pgx directly?
The cost of a bad structure is invisible early and expensive late. It shows up as:
“I can’t test this without spinning up Postgres”
“I don’t know where to add this validation”
“This PR touched the handler, the database layer, and the config — why?”
“The new engineer has been reading the codebase for two days and still doesn’t know where to look”
But there is a second, more modern reason to care about structure: coding agents.
When you work with AI coding assistants on multiple projects, you find yourself repeating the same architectural decisions in an ad-hoc, unstructured way — every new repo, every new session. “Put the business logic here, don’t let the handler call the database, declare the interface in the usecase, not the repository…” You say the same things differently every time, and the results vary.
A well-defined structure solves both problems at once. For humans, it creates familiarity and navigability. For agents, it becomes a precise, repeatable scaffold you can hand over in a single prompt. You make the decision once, encode it in a framework, and stop re-explaining it forever.
The Go community has converged on one thing for the outer shell: cmd/ for entrypoints and internal/ for private application code. What lives insideinternal/ is still up for debate. This post proposes a concrete inner shape — one that is lightweight, familiar once you learn it, and extensible by design.
The core idea: make the decision once. Encode it. Then stop deciding.
The structure I use is a four-layer model with explicit, non-overlapping responsibilities. It is inspired by hexagonal architecture, but stripped of its jargon and its boilerplate.
Each layer has one job. Here is what it owns, and what it is explicitly forbidden from touching:
Layer
Owns
Never touches
cmd/
Config load, transport wiring
Business logic, repo imports
Transport
DI composition, cobra command or HTTP router setup
Business rules, direct I/O
Handler
Input parsing, validation, idempotency pre-check
Database calls, HTTP clients
Usecase
All business logic + interface declarations
http.Request, SQL drivers, file paths
Repository
Concrete I/O implementation
Business logic of any kind
Each component in your service gets its own package with these four files:
internal/app/<component>/
├── transport.go ← DI wiring
├── handler.go ← input normalization + validation
├── usecase.go ← interfaces + business logic
└── model.go ← domain types
The repository implementations live separately:
internal/repository/
├── store/ ← Postgres repos
├── httpclient/ ← external HTTP APIs
└── ...
The directory tree is the architecture. You should be able to look at internal/ and immediately understand how many components exist and what external systems they talk to.
Let me make this concrete. Consider an ingest component that reads a manifest file, streams audio from a cloud bucket, and records the result in a database.
infra := repository.NewInfra(cfg) // opens DB pool, HTTP clients — once, here
ingestT := ingest.NewTransport(cfg, infra)
root :=&cobra.Command{Use: "pipeline"}
root.AddCommand(ingestT.Command())
root.Execute()
}
main.go does three things: load config, build infra, register commands. That is it. It never imports a repository package directly. It never contains business logic. If main.go is growing, something is wrong.
cmd.Flags().String("manifest", "", "path to manifest file")
return cmd
}
Transport is the only place that knows both the usecase and the concrete repository implementations. This is the composition root — the one place where the wiring lives.
inputs, err :=parseManifest(manifestPath) // raw string → []FileInput domain objects
if err !=nil {
return fmt.Errorf("invalid manifest: %w", err)
}
for _, input :=range inputs {
if err := h.usecase.Ingest(ctx, input); err !=nil {
return err
}
}
returnnil
}
The handler never calls a repository. Its only job is: parse raw input → validate → call usecase. Raw CLI strings and HTTP request bodies never cross into the usecase layer.
// pure business logic — no http.Request, no pgx, no file paths
}
This is the most important insight in the whole pattern. The usecase declares the interfaces it needs, not the repository. The repository satisfies them implicitly — Go’s structural typing means no implements keyword, no adapter boilerplate.
FileRepo never imports from app/ingest. It just happens to implement the methods that ingest.FileStore requires. Go’s compiler verifies this at the call site in transport.go — and nowhere else.
On day one, you point someone at internal/ and they immediately see two buckets: app/ (business components) and repository/ (infrastructure). Within a component they navigate by question:
I want to understand…
I look at…
What commands / endpoints exist
cmd/
How a component is wired together
<component>/transport.go
What inputs are accepted and validated
<component>/handler.go
What the service does (business rules)
<component>/usecase.go
What interfaces the usecase depends on
Top of <component>/usecase.go
Where data actually comes from / goes
repository/
Pure algorithmic logic (no I/O)
<component>/<algorithm>.go
A new engineer doesn’t read the whole codebase. They navigate by layer. If they want to understand the chunking algorithm, they open chunk/usecase.go and find the logic. If they want to know what Postgres tables are involved, they open repository/store/chunk_repo.go. They never need to hold both in their head at the same time.
This structure makes code reviews faster because layer violations are immediately visible:
A PR that modifies handler.go should never touch repository/. If it does, business logic has leaked into the wrong layer.
A PR that changes a usecase interface should have a corresponding change in transport.go (where the concrete type is passed). If it doesn’t, something won’t compile.
A PR that adds a new external dependency should add a new file in repository/, a new interface in the relevant usecase.go, and a new wiring line in transport.go. The change is mechanical and reviewable.
Import paths serve as a free architecture linter. If you see internal/repository/store importing from internal/app/ingest, that is visually, immediately wrong — no tooling required.
If a file is growing past 300 lines and it touches both I/O and business logic, the structure tells you to split it. Not as a style preference. As a layer violation.
Testability without infrastructure. The usecase declares interfaces. The interfaces can be mocked with a trivial struct. Business logic tests run in milliseconds with no database, no network, no running services. More importantly, I wrote pure-logic files for complex algorithms — chunking boundary detection, QA gating rules, retrieval evaluation metrics — with zero I/O. Those files are tested with go test ./... and nothing else. This is not theoretical. It saved hours when iterating on the chunking algorithm.
New component = familiar scaffold. When I added a sixth component to the pipeline, I created four predictably named files and one repository file. There was no architecture decision to make, no “where does this go?” conversation. I knew exactly what each file was for before I opened it.
One struct, N interfaces. Because Go’s interfaces are implicit, a single HTTP client struct passed to four different transports satisfies four different usecase interfaces with zero adapter code. No wrapper, no Adapter struct, no code generation. The compiler verifies it at the call site.
Import violations are self-evident. A repository package importing from an app package stands out in code review. No linter rule needed.
Upfront cost. For a service with one component and one external dependency, this structure is overkill. You are creating four files where one main.go would do. The structure pays for itself at the second component, not the first.
Data mapping overhead. You map raw inputs to domain types in the handler, and domain types to database rows in the repository. For simple flat structs this is trivial. For complex nested types it becomes tedious. This is the honest tax of keeping layers clean.
Interface proliferation. If three usecases all need GetUserByID, you declare three interfaces that look nearly identical. This is idiomatic Go — the Go standard library does it everywhere — but it surprises engineers coming from languages where interfaces live in a shared package.
Transport is an unusual name. Most architecture literature calls this the “delivery layer” or “adapter.” Naming it Transport and having it own dependency injection wiring is deliberate, but it takes some orientation for engineers who know hexagonal architecture from other contexts.
If you’ve read about hexagonal architecture (also called Ports and Adapters), this structure will look familiar. If you haven’t, I’ll explain it without the jargon.
The idea is simple: your business logic should be at the centre, isolated from everything that could change — the database, the HTTP framework, the CLI library. You define ports (interfaces) that describe what the business logic needs, and adapters (implementations) that plug into those ports from outside.
This structure implements that idea. Here’s the direct mapping:
style API fill:#0f172a,stroke:#475569,color:#94a3b8
The usecase defines the ports (its interfaces). The handler and repository are the adapters. The transport wires them together.
What makes this version lightweight:
No shared interfaces/ package. Interfaces live in the usecase file that consumes them. The package that uses an interface owns it — not the package that implements it.
No adapter wrapper structs. Go’s structural typing means your Postgres repo satisfies the FileStore interface without declaring implements FileStore anywhere. The compiler verifies it.
No DI container or code generation. Transport files are plain Go functions calling plain Go constructors.
No framework opinion. You can use cobra for CLI, chi or net/http for HTTP, or both. The usecase doesn’t know either exists.
The result is an architecture that covers the full value of hexagonal design — isolation, testability, replaceability — without its ceremonial cost.
The most practical test of a structure is: how hard is it to add something new?
Adding a new transport type (e.g., HTTP API alongside an existing CLI):
Create a new http_transport.go in the same component package. It builds the same usecase (same constructor, same dependencies) and wires it to an HTTP handler instead of a cobra command. The usecase and repository are completely unchanged.
// ingest/http_transport.go — added later, zero changes to usecase or repo
h := ingest.NewHTTPHandler(uc) // new handler for HTTP, same usecase
return&HTTPTransport{handler: h}
}
Swapping a database:
Implement the same usecase interface against the new database. Wire the new implementation in transport.go. The usecase file has zero changes, because it only knows about an interface.
Adding a new component:
Create a new internal/app/<component>/ package with the same four files. Add one line to cmd/main.go to wire it. Every engineer who has worked on any other component will immediately know how to navigate the new one.
Adding a new repository dependency to an existing usecase:
Declare a new interface in usecase.go. Add a new field to the usecase struct and constructor. Implement the interface in a repository file. Wire it in transport.go. Nothing else changes — and the compiler will tell you exactly what is missing.
Extension in this structure is additive and mechanical. You are filling in a known shape, not making architectural decisions.
This structure is not always the right answer. Here is an honest guide.
Use the full four-layer structure when:
Your service has ≥ 3 distinct processing components
It talks to ≥ 2 external systems (a database, an HTTP API, a storage bucket…)
Multiple engineers will work on it
You want to unit-test business logic without infrastructure
You expect to swap or extend infrastructure over time
Use a simplified variant (collapse transport + handler into main.go) when:
You have 1-2 components and a clear, stable domain
The team is one person building a personal tool
You’re in prototype mode and the shape of the domain isn’t clear yet
Write a single main.go and revisit when:
It’s a one-off script with a single purpose
The entire business logic fits in under 200 lines
No external dependency is likely to change
The pragmatic middle ground: Even on small projects, I now start with at least cmd/ + internal/ + a usecase.go file. If transport and handler are overkill today, I collapse them into main.go. When the second feature arrives, the migration is mechanical — move code, add files, no logic changes. The habit costs nothing and pays off every time a project grows.
style AY fill:#1e293b,stroke:#64748b,color:#94a3b8
Dependencies only flow inward. Outer layers know about inner layers. Inner layers know about nothing outside themselves.
If a repository package imports from an app package, business logic has leaked into infrastructure. If one app component imports from another, you have hidden coupling that will cause a cascade when either changes. If cmd/main.go imports a repository package directly, the composition root has leaked out of the transport layer.
These violations are architectural, not stylistic. The fix is to move code, not rename variables. Enforce this rule in code review. No linter required — import paths make violations visually obvious.
The prompt works because it describes the architecture as hard constraints, not suggestions. The LLM fills in your component names and dependency types while the structural decisions stay fixed. Every generated file will follow the same shape — and that shape will look familiar the next time you use it.
You can re-run it for each new component you add: replace the component list, keep the rules.
The generated scaffold is not a finished service. It is a map. Every file will tell you exactly what belongs in it, and every engineer who reads it will know where to navigate.
Structure is a decision you make once. Chaos is a decision you make every day.
The four-layer model — cmd/ → Transport → Handler → Usecase → Repository — covers the full value of hexagonal architecture without its ceremonial cost. It is lightweight because it uses Go’s implicit interfaces. It is extensible because every new component follows the same shape. It is readable because the directory tree is the architecture diagram.
The biggest win is not modularity, testability, or replaceability — though you get all of those. The biggest win is familiarity. Every component looks the same. A new engineer knows where to look. A code reviewer knows what a PR should touch. An extension is mechanical, not architectural.
Structure your service before it structures you. The decision checklist and prompt at the end of this post are your starting point.
From building a production RAG pipeline over an internal engineering corpus. One decision I made at ingest time turned out to pay for itself twice — as the strongest retrieval signal in the system, and as a free query-intent classifier. This is that decision, and nothing else.
The default RAG recipe is three lines: embed every document, embed the user’s question, return the nearest documents by cosine similarity. It works in demos, and it hides a mismatch that quietly caps its quality: a question and the document that answers it do not look alike.
A question is short and interrogative — “Why does the consumer crash on unspecified events?” The document that answers it is long and declarative: paragraphs about enum handling, a code block, a note about a dead-letter queue — and it may never contain the word “crash.” In embedding space they land near each other, but not as near as you’d want.
You can measure the gap. Take a corpus, generate natural questions, compare three ways:
Comparison
Typical cosine
Question ↔ the document body that answers it
0.69
Question ↔ a different question on the same topic
0.86
Question ↔ a question on an unrelated topic
0.53
Two questions about the same thing sit at 0.86. A question and its answering body sit at 0.69 — a ~0.17 gap, right in the band where ranking is decided. Every time you match a question against a body, you’re leaving that gap on the table.
So: if the tightest match is question-to-question, match the user’s question against questions. At query time you only have one — the user’s. Where do the rest come from? Ingest time.
When a document is ingested, an LLM writes down the questions it answers, each tagged with a category:
{"synthetic_queries": [
{"category": "diagnostic", "question": "Why do unspecified event types route to the dead-letter queue?"},
{"category": "implementation", "question": "How do I change the retry backoff for the batch consumer?"},
{"category": "conceptual", "question": "What is the difference between a batch handler and a stream handler?"}
]}
These synthetic questions are embedded and stored as a first-class retrieval target of their own — one row per question, each pointing back at its document. At query time, the user’s question is compared against them. You’re back in the 0.86 regime instead of the 0.69 one.
This is HyDE turned inside out. HyDE fixes the same asymmetry by generating a fake answer at query time — an LLM call on every request. Synthetic questions do the mirror image, once, offline: generate fake questions per document at ingest, embed them, store them. At query time there’s no extra LLM call — you just search a table.
One detail makes it rigorous: modern embedding models are dual-encoders with a task type. Embed document bodies as RETRIEVAL_DOCUMENT, synthetic questions and the runtime query as RETRIEVAL_QUERY. Comparing the query against the questions is then a clean query-vs-query match; comparing it against bodies is the asymmetric case the task types were built to reconcile. You embed the user’s question once and let it search both.
The payoff you didn’t plan for: free intent classification
Here’s where the one decision pays off a second time.
Different questions want different treatment. “Why is the consumer crashing right now?” wants the freshest runbook. “Why did we choose Postgres over DynamoDB?” wants the original design doc, however old. To treat them differently you have to know which kind of question was asked — and the usual way to find out is a query router: an extra LLM call at the top of the pipeline, costing latency and a per-query call, and a second place where labels drift.
You don’t need it, because you already classified at ingest time — every synthetic question carries a category from a small taxonomy: diagnostic, implementation, conceptual, navigational, temporal. So after the question search runs, you don’t just know which documents matched — you know which kinds of questions matched, and how strongly. Sum them, weighted by similarity:
For “why is the consumer crashing on unspecified events?”, the matches vote diagnostic 1.69, implementation 0.87, conceptual 0.71 — dominant intent diagnostic, inferred with zero LLM calls in under a millisecond. Trust it only when the winner clears a majority (top/total > 0.6); otherwise fall back to neutral. That’s the property a router can’t match — on a query the corpus can’t answer, inference says “I don’t know” and behaves like a plain hybrid retriever, instead of emitting a confident wrong label.
Query router
Free inference
Extra LLM call per query
yes (~200–500ms)
no
Cost per query
one LLM call
zero
Unanswerable query
confident wrong label
falls back to neutral
Source of truth for labels
router + consumers, can drift
one category column
The synthetic questions are, in effect, an inverted intent index: instead of asking “what kind of question is this?” you precomputed “what kinds of questions does each document answer?” — and the user’s question becomes whichever kinds it most resembles. That intent then does its job: it picks a recency half-life, so temporal answers decay in weeks while conceptual ones stay valid for years — same retriever, age applied the way each kind of question wants.
Two honest caveats. If the ingest LLM tags questions unevenly — say 80% implementation — every query starts looking implementation-heavy, and the intent signal degrades; you have to monitor the category distribution. And a query that matches few synthetic questions gets an unreliable vote; the confidence floor catches it, but “neutral” means you’ve lost the tuning for exactly the queries the corpus serves worst. The signal is only as good as the questions behind it.
The whole thing is one move with two payoffs: generate categorized synthetic questions per document at ingest, and embed them as a first-class retrieval target. It gives you the tightest retrieval signal available (question-to-question, ~0.86 vs ~0.69), and it hands you free query-intent classification because those questions carry their category.
The general lesson outlasts the trick: before adding a component to your hot query path — a router, a classifier, a second model call — check whether something you already compute at ingest time secretly contains the answer. Ingest is batched, retryable, and off the user’s critical path. The cheapest query-time work is the work you already did.
This post draws from real engineering experience building a production CI pipeline that uses LLMs as a core processing component inside an automated, incremental analysis system. The pipeline runs on every commit, makes structural decisions using LLMs, synthesizes structured output, and publishes incremental diffs to object storage. Everything in this post comes from actual incidents, debugging sessions, and architectural decisions made while fighting non-determinism at production scale.
A deterministic system, given the same inputs, always produces the same outputs. A Postgres query, a sorting algorithm, a hash function — these are deterministic. You can test them, cache them, replay them. You can build on them.
LLMs are not deterministic. Even with temperature=0 and a fixed model version, an LLM may produce subtly different outputs across calls with identical prompts. This is partly by design (sampling, parallelism, hardware differences), partly a side effect of model serving infrastructure, and partly an intrinsic property of the probabilistic nature of the underlying architecture.
This is well-known in the context of user-facing chatbots. What is less well-understood is what it means when you embed an LLM as a component inside a production software pipeline — one that is expected to behave like any other service: stable, cacheable, testable, incrementally correct.
The challenge is not just that the LLM might say something different. The challenge is that different prose produces different downstream state, and that downstream state is what your pipeline acts on.
When you first embed an LLM call into a pipeline, it feels simple:
flowchart LR
Input --> LLM["[LLM]"]
LLM --> Output
But production pipelines look more like this:
flowchart TD
Input --> A["LLM decision A"]
A --> IS["intermediate state"]
IS --> CL["Cache lookup using state"]
CL --> B["LLM decision B\n(conditioned on A)"]
B --> ID["Identity derived from B's output"]
ID --> PA["Persisted artifact keyed by identity"]
PA --> IDD["Incremental diff against previous artifact"]
IDD --> C["LLM decision C\n(conditioned on diff)"]
C --> FO["Final output"]
Each LLM call in this chain is a source of variance. And variance at step A propagates through every downstream step. By the time you reach the final output, you may have no idea whether the result changed because the real world changed, or because the LLM phrased something differently.
Most production pipelines are not full-recompute systems. They are incremental: they track what changed, what was processed, what is stale. Git diffs, database change streams, event queues — incremental processing is how large systems stay affordable.
LLM non-determinism attacks the most critical assumption of any incremental system: that unchanged inputs produce unchanged outputs. When this assumption breaks, you get:
False positives: “nothing changed but the system thinks something did”
False negatives: “something changed but the system did not notice”
Cascading re-computation that defeats the whole purpose of incrementalism
Runaway costs in pay-per-call LLM APIs
Continuous deployment loops triggered by phantom changes
Most bugs make something crash or produce obviously wrong output. Non-determinism usually produces output that looks correct. The analysis looks fine. The test passes. The CI is green. But on the next run, with identical inputs, you get slightly different output — and that tiny difference causes downstream components to do unnecessary work.
This kind of bug is extremely hard to find in code review. There is no test that reliably catches it because the tests themselves are non-deterministic.
When a non-deterministic component sits inside a pipeline that runs on every commit, the cost compounds fast. Thirty unnecessary LLM calls per CI run, at $0.003-$0.006 per call, is $0.09-$0.18 per run. Across hundreds of commits per month, across multiple repositories, this is real money — before you even count the engineering time spent debugging “why did this change?“
Systems that automatically generate and maintain structured reports from source code or other structured inputs are particularly vulnerable. Reports have identity: a report named variant-a3f8.md is expected to remain variant-a3f8.md across runs unless something actually changed. If the LLM renames an internal concept from “error path” to “failure branch,” and the report ID is derived from that prose, you now have a stale old report and a spurious new one.
Any CI/CD pipeline that uses an LLM to make decisions (code review, test generation, diff analysis, deployment approvals) is exposed. A non-deterministic approval generates a non-deterministic pipeline state, which can cause flaky builds, unnecessary rollbacks, or missed deployments.
Tools that statically analyze code and use LLMs to resolve ambiguity (e.g., which concrete implementation is called through an interface) must cache their decisions. If the same ambiguity is re-resolved on every run and the answer varies slightly, downstream graph structures change, and the entire analysis becomes unstable.
Pipelines that use LLMs to classify or label data incrementally face the problem that re-classification of unchanged data can silently flip labels. If downstream models are trained or evaluated on these labels, model performance becomes non-reproducible.
Agents that take actions across multiple steps — web browsing, tool use, code execution — amplify non-determinism geometrically. A different choice in step 2 produces a different context for step 3, which produces different tool calls in step 4. The final state of a multi-step agent workflow is highly sensitive to early variation.
RAG systems retrieve documents and synthesize answers. The retrieved documents depend on an embedding match, and the synthesis depends on which documents were retrieved. If the retrieval order or content changes slightly, the synthesized answer changes. If the system stores those answers as canonical truths, you have silent drift.
Non-determinism is a real and costly problem — but it is not a universal problem. Many categories of software use LLMs heavily and are genuinely unaffected by output variance. Understanding why they are immune helps clarify what actually makes the affected cases hard.
The underlying pattern: non-determinism only matters when the LLM’s output is used as input to a system that has memory, identity, or state. If the output is ephemeral and consumed immediately without being stored, compared, or built upon, variance is harmless.
A user asks a question, the LLM answers, the answer is shown to the user, and the conversation ends. There is no downstream state that persists. The next time the same question is asked, no system is comparing the new answer to the old one. The user might notice the answer is different, but no pipeline breaks.
Examples: customer support chatbots, developer assistants, knowledge base Q&A, search-augmented chat.
Why it is safe: The answer is the final product. It is not a key, an identifier, a cache lookup, or the input to another automated system. Variance in prose is the expected behavior of a conversational interface.
Systems that use LLMs to draft content that a human then reviews, edits, and approves before publication are effectively insulated from non-determinism. The human is the reconciliation layer. They see the draft, decide if it is good enough, and take responsibility for the output.
Examples: marketing copy generators, email drafting tools, blog post drafters, PR description generators, changelog summarizers.
Why it is safe: The human review step converts probabilistic output into a deliberate, deterministic human decision. The system never has to compare two LLM-generated drafts and decide whether they represent the same intent.
Applications where the explicit design goal is variety — where users want different outputs each time — are not just unaffected by non-determinism, they depend on it. Controlled randomness is a feature, not a bug.
Examples: story generators, image prompt expanders, brainstorming tools, game dialogue systems, name generators, creative writing assistants.
Why it is safe: There is no baseline to drift from. Each generation is independent and its quality is judged on its own merits, not compared to previous generations.
Systems that use an LLM to summarize or rerank search results for a user, without storing those summaries or feeding them into further automated processing, are effectively stateless. Each search is a fresh query. The user gets an answer and moves on.
Why it is safe: The LLM output is presentation only. It is never written to a database, compared against a previous version, or used as a key in any system.
Translation and transcription systems that display output to a user but do not store it in a structured system are low-risk. Small variations in phrasing across translations are acceptable — they are expected in natural language.
Examples: real-time captioning, document translation for reading, meeting transcription.
Why it is safe: Translation variance is a known property of natural language that users and downstream humans tolerate. The output is not parsed, hashed, or used as an identifier.
4.6 Classification With Human-in-the-Loop Correction
Pipelines where LLM classification feeds into a human review queue — where every output is reviewed before being acted upon — have the same insulation as content generation for human review. The human catches inconsistencies.
Examples: content moderation with human review, medical coding with clinician review, legal document classification with attorney review.
Why it is safe: The human step is the system’s source of truth. The LLM is a tool to reduce human workload, not an authoritative decision maker.
4.7 Summarization of Streaming Data (No State Persistence)
Real-time summarization of logs, events, or telemetry that is displayed in a dashboard and discarded does not accumulate state. Each window of data produces a fresh summary. There is no previous summary to compare against.
Why it is safe: The summary is a transient view of current state, not a persisted record. No downstream system depends on the summary being identical across repeated calls.
The clearest way to assess risk is to ask a single question:
flowchart TD
Q["Does the LLM's output become state that another system\nwill read, compare, or build upon in a future operation?"]
Q -->|YES| Y["Non-determinism is a design concern."]
Q -->|NO| N["Non-determinism is probably fine."]
“State” here includes:
Files written to disk or object storage
Database records
Cache entries used as keys
Identifiers or IDs derived from LLM output
Flags or decisions that trigger downstream automation
Embeddings stored for later retrieval and comparison
If the LLM output is consumed immediately by a human or displayed ephemerally, you are in the safe zone. If it is persisted anywhere and later compared to new LLM output to determine “what changed,” you are exposed.
A useful heuristic: if you could run the same LLM call twice and the system would behave differently on the second call based on what the first call produced, non-determinism is a risk you need to manage.
5. A Real-World Case Study: An LLM-Powered CI Analysis Pipeline
The system that motivates this post is a production CI pipeline for a large Go codebase. Its job is to automatically generate and maintain structured analysis reports for every entry point in the codebase — incrementally, on every commit. The pipeline combines static analysis with LLM-driven inference and persists its output to object storage, diffing against the previous run to determine what needs to be regenerated.
There are three distinct places where LLM calls are made:
1. Dispatch Resolution (Pass 1 + Pass 2)
Go uses interfaces extensively. When a method is called through an interface, there may be multiple concrete implementations. The pipeline uses Class Hierarchy Analysis (CHA) to enumerate candidates, then asks an LLM to resolve which concrete type is actually invoked in a given context.
For each entry point, the LLM is asked to identify distinct behavioral variants — “what are the different execution paths through this code?” It returns a list of variants with titles, descriptions, and the set of participating code nodes.
3. Report Synthesis (Pass 2 + 3)
For each variant, the LLM synthesizes a human-readable overview and a detailed structured report.
6. The Taxonomy of Non-Determinism in LLM Pipelines
After building and operating this pipeline, the non-determinism we encountered falls into six distinct categories. Understanding the category is the first step to fighting it.
What it is: The LLM uses different wording to describe the same concept across runs.
Example from our pipeline:
Run 1: variant_condition: "when the message fails to unmarshal"
Run 2: variant_condition: "on unmarshal error"
Run 3: variant_condition: "if JSON deserialization fails"
All three describe the same code branch. But if the artifact identity is derived from variant_condition, each run produces a different artifact ID. The old report becomes stale (but not deleted), and a near-duplicate new report is created.
Blast radius: Medium — cosmetically annoying, but also causes real cost from variant re-synthesis.
What it is: The LLM returns the same information but in a different order. If the pipeline derives any kind of index from position, this causes false positives.
What it is: For the same interface call with the same candidate set, the LLM selects a different concrete implementation on different runs.
Example:
Run 1: resolves (BatchMessageProcessor).ProcessBatch → (*payments/handler.Handler).ProcessBatch
Run 2: resolves the same call → (*inventory/handler.BatchHandler).ProcessBatch
Even if the LLM is “usually right,” inconsistency here changes the call graph structure, which changes the analysis tree hash, which marks entry points as modified even when no code changed.
Blast radius: Very high — affects the entire downstream pipeline.
What it is: The LLM returns output in a subtly different format that the parser handles differently.
Example from our pipeline: For a dispatch call involving (hash.Hash).Sum, the LLM consistently returned crypto/sha256.(*digest).Sum — the pre-Go 1.24 internal type name. After a Go toolchain upgrade, the actual FQN changed to (*crypto/internal/fips140/sha256.Digest).Sum. The LLM didn’t update its knowledge, so its output failed validation every time — but the failure was silent, and the site was re-attempted on every CI run, never successfully cached.
Blast radius: Medium — causes persistent cache misses for specific sites, cumulative cost.
What it is: Non-determinism that appears to come from the LLM but actually originates in upstream infrastructure.
Example: The static call graph builder uses the Go type system to enumerate candidates. Some FQNs (particularly for stdlib internals, platform-specific types, and generic instantiations) can be generated differently depending on the Go toolchain version, build tags, or even iteration order over maps in the compiler. If the candidate list changes between runs, the dispatch cache key changes, the persisted entry is not reused, and the LLM is re-invoked.
This is a case where the non-determinism appears in the LLM layer but is actually caused by the calling layer.
Blast radius: High and deceptive — it looks like the LLM is behaving differently, but the LLM is actually consistent. The problem is the key that addresses its cache.
The most important thing to understand about LLM non-determinism is that it doesn’t stay local. It cascades.
Here is a concrete cascade from our pipeline:
flowchart TD
S1["Step 1 — Dispatch resolution\nSame LLM target on both runs, but phrased differently on run 2\n→ cache key doesn't match → cache miss"]
S2["Step 2 — Call graph construction\nTarget A included in both runs (same result)\nbut map iteration order differs\n→ computeAnalysisTreeHash produces different hash"]
S3["Step 3 — Diff against previous analysis tree\nHash differs → entry point marked Modified\n→ IdentifyVariants called (1 LLM call)"]
S4["Step 4 — Variant identification\nSame 4 variants on both runs\nbut different variant_condition text\n→ 4 old IDs gone, 4 new IDs appear"]
S5["Step 5 — Report synthesis\n4 new variant reports synthesized (4 LLM calls)\n4 old reports marked stale"]
S6["Step 6 — Publish\n4 new files uploaded, 4 old deprecated\nManifest updated"]
S7["Step 7 — PR creation\nBot opens PR: 14 entry points modified\nPR merged → next CI run → same drift → infinite loop"]
S1 --> S2 --> S3 --> S4 --> S5 --> S6 --> S7
This is not a hypothetical. It happened in production.
flowchart TD
A["LLM Prose Drift"]
B["Different variant_condition text"]
C["Different variant ID\n(hash of text)"]
D["Old variant treated as deleted"]
E["New variant treated as new"]
F["Re-synthesis cost"]
G["Bot PR opened"]
H["Merge bot PR"]
I["Trigger next CI run"]
J["Same drift on next run"]
A --> B --> C --> D --> E --> F --> G --> H --> I --> J --> A
Observed: A CI run reported modified: 14, unchanged: 24 even though the target repository had no new source file changes between this run and the previous one.
Investigation: The pipeline does not use git diff to decide Modified vs Unchanged. Instead, it:
Loads the previous analysis snapshot from the artifact store
Rebuilds fresh analysis trees from the current source
Compares analysis tree hashes
When the dispatch cache was not fully reused across runs (due to validation failures we’ll discuss in a moment), some interface calls were re-resolved. Even though the LLM chose the same targets, the rebuilt call graphs had slightly different internal structure, causing analysis tree hash mismatches.
The critical insight: the pipeline concluded “modified” from analysis output drift, not from source-code drift.
Root cause: Dispatch cache reuse failures caused partial call graph non-determinism, which caused analysis tree hash drift, which caused false-positive Modified classifications.
Observed: A one-character string literal change in a single handler function ("unmarshaling message" → "unmarshaling inventory v2 message") triggered re-synthesis for 15 entry points. Only 1 was genuinely affected.
The system is supposed to be incremental: it has a change detection layer that computes a CanonicalSourceHash for each function (AST text, comments stripped) and only regenerates reports for variants whose participating_nodes intersect the changed nodes.
Root cause: The dispatch cache key is context-free.
This key contains the interface method and the sorted candidate set, but not the caller. This means every call site in the entire codebase that invokes the same interface method against the same candidates shares one cache entry.
The shared infrastructure pattern made this catastrophic:
internal/consumer/v2/consumer_pull.go
│
│ c.processor.ProcessBatch(ctx, batch) ← one call site
Six entry points, six different concrete types, one shared call site, one shared dispatch resolution. When BatchHandler.ProcessBatch changed, its CanonicalSourceHash changed, and the changed node appeared in every entry point that included the shared call site — all six entry points and their variants.
Expected result:new:0, modified:1, deleted:0, unchanged:37 Actual result:new:0, modified:15, deleted:0, unchanged:23 Extra LLM calls: ~84 unnecessary calls
The fix: include the caller FQN in the dispatch key, so each call site gets its own resolution scoped to its context.
8.3 Incident: 30 Dispatch Sites Missing Cache on Every Run
30 sites miss the cache on every single run, resulting in 30 unnecessary LLM calls per commit. This creates a permanent cost floor that no amount of caching can reduce.
Root cause split into two categories:
Category 1 (~2 sites): Parse failure — result never written to cache.
The LLM for (hash.Hash).Sum consistently returned crypto/sha256.(*digest).Sum — the pre-Go 1.24 internal type name. After a Go toolchain upgrade, the actual FQN changed to (*crypto/internal/fips140/sha256.Digest).Sum. The LLM didn’t know about this change. Its response failed validation, was never written to cache, and the same call was made on every subsequent run with the same result.
Permanent loop:
Run N: LLM call → parse failure → not written to cache
Run N+1: LLM call → parse failure → not written to cache
Run N+2: ...
Category 2 (~28 sites): Written to cache, but fails validation on next read.
Sites would parse successfully and be written to the persisted cache. But the following run would still miss them. Investigation pointed to two sub-causes:
Key mismatch: The constructorHash or candidatesHash component of the cache key was computed differently on read vs write (due to non-deterministic candidate enumeration for certain types).
Graph lookup failure: A selected FQN in the persisted entry was absent from the current call graph’s byFQN map — meaning the FQN format changed subtly between toolchain invocations.
Blast radius beyond cost: Because ~30 sites resolved differently on each run, 3 entry points were classified as Modified on every CI run, triggering a bot PR on every commit merge — including merges of the bot PR itself. Infinite bot PR loop.
The key insight for variant identity: you don’t need the LLM to produce consistent IDs. You need to reconcile LLM output against code-derived anchors before any identity-sensitive operation.
MM["Medium match → reuse previous ID\n(if unambiguous)"]
NM["No match → generate new stable ID\nfrom anchors"]
DED["Deduplicate by identity key\n(keep canonical, drop duplicates)"]
SYN["Synthesize via LLM\nusing reconciled IDs"]
IS --> NPN --> IBF --> CIK --> MAT
MAT --> SM --> DED
MAT --> MM --> DED
MAT --> NM --> DED
DED --> SYN
This way, even if the LLM describes the same variant with different words every run, the variant ID is stable because it’s derived from the code structure, not the prose.
A hash of the dependency injection wiring (the context the LLM uses to make its decision)
A resolver version (for explicit invalidation when the LLM or prompt changes)
A persisted entry is only reused if all of the following are true:
schema_version matches
resolver_version matches
wiring_hash matches (DI context hasn’t changed)
candidates_hash matches (the candidate set hasn’t changed)
All selected targets still exist in the current call graph
This brought dispatch cache hit rate from 0% (first run) to 83% (144/174 sites on subsequent runs), with the remaining 30 misses explained by the incidents above.
The key design decision: tie the cache key to the context that informs the LLM’s decision. If the DI wiring changes, the decision may change too — so the cache must be invalidated. This is similar to how a database query cache invalidates when its input tables change.
Before any LLM work starts, a Git-based precheck can determine whether the commit range contains any code changes that could plausibly affect the generated output.
The precheck is designed to fail open: if any uncertainty exists (no checkpoint, unreachable SHA, inconclusive classification), it proceeds to the full pipeline. The optimization is only applied when confidence is high.
Critically, the precheck uses the full range from last processed SHA to HEAD — not just the current commit. This ensures that skipped intermediate commits are still considered when evaluating whether to run.
When the LLM is asked to identify variants for an entry point, it may return semantically identical variants worded differently. Code-anchor deduplication catches these before they reach synthesis:
// Low-confidence → all candidates used (CHA pass-through)
// But the result IS written to cache, breaking the retry loop
returnDispatchResult{Confidence: "low"}, true
}
A low-confidence result has the same runtime behavior as a parse failure (all CHA candidates are used), but it stops the infinite retry loop. The LLM is not re-invoked for this site on the next run.
LLM providers can take arbitrarily long for individual responses, especially during degraded service. Without a per-attempt timeout, one slow call can block your entire pipeline.
Typical value: 60-120 seconds per attempt. With 3 retries and exponential backoff, worst-case per-call time is bounded to ~8-16 minutes rather than the provider’s connection timeout (often 10+ minutes per attempt).
A common mistake is using nested concurrency limiters. If you have an outer limit of 4 (for clusters of work) and an inner limit of 4 (for items within each cluster), you can have 4×4=16 concurrent LLM calls when you intended to have 4.
Use a single shared semaphore for all LLM-backed operations:
Pass this limiter through your entire call stack. The configured concurrency limit should mean exactly: the maximum number of concurrent LLM calls across the entire pipeline, not per level.
When the LLM service is degraded, you’ll get a flood of 503 or deadline errors. Without a circuit breaker, your pipeline will exhaust its retry budget on every call, waste 10+ minutes, and eventually fail.
A failure budget is simpler than a full circuit breaker and sufficient for most batch pipelines:
The LLM’s interpretation of a prompt changes when the prompt changes. This is obvious, but the consequence is less obvious: when you update a prompt, you should invalidate all cached decisions that were made with the old prompt.
Embed a resolver_version in your cache keys and increment it whenever your prompt changes meaningfully. This ensures that the first run after a prompt change re-resolves everything from scratch (acceptable), rather than mixing old and new resolutions in the same artifact (not acceptable).
Not all LLM calls should be cached the same way. Distinguish between:
Call Type
Cache Strategy
Structural decisions (dispatch)
Persist to durable storage; invalidate on context change
Content synthesis (reports)
Cache by code hash; regenerate when code changes
One-shot creative tasks
Don’t cache; accept variance
The key insight: cache decisions, not prose. A dispatch decision (“this interface resolves to type X”) is a structural fact that should remain stable. The prose explanation of why X was chosen is cosmetic and doesn’t need to be cached.
Artifact ID = hash(code-derived anchors: entry_fqn + branch_fqn + participating_nodes)
Artifact content = LLM-generated prose (can vary without affecting ID)
This separation means prose can be freely re-generated (for quality improvements) without causing identity churn. It also means the same artifact can be updated with new prose without creating a new file.
The LLM’s knowledge of your codebase is frozen at training time. It does not know about:
New packages added to your repository
Renamed interfaces or restructured types
Framework upgrades that change FQN conventions
This means the LLM can produce plausible-looking but incorrect output for code patterns it hasn’t seen, and the incorrectness may be subtle enough to pass validation. In our case, the (hash.Hash).Sum incident showed the LLM consistently returning a pre-upgrade FQN that looked valid but failed against the current call graph.
The mitigation is: validate LLM output against current code facts, not against what the LLM should know.
Even with temperature=0, some providers do not guarantee identical outputs across all invocations. Hardware differences, model serving infrastructure, and internal implementation details can produce different outputs for identical inputs. The only mitigation is caching — but caching only helps for calls you’ve made before.
The false-positive re-synthesis incident (15 entry points from 1 change) is only partially fixable with a context-scoped dispatch key. The deeper fix is context-sensitive pointer analysis (k-CFA instead of CHA). CHA is O(n) over the class hierarchy; k-CFA is polynomial and can be extremely slow on large codebases. For most production systems, CHA + LLM dispatch is the right tradeoff, but it cannot achieve the precision of a true context-sensitive analysis.
Every time you improve your prompt, you pay the cost of re-resolving all previously cached decisions. This is unavoidable — a better prompt may produce genuinely different and better results, so old cached results are legitimately stale. The mitigation is resolver_version in cache keys, which makes invalidation explicit and controllable.
Building software on top of LLMs that is reliable enough for production CI/CD requires treating the LLM as a probabilistic black box that produces proposals, not facts.
Code owns identity. Never derive artifact IDs, cache keys, or artifact names from LLM-generated prose. Use code-derived anchors (FQNs, hashes, structured types).
Validate, normalize, reconcile before using. Every LLM response should pass through a validation layer that checks code-derived facts, a normalization layer that canonicalizes format, and a reconciliation layer that matches against previous state.
Cache decisions, not prose. Persist structural decisions (dispatch resolutions, classification results) to durable storage with rich invalidation keys. Don’t cache prose — it should be freely regenerable.
Measure non-determinism directly. Instrument for cache hit rate, LLM calls per run, parse failures, and artifact churn rate. Non-determinism is invisible without these metrics.
Fail open safely. When caching or validation fails, fall back to a deterministic default (e.g., CHA pass-through, all candidates used). Never re-retry forever — persist the failure result.
Separate early gates from semantic gates. Git-based prechecks can save LLM cost for obviously irrelevant changes, but they do not replace semantic diffing. Use both, in sequence.
The patterns described in this post draw on established ideas from several fields:
Content-addressable storage (CAS): the idea of keying artifacts by hash of their content, used in Git, Nix, and build systems like Bazel
Incremental computation: systems like Salsa (Rust compiler) and Build Systems à la Carte provide formal frameworks for thinking about what must be recomputed when inputs change
Optimistic concurrency control: validating cached results against current state before use, rather than assuming they’re valid
Circuit breakers: from Michael Nygard’s “Release It!” — stopping cascading failures by detecting and short-circuiting failing dependencies
Context-sensitive program analysis: Andersen’s algorithm, k-CFA — the theoretical foundation for understanding why CHA + LLM is an approximation
The incidents described in this post were observed in a production CI pipeline running on a large Go codebase with hundreds of entry points and thousands of LLM calls per week. All code examples are simplified for clarity. If you’re building a similar system, the most important investment is not in prompt engineering — it is in the deterministic infrastructure around your LLM calls.