The problem
LLM pipelines donât fail loudly. Change a prompt, swap a model, bump a temperature: the code still runs, still returns a string, still passes every type check. What changes is quality, and quality doesnât throw an exception. Standard CI is built around code that either works or doesnât, so it has nothing to catch this class of regression. llmgate treats LLM output as a first-class thing to test against a baseline, the same way youâd test a functionâs return value.
Tracing
@llmgate.trace is a decorator applied to any function that returns a
string, or that returns an OpenAI/Anthropic response object directly. No
wrapping or reshaping required. Every call made through a traced function
is logged to .llmgate.db, a local SQLite database. No daemon, no
background process, no network call. The write happens inline, as part of
the function call.
Runs are grouped by an identifier set via the LLMGATE_RUN_ID environment
variable. In local development this is whatever version string you
choose. In CI, the natural choice is the git SHA, which gives every commit
its own addressable run of recorded outputs.
Assertions
Three assertion helpers ship with the library, applied directly to traced output:
assert_contains(output, substring): fails if a required string is missing.assert_output(output, predicate, message): fails if an arbitrary predicate over the output returns false, with a custom failure message.assert_similarity(output, baseline, threshold=0.85): fails if output diverges from a baseline string past a similarity threshold.
These run inline in your eval suite, the same way youâd write any test assertion. They catch missing required content or malformed output at the point of execution, before a diff is even run.
Diffing runs
The core operation is llmgate diff <baseline> <current>. It pulls both
named runs from .llmgate.db and computes token-level similarity between
corresponding outputs. If similarity drops below --threshold (default
0.8), the command exits 1. A numeric comparison between two sets of
previously recorded outputs, gated by a threshold you control per-project.
--no-fail runs the same comparison but always exits 0, turning the check
into a report instead of a gate. Useful when youâre establishing a
baseline and donât want early noise blocking merges.
Two inspection commands round out the CLI. llmgate runs lists every
recorded run with summary stats. llmgate show <run-id> drills into the
individual calls that make up a specific run, which is where you go after
a diff fails to see exactly which output changed.
Storage and CI integration
Everything lives in .llmgate.db. Commit it to the repo or cache it as a
CI artifact between jobs; llmgate only reads and writes a local file, so
it doesnât care which. No account to create, no service to authenticate
against, so nothing about the tool depends on network reachability during
a CI run. Data leaves the machine only when you push the .db file
yourself.
A typical GitHub Actions integration runs the eval suite with
LLMGATE_RUN_ID set to github.sha, producing a new run keyed to that
commit, then calls llmgate diff against github.base_ref to compare the
PRâs outputs against the target branchâs recorded baseline. A non-zero
exit fails the check the same way any other CI step would.
Design consequence
Every other piece of the tool follows from one choice: no server. A hosted eval platform can add authentication, team dashboards, retention policies, a query language for browsing past runs. llmgate canât, because it doesnât run anywhere but the machine and CI job that call it. Thatâs the trade. What it buys back is deployability: adding llmgate to a pipeline is a pip install and a decorator, not a vendor contract, a data-retention conversation, or a new service to keep available. For a check that runs on every PR, thatâs usually the more important property.