Prompt changes and model swaps break things silently. Nothing throws. The output is just worse, and nobody notices until a user does. There’s no compiler for prompt quality. llmgate traces what your LLM calls actually output, then diffs runs against each other before you merge.
Wrap a function in @llmgate.trace and every call gets logged to a local
SQLite file, .llmgate.db. Tag the run with LLMGATE_RUN_ID, a version
string or the git SHA if you’re in CI. Assert against the output:
assert_contains for required substrings, assert_output for an
arbitrary predicate with a failure message, assert_similarity for a
threshold comparison against a baseline. Run llmgate diff main feature-branch and it computes token-level similarity between the two
runs’ outputs, exiting 1 if anything regressed past the threshold, 0.8 by
default, configurable with --threshold. That exit code is the whole
integration point. Your PR fails the same way a failing unit test fails
it.
Most eval tooling in this space wants to be a hosted dashboard: send your traces up, view them in a web UI, pay per trace. llmgate stores everything in a SQLite file that lives in your repo or gets cached as a CI artifact between runs. Nothing leaves your machine unless you push that file yourself. There’s no backend to design around, so there isn’t one.
llmgate runs lists everything recorded, with stats attached. llmgate show <run-id> inspects individual calls in a run, for when a diff flags a
regression and you need to see what actually changed. --no-fail on the
diff command downgrades the check to a report instead of a gate, for
visibility without blocking merges yet.
The tracing decorator works with any function that returns a string, or
with raw OpenAI/Anthropic response objects. You don’t restructure your
pipeline to fit it. Install with pip install llmgate, wire the trace
decorator into your eval suite, add a diff step to your GitHub Actions
workflow keyed off github.base_ref and github.sha. No config file.
Nothing left to configure.