Agent Economics Lab

Explicit composition and source adapters

The modularity claim is intentionally narrower than a plugin ecosystem:

Normalize exported evidence into one stable bundle. Compose typed checks as an explicit ordered tuple. Record the source, evidence digest, decision-contract digest, enabled checks, and missing coverage in every result.

There is no automatic discovery, entry-point execution, dependency-injection container, live vendor authentication, or policy DSL.

Architecture

vendor or internal export
        |
        v
source adapter function ------ source ID + version
        |
        v
EvidenceBundle --------------- canonical SHA-256 digest
        |
        v
AssuranceEngine(checks) ------ fixed coverage + ordered checks
        |
        +---- gate results ---- PASS / FAIL + typed consequence
        |
        +---- diagnostics ----- warning evidence; never routing authority
        |
        v
AssuranceCase ---------------- evidence + contract digests
        |                       INCOMPLETE / SCALE / ASSIST / STOP
        |
        +---- renderer.markdown@1
        +---- renderer.json@1

The frontier path is a second, parallel surface with its own three renderers (renderer.frontier-markdown@1, renderer.frontier-json@1, renderer.frontier-svg@1), documented in the frontier. The diagram above is the single-arm decision path only.

The approach complements existing platforms. Galileo already supports custom code-based metrics, LangSmith supports trace export and OpenTelemetry routing, and OpenTelemetry publishes evolving GenAI semantic conventions. Those tools remain upstream evidence providers. This project owns the downstream economic assurance manifest and decision boundary.

Canonical evidence boundary

EvidenceBundle contains:

• normalized, deterministically ordered TraceEvent records;
• one Outcome per task;
• a model rate card;
• a named counterfactual baseline;
• versioned policy values;
• source adapter ID/version; and
• a content digest independent of source adapter and input row order.

The built-in adapters are:

source.csv@1
source.normalized-json@1
source.claude-code-jsonl@1
source.claude-code-session-tree@1
source.otel-genai@1

agent-economics capabilities prints this list from the build rather than from this page, and is the authority when the two disagree.

The public SWE-bench demonstration also records source.public-swebench-mini-agent@1. That mapper is intentionally scoped to one pinned public case and is not advertised as a generic mini-SWE-agent converter.

The JSON adapter is the offline interchange seam for vendor mappers. It does not claim that arbitrary Galileo, LangSmith, or OTLP files already match the canonical schema. A vendor contribution should be a small, tested mapping into the bundle, not a new economics engine.

Adapter equivalence is a required test:

csv_case = evaluate_bundle(csv_bundle)
mapped_case = evaluate_bundle(vendor_export_bundle)

assert csv_bundle.digest == mapped_bundle.digest
assert csv_case.decision == mapped_case.decision
assert csv_case.breaches == mapped_case.breaches

Adapters must reject duplicate event IDs and outcome IDs. They should discard raw prompt, response, and tool-argument values by default unless the assurance check explicitly requires them and the data owner approves retention.

Check contract

Each CheckSpec declares:

CheckSpec(
    id="gate.example",
    version="1",
    mode=CheckMode.GATE,
    covers=frozenset({Coverage.OUTCOME_QUALITY}),
    run=check_function,
    failure_route=Decision.ASSIST,
)

• id and version are recorded in the report.
• mode=GATE may return a typed ASSIST or STOP consequence on failure.
• mode=DIAGNOSTIC may add findings but is rejected if it attempts to route.
• covers declares which required assurance dimension the check supplies.
• failure_route binds a gate’s declared ASSIST or STOP consequence.
• run is an ordinary pure function over an immutable evaluation view.

Duplicate check IDs fail at engine construction. A check that emits results under another ID fails evaluation. A diagnostic that tries to return FAIL or a routing consequence also fails evaluation.

Implementation fingerprint

Every CheckSpec also exposes an implementation_digest: a SHA-256 over the normalized source text of run, recorded per check in the decision-contract manifest.

It closes a gap that declared metadata cannot. Deleting a required gate is caught by the coverage contract, because the fixed contract still demands the dimension. Replacing that gate with a permissive implementation that keeps the same ID, version, coverage, and failure route is not: required coverage still looks satisfied, and the gate simply stops failing. Before the fingerprint existed, that substitution produced a byte-identical contract digest. Now it does not.

from agent_economics import implementation_fingerprint

implementation_fingerprint(my_check_function)

Scope, stated precisely:

• It detects that an implementation changed. It does not prove that any implementation is correct, and it is not a signature.
• Normalization strips the common indent and per-line trailing whitespace, so the value depends on source text rather than nesting depth. It is stable across CPython versions, which is what keeps published artifacts byte-reproducible on the supported matrix.
• Comment and formatting edits inside a check change the digest. That is intended: the contract records which code produced a verdict.
• Checks generated by a shared factory share source text and therefore share a fingerprint. Closure values are not captured.
• Callables with no retrievable source (built-ins, C functions, functools.partial, interactively defined functions) raise ValueError rather than entering a contract with an unbindable implementation.

Fail-safe deletion

The default engine requires six coverage dimensions:

outcome_quality
unit_economics
tail_risk
business_value
counterfactual
runtime_caps

Deleting the structural repetition diagnostic is safe because it covers no required dimension. Disabling the acceptable-rate gate leaves the fixed contract without a provider for outcome_quality, so the engine returns:

Decision: INCOMPLETE
Missing coverage: outcome_quality

This is the key enterprise guarantee: modularity cannot be used to manufacture a green result by quietly deleting an inconvenient required check.

Add a domain gate

A local gate needs no registry or core edit:

def no_failed_events(view):
    failed = [event.event_id for event in view.events if event.status != "ok"]
    return CheckOutput(results=(CheckResult(
        check_id="gate.no-failed-events",
        status=CheckStatus.FAIL if failed else CheckStatus.PASS,
        message=f"failed events: {failed}",
        on_failure=Decision.STOP if failed else None,
    ),))

custom_gate = CheckSpec(
    id="gate.no-failed-events",
    version="1-local",
    mode=CheckMode.GATE,
    covers=frozenset(),
    run=no_failed_events,
    failure_route=Decision.STOP,
)

case = evaluate_bundle(evidence, default_checks() + (custom_gate,))

Run the complete add/delete proof with make modularity.

For CI, request JSON and decision-specific exit codes:

python3 -m agent_economics evaluate ... --format json --ci

Decision	Exit code
SCALE	0
INCOMPLETE	2
ASSIST	3
STOP	4

What remains fixed within a major version

The task/outcome join, complete effective-cost reconstruction, unit calculations, decision reducer, and canonical evidence digest remain a small inspectable kernel. They are not dynamically replaceable. Cost-source ledgers or generalized module discovery should be added only when two independent integrations demonstrate a real incompatibility with this boundary.

This site is open source. Improve this page.