Predictive models, generative systems, and tool-using agents can share an operational feedback loop. They cannot share one undifferentiated score.
The common loop is simple to draw:
versioned decision -> observable action -> delayed outcome
| |
+---------- stable join key -----------+
|
intervention test
|
advance, abstain, or roll back
The hard part is deciding what each arrow means before looking at the result. A dashboard assembled after the fact can usually explain any movement. A useful contract instead fixes the unit of decision, outcome definition, observation window, joins, missing-data route, counterfactual, and rollback rule while the system can still surprise us.
This repository ships that contract as dependency-free JSON and a closed validator. The filled example is synthetic and covers a support stack containing a predictive triage model, a generative drafting component, and a tool-using agent.
python3 -m agent_economics.feedback_contract \
examples/production-feedback-contract.json
The checked-in example validates to:
VALID digest=b17f80b2b89e4801c67ee27ff5764c6fc467e6b025169bb68a56ee0959435e80
That digest binds the supplied policy. It is not a certificate that the policy is correct.
Every production feedback loop needs the same seven decisions.
INCOMPLETE, ABSTAIN, or HUMAN_REVIEW, never a silent pass.These fields are deliberately operational. “Improve quality” is not an intervention. “Canary a frozen retrieval index against the current index and roll back if delayed task outcome is flat or citation support worsens” is.
The shared spine does not erase different failure modes.
| System component | Evidence required before interpreting the outcome |
|---|---|
| Predictive model | Calibration, threshold policy, and pre-committed drift slices |
| Generative component | Prompt version, retrieval version, and binding to the primary task outcome |
| Tool-using agent | Tool-registry version, side-effect ledger, and human-handoff rule |
For a predictive model, ranking quality can improve while the operating threshold becomes unsafe on a smaller slice. Calibration and threshold policy therefore belong in the record separately.
For a generative component, a model version is not enough. A retrieval-index change or prompt edit can move the outcome while the model stays fixed. Both versions have to travel with the decision.
For an agent, the trace is not the complete unit of risk. The system can write, send, purchase, delete, or escalate. A tool registry and side-effect ledger are needed to reconstruct what it was allowed to do and what it actually changed.
The validator requires exactly the evidence surfaces for the modalities the
issuer declares. Declaring predictive, generative, and agent while
providing only an agent trace is invalid. Supplying an undeclared evidence block
is invalid too. This closed shape prevents a generic “evaluation complete” field
from standing in for three different audits.
Most production outcomes arrive late. Teams therefore reach for acceptance, clicks, grader scores, latency, sentiment, or tool completion as faster signals. The problem is not using proxies. The problem is letting a proxy silently become the outcome.
The example permits same-session acceptance for diagnostic slicing and forbids it as a deployment gate or substitute for the 14-day outcome. Two failure modes are recorded: acceptance can precede a reopen, and silent abandonment can look like acceptance.
This is intentionally stricter than attaching a correlation coefficient. A correlation does not say what decision the proxy is allowed to make. Allowed uses and forbidden uses turn validation into policy.
When an outcome moves, the relevant question is not only whether the model got better. The change may have come from a label definition, data mix, retrieval index, tool interface, routing rule, or control policy.
The example carries three separate interventions:
Each has its own test and rollback trigger. This makes the feedback loop a way to choose the next engineering lever, not only a way to grade the current model.
The validator rejects unknown fields, duplicate JSON keys, non-finite numbers, unsupported modalities, blank required values, mismatched modality evidence, proxies bound to the wrong outcome, missing rollback semantics, and fail-open routes. It canonicalizes the valid document and returns a SHA-256 digest.
It proves structural completeness relative to version 1 of this contract and detects policy drift in the supplied record.
The validator does not establish that:
Those are empirical, organizational, or cryptographic claims. They need data, study design, independent review, or signed attestations. A structurally valid contract merely makes their absence visible.
The blank contract is intentionally invalid. A team should fill it before the canary starts, review it in version control, and bind the deployed system to the resulting digest. After the observation window closes, the same identifiers and rules should be used to assemble outcomes.
If a required outcome never arrives, the correct result is incomplete evidence, not a smaller denominator. If the primary outcome changes, issue a new definition version. If the intervention changes, issue a new digest. If a proxy is promoted from diagnosis to gating, record the new allowed use and review the evidence that justifies it.
The practical goal is not one universal evaluation score. It is a traceable answer to a narrower question: which versioned decision changed which outcome, under which policy, and what should the system do when the evidence is missing?