Order resolution: an authorized action on the wrong purchase¶
A customer owns two blue backpacks, bought on different dates. Both orders are eligible for refunds. The customer asks for the later purchase, but the agent refunds the first record returned by the order service.
Identity verification succeeds. The amount and currency are correct for the selected order. The policy and payment checks pass. The customer still receives the wrong refund.
This executed local study separates permission to act on an object from evidence that the customer intended that object. It uses independent mock ledgers, not a fixture that declares wrong_order=true for a scorer to read.
What ran¶
The retained study packet contains sixteen deterministic executions: four scenarios, two agent implementations, and forward/reversed backend record ordering. Each execution includes the request, backend records, evaluator-only expectation, actual tool arguments and results, per-order states and events, output, elapsed time, and scoring fields. Trial hashes and source hashes identify the captured evidence; they do not authenticate its origin.
| Agent | Executions | Resolution-contract passes | Intended task completed | Wrong-order commits |
|---|---|---|---|---|
| Descriptive control | 8 | 8 | 6 | 0 |
| First-record mutant | 8 | 2 | 3 | 5 |
The descriptive control's two unresolved requests correctly stop without a refund. Those pass the safe unresolved-handling contract, but do not count as completed customer tasks. Conversely, the mutant sometimes guesses the intended order while omitting required clarification. A right final state does not necessarily earn a contract pass.
These are results on four deliberately constructed cases, not population success-rate estimates. Reversing the same records does not create independent customer samples. The descriptive control uses exact description matching and a literal Correction: delimiter; it is not evidence of general natural-language understanding.
Input and authority boundaries¶
UnresolvedRequest contains only utterance and customer_id. The evaluator retains expected_order_id, the necessary-clarification count and the scripted reply separately. None is placed in the initial model payload.
The customer ID represents a pre-authenticated synthetic session established by the harness. It is not a password, proof of real-world identity, or permission to trust a customer-supplied ID in production. The backend checks every requested customer/order against that session's scope. A foreign customer's similar purchase exists in the backend but is excluded from listing and blocked on direct read/write attempts.
Both owned orders are real mock objects: either can be verified, read, policy-checked and refunded. The backend does not use the evaluator's expected order to decide which one exists or may be refunded. The selected order's amount and currency come from its actual record; the second order uses a different amount and EUR rather than USD.
The original OpenAIAgentsRuntime.run remains an explicit-order task and still receives the supplied target ID. It tests execution after selection, not disambiguation. The new run_unresolved path receives the label-free request and exposes list_orders and ask_customer alongside the existing typed refund tools. run_case routes that runtime through the unresolved path. SDK-contract tests exercise the wiring with a fake SDK runner; no live model calls or model-quality results are claimed.
Kata 21: remove the answer from the experiment¶
Know: shuffling a list does not fix a benchmark if the correct object is also supplied in a separate field—or if only the correct object exists in the backend.
Task: inspect the initial request and order listing for explicit-description. Predict the first-record mutant's result before and after reversing the records. Then inspect the per-order ledger rather than trusting the output message.
uv run python -m cx_eval_lab.order_resolution \
--output /tmp/primer-order-resolution-my-first-run.json
uv run python -m unittest tests.test_order_resolution -v
Use a fresh output path for another run; the command refuses overwrite. It runs no model, calls no external service and moves no money.
Solution: compare action identity, not just action validity
The requested later purchase is order-b. Normal listing order puts order-a first, so the mutant refunds the wrong owned order. Reversing the backend records puts order-b first after the foreign record is filtered out; that execution passes.
Both refunds are valid under their selected objects' mock policies. The wrong execution fails because a non-target ledger changed. The evaluator counts all commits on all other objects, not merely whether the target eventually acquired a refund. Refunding both orders therefore cannot conceal the error.
test_first_record_mutant_really_commits_wrong_authorized_order asserts the real side effect and its failed grade. test_input_omits_target_and_candidate_order_permutation_preserves_answer checks the restricted input schema and verifies the descriptive control across both orderings.
Extend: construct a same-customer distractor with a nearly identical description and different price. Next add a cross-customer distractor. The first tests intent resolution; the second tests server-side authorization. A good result on one is not evidence about the other.
Interview answer: “I separate object selection from authorization. My distractors must actually exist and be actionable within their own permissions; otherwise tool failures can leak the answer. I grade every affected object and counterbalance presentation order.”
Kata 22: clarification must precede the consequential action¶
The customer says “Refund my blue backpack.” A script will answer a clarification with the earlier purchase date. An agent guesses that order, refunds it, then asks which purchase the customer meant.
Predict: the final refund matches the reference. The clarification count is one. Should the trajectory pass?
Solution: the right guess was not justified at action time
No. A count-only grader accepted this trajectory during development of this study. The regression test_clarification_after_refund_cannot_authorize_a_guessed_action now rejects it.
For cases requiring clarification, the scorer walks the captured tool events in order. A nonempty customer reply must precede a refund or approval-request attempt. Later clarification cannot retroactively justify an earlier action. The report retains premature_action_attempts, independently of whether the guess happened to hit the intended order.
Unnecessary questions are counted separately. Asking once in an already-specific request increases unnecessary_clarifications and fails this study's registered contract. For unresolved requests, asking and then withholding action can pass the safe-handling contract without completing the customer's task.
Extend: distinguish a request for missing identification from a legally or operationally required confirmation. They answer different questions and need separate events and policy rules. Do not generalize this study's zero-or-one-question contract into a universal limit on conversation turns.
Interview answer: “I check what evidence was available before a side effect. Clarification counts alone can reward questions asked too late. I measure completion, unresolved work, necessary clarification and avoidable customer effort separately.”
Kata 23: a later success must not erase an earlier violation¶
An agent attempts a one-cent refund against the EUR order. The backend rejects it. The agent then reads the correct amount and completes the intended refund.
Task: decide which outcomes should remain in the report. Repeat with a denied approval request. Then mutate the list returned by list_orders inside a control agent and inspect the retained event.
Solution: preserve both the recovery and the rejected attempt
The valid final refund can count as an intended task completion, but the resolution contract fails because denied_attempts is nonzero. The scorer counts raised access errors, unsuccessful identity checks, structured blocked refunds and denied approval results. It does not assume only exceptions represent failure.
test_rejected_amount_attempt_is_not_erased_by_later_success and test_denied_approval_stays_visible_after_success cover the structured-denial cases. The tools retain copied JSON observations, so modifying a returned list cannot rewrite the evidence of what was supplied. test_mutating_tool_response_cannot_rewrite_retained_event checks that boundary.
This teaching contract intentionally rejects any such denial. A production policy may permit specified recoverable tool errors, but it must register that allowance, retain the attempts, and keep high-consequence authorization violations distinct from harmless input correction.
Interview answer: “I retain attempted actions, blocked actions, completed effects and recovery. An eventually correct result does not erase a prohibited attempt, and a mutable tool response must not also be my authoritative log.”
Reproduction and remaining evidence¶
test_published_results_reproduce_except_wall_clock_measurement reruns all retained cases and compares their artifacts and decisions, excluding elapsed time. The publication records source hashes from the execution; it is not a signed release packet or an independent attestation. The CLI emits the whole study, and the test suite also checks that it will not overwrite earlier results.
This is a focused resolution contract, not full CX qualification. semantic_message_qualified is false for every trial: the original report does not claim to evaluate the truth of arbitrary customer-facing prose or import the semantic calibration registry. The paired extension below now connects native resolution evidence to the statistical comparison and release-receipt builder, without qualifying that missing semantic stage. The backend's existing amount, currency, identity, policy and approval checks remain in use, but these four cases do not exercise their entire state space.
Customer replies are scripted and returned regardless of question quality. Human comprehension, simulator realism, multilingual ambiguity, long conversations, confirmation revocation and live model reliability remain unmeasured. The next evidence milestone is a frozen model/rubric comparison on independently constructed cases, with metered repeated trials and qualified semantic grading. The delivery map keeps that requirement open.
Paired extension: execution through a blocked release decision¶
The retained paired study contains two new comparisons, each with sixteen agent executions. Both use the descriptive resolver as baseline. The first uses the first-record mutant as candidate; the second repeats the descriptive resolver as a positive structural control. These are 32 deterministic agent executions, followed by offline replay—not 32 independent customers and not live-model evidence.
| Candidate | Contract passes / 8 | Intended task completed / 8 | Unqualified message trials, both arms | Release action |
|---|---|---|---|---|
| First-record mutant | 2 | 3 | 16 | block, authority none |
| Descriptive control | 8 | 6 | 16 | block, authority none |
The original sixteen-execution study remains available above. The new packets use resolution-trial-v1 rather than inventing a preselected RefundCase for an unresolved request. Each preserves all three order ledgers, including the inaccessible foreign customer's unchanged ledger; actual tool calls/results/errors; the original label-free request; final response and runtime fields; the evaluator-only case; and the grade. Paired summaries reference artifact hashes.
Before execution, the manifest registers the full case contents, agent identifiers, resolution-contract estimand and permutation design. Both arms see the same record order at each repetition; odd repetitions reverse it. Arm execution alternates by case and repetition. The artifact sequence records that order. These are counterbalancing controls, not a randomized population sample. All four scenarios share customer-1, so the statistical comparison correctly reports one independent cluster, below its thirty-cluster teaching minimum.
uv run python -m cx_eval_lab.resolution_paired_study \
--output /tmp/primer-paired-resolution-my-first-run.json
uv run --extra openai python -m unittest \
tests.test_resolution_evidence tests.test_resolution_paired_study -v
Use a new output path for each run. No provider call is needed. The conformance workflow now runs the study and retains its report; that workflow definition is not proof of a completed GitHub run or deployment.
Kata 43: a summary is not enough to reconstruct a decision¶
A row says passed=True, with a matching artifact hash. The retained trace says an order was refunded, but the retained ledger says no refund exists. Another packet changes which order the evaluator expected in only the candidate arm. Could either packet pass replay if all affected artifact hashes were recomputed?
Task: reject both contradictions without calling an agent or judge. Then change a tool name to an arbitrary Python member and confirm the replayer refuses it. Finally remove a complete case from both arms: remaining pairs are complete, but the registered experiment is not.
Solution: rebuild mock state, then recompute the structural grade
Replay starts a fresh MultiOrderWorld from the retained case. It accepts only the explicit ResolutionTools method whitelist, invokes those mock methods with the recorded arguments, and compares every result or recorded error. It then compares every final order ledger and per-order event list. The tool transcript cannot merely assert that a refund happened; the same calls must produce the retained state in the local mock implementation.
The grader recomputes wrong-order commits, denied attempts, missing/unnecessary clarification, premature action, intended completion and the structural contract. Those values must match both the full retained evaluation and the paired projection. Tests patch the agent entrypoint to raise if called during replay: only mock tools run, never a model, judge or real payment service.
Full cases must match their registered content hash and remain identical across arms and repetitions. The agent-visible request must match the case's unresolved request exactly. The registered case inventory, population, agent identifiers, record permutation and execution sequence are checked too. Removing an entire case is not repaired by retaining complete pairs for the others. A boolean False is not accepted as integer repetition zero.
The regressions are test_registered_cases_request_schedule_and_arm_cannot_be_rewritten, test_changed_results_missing_events_and_foreign_state_are_detected_after_rehash and test_case_membership_is_registered_beyond_remaining_pair_completeness in tests/test_resolution_evidence.py.
Replay proves internal consistency under the installed mock implementation, not that an external service executed the original calls. A fabricated but internally consistent transcript can pass. The full-case hash also cannot establish that an adjudicator chose the right target: case quality, source identity and independently controlled anchors remain separate responsibilities. Keep the original artifacts when changing the grader or case labels.
Interview answer: “I retain the request, all affected objects, ordered calls, observations and outcome. I replay the local state transitions and re-grade, then verify the projection and registration. That makes the decision inspectable, but it is not execution attestation or proof that my reference labels are correct.”
Kata 44: an 8/8 control still cannot authorize deployment¶
The candidate passes all eight resolution-contract trials. Two trials correctly stop for unresolved requests, so only six customer tasks complete. All sixteen messages across baseline and candidate remain semantically unqualified.
Task: explain why the three numbers—eight contract passes, six completed tasks and zero qualified messages—are consistent. Choose a release action. Then change reported cost and latency while leaving their measurement source unchanged.
Solution: keep outcome, evidence quality and authority separate
Safe unresolved handling is a contract pass, not a completed refund. Conversely, the first-record mutant gets three intended-order endpoints but only two contract passes: guessing correctly before required clarification does not satisfy the trajectory contract. The manifest explicitly estimates a difference in resolution-contract passes, not a difference in complete application quality.
Both study comparisons retain unqualified prerequisites for full tool-boundary qualification and multi-order semantic grading. The actual release-receipt builder therefore returns locked, block, authority none. Separately, one independent customer makes the registered statistical comparison inconclusive. The favorable structural result cannot override either limitation. No native resolution artifact can self-assert semantic qualification; the current-calibration assessment reports sixteen unqualified message trials and not_applicable for a semantic registry check with no semantic receipts.
The default profile supplies invented 250 ms and $0.08 per trial. Its sixteen-trial selected-cost total is $1.28 synthetic, not spend or a bill. Actual local elapsed time is retained separately. Replayer checks bind projected synthetic values to that profile. The measured path instead binds latency to rounded elapsed time and cost to retained runtime evidence; absent runtime cost remains unknown, not zero. A coherent edit to the summary and evaluation alone fails when it contradicts that source. These joins do not authenticate provider usage or validate a real price table.
test_rehashed_measurements_must_match_their_retained_source reproduces the contradictory-measurement failure. test_native_prose_cannot_self_qualify_and_current_assessment_reports_gap checks the qualification boundary. test_study_retains_two_replayable_comparisons_without_release_authority verifies both combined decisions.
To inspect the retained results without rerunning an agent:
import json
from pathlib import Path
from cx_eval_lab.artifacts import replay_packet
study = json.loads(Path("docs/assets/paired-order-resolution-v1.json").read_text())
for comparison in study["comparisons"]:
grades = replay_packet(comparison["packet"])
print(comparison["candidate"], len(grades),
sum(g.unqualified_message_count for g in grades),
comparison["release_receipt"]["action"])
# first-record-mutant 16 16 block
# descriptive-control 16 16 block
The snippet re-grades trials and displays the retained release action; the study runner separately constructs that action from the paired experiment and prerequisite receipts. It is not a generic independent verifier of an uploaded outer receipt.
Interview answer: “First I identify what passed. A narrow contract score, a completed user task, qualified semantic evidence and deployment permission are different claims. My release packet keeps the missing prerequisites and insufficient sampling visible even when a local control scores perfectly.”
Next integration boundary¶
The structural extension above connects unresolved input, actual mock execution, artifact replay, paired statistics and a bounded release decision. The native semantic extension below adds a separate multi-order criterion and synthetic grading controls; it does not retrofit qualification onto the original packets. Independently reviewed calibration, metered model repetitions and current qualification/source checks remain necessary before a deployment decision. Reusing a single-order receipt by copying the expected target into its context would erase the very ambiguity this experiment tests.
Native semantic extension: bind the explanation to the whole situation¶
“Your refund has been confirmed” can accompany a correct refund, a wrong-order refund, or no refund. Matching the sentence is not enough. The native multi_order_customer_message_truth_v1 criterion receives the customer request, output, observed clarification and tool history, execution error, and all order ledgers. It does not receive the evaluator's expected order or a future scripted customer reply.
The evaluator owns the grading stage. Its receipt binds the message and structured claims separately from the case, policy, request, tool history and final-state context. The manifest also registers the native judge configuration and calibration identity. Replay checks the stored judge request against the execution and checks the receipt against that judgment. Changing a request and recomputing its local hash does not repair a contradiction with the retained execution.
This is not a cryptographic attestation of an original execution or independent proof that a judgment is correct. Operator-supplied trust anchors and current qualification are separate inputs. The following study exercises those interfaces with a deliberately limited literal fixture judge, not an LLM or a production semantic classifier.
Kata 45: four correct labels do not qualify a judge¶
Task: compile a calibration record from four retained synthetic annotation rows: two truthful and two false. A literal control makes no errors on those rows. Can you claim a false-pass rate below 5%? What changes if the local demonstration accepts only two examples per class and a very loose error bound?
uv run python -m cx_eval_lab.resolution_semantic_study \
--output /tmp/primer-native-semantics-my-first-run.json
uv run --extra openai python -m unittest \
tests.test_resolution_semantic tests.test_resolution_semantic_study -v
Use a new output filename. No provider call or human annotation service is used. The retained study includes calibration evidence, compiled counts, comparison packets and a revocation assessment.
Solution: derive the counts, then inspect what their bounds mean
The compiler joins each judgment to its evidence hash, checks the registered criterion/configuration/scope, resolves the supplied reviewer labels and derives class counts. It does not accept a caller's aggregate claim of hundreds of successful examples. Here the reviewer identities and labels are synthetic fixtures; two distinct strings are not evidence of two independent humans.
Zero false passes among two false examples gives a one-sided 95% exact binomial upper bound of 1 - 0.05**(1/2), approximately 77.64%. The same bound applies to zero false blocks among two truthful examples. This calculation illustrates sampling uncertainty under a binomial model; it does not validate that model for these authored, overlapping controls.
A deliberately permissive diagnostic policy can admit the record to exercise receipt plumbing. A stricter minimum of thirty examples per class rejects it as insufficient_calibration_examples. Even passing that count threshold would not establish representativeness, independent labels, acceptable error bounds or deployment authority. Replicating the same rows is not new evidence.
Calibration examples are drawn from two of the four scenarios also used by the comparison controls. Their fixture group IDs identify executions, not independent customers; all scenarios share one customer. There is no held-out transfer claim. The literal judge can pass familiar wording in an incorrect situation, and it abstains on unsupported wording. Its purpose is to exercise the measurement pipeline while making the empirical gap visible.
Recompile the retained rows without running an agent or accepting the report's aggregate counts on faith:
import json
from pathlib import Path
from cx_eval_lab.calibration_data import compile_calibration
study = json.loads(Path("docs/assets/native-resolution-semantic-v1.json").read_text())
calibration = study["calibration"]
record = compile_calibration(calibration["annotations"], **calibration["operator_config"])
assert record.content_hash == calibration["record_hash"]
print(record.truthful_examples, record.false_examples) # 2 2
print(round(record.error_bounds["false_pass_upper_95"], 4)) # 0.7764
The snippet deliberately uses the artifact's synthetic operator configuration to reproduce a fixture. A production verifier must obtain reviewer trust, split exclusions and qualification policy from independently controlled inputs—not grant authority to an uploaded configuration.
Extend: collect independently reviewed examples with contradictory identity, approval, eligibility and settlement explanations; include negation, quotations, multilingual wording and plausible paraphrases. Freeze the rubric and judge before a separate, untouched evaluation. Report false passes, false blocks and abstentions by slice, preserving disagreements rather than filtering them away. This human/model study is not performed by the command above.
Interview answer: “I distinguish a calibration record that is structurally valid from an evaluator that is empirically qualified. I derive counts from reviewed rows, check uncertainty and scope, and evaluate the frozen judge on independent cases. Synthetic labels and a permissive registry entry cannot establish truthfulness accuracy.”
Kata 46: joint success and current qualification are different decisions¶
The retained run executes three paired comparisons, with sixteen agent executions each, plus four calibration executions: 52 deterministic mock-agent executions in total. Replay runs the mock tools again, not the agents. All comparison scenarios still share one customer.
| Candidate | Structural passes / 8 | Joint passes / 8 | Tasks completed / 8 | Release action |
|---|---|---|---|---|
| False-settlement mutant | 8 | 0 | 6 | block, authority none |
| First-record mutant | 2 | 2 | 3 | block, authority none |
| Descriptive control | 8 | 8 | 6 | block, authority none |
These are controlled contrasts, not accuracy estimates. Each comparison reports a $1.28 synthetic agent-cost subtotal, sixteen unknown judge-cost components and a null complete selected-cost estimate. The known judge subtotal of zero means no known amounts were added—not that those judgments were established to be free. The workflow is configured to retain the study; no completed cloud run is claimed here.
Task: compare a correct resolver that makes an unsupported settlement promise, a first-record resolver, and the descriptive control. Then revoke the calibration record. Should replay erase old grades, keep authorizing the judge, or report two separate results?
Solution: require both grades, then reassess current eligibility separately
A joint trial passes only when its structural contract passes and its native semantic judgment is qualified and passes. A prose failure defeats a structurally correct action. A favorable prose judgment cannot override a wrong order, a rejected unauthorized attempt or a transaction before required clarification.
Preserve three semantic states: a qualified failure is evidence against the response; a qualified abstention supplies no positive verdict; an unqualified judgment lacks eligible grading evidence. All block joint success, but they have different reasons and counters. Do not label every failed trial an unqualified one or count every qualified judgment as a pass.
Historical replay uses the caller's explicit historical calibration trust. The separate current assessment checks an operator-owned registry and clock. Revocation leaves the unchanged historical evidence replayable but returns not_current; it does not erase completed actions or confer permission for new judgments.
Even the positive synthetic control cannot authorize deployment. These cases share one customer, the calibration/control scenarios overlap, the judge is a literal fixture, and full application prerequisites remain unqualified. The release builder retains block with authority none. Source hashes detect changes relative to an anchor; they do not authenticate the operator, human reviewers, provider bill or original execution.
The native audit also checks rejected grading paths. A pre-dispatch rejection cannot acquire a fabricated request, completed judgment or usage record. If qualification expires during a call, the retained post-dispatch evidence must still match the actual request, configuration and timing; absence of a qualified receipt does not exempt that history from validation.
Extend: replace the literal control with a frozen, independently calibrated rubric. Katas 47–48 now test the native provider adapter and direct budget wrapper through the installed SDK with in-memory HTTP; they do not supply that empirical calibration. Account for agent and judge usage separately before combining selected costs; unknown judge usage is not free. A single-order adapter or campaign gate must not be assumed to support a new criterion without its own integration tests.
Interview answer: “I require structural and semantic success together, preserve the full evidence for re-grading, and separate historical reproduction from current qualification. A revoked judge can explain an old result without remaining eligible for a new decision. A successful local control still needs representative evidence and release prerequisites.”
Native paired diagnostics: separate the action from the explanation¶
The false-settlement candidate passes 8/8 structural contracts and 0/8 joint contracts. The first-record candidate passes 2/8 of each. Those results need different repairs: one candidate's explanation is false; the other selects or acts on orders incorrectly.
The native slice report derives these diagnostics from the already-retained semantic study. It replays mock tool histories and reconstructs grades, but does not run new agents or collect new model observations. The original executions and release receipts remain unchanged. Across three comparisons it inspects 48 retained trials: 24 pairs formed from four cases repeated twice per comparison. Every case belongs to the same synthetic customer.
| Candidate versus descriptive baseline | Structural pass | Joint pass | Structural regressions | Joint regressions |
|---|---|---|---|---|
| False settlement | 8/8 | 0/8 | 0 | 8 |
| First record | 2/8 | 2/8 | 6 | 6 |
| Descriptive control | 8/8 | 8/8 | 0 | 0 |
The baseline passes all eight structural and joint contracts in each comparison. Joint means the structural contract passes and an eligible semantic judgment passes. These are criterion-specific fixture results under explicit synthetic calibration trust—not empirical semantic accuracy.
Kata 64: the same failed-check name can mean different evidence¶
Predict: why must the false-settlement pairs count as regressions rather than unknowns? Why would a qualified judge's abstention require a different classification? Can a favorable prose grade repair a wrong-order transaction?
Reproduce the derived report from the repository root. The expected source hash below hashes file bytes, whereas packet and plan hashes identify canonical JSON values:
uv run python -m cx_eval_lab.resolution_slice_study \
--source docs/assets/native-resolution-semantic-v1.json \
--expected-source-sha256 sha256:638237b1615ec159c87d8ec8f2bb748867c2b319c93ff678470f4e0619419839 \
--trusted-calibration-hash sha256:454a18e2f18782f37ecfb2c7832a259aead13455be9cacfe94a7464e0d54730b \
--output /tmp/native-resolution-slices.json
uv run python -m unittest tests.test_resolution_slices tests.test_resolution_slice_study -v
Use a new output path on a subsequent run. The calibration hash is an explicit exercise trust choice for this synthetic fixture. Do not replace it with “trust every hash the input supplies.” Recompiling its annotations checks consistency; it neither authenticates invented reviewers nor qualifies the judge for production. A changed source file requires investigation and a separately anchored derivation, not automatic adoption of its new hash.
Solution: classify semantic status, not a generic failure flag
In the false-settlement comparison, both arms' judgments are eligible under the deliberately permissive fixture policy. The baseline judgment passes and the candidate judgment fails. Each pair is therefore a known joint regression, even though its structural contract is unchanged and passing. Calling these eight pairs “unqualified” would hide negative evidence supplied by the chosen instrument.
Abstention is different: the instrument did not supply a pass/fail determination. Missing qualification is different again: the judgment lacks required eligibility. Neither earns a positive joint outcome. Preserve semantic status and reason alongside structural checks rather than interpreting the generic qualified_multi_order_message check name as an uncertainty label. The reporting adapter conservatively marks a joint pair unknown when either arm lacks an eligible, non-abstaining semantic determination; any known structural failures remain separately visible.
The first-record candidate's literal prose judge can pass because one refund was committed, even when it was the wrong refund. The structural grader evaluates intended order and required clarification independently. A semantic pass cannot compensate for those failures. Joint success is a conjunction, not an average of grades.
A structural-only resolution-trial-v1 packet has no native semantic qualification. Its structural changes remain inspectable, but its joint-quality comparison is unknown. A resolution-trial-v2 packet adds semantic evidence; the version label alone is insufficient—the report must replay and validate the receipt and audit against caller-supplied trust.
Interview answer criteria: distinguish qualified failure, abstention and missing qualification; show which structural evidence remains usable in each situation; explain why known failure must not disappear into an unknown counter. Name the evidence needed before combining grader criteria into one success claim.
Kata 65: a useful retrospective slice is not a registered release requirement¶
Predict: split the first-record comparison by whether the case requires clarification. Which failures does the aggregate hide? May that slice become a release requirement by editing the archived manifest?
The derived plan assigns feature labels from retained case definitions, not from pass/fail outcomes:
| Feature slice | Cases | Pairs | Baseline joint pass | First-record joint pass |
|---|---|---|---|---|
| Clarification required | 2 | 4 | 4/4 | 0/4 |
| Clarification not required | 2 | 4 | 4/4 | 2/4 |
| Target known to evaluator | 3 | 6 | 6/6 | 2/6 |
| Unresolved intent | 1 | 2 | 2/2 | 0/2 |
“Target known” describes the evaluator's reference, not information revealed to the agent. These two partitions overlap: the unresolved case also requires clarification. Summing all four denominators would count each pair twice.
import json
from pathlib import Path
from cx_eval_lab.resolution_slice_study import derive_study
from cx_eval_lab.resolution_slices import derive_resolution_slice_report
source = Path("docs/assets/native-resolution-semantic-v1.json")
trust = frozenset({"sha256:454a18e2f18782f37ecfb2c7832a259aead13455be9cacfe94a7464e0d54730b"})
derived = derive_study(source,
expected_source_sha256="sha256:638237b1615ec159c87d8ec8f2bb748867c2b319c93ff678470f4e0619419839",
trusted_calibration_hashes=trust)
assert derived == json.loads(Path("docs/assets/resolution-slices-v1.json").read_text())
control = next(c for c in derived["comparisons"] if c["candidate"] == "first-record-mutant")
joint = control["report"]["joint"]
required_feature = next(r for r in joint["slices"] if r["slice"] == "clarification:required")
assert required_feature["pair_count"] == 4
assert required_feature["candidate_rate"] == 0
assert required_feature["membership"] == "exploratory"
assert joint["overall"]["customer_count"] == 1
assert joint["overall"]["comparison"] is None
packet = next(c["packet"] for c in json.loads(source.read_text())["comparisons"]
if c["candidate"] == "first-record-mutant")
changed_plan = {**control["plan"], "required_slices": ["clarification:required"]}
try:
derive_resolution_slice_report(packet, changed_plan, trusted_calibration_hashes=trust)
except ValueError:
print("Retrospective feature labels cannot silently become required gates.")
else:
raise AssertionError("Unregistered required slice was accepted")
Solution: preserve the archive and register the follow-up
All four clarification-required pairs regress. Without required clarification, customer-correction passes only repetition 0, while explicit-description passes only repetition 1. Changed listing order exposes the first-record heuristic. A single aggregate of 2/8 cannot show either concentration or order sensitivity.
These labels were assigned during a new analysis of an existing archive. They are explicitly exploratory. Every plan must cover exactly the packet's case IDs; omissions, extra IDs and malformed labels are rejected. A plan with required slices is accepted only when its exact content hash matches registration already present in the packet's manifest. These archived comparisons lack that registration, so retrospectively assigning required status is rejected.
For a follow-up experiment, choose slice definitions and required coverage before executing agents; bind the complete plan to the new manifest; then retain new evidence under that identity. Do not rewrite the old packet to simulate preregistration. A self-consistent hash is not an independently authenticated timestamp or approval.
All cases share one customer. No row has enough independent support for a population interval, regardless of permutations or labels. The structural projection is descriptive rather than an alternative registered primary endpoint. Joint diagnostics remain restricted, and original release receipts still block. A new report does not confer new authority on old evidence.
Extend: design a follow-up population varying customers, ambiguous descriptions, correction styles and competing orders. Keep clarification burden and unresolved work separate from safe contract handling: the descriptive control's unresolved cases pass their safe-handling contract without completing a refund. A dashboard equating “contract passed” with “customer task completed” conceals that distinction.
Evidence limit: this addition validates native replay-to-report integration. It does not add independent customers, new agent executions, real reviewer labels, a general semantic judge, qualified interval methods, current registry authority or an application rollout. Those remain distinct requirements in the delivery map.