Offline nested-support saved responses
d4d evaluate support-results checks saved evidence under the explicit
nested_support_result_v1 protocol. It does not call a provider, execute a
planned request, use a judgement cache, or change an old plan. The v1/v2 planners
and SupportJudgeV2 remain separate instruments.
Prepare a descriptor from an existing d4d-support-plan-v2 directory with
--protocol nested_support_result_v1 and repeatable --select TARGET_ID
ATTEMPT_ID. Each selected target has one unique attempt. Fitness targets are
not eligible. The descriptor copies exact plan bytes and the selected records'
required artifact closure, reconstructs the schema specification and nested
inventory, and checks each selected complete request against its recipe. Schema
imports come only from captured bytes. Original source paths remain metadata;
they are never recreated or used to load today's schema.
The saved-response envelope is nested_support_response_v1. It contains exactly
format, descriptor_sha256, target_id, attempt_id, request_sha256, and
message_base64. The last field encodes the untouched native-message JSON bytes.
package_response below is a pure helper for an external capture harness. These
binding hashes are caller declarations: they detect unchanged envelope
replay into another request/attempt/record. They do not authenticate a provider
call or prevent a caller from inventing and relabelling evidence. The separate
registered executor captures the request,
admission, raw response and settlement at the call boundary. Its recheck
establishes captured invocation consistency, not authenticated provider
provenance; these saved-response commands remain offline.
The native-message adapter derives fields from those raw bytes. It requires an
assistant message, exact requested/reported model equality, end_turn, exactly
one text block containing a strict JSON object with exactly verdict and a
nonblank reason, and valid nonnegative integer input/output token counts.
Markdown fences, trailing prose, duplicate keys, nonfinite numbers, tool/refusal
blocks, unknown completion states and model aliases are refused. Optional
thinking/redacted-thinking blocks remain evidence. Missing usage remains
missing; it never becomes zero. There is no retry, best-of-N selection, model
alias mapping, or semantic truth certification.
accept preserves rejected raw evidence and exits nonzero for rejection. A
direct native-message file without an envelope is preserved and rejected with
an explicit missing-envelope reason. recheck reconstructs the outcome from
raw artifacts and returns success only when the saved assessment matches; a
correctly preserved rejection is still a valid artifact, not an accepted score.
Existing output directories, symlinks and aliases into input evidence are
refused. No command overwrites an existing destination.
report independently rechecks supplied results, refuses duplicate attempts and
foreign descriptors, and separates selected missing/rejected/accepted counts by
relationship edge and attribute value. Full-plan counts are labelled as declared
by the pinned manifest; unselected record artifacts are not reconstructed.
Fitness remains unscored. Even complete selected-response accounting leaves all
original calibration, context, instrument, transport and paid-readiness blockers
in place. This does not certify full-record schema validation or scientific
accuracy. Parents #3342/#2929 and calibration #3343 remain open.
Invented fixture round trip
This example creates a new temporary Git fixture and makes no network or model
calls. The model name, usage, cap and verdict are invented software-test values.
They are not a quality measurement, calibrated instrument or production budget.
Run from an environment where this repository's d4d package is installed.
demo=$(mktemp -d)
python - "$demo" <<'PY'
import hashlib, json, subprocess, sys
from pathlib import Path
import yaml
from data_sheets_schema.support_plan import build_plan
root = Path(sys.argv[1]) / 'source'
root.mkdir()
(root / 'schema.yaml').write_text('''id: https://example.invalid/fixture
name: fixture
prefixes:
linkml: https://w3id.org/linkml/
ex: https://example.invalid/
imports: [linkml:types]
default_prefix: ex
classes:
Dataset:
attributes:
title:
range: string
creators:
range: Creator
multivalued: true
inlined_as_list: true
Creator:
attributes:
name:
range: string
''')
record = yaml.safe_dump({'title': 'River archive', 'creators': [{'name': 'Alex'}]}).encode()
bundle = b'FILE: invented.txt\nAlex prepared the River archive.\n'
(root / 'record.yaml').write_bytes(record)
(root / 'bundle.txt').write_bytes(bundle)
(root / 'provenance.yaml').write_text(yaml.safe_dump({
'run': {'project': 'River', 'label': 'fixture_rep1', 'method': 'fixture'},
'model': {'model': 'invented-generator'},
'inputs': {'bundle_path': 'bundle.txt', 'bundle_md5': hashlib.md5(bundle).hexdigest()}}))
(root / 'roster.json').write_text(json.dumps({'jobs': [{
'id': 'river-r20', 'input': 'record.yaml', 'project': 'River',
'label': 'fixture_rep1', 'method': 'fixture', 'cohort': 'invented',
'generation_rep': 1, 'purpose': 'primary', 'rubric': 'rubric20',
'output': 'unmeasured.json', 'provenance': 'provenance.yaml'}],
'pinned_files': {'record.yaml': hashlib.sha256(record).hexdigest()}}))
subprocess.run(['git', 'init', '-q', str(root)], check=True)
subprocess.run(['git', '-C', str(root), 'add', '.'], check=True)
subprocess.run(['git', '-C', str(root), '-c', 'user.name=Offline fixture',
'-c', 'user.email=offline@example.invalid', 'commit', '-qm', 'Invented inputs'], check=True)
build_plan(root / 'roster.json', root.parent / 'plan', root=root, profile='neutral',
schema_path=root / 'schema.yaml', model='fixture-judge', max_tokens=128,
plan_version=2, artifact_kind='full')
PY
target=$(python - "$demo/plan/manifest.json" <<'PY'
import json, sys
m = json.load(open(sys.argv[1]))
print(next(t['id'] for t in m['targets'] if t['axis'] == 'grounding_v3'))
PY
)
d4d evaluate support-results prepare --plan "$demo/plan" \
--output "$demo/descriptor" --protocol nested_support_result_v1 \
--select "$target" invented-attempt
python - "$demo" <<'PY'
import json, sys
from pathlib import Path
from data_sheets_schema.nested_support_results import package_response
root = Path(sys.argv[1])
raw = json.dumps({'id': 'invented-message', 'type': 'message', 'role': 'assistant',
'model': 'fixture-judge', 'stop_reason': 'end_turn',
'usage': {'input_tokens': 10, 'output_tokens': 20},
'content': [{'type': 'text', 'text': json.dumps({
'verdict': 'supported', 'reason': 'Invented response for a software test.'})}]}).encode()
with (root / 'response-envelope.json').open('xb') as stream:
stream.write(package_response((root / 'descriptor/descriptor.json').read_bytes(),
attempt_id='invented-attempt', native_message=raw))
PY
d4d evaluate support-results accept --descriptor "$demo/descriptor" \
--response "$demo/response-envelope.json" --attempt invented-attempt --output "$demo/result"
d4d evaluate support-results recheck --result "$demo/result"
d4d evaluate support-results report --descriptor "$demo/descriptor" --result "$demo/result"
The final report shows one selected accepted response, other plan targets unselected, fitness unscored, and paid readiness still false. All bytes remain available in the temporary directory for inspection.