Skip to content

Offline typed support and fitness plans

d4d evaluate support-plan prepares the offline deliverable in #3341 for the common-instrument run in #2929. It makes no provider calls, estimates no empirical accuracy, and does not authorize execution. Run it from a complete Git checkout:

d4d evaluate support-plan \
  --roster notes/reference_rescore_2026-09-11/manifest.json \
  --profile bridge2ai \
  --output /tmp/d4d-support-plan

The output directory must be new and its parent must exist. The command deduplicates primary rubric jobs by input while rejecting conflicting identities. The reference roster's pinned_files hashes bind the full records. A new roster can use the same format, including an explicit provenance path per job for nonstandard layouts. The default provenance location is data/d4d_concatenated/{method}_core/{label}/{project}_provenance.yaml. The provenance's project, method and label must agree with the roster.

Every record, provenance file, original support bundle, available rubric join result, merged schema/import dependency, vocabulary, complete specification and system prompt is copied into content-addressed artifacts/. The manifest records planning-code hashes and commit, explicit profile selection, model-selection basis, same-family status, original paths and hashes, and any recovery commits. Historical reference records and bundles are recovered from full Git history by their recorded hashes when current bytes have drifted. Originals are never rewritten. Supplied provenance SHA256 pins are honored before reading identity or bundle declarations, recovering matching bytes when necessary. Historical rosters without that pin explicitly record the basis captured_current_unpinned_by_roster; a captured snapshot is not a claim that the older roster bound its lineage. Git reads ignore inherited repository/worktree/index overrides, so code and input commit identities refer to their intended roots. A missing or unrecoverable pin fails the build. A shallow clone must first obtain full history; a sparse checkout must materialize the roster, provenance, schema, current bundles and referenced rubric result paths. Unavailable rubric results are listed as an additional blocker rather than silently described as a completed join.

The plan retains every populated top-level field on both axes. Zero and false are populated. Nested list items are still one field request, as in the current v2 judge; resolving #3342 may change these counts and requires a new plan. Every target belongs to its own record and value, with its own judgement context. Value hashes use typed-yaml-v1: recursively tagged scalars, mappings and lists, including dates and timezone-bearing datetimes. This distinguishes a native date from its quoted string spelling and does not change the live judge's YAML prompt or existing cache keys. No historical cache reuse or replicate propagation is credited. Intended result/cache paths are reserved in the manifest; the planner does not create verdict files.

Inspect an exact target without any model call:

python -c 'import json; m=json.load(open("/tmp/d4d-support-plan/manifest.json")); print(m["targets"][0]["id"])'
d4d evaluate support-request --plan /tmp/d4d-support-plan --target 'ID_FROM_ABOVE'

The request command verifies artifact and full-request hashes before printing the reconstructed arguments. Source text is stored once and referenced by multiple recipes. These are the exact shared request-builder arguments at _call_with_retry, before provider transport, encoded with the plan's canonical JSON convention. They are not claimed to be SDK wire bytes. Provider routing, reasoning policy, output-limit adjustments, retries and accounting still need a separate execution registration. Transport must be reviewed before authorizing a run. Materialization needs neither the original files nor the current prompt text.

The manifest always blocks paid use on #3342 granularity, #3343 independent empirical calibration, transport registration and paid authorization. Software tests and this plan cannot clear these blockers. Positive and supported negative controls must be independently curated and measured before the cohort is judged. A proposed canary selects the first sorted record for each project; it is a plan, not a completed canary. Existing audit registrations, frozen caches and protected worktrees are not modified or borrowed as budget authority.

Estimates and local prices

Each target's input estimate is ceil(canonical request UTF-8 bytes / 4). This is an explicit heuristic, not a model tokenizer or guaranteed spend bound. The output ceiling uses --max-tokens (default 8000), including whatever reasoning the selected transport charges within that limit. One attempt per target is assumed; retries are excluded. The three scenarios are:

  • uncached: all input charged as ordinary input;
  • cache miss on every request: every support prefix charged as cache write;
  • warm within record: one support-prefix write per record and subsequent field requests as reads, with no reuse assumed between records.

The support prefix comprises the system prompt and source block. Actual cache support, minimum size and expiry are unverified. Fitness has no cached source block. Output ceilings are the same across these scenarios. Canary usage must replace these assumptions before estimating a full paid fanout.

Dollars remain null when prices are missing. Pass --model for an explicit model; otherwise evaluation_model_settings() resolves and records the generation-default basis. An optional --prices path.json accepts this shape:

{
  "model": "THE_EXACT_SELECTED_MODEL",
  "currency": "USD",
  "per_tokens": 1000000,
  "source": "Your verified local price source",
  "as_of": "YYYY-MM-DD",
  "rates": {}
}

Fill rates with the known nonnegative numeric rates for input, output, cache_write and cache_read. Omit unknown rates; a scenario requiring an unknown rate remains unknown, not zero. The planner pins the file and labels it user supplied and unverified; it never retrieves prices or imports an old study budget. Keep generated plans local until their source contents and publication scope have been reviewed. Planning does not complete the empirical acceptance criteria of

2929 or #3343.

Nested version 2

The default remains historical d4d-support-plan-v1. Opt in to the draft #3342 nested target contract with a fresh output directory:

d4d evaluate support-plan \
  --plan-version 2 --artifact-kind full --class-name Dataset \
  --roster notes/reference_rescore_2026-09-11/manifest.json \
  --profile bridge2ai \
  --output /tmp/d4d-nested-support-plan

Version 2 requires an explicit artifact kind (full, core, or collection). A core plan must select --class-name CoreDataset; the default full Dataset class is refused for core. A collection or external root requires its explicitly selected --class-name and --schema. A roster job's optional artifact_kind must agree, as must the corresponding provenance output path when declared. One plan uses one kind/class/schema/profile. Records are not unwrapped and paths address their exact pinned bytes.

The required profile selects its existing pinned vocabulary resource, ignoring ambient profile selection. --vocabulary FILE can explicitly override that snapshot for version 2 only. Its YAML must contain a vocabularies mapping from registry names to nonempty identifier/label mappings. The planner records the override basis and hashes the entire file. No registry is fetched. Missing values_from vocabularies remain named blocked paths; they are not replaced by empty authoritative lists. Both nested support and top-level fitness use the same captured vocabulary and schema closure. The nested schema view is independent of mutable shared LinkML caches, so capture order cannot alter its identity.

The output format is d4d-support-plan-v2. It uses grounding_v3 for the draft nested support contract and keeps fitness as a separate top-level measurement. Counts distinguish support relationship_edge, support attribute_value, and top-level fitness. The request count is an accounting total, not a pooled measurement denominator. Every nested target records its many-to-one root-field mapping and the corresponding fitness target id; a top-level fitness verdict must not become multiple nested verdicts.

Each record has a complete hashed target inventory, including its deduplicated specification catalog, all blocked pointers/codes, and explicit omissions from projected context. Every request pins its target/context/specification identities and reconstructs from a content-addressed recipe through the same d4d evaluate support-request command as version 1. The request's SHA-256 uses the planner's canonical UTF-8 JSON encoding, not an SDK wire encoding or a cache key. Raw record, provenance, original bundle, schema/imports, vocabulary, instrument source files and available rubric join outputs are separately pinned. Recovery uses the same original-hash checks as version 1 and never rewrites the source repository. Unknown top-level fields remain explicit blocked support and fitness paths; malformed nested shapes remain blocked support paths while their declared top-level field can still receive a separate fitness question.

The three token/cost scenarios are reported both overall and by measurement stratum. A warm support scenario credits one source-prefix write per record, then reads across both nested support kinds. Fitness has no cached source block. Estimates still use a byte-count heuristic, optional local prices and a requested output ceiling; they are not measured usage, a transport guarantee, or a spend cap. No judgement-cache reuse is credited.

Planning remains offline. Downstream software is available through the explicit saved-response protocol, registered execution, captured calibration evidence workflow and fitness/support index. These commands are separate opt-ins; preparing a plan does not invoke them or reuse judgement caches. The version numbers name different contracts: planner v2 uses grounding_v3, while saved nested-support results and execution each have their own v1 protocol.

Context-projection review, independently adjudicated scientific controls, #3343 empirical calibration, an explicit per-run transport/decision registration and actual paid authorization remain requirements. Software availability does not clear a captured plan's readiness blockers. Actual unresolved target paths and missing rubric join artifacts add their own blockers. Schema validation of the record is separate from this inventory; parent #3342/#2929 work remains open.