Skip to content

Eval Seed/Build Matrix

The eval corpus now carries an explicit offline seed/build contract in each examples/eval/*/reference/policy.json.

The matrix covers this deterministic path:

go run ./cmd/icot --no-llm --no-transcript --from-example ./examples/eval/<fixture> --example <tmp>/<fixture>
go run ./cmd/openudon build --example <tmp>/<fixture>

The repository test uses the same behavior through package APIs and writes only to a temporary directory. It does not call an LLM provider, retrieve remote OpenAPI documents, or execute generated workflows.

Run the matrix directly with:

make eval-seed-build

For iCoT reliability reporting, run the provider-free scorecard:

go run ./cmd/icot scorecard --root examples/eval --out eval/runs/icot-scorecard-local

The scorecard writes openudon.icot-scorecard.v1 JSON with the expected outcome, observed outcome, fixture class, first failure family, and failure codes for each fixture. It uses the same no-LLM, package-local seed/build path as the matrix. Reports also include run ID, prompt version, readiness classifier version, generation time, git commit, and scorecard command provenance, and write a scorecard.json.sha256 digest sidecar after report consistency validation. Scorecard retention metadata marks these reports as release_evidence, without provider output, safe to archive, and not requiring redaction before sharing.

For M40 natural-language authoring coverage, include checked-in variant metadata:

go run ./cmd/icot variants validate --root examples/eval
go run ./cmd/icot variants coverage --root examples/eval
go run ./cmd/icot scorecard --root examples/eval --include-variants --out eval/runs/icot-authoring-scorecard-local
go run ./cmd/icot report verify --file eval/runs/icot-authoring-scorecard-local/scorecard.json

Variant files live at examples/eval/*/reference/authoring-variants.json. Positive variants reuse the fixture's reviewed reference workflow with a different operator brief and must still build provider-free. Missing-detail and unsafe-negative variants must stop with the declared needs_input, build_fail, or icot_fail outcome and failure family. needs_input variants must also declare expected_top_issue_code and expected_top_issue_slot; the scorecard compares those against the observed top readiness issue so a variant cannot pass by asking the wrong follow-up question. The variant scorecard adds provider-family, variant-class, provider/failure-family, and top readiness issue summaries without changing the default seed/build contract. It also records explicit counters for missing-detail or unsafe-negative variants that unexpectedly pass, plus a diagnostic-gap counter for needs_input variant results that lack top issue details. Missing-detail variants may set seed_from_reference plus clear_fields or clear_slots so the deterministic path preserves the reviewed source/operation and removes only the intended business/request detail.

icot variants validate is a fast metadata check for the same files. It catches schema errors, unknown expected failure families, missing or unknown expected top issue metadata, duplicate IDs, and reference-seeded clear slots that no longer match the reviewed reference intent.

icot variants coverage checks the same corpus by provider family and requires at least one positive, missing-detail, and unsafe-negative variant per provider family before the provider-free scorecard evidence is accepted.

This scorecard remains provider-free reference/variant package evidence. It does not show that a live LLM generated the workflow from the variant brief. For optional real authoring evidence, run:

go run ./cmd/icot authoring-eval --root examples/eval --include-variants --provider copilot-api --model gpt-5.4-mini --out eval/runs/icot-authoring-eval-local

icot authoring-eval writes openudon.icot-authoring-eval.v1 with provider/model, run ID, commit, command, prompt/readiness versions, LLM call count, generated paths, first failure family, drift counts, credential-scan status, and per-variant pass/fail. It also writes an authoring-eval.json.sha256 digest sidecar. Failures include a structured failure category for provider availability, timeouts, malformed model JSON, model refusal, incomplete drafts, lint/build failures, credential scan failures, and reference drift. Generated project files, intents, transcripts, and the report JSON are checked for credential-like literals. Retention metadata marks authoring-eval reports as local_ephemeral, containing provider output, not safe to archive, and requiring redaction review before sharing. Keep that report local/manual unless it has been reviewed for release-note evidence.

The make icot-authoring-scorecard and make release-saas-check paths run scorecard verification automatically. Optional real authoring-eval evidence remains local/manual; after generating it, verify the JSON and digest sidecar explicitly:

go run ./cmd/icot report verify --file eval/runs/icot-authoring-eval-local/authoring-eval.json

Policy Fields

seed_build.expected declares the required outcome:

  • pass: iCoT seeding and build must both succeed.
  • build_fail: iCoT seeding should succeed, but build is expected to reject the package.
  • icot_fail: iCoT seeding itself is expected to fail.

seed_build.class declares how the fixture should be interpreted:

  • strict-positive: golden OpenUdon-native behavior. These fixtures must build green.
  • expected-negative: a deliberate rejection or repair fixture. Failure is acceptable only when it matches the declared expectation.
  • advisory: reducibility or drift evidence. Advisory fixtures are useful, but they do not block strict positive coverage.

seed_build.allowed_failure_codes is optional. When present, at least one observed failure code must match the allow-list. Current build errors that happen before a full quality report are represented as build:error.

Current Split

Strict positive fixtures must pass the matrix. This includes the native SaaS, pagination, credential-binding, multi-service, runtime-only, and helper-backed examples.

Expected-negative fixtures either build a reviewable package that documents the problem or fail for the declared policy reason. For example, cmd-disallowed-deploy is expected to fail build because the project policy denies cmd.

Advisory n8n reducibility fixtures remain separated from strict native behavior. They currently seed and build cleanly from bounded, package-local OpenAPI evidence, but they remain advisory until the project deliberately graduates them into strict OpenUdon-native coverage.

Maintenance

When adding or changing an eval fixture:

  • Add reference/policy.json with seed_build.expected, seed_build.class, and a short reason.
  • Add reference/authoring-variants.json when the fixture is part of natural-language authoring coverage; keep variants provider-free and free of secrets, channel IDs, email addresses, or live provider outputs. For needs_input variants, declare the exact expected top readiness issue code and slot.
  • Keep strict positive fixtures buildable from package-local artifacts.
  • Use advisory only when a fixture is evidence for reducibility or upstream drift rather than a strict OpenUdon-native contract.
  • Use expected-negative only for deliberate rejection, clarification, or repair behavior.
  • Keep allowed_failure_codes narrow, and remove allowances when the gap is fixed.