Inspect the EvoHarnessBench skills

evolve-eval is the evaluation skill and command-line entry point supplied by the authenticated evaluation package. Install it through the Evaluation tab using your service key.

Use __BASE_URL__ as the hosted catalog. Keep credentials in environment variables; never print, persist, place them in prompts, or pass them on command lines.

For an evaluation run, read references/guided-workflow.md and do not skip its four user checkpoints. For a documentation, API, or result question, answer directly without starting an evaluation; read references/http-and-sdk.md, inspect the installed SDK or service schema when needed, and tailor the smallest working query to the field the user wants.

Read only what the hosted evaluation needs

• references/modes-and-metrics.md for track, mode, and matrix choices.
• references/runtime-examples.md for EOG or ALE.
• references/http-and-sdk.md for execution and recovery.

The default OpenAI model is gpt-5 unless the user explicitly selects another model or an official benchmark fixes a separate simulator, judge, or model-menu seat.

Check the hosted target before evaluating

evolve-eval targets

Use a hosted endpoint only when its catalog entry says runnable: true and advertises the requested track, mode, action surface, and harness. If the hosted service does not support the requested configuration, explain that limitation before starting a run.

Local datasets

Download the datasets from the EvoHarnessBench collection on Hugging Face for local inspection and analysis. To run and score your agent, please use the hosted API service.

Guide the user from a seed benchmark to locally validated evolving datasets. Preserve the seed, work in a separate output directory, and never publish or upload without a separate explicit request.

Route the request

• For Terminal-Bench 2 at commit 2fd12b88aafdd04a52c298e3940bcb189f9766d6, read references/terminal-bench-2.md and follow its exact blocks.
• For APEX-Agents at commit 92c86856, read references/apex-agents.md and follow its exact blocks.
• For every other seed, read references/interactive-workflow.md, then references/adapter-contract.md. Read references/tracks.md while designing annotations and references/validation.md before validation or promotion.
• Read references/pipeline.md when explaining or checking the paper-aligned construction algorithm.

Do not load either exact example reference for an unrelated seed. TB2 and APEX are regression examples, not templates to copy blindly.

General workflow

For an unfamiliar seed, pause for four explicit user approvals. Treat every checkpoint as a repair loop: diagnose failures, propose a deterministic correction, regenerate the checkpoint report, and ask again. Do not advance while the report says repair_required, and do not collapse the approvals into one confirmation.

1. Seed suitabilityInspect read-only and report the source pin, task and grader shape, evidence, repeated capabilities, supported tracks, gaps, and separate output path. Tools and skills may be supported independently; agents require valid tool annotations and ownership.
2. AnnotationsPropose closed catalogs, deterministic matching rules, evidence samples, exclusions, ambiguous cases, skill procedures, and the exact tool-owner partition. Labels must be grounded in public prompts, descriptions, or approved anchors. The frozen adapter must label every task without model calls.
3. Ranking and stagingUse the packaged engine to show frequencies, release order, proposed versions, splits, exclusions, and smoke commands. Ask the user to approve this plan before building.
4. Validation and promotionBuild twice, compare every file, run the seed’s real verifier, and present the report. Ask separately before promoting or replacing an accepted destination.

Record each answer in approvals.json; record every user-approved annotation departure in deviations.json. Annotation approval must store the current annotation_input_sha256 printed by inspect, and staging approval must store plan_sha256. Any upstream edit invalidates downstream approval and returns the conversation to the affected checkpoint.

Approval may select a deterministic capability definition, such as APEX MCP operations as tools. It may not waive outcome authority, evidence, determinism, cumulative growth, earliest-solvable assignment, split floors, canonical row or materialization schemas, procedure preservation, or runtime validation.

Use scripts/evolve_core.py for shared construction mechanics. Seed-specific code should only normalize source data into the adapter contract. A failed or verifier-less build remains preflight-only; do not call it validated or promote it.