MedGaugeCapabilities / B · Embodiment (tools)
B.5 Tool optimisation Active
Given a tool and recorded trajectories that use it, the agent revises the tool to reduce steps or raise success rate without breaking its interface. Graded by comparing pass^3 and step counts before and after; the interface tests must still pass.
Evidence: none. No benchmark card lists this capability yet. No level is issued: methodology 0.1.0 sets no level thresholds.
- Domain
- B · Embodiment (tools)
- Evidence form
- program
- Veto constraints
- —
- Depends on
- B.1 Tool creationB.4 Tool result handling
- Required by
- —
- Internal task families
- —
- External anchors
- tau-bench, SkillsBench
- Version
- 0.1.0 · 0.1.0
Definition in detail
Recorded trajectories show where a tool makes agents take extra steps or fail. Revising the tool, rather than the prompt, fixes the cause for every later run, provided callers are not broken.
Inputs a task gives
- An existing tool with its source and tests
- Recorded trajectories that use the tool, including failed runs
- Interface tests that the revised tool must keep passing
Artifact required
A revised tool source with an unchanged interface and a change log listing each change with the trajectory evidence that motivated it.
How it is graded
A program runs the same task family with the original and the revised tool, on the same model and harness, and compares pass^3 and mean step count. The interface tests must still pass; a drop in pass^3 or a broken interface test fails the item.
In scope
- Reducing steps or failure points that the trajectories reveal
- Keeping the interface and output format stable
- Linking each change to trajectory evidence
Out of scope
- Writing a new tool from a requirement - graded under B.1
- Changing which components a harness loads - graded under B.6
Typical failures
Expected failure patterns; not measured rates.
- Fitting the tool to the recorded trajectories so that it fails on new items
- Changing an output format and breaking callers
- Cutting steps by removing validation that prevented wrong results
Standards: std-atif
Benchmarks
No benchmark card lists this capability.
Grading references
No grading reference is mapped to this capability.