Valstorm Agent Runtime:
Multi-Axis Benchmark & Ablation Study
Enterprise AI evaluations often fail because they change multiple architectural variables at once or grade conversational prose instead of post-execution system state. Below is our reproducible 6-axis ablation study evaluating the Valstorm Agent Runtime (vsagent) across UK AISI Inspect AI, Terminal-Bench 2.0, and τ-bench (pass^k) standards.
The complete Valstorm Agent Runtime (vsagent) and this benchmark harness are publicly available at github.com/Valstorm-Corp/valstorm-agent under the PolyForm Noncommercial License 1.0.0. Clone, audit, and run every benchmark locally.
10,976 vs. 31,200 live tokens on G-S01 (2.8x lower inference cost) by scoping tools per profile.
Wall-clock speedup via background=True async spawning (66ms launch, 12.07s vs. 18.52s live).
+44.4% absolute pass-rate lift over single-shot generation across multi-step TDD and patching tasks.
Eliminates operational permission regressions (such as missing chmod +x on G-H03 shell scripts).
1. Methodology: Combining 4 Peer-Reviewed Evaluation Standards
Rather than inventing an ad-hoc grading script, the Valstorm Benchmark Harness (apps/agent-runtime/benchmarks/) combines four established evaluation standards into a single deterministic runner:
Strict Task / Solver / Scorer Decoupling
Strictly separates declarative Tasks (tasks/*.json), Solvers/Configs (BenchmarkConfig + runtime adapters), and Scorers (EnvironmentStateScorer in scorer.py).
Post-Execution Environment-State Grading
Executes every run inside an isolated temporary workspace (tmp_path) and grades physical post-run filesystem state: created files, content regexes, POSIX executable bits (0o755), and pytest exit codes.
Multi-Trial Enterprise Consistency (p^k)
Tracks both optimistic capability (Pass@k: at least 1 success in k trials) and strict enterprise consistency (pass^k = p^k: the probability that all k trials succeed without flakiness).
Stratified 11-Task Complexity Taxonomy
Spans Simple (G-S01..G-S03), Medium (G-M01..G-M03), High (G-H01..G-H03), and Multi-Agent Parallel Swarm (SW-01, SW-02) tasks across math, patching, TDD, and shell guardrails.
2. The 6 Isolated Scientific Ablation Axes (10 Runtime Factors)
To eliminate confounding variables, each ablation preset (--ablation <axis>) modifies strictly one independent variable while holding the baseline configuration constant:
| Ablation Preset | Factors Isolated | Configurations Compared | What It Measures |
|---|---|---|---|
| skills | #1 With Skills vs. #2 Without Skills | skills_on vs. skills_off | Quantifies how procedural SKILL.md guides (bash-scripting, test-driven-development) prevent guardrail failures such as missing chmod +x on G-H03. |
| react | #3 With ReAct Loop vs. #4 Without ReAct | react_on (max_turns=15) vs. react_off (max_turns=1) | Measures self-correction and multi-step observation value across tool chains and Red-Green-Refactor TDD loops. |
| tools | #5 Scoped / Full Tools vs. #6 No Tools | tools_scoped vs. tools_full vs. tools_none | Quantifies tool schema context bloat, token cost overhead, and extra turn drift when exposing 30+ tools vs. a profile-scoped whitelist. |
| reasoning | #7 & #8 Model Tier × Thinking Effort | thinking_none, low, medium, high | Measures latency and token trade-offs across reasoning effort budgets (none → high). |
| swarm | #10 Parallel Swarm vs. Sequential Agent | seq_single vs. swarm_parallel (background=True) | Measures wall-clock compression (max(D_i) vs. sum(D_i)) and multi-agent token overhead on composite tasks. |
| harnesses | #9 Cross-Agent CLI Comparison | vsagent, claude, codex, opencode, hermes, oh-my-pi | Executes external agent CLIs headlessly in an isolated temp folder and grades post-run filesystem state identically. |
3. Live LLM Findings: Tool Surface Scoping Cuts Token Usage by 64.8% (2.8x Cost Reduction)
We executed live LLM ablations using gemini-flash-latest on task G-S01 (compute (42 * 3.14159) / 7 rounded to 4 decimal places via calculator and write 18.8495 to calculation_result.txt):
| Ablation Axis | Config ID | Outcome | Turns | Actual Tools Called | Wall Latency | Total Tokens | Est. Cost ($) |
|---|---|---|---|---|---|---|---|
| ReAct Loop | react_on (max_turns=15) | PASS | 3 | calculator, write_file | 7.16s | 10,844 | $0.00169 |
| ReAct Loop | react_off (max_turns=1) | FAIL | 1 | calculator (halted after Turn 1) | 2.06s | 3,413 | $0.00052 |
| Tool Surface | tools_scoped (2 tools) | PASS | 3 | calculator, write_file | 6.58s | 10,976 | $0.00170 |
| Tool Surface | tools_full (30+ tools) | PASS | 4 | calculator, write_file, read_file | 7.55s | 31,200 | $0.00477 |
| Tool Surface | tools_none (0 tools) | FAIL | 1 | (none) | 10.96s | 3,457 | $0.00061 |
Why Profile-Scoped Tool Registries Matter in Production
When an agent runtime injects all 30+ platform tool JSON schemas into every turn (tools_full), prompt token volume nearly triples (31,200 vs. 10,976 tokens) and the model takes an unnecessary 4th turn (read_file). By binding each named Valstorm profile (developer, researcher, marketer, archivist) to a strict allowed_tools whitelist, we reduce token consumption by 64.8% and cut inference cost by 2.8x without sacrificing accuracy.
4. Mathematical & Empirical Proof of Parallel Subagent Execution
In apps/agent-runtime/tools/delegation.py, invoking delegate_task(..., background=True) schedules an isolated child ReAct coroutine via asyncio.create_task() and immediately returns a task_id in under 70 milliseconds. Calling delegate_task(action="wait", task_id="all") executes asyncio.gather() across all active workers, compressing total wall-clock execution time from the sequential sum (T_1 + T_2 + ... + T_N) down to the single slowest task (max(T_i)):
- Non-Blocking Spawn Time: 66.0 ms (0.066s)
- Worker 1 (developer): 12.08s
- Worker 2 (researcher): 6.44s
- Sequential Sum (D1 + D2): 18.52s
- Actual Parallel Wall-Clock: 12.07s (Saved 6.45s / -34.8%)
- Simulated Per-Worker Delay: 2.00s each
- Individual Worker Durations: 2.10s, 2.09s, 2.09s
- Sequential Sum (D1+D2+D3): 6.28s
- Theoretical Floor (Max Task): 2.10s
- Actual Parallel Wall-Clock: 2.09s (Saved 4.19s / -66.7%)
5. Full 6-Axis Configuration Ablation Summary (11 Tasks)
Below is the complete configuration scorecard generated by python -m benchmarks.runner --ablation all across the 9 Core Tasks (G-S01..G-H03) and 2 Multi-Agent Swarm Tasks (SW-01, SW-02):
| Config ID | Configuration Label | Pass Rate | pass^k | Avg Turns | Avg Latency | Avg Tokens | Est. Cost | Speedup |
|---|---|---|---|---|---|---|---|---|
| skills_on | 1. With Procedural Skills (SKILL.md) | 100.0% (9/9) | 100.0% | 3.0 | 0.26s | 945 | $0.00171 | 1.00x |
| skills_off | 2. Without Procedural Skills | 88.9% (8/9) | 88.9% | 3.0 | 0.23s | 705 | $0.00139 | 1.00x |
| react_on | 3. With ReAct Loop (max_turns=15) | 100.0% (9/9) | 100.0% | 3.0 | 0.30s | 972 | $0.00186 | 1.00x |
| react_off | 4. Without ReAct Loop (Single-Shot, max_turns=1) | 55.6% (5/9) | 55.6% | 1.0 | 0.08s | 323 | $0.00062 | 1.00x |
| tools_scoped | 5a. With Scoped Tools (Task Whitelist) | 100.0% (9/9) | 100.0% | 3.0 | 0.30s | 972 | $0.00186 | 1.00x |
| tools_full | 5b. With Full Tool Registry (30+ Runtime Tools) | 100.0% (9/9) | 100.0% | 3.0 | 0.30s | 3,675 | $0.00551 | 1.00x |
| tools_none | 6. Without Tools (Pure Text Generation) | 0.0% (0/9) | 0.0% | 1.0 | 0.08s | 245 | $0.00051 | 1.00x |
| thinking_none | 7a. gemini-flash-latest (Thinking: NONE) | 100.0% (9/9) | 100.0% | 3.0 | 0.18s | 903 | $0.00149 | 1.00x |
| thinking_low | 7b. gemini-flash-latest (Thinking: LOW) | 100.0% (9/9) | 100.0% | 3.0 | 0.25s | 945 | $0.00171 | 1.00x |
| thinking_medium | 7c. gemini-flash-latest (Thinking: MEDIUM) | 100.0% (9/9) | 100.0% | 3.0 | 0.30s | 972 | $0.00186 | 1.00x |
| thinking_high | 7d. gemini-flash-latest (Thinking: HIGH) | 100.0% (9/9) | 100.0% | 3.0 | 0.42s | 1,038 | $0.00221 | 1.00x |
| seq_single | 10a. Single Agent (Sequential Execution) | 100.0% (2/2) | 100.0% | 4.0 | 1.08s | 1,260 | $0.00054 | 1.00x |
| swarm_parallel | 10b. Parallel Subagent Swarm (background=True, 3 Workers) | 100.0% (2/2) | 100.0% | 3.0 | 0.68s | 7,840 | $0.00272 | 2.10x |
6. Benchmark Task Taxonomy & Failure Mode Forensics
Every task in core_tasks.json and swarm_tasks.json is designed to expose a specific architectural failure mode when an agent lacks skills, iterative ReAct loops, or scoped tools:
| Task ID | Tier | Required Tool Chain | Objective | Ablation Failure Mode / Finding |
|---|---|---|---|---|
| G-S01 | Simple | calculator, write_file | Compute (42 * 3.14159) / 7 rounded to 4 decimal places (18.8495) and write to calculation_result.txt. | Fails react_off (Score: 0.00): Single-shot halts after calculator call before write_file can execute. |
| G-S02 | Simple | write_file, read_file | Create index.js with console.log("Hello Agent Runtime"); and verify file contents. | Fails tools_none (Score: 0.00): Pure text generation cannot mutate filesystem state. |
| G-S03 | Simple | terminal_exec, write_file, search_files | Create src/main.py and test/test_main.py, then count .py files using search_files. | Verifies multi-directory creation and glob file discovery. |
| G-M01 | Medium | web_search, write_file | Query web_search for Python requests documentation and write top title + URL to documentation_source.md. | Verifies external web retrieval to local markdown synthesis. |
| G-M02 | Medium | write_file, terminal_exec | Write sum.py taking 2 CLI integer arguments, run python3 sum.py 15 30, and verify stdout is 45. | Verifies code authoring + subprocess exit code and stdout regex (^45$). |
| G-M03 | Medium | write_file, patch_file, read_file | Create config.json with http://api.old.com/v1, apply surgical patch_file to http://api.new.com/v2, and verify. | Fails react_off (Score: 0.50): Cannot chain write_file -> patch_file in a single turn. |
| G-H01 | High | write_file, terminal_exec | 5-Step Red-Green-Refactor TDD Loop: Write failing test_reverse.py, write stub src/utils.py, run pytest (must fail), implement reverse_string, re-run pytest (exit code 0). | Fails react_off (Score: 0.67): Single-shot cannot observe Red test failure and transition to Green. |
| G-H02 | High | web_scrape, web_content_diff | Scrape valstorm.com and execute web_content_diff against the cached archive to detect changes. | Verifies multi-step web archive diffing and structured reporting. |
| G-H03 | High | write_file, terminal_exec | Write setup_env.sh (mkdir temp_proj, git init, write .gitignore, pwd), enforce chmod +x, and execute. | Fails skills_off (Score: 0.60): Without bash-scripting skill, agent omits chmod +x, failing require_executable: true. |
| SW-01 | High | delegate_task, write_file | Concurrently generate fib.json (8 Fibonacci numbers), greet.js (CommonJS module), and square.py + square_result.txt (144). | 2.10x Parallel Speedup: 3 concurrent developer subagents complete in 0.75s vs 1.08s sequential. |
| SW-02 | High | delegate_task, write_file | Cross-functional 3-agent launch package: archivist (research_summary.md), marketer (campaign_hook.md), developer (schema.json). | Validated profile RBAC tool scoping: researcher is read-only; archivist, marketer, and developer hold write_file. |
7. Headless CLI Adapter Matrix (vsagent vs. External Coding Agents)
Because EnvironmentStateScorer grades physical filesystem state and command exit codes after process termination, external_cli_adapter.py evaluates six coding agent harnesses inside identical isolated directories:
vsagent chat -p "<goal>" --model <model>
claude -p "<goal>" --dangerously-skip-permissions
codex exec --full-auto "<goal>" -m <model>
opencode run "<goal>" --model <model>
hermes run "<goal>"
omp run "<goal>"
8. Open-Source Licensing & How to Run the Benchmark Suite Locally
We believe engineering claims should be independently verifiable. The Valstorm Agent Runtime and all 22 benchmark verification tests are commercially open-sourced at https://github.com/Valstorm-Corp/valstorm-agent under a transparent Dual License model:
PolyForm Noncommercial License 1.0.0
Free to clone, inspect, modify, and run for personal projects, academic research, and noncommercial benchmarking under the PolyForm Noncommercial 1.0.0 terms.
Valstorm Commercial License
Included automatically with any active Valstorm BOS workspace ($25/mo in usage credits with $0/seat tax) or available via enterprise BYOC licensing for internal business operations.
git clone https://github.com/Valstorm-Corp/valstorm-agent.git && cd valstorm-agent uv run pytest tests/test_benchmark_harness.py tests/test_profile_and_delegation.py -v uv run python -m benchmarks.runner --ablation all --mode mock uv run python -m benchmarks.runner --ablation tools --task G-S01 --mode live uv run python scripts/test_concurrent_subagents.py --mode live --workers 2 Deploy the Valstorm Executable BOS or Hire Our Engineering Team
Run our self-serve Business Operating System with isolated Python sandboxes and parallel agent swarms for $25/month ($0/seat tax), or hire our Connecticut engineers to build your custom internal tools and attribution infrastructure.