Valstorm Agent Runtime (v4.0):
System-1 Decision Model & Canvas Navigator
Traditional agent architectures rely exclusively on slow, autoregressive System-2 LLM generation for every micro-decision—even when selecting from a closed set of UI controls or choosing a reasoning budget. Below is our reproducible 8-axis ablation study evaluating the Valstorm Agent Runtime (vsagent v4.0.0) with our System-1 Calibrated Decision Model & UI/Canvas Navigator across UK AISI Inspect AI, Terminal-Bench 2.0, τ-bench (pass^k), and GAIA/SWE-bench standards.
The complete Valstorm Agent Runtime (vsagent v4.0.0), the System-1 Decision Model, and this benchmark harness are publicly available at github.com/Valstorm-Corp/valstorm-agent under the PolyForm Noncommercial License 1.0.0. Clone, audit, and reproduce every benchmark locally.
4.00s vs. 13.35s (-70% latency) and -48.5% tokens on UI-01 Cloud Vaults E2E with zero selector hallucination.
Out-of-band routing preserves 100% of provider KV prefix cache while automatically escalating shell hazards to high thinking.
0.15s vs. 1.18s and -89.4% tokens (542 vs. 5,108) across UI-01..UI-04 with 100% pass rate.
Wall-clock speedup via background=True async spawning (66ms launch, 12.07s vs. 18.52s live).
Executive Summary & Primary Empirical Findings (v1.0 → v4.0)
By replacing multi-turn System-2 LLM ReAct DOM/canvas stepping with our parallel System-1 Decision Model (system_one_ui_nav evaluating Choice, Score, Noul, and Box in a single <15ms local pass):
- • Cloud Vaults E2E (
UI-01): Wall-clock dropped from 13.35s to 4.00s (3.34x faster / -70% latency), ReAct turns cut from 4 to 2 (-50%), tokens reduced by -48.5% (9,871 vs. 19,150), and cost cut by 47%. - • ArtCraft 2D/3D Canvas (
UI-02): Navigatingtool_img_to_3d_mesh→actor_hero_pose_rig→cam_main_shot_rig→btn_render_video_scenecompleted in 6.26s vs. 10.30s (1.65x faster) with -47.4% tokens. - • Deterministic Suite (
UI-01..UI-04): Reduced average latency from 1.18s to 0.15s (7.87x speedup) and tokens from 5,108 to 542 (-89.4%) with 100.0% pass rate.
Because reasoning_effort is an out-of-band request parameter rather than part of the prompt text or tool schema array, dynamically routing thinking effort (none, low, medium, high) via a <3ms System-1 evaluation preserves 100% of the provider KV prefix cache:
- • Arithmetic Live (
G-S01): Routed tonone, completing in 5.64s vs. 12.45s (2.21x faster) and saving 28.9% tokens by eliminating duplicate calculator calls. - • Hazard Detection (
G-H03): Staticnonefailed with 88.9% pass rate (omittingchmod +x). Adaptive routing detected the shell hazard (hazard_p >= 0.68), escalated tohigh, and achieved 100% pass rate while saving 20.1% cost vs. static high.
1. Evaluation Methodology & 5 Industry Standards
Rather than inventing an ad-hoc grading script, the Valstorm Benchmark Harness (apps/agent-runtime/benchmarks/) combines five established evaluation standards into a single deterministic runner:
Strict Decoupling
Strictly separates declarative Tasks (tasks/*.json), Solvers/Configs (BenchmarkConfig + runtime adapters), and Scorers (EnvironmentStateScorer in scorer.py).
State Grading
Executes every run inside an isolated temporary workspace (tmp_path) and grades physical post-run filesystem state: created files, content regexes, POSIX executable bits (0o755), and pytest exit codes.
Enterprise Consistency
Tracks both optimistic capability (Pass@k: at least 1 success in k trials) and strict enterprise consistency (pass^k = p^k: probability that all k trials succeed without flakiness).
15-Task Taxonomy
Stratified across Simple (G-S01..G-S03), Medium (G-M01..G-M03), High (G-H01..G-H03), Swarm (SW-01..SW-02), and UI/Canvas (UI-01..UI-04) tasks.
Choice, Score, Noul & Box Telemetry
Provides millisecond-latency local scoring over closed sets: evaluates probability distributions (Choice), normalized utility (Score), epistemic confidence (Noul >= 0.75), and 2D/3D spatial coordinates (Box).
2. The 8 Isolated Scientific Ablation Axes (14 Runtime Factors)
To eliminate confounding variables, each ablation preset (--ablation <axis>) modifies strictly one independent variable while holding all other factors constant:
| Ablation Preset | Factors Isolated | Configurations Compared | What It Measures |
|---|---|---|---|
| decision_model | #11 System-1 UI Navigator vs. Full LLM ReAct | system_one_ui_nav vs. llm_react_ui_nav | Measures wall-clock latency, turn count, token burn, and selector hallucination across DOM UIs and ArtCraft 2D/3D creative crafting canvases. |
| adaptive_thinking | #8 System-1 Adaptive Router vs. Static Thinking | thinking_adaptive_s1 vs. thinking_high_static vs. thinking_none_static | Measures latency reduction and pass rate maintenance when dynamically allocating reasoning effort without invalidating the provider KV prefix cache. |
| skills | #1 With Skills vs. #2 Without Skills | skills_on vs. skills_off | Quantifies how procedural SKILL.md guides prevent guardrail failures such as missing chmod +x on G-H03. |
| react | #3 With ReAct Loop vs. #4 Without ReAct | react_on (max_turns=15) vs. react_off (max_turns=1) | Measures self-correction and multi-step observation value across tool chains and Red-Green-Refactor TDD loops. |
| tools | #5 Scoped Tools vs. #5 Full Registry vs. #6 No Tools | tools_scoped vs. tools_full vs. tools_none | Quantifies token waste and distraction when loading all 30+ platform tools versus injecting only profile-whitelisted tools. |
| thinking | #7 Reasoning Effort Tiers | none vs. low vs. medium vs. high | Measures latency tax, token consumption, and pass rate impact across static Gemini 2.5 thinking budgets. |
| swarm | #10 Parallel Asyncio Swarm vs. Sequential Single-Agent | swarm_parallel (background=True) vs. seq_single | Measures wall-clock speedup when spawning concurrent child subagent coroutines in <70ms. |
| external_clis | #9 Headless CLI Adapters | vsagent vs. claude vs. codex vs. opencode vs. hermes vs. omp | Benchmarks vsagent against 5 external coding agent CLI harnesses inside identical sandbox environments. |
3. Live Empirical Findings: System-1 Decision Model & Canvas Navigation
A. System-1 Closed-Set UI & 2D/3D Canvas Navigation (3.34x Faster, -48.5% Tokens)
In traditional agents, every DOM click or canvas node interaction triggers an entire roundtrip LLM call, generating hundreds of tokens of conversational reasoning before executing a single click. Our System-1 Decision Model (ui_navigate) evaluates closed-set options via local calibration in <15ms:
| Live Benchmark Task | Execution Paradigm | Turns | Wall Latency | Tokens | Cost ($) | Speedup / Savings |
|---|---|---|---|---|---|---|
| UI-01 (Cloud Vaults E2E) | System-1 Decision Model (system_one_ui_nav) | 2 | 4.00s | 9,871 | $0.00159 | 3.34x faster (-48.5% tokens) |
| UI-01 (Cloud Vaults E2E) | System-2 Full ReAct (llm_react_ui_nav) | 4 | 13.35s | 19,150 | $0.00300 | 1.00x baseline |
| UI-02 (ArtCraft 2D/3D Scene) | System-1 Decision Model (system_one_ui_nav) | 2 | 6.26s | 10,821 | $0.00174 | 1.65x faster (-47.4% tokens) |
| UI-02 (ArtCraft 2D/3D Scene) | System-2 Full ReAct (llm_react_ui_nav) | 4 | 10.30s | 20,566 | $0.00322 | 1.00x baseline |
B. System-1 Adaptive Thinking-Level Routing (2.21x Faster, 0% KV Cache Invalidation)
Because reasoning budget (thinking_config) is passed as an out-of-band request parameter rather than injected into the system prompt text, dynamically routing thinking effort between none, low, medium, and high preserves 100% of the provider KV prefix cache:
| Routing Strategy | Task G-S01 Latency | Task G-S01 Tokens | G-H03 Shell Guardrail | Suite Pass Rate | Key Observation |
|---|---|---|---|---|---|
| thinking_adaptive_s1 | 5.64s (2.21x faster) | 9,685 (-28.9%) | PASS (1.00) | 100.0% (9/9) | Routes math to none; detects shell hazard (hazard_p >= 0.68) and escalates G-H03 to high. |
| thinking_high_static | 12.45s | 13,619 | PASS (1.00) | 100.0% (9/9) | Overthinks simple math, triggering duplicate calculator call; +54.7% latency tax. |
| thinking_none_static | 5.20s | 9,540 | FAIL (0.60) | 88.9% (8/9) | Fast on simple tasks but omits chmod +x on shell scripts without skills, failing G-H03. |
C. Tool Surface Scoping (64.8% Token Reduction, 2.8x Cost Savings)
Restricting the agent's tool registry to persona-scoped tools (tools_scoped: 2 tools) versus injecting the full platform registry (tools_full: 30+ tools) reduced total token consumption from 31,200 to 10,976 tokens (-64.8%) and reduced per-run inference cost from $0.00477 to $0.00170 (2.8x reduction) on task G-S01.
4. Mathematical & Empirical Proof of Parallel Subagent Execution
In apps/agent-runtime/tools/delegation.py, invoking delegate_task(..., background=True) schedules an isolated child ReAct coroutine via asyncio.create_task() and immediately returns a task_id in under 70 milliseconds. Calling delegate_task(action="wait", task_id="all") executes asyncio.gather() across all active workers, compressing total wall-clock execution time from the sequential sum (T_1 + T_2 + ... + T_N) down to the single slowest task (max(T_i)):
- Non-Blocking Spawn Time: 66.0 ms (0.066s)
- Worker 1 (developer): 12.08s
- Worker 2 (researcher): 6.44s
- Sequential Sum (D1 + D2): 18.52s
- Actual Parallel Wall-Clock: 12.07s (Saved 6.45s / -34.8%)
- Simulated Per-Worker Delay: 2.00s each
- Individual Worker Durations: 2.10s, 2.09s, 2.09s
- Sequential Sum (D1+D2+D3): 6.28s
- Theoretical Floor (Max Task): 2.10s
- Actual Parallel Wall-Clock: 2.10s (3.00x Speedup)
5. Full 17-Configuration Empirical Scorecard
The table below displays average metrics across all configurations in the Valstorm v4.0 ablation suite:
| Config ID | Configuration Label | Pass Rate | pass^k | Avg Turns | Avg Latency | Avg Tokens | Est. Cost | Speedup |
|---|---|---|---|---|---|---|---|---|
| system_one_ui_nav | 11a. System-1 Decision Model UI/Canvas Navigator | 100.0% (4/4) | 100.0% | 2.0 | 0.15s | 542 | $0.00046 | 7.87x vs. LLM Step |
| llm_react_ui_nav | 11b. System-2 Full LLM ReAct UI/Canvas Navigation | 100.0% (4/4) | 100.0% | 4.5 | 1.18s | 5,108 | $0.00456 | 1.00x |
| thinking_adaptive_s1 | 8a. System-1 Adaptive Thinking Router (adaptive) | 100.0% (9/9) | 100.0% | 3.0 | 0.30s | 727 | $0.00151 | 1.00x |
| thinking_high_static | 8b. Static HIGH Thinking (Without Skills) | 100.0% (9/9) | 100.0% | 3.0 | 0.42s | 798 | $0.00189 | 1.00x |
| thinking_none_static | 8c. Static NONE Thinking (Without Skills) | 88.9% (8/9) | 88.9% | 3.0 | 0.16s | 663 | $0.00116 | 1.00x |
| skills_on | 1. With Procedural Skills (SKILL.md active) | 100.0% (9/9) | 100.0% | 3.0 | 0.30s | 972 | $0.00186 | 1.00x |
| skills_off | 2. Without Skills (Zero SOP Guidance) | 88.9% (8/9) | 88.9% | 2.9 | 0.28s | 725 | $0.00139 | 1.00x |
| react_on | 3. With ReAct Loop (max_turns=15) | 100.0% (9/9) | 100.0% | 3.0 | 0.30s | 972 | $0.00186 | 1.00x |
| react_off | 4. Without ReAct Loop (Single-Shot, max_turns=1) | 55.6% (5/9) | 55.6% | 1.0 | 0.08s | 323 | $0.00062 | 1.00x |
| tools_scoped | 5a. With Scoped Tools (Task Whitelist) | 100.0% (9/9) | 100.0% | 3.0 | 0.30s | 972 | $0.00186 | 1.00x |
| tools_full | 5b. With Full Tool Registry (30+ Runtime Tools) | 100.0% (9/9) | 100.0% | 3.0 | 0.30s | 3,675 | $0.00551 | 1.00x |
| tools_none | 6. Without Tools (Pure Text Generation) | 0.0% (0/9) | 0.0% | 1.0 | 0.08s | 245 | $0.00051 | 1.00x |
| thinking_none | 7a. gemini-flash-latest (Thinking: NONE) | 100.0% (9/9) | 100.0% | 3.0 | 0.18s | 903 | $0.00149 | 1.00x |
| thinking_low | 7b. gemini-flash-latest (Thinking: LOW) | 100.0% (9/9) | 100.0% | 3.0 | 0.25s | 945 | $0.00171 | 1.00x |
| thinking_medium | 7c. gemini-flash-latest (Thinking: MEDIUM) | 100.0% (9/9) | 100.0% | 3.0 | 0.30s | 972 | $0.00186 | 1.00x |
| thinking_high | 7d. gemini-flash-latest (Thinking: HIGH) | 100.0% (9/9) | 100.0% | 3.0 | 0.42s | 1,038 | $0.00221 | 1.00x |
| seq_single | 10a. Single Agent (Sequential Execution) | 100.0% (2/2) | 100.0% | 4.0 | 1.08s | 1,260 | $0.00054 | 1.00x |
| swarm_parallel | 10b. Parallel Subagent Swarm (background=True, 3 Workers) | 100.0% (2/2) | 100.0% | 3.0 | 0.67s | 7,840 | $0.00272 | 2.09x |
6. Benchmark Task Taxonomy & Failure Mode Forensics (15 Tasks)
Every task in core_tasks.json, swarm_tasks.json, and ui_navigation_tasks.json is designed to expose a specific architectural failure mode:
| Task ID | Tier | Required Tool Chain | Objective | Ablation Failure Mode / Finding |
|---|---|---|---|---|
| UI-01 | Medium | ui_navigate, assert_noul | Valstorm Desktop Cloud Vaults E2E: Open Cloud Vaults tab -> select Q4_Strategy.md -> click Share Document + verify 3 parallel Noul assertions. | 3.34x Live Speedup (4.00s vs 13.35s) and -48.5% tokens using system_one_ui_nav over llm_react_ui_nav. |
| UI-02 | High | ui_navigate, canvas_graph | ArtCraft 2D/3D Video Crafting (storytold/artcraft): tool_img_to_3d_mesh -> actor_hero_pose_rig -> cam_main_shot_rig -> btn_render_video_scene. | 1.65x Live Speedup (6.26s vs 10.30s) and -47.4% tokens with 0% selector hallucination. |
| UI-03 | Medium | ui_navigate, epistemic_detect | CRM DataGrid Blocker Self-Healing: Detect active sync warning modal (blocker_detected Noul >= 0.75), route to btn_dismiss_sync_modal, select High-Value Leads filter, and pin view. | Verifies epistemic blocker detection (route_status="handle_blocker") before proceeding with normal workflow steps. |
| UI-04 | High | ui_navigate, timeline_scrub | ArtCraft Multi-Layer Compositing & Timeline Scrubbing: Remove foreground alpha matte -> composite 3D studio backdrop -> scrub timeline playhead to keyframe 24 -> render final shot. | Verifies multi-category closed-set routing across tools, layers, and timeline controls in a single trajectory. |
| G-S01 | Simple | calculator, write_file | Compute (42 * 3.14159) / 7 rounded to 4 decimal places (18.8495) and write to calculation_result.txt. | Fails react_off (Score: 0.00): Single-shot halts after calculator call before write_file can execute. 2.21x faster under thinking_adaptive_s1. |
| G-S02 | Simple | write_file, read_file | Create index.js with console.log("Hello Agent Runtime"); and verify file contents. | Fails tools_none (Score: 0.00): Pure text generation cannot mutate filesystem state. |
| G-S03 | Simple | terminal_exec, write_file, search_files | Create src/main.py and test/test_main.py, then count .py files using search_files. | Verifies multi-directory creation and glob file discovery. |
| G-M01 | Medium | web_search, write_file | Query web_search for Python requests documentation and write top title + URL to documentation_source.md. | Verifies external web retrieval to local markdown synthesis. |
| G-M02 | Medium | write_file, terminal_exec | Write sum.py taking 2 CLI integer arguments, run python3 sum.py 15 30, and verify stdout is 45. | Verifies code authoring + subprocess exit code and stdout regex (^45$). |
| G-M03 | Medium | write_file, patch_file, read_file | Create config.json with http://api.old.com/v1, apply surgical patch_file to http://api.new.com/v2, and verify. | Fails react_off (Score: 0.50): Cannot chain write_file -> patch_file in a single turn. |
| G-H01 | High | write_file, terminal_exec | 5-Step Red-Green-Refactor TDD Loop: Write failing test_reverse.py, write stub src/utils.py, run pytest (must fail), implement reverse_string, re-run pytest (exit code 0). | Fails react_off (Score: 0.67): Single-shot cannot observe Red test failure and transition to Green. |
| G-H02 | High | web_scrape, web_content_diff | Scrape valstorm.com and execute web_content_diff against the cached archive to detect changes. | Verifies multi-step web archive diffing and structured reporting. |
| G-H03 | High | write_file, terminal_exec | Write setup_env.sh (mkdir temp_proj, git init, write .gitignore, pwd), enforce chmod +x, and execute. | Fails skills_off & thinking_none_static (Score: 0.60): Omits chmod +x. Recovered (1.00) by thinking_adaptive_s1, which detects the .sh hazard (hazard_p >= 0.68) and escalates thinking to high. |
| SW-01 | High | delegate_task, write_file | Concurrently generate fib.json (8 Fibonacci numbers), greet.js (CommonJS module), and square.py + square_result.txt (144). | 2.09x Parallel Speedup: 3 concurrent developer subagents complete in 0.67s vs 1.08s sequential. |
| SW-02 | High | delegate_task, write_file | Cross-functional 3-agent launch package: archivist (research_summary.md), marketer (campaign_hook.md), developer (schema.json). | Validated profile RBAC tool scoping: researcher is read-only; archivist, marketer, and developer hold write_file. |
7. Cross-Agent Headless CLI Adapter Architecture
To benchmark vsagent fairly against external coding agents without bias, apps/agent-runtime/benchmarks/adapters/external_cli_adapter.py wraps six headless CLI harnesses inside identical temporary workspaces (tmp_path) and grades all six with the exact same EnvironmentStateScorer:
Valstorm Agent Runtime in-process adapter with full state scoring.
Anthropic Claude Code headless execution mode.
OpenAI Codex CLI headless automated runner.
OpenCode terminal agent adapter.
Hermes coding agent headless wrapper.
Oh-My-Pi automation CLI adapter.
8. Open-Source Repository & Dual-Licensing Architecture
To maintain transparency and allow external AI researchers to reproduce our findings, the Valstorm Agent Runtime (vsagent v4.0.0) and this complete benchmark harness are source-available at github.com/Valstorm-Corp/valstorm-agent under a Dual License model:
PolyForm Noncommercial License 1.0.0
Anyone may clone, inspect, modify, and run the agent runtime and benchmark suite for personal projects, academic research, evaluation, and noncommercial benchmarking without paying licensing fees.
Valstorm Commercial License
Deploying vsagent in production business operations, integrating into client-facing platforms, or executing automated revenue workflows is covered under active Valstorm BOS workspaces ($25/mo with unlimited seats) or enterprise BYOC agreements.
9. Reproduce These Benchmarks Locally (vsagent v4.0.0)
Clone the public repository and run the automated verification suite (31/31 pytest tests passing) and ablation benchmarks:
Valstorm Cloud provides hosted AST security sandboxing, 1-click subagent swarms, and CRM database bindings for $25/month.