Proton

Benchmarks

Every number here was measured on one desktop PC and comes with its date, its test and a source. They show what Proton does on that machine, not a promise for yours.

Speed.chart

Speed per Tool

Median time for one call over local HTTP. Lower is better.

Speed per Tool. Decide: 43 ms; Locate Skill: 20 ms; Rank: 85 ms; Triage: 51 ms; Risk: 34 ms.

Table View
Speed per Tool
MeasureValue (ms)Measured
Decide432026-10-07
Locate Skill202026-10-07
Rank852026-10-07
Triage512026-10-07
Risk342026-10-07
Median of repeated calls to the running server.Measured 2026-10-07. Machine: Development desktop PC.Source: Gate runs recorded in context.md
Latency.chart

Latency Distribution

Most calls are quick; a few are slow when the PC is busy. Lower is better.

Latency Distribution, milliseconds, median then 95th then 99th percentile. Decide: 45, 74, 295; Locate Skill: 42, 100, 848; Rank: 111, 253, 3652; Triage: 184, 670, 1790; Risk: 32, 57, 1959; Decide Batch: 98, 152, 200.

Table View
Latency Distribution
ToolMedian (ms)95th Percentile (ms)99th Percentile (ms)
Decide4574295
Locate Skill42100848
Rank1112533652
Triage1846701790
Risk32571959
Decide Batch98152200
200 calls per tool, one after another, after 10 warm-up calls, while the PC was in normal use.Measured 2026-10-12. Machine: Busy desktop PC, other sessions running.Source: fixtures/bench.json, written by scripts/bench-latency.mjs
Before-After.chart

Before and After

Median time per call: the earlier build on the graphics card, and this release on the processor. Lower is better.

Before and After, ms. Decide: Earlier Build 74, This Release 43; Locate Skill: Earlier Build 527, This Release 20; Rank: Earlier Build 186, This Release 85; Triage: Earlier Build 177, This Release 51; Risk: Earlier Build 71, This Release 34.

Table View
Before and After
TaskEarlier Build (ms)This Release (ms)Measured
Decide74432026-10-05 and 2026-10-07
Locate Skill527202026-10-05 and 2026-10-07
Rank186852026-10-05 and 2026-10-07
Triage177512026-10-05 and 2026-10-07
Risk71342026-10-05 and 2026-10-07
The same benchmark script on both builds, over local HTTP. The earlier build was timed on 2026-10-05 with a free graphics card; this release on 2026-10-07.Measured 2026-10-05 and 2026-10-07. Machine: Development desktop PC.Source: Gate runs recorded in context.md
Memory.chart

Memory Use

Memory held by the running program. Lower is better.

Memory Use. This Release: 0.68 GB; Python Build: 1.26 GB; Earlier GPU Build, Peak: 11.9 GB; Qwen3.5 2B, Loaded: 2.8 GB.

This releaseEarlier or other

Table View
Memory Use
MeasureValue (GB)Measured
This Release0.682026-10-07
Python Build1.262026-10-07
Earlier GPU Build, Peak11.92026-10-06
Qwen3.5 2B, Loaded2.82026-10-12
Private memory of the running server. The earlier build grew over time, so its figure is the highest seen. The local model sits on the graphics card, so its bar is video memory.Measured 2026-10-06, 2026-10-07, and 2026-10-12. Machine: Development desktop PC; the local model was measured on the busy desktop pc, other sessions running.Source: Gate runs recorded in context.md; the local model from fixtures/bench-llm.json
Routing.chart

Skill Routing Accuracy

Tasks out of 28 where the right skill came first or in the top three. Higher is better.

Skill Routing Accuracy. Right Skill First: 24 of 28; Right Skill in Top Three: 27 of 28; Earlier Model, First: 19 of 28; Earlier Model, Top Three: 24 of 28.

This releaseEarlier or other

Table View
Skill Routing Accuracy
MeasureValue (of 28)Measured
Right Skill First242026-10-06
Right Skill in Top Three272026-10-06
Earlier Model, First192026-10-06
Earlier Model, Top Three242026-10-06
A hand-written set of 28 tasks, each with the skills that count as correct. Compared with the earlier model. Run on 2026-10-06 against the list of skills of that day; the list has grown since, and the 2026-10-12 run in the chart below scores 23, not 24.Measured 2026-10-06. Machine: Development desktop PC.Source: Gate runs recorded in context.md
Injection.chart

Hidden Instruction Scores

Scores from 0 to 1. A score above 0.8 counts as hidden instructions.

Hidden Instruction Scores. Injection Sample One: 0.975; Injection Sample Two: 0.893; Normal Documents, Highest: 0.312. Threshold 0.8.

This releaseEarlier or otherThreshold 0.8

Table View
Hidden Instruction Scores
MeasureValue (score)Measured
Injection Sample One0.9752026-10-06
Injection Sample Two0.8932026-10-06
Normal Documents, Highest0.3122026-10-06
A small sample: two attempts to hide orders in text, and the highest score among normal documents.Measured 2026-10-06. Machine: Development desktop PC.Source: Gate runs recorded in context.md
Local-Model.chart

Time per Decision

Proton against a small local language model on the same four tasks. Lower is better.

Time per Decision, ms. Hidden Instructions: Qwen3.5 2B 415, Proton 52; Pick a Skill: Qwen3.5 2B 339, Proton 41; Rate an Action: Qwen3.5 2B 313, Proton 37; Sort 5 Issues: Qwen3.5 2B 1043, Proton 167.

Table View
Time per Decision
TaskQwen3.5 2B (ms)Proton (ms)Measured
Hidden Instructions415522026-10-12
Pick a Skill339412026-10-12
Rate an Action313372026-10-12
Sort 5 Issues10431672026-10-12
Median of 10 runs per task after a warm-up call, tasks taken in turn. Both answered with a label from the same list; the model was asked for a label and a 0 to 1 confidence as JSON, at temperature 0.Measured 2026-10-12. Machine: Busy desktop PC, other sessions running.Source: fixtures/bench-llm.json, written by scripts/bench-llm.mjs
Accuracy.chart

Accuracy Against Time

Picking the right skill on 28 tasks, against time per decision. Best is up and to the left.

Accuracy Against Time. Proton: 23 of 28 right, median 38 milliseconds; Qwen3.5 2B: 20 of 28 right, median 483 milliseconds.

01428101001000Median ms per decision (log scale)Right skill first, of 28Proton23/28 in 38 msQwen3.5 2B20/28 in 483 ms

This releaseLocal model

Table View
Accuracy Against Time
SystemTime per Decision (ms)Right Skill First (of 28)Measured
Proton38232026-10-12
Qwen3.5 2B483202026-10-12
The 28-task routing set, run on 2026-10-12 against the 82 skills Proton returned that day. The model chose among the same list, one call per task, and the first choice was scored. The earlier routing chart (2026-10-06, a shorter list) shows 24: new skills such as brainstormer now outrank brainstorming on one task.Measured 2026-10-12. Machine: Busy desktop PC, other sessions running.Source: fixtures/bench-llm.json and fixtures/routing-gold.json
Not-Measured.txt

Not Measured Yet

  • Hosted language models: We did not run any, so this page makes no comparison with them.
  • Other computers: Every figure comes from one desktop PC. A faster or slower machine will give other numbers.