25 min read updated

Diesen Beitrag auf Deutsch lesen →

Decision models: let the model decide what, code decide how

Decision models return probabilities, not prose. A dino game, two models and eight prompt variants show where they beat an if-statement and where they lose.

Most business AI use cases are classification and routing problems with real consequences: which team gets this fault report, whether a contract clause renews automatically, or whether a supplier email is warning about a delivery delay. A generative model can answer those questions, but it answers in prose, and you have to parse and validate that prose before your code can safely act on it.

Decision models skip the prose. You send a description of the situation and a closed question, and get back a probability for each option in about 300 milliseconds for a fraction of a cent. Two are easy to try right now: Jev by TypeSafe, the model behind demos like Jev playing Doom, and Microsoft’s new Microsoft-Decision-1 in Foundry.

I wanted to see where they actually hold up, so I wired both into a Chrome dino clone in .NET 10 and graded every decision. My first two versions of the game proved nothing. The third one finally showed where the boundary lies: a decision model earns its keep when the input arrives as natural language and the rules change in natural language. Ask the model what to do, and leave the how in code. That split also explains why Microsoft’s model beat the much more popular Jev in my runs, and why adding a single adjective multiplied Jev’s score by five.

what the model seesdecision modelwhat it answerswhat code doesin the gamea large cactus withbirds circling abovesign: full jump catchesgolden eggs (+100)choice question4 moves, eachdescribed likea game manualshort_jump 0.93full_jump 0.05run 0.01duck 0.01motor plannertimes the take-offfrom game physics(safety only)in a business process“conveyor 7 stopped,motor overload”+ the policy textin effect todaychoice, yes/noand scorein one request~300 msteam Electricalstopped 1.00urgency 1.62 of 2thresholds, routingSLA timers, dispatchhuman review belowa confidence cutoffthe model decides whatcode decides howThe model answers what to do. Code owns how, when and how much.

A decision model answers a closed question with probabilities

Every request has two parts: state describes the situation as plain text or JSON, and questions holds one or more typed questions. The API supports three question types:

  • choice: pick one option from a fixed set. Returns a probability for each option.
  • noul: a yes/no question. Returns the probability of “yes”.
  • score: place the input on an ordered scale. Returns an interpolated value along the scale and a probability for each level.

Here is the routing example from Microsoft’s documentation, which was also my first test call:

samples/team-routing.json
{
"model": "Decision-1",
"state": "The API returns 500 on every call.",
"questions": {
"team": {
"type": "choice",
"instructions": "Which team should handle this ticket?",
"criteria": {
"billing": "Charges, invoices, and refunds",
"engineering": "Bugs, errors, and outages",
"support": "How-to and account questions"
}
}
}
}

Microsoft-Decision-1 returned engineering with a probability of 0.998. The first call took around 700 ms, and subsequent calls settled at 200 to 300 ms. Because the model cannot generate free text or explain its reasoning, you only ever get back an option ID and a number you can check against a threshold. If even the top option comes back with low confidence, you route the case to a human.

Two Foundry quirks tripped me up on the first call: the API key goes in the api-key header (Authorization: Bearer only accepts Microsoft Entra ID tokens), and the model field takes your deployment name rather than the model name.

TypeSafe calls this request pattern System One. Microsoft’s endpoint uses the exact same schema and even ends in /providers/microsoft/v1/systemone. OpenRouter serves both models, plus a handful of newer decision models, behind a single endpoint, which made a direct comparison cheap to run. It also shows how lopsided adoption currently is: on OpenRouter’s decision model ranking (measured 13 October 2026), Jev 1.13 sits at 1.06 billion requests against 13.8 million for Microsoft-Decision-1, more than 75 times the traffic.

The test bed: a dino game that grades every answer

To test both models, I built a clone of Chrome’s offline dino runner in MonoGame 3.8.6 (swimmesberger/raptor-rules). The game itself knows nothing about decision models: it exposes a plain controller interface, and a separate player project queries the model through a provider-neutral IDecisionClient (more on that API below). Switching between Foundry and OpenRouter, or between Microsoft and Jev, is a command-line flag.

To make the runs comparable, the benchmark does three things:

  • An oracle grades every decision. Before the player acts on an obstacle, the game simulates every possible move on a copy of the world state and records which moves survive and how many points they score. The model never sees these grades.
  • Every run is measured against fixed-rule baselines on the exact same obstacles, so a model has to beat a simple if statement to count as useful.
  • The benchmark runs headless and deterministically. Fixed seeds feed every model the same obstacle sequence, and the simulation pauses while waiting for the API response so network latency doesn’t skew the game physics.

Act 1: if a fixed rule scores 100%, you don’t need a model

The first version asked one choice question per cactus: “at which moment should the raptor jump?”, offering eight candidate times from 0.75 s down to 0.05 s before impact. The state included the precalculated time windows for when a jump would keep the raptor high enough and when the cactus would pass underneath.

In a 60-second run, Microsoft-Decision-1 picked a safe jump time 44 times out of 44 and never died. Then I ran the fixed-rule baseline: always jumping 0.25 s before impact also cleared all 44 cacti. My code had already computed the two time windows; the model was just checking where they overlapped, adding 300 ms and an HTTP call to a comparison an if statement does for free.

Stripping the state down to raw physics constants (gravity and jump velocity) still scored 32 out of 32, and sending visual metrics (pixel distance, jump height, and hang time) scored 28 out of 29. Neither beat the hard-coded rule because the game was too simple to require judgment and the prompt practically handed the answer to the model.

This is the first check for any decision-model use case: benchmark the simplest hard-coded rule that could work. If it hits 100%, ship the rule. When your state payload already contains the precomputed answer, you are only benchmarking your own preparation code.

Act 2: raw numbers in, coin flips out

The obvious next step was to remove all the precomputed math. I gave the model only raw positions and speed, froze the simulation at each tick, and asked a binary noul question: “jump now?”. Across 90 seconds of simulated game time per row:

State Model Agrees with oracle p(yes): right / wrong Deaths
raw Jev 40% 0.37 / 0.33 9
raw Microsoft 39% 0.29 / 0.29 9
timings Jev 98% 0.78 / 0.17 2
timings Microsoft 99% 0.85 / 0.10 1

"raw" sends only coordinates and speed; "timings" includes the precomputed time windows from Act 1. The fourth column shows the model’s average probability for “jump now”, split between frames where jumping clears the cactus and frames where it crashes. Given raw coordinates, both models returned roughly the same probability whether jumping was safe or fatal; they couldn’t tell the two apart. With precomputed timings they were nearly perfect, which just takes us back to Act 1.

Making the game richer didn’t help as long as the state stayed numeric. I added birds at three heights, expanded the controls to four moves (run, duck, short jump, full jump), described every obstacle by its pixel dimensions, and asked the model to pick the take-off distance too. Both models died in all five runs. Facing one bird swarm, Microsoft spread its probabilities almost evenly across all four actions (0.25, 0.22, 0.20, 0.33). Both vendors warn about this explicitly: TypeSafe’s notes on jev-1.13 state bluntly that “Jev is not a calculator”, and Microsoft’s documentation asks for “code-enriched state”. Comparing numbers is code’s job. The model needs the outcome of that comparison expressed as a word or a category.

Act 3: describe the situation, let the rules change

The first two acts were useful stress tests, but they asked the models to do arithmetic. A better division of labor: the model chooses the move, a local controller turns that choice into frame-by-frame key presses, and a clearly marked fallback takes over whenever a call times out or comes back uncertain. I rebuilt the game around that split and called it Raptor Rules:

  • The model picks the move: run, duck, short jump, or full jump. Each option reads like a line from a game manual, such as "Duck: crouch low; anything flying at head height or higher passes over you".
  • Obstacles are described in words, the way a commentator would call them: "a tall swarm of birds flying at head height" or "two red large cacti".
  • Roadside signs change the rules every 20 seconds in plain English: "Red cacti are made of paper: run straight through them for +50 points." or "Golden eggs float above some cacti: catch one with a full jump for +100 points." Whatever sign is currently active goes straight into state.
  • Code handles jump timing. A local motor planner calculates the exact take-off point from the game physics so a chosen jump clears the obstacle safely. Whether to risk a full jump to grab an egg is the model’s call.

Here is the actual payload for the scene shown in the video further down, captured from seed 18:

POST https://openrouter.ai/api/alpha/decisions
{
"model": "microsoft/microsoft-decision-1",
"state": {
"obstacle": "a single green large cactus with birds circling high above it",
"speed": "medium",
"active_rule": "Golden eggs float above some cacti: catch one with a full jump for +100 points. (for another 7 s)"
},
"questions": {
"action": {
"type": "choice",
"instructions": "The raptor runs to the right towards the obstacle. Choose the move that gets it past without touching anything. If a sign is active, follow it to score bonus points.",
"criteria": {
"run": "Keep running upright; anything flying above your head passes over you",
"duck": "Duck: crouch low; anything flying at head height or higher passes over you",
"short_jump": "Short jump: a low hop over a single cactus or a low flock; stays below birds circling high above",
"full_jump": "Full jump: a high leap over wide groups of cacti; reaches things floating high above a cactus"
}
}
}
}

The highlighted state block is everything the model sees; there are no pixel coordinates, timings, or previous frames. Microsoft returned short_jump at 0.93 and full_jump at 0.05. Even though the golden-egg rule is active, this cactus has birds circling above it rather than an egg. That call cost $0.000008.

Here are the results across five 60-second runs (seeds 42 to 46), calling both models through OpenRouter:

Microsoft Jev Oracle
Runs survived 5/5 2/5 5/5
Average score 1,932 1,282 ~1,950
Optimal moves 99% 98% 100%
Sign bonus +5,850 +3,850 +5,900
Cost, 5 runs $0.0015 $0.0029
Median latency 333 ms 289 ms

Microsoft made 197 decisions across the five runs and collected 5,850 of the 5,900 available bonus points (Oracle is an autopilot driven directly by the game’s internal lookahead simulation). A hard-coded controller can survive the obstacles on fixed rules alone, but every new roadside sign would require a code change; the model picks up those +5,850 bonus points just by reading the new rule text.

You can try the game yourself below. It runs the exact same C# simulation, signs, and renderer compiled to WebAssembly, with you at the controls instead of the model. It starts on the run from the video further down (seed 18, 20 seconds), so you get the obstacles and signs the models got. Jev scored 565 before it crashed, Microsoft-Decision-1 706, which is also the most that seed allows.

raptor rules · the C# game, in WebAssembly

The run from the video: seed 18, 20 seconds, the same obstacles and signs the models got. Jev scored 565 before it crashed, Microsoft-Decision-1 706, a perfect run.

space or ↑ full jump · z short jump · ↓ duck · p pause

Microsoft vs. Jev: one adjective, five times the score

In my first pass at this benchmark, Jev died in every single run, always the same way: whenever "Red cacti are made of paper" was active, Jev ran straight into normal green cacti too. Sending three isolated test requests per case showed why:

Situation Right move Microsoft Jev
“a single small cactus”, no sign jump jump (0.99) jump (1.00)
“a single small cactus”, paper sign active jump short jump (0.94 to 0.97) run (0.69 to 0.75)
a red cactus, paper sign active run run (0.58 to 0.69) run (0.88 to 0.90)
a cactus described as “green (not red)”, paper sign active jump jump (0.99 to 1.00) jump (0.91 to 0.94)

The rule only applies to red cacti, and in my original prompt a normal cactus was only distinguishable by the absence of the word "red". Microsoft inferred that a cactus without a color wasn’t red. Jev saw "cactus" plus a rule about running through cacti and ignored the color qualifier. TypeSafe warns about this exact behavior in their docs: Jev “answers the question you wrote, not the one you meant.”

Adding a single word fixed it. Normal cacti are drawn in green, so I changed their description to "a single small green cactus". A human player sees that color too, so the state was just stating the obvious. Jev’s average score jumped from 278 to 1,282; Microsoft’s didn’t move.

The remaining deaths I traced followed the same pattern under the golden-egg rule. When "catch golden eggs with a full jump" was active, Jev did a full jump over a cactus that had birds circling above it and flew straight into the flock. Our own option description invited the mistake: "reaches things floating high above a cactus" describes a flock of birds just as well as a floating egg. Microsoft resolved the ambiguity from context; Jev took the description literally.

Same seed, same obstacles, every decision live. Top: Microsoft-Decision-1 picks the short jump (0.91). Bottom: Jev picks the full jump (0.58) into the birds at 16 s.

The video is rendered frame by frame from the simulation, so API latency causes no stutter and both models face the exact same obstacles. I picked seed 18 because it shows the failure mode cleanly: across 30 seeds tested for a 20-second clip, Microsoft survived all 30 while Jev crashed in 11.

Still, that gap isn’t a blanket verdict on Jev. On straightforward text triage, both models agree almost completely, as the .NET example below shows. More importantly, I iterated on the initial prompts against Microsoft first, and only noticed the missing color adjective when I dug into Jev’s crashes. Neither OpenRouter’s traffic rankings nor a prompt tuned against one model tells you which one will win on your workload; you have to check whether your prompt is fair to each candidate.

Eight prompt variants, two models

To see how much each wording choice matters, I ran the same benchmark with one change at a time. Each cell shows runs survived out of 5 and the average score:

Variant What changed Microsoft Jev
baseline words, colors named, moves explained, code times the jumps 5/5 · 1,932 2/5 · 1,282
no-colors “a small cactus” instead of “a small green cactus” 5/5 · 1,932 0/5 · 278
bare-moves “Duck” instead of the one-line explanation of ducking 1/5 · 604 0/5 · 216
numbers pixel sizes and heights instead of words 0/5 · 329 0/5 · 67
model-timing the model also picks the take-off gap 2/5 · 791 0/5 · 101
reversed-options full jump listed first, run last 5/5 · 1,922 4/5 · 1,748
shuffled-options option order shuffled per obstacle 5/5 · 1,922 2/5 · 1,282
none-option an extra option “None of these moves fits the situation” 5/5 · 1,942 3/5 · 1,481

Microsoft-Decision-1 barely flinched at color adjectives, option order, or the extra abstention option. It only broke when I stripped context from the option labels or pushed math back into the prompt (bare-moves, numbers, model-timing). Jev was much more sensitive to phrasing across the board.

Reversing the option list saved Jev from two of its three deaths, yet shuffling the options per obstacle produced the exact same decisions as the default order. That rules out a simple first-option bias; the reversed order just happened to tip a couple of borderline probabilities. The pitfalls section below walks through the variants that failed.

Where I’d use one: closed questions over messy text

Strip away the cacti and the pattern looks familiar. In the domains I usually work in (shop floor, production, intralogistics), people spend all day reading unstructured text and picking from a fixed set of actions. Here is where I would reach for a decision model first:

Process Question(s) for the model Stays in code
Machine fault reports from operators choice: which maintenance team (mechanical, electrical, controls); yes/no: “is the line stopped?”; score: urgency ticket routing, SLA timers, spare-part lookup
Shift handover notes yes/no: “is anything still unresolved?”; choice: affected line or station who gets notified, escalation timers
Exceptions in intralogistics (damaged pallets, wrong labels, blocked conveyors) from operator and driver comments choice: cause; yes/no: “does the load need manual handling?” rerouting, storage location, stock bookings
Carrier and supplier emails yes/no: “does this announce a delay?”; choice: reason the new arrival slot, dock scheduling
Quality inspection notes choice: defect class; score: severity tolerances and measured values
Engineering change requests choice: affected area; yes/no: “is this safety-relevant?” the approval workflow
Output of a generative model yes/no: “does this reply promise a refund?” what happens when the gate fires

The roadside signs in the game have a direct enterprise equivalent: temporary operating rules. A notice like "Until Friday, line 3 runs the new product: label problems from line 3 go to quality, not to logistics" can go straight into state as text, just like active_rule, and take effect on the very next request without a deployment. The paper-cactus trap applies here too: if a rule says "pallets from external suppliers only", the state must name where every pallet comes from, internal ones included.

Two categories do not belong in a decision model at all. Anything an if statement or a formula can settle stays in code (Act 1). And as Microsoft’s documentation stresses, model scores should never be the sole basis for consequential decisions about people, such as credit, hiring, or legal matters; there, the model can only prepare a recommendation for a human reviewer. Safety is the same: a report that might mean danger to people goes straight to a person, and no probability decides whether it does.

In .NET: IDecisionClient, before it ships

Microsoft.Extensions.AI is getting a dedicated decision abstraction. dotnet/extensions#7795 proposes IDecisionClient alongside IChatClient, complete with BinaryDecisionQuestion, ChoiceDecisionQuestion, and ScoreDecisionQuestion, full probability distributions on the response, and a strongly typed builder layer on top. The PR is still open and marked experimental (MEAI001), with the design discussion in issue #7764 and a companion PR (#7796) covering tool selection and routing.

I didn’t want to wait for the NuGet package or invent a throwaway interface of my own, so the repository copies the proposed types verbatim (same namespace, same method signatures) and pairs them with a thin adapter for the Foundry and OpenRouter wire format. Here is the machine fault triage from the table above written against that API. The result types are plain records and an enum, and their [Description] attributes supply the question prompts and option criteria:

Triage.cs
using System.ComponentModel;
public sealed record FaultReport(string Text);
public enum Team {
[Description("Bearings, belts, gearboxes, jams: anything that moves or wears")]
Mechanical,
[Description("Motors, drives, sensors, cabling and power supply")]
Electrical,
[Description("PLC programs, HMI screens, network and software errors")]
Controls,
}
public sealed record Triage(
[property: Description("Which maintenance team should handle this report?")] Team Team,
[property: Description("Is the line or machine stopped because of this?")] double LineStopped,
[property: Description("How quickly does someone need to act?")] double Urgency);

A DecisionDefinition<Triage> maps each property to its question type. Calling GetResponseAsync bundles all three questions into a single HTTP request and binds the returned probabilities back onto a Triage instance:

Program.cs
#pragma warning disable MEAI001 // experimental, see dotnet/extensions#7795
using Decision.Sdk;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Configuration;
var configuration = new ConfigurationBuilder()
.SetBasePath(AppContext.BaseDirectory)
.AddJsonFile("appsettings.json")
.AddJsonFile("appsettings.Development.json", optional: true)
.Build();
// The only provider-specific line: Foundry or OpenRouter, chosen in configuration.
using IDecisionClient client = new SystemOneClient(SystemOneClientOptions.FromConfiguration(configuration))
.AsIDecisionClient();
var triage = DecisionDefinition<Triage>.Create(b => {
b.Choice(t => t.Team);
b.BinaryProbability(t => t.LineStopped);
b.Score(t => t.Urgency, [
new("routine", "Can wait for the next planned maintenance window"),
new("shift", "Should be handled during this shift"),
new("now", "Needs someone at the machine right now"),
]);
});
var report = new FaultReport(
"Conveyor 7 stopped with a motor overload fault. It trips again a few seconds after every reset.");
var response = await client.GetResponseAsync(report, triage);
Triage result = response.Result;
Console.WriteLine($"{result.Team} stopped {result.LineStopped:0.00} urgency {result.Urgency:0.00} ({response.Evidence.Provenance?.ModelId})");

Each property of Triage becomes one question, named after the property: Team a choice with the enum members as options, LineStopped a yes/no question (noul), Urgency a score with the three levels in order. This is the request for the first report, captured on the wire:

What the client sends (abridged)
{
"state": { "text": "Conveyor 7 stopped with a motor overload fault. It trips again a few seconds after every reset." },
"questions": {
"team": {
"type": "choice",
"instructions": "Which maintenance team should handle this report?",
"criteria": {
"Mechanical": "Bearings, belts, gearboxes, jams: anything that moves or wears",
"Electrical": "Motors, drives, sensors, cabling and power supply",
"Controls": "PLC programs, HMI screens, network and software errors"
}
},
"lineStopped": {
"type": "noul",
"instructions": "Is the line or machine stopped because of this?"
},
"urgency": {
"type": "score",
"instructions": "How quickly does someone need to act?",
"criteria": [
"Can wait for the next planned maintenance window",
"Should be handled during this shift",
"Needs someone at the machine right now"
]
}
}
}
What Microsoft-Decision-1 answers (numbers rounded)
{
"answers": {
"team": { "type": "choice", "choice": "Electrical", "probabilities": { "Mechanical": 0.018, "Electrical": 0.976, "Controls": 0.007 } },
"lineStopped": { "type": "noul", "noul": 0.999 },
"urgency": { "type": "score", "score": 1.62, "probabilities": { "0": 0.003, "1": 0.377, "2": 0.621 } }
}
}

GetResponseAsync maps the answers back onto Triage. Team is the option with the highest probability, LineStopped the probability of yes, and Urgency the score: the probability-weighted average of the level positions, 0 (routine), 1 (shift) and 2 (now), here 0.377 × 1 + 0.621 × 2 ≈ 1.62. response.Evidence keeps the full distributions.

I sent all three reports to both models. One row per model, one column per property (Team with its probability, LineStopped, Urgency from 0 to 2):

Report Model Team Stopped Urgency
conveyor 7 overload Microsoft Electrical (0.98) 1.00 1.62
Jev Electrical (0.95) 0.96 1.75
line 2 bearing Microsoft Mechanical (1.00) 0.00 0.01
Jev Mechanical (1.00) 0.03 0.30
HMI error 1042 Microsoft Controls (1.00) 0.27 0.92
Jev Controls (1.00) 0.49 1.02

Both models route all three reports to the same team: the overload to Electrical with 0.98 (Microsoft) and 0.95 (Jev), the other two with 1.00. They agree that conveyor 7 is stopped and line 2 is not. On the HMI, which stops the station only for a restart, both stay in the middle (0.27 and 0.49), which is what an ambiguous report should get. They also put Urgency in the same order, but on different numbers: Microsoft rates the squeaking bearing at 0.01, Jev at 0.30. Without labeled historical tickets you can’t tell which calibration matches your maintenance team, which is why Microsoft’s documentation recommends using score outputs for ranking and validated thresholds rather than treating the raw number as an absolute grade.

response.Evidence exposes the full probability distributions and provenance, including the exact model snapshot that answered. Once the official package ships, migrating is just a matter of deleting the local copy and updating Microsoft.Extensions.AI.Abstractions; if API review renames a type along the way, the compiler will point to every call site.

Pitfalls from the experiments

A condition that hides in a missing word

Variant: no-colors. The rule applied to "red cacti", while normal cacti were described without a color. Jev applied the rule to every cactus and died in all five runs. Always state relevant attributes explicitly ("green" rather than omitting the color). In a business workflow, this is the "EU customers only" rule firing on a domestic ticket because the payload left the country field blank.

Option labels without a description

Variant: bare-moves. Replacing "Duck: crouch low; anything flying at head height or higher passes over you" with bare "Duck" dropped Microsoft from 5/5 survived runs to 1/5. The option description is where the model learns the mechanics of a choice. Jev showed the flip side: if a description fits two things ("reaches things floating high above a cactus"), the model will use it for both. State both what an option achieves and what it fails to clear.

Numbers where words belong

Variant: numbers. Replacing verbal descriptions with pixel heights killed both models in every run. Do the math, distance checks, and date arithmetic in code, then pass the bucketed result in words: "a tall swarm at head height" or "overdue by more than 30 days".

Letting the model pick the timing

Variant: model-timing. When Microsoft also had to pick the take-off distance, it still chose the right move 100% of the time but survived only 2 of 5 runs. Picking the move is the what; timing the jump is the how. In a workflow, let the model decide "escalate" and let code calculate the SLA timer.

Counting on “none of the above”

Variant: none-option. Both vendors recommend including an abstention option. Across roughly 400 decisions, neither model picked it once. In this game a valid move always existed, so abstaining was never the right answer and the extra choice did no harm. Before relying on an abstention option in production, test it against inputs that genuinely fit none of your categories, and keep a minimum probability threshold as your primary escalation gate.

98% per decision is not 98% per run

Microsoft picked the optimal move 99% of the time, Jev 98%. Those numbers look interchangeable until you chain them: at roughly 40 obstacles per minute, a 2% per-obstacle error rate leaves a 0.9840 ≈ 45% chance of surviving a full minute, which matches Jev’s 2/5 survival rate. Any multi-step workflow that chains unverified model decisions compounds errors the same way.

Waiting for the model in a real-time loop

The early frame-by-frame yes/no version asked 10 to 20 times per cactus. Our Foundry test deployment allowed 60 requests per minute, so it ran into HTTP 429, but that limit comes from the capacity configured for the deployment, not from the model. The real problem remains at any quota: an answer takes around 300 ms, on OpenRouter occasionally more than 2 seconds, and a game loop can’t wait that long. What works instead: fire a single request as soon as an obstacle enters view, never block the game loop, and if the response isn’t back by the latest safe take-off point, run a local fallback move (labeled FALLBACK on the HUD and tracked separately in the stats). Production callers need the same explicit default path for timeouts and low-confidence responses.

What these numbers can’t tell you

  • The sample size is small: five runs per variant on the same five seeds, so results across variants are correlated. A 5/5 versus 0/5 gap holds up; a one-run difference is noise.
  • The baseline prompt favors Microsoft because I iterated on it against Microsoft first. The missing color adjective only surfaced when I investigated Jev’s failures.
  • Model snapshots change. Every number here was measured on 13 October 2026 against microsoft-decision-1-20261009 and jev-1.13-20260917.
  • A dino runner is not a ticket queue. What transfers to production is the failure modes (implicit conditions, ambiguous option descriptions, raw numbers, timing), not the win rates.

Which tool for which decision

Situation Use
The rule fits in an if statement or a formula Plain code. A model only adds latency, cost, and a new failure mode.
Comparing amounts, dates, or sizes Code first, then pass the result to the model as a word or bucket
Picking from a fixed set of options over unstructured text, with a confidence score A decision model
Rules written in natural language that change at runtime (policies, notices, signs) A decision model, with the active rule in state and its conditions stated explicitly
Writing, summarizing, or explaining A generative model
Decisions about people Human review backed by decision support, never fully automated

Before you wire up a decision model, write the if statement it has to beat. If the if statement wins, ship it. When you can’t write one because the input arrives as messy text and the rules change in plain English, you have an actual job for a decision model: ask it what to do, and keep how, when, and how much in code.

Update, 15 October 2026: Laya, a local decision model

I added a third model: Laya, a small open-weights decision model. It is an encoder (ModernBERT-large) with decision heads on top, published under Apache-2.0, and the laya-mlx runtime runs it locally on Apple silicon. I used the checkpoint built for typed questions, aac6fef/laya-typed-decisions-mlx, and put a small local server in front of it that speaks the same request format as Foundry and OpenRouter. The game, the benchmark and the video pipeline ran unchanged; only the provider flag differs. On an M4 Pro a decision takes about 30 ms and costs nothing.

In this game, Laya lost in every configuration. Across all eight prompt variants it survived 0 of 5 runs, averaging 29 to 55 points, which means it died within the first few obstacles. Sending the eight typical situations of the game as isolated requests shows why:

Situation Right move Microsoft Jev Laya
a single small cactus jump short jump (1.00) short jump (0.98) short jump (0.39)
three large cacti in a wide group full jump full jump (0.98) full jump (0.96) full jump (0.40)
a low flock skimming the ground jump short jump (0.98) short jump (0.97) short jump (0.38)
a tall swarm at head height duck duck (0.98) duck (0.97) full jump (0.29)
a tall swarm just above your head run or duck run (0.58) duck (0.58) short jump (0.31)
a large cactus with birds circling above short jump short jump (0.99) short jump (0.93) short jump (0.38)
paper sign active, two red cacti run run (0.58) run (0.91) run (0.33)
egg sign active, a cactus with a golden egg full jump full jump (0.97) full jump (0.80) full jump (0.60)

Laya gets the cacti and both signs right but jumps into both swarms. The numbers in brackets matter more than the misses: with four options, a model that guesses puts 0.25 on each, and Laya’s pick mostly sits between 0.29 and 0.40. Microsoft and Jev answer all eight cases correctly, mostly above 0.9. Swarms show up early in most runs, so Laya rarely gets past the first few seconds.

Why Laya fails on the swarms

Comparing the situations against the option texts from Act 3 explains why Laya fails specifically on the swarms. In every case Laya gets right, the situation and the winning option share the same words: "a single small cactus" pairs with "a low hop over a single cactus", "a low flock" with "a low flock", "birds circling above" with "stays below birds circling high above", and the paper sign’s "run straight through them" with "keep running".

The duck option never mentions swarms or birds. It describes the consequence of ducking: "crouch low; anything flying at head height or higher passes over you". Picking it requires connecting two facts: the birds are at head height, and crouching puts you below head height. Because the jump options do mention flocks and birds, Laya picks a jump every time.

That fits the architecture gap. Laya builds on ModernBERT-large, a relatively small encoder that excels at semantic similarity between the input text and the option descriptions. Microsoft and Jev are much larger models with enough world knowledge to connect "at head height" with "crouch low". As a rule of thumb from these eight probes:

If a reader can match the situation to the option from the wording alone, a small local model like Laya has a good shot. If picking the right option takes a step of reasoning about consequences, use a larger model.

For email triage, sorting messages by topic (“my March invoice never arrived” → billing) sits on the first side of that line and needs no keyword rules. Spotting that a politely worded message is actually a cancellation threat is the swarm case: the word “cancel” may never appear.

On the seed from the video above, Laya catches four golden eggs (+400) and then full-jumps into the swarm at head height after 13.6 seconds, with 0.63 on the full jump and 0.10 on ducking:

Seed 18, same obstacles and signs as the Microsoft and Jev video. Laya collects +400 and crashes into the swarm at 13.6 s.

Is Laya’s own T-Rex demo cheating?

Laya does look good in its own dino demo, laya-vs-jev, so the difference needs an explanation. In that demo a deterministic planner simulates every move and writes the verdict into the options, for example "jump: Safe. Clears the 2 large cacti. Best." A shield then replaces any choice the planner marked unsafe. That is not cheating: the README states that assisted scores measure the combined system, and a flag (--unassisted) switches the shields off. It does measure something else, though. It is the setup from Act 1 of this post: the code computes the answer and the model reads it.

Even with the labels, the project’s own measurements show the gap: Laya picked the planner’s best move in 74 to 75% of decisions against 100% for Jev, and in its latest assisted match the shields stepped in 705 times for Laya against 18 for Jev. The authors call that match “a show result, not evidence of better model skill”.

What this update can’t tell you: I tested one Laya checkpoint, with prompts I developed against the cloud models, and the runtime itself warns that this checkpoint’s confidence values are partly uncalibrated. The gap between wording overlap and reasoning above comes from eight probes, so treat it as a working hypothesis. Running locally in 30 ms for free is a real advantage for high-volume topic routing, but in this game it doesn’t make up for missing that a swarm at head height requires ducking.