Training a Jev-style decision model in Domino and pairing it with an LLM

Built in Domino is a series of first-hand accounts from Domino users about what they built, how they built it, and what they learned along the way. Each installment walks through a real workflow, the decisions behind it, and the practical steps to reproduce something similar on your own Domino deployment.

A decision model reads language and hands back real probabilities for specific choices without generating a single word of text. This post covers how to train, deploy, and wire an open-source decision model into Domino to run background agent logic quickly, cheaply, and reliably.

Key Takeaways

Forcing a chatbot to generate JSON for routing and classification is slow, expensive, and fragile.

Decision models run in a single forward pass, returning calibrated probabilities rather than text digits an LLM guessed.

Pairing a decision model with an LLM lets you handle background logic cheaply and save the expensive model strictly for writing text.

Domino handles the entire lifecycle, letting you train, deploy, and monitor these dual-brain agent systems in one project without wrestling with disconnected tools.

Before unpacking the architecture, it helps to look at the exact problem this approach solves.

The industry default: Throw an LLM at it

A ticket routing system built on a standard LLM often looks like this:

You are a support routing assistant. Read the ticket and respond with JSON
like {"team": "billing" | "technical" | "sales", "confidence": 0-1}.
Respond with JSON only. ONLY JSON. Please.

This works, kind of. But forcing an LLM to generate structured output comes with severe problems baked in:

1. It is slow for no reason. An LLM writes its answer one token at a time, and each token is a full pass through the model. A tiny JSON blob still requires 15 to 20 sequential passes.
2. You have to parse it. Sometimes you get JSON. Sometimes you get JSON wrapped in a friendly sentence. Sometimes you get "team": "Billing & Payments", a label that does not exist.
3. The confidence is fake. That 0.9 is two characters the model decided to type because they looked plausible. You cannot calibrate a number that was written instead of measured.
4. It does not scale with questions. If you want the team, the urgency, and the sentiment, that is three calls, or one prompt where the questions bleed into each other and get answered as a blob.

None of this is a knock on LLMs. They're very good at writing and open-ended reasoning. Routing a ticket is neither of those things.

TypeSafe launched Jev on September 15, 2026, and gave this problem a name. They call Jev a "System One model": you hand it some state and a set of typed questions, and it hands back probabilities. Not text generation or parsing. I thought the idea was important enough to understand properly, so I built one.

It's called the Domino Decision Model (DDM). It's open source, runs locally, and speaks the same request format as Jev's System One API. The weights and code are on Hugging Face.

Fair warning up front: the model I trained for this post is a proof of concept designed to prove the architecture works end to end, not to beat Jev at anything.

Language in, decisions out

A decision model reads language. Uniquely, it does not write back. You send it a state and a set of typed questions and the decision model sends back a probability for every allowed answer. DDM supports three core question types:

A request looks like this:

{
  "state": "My Stripe integration has failed for 3 days and I'm losing sales. Help ASAP.",
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team should handle this?",
      "criteria": {
        "billing": "Payment or subscription issues",
        "technical": "Bugs or integration problems",
        "sales": "Pricing or account questions"
      }
    },
    "urgent": {
      "type": "noul",
      "instructions": "The message conveys urgency or time-sensitivity."
    }
  }
}

and the response:

{
  "answers": {
    "team": {
      "type": "choice",
      "choice": "technical",
      "confidence": 0.78,
      "probabilities": {"billing": 0.15, "technical": 0.85, "sales": 0.0}
    },
    "urgent": {"type": "noul", "noul": 0.97}
  }
}

The model cannot answer "Billing & Payments" because that is not one of your options. The model can only pick from the list you give it, so there is nothing to parse. Each probability come straight out of the network, and because the model is trained to predict true likelihoods, you can measure weather those probabilities are accurate.

That changes how you build. When a decision takes milliseconds and returns a probability you can trust, you no longer have to ration it. You can set a confidence threshold in your code, handle the clear cases automatically, and send only the uncertain ones to something smarter, like an LLM or a human. A decision stops being a small, expensive conversation and becomes as cheap as a function call.

Here's a practical tip: question ids are never shown to the model, so put the meaning in instructions and criteria. Naming a question urgent tells the DDM nothing, but the instructions “the message conveys urgency” does.

An LLM vs a decision model that reads language

People blur these two together. Under the hood, a decision model can use the exact same transformer layers as an LLM. It can even start from an LLM's weights. The difference is in what happens at the end.

An LLM runs in a loop:

tokens = prompt
while not done:
    logits = model(tokens)[-1]   # scores over ~150,000 vocabulary tokens
    tokens.append(sample(logits))  # pick one, feed it back in, repeat
answer = parse(decode(tokens))     # and hope

A decision model runs once:

hidden = model(prompt)                          # one forward pass
probs = softmax(score_options(hidden))          # one probability per YOUR option

The LLM's output is text you have to interpret. The decision model's output is a number you can put in an if statement.

What a lot of "open-source Jev" gets wrong

In the few weeks since Jev launched, the open source community exploded with alternatives. A lot of early attempts were just Qwen fine-tunes trained to emit decision-shaped JSON.

Underneath, that's still a chat model. It still generates token by token. Its confidence is still typed out. It still needs a parser. It still answers one question per call, or several questions in one prompt where they can see and influence each other. You get the Jev format without any of the reasons Jev exists.

That said, the space is moving fast. In the last two weeks, several projects got the memo and did it right. Instead of forcing an LLM to generate text, they keep the pretrained backbone for world knowledge, throw away the text generation mouth, and put a proper decision readout on top. Jared Palmer's kev puts a decision readout on a Qwen backbone, and lev does the same on LiquidAI's LFM2.5-350M.

To tell whether a new clone is a true decision model or a chatbot in disguise, ask two things:

• Does it generate tokens to answer? If the answer comes from decoding text, it's an LLM in a decision costume.
• Can it return an option you didn't supply? A real decision model can't, by construction.

How DDM works

DDM follows the architecture that the public evidence about Jev points to, mostly from Archer Hume's black-box study, Jev's Architecture Unmasked, which probed the Jev API with about 10,000 calls. TypeSafe hasn't published their design. What follows is my best reconstruction from that evidence.

Step 1. Pack everything into one sequence

The state goes first, then each question with its options. Special tokens mark the boundaries, and user text can never produce those tokens, so nobody can forge a fake option by typing "option 4:" into a ticket.

[ state ][ Q1 + options + <dec> ][ Q2 + options + <dec> ][ Q3 ... ]

Step 2. Isolate the questions with an attention mask

Every question can read the state and itself. No question can read another question. Here's the mask for a 3-token state and two 2-token questions (■ means "can attend"):

               s1 s2 s3 | a1 a2 | b1 b2
state      s1   ■  ·  · |  ·  · |  ·  ·
           s2   ■  ■  · |  ·  · |  ·  ·
           s3   ■  ■  ■ |  ·  · |  ·  ·
question 1 a1   ■  ■  ■ |  ■  · |  ·  ·
           a2   ■  ■  ■ |  ■  ■ |  ·  ·
question 2 b1   ■  ■  ■ |  ·  · |  ■  ·
           b2   ■  ■  ■ |  ·  · |  ■  ■

Each question's position id restarts right after the state, so a question computes exactly what it would compute if you sent it alone. I tested this: a question's probabilities match to within 0.0000001 whether it's sent solo, packed with siblings, or packed next to a sibling containing a "secret" it's not allowed to see. The state gets encoded once and shared by every question, which is why asking 20 questions has a similar cost to asking one.

Step 3. Read the answers out directly

No loop. For each question, the model compares the hidden state at the <dec> token against the hidden state at the end of each option, and a softmax turns those scores into probabilities. Yes/no questions get a single output between 0 and 1.

Step 4. Calibrate

Training uses log loss, which gives the lowest penalty when the model predicts the true probability. This rewards the model for being honest about its uncertainty instead of guessing with confidence. After training, DDM also fits a temperature per question type on validation data and bakes it into the checkpoint.

The backbone itself is boring on purpose: RMSNorm, rotary embeddings, grouped-query attention, SwiGLU, and an optional sparse mixture of experts. Size is a config choice, from a 0.4M-parameter toy that trains on a laptop CPU to a 13.6B-parameter MoE preset for when you have the GPUs and the patience.

How to choose a readout

The main design question was how to turn hidden states into option probabilities. I looked at two ways to do it.

Approach A, a fixed-slot classifier head. The output layer always produces 256 logits. Slot 1 means "the first option listed," slot 2 means "the second," and so on. Unused slots get masked out before the softmax. It's the normal PyTorch classifier setup everyone already knows.

Approach B, a pointer readout. Score each option against the question using the option's own hidden state. If you give it 3 options, you get 3 scores. If you give it 40, you get 40. The weights don't care how many options there are.

I went with the pointer readout. The fixed head is easier to write, but it teaches the model that "slot 3" is a distinct concept, and slot 3 means nothing. The pointer readout asks the right question: how well does this option answer this question given this state. I kept Jev's 255-option limit for compatibility. That limit is a validation rule; the architecture doesn't need it.

One caveat: options are still read as a list, so order can still nudge the probabilities a little. Jev has the same effect. To mitigate it, the DDM actively shuffles option order during training.

What it takes to build and deploy DDM in Domino

I did all the training and testing in Domino. Because the data, compute, and environments all live in one place, the whole loop fits neatly in a single Domino Project. You train the model using a standard script in a Domino Job (python train.py --preset tiny), test it interactively in a Domino Workspace, and deploy the PyTorch artifact as a standard Domino endpoint with a simple predict() function. Going from raw data to a live API takes minutes..

Once deployed, you can give your agent two helpers. The decision model acts as a cheap, lightning-fast brain for the background decisions, allowing the LLM to keep the writing. Our LLM Gateway 2.0 handles the LLM side, centralizing access to external providers with usage tracking and budgets.

Every decision routes to the DDM endpoint. The agent calls an LLM only when the DDM is unsure or needs prose. The setup looks like this:

Here is how that dual-model flow looks in code. The implementation sets up a four-step loop where DDM handles triage and verification, calling the LLM only when confidence dips or prose is required:

def handle_ticket(ticket):
    # 1. One DDM call, three decisions, one forward pass
    triage = decide(ticket, {
        "team": {"type": "choice", "instructions": "Which team?", "criteria": TEAMS}, 
        "urgent": {"type": "noul", "instructions": "Blocking revenue."}
    })
    
    # 2. Only pay inference cost for the LLM when the DDM isn't sure 
    team = triage["team"]["choice"]
    if triage["team"]["confidence"] < 0.6 or team == "unknown":
        team = ask_llm(f"Pick one of {list(TEAMS)}. Reply with the key only.\n\n{ticket}").strip()

    # 3. The LLM writes the words
    draft = ask_llm(f"Write a friendly reply from the {team} team to:\n\n{ticket}")

    # 4. The DDM checks the LLM's homework
    check = decide({"ticket": ticket, "draft_reply": draft}, {
        "on_topic": {"type": "noul", "instructions": "The reply addresses the problem."}
    })
    
    if check["on_topic"]["noul"] < 0.7:
        return escalate_to_human(ticket, triage, draft)
        
    return send_reply(ticket, team, draft, urgent=triage["urgent"]["noul"] > 0.5)

Notice what the LLM is not doing here: first-pass routing, urgency detection, or policy checks. Those are two forward passes on a small model. The LLM budget goes to the one thing only an LLM can do.

Scoring every production trace

Domino lets you evaluate deployed agents out of band: a scheduled Job pulls production traces, scores them, and logs the scores back, so evaluation never adds latency to the live agent. Most teams use an LLM as the judge, which gets expensive fast, so they score a sample and hope that sample serves as a reasonable production subset to make decisions from.

With a DDM, you can score everything and save the LLM judge for the traces the DDM isn't sure about. The search_agent_traces and log_evaluation calls come from the Domino Python SDK for agents:

from domino.agents.tracing import search_agent_traces
from domino.agents.logging import log_evaluation

EVAL_QUESTIONS = {
    "faithful": {"type": "noul",
                 "instructions": "The final answer is supported by the retrieved context."},
    "task_done": {"type": "score", "instructions": "How completely did the agent resolve the request?",
                  "criteria": ["Not at all", "Partially", "Mostly", "Fully"]},
}

traces = search_agent_traces(agent_id=AGENT_ID, agent_version=VERSION)

for trace in traces.data:
    state = {"inputs": trace.spans[0].inputs, "outputs": trace.spans[0].outputs}
    scores = decide(state, EVAL_QUESTIONS)

    faithful = scores["faithful"]["noul"]
    if 0.3 < faithful < 0.7:  # the DDM is unsure, so escalate this one trace to an LLM judge
        verdict = ask_llm(f"Is this answer supported by its context? Reply yes or no.\n\n{state}")
        faithful = 1.0 if verdict.strip().lower().startswith("yes") else 0.0

    log_evaluation(trace_id=trace.id, name="faithfulness", value=faithful)
    log_evaluation(trace_id=trace.id, name="task_completion", value=scores["task_done"]["score"])

Run that as a scheduled Domino Job, and every trace gets scored, with the expensive judge reserved for the gray zone. The agent ID and version come from the Evaluation setup button on the deployed agent's page.

The pattern in both use cases is the same: decide cheaply, escalate on uncertainty. That pattern only works when your confidence numbers are real, which is the reason decision models exist.

Limitations and what's next

DDM today is a proof of concept with some limitations:

• It trains from scratch. It only knows what's in your data. That's great for a bounded domain like your tickets or your traces, but it's never going to match Jev's general knowledge. The biggest upgrade is starting from a pretrained open-weight backbone and keeping everything else (the packing, the mask, the pointer readout) as is.
• The attention mask is simple. It's fine for short and medium contexts. Long contexts need FlexAttention or a shared prefix cache.
• There's no dedicated server yet. For now, it's a Domino endpoint wrapping a Python client, which is fine for a POC.

Try it

Everything is on Hugging Face. The repo includes the model code, the training script, the Jev-style client, a README that goes deep on the architecture, and a dataset guide you can hand to a coding agent to generate training data for your own domain.

If you build something with it, break it, or find a better readout than mine, tell me. I'd love to hear how it goes.

For the full agent lifecycle around this pattern, the Deploying agentic AI systems blueprint covers deployment and monitoring, and Automated GenAI tracing covers instrumenting traces for the DDM scores.

And the next time you're about to write "Respond with JSON only. ONLY JSON. Please.", ask yourself whether you need a writer or a decider. Most of the time, you need a decider

FAQ

What is a decision model?

A decision model is a neural network that reads text and returns calibrated probabilities for a predefined set of answers. It outputs data that is ready for programmatic logic without generating any text.

How is a decision model different from an LLM?

An LLM generates text one token at a time in a continuous loop. A decision model processes the input in a single forward pass and outputs a probability array for the specific options you provide.

Can a decision model replace LLM-as-a-judge?

Yes, it can handle the vast majority of agent evaluation tasks. Because it is fast and cheap, it can score every single production trace and only escalate the edge cases to a more expensive LLM judge.

Is DDM production-ready?

The current version of DDM is a proof of concept built to demonstrate the underlying architecture. While it successfully proves the concept end to end, it requires training from scratch on your own data and does not have a dedicated serving framework yet.