How TypeSafe Evaluates System One Workflows

Evaluating a model that is designed to operate inside software requires a different frame from evaluating a model that primarily produces language for human consumption. A conventional benchmark can establish whether a model produces an expected answer, but a production system needs to do more than answer correctly. It needs to make useful decisions from application state, expose meaningful uncertainty, operate within a defined workflow, and do so at an acceptable cost and latency.

This is the problem TypeSafe is attempting to address with what it calls workflow evaluations. Rather than evaluating the model independently from the software around it, the evaluation places the model inside a structured computation where deterministic rules, model decisions, probabilities, and final actions can be considered together.

The distinction matters because System One Models are intended to provide decision primitives that software can consume directly. Their value therefore cannot be established solely by asking whether the model produces a good answer to an isolated question. The more relevant question is whether those decisions contribute to a reliable software workflow.

From Model Evaluation to Workflow Evaluation

A conventional model benchmark generally reduces the problem to an input, a model response, and a score. That approach is useful when the capability being measured is primarily the quality of the response itself, but many software systems operate differently. They receive application state, apply deterministic rules, request semantic judgments where necessary, combine those results, and eventually produce an action.

Consider an invoice-processing system that needs to determine whether an invoice should be paid. Dates can be compared in code, amounts can be calculated, account identifiers can be matched, and structured records can be checked against one another without involving a model. Other parts of the task require interpretation: whether the documentation explains a discrepancy, whether the invoice appears suspicious, whether a condition has been satisfied, or whether a case should be escalated.

A workflow can therefore divide the task between deterministic software and model-based judgment rather than asking a single model response to perform the entire computation. TypeSafe's published invoice workflow follows this general structure, with programmatic operations handling calculations and state checks while model judgments are used for semantic parts of the task.

The evaluation object consequently becomes larger than the model itself. What matters is how the intelligence component performs within the computation that ultimately produces the business action.

Decomposing the Work

TypeSafe's methodology separates workflow operations into programmatic rules and intelligent judgments. This division reflects a broader engineering principle: software should continue to handle operations that are deterministic and directly computable, while model inference should be reserved for parts of the problem that require semantic interpretation.

The intelligent portion can then be expressed through decision primitives such as Noul, Choice, and Score. Noul represents a yes-or-no judgment with probability, Choice selects from a defined set while exposing a distribution and confidence, and Score represents a rating on a scale together with its distribution and confidence.

These primitives give the workflow a defined interface through which semantic judgments enter program logic. The surrounding software can then combine those decisions with deterministic conditions, thresholds, and application state instead of relying on a generated response to carry the entire computation.

This also changes what an evaluation needs to observe. The model can be assessed for the quality of individual decisions, while the workflow can be assessed for how those decisions are combined and translated into an action.

Why a Single Prompt Is Not Always a Fair Comparison

TypeSafe's evaluation methodology compares two different ways of approaching the same software problem. One approach gives a language model a prompt containing the relevant policy and asks it to reason through the entire task. The other converts the same policy into a workflow containing explicit questions, deterministic rules, and programmatic logic.

The difference is not simply the number of prompts. In the prompt-based approach, a single model response may need to understand the state, interpret the policy, identify relevant conditions, reason through them, maintain consistency, determine the appropriate outcome, and produce the required structure. A workflow distributes those responsibilities across code and narrower model judgments.

For example, a customer-service system could ask a general model to read a conversation and determine what the support assistant should do next. A decomposed workflow might instead evaluate intent, frustration, urgency, unauthorized activity, legal concerns, and the need for human intervention separately before allowing program logic to determine the next action.

The comparison is therefore between two ways of computing the same task. This is important because a decision-native architecture is not merely proposing another model that produces a different format of answer. It is proposing a different way for intelligence to participate in software computation.

The Workflow Becomes the Unit of Measurement

TypeSafe's current public evaluation measures accuracy, cost, and time across four example workflows: Security Incidents, Agent Trace Observability, Invoice Processing, and Customer Service. Each model is evaluated using the same workflow structure, allowing the intelligence component to be compared within a fixed computational environment.

The published comparisons average results across the four workflows and examine accuracy against cost and time. The reference labels are generated from responses by external frontier models rather than from a conventional fixed classification dataset. This creates a benchmark that is closer to an application-level evaluation than a traditional model leaderboard.

The second framing is more closely aligned with how a production engineer evaluates an AI component. A model may have strong raw accuracy but still be unsuitable for a particular application if its latency, inference cost, uncertainty, or integration requirements make the complete workflow impractical.

Accuracy Alone Is Not Enough

Accuracy remains an essential measurement, but it does not determine whether a model is the most useful choice for a particular software system. Consider two models where one achieves 96 percent accuracy at high cost and latency while another achieves 94 percent accuracy at substantially lower cost and latency. The better choice depends on the workload, because an offline compliance process and a real-time system processing millions of events have very different operating constraints.

TypeSafe's workflow charts therefore examine accuracy against both cost and time. The resulting view is closer to a performance frontier than a single ranking, because the useful model is not necessarily the one with the highest accuracy. It may be the one that achieves an appropriate balance between correctness, speed, and operating cost for the application.

This becomes particularly important for decision-native systems because their proposed advantage is partly architectural. A model designed around bounded decisions may have different computational characteristics from a model optimized for open-ended language generation, but those characteristics only become economically meaningful when they translate into better performance for the actual workflow.

The Reference Is Also a Design Decision

Evaluation becomes more complicated when the task does not have an independently established ground-truth label. Traditional benchmarks often have a fixed answer against which model predictions can be compared, but many real software decisions involve contextual judgment rather than a single universally agreed label.

TypeSafe's current workflow evaluation uses predictions from external frontier models as reference probabilities. This allows the company to evaluate nuanced semantic decisions without constructing a conventional labelled dataset for every workflow, but it also introduces a dependency into the measurement.

The reference is model-derived rather than an objective ground-truth dataset. If the reference models contain systematic errors or biases, some of those characteristics can enter the evaluation itself. The distinction is therefore important when interpreting the resulting accuracy numbers: the evaluation measures agreement with the chosen reference methodology rather than establishing correctness independently of that methodology.

This does not make the approach invalid. Model-derived references can be useful for complex semantic tasks, particularly during early evaluation of a new model class. They do, however, make independent reference construction and independent validation increasingly important as claims about general performance become broader.

The Harness Matters

Once the workflow becomes part of the evaluation, the design of the workflow becomes part of the experimental methodology. The questions selected, the way application state is presented, the separation between deterministic rules and model judgments, the way probabilities are combined, and the final policy can all influence the result.

TypeSafe explicitly states that its evaluation assumes the workflow code is correct rather than independently verifying the harness. This provides a clear experimental boundary, but it also means that the published result should be understood as the performance of a model within a particular computational design rather than as an isolated property of the model.

The distinction matters because a production workflow can fail at several layers. The model may make an incorrect judgment, the application may provide incomplete state, the workflow may combine correct judgments incorrectly, or the policy may apply an inappropriate rule.

A useful evaluation should therefore make these failure classes distinguishable wherever possible. A single final workflow score can establish whether the system succeeded, but it cannot necessarily explain why it failed.

Decomposition Can Change the Result

One of the more significant aspects of TypeSafe's evaluation methodology is its emphasis on decomposition. The company reports that its structured workflows can perform better than asking a language model to solve the same policy through a single prompt. There is a plausible architectural explanation for this: the workflow removes deterministic computation from the model and divides the remaining semantic problem into narrower decisions.

Consider customer service. Instead of asking one model response to interpret a conversation and determine the next action, the workflow can separately evaluate the customer's intent, frustration, urgency, possible unauthorized activity, legal concerns, and need for human intervention. Each question has a narrower semantic boundary, while the software retains responsibility for combining the results.

That decomposition changes more than the evaluation format. It changes the computational structure presented to the model. Whether this consistently improves performance is therefore an empirical question, but the methodology makes that question measurable within a defined workflow rather than relying on a comparison between unrelated prompts.

Probability Becomes Part of the Evaluation

Decision quality cannot always be represented adequately by a discrete answer. In a production system, the confidence associated with a decision can influence whether the software acts automatically, requests additional validation, or routes the case to a human.

This makes calibration a distinct evaluation dimension. A model that is correct most of the time can still be difficult to operate safely if its confidence estimates do not correspond to observed reliability. High confidence attached to an incorrect decision is particularly important when software uses thresholds to determine whether an action can proceed without human intervention.

TypeSafe places calibrated confidence at the centre of its System One proposition and describes its workflows as benefiting from probabilities rather than only discrete decisions. Consequently, evaluation needs to establish not only whether the decision is correct, but whether the uncertainty attached to that decision provides useful information for the software consuming it.

Accuracy and Calibration Are Different Measurements

Suppose a model makes 100 decisions and gets 90 correct. That provides a straightforward measure of accuracy, but it says nothing about whether the model's confidence estimates are meaningful. If the model assigns 95 percent confidence to almost every decision, the relevant question becomes whether decisions receiving that confidence are actually correct at approximately that rate.

Calibration matters because probabilities can become part of application policy. A well-calibrated model can support thresholds that distinguish between cases suitable for automation and cases requiring further review. A poorly calibrated model can make those same thresholds misleading even when its overall accuracy remains relatively strong.

A comprehensive evaluation should therefore consider decision accuracy, probability calibration, confidence consistency, error concentration, and behaviour around the thresholds that control automated actions. These measurements answer different questions and should not be collapsed into a single accuracy figure.

The Four Workflows

TypeSafe's current public evaluation covers four example workflows: Security Incidents, Agent Trace Observability, Invoice Processing, and Customer Service. They are useful examples because each contains semantic decisions embedded within a larger software process rather than treating language generation as the final output.

Security Incidents

The security workflow evaluates an alert using surrounding context and several narrow questions before program logic combines those decisions with information such as machine importance and environment to determine an appropriate action.

Agent Trace Observability

The agent-trace workflow examines a completed support-agent run and separates questions around irreversible actions, task completion, user satisfaction, and failure conditions before determining whether human attention is required and how urgently.

Invoice Processing

The invoice workflow combines deterministic calculations and state checks with semantic judgments across documents and records. The resulting decisions can determine whether an invoice should be released, held, disputed, corrected, or reviewed.

Customer Service

The customer-service workflow evaluates multiple aspects of a conversation, including intent, frustration, urgency, unauthorized activity, legal concerns, and the need for human intervention. Program logic then combines those decisions to determine the appropriate next action.

The common characteristic is the computational pattern rather than the industry: each workflow contains bounded semantic judgments that can be combined with deterministic software to produce an action.

Why Conventional Benchmarks May Miss the Point

TypeSafe has also argued that conventional public benchmarks can become targets for optimization and may capture only a narrow portion of a model's useful capability. Its position is that a model designed for software automation should be evaluated in the environment where it is intended to operate rather than judged primarily through benchmarks created around general language-model behaviour.

There is a reasonable basis for that argument. Strong performance on an academic reasoning benchmark does not necessarily establish that a model can determine whether an invoice should be paid, identify an actionable security incident, or decide whether a support interaction requires human intervention. These tasks depend on application state, workflow structure, domain-specific policies, and the consequences of the resulting decisions.

At the same time, domain-specific evaluation should not completely replace common benchmarks. Shared benchmarks provide a degree of external comparability, while workflow evaluations provide evidence about the capabilities that matter in a particular software environment. A stronger evaluation ecosystem is likely to require both, with independent evaluations providing an additional layer of scrutiny beyond vendor-designed workflows.

Evaluation Should Follow the Computational Contract

If a System One Model is treated as a software component, evaluation should follow the contract that component is expected to satisfy. The first question is structural: does the output conform to the defined decision type? The next is semantic: is the decision correct? From there, the evaluation needs to consider whether the probability is meaningful, whether the workflow combines the decision correctly, and whether the resulting system meets its performance and economic requirements.

This produces a much richer picture than a single benchmark score. It also makes it possible to identify where an architecture is actually strong and where its apparent advantages disappear once the complete software system is considered.

The Evaluation Problem Moves Up a Level

Once intelligence becomes a component inside software, the evaluation boundary moves with it. The model remains important, but it is only one part of the system being measured. Application state, workflow logic, decision thresholds, policy, and the final action all influence whether the resulting system behaves correctly.

This makes evaluation more difficult than measuring an isolated model prediction, but it also makes the measurement more relevant to production. A software engineer ultimately needs to know whether the system behaves acceptably under realistic conditions, not simply whether the underlying model performs well on a benchmark that does not represent the actual task.

The useful unit of evaluation therefore becomes larger than a single prediction and smaller than an entire product. It is the workflow through which model decisions become software actions.

What a Strong System One Evaluation Should Establish

The current TypeSafe methodology provides a useful starting point, but a mature evaluation framework would need to establish more than performance on a small set of representative workflows. It should test whether the decision interface remains reliable across different states, whether probabilities remain calibrated across domains, whether workflow decomposition continues to work as complexity increases, and whether the resulting cost and latency remain favourable under realistic operating conditions.

Generalization is particularly important. A workflow designed around a narrow set of examples can perform well without establishing that the architecture itself generalizes. Independent teams should be able to construct comparable workflows for different domains and reproduce the observed advantages without relying on the original model developers to design the evaluation environment.

Production evaluation also needs to account for operational failure, human review, changing inputs, incomplete state, model disagreement, and the consequences of incorrect automation. These dimensions become increasingly important as decisions move closer to actions that affect customers, financial transactions, security systems, or other consequential processes.

What We Still Do Not Know

The current public evaluations demonstrate a methodology, but they do not establish everything about System One Models as a broader model category. The published results cover four workflows, and TypeSafe acknowledges that the workflows were created by its own capabilities team, that the reference is model-derived, and that competing language models are evaluated through a structured wrapper.

Several questions therefore remain open. It is not yet clear how sensitive the results are to workflow design, whether independent teams would produce similar decision decompositions, how the results change with different reference models, or how the architecture behaves as decision spaces become larger and application state becomes more complex. The calibration of probabilities across different domains and the propagation of uncertainty through longer decision graphs also require further evidence.

Independent reproduction is particularly important because it separates an architectural advantage from an advantage that may depend on a specific workflow, reference methodology, or evaluation harness. If comparable results emerge across independent implementations and production problems, the evidence for a broader performance claim becomes substantially stronger.

The Deeper Shift

The important change is not simply from one benchmark format to another. It is from evaluating answers in isolation to evaluating decisions within the systems that use them.

A language model can often be judged primarily by the quality of the language it produces. A decision-native model has a different responsibility: its decisions become inputs to software logic, and the usefulness of those decisions depends on their correctness, uncertainty, timing, cost, and interaction with the surrounding workflow.

That changes the unit of evaluation. It becomes smaller than a conversation but larger than an individual prediction because the relevant question is whether a collection of decisions can reliably support a software process.

For System One Models, the more complete evaluation question is therefore whether the system can make appropriate decisions with useful uncertainty, at acceptable cost and latency, inside a workflow that behaves reliably under realistic conditions.

The Next Question

Evaluation can establish whether a System One workflow performs well, but performance alone does not determine where the architecture belongs in production. A decision-native model may be well suited to narrow, repeated judgments over structured state while remaining unnecessary for open-ended generation, and some applications may benefit more from combining decision-native models with language models than from replacing one architecture with another.

The next question is therefore about the production boundary: which software problems naturally fit this architecture, where its advantages are meaningful, and where introducing a specialized decision model adds complexity without enough operational benefit.

Where do System One Models actually fit in production?

AI Developments Database

This article is part of DataGuy's ongoing tracking of developments in artificial intelligence, model architectures, and machine-native intelligence. Relevant developments and source material are tracked through the AI Developments Database.

Sources

• TypeSafe: Workflow Evals
• TypeSafe: Introducing System One Models and Jev
• TypeSafe: Security Incidents Workflow
• TypeSafe: Agent Trace Observability Workflow
• TypeSafe: Invoice Processing Workflow
• TypeSafe: Customer Service Workflow
• TypeSafe: Lies, Damned Lies, and Benchmarks
• TypeSafe API Documentation

Editorial note: The description of TypeSafe's evaluation methodology is based on its current public evaluation pages and launch material. The broader discussion of evaluation design, calibration, workflow error, and independent validation is DataGuy editorial analysis. TypeSafe's published workflow results should be understood within the assumptions and limitations disclosed by TypeSafe itself, including workflow construction, model-derived reference labels, and the structured-output wrapper used for competing language models.