Support-ticket routing is a common decision task. Each ticket must be assigned to a queue, a team, or a priority level. The useful output is a discrete choice plus a confidence score, not a generated paragraph. That makes the problem a natural fit for a decision-first model.
The question this post examines is not whether any model can classify tickets in a demo. The question is whether a small decision model can process a realistic batch of 1,000 tickets under conditions that approximate product use, while leaving a clear record of inputs, decisions, scores, and timing. The answer depends on accuracy under the chosen label set, calibration of the confidence scores, latency at batch scale, and behavior on ambiguous or out-of-distribution tickets. Those dimensions must be measured rather than assumed.
This post describes the experimental design used on The Decision Bench. It does not report final accuracy or latency numbers. Those results belong in a follow-up once the data have been collected and the conditions have been fixed.
What “Sort” Means in This Context
Sorting 1,000 support tickets means assigning each ticket to one of a predefined set of categories or queues. Typical categories might include billing, technical support, account access, feature request, and spam or abuse. The exact taxonomy is chosen before the experiment begins and is held fixed.
The model receives the ticket text (subject plus body, or a truncated version if length limits apply) and returns a structured decision: the chosen category and an associated confidence or probability score. Secondary scores, when available, are also recorded. No free-form explanation is required or stored as the primary output.
This definition keeps the task inside the decision-first regime. The downstream system can act on the label and the score without parsing generated text. It also makes logging straightforward: one input, one primary label, one primary score, optional secondary scores, and a timing measurement.
Preparing the Ticket Set
A usable test set of 1,000 tickets needs several properties.
The tickets should reflect real language variation: short and long messages, incomplete sentences, mixed languages if that appears in production, and the usual noise of customer-written text. Synthetic tickets generated from templates tend to under-state the difficulty of boundary cases.
Labels must be assigned consistently. If the experiment uses human-annotated ground truth, the annotation guidelines are written down and the inter-annotator agreement is checked on a sample. If the experiment uses an existing production log with historical labels, the label source and any known biases are noted.
I keep a held-out portion of the 1,000 tickets untouched until the main evaluation run. Early experimentation and prompt or threshold tuning happen only on a smaller development slice. This separation prevents accidental overfitting to the evaluation set.
Length distribution is also recorded. Extremely long tickets may be truncated by the model’s context window or by preprocessing. The truncation rule is stated in the log so that later readers know whether the model saw the full text.
Logging Every Decision
The same checklist used for single-demo runs is applied to the full batch:
Model identifier and version
Runtime and hardware description
Exact input text for each ticket (or a cryptographic hash plus the stored original if size becomes an issue)
Primary label returned
Primary confidence or probability
Secondary scores when the runtime exposes them
Wall-clock time per ticket or per mini-batch
Any error or timeout that prevented a decision
Batch timing is measured in two ways. End-to-end wall-clock time for the full 1,000 tickets gives the throughput figure. Per-ticket or per-mini-batch timings reveal whether latency is stable or whether long tails appear under sustained load.
I prefer to write one structured log line per ticket. Aggregate statistics are computed afterward. Keeping the per-ticket record makes it possible to isolate the exact inputs that produced low-confidence decisions, high-confidence errors, or unexpected latency spikes.
Evaluation Criteria
Accuracy against the ground-truth labels is the first number most people look at. It is necessary but not sufficient.
Calibration matters. If the model reports high confidence on a subset of tickets, the error rate inside that high-confidence subset should be examined separately. A model that is accurate overall but over-confident on its mistakes creates a different operational risk from a model whose confidence tracks actual difficulty.
Coverage at a confidence threshold is equally practical. Many routing systems only auto-assign tickets above a chosen score and send the rest to human review. The fraction of tickets that clear the threshold, and the accuracy inside that fraction, determine whether the model reduces workload or simply shifts it.
Ambiguous and out-of-distribution tickets are collected deliberately. Tickets that human annotators found hard to label, tickets that sit near category boundaries, and tickets that contain unusual formatting or language are flagged in the log. Performance on those slices is reported separately from the overall average.
Latency under batch conditions is recorded with the same care used for single inferences. Mean, median, and upper percentiles are all kept. A model that is accurate but occasionally takes many times the median latency may still be unsuitable for a near-real-time routing queue.
Practical Limits of the Experiment
A 1,000-ticket test is large enough to expose many failure modes and small enough to inspect manually when something looks wrong. It is not large enough to stand in for months of production traffic. Distribution shift, seasonal changes in ticket language, and adversarial or abusive inputs remain outside the scope of a single static set.
The experiment also inherits the limits of the chosen label taxonomy. A model that performs well on five coarse queues may perform differently if the taxonomy is later expanded to twenty finer categories. The results are therefore conditional on the label set that was fixed at the start.
Hardware and runtime conditions are part of the result. A throughput figure measured on a high-end desktop does not automatically transfer to a constrained server process or to a browser-based triage tool. The log states the environment so that the numbers remain interpretable.
What Success Would Look Like
Success is not a single accuracy number. It is a combination of:
Acceptable accuracy on the defined taxonomy
Confidence scores that separate easy from hard tickets well enough to support a review threshold
Stable latency across the batch
Transparent logging that lets an engineer find and examine every high-confidence error
If those conditions are met, the model becomes a candidate for a limited pilot. If any of them fail, the failure is useful data. The Decision Bench treats both outcomes as evidence rather than as marketing claims.
How This Test Connects to the Rest of the Site
Bench Tests posts supply the measurement methods. Deploy Notes examine what changes when the same model moves to ONNX, FastAPI, or mobile. Field Tests place the routing decision inside a larger workflow. Failure Log entries will begin from the exact tickets that produced surprising scores or incorrect high-confidence labels.
The common thread remains the same. Show the input. Show the decision. Show the score. Show the timing. Keep the conditions visible. A small decision model may or may not be ready to sort 1,000 support tickets under product constraints. The only way to know is to run the experiment and read the record.
