Email triage is a classic decision task. Each message must be placed into one of a small number of buckets—inbox, spam, or phishing—so that downstream filters, quarantine systems, or human review queues can act. The useful output is a discrete label plus a confidence score. A generated paragraph explaining the choice is rarely consumed by the systems that actually move the mail.
Laya, as a decision-first model, is a natural candidate for this scene. The question is not whether it can produce a label on a handful of clean examples. The question is whether it can process a realistic collection of messages, keep confidence scores meaningful across the three classes, and leave a log that supports later calibration and failure analysis. This post describes the field-test design used on The Decision Bench. It does not report final accuracy or false-positive rates. Those numbers require a fixed corpus, a fixed model version, and a completed measurement pass.
Defining the Three-Way Decision
The label set is deliberately small: inbox, spam, phishing. Inbox covers legitimate messages the user expects. Spam covers bulk or unwanted commercial mail that is not actively malicious. Phishing covers messages that attempt to steal credentials, deliver malware, or otherwise deceive the recipient into a harmful action.
Keeping the taxonomy coarse has two advantages. First, it matches many production triage systems that only need to decide whether a message is safe to deliver, should be quarantined, or should be dropped. Second, it makes the decision boundary inspectable. When the model confuses spam with phishing, or phishing with inbox, the error is immediately meaningful.
The model receives the subject line and body text (or a truncated version when length limits apply). Headers, attachment names, and URL lists can be added as extra features in later variants; the base test uses text only so that the contribution of the language model itself remains clear. The output is a single primary label and its confidence score. Secondary scores are recorded when the runtime exposes them.
Building a Usable Test Corpus
A useful corpus mixes sources and difficulty levels.
Legitimate inbox messages should include ordinary personal and work correspondence, newsletters the user has subscribed to, and transactional mail (receipts, shipping notices). Spam should include both obvious commercial blasts and more polished marketing that sits near the inbox boundary. Phishing should include credential-harvesting attempts, fake account alerts, and messages that impersonate known services.
I keep a held-out slice untouched until the final evaluation run. Early threshold tuning and error analysis happen only on a development portion. This separation prevents the evaluation numbers from reflecting accidental overfitting to the test messages.
Length distribution, language variety, and the presence of URLs or obfuscated text are recorded as metadata. Extremely long messages may be truncated; the truncation rule is stated so that later readers know what the model actually saw. Messages that human reviewers themselves found ambiguous are flagged; performance on that slice is reported separately from the overall average.
Logging Every Classification
The logging checklist follows the same pattern used in other field tests:
Model version and runtime
Exact subject and body (or a stable hash plus stored original)
Primary label returned
Primary confidence
Secondary scores when available
Wall-clock latency
Any human-assigned ground-truth label
Flags for ambiguous, adversarial, or out-of-distribution examples
One structured record per message keeps the analysis flexible. Aggregate accuracy is computed later. The per-message record makes it possible to pull every high-confidence phishing miss or every low-confidence inbox message without re-running the model.
Latency is measured from the moment the text is ready for inference until the structured decision is available. Cold-start and warm-run figures are kept distinct when the model is loaded once and then applied to a batch.
Evaluation Slices That Matter
Overall accuracy is the first number most people ask for. It is not the most operationally important.
Precision and recall on the phishing class usually carry higher cost. A false negative (phishing labeled as inbox) can reach the user; a false positive (legitimate mail labeled as phishing) creates review load and potential delivery delays. I therefore report phishing-specific metrics alongside the three-way confusion matrix.
Confidence behavior is examined by slice. High-confidence inbox predictions should be overwhelmingly correct if the score is to be used for auto-delivery. High-confidence phishing predictions should be reliable enough to justify automatic quarantine. Low-confidence messages form the natural review queue; the size of that queue and the error rate inside it determine whether the model reduces human effort or simply relocates it.
Ambiguous and adversarial messages are given their own slice. Short messages with little content, messages that mix legitimate and suspicious signals, and messages that use homoglyphs or unusual encoding are collected deliberately. Performance on those examples is reported separately so that average numbers do not hide brittle spots.
Practical Limits of the Field Test
A static corpus, even a carefully built one, does not capture concept drift. Phishing campaigns change. Spam templates evolve. Seasonal traffic shifts the base rates of the three classes. Results on a fixed set are therefore a snapshot, not a permanent guarantee.
The text-only feature set also omits signals that production systems often use: reputation of sending domains, authentication results (SPF, DKIM, DMARC), URL redirection chains, and attachment analysis. A model that performs well on text may still need those additional signals before it can be trusted as a sole decision maker.
Finally, the three-way taxonomy itself is a simplification. Some organizations need finer categories (bulk newsletter versus true spam, spear-phishing versus generic campaigns). Extending the label set changes both the difficulty and the calibration behavior; the results of the coarse test do not automatically transfer.
What Success Would Look Like
Success is not a single accuracy figure. It is a combination of:
Acceptable precision and recall on the phishing class under the chosen operating threshold
Confidence scores that separate clear cases from ambiguous ones well enough to support an auto-action versus review split
Stable latency across the corpus
A complete log that lets an engineer inspect every high-confidence error
If those conditions are met, the model becomes a candidate for a limited pilot behind a human review queue. If any of them fail, the failure is useful data. The Decision Bench treats both outcomes as evidence.
How This Scene Fits the Rest of the Site
Field Tests place decision models inside concrete workflows. Bench Tests supply the accuracy, calibration, and latency methods that are reapplied to the email corpus. Deploy Notes examine what changes when the same classifier moves from a Python process to ONNX, FastAPI, browser, or Android. Failure Log entries begin from the exact messages that produced high-confidence mistakes or unexpected score distributions.
The governing rule stays the same. Show the message. Show the decision. Show the confidence. Show the timing and the conditions. When those elements remain visible, the question “Can Laya sort inbox, spam, and phishing?” receives an answer grounded in evidence rather than in a polished demo.
No comments yet — be the first to share a thought.