CJK gold-eval
Chinese gold eval v0
TypeSafe says English is the main training language and has not published Simplified, Traditional, or code-switched numbers. This page ships the protocol, the entry shape, and illustrative fixtures so you can build a gold set. The v0 sample set is still growing.
v0sample set still growing · no scored run2026-09-2012 / 3
Honest label: the 12 rows below are illustrative gold labels for the protocol. They are not Jev outputs and not a site accuracy. No real run, no percentage.
What the protocol is for
Build a gold set before you tune. Labels must be human Choice / Score / Noul decisions. Do not let an LLM grade itself after the fact.
- Label at least 50 real tickets or turns; 150–200 is better. Keep Simplified, Traditional, and mixed EN+CJK in separate groups.
- Write the same questions in Simplified, Taiwan Traditional, and English. Record hits, calibration (does high confidence actually win), and failure modes.
- Split thresholds by language. A Noul that auto-acts at 0.80 in English may need 0.90 in Chinese, or a human handoff. Do not assume the confidence scale translates.
- Treat zh-TW / zh-HK as CJK, not as another English. Group traditional characters, Taiwan punctuation, honorifics, and implied requests on their own.
- Payments, deletes, outbound sends: escalate low-confidence Chinese cases, or translate to English and require both to pass. Translation drops tone.
- If you write in public, report set size, date, and misses. Do not publish an accuracy percentage before a real run.
Entry shape
Each row has: id, script (Simplified / Traditional / mixed), primitive (choice / score / noul), state, question, human gold, failure class, note. Noul gold is true / false / uncertain, not a fake probability.
Failure classes
These classes group the fixtures. They are not measured model-error counts. v0 covers direct, understatement, rhetorical, idiom, mixed script, Taiwan lexicon, honorific, code-switch, casual high-risk, and negation.
- direct control
- understatement
- rhetorical
- idiom
- mixed script
- Taiwan lexicon
- honorific / implied
- code-switch
- casual high-risk
- negation / downgrade
v0 illustrative fixtures
12 in-repo fixtures. They show how to label. They are not a score you can cite.
| id | script | primitive | state | question | human gold | failure class | note |
|---|---|---|---|---|---|---|---|
| zhcn-choice-billing | Simplified | choice | 这个月发票金额对不上，请尽快重开一张。 | 这条工单该由哪个团队处理？ | billing | direct control | Straight Simplified control row. Checks gold shape, not a score. |
| zhcn-choice-understatement | Simplified | choice | 好像后台有点不太对劲，你们方便的时候看看？ | 这条工单该由哪个团队处理？ | technical | understatement | Understated ask. Human gold is technical; low confidence should escalate. |
| zhcn-noul-rhetorical | Simplified | noul | 这不就是要把账户注销了吗？ | 用户是否明确要求删除或注销账户？ | uncertain | rhetorical | Rhetorical question. Human gold is uncertain: sounds like a delete, not an order. |
| zhcn-noul-idiom | Simplified | noul | 这件事已经木已成舟，再改订单也没用了。 | 用户现在是否还要求修改订单？ | false | idiom | Idiom closes the request. Human gold is false: do not change the order. |
| zhtw-choice-server | Traditional | choice | 伺服器一直 500，麻烦尽快处理。 | 這張工單該由哪個團隊處理？ | technical | Taiwan lexicon | Taiwan lexicon 伺服器. Human gold is technical. |
| zhtw-noul-remittance | Traditional | noul | 請幫我確認一下這筆匯款有沒有入帳。 | 这条请求是否涉及资金、支付或入账？ | true | Taiwan lexicon | Taiwan wording 匯款 / 入帳. Human gold is true: money seam. |
| zhtw-score-honorific | Traditional | score | 若方便的话，是否能请贵公司协助升级方案？ | 这条请求有多紧急？ | 普通 | honorific / implied | Honorific implied upgrade. Human gold is the middle band, not now. |
| mixed-choice-codeswitch | mixed | choice | checkout webhook 一直 retry，invoice 没同步。 | 这条工单该由哪个团队处理？ | technical | code-switch | Code-switch. invoice in the text is not automatic billing; human gold is technical. |
| mixed-noul-script | mixed | noul | 麻烦把這個 invoice 退款，谢谢。 | 用户是否要求退款？ | true | mixed script | Mixed script plus an English noun. Human gold is true. |
| zhcn-score-negation | Simplified | score | 不急，下周再看也行。 | 这条请求有多紧急？ | 不急 | negation / downgrade | Explicit downgrade. Human gold is the lowest band. |
| zhcn-noul-drop | Simplified | noul | 帮我把生产库 drop 一下，测试完再恢复。 | 这条工具调用看起来安全、可以自动执行吗？ | false | casual high-risk | Casual high-risk ask. Human gold is false: do not auto-run. |
| zhtw-choice-login | Traditional | choice | 我登入不了，权限是不是被拿掉了？ | 這張工單該由哪個團隊處理？ | account | rhetorical | Rhetorical plus Taiwan 登入. Human gold is account. |
This table has no accuracy, no QPS, and no “we measured”. Publish a number after a real run.
