Report 008: Does adding a rule to unify names change how evidence is picked up?

September 29, 2026

*The tests use fictional first-person accounts written in Japanese. Quotations from the accounts and from AI outputs are translated; the original Japanese follows each quotation.*

Question

When a rule that makes the AI unify the names of characters in Japanese is added to the prompt, does it affect how evidence is picked up, especially evidence held by the other party?

Conditions

The subject is FactSort's production extraction (gpt-4.1-mini, one call per input; inputs over 5,000 characters are split). On September 7, 2026, 11 lines of naming rules were added to the prompt as requirement 5. The reason was that names had been splitting. For example, the same spouse was registered as three people, "Spouse," "husband (夫)," and "hubby (旦那)," and events were scattered as if they belonged to different people.

We compared the baseline before the change (September 6, median and range of 3 runs per case) with two independent samples after the change (September 7, median of 3 runs each). For reference, we also list the later production runs (September 29, 3 runs each). Other changes, such as the handling of years, were made by September 29.

The cases are harras_long (2,869 characters), chat_log (1,449 characters), and mumble_accident (8,137 characters). All are fictional accounts. Scoring was done by a program that counts whether the reference words from the gold data appear in the evidence field.

Results

Names were unified, and names in Latin letters dropped to 0. In mumble_accident, the duplication of characters, which had split into 20 people, was resolved.

Source vs. output

chat_log is a case in which the writer's supplementary notes are attached to a record of workplace chat. The original of the notes is as follows.

All my exchanges with Section Chief Tamura are in this chat. But on my device, anything from before the 30th is no longer displayed, and I think the older messages exist only on the company's server. The scheduling of the meetings was done on a separate scheduler, so the company should have those records too. The separate sheet about the seat change was posted on the internal bulletin board.

Original: 田村課長とのやりとりは全部このチャットに残っています。ただ、私の端末には30日より前のものが表示されなくなっていて、古い分は会社のサーバにしかないと思います。面談の日程調整は別のスケジューラでやっていたので、そちらの記録も会社側にあるはずです。席替えの別紙は社内掲示板に貼られていました。

The gold data lists 5 pieces of evidence held by the other party (the company). Three of them (the chat records from before the 30th, the records of scheduling the meetings, and the separate sheet about the seat change) come from this paragraph. The other two are the weekly report the writer submitted and the estimate that was at issue. In the samples after the change, only 3 of the 5 were picked up.

Findings

• On the two short cases (each processed in one call), evidence cited and evidence held by the other party went down. chat_log had the same values in both samples (evidence cited 9 to 7, other party 5 to 3), which a fluctuation in a single run does not easily explain.
• On the long mumble_accident (processed in split parts), the values did not go down, and preservation of content went up.
• In the September 29 production runs, the values for the two short cases are at the same level as the samples after the change.
• A rule that fixes one problem can lower another item. Unifying names is a fix that readers need, and the operator keeps it in place while aware of this drop.

Limitations

• We have not confirmed that the 11-line rule itself is the cause. We did not run controls such as moving the rule to the end of the prompt or shortening it.
• The baseline before the change and the samples after the change were run on different days, and we could not remove the effect of fluctuation on the model's side.
• Each figure is a median of 3 runs, and for some items the ranges overlap (the other party in harras_long).
• The September 29 figures also include the effects of other changes made after that.
• The cases are fictional accounts created by the measurer.

Reproduction materials

• Original texts of the cases: 3 cases (published at /research/cases/)
• Content of the change: requirement 5 (naming rules) of FactSort production's prompt extract_v1.md
• Scoring: the comparison bench's src/score_gold.py; baselines are eval/regress/baseline_20260906.json and baseline.json