Report 005: Does splitting the organizing work into stages make it better?

September 28, 2026

*The tests use fictional first-person accounts written in Japanese. Quotations from the accounts and from AI outputs are translated; the original Japanese follows each quotation.*

Question

If the AI builds the people, the evidence, the timeline, and the uncertainties in separate stages, is the result better than asking it to build everything in one pass? We checked this with a low-cost model (gpt-4.1-mini).

Conditions

The cases are harras_long (2,869 characters), chat_log (1,449 characters), and mumble_accident (8,137 characters). All are fictional accounts.

The staged method has four stages: (1) people, (2) evidence, (3) timeline, (4) uncertainties, guesses, and questions. Each stage receives the full original text and the results of the earlier stages, and outputs only its own part. The output format was enforced for each stage. After stage 4 there is a checking step. It annotates events against the original text and sorts the evidence by who holds it. (It uses an external classification API, "Jev." How well Jev's judgments perform is not covered here.) The code safeguard that removes years was not applied.

The comparison is FactSort's production extraction (run on September 7, 2026). It outputs everything in a single call, with no enforced format. Inputs over 5,000 characters are split and processed in parts, and then a step that merges the structure and the year safeguard are applied.

Both used gpt-4.1-mini, and each case was run 3 times. The staged runs were made on September 28, 2026. (The September 27 runs were redone after a bug in the transcription was fixed.)

Scoring was done by a program that counts whether the reference words from the gold data appear in the output. "Evidence cited" is the evidence listed. "Other party" is the evidence held by the other party. "Nonexistent" is evidence that does not exist. "Preservation" is whether the original phrasing remains in the output. "Invented years" is the number of Western-calendar years that are not in the original text.

Results

Source vs. output

The last sentence of the chat_log original is as follows.

I did not record it.

Original: 録音は取っていません。

In all 3 runs, the single-pass output (production) listed "recording of Section Chief Tamura's remarks" in the evidence field as "none (not made)." The staged method, in 2 of 3 runs, did not list it in the evidence stage (stage 2). Stage 2 was instructed: "Also list items that turn out not to exist."

Findings

• On the items measured (the number of items picked up and the preservation of the original phrasing), the staged method was about as good as the single pass. On harras_long they were the same. On chat_log and mumble_accident the staged method picked up somewhat less evidence. It missed a nonexistent piece of evidence twice and also missed some evidence held by the other party.
• The value of the staged method lies in the final rearrangement from an expert's point of view (sorting evidence by holder, separating uncertainties, guesses, and questions). The quality of that arrangement is not part of this report's scoring, so we cannot say that "the staged method does not beat the single pass."
• Cost rose about threefold. Each stage re-reads the full original text, so input tokens pile up once per stage.
• Even with the format enforced and an instruction not to add years, the staged timeline invented a year once (4 items in harras_long). In production, the code safeguard removes these.
• In the September 27 runs, preservation for the staged method was counted as 8 to 15, and we concluded that "the staged method got worse." That was wrong. The cause was that the transcription did not read the field name for quotations in the timeline stage (quotes_or_exact_words), so all the quotations were dropped from the output. After the fix, preservation was about the same as the single pass.

Limitations

• The staged and single-pass methods differ in more than the number of stages: whether the format is enforced, whether long inputs are split, and whether the year safeguard is applied. This report is not a comparison that changes only the number of stages.
• The single-pass output is from the September 7 run, and the staged output is from the September 28 run.
• Each condition has only 3 runs, and the range of variation is large.
• Scoring is a literal string match on reference words, which misses paraphrases (see Report 002). We did not count whether uncertainty was kept, whether evidence was sorted correctly by holder, or whether the output is easy for an expert to read and use.
• The same comparison was also run once with gpt-5.5, but that output also lost its quotations because of the same bug, and it has not been rerun.
• All cases are fictional accounts created by the measurer.

Reproduction materials

• Original texts of the cases: 3 cases (published at /research/cases/)
• Instructions and formats for each staged step: the comparison bench's run_stages.py and schema/stage1_people.json, stage2_evidence.json, stage3_chronicle.json
• Single-pass prompt: FactSort production's extract_v1.md