FactSort Lab. Research

When AI organizes an account of a problem,
what gets added, and what gets dropped?

A year that was never written gets attached to "last October." A wavering memory gets settled into one version. Evidence known not to exist disappears from the list. Advice nobody asked for gets written in.

FactSort Lab. measures how often these things happen, using fictional accounts, and publishes one report per measurement.

All research reportsMethod
5 / 8
Outputs that included a legal assessment when only asked to organize
Report 004
0 / 7
Times plain-prompt ChatGPT listed "nonexistent" evidence in its evidence section
Report 003
2 → 0
Years attached to "last October"; zero after the prompt was changed
Report 001
Latest research

Latest research

All reports (8)
How it is builtSeptember 29, 2026

Does adding a rule to unify names change how evidence is picked up?

After a rule to unify the names of people was added, fewer items of evidence held by the other party were picked up in two short cases (in chat_log, from 5 of 5 to 3). The long case showed no drop.
What gets addedSeptember 28, 2026

Does telling the AI "do not add years" stop it from inventing years?

Some models stopped and some did not.

 What gets addedSeptember 28, 2026

When asked only to organize, does AI also give advice?

With a plain prompt, 5 of 8 outputs included a legal assessment.
Research themes

Themes

What gets added

Years not in the source, advice and scripts nobody asked for.

What gets dropped

Wavering memories, evidence known not to exist.

How it is built

Splitting into stages, adding rules to a prompt.

Human practice

What intake desks and investigation forms have asked people to write separately.

Method

How we measure

  1. We write fictional accounts and prepare the correct organization (gold data) before measuring
  2. One measurement is one report, written as question, conditions, results, source vs. output, and limitations
  3. We state the model, the exact prompt, the number of runs, and the date
  4. We state conclusions only within what we measured, and keep corrections on record when we find errors
  5. The test accounts are published (in Japanese)
  6. Reports are published in Japanese and English
Readings

Readings

All readings
Tool

The organizing tool

We offer a free organizing tool built on the results of these reports. It sorts an account into people, timeline, records (those at hand, those the other party should hold, and those known not to exist), points that waver, guesses and facts, and the writer's own questions.

  • It is designed not to add facts or years that are not in the source. Wavering memories are kept as they are
  • It does not give legal advice or predictions. It lists possible places to consult and points to emphasize when explaining
  • The only AI it uses is OpenAI's gpt-4.1-mini. Outputs may contain errors
  • Names of people and organizations and contact details are replaced with labels before storage and analysis (some may be missed)
  • Input is deleted from this site's server after 72 hours. OpenAI keeps API inputs for up to 30 days for abuse monitoring and does not use them for training
Go to the tool

Each person can run up to 3 analyses per day, at least 10 minutes apart. There is also an overall daily limit. The tool is for people aged 13 and over. Please do not enter real names or other information that identifies people.