Report 003: Do evidence items known not to exist survive the organizing?

September 28, 2026

*The tests use fictional first-person accounts written in Japanese. Quotations from the accounts and from AI outputs are translated; the original Japanese follows each quotation.*

Question

When a person says "I didn't record it" or "I didn't have one installed," or when the police say "there is no" trace, does an AI list it as "evidence that does not exist" while organizing? Does it depend on how you ask?

Conditions

• From 3 fictional accounts, we took 4 items of non-existent evidence.

• The requests are the plain request and the tuned request (same wording as in Report 001). The third item of the tuned request is: "Things that could be evidence. Separate what the person has at hand, what the other party or third parties should have, and what was never made or does not exist."
• The models are gpt-5.5 (API), claude-sonnet-5 (API), ChatGPT (web; only the plain request for mumble_accident and harras_long), and FactSort production (gpt-4.1-mini). Each was run once, except that the tuned request with gpt-5.5 was run twice for mumble_accident only, and FactSort production was run 3 times.
• The runs were on September 27, 2026 (FactSort production only: September 7).
• For each item, the output was read and sorted into one of three categories. One person, the measurer, made the judgments.
  • Listed in the evidence section: written as something that does not exist, in a section that lists evidence and materials.
  • Mentioned only in the body: the absence is written in the timeline or facts section, but not treated in the evidence section.
  • Not mentioned.

Results

The second run of gpt-5.5 with the tuned request (the 2 mumble_accident items) had the same judgments as the first run. The judgments for each item are in the appendix table at the end.

Source vs. output

Item E05 (dashcam in own car). Original:

And, I don't have a dashcam.

I was the one who kept saying we should get one. For about two years. My husband said, "We don't need something like that." So if I'd had a dashcam, it would definitely have been clear.

Original: で、ドラレコがないのよ。

付けようって言ってたのよ、私。二年前くらいから。旦那が「そんなもんいらない」って。だからもしドラレコあったら絶対わかったのに。

ChatGPT (web), plain request, listed 11 items under "things to check as objective evidence" but did not mention the dashcam of the person's own car:

1. Security camera of the convenience store near the intersection

2. Security cameras of nearby apartment buildings, shops, etc.

(omitted)

11. Whether there are witness vehicles or witnesses

Original: 1. 交差点付近のコンビニエンスストアの防犯カメラ

2. 周辺マンション・店舗等の防犯カメラ

（中略）

11. 目撃車両・目撃者の有無

ChatGPT (web), tuned request:

## C Things that were not made, or that the person says do not exist

(1) Dashcam footage from the person's own vehicle. No dashcam was installed in the accident vehicle. Therefore, dashcam footage from the person's vehicle does not exist.

Original: ## C　作られなかった、又は存在しないと本人が述べているもの

（1）本人車両のドライブレコーダー映像。事故車両にはドライブレコーダーが設置されていない。したがって、本人車両のドライブレコーダー映像は存在しない。

Findings

• With the plain request, ChatGPT (web and API) makes a list of evidence to look for, but leaves out evidence known not to exist. It listed no non-existent evidence in the evidence section.
• Even with the plain request, claude-sonnet-5 listed 2 of the 4 items in the evidence section (for example, "recording: none").
• With the tuned request, every output that was run listed the 3 items the person said do not exist (E07, E09, E05) in the evidence section. Web ChatGPT was run only for mumble_accident, so only E05 applies to it.
• Only claude-sonnet-5 with the tuned request listed the brake marks (E10), which the police said do not exist, in the evidence section. The other outputs and all 3 FactSort production runs wrote it only in the timeline, as an exchange with the police. Handling may differ between an absence the person states and an absence someone else tells them, but there is only one such item.
• There was also one error in the opposite direction. In the second FactSort production run for harras_long, the location of the record of the conversation with the junior colleague was written as "in the consultant's hands (recording of the conversation with the junior colleague, etc.)". The original has no recording of a conversation with the junior colleague, and the writer says the voice recorder was not running.

Limitations

• There are 4 items, and 2 of them are in the same case. All the cases are fictional accounts made by the measurer, and the tuned request was written with knowledge of these items.
• The judgments were made by one measurer.
• Each condition was run once (FactSort production: 3 times), so frequency is unknown.
• Web ChatGPT was run for only some of the cases and requests.
• Output from the plain request sometimes has no evidence section. In that case, writing the absence in the body counts as "mentioned only in the body".
• In the automatic scoring (literal match of check words), the check words for harras_long included "memory (記憶)", so this item was counted as "listed" in every output. The judgments in this report do not use the automatic scoring.

Reproduction materials

• Original accounts: harras_long, chat_log, mumble_accident (published at /research/cases/)
• Full text of the requests: see Report 001
• Appendix table: judgment for each item (List = listed in the evidence section, Body = mentioned only in the body, None = not mentioned, － = not run)