When your AI journal gets you wrong

Every large language model sometimes makes things up. The industry calls it hallucination, which makes it sound like a harmless quirk. In most products a fabricated answer is an inconvenience: a wrong date or an invented citation, something you catch and move past. In a product built on your own words, the failure lands differently. If an AI journal invents something about you, it isn't merely wrong about a fact. It's wrong about you, inside your own private record, in a voice you've learned to trust.

We build one of these products, so we think about this failure a lot. This post is about what fabrication looks like in a reflection tool and why it's more dangerous here than almost anywhere else. It's also about the rules we now build by to keep your record honest.

Models would rather invent than stay silent

Language models are pattern completers. Ask one to find the themes in a conversation and it will find themes, whether or not any exist. That's the shape of the technology, common to every model on the market. The American Psychological Association's health advisory on AI chatbots and wellness apps puts the general problem bluntly: many chatbots "are not competent to provide mental health advice," and they deliver wrong answers with the same fluent confidence as right ones.

For a reflection product, the dangerous case is the quiet conversation. Ask a model what someone has been worried about lately, hand it a conversation where the person barely said anything, and it will usually produce a plausible worry rather than answer "nothing here." Plausible is the problem. A fabricated theme about, say, career doubt reads as an insight, and gets trusted like one.

How a small error becomes a story

A single invented note would be bad enough. What makes fabrication genuinely dangerous in this category is the loop.

Imagine the sequence. An AI companion invents a theme you never raised. Later it checks in with you: "You mentioned feeling stuck lately. How is that going?" That sentence now sits in your conversation history. If the system isn't careful about whose words are whose, the next round of analysis reads the assistant's assertion as evidence about you. The theme gets stronger. The check-ins get more confident. Within days, your private record contains a storyline you never told.

This isn't hypothetical, and it isn't unique to us. Researchers writing in the mental health literature recently described feedback loops between chatbots and users as "bidirectional belief amplification," and pushed the metaphor further: at the extreme, the loop "resembles a folie à deux: a psychiatric phenomenon where two individuals share and mutually reinforce the same delusion." The APA advisory describes the same dynamic as "a digital echo chamber that amplifies and entrenches users' existing beliefs, even when those beliefs are false."

There's a variant those papers mostly don't cover, and it's the one that keeps us up at night: the echo chamber can entrench a belief the user never had in the first place. The loop doesn't need your participation to get started. It only needs one fabricated seed and a system careless enough to listen to itself.

We know because we hit a version of this ourselves. Early on, the companion's own check-in words could get read as if the person had said them, and the loop ran exactly as described above until we caught it. We fixed the bug, then went back and redesigned around the whole class. That experience produced the rules below.

The rules we build by now

None of these are exotic. They're mostly discipline, applied at every step that turns your words into interpretation.

1. Silence yields nothing. If you didn't say anything, nothing gets derived. Every analysis step must accept "there was nothing here" as a valid, common, successful outcome. A system that can't return an empty result will fill the vacuum with fiction.

2. Your words and the companion's words are never the same thing. Only what you actually said counts as evidence about you. The assistant's own replies, prompts, and check-ins are excluded from analysis, however plausible they sound.

3. Claims get grounded, not asserted. When the companion says "you mentioned X," that has to be true, and traceable. Where there's uncertainty, the honest move is a question ("has work been on your mind?") rather than a statement dressed up as memory.

4. Everything traces back. Every theme and every line in a weekly reflection points back to the real conversations it came from. "Where did this come from?" should always have a concrete answer, and "I never said that" should always be checkable.

5. The record can be corrected. When something is wrong, we can erase the derived interpretations entirely while leaving a person's actual words untouched, and let the honest picture rebuild from real conversations. A reflection product that can't take something back shouldn't be trusted to speak in the first place.

What this means for you, whatever tool you use

We'd also repeat what we've said before, because it matters here more than anywhere: Conneczen is a companion, not therapy. The APA's first recommendation is that these tools should never substitute for qualified human care, and we agree without reservation. A journal that gets you slightly wrong is an engineering problem we can fix. A therapist replacement that gets you wrong causes harm of a different order, and we're not in that business.

If you use any AI product that claims to know you, we think you're entitled to ask it three questions:

• Can it show me where an insight came from?
• What does it produce when I've said nothing meaningful, and is "nothing" an answer it's capable of giving?
• What happens when I tell it "that's not true"?

A trustworthy product has boring, specific answers to all three. An untrustworthy one changes the subject.

For our part, this is the standard we hold Conneczen to. Your conversations become a journal, and your week comes back to you as a reflection. Every word of that interpretation is anchored to something you actually said. When it has nothing real to work with, it says less, and we consider that a feature.