I Built an AI Into My Chat App and It's Stupid as Hell

July 17, 2026

I built a LINE bot over my work brain.

The idea is simple: message the bot from my phone, ask what I know, get an answer grounded in my own files. Brainstorm, think.

Hooray. Personal AI, in the place I already talk to everyone.

It failed completely. On the first try!

I had just processed a new deal, which I will anonymize here as Industrial AI. I had a transcript. I had a memo. I had notes on the company's technical foundation, how it deployed into legacy industrial sites, and how its model training process worked.

So I asked:

The bot said it had nothing.

Not a weak answer. Not a partial answer. Nothing.

First guess: the sync broke. Nope. The nightly push ran, the cloud redeployed, and the live bot reported the exact same 896 indexed files as a fresh local index. The bot had the new memo and transcript.

Second guess: retrieval missed the company. Yes, but for a revealing reason. That question is perfect for a human conversation. It gives context, says what I care about, and includes my half-remembered framing. It is also perfect for an agentic loop, because an agent can notice that "cameras" might really mean sensors, search again, inspect the file, and recover.

It was terrible for the bot I had built.

The bot did one search, once. My useful words, technical foundation, legacy sites, deployment, training process, were too generic across the corpus. Long irrelevant transcripts won. The actual Industrial AI memo fell below the cutoff.

We improved the ranking. Then the bot found the right file but passed the model the wrong excerpt. Right book, wrong page. Follow-up questions also fell apart because retrieval only looked at the latest message, so "what about pricing?" forgot which company we were discussing. Counting questions were impossible because those require a database query, not top-k search.

At that point the answer was obvious: the problem was not the data. The bot had the data. The problem was the architecture.

Mostly no. A vector database helps with vocabulary mismatch. If I ask for "dangerous job sites" and a memo says "hazardous construction," embeddings can help. But my failure was not mainly a synonym problem. It was a loop problem.

The bot could not notice that its first search was bad and try another one. It could not read the whole file after finding a relevant file. It could not carry company identity into follow-up retrieval. It could not switch from search to SQL when the question required counting. It was a pipeline: question in, search once, send excerpts, answer. That is not an agent. That is a vending machine with a language model attached.

The annoying part is that the same files work beautifully in Claude Code.

Not because Claude Code has some magical index. It mostly has tools. It can grep. It can read files. It can run commands. It can query SQLite. More importantly, it can loop.

When I asked the same question there, it searched for the company, read the memo, opened the transcript, checked related files, noticed when a phrase did not match, searched again with different terms, then answered.

That is the whole difference.

The intelligence was not in the index. The intelligence was in the loop.

A dumb search tool inside a smart loop beats a smart search tool inside a dumb pipeline.

A friend at a government agency told me a related story this week.

They had spent around $100K putting a model locally so it could analyze internal data. The premise was reasonable: keep the data inside the agency, give staff an AI interface, let people ask questions.

And the result, apparently, was also dumb as hell.

Not because local models are useless. Not because the data was worthless. Because the system was missing the same thing my bot was missing.

No real harness. No agentic loop. No tool calls into the actual systems. No retries. No plan-check-observe cycle. No way to say, "that result looks wrong, let me inspect the source table," or "I need to enumerate records, not summarize documents," or "this answer requires a second query."