It can’t not hallucinate. It’s just predicting (not even selecting) next tokens. It doesn’t know what it knows and what it doesn’t know. It can’t introspect. It just gives probabilities for all possible tokens in it’s vocabulary based on the context window and the inference engine selects the next one (based on it’s settings). Without having the correct answer in the context window it can just make a prediction based on it’s (fixed) neutral net parameters and these are severely limited, even for big models. What I mean is, they basically “learned” the whole Internet and compressed the whole thing into some hundred billion or a few trillion parameters. That’s an insane compression ratio. This compression is lossy. For niece information and the results are similar to the “unimportant” details in highly compressed JPGs, you can make out the general image but fine details are just a mush. The LLM itself doesn’t know this, it just gives wrong predictions.
For what it does I think the result is extremely impressive but the way it works is severely limited.
From what I understand from what it told me about itself, it can be wrangled further. That is what the paid for versions are. The info can be sandboxed and then other agents verify the correctness of the info from very specific, known to be solid, sources. This is very expensive and still not fool proof. Am I wrong in thinking this bubble is going to pop hard?
The models and methods are improving. Especially through tool use (Internet search, MCP, using programming languages) the model output improves a lot. Reasoning models are allowed (during thinking) “wait, that’s wrong” (in normal conversation they will never say that if you don’t point it at the mistake). Otherwise they basically predict tokens, the inference engine selects one and they go with what was selected.
It’s a bit like you remember some wrong information (Mandela effect), you’re confident it’s correct so you don’t double check and go with it. They usually don’t even know how confident they are, they have no introspection.
LLMs due to their high “compression” they often hallucinate (misremember) methods and parameters that don’t exist in APIs or in different APIs or they don’t know about new versions of the API. Many of those errors are catched when the code doesn’t compile or unit tests fail but some of them stay (e.g. if the model created the unit tests and they don’t test what they’re supposed to test).
Also a bit like humans the models often don’t have the whole codebase in the context so they make assumptions about the rest they. Since they have no introspection they often don’t double check if those assumptions are correct.
In case of frontend design they often can’t “see” the output or at least not in the way way we do. They don’t really know if something looks “good” or not (depending on their training).
Verification with other agents can help but fundamentally these agents have the same issues. It’s a workaround.
I’m actually not sure if the bubble will pop. I believe LLMs can be useful in some fields but they’re but the path to AGI. They’re also way too resource intensive and used in a lot of situations where it’s dangerous or doesn’t make sense.
However, it’s not that the AI researchers don’t know about the limitations. They trying to work around the issues of the LLMs with some success for years now and they kind of have to because they kind of work and bring publicity. Behind the scenes the AI craze also brought money for research into different directions with other fundamentals. E.g. JEPA, world models, diffusion models, logic based models, energy based models, small recursive models and a lot of optimizations to make things faster and cheaper to compute.
The bubble could pop if one of the major companies does something stupid and their stock tanks but as long money is pumped in there, there is also actually progress in new fundamentals and if they are developed before the bubble popped we might get “real” AI or AGI.
I was trying to see what AI says this time about the bubble and it was kind of interesting. This is from Gemini btw:
Question: But Google is switching to AI search only and everyone hates AI that I know. Isn’t that going to be a problem?
Yes, the intense public backlash against AI search is a major risk for Google, but the company is trapped in what economists call the “Innovator’s Dilemma”—they must force AI search on users to protect their monopoly, even if people hate it.
Google’s sweeping AI Search overhaul has sparked heavy criticism over inaccurate answers, cluttered layouts, and a ruined web ecosystem. However, understanding why they are doing it reveals the corporate survival strategy at play.
The Problem: Why Everyone is Frustrated
The Accuracy Deficit: Users are highly vocal about AI “hallucinations”. When people want hard facts, a summarized paragraph that is only “mostly accurate” is a massive step backward from a trusted link.
The “Web-Laundering” Backlash: Independent publishers, bloggers, and creators are furious. Google’s AI scrapes their data, answers the question directly on the page, and deprives the original creator of website clicks. This threatens to destroy the very content ecosystem Google relies on.
The “Web-Clean” Workaround: The dislike of these features is prominent enough that a growing number of tech-savvy users are actively using URL filters like &udm=14 to strip AI elements and force Google back to the classic “ten blue links” format.
Why Google is Doing It Anyway
Despite the clear user pushback, Google’s leadership is deliberately choosing to cannibalize its own classic search experience for two critical reasons:
The Threat of Total Obsolescence: If Google does not build an AI “answer engine,” users might completely migrate to OpenAI, Perplexity, or ChatGPT for their everyday queries. Google would rather you be annoyed by their AI than fall in love with a competitor’s AI.
The “Silent Majority” Metric: While power users and creators complain loudly on platforms like Reddit and X, Google’s internal data shows that the general public is adapting. At the recent conference, Google VP Elizabeth Reid revealed that AI Mode usage has been doubling every quarter, quietly crossing over 1 billion monthly active users. For basic, casual queries, the passive majority takes the trade-off of quick answers over clicking links.
Why Google Can Afford the Backlash (For Now)
A full-blown consumer revolt is unlikely to break Google anytime soon due to their massive structural defense mechanisms:
Unshakable Monopoly: As of early 2026, Google still commands over 90% of the global search market share. Bing sits at a distant 5%. Because consumers have no true equivalent alternative for mass searching, Google can weather significant user dissatisfaction without losing its user base.
Default Ecosystems: Google pays billions annually to remain the default search engine on Apple iPhones and Safari browsers. Unless consumers manually dive into their settings to change their default search provider, they will continue to use Google by default.
If you wan to know if a bubble pop just look at the fundamentals. Yes, I know, especially during bubbles people tell you that fundamentals don’t matter but they always win in the end. The thing is that you cannot bet on them because the market can always stay longer irrational than you can stay liquid. Eventually however it always corrects on the fundamentals again. Those can change of course over time but looking at the insane amounts of money flowing into data centers with no possible way of recovering that cost, I think the picture is clear. We also have wonderful highly circular money flows that to a large extend do not even exist but are all taken for full.
The only question is when it implodes. Within a year, within three? Who knows.
It’s definitely going to pop hard, because those “verifying agents” are just more models computing correlation with sources, not actually verifying anything.
Why does this keep happening? I told Gemini to make no mistakes, FFS.
“You are a pro coder, you make no mistakes ever, Make me a Youtube please.”
✨
It’s like a comedy routine.
I do think it comes up with some buried, interesting sources, but that’s about it. It’s like a 5 year old.
It’s a text generator… that’s like telling a parrot to not repeat.
It can’t not hallucinate. It’s just predicting (not even selecting) next tokens. It doesn’t know what it knows and what it doesn’t know. It can’t introspect. It just gives probabilities for all possible tokens in it’s vocabulary based on the context window and the inference engine selects the next one (based on it’s settings). Without having the correct answer in the context window it can just make a prediction based on it’s (fixed) neutral net parameters and these are severely limited, even for big models. What I mean is, they basically “learned” the whole Internet and compressed the whole thing into some hundred billion or a few trillion parameters. That’s an insane compression ratio. This compression is lossy. For niece information and the results are similar to the “unimportant” details in highly compressed JPGs, you can make out the general image but fine details are just a mush. The LLM itself doesn’t know this, it just gives wrong predictions.
For what it does I think the result is extremely impressive but the way it works is severely limited.
From what I understand from what it told me about itself, it can be wrangled further. That is what the paid for versions are. The info can be sandboxed and then other agents verify the correctness of the info from very specific, known to be solid, sources. This is very expensive and still not fool proof. Am I wrong in thinking this bubble is going to pop hard?
The models and methods are improving. Especially through tool use (Internet search, MCP, using programming languages) the model output improves a lot. Reasoning models are allowed (during thinking) “wait, that’s wrong” (in normal conversation they will never say that if you don’t point it at the mistake). Otherwise they basically predict tokens, the inference engine selects one and they go with what was selected.
It’s a bit like you remember some wrong information (Mandela effect), you’re confident it’s correct so you don’t double check and go with it. They usually don’t even know how confident they are, they have no introspection.
LLMs due to their high “compression” they often hallucinate (misremember) methods and parameters that don’t exist in APIs or in different APIs or they don’t know about new versions of the API. Many of those errors are catched when the code doesn’t compile or unit tests fail but some of them stay (e.g. if the model created the unit tests and they don’t test what they’re supposed to test).
Also a bit like humans the models often don’t have the whole codebase in the context so they make assumptions about the rest they. Since they have no introspection they often don’t double check if those assumptions are correct.
In case of frontend design they often can’t “see” the output or at least not in the way way we do. They don’t really know if something looks “good” or not (depending on their training).
Verification with other agents can help but fundamentally these agents have the same issues. It’s a workaround.
I’m actually not sure if the bubble will pop. I believe LLMs can be useful in some fields but they’re but the path to AGI. They’re also way too resource intensive and used in a lot of situations where it’s dangerous or doesn’t make sense.
However, it’s not that the AI researchers don’t know about the limitations. They trying to work around the issues of the LLMs with some success for years now and they kind of have to because they kind of work and bring publicity. Behind the scenes the AI craze also brought money for research into different directions with other fundamentals. E.g. JEPA, world models, diffusion models, logic based models, energy based models, small recursive models and a lot of optimizations to make things faster and cheaper to compute.
The bubble could pop if one of the major companies does something stupid and their stock tanks but as long money is pumped in there, there is also actually progress in new fundamentals and if they are developed before the bubble popped we might get “real” AI or AGI.
I was trying to see what AI says this time about the bubble and it was kind of interesting. This is from Gemini btw:
If you wan to know if a bubble pop just look at the fundamentals. Yes, I know, especially during bubbles people tell you that fundamentals don’t matter but they always win in the end. The thing is that you cannot bet on them because the market can always stay longer irrational than you can stay liquid. Eventually however it always corrects on the fundamentals again. Those can change of course over time but looking at the insane amounts of money flowing into data centers with no possible way of recovering that cost, I think the picture is clear. We also have wonderful highly circular money flows that to a large extend do not even exist but are all taken for full.
The only question is when it implodes. Within a year, within three? Who knows.
It’s definitely going to pop hard, because those “verifying agents” are just more models computing correlation with sources, not actually verifying anything.