The models and methods are improving. Especially through tool use (Internet search, MCP, using programming languages) the model output improves a lot. Reasoning models are allowed to admit mistakes (during thinking) “wait, that’s wrong” (in normal conversation they will never say that if you don’t point it at the mistake). Otherwise they basically predict tokens, the inference engine selects one and they go with what was selected.

It’s a bit like you remember some wrong information (Mandela effect), you’re confident it’s correct so you don’t double check and go with it. They usually don’t even know how confident they are, they have no introspection.

In software development scenarios LLMs, due to their high “compression”, often hallucinate (misremember) methods and parameters that don’t exist in APIs or in different APIs or they don’t know about new versions of the API. Many of those errors are catched when the code doesn’t compile or unit tests fail but some of them stay (e.g. if the model created the unit tests and they don’t test what they’re supposed to test).

Also a bit like humans the models often don’t have the whole codebase in the context so they make assumptions about the rest. Since they have no introspection they often don’t double check if those assumptions are correct.

In case of frontend design they often can’t “see” the output or at least not in the way way we do. They don’t really know if something looks “good” or not (depending on their training).

Verification with other agents can help but fundamentally these agents have the same issues. It’s a workaround.

I’m actually not sure if the bubble will pop. I believe LLMs can be useful in some fields but they’re but the path to AGI. They’re also way too resource intensive and used in a lot of situations where it’s dangerous or doesn’t make sense.

However, it’s not that the AI researchers don’t know about the limitations. They trying to work around the issues of the LLMs with some success for years now and they kind of have to because they kind of work and bring publicity. Behind the scenes the AI craze also brought money for research into different directions with other fundamentals. E.g. JEPA, world models, diffusion models, logic based models, energy based models, small recursive models and a lot of optimizations to make things faster and cheaper to compute.

The bubble could pop if one of the major companies does something stupid and their stock tanks but as long money is pumped in there, there is also actually progress in new fundamentals and if they are developed before the bubble popped we might get “real” AI or AGI.

If you wan to know if a bubble pop just look at the fundamentals. Yes, I know, especially during bubbles people tell you that fundamentals don’t matter but they always win in the end. The thing is that you cannot bet on them because the market can always stay longer irrational than you can stay liquid. Eventually however it always corrects on the fundamentals again. Those can change of course over time but looking at the insane amounts of money flowing into data centers with no possible way of recovering that cost, I think the picture is clear. We also have wonderful highly circular money flows that to a large extend do not even exist but are all taken for full.

The only question is when it implodes. Within a year, within three? Who knows.

It’s definitely going to pop hard, because those “verifying agents” are just more models computing correlation with sources, not actually verifying anything.

Good to know, this comment chain finally convinced me to try it out.