LeCun, World Models and AGI

For quite some time now, Yann LeCun has argued that the future of AI will be shaped not by “Generative AI” but by JEPA (Joint-Embedding Predictive Architecture)–based “World Models.” In many ways I find his case convincing; I've touched on it in earlier posts too. Recently he gave a talk at New York University.

To summarize his ideas, simplifying a little:

At the level it has reached today, AI looks like it knows a great deal — but it doesn't truly understand the world. A cat, or a ten-year-old child, can easily do things robots still can't. The reason is the lack of a World Model.

Most of today's AI merely reacts; it doesn't think.

A World Model means a system holding the current state of the world as an abstract representation, being able to run “what happens if I take this action?” simulations in its head, and planning accordingly. Humans do this constantly. Today's generative AIs, on the other hand, only react.

Does it have to predict every pixel?

Generative models like ChatGPT and Sora are genuinely good at predicting and producing every pixel and every detail of the world. But the real world is unpredictable and full of randomness. So generative models drown in unnecessary detail, and planning becomes hard.

The JEPA architecture, by contrast, is concerned not with the future itself but with an abstract representation of the future. That is, not “how many pixels are in this frame?” but “will this object fall, collide, or stay balanced?” By discarding the irrelevant detail, this approach learns the physical and causal structure.

It's not about the pixel count of a frame; it's about whether the object will fall.

This is why LeCun and those who think like him argue that if human-level AI is the goal, LLMs need to be abandoned, and that the road to AGI runs through World Models.

It doesn't sound unreasonable to me at all.