Making LLM Agents Actually Plan: A Roadmap

Your LLM agent is impressive. It writes clean code, summarizes dense papers, drafts emails in your voice. Then you ask it to do something that requires more than one step — and it falls apart.

You tell it to plan a trip. It books a flight arriving at 11 PM and a dinner reservation at 7 PM the same evening. You ask it to set up a deployment pipeline. It schedules the deploy step before the tests. You ask it to plan a dinner party for eight guests with dietary restrictions. It produces a beautiful timeline where two dishes need the oven at the same time, appetizers finish after the main course, and the "vegan" option contains cheese.

Each individual step looks reasonable. The flight is real. The restaurant is nice. The recipe is good. But the steps don't fit together. Constraints are violated. Dependencies are ignored. State isn't tracked.

These aren't language failures. They're planning failures.

LLMs are extraordinarily good at generating plausible-sounding next steps. They are extraordinarily bad at ensuring those steps form a coherent, constraint-satisfying sequence. And if you're building agentic AI systems — anything that coordinates multi-step tasks in the real world — this distinction is the difference between a demo and a product.

So what would it actually take to fix this? Think about what went wrong in the dinner party example. The problem wasn't that the LLM didn't know good recipes — it did. The problem was that nobody checked whether two dishes could physically share one oven at the same time, or whether the appetizers would actually be ready before the main course. What was missing wasn't knowledge. It was a system that tracks what's true right now, what each action changes, and whether the whole sequence actually works end to end.

That kind of system exists. And it's been around a lot longer than LLMs.

What the Planning Community Brings to the Table

There is an entire subfield of AI dedicated to exactly this problem. It's called automated planning — it has been around for decades, but the rise of LLM-based agents has brought it into the spotlight from a completely new angle.

Think of it like a GPS navigation system. You tell it where you want to go, it knows every road and turn restriction, and it finds a route that's guaranteed to get you there — no wrong turns, no dead ends. Planning solvers work the same way, but for sequences of actions: you tell them what the world looks like now, what actions are available, and what you want to achieve, and they search for a valid sequence to reach that goal. The best modern solvers handle millions of states efficiently. They've been honed through international competitions and decades of research, and they're now finding a second life as the backbone of reliable agentic systems.

But just like a GPS needs a map, a planning solver needs a formal description of the problem — one that spells out every possible state, every action, and every constraint in precise, unambiguous terms. The planning community uses a language called PDDL (Planning Domain Definition Language) for exactly this. Think of it as the "map format" for planning: a structured specification of what's possible and what's desired. Writing PDDL requires expertise. It's not something you hand to a product manager.

And that's exactly where the opportunity is. LLMs are excellent at understanding messy, natural language descriptions of problems. Formal planners are excellent at finding guaranteed-correct solutions — but only if someone writes the formal description first. What if LLMs could be the bridge? Not as replacements for planners — but as translators between human language and formal planning tools.

The Two Paradigms

This idea — LLMs as a bridge to formal planning — sounds straightforward, but it plays out very differently depending on one critical question: does someone already have the formal description, or does the system need to create it from scratch? The answer splits the entire field into two paradigms, and understanding this split is the key to understanding everything that follows.

Paradigm 1 is powerful and already producing strong results. If you have a domain expert who can write PDDL, you can get dramatic improvements by using LLMs to generate heuristics, policies, or candidate plans that formal tools then verify.

Paradigm 2 is harder and more ambitious. A user describes a task in plain English — "My robot can carry two items, fragile items go on top, it needs to recharge every 30 minutes" — and the system converts that to a formal model, validates it, solves it, and returns a verified plan. No PDDL expertise required. This is the frontier, and it's where the most exciting unsolved problems live.

These two paradigms didn't emerge simultaneously. They represent a progression — the field didn't jump straight to "English in, verified plan out." It got there through a series of hard lessons, each building on the failures of the last.

The Evolution

In just three years, the research question has shifted entirely — from "can LLMs plan?" to "how do we build systems where LLMs and planners work together?" Here's how that happened.

The first wave asked the obvious question: can LLMs just plan on their own? The answer was a clear no. Rigorous benchmarks showed that frontier models solved about 12% of planning problems correctly. And when researchers renamed predicates to meaningless tokens — turning "move-block" into "xyzzy-37" — performance collapsed to zero. The LLMs weren't reasoning about actions and consequences. They were pattern-matching against their training data.

That failure turned out to be productive. Instead of giving up, the second wave asked a better question: what if LLMs don't need to plan alone? Researchers started pairing LLMs with formal planners — the LLM proposes, the planner verifies. LLMs generated candidate plans that formal verifiers checked. LLMs wrote Python heuristic functions that guided classical search. Results jumped from 12% to 82% on the same benchmarks.

But all of that still required someone to write PDDL by hand. The third wave tackled that bottleneck head-on: what if the LLM could create the formal model itself, from nothing but an English description? Multi-agent systems where LLMs translate natural language to formal models started achieving real results — 100% on some domains, though orchestration failures remained a bottleneck.

Which leads to the emerging fourth wave: can we build truly agentic systems for planning — systems that learn, adapt, and improve their own coordination strategies? That's the frontier this series builds toward.

The Roadmap

This series walks through the entire landscape, from foundations to the research frontier. Here's where we're going.

Meet RoboSort

All of this — paradigms, evolution, formal guarantees — can feel abstract until you see it in action. So throughout this series, we'll ground every concept in a single running example: a warehouse robot named RoboSort working at the PackBot Warehouse.

Interactive — click "▶ Auto Demo" to watch a planner build a tower, then see the naive LLM try (and fail)

The setup is simple: three shelves holding tower parts — two legs, a beam, a roof, and a flag — plus a build zone where RoboSort must assemble them into a stable tower. The catch: pieces need support underneath. Place the beam before the legs are down? It falls. Place the roof with nothing below? Same thing. Order matters — and the naive LLM gets it catastrophically, visibly wrong.

This sounds easy. It isn't. If RoboSort places pieces in the wrong order, the structure collapses. If it skips a foundation piece, everything above is unsupported. If it doesn't track what's already been placed, it tries impossible assemblies. Every post in this series will revisit this warehouse — growing from 5 pieces to 50, adding aisles, conveyor belts, charging stations, and eventually new facilities with novel rules described only in English.

Beyond the Warehouse: Planning in Agentic AI

You might be thinking: "Warehouse robots are interesting, but I'm building a coding agent / travel bot / workflow system." Fair enough. RoboSort makes the planning problem concrete, but the constraints-and-sequencing challenge shows up everywhere. If you're building agents that coordinate multi-step tasks, here's how the exact same planning principles apply to your domain.

What's Ahead

The core thesis of this series is simple: LLMs alone cannot plan reliably, but the combination of LLMs and formal planning tools is extraordinarily powerful. The planning community has built solvers that guarantee correctness. LLMs bring natural language understanding, flexibility, and the ability to bridge the gap between how humans describe problems and how solvers need to receive them. The convergence of these two communities is reshaping what agentic AI can reliably accomplish.

By the end of these seven posts, you'll understand exactly how to make your LLM agent plan reliably — and why the answer involves a well-established AI subfield that the LLM community has been rapidly integrating over the past few years. You'll know what PDDL is, how heuristic search works, why Tree of Thoughts is expensive and unsound, what LLM-Modulo actually does, how NL2Plan converts English to formal models, and what the frontier of agentic AI for planning looks like.

Let's get started.