LLMs Try to Plan (It Goes Badly)

Posts 2 and 3 built the foundations: PDDL formalizes a planning problem, and heuristic search solves it. Modern planners handle billions of states with mathematical guarantees. The obvious question follows: do we even need any of that? Can't GPT-4 just plan?

The question is fair. LLMs can pass the bar exam, write working code, and explain quantum mechanics to a five-year-old. Surely sequencing five actions in the right order is within reach.

Between 2022 and 2024, a wave of rigorous research tested exactly this. The results weren't just disappointing — they were diagnostic. They didn't just show that LLMs fail at planning. They showed why they fail, and the "why" turned out to be the most important finding. It redirected the entire field.

This post presents the evidence. We'll run our RoboSort warehouse through the same gauntlet the researchers used, watch frontier models fail at problems a classical planner solves in milliseconds, and understand why the popular fixes — chain-of-thought, self-critique, Tree of Thoughts — don't actually work.

The Benchmark: PlanBench

In 2023, Valmeekam et al. introduced PlanBench, the first rigorous benchmark for evaluating LLMs on classical planning tasks. The setup was deliberately simple: Blocksworld problems where a robot arm must stack colored blocks into a target configuration. Five blocks. A few moves. The kind of problem a STRIPS planner from 1971 could solve.

They tested GPT-4, GPT-3.5, and several open-source models. The task: given an initial state and a goal state, produce a valid sequence of actions. Not optimal — just valid. Any plan that reaches the goal without violating preconditions counts as a success.

The numbers were stark. GPT-4 — the most capable model at the time — generated valid plans for only about 30% of instances. GPT-3.5 managed around 3%. Open-source models scored near zero. Meanwhile, Fast Downward, a classical planner from 2004, solved 100% of instances in under a second.

But the raw accuracy numbers weren't the most devastating finding. That came when the researchers asked a deeper question: is the LLM actually reasoning, or is it pattern-matching?

RoboSort at the Benchmark

Let's see what this looks like in our warehouse. We gave a frontier LLM the same RoboSort problem from Posts 1–3: five pieces, three shelves, one build zone, support constraints. Build a tower. Here's what happened.

RoboSort facility · initial state (the LLM sees this as text in the prompt)

The model knows what support constraints are — it can explain them eloquently. But knowing a constraint and enforcing it during sequential generation are fundamentally different cognitive operations. The LLM generates step-by-step, each token predicted from context. It doesn't maintain a world model. It doesn't track which pieces are placed. It doesn't verify preconditions before committing to an action.

The Smoking Gun: Mystery Blocksworld

If LLMs were genuinely reasoning about actions and consequences — if they truly understood what "pick up" does to the state of the world — then the names of the predicates shouldn't matter. "pick-up-block" and "xyzzy-37" should be equivalent: different labels for the same operation. The planner doesn't care what you call things. Does the LLM?

Valmeekam et al. tested this with Mystery Blocksworld. They took the exact same planning problems and replaced every predicate and action name with meaningless tokens. on(A,B) became snurg(q3,q7). pick-up became florp. The logical structure was identical. Only the labels changed.

(:action pick-up
  :parameters (?x - block)
  :precondition (and
    (clear ?x)
    (on-table ?x)
    (arm-empty))
  :effect (and
    (holding ?x)
    (not (clear ?x))
    (not (on-table ?x))
    (not (arm-empty))))

(:action florp
  :parameters (?x - grindle)
  :precondition (and
    (zarb ?x)
    (plonk ?x)
    (sniv-empty))
  :effect (and
    (clutching ?x)
    (not (zarb ?x))
    (not (plonk ?x))
    (not (sniv-empty))))

The result was devastating: performance collapsed to zero. Not "lower." Not "somewhat degraded." Zero.

This is the smoking gun. A system that reasons about state transitions wouldn't care whether the action is called "pick-up" or "florp." The preconditions are the same. The effects are the same. The state space is identical. But the LLM's performance was entirely dependent on recognizing the names — because it wasn't reasoning at all. It was retrieving similar-looking action sequences from its training data.

RoboSort Goes Mystery

The same test with our warehouse robot is equally revealing. When we describe the problem with meaningful names — "pick up Leg1 from Shelf A" — the LLM can pattern-match against warehouse logistics data and occasionally get it right. But rename the pieces:

Pieces: L1, L2, Beam, Roof, Flag
Locations: Shelf-A, Shelf-B,
           Shelf-C, Build-Zone
Constraint: legs support beam,
            beam supports roof

LLM: "Place L1, L2, then Beam,
     then Roof, then Flag"
     → Sometimes correct (~30%)

Pieces: Q3, Q7, Zrint, Plonk, Vex
Locations: Zone-W, Zone-X,
           Zone-Y, Zone-Z
Constraint: Q3,Q7 glarb Zrint,
            Zrint glarbs Plonk

LLM: "Move to Zone-X, florp Zrint,
     move to Zone-Z, snarg Zrint..."
     → Incoherent actions

The Prompting Fixes That Don't Fix

After PlanBench, the natural response was: maybe the LLM just needs help. Better prompts. More structure. Time to think. The research community systematically tested every major prompting strategy. Here's what happened.

Chain-of-Thought (CoT)

The idea: make the LLM reason step-by-step. Instead of generating the plan directly, have it explain its reasoning at each step. "First, I need to place the foundation. The legs go first because they support the beam..."

The result: marginal improvement at best, and sometimes worse. CoT helps the LLM generate more verbose explanations of why its plan is correct — while the plan remains incorrect. The model articulates the constraint ("beam must go before roof") and then violates it in the very next step. It's like a student who writes a perfect essay about traffic laws and then runs a red light.

Self-Critique / Verify-and-Refine

The idea: have the LLM check its own work. Generate a plan, then ask the same model (or another instance) to verify it. If it finds errors, revise and try again.

The result: self-critique makes things worse. This was the most surprising finding. Valmeekam et al. showed that when LLMs evaluate their own plans, they exhibit a "yes-man" bias — the critic tends to approve whatever the generator produced. Worse, when the critic does flag an issue, the "fix" often introduces new violations. After several rounds of self-critique, accuracy decreased.

Tree of Thoughts (ToT)

The idea: explore multiple reasoning paths in parallel, evaluate them, and pick the best. Instead of one linear chain of thought, branch out, score each branch, and select the most promising.

The result: expensive and unsound. ToT can marginally improve results on some instances, but it multiplies computational cost by 10–100x without providing any guarantee. You're searching a tree of LLM-generated candidates — but the evaluation function is also an LLM, which can't reliably distinguish valid from invalid plans. It's search without a sound heuristic. Post 3 showed why that doesn't work: you need a heuristic that actually correlates with distance to goal. An LLM's confidence score doesn't.

Seeing the Failure: LLM vs. Planner on RoboSort

Let's make this concrete. Below, we run the same RoboSort tower assembly problem through an LLM (left) and a classical planner (right). Watch the LLM generate a plausible-looking but invalid plan, while the planner finds a guaranteed-correct sequence in milliseconds.

After executing each plan · what RoboSort actually does on the floor

The planner's output might look boring — it's just a sequence of formal actions. But every single step has been verified: every precondition checked, every effect applied, every state transition valid. The LLM's output reads better — natural language, confident tone — but it's wrong. This is the fundamental tension: fluency is not validity.

Why LLMs Fail at Planning: The Architectural Argument

The evidence points to a fundamental architectural mismatch. Planning requires three capabilities that autoregressive language models lack:

1. State tracking. A planner maintains an explicit world state — what's true right now — and updates it after every action. An LLM has no world model. It has a context window of tokens. When RoboSort places L1, the planner knows the gripper is empty and L1 is on the platform. The LLM has to infer this from the text it already generated — and it often gets it wrong.
2. Precondition verification. Before executing place-on-piece(Roof, Beam), the planner checks: is Beam placed? Is Roof in gripper? Is Robot at build zone? All must be true, or the action is blocked. The LLM has no mechanism to perform this check — it simply generates the next most likely token.
3. Backtracking. When a planner reaches a dead end — a state from which no action sequence reaches the goal — it backtracks and tries a different path. Autoregressive generation is one-shot: each token is committed permanently. There is no "undo." Tree of Thoughts simulates branching but without a sound evaluation function, it's random search with extra steps.

The Taxonomy: What Should LLMs Actually Do?

The failures of 2022–2023 didn't kill the idea of LLMs in planning. They refined it. The research community converged on a taxonomy of roles — things LLMs are genuinely good at, paired with formal tools that handle what LLMs can't.

The critical shift: stop asking LLMs to plan. Start asking them to help plan. The LLM's genuine strengths — understanding natural language, generating code, translating between formats — are exactly the capabilities that formal planners lack. The planner's strengths — state tracking, constraint verification, optimality guarantees — are exactly what LLMs lack. The marriage is natural. The remaining posts in this series explore how it works in practice.

What's Ahead

This post established the hard evidence: LLMs cannot reliably plan. Not with better prompts. Not with self-critique. Not with tree search. The architectural mismatch is fundamental — autoregressive generation lacks state tracking, precondition verification, and backtracking.

But the story doesn't end with failure. The taxonomy of roles shows the path forward. Instead of replacing planners, what if LLMs amplified them? What if an LLM could look at a PDDL domain and generate a Python function that estimates distance to goal — a heuristic — and hand it to a classical search engine?

That's exactly what happened. And the results went from 12% to 82%.