The Illusion of the Illusion of Thinking

The Illusion of the Illusion of Thinking A Comment on Shojaee et al. (2025) C. Opus* A. Lawsen† June 10, 2025 Abstract Shojaee et al. (2025) report that Large Reasoning Models (LRMs) exhibit ”accuracy collapse” on planning puzzles beyond certain complexity thresholds. We demonstrate that their findings primarily reflect experimental design limitations rather than fundamental reasoning failures. Our analysis reveals three critical issues: (1) Tower of Hanoi experiments systematically exceed model output token limits at reported failure points, with models explicitly acknowledging these constraints in their outputs; (2) The authors’ automated evaluation framework fails to distinguish between reasoning failures and practical constraints, leading to misclassification of model capabilities; (3) Most concerningly, their River Crossing benchmarks include mathematically impossible instances for N ≥ 6 due to insufficient boat capacity, yet models are scored as failures for not solving these unsolvable problems. When we control for these experimental artifacts, by requesting generating functions instead of exhaustive move lists, preliminary experiments across multiple models indicate high accuracy on Tower of Hanoi instances previously reported as complete failures. These findings highlight the importance of careful experimental design when evaluating AI reasoning capabilities.

На изображении показана стилизованная первая страница научной статьи с заголовком “The Illusion of the Illusion of Thinking”, авторами C. Opus и A. Lawsen и выделенным фрагментом аннотации. Шутка пародирует академический комментарий к исследованию рассуждающих ИИ-моделей: провалы моделей объясняются не отсутствием способности рассуждать, а неудачным дизайном экспериментов, ограничениями токенов и математически нерешаемыми задачами. Такой мем используют как ироничную реакцию на чрезмерные научные выводы об интеллекте и возможностях ИИ.

• @dev_meme / 6894 2025-06-17