This benchmark evaluates large language models (LLMs) using 436 NYT Connections puzzles. Three different prompts, not optimized for LLMs through prompt engineering, are used. Both uppercase and lowercase puzzles are assessed.
| Model | Score |
|---|---|
| o1 | 90.7 |
| o1-preview | 87.1 |
| o1-mini | 42.2 |
| Multi-turn ensemble | 37.8 |
| Gemini 2.0 Flash Thinking Exp 01-21 | 37.0 |
| GPT-4 Turbo | 28.3 |
| GPT-4o 2024-11-20 | 27.9 |
| GPT-4o 2024-08-06 | 26.5 |
| Llama 3.1 405B | 26.3 |
| Claude 3.5 Sonnet (2024-10-22) | 25.9 |
| Claude 3 Opus | 24.8 |
| Grok Beta | 23.7 |
| Llama 3.3 70B | 23.7 |
| Gemini 1.5 Pro (Sept) | 22.7 |
| Deepseek-V3 | 21.0 |
| Gemini 2.0 Flash Exp | 20.0 |
| Gemma 2 27B | 18.8 |
| Qwen 2.5 Max | 18.6 |
| Gemini 2.0 Flash Thinking Exp | 18.6 |
| Mistral Large 2 | 17.4 |
| Qwen 2.5 72B | 14.8 |
| Claude 3.5 Haiku | 13.7 |
| MiniMax-Text-01 | 13.6 |
| Nova Pro | 12.5 |
| Phi-4 | 11.6 |
| DeepSeek-V2.5 | 9.9 |
- A temperature setting of 0 was used
- Partial credit is awarded if the puzzle isn't completely solved.
- Only one attempt is allowed per puzzle. Humans solving puzzles on the NYT website get four attempts and a notification when they're one step away from the solution.
- Multi-turn ensemble is my unpublished system. It utilizes multiple LLMs, multi-turn dialogues, and other proprietary techniques. It is slower and more costly to run but it does very well. It outperforms non-o1 LLMs on MMLU-Pro and GPQA.
- Jan 28, 2025: Qwen 2.5 Max (18.6) added.
- Jan 22, 2025: Phi-4 (11.6), Nova Pro (12.5), Gemini 2.0 Flash Thinking Exp 01-21 (37.0) added.
- Jan 16, 2025: Gemini 2.0 Flash Thinking Exp, o1, MiniMax-Tex-o1 added. Gemini 2.0 Flash Thinking Exp sometimes hits the output token limit.
- Dec 27, 2024: GPT-4o 2024-11-20, Llama 3.3 70B, Gemini 2.0 Flash Exp, Deepseek-V3 added. Gemini 2.0 Flash Thinking Exp could not be benchmarked because its output gets cut off for some puzzles.
- Claude 3.5 Haiku added. 13.7.
- Claude 3.5 Sonnet (2024-10-22) added. Improves from 25.9 from 24.4.
- Grok Beta added. Improves from 21.3 to 23.7. It's described as "experimental language model with state-of-the-art reasoning capabilities, best for complex and multi-step use cases. It is the successor of Grok 2 with enhanced context length."
- Also check out the LLM Step Game, LLM Thematic Generalization Benchmark, LLM Creative Story-Writing Benchmark, LLM Confabulations , LLM Deceptiveness and Gullibility Benchmark, and LLM Divergent Thinking Creativity Benchmark.
- Follow @lechmazur on X (Twitter) for other upcoming benchmarks and more.