This project is a pipeline for solving and visualizing exam questions using LLMs. It consists of a Python backend to solve exams (only text data) with LLMs and a static frontend for viewing results. The results reveal a clear hierarchy: while cloud-based reasoning models like GPT 5 Mini and DeepSeek V3 lead with 96.1% accuracy, local models show the trade-off between privacy and capability.
| Model | Accuracy | Avg Time | Tokens/Q |
|---|---|---|---|
| GPT 5 Mini | 96.1% | 17.4s | 1.3k |
| DeepSeek V3 | 96.1% | 31.5s | 1.2k |
| Qwen 3.5 9B | 45.1% | 187.5s | 1.8k |
| Qwen 3.5 4B | 43.1% | 163.4s | 2.3k |
| Qwen 3.5 0.8B | 3.9% | 84.4s | 3.1k |
The project is primarily an experiment focused on testing the performance of local LLMs. The local environment was run on a 32GB RAM Lenovo Yoga powered by an AMD Ryzen AI 7 350 with Radeon 860M (2.00 GHz), utilizing the NPU through FastFlowLM.
In contrast to the cloud-based reasoning models, local models frequently hit safety limits due to hallucination loops. Our data shows that correct answers typically have thinking traces between 400 and 1,800 tokens. Any trace exceeding 3,000 tokens almost universally indicates a failure or a logic loop, suggesting that "longer isn't always better" for small models. Capping tokens to 2,000 was found to be the optimal balance to prevent infinite loops without sacrificing successful reasoning paths.
Conclusion: Local models are not yet a practical replacement for cloud models in intensive logic tasks. Cloud APIs offer a level of accuracy, speed, and parallelization capability (including batch processing) that currently remains far beyond the reach of local hardware.
The ground truth dataset (frontend/static/2026/2026-qa-all-phases.json) consists of 4 separate exams (corresponding to the 4 levels of the competition) which were merged into a single JSON file using Gemini 3 Pro. The original problems were sourced from the Concurso Primavera official site. While Gemini initially misindexed many questions (e.g., labeling Q1 as Q5), these indexing errors were manually corrected before running the pipeline.
The final dataset of 51 multiple-choice questions excludes 25 problems requiring image analysis and 4 that were incorrectly parsed (as defined in backend/aggregate.py). PDF parsing was a major bottleneck in the project; the library docling failed to produce usable results, and Gemini also struggled to provide consistently accurate parsing, necessitating the manual corrections mentioned above.
The backend scripts are located in the backend/ directory. All results are saved directly into the frontend folder, keeping the frontend independent and ready for static hosting (e.g., GitHub Pages) without a live backend server.
solver.py: Run this to solve exams (only text data) with LLMs. You can change parameters directly in the script. It supports OpenAI-compatible APIs and has a 4k token limit. Run with anexam_idargument for parallel execution.aggregate.py: Should only be run if the solver was executed via parallel instances to merge partial results and apply the blacklist.
The frontend/ directory is self-contained and designed to be deployed directly to GitHub Pages. It handles data loading and rendering from static/2026/ using app.js and index.html.