Increased timeouts in submission evaluations in March 2026

[bmesuere]

Between March 6th and March 19th, 2026, some student submissions experienced more timeouts than usual. Submissions that should have been evaluated correctly were instead marked as timed out. This affected evaluations across all exercise types, with SQL and C# exercises being the most visibly impacted.

Incident Summary

• Date and Time: March 6th, 2026 (evening)
• Duration: ~13 days (resolved March 19th)
• Impact: Roughly 10% of SQL evaluations were incorrectly marked as timed out between March 6th and March 17th. C# and other exercise types were also affected, though to a lesser extent. Resubmitting often resolved the issue for users.

Root Cause Analysis

There were 2 separate issues that contributed to the increased timeouts.

Issue 1: Platform update increased baseline memory usage

A platform update deployed on March 6th slightly increased the baseline memory used during evaluations. Exercises previously running just below their set memory limit now occasionally exceeded it. This primarily affected SQL and C# exercises, which were already running close to their allowed memory limits. The increased memory pressure led to slower executions, causing evaluations to exceed their time limits, making them appear as timeouts.

Issue 2: Higher platform load due to a large upcoming assessment

Starting March 9th, platform load increased due to an upcoming assessment for a large group of students. Due to the nature of these exercises, there was a higher than normal load on our worker servers, slowing down execution across all exercise types.

Mitigation and Future Prevention

We increased the memory limits for exercises that were running too close to their threshold. We also manually re-evaluated submissions that had been incorrectly marked as timed out, so students do not need to resubmit.

To prevent similar incidents in the future, we are taking the following steps:

• Better monitoring: We are improving our monitoring to detect unusual timeout patterns earlier, so we can act before issues are widely felt.
• Configurable minimum memory limits: We are making minimum memory allocations configurable per judge and per programming language within judges, so that memory floors can be tailored to the needs of specific exercise types.
• Reduced concurrent evaluations: We are reducing the number of evaluations that run simultaneously on a single server, lowering the chance of resource contention during peak periods.

Timeline

• March 6th (evening): Platform update deployed
• March 9th — Increased platform load begins
• March 16th (morning): First user reports of unexpected timeouts
• March 17th, 8:00: Root cause identified
• March 17th, 9:00: Memory limits increased for largest SQL and C# exercise repositories
• March 17th–19th: Memory limits increased for all remaining affected repositories; affected submissions manually re-evaluated