Skip to content

Fix spurious "Nested loops over ThisThreadPool" error - #2557

Merged
tapios merged 3 commits into
mainfrom
ts/race-fix
Jul 29, 2026
Merged

Fix spurious "Nested loops over ThisThreadPool" error#2557
tapios merged 3 commits into
mainfrom
ts/race-fix

Conversation

@tapios

@tapios tapios commented Jul 27, 2026

Copy link
Copy Markdown
Member

jl_in_threaded_region is a process-global flag, but parallelize_over(f, ::ThisThreadPool) read it twice. If an unrelated task entered a threaded region between the two reads, the first read said "not in a region" and the second said "in a region", leading to an error without actual nesting.

This affects any code that broadcasts Fields from several concurrent tasks. The plane/inertial_gravity_wave example does that via ThreadsX.mapreduce, and fails intermittently (PR #2556, build 7117).

CPU Multithreading & Scope Fixes

  • Scope Resolution: Loops resolve their scope once up front with resolved_scope, and thread rank comes from task-local storage (:climacore_pool_threads) rather than global state.
  • Thread Pool Partitioning: Concurrent loops divide the CPU pool dynamically, each taking an equal share of threads instead of letting the first loop claim all threads. This is ~2.6x faster with two concurrent tasks on 8 threads.
  • Explicit Scope & Task Spawning: Explicit ThisThreadPool() calls now resolve properly. Pool threads are spawned explicitly into :default with @spawn :default (protecting against execution inside :interactive tasks), and parallel worker errors are collected and thrown via CompositeException.

GPU Performance Optimizations (ClimaCoreCUDAExt)

  • Device Attribute Caching: Cache invariant CUDA device attributes (MAX_THREADS_PER_BLOCK, WARP_SIZE, SM_COUNT, etc.) to eliminate repeated C driver calls (cuDeviceGetAttribute) on every launch. Reduces host-side broadcast launch latency by ~54x (from ~477 μs down to ~8.8 μs per launch on an A100).
  • Occupancy Query Caching: Cache invariant kernel launch configurations (CUDA.launch_configuration) in an IdDict to eliminate repeated C driver occupancy calculations (cuOccupancyMaxPotentialBlockSize) on every broadcast and reduction launch. Saves an additional ~2 μs of host CPU launch overhead per kernel call (~4.2% faster reductions).
  • Task-Local Reduction Buffers: Intermediate GPU reduction arrays in reduce_points are cached in task-local storage to avoid allocating device memory (similar) during repeated scalar reductions.
  • Warp Alignment: Ensure thread block sizes for slice loops are rounded to warp multiples (32) to prevent warp divergence.

CI & Workflow Improvements

  • Documentation Deploy Concurrency: Added a concurrency group (documentation-deploy) to .github/workflows/docs.yml to queue documentation preview deployments sequentially, preventing git --force-with-lease rejections (! [rejected] HEAD -> gh-pages (stale info)) when multiple CI jobs run concurrently.

@tapios
tapios force-pushed the ts/race-fix branch 2 times, most recently from ee71a71 to 15810d2 Compare July 28, 2026 22:01
Comment thread src/DataLayouts/scopes.jl Outdated
Comment thread ext/cuda/loops.jl Outdated
Comment thread ext/cuda/loops.jl Outdated
Comment thread ext/cuda/cuda_utils.jl Outdated
Comment thread src/DataLayouts/loops.jl Outdated
Comment thread src/DataLayouts/loops.jl Outdated
…s per launch; reduces launch latency 54x - Cache GPU reduction intermediate arrays in task-local storage to avoid allocations.
@tapios
tapios enabled auto-merge July 29, 2026 06:18
Documenter deploys with `git push --force-with-lease`, so a run whose
clone of `gh-pages` is stale by the time it pushes fails with "cannot
lock ref 'refs/heads/gh-pages'". Two overlapping docbuilds of the same
ref race each other this way.

docs.yml was the only workflow without a concurrency group; add the one
the other workflows use, so a new run supersedes an in-progress one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tapios
tapios merged commit 0c39d5d into main Jul 29, 2026
23 of 31 checks passed
@tapios
tapios deleted the ts/race-fix branch July 29, 2026 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants