Add library improvements (extracted from benchmark PR)

[claude]

This PR extracts the library improvements from #122, separating them from the benchmarking code.

Library Changes

• In-memory gradient collector: Add InMemoryCollector for small runs that don't need disk I/O
• Better distributed error logging: Improved error messages in bergson/distributed.py with Python 3.10+ compatibility
• Auto batch size CLI tool: New bergson auto_batch_size command to automatically find optimal batch sizes
• Build metadata in reduce: Include build metadata in reduce runs for better tracking
• max_tokens support: Add max_tokens parameter support with Dataset
• CLAUDE.md updates: Document project conventions and development practices
• Test coverage: Add tests for new functionality

Files Changed

Modified files:

• .pre-commit-config.yaml
• CLAUDE.md
• README.md
• bergson/__main__.py
• bergson/config.py
• bergson/distributed.py
• bergson/query/attributor.py
• bergson/reduce.py
• bergson/score/score.py
• bergson/score/score_writer.py
• bergson/score/scorer.py
• bergson/utils/utils.py
• bergson/utils/worker_utils.py
• examples/slurm/score_worker.sh
• pyproject.toml
• tests/test_build.py
• tests/test_score.py

New files:

• bergson/cli/auto_batch_size.py
• bergson/collector/in_memory_collector.py
• bergson/utils/auto_batch_size.py

Related

Extracted from #122 (benchmarking PR)

Co-authored-by: Lucia Quirke luciaquirke@users.noreply.github.com

[luciaquirke]

@claude add allowed_bots: 'claude' to the review workflow

[claude]

Claude finished @luciaquirke's task in 41s —— View job

---

Unable to complete

• Find the review workflow file
• Add allowed_bots: 'claude' to the workflow
• Commit and push changes - Cannot push workflow file changes

Issue

I successfully made the change to add allowed_bots: 'claude' to .github/workflows/claude-code-review.yml, but I'm unable to push workflow file modifications due to GitHub App permissions. GitHub prevents apps from modifying workflow files without the workflows permission for security reasons.

What needs to be done

You'll need to manually add this line to .github/workflows/claude-code-review.yml at line 42 (after the prompt parameter):

allowed_bots: 'claude'

The full section should look like:

      - name: Run Claude Code Review
        id: claude-review
        uses: anthropics/claude-code-action@v1
        with:
          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
          plugin_marketplaces: 'https://github.com/anthropics/claude-code.git'
          plugins: 'code-review@claude-code-plugins'
          prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ github.event.pull_request.number }}'
          allowed_bots: 'claude'
          # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
          # or https://code.claude.com/docs/en/cli-reference for available options

You can find more information about this limitation in the FAQ.

---