Replace manual batchee grouping in harmony-autotester with batchee function - #121
Replace manual batchee grouping in harmony-autotester with batchee function#121kecunning wants to merge 1 commit into
Conversation
📝 WalkthroughWalkthroughThe Sambah test now uses ChangesSambah batch grouping test
Estimated code review effort: 1 (Trivial) | ~2 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/sambah/test_sambah.py`:
- Line 40: Update the grouping setup around _group_batch_indices so grouped
results retain the original granule dictionaries rather than only filename
strings. Pass granules into _group_batch_indices if supported, or remap each
grouped filename back to its corresponding granule before the later selection
and generate_partial_spatial_box calls.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: c48619e3-5a5d-433f-9294-ab5a3497319d
📒 Files selected for processing (1)
tests/sambah/test_sambah.py
|
|
||
| for k, v in zip(batch_indices, granules, strict=False): | ||
| grouped[k].append(v) | ||
| grouped = _group_batch_indices(batch_indices, granule_names) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repository files matching test_sambah.py:"
fd -a 'test_sambah\.py$' . || true
echo
echo "Relevant snippets:"
for f in $(fd 'test_sambah\.py$' .); do
echo "--- $f"
wc -l "$f"
ast-grep outline "$f" --view compact || true
sed -n '1,120p' "$f" | cat -n
done
echo
echo "Search for helper and generate_partial_spatial_box usages:"
rg -n "_group_batch_indices|generate_partial_spatial_box|granule" .Repository: nasa/harmony-autotester
Length of output: 9870
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Find batchee harmony util implementations:"
fd -a 'util\.py$' . || true
rg -n "def _group_batch_indices|_group_batch_indices" .
echo
echo "Inspect files containing _group_batch_indices:"
for f in $(rg -l "_group_batch_indices" .); do
echo "--- $f"
wc -l "$f"
sed -n '1,220p' "$f" | cat -n
done
echo
echo "Inspect relevant helper code:"
sed -n '1,45p' tests/umm_g_utilities.py | cat -nRepository: nasa/harmony-autotester
Length of output: 8147
🌐 Web query:
batchee harmony._group_batch_indices
💡 Result:
The term _group_batch_indices refers to an internal grouping function within the NASA Batchee Python package [1][2]. Batchee is a service designed to intelligently group filenames together to enable efficient batch operations, such as concatenation, within the NASA Harmony service orchestrator [3][4][5]. In recent development (notably around June 2026), the grouping logic previously duplicated across different modules—specifically in batchee/harmony/service_adapter.py and batchee/tempo_filename_parser.py—was refactored into a single, reusable function [1][2]. This utility function, often identified by names similar to _group_batch_indices (or located within the utility modules like batchee/harmony/util.py), centralizes the logic that determines how input files are organized into batches for downstream processing [2]. This refactoring was performed to improve code maintainability and allow the same grouping logic to be utilized by other components, such as the harmony-autotester [1].
Citations:
- 1: Separate grouping logic into reusable function batchee#215
- 2: nasa/batchee@04b3c7e
- 3: https://github.com/nasa/batchee
- 4: https://pypi.org/project/batchee/
- 5: https://harmony.earthdata.nasa.gov/docs
🌐 Web query:
site:github.com/nasa/batchee "_group_batch_indices"
💡 Result:
group_batch_indices is a Python function within the NASA Batchee package, a tool designed to intelligently group filenames to facilitate batch operations such as concatenation [1][2][3]. The function was introduced as a refactoring measure to consolidate duplicated grouping logic that previously existed across multiple modules, such as the TEMPO filename parser and the service adapter [1][2]. By moving this logic into a reusable function, the development team improved code maintainability and enabled consistent grouping behavior throughout the package [1][2]. The Batchee package itself is used within the NASA Harmony service to analyze filename patterns and organize related files into logical batches for processing [3][4].
Citations:
- 1: Separate grouping logic into reusable function batchee#215
- 2: nasa/batchee@04b3c7e
- 3: https://github.com/nasa/batchee
- 4: https://github.com/nasa/batchee/blob/develop/README.md
🌐 Web query:
github.com/nasa/batchee batchee harmony util.py _group_batch_indices source
💡 Result:
The function _group_batch_indices in the nasa/batchee repository is located in the file batchee/harmony/util.py [1][2]. This function was introduced as part of a refactoring effort (referenced in issue #215 and pull request #220) to consolidate duplicated grouping logic that previously existed in other modules, such as batchee/tempo_filename_parser.py and batchee/harmony/service_adapter.py [3][2]. The purpose of _group_batch_indices is to provide a reusable utility for grouping filenames and their corresponding batch indices, which is a core function of the Batchee service [3][4][5]. For the exact source code, you can view the file directly in the repository at: https://github.com/nasa/batchee/blob/main/batchee/harmony/util.py [2][6].
Citations:
- 1: nasa/batchee@e34228f
- 2: TRADE-1167: Refactored grouping logic into separate function for reuse batchee#220
- 3: Separate grouping logic into reusable function batchee#215
- 4: https://github.com/nasa/batchee
- 5: https://harmony.earthdata.nasa.gov/docs
- 6: nasa/batchee@04b3c7e
🌐 Web query:
raw.githubusercontent.com nasa/batchee main batchee harmony util.py _group_batch_indices
💡 Result:
In the nasa/batchee project, the function _group_batch_indices is a utility located in the batchee/harmony/util.py file [1][2]. This function was introduced as part of the effort to refactor and consolidate redundant filename grouping logic that was previously duplicated across different modules, such as batchee/tempo_filename_parser.py and batchee/harmony/service_adapter.py [3][4]. The refactoring, tracked under issue #215 and implemented via pull request #220 in June 2026, moved this logic into a single, reusable function to improve maintainability and support automated testing [3][4][1]. The batchee package itself is a NASA Harmony service designed to group files into batches for operations like concatenation [5][6].
Citations:
- 1: nasa/batchee@e34228f
- 2: nasa/batchee@04b3c7e
- 3: Separate grouping logic into reusable function batchee#215
- 4: TRADE-1167: Refactored grouping logic into separate function for reuse batchee#220
- 5: https://github.com/nasa/batchee
- 6: https://github.com/nasa/batchee/blob/develop/README.md
🌐 Web query:
raw.githubusercontent.com nasa/batchee main batchee harmony/util.py _group_batch_indices
💡 Result:
The _group_batch_indices function is a utility component found in the batchee/harmony/util.py file of the NASA batchee repository [1][2]. The batchee project is a Python tool designed for the NASA Harmony service to intelligently parse and group filenames (such as TEMPO L2 data files) based on specific patterns, facilitating batch operations like concatenation [3][4]. The introduction of _group_batch_indices was part of a refactoring effort (implemented in June 2026) to consolidate redundant grouping logic that had previously been duplicated across different modules within the batchee project, such as service_adapter.py and tempo_filename_parser.py [5][2]. By centralizing this logic into a reusable utility function, the developers improved code maintainability and enabled easier testing [5][2]. It should be noted that this repository is distinct from the Apache BatchEE project, which is a separate Java-based implementation of JBatch [6].
Citations:
- 1: nasa/batchee@e34228f
- 2: nasa/batchee@04b3c7e
- 3: https://github.com/nasa/batchee
- 4: https://github.com/nasa/batchee/blob/develop/README.md
- 5: Separate grouping logic into reusable function batchee#215
- 6: https://geronimo.apache.org/batchee/
🌐 Web query:
"def _group_batch_indices" "batchee"
💡 Result:
A search of common Python libraries and code repositories indicates that _group_batch_indices is not a standard function in major Python libraries such as NumPy, PyTorch, or the Python standard library [1][2][3][4][5]. The name suggests a helper function used to organize or group indices for batching operations, typically found in custom data loading or optimization logic. While similar functionality exists in libraries—such as PyTorch's GroupedBatchSampler [1] or various grouping implementations in Xarray [5]—the exact function name _group_batch_indices appears to be a user-defined or internal implementation detail specific to a private codebase or a specific research script [6]. If you encountered this function in a specific project, it is likely intended to partition data indices into groups based on certain criteria (such as sequence length or category) to facilitate efficient mini-batch processing [6]. To understand its specific behavior, you should inspect the source code where it is defined, typically within the project's data loading or utility modules.
Citations:
- 1: https://github.com/facebookresearch/detectron2/blob/main/detectron2/data/samplers/grouped%5Fbatch%5Fsampler.py
- 2: https://github.com/vllm-project/vllm-gaudi/blob/dbb090ca/vllm_gaudi/extension/unified_batch.py
- 3: https://github.com/pytorch/pytorch/blob/ec673ecd/torch/_inductor/fx_passes/group_batch_fusion.py
- 4: https://github.com/pytorch/pytorch/blob/2d31336e/torch/_inductor/fx_passes/bucketing.py
- 5: https://github.com/pydata/xarray/blob/v2026.04.0/xarray/core/groupby.py
- 6: https://gist.github.com/alexshtf/a434024013c07c4100bbe84e7bc4d580
🌐 Web query:
github batchee harmony/util.py _group_batch_indices source
💡 Result:
The function _group_batch_indices is not a standard or public API in the nasa/batchee project, nor is it a widely recognized function in the Python ecosystem. Research indicates that _group_batch_indices was introduced as a private, internal utility function within the NASA Batchee project (a service for grouping files for batch operations in Harmony) [1]. Specifically, it was implemented as part of a refactoring effort (ticket TRADE-1167) to move duplicated grouping logic into a reusable location within the package's internal utility module, located at batchee/harmony/util.py [2][3][4][5]. Because this function is intended as an internal helper for the Batchee package itself—to assist in grouping input arguments and batch indices [2]—it is not part of the public-facing API and may not be documented in standard project manuals [6]. Users interacting with Batchee typically utilize the command-line interface or the primary Python API for file grouping rather than accessing internal utility functions [1][6].
Citations:
- 1: https://github.com/nasa/batchee
- 2: Separate grouping logic into reusable function batchee#215
- 3: TRADE-1167: Refactored grouping logic into separate function for reuse batchee#220
- 4: nasa/batchee@e34228f
- 5: nasa/batchee@04b3c7e
- 6: https://pypi.org/project/batchee/
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Check installed/bound batchee source packages:"
python3 - <<'PY'
import importlib.util
from pathlib import Path
for name in ["batchee.harmony.util", "batchee.util", "batchee"]:
spec = importlib.util.find_spec(name)
print(f"{name}: {spec.origin if spec else None}")
PY
echo
echo "Search repository requirements and installed eggs:"
rg -n "batchee|Harmony" -g 'requirements*.txt' -g 'pyproject.toml' -g 'Pipfile' -g 'poetry.lock' -g 'setup.py' -g '*.egg-link' . || trueRepository: nasa/harmony-autotester
Length of output: 459
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "All dependency and config files:"
git ls-files | rg '(^|/)(requirements.*\.txt|pyproject\.toml|setup\.py|setup\.cfg|Pipfile|poetry\.lock|environment.*\.ya?ml|uv\.lock|.*\.egg-info/.*|.*\.egg-link|.*\.egg-info)$' || true
echo
echo "Dependency mentions:"
rg -n "batchee|harmony|batchee" -g 'requirements*.txt' -g 'pyproject.toml' -g 'setup.py' -g 'setup.cfg' -g 'Pipfile' -g 'poetry.lock' -g 'environment*.yml' -g 'environment*.yaml' -g '*.egg-info/*' . || true
echo
echo "Import of get_batch_indices target:"
fd -a 'tempo_filename_parser\.py$' . || true
for f in $(fd 'tempo_filename_parser\.py$' .); do
echo "--- $f"
sed -n '1,180p' "$f" | cat -n
doneRepository: nasa/harmony-autotester
Length of output: 511
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Inspect package manifests:"
for f in pyproject.toml tests/sambah/requirements.txt bin/requirements.txt tests/casper/requirements.txt tests/hybig/requirements.txt tests/net2cog/requirements.txt; do
echo "--- $f"
if [ -f "$f" ]; then
cat -n "$f"
fi
done
echo
echo "Search for _group_batch_indices in repository and dependency declarations:"
rg -n "_group_batch_indices|def get_batch_indices|get_batch_indices" . || true
echo
echo "Probe installed batchee package if available:"
python3 - <<'PY'
import importlib, inspect, textwrap, sys
try:
m = importlib.import_module("batchee.harmony.util")
print("module:", m.__file__)
src = inspect.getsource(m)
for name in ["_group_batch_indices", "get_batch_indices"]:
print("---", name)
try:
obj = getattr(m, name)
print(inspect.getsource(obj))
except Exception as e:
print(type(e).__name__, e)
except ModuleNotFoundError as e:
print("ModuleNotFoundError:", e)
PYRepository: nasa/harmony-autotester
Length of output: 2532
🌐 Web query:
raw.githubusercontent.com nasa/batchee v1.5.2 batchee harmony util.py
💡 Result:
The nasa/batchee project is a Python package developed for the NASA Harmony service that groups filenames together to enable efficient batch operations, such as concatenation [1][2]. The file path specified, batchee/harmony/util.py, is a component of this package that contains utility functions used by the Harmony service adapter [3]. In version 1.5.2, as well as in recent development, batchee/harmony/util.py has been utilized to house refactored logic for filename grouping [3][4][5]. Specifically, recent updates (such as those related to pull request #220) involved refactoring grouping logic into reusable functions within util.py to reduce code duplication across different modules of the package, including the harmony/service_adapter.py and tempo_filename_parser.py modules [3][4][5]. Batchee is part of the SAMBAH (Subsetter And Multi-dimensional Batched Aggregation in Harmony) service chain, which processes data granules for Earth science applications [6][7]. The package is available on PyPI [8] and maintained on GitHub [1].
Citations:
- 1: https://github.com/nasa/batchee
- 2: https://github.com/nasa/batchee/blob/develop/README.md
- 3: nasa/batchee@e34228f
- 4: TRADE-1167: Refactored grouping logic into separate function for reuse batchee#220
- 5: Separate grouping logic into reusable function batchee#215
- 6: https://harmony.earthdata.nasa.gov/docs
- 7: https://stitchee.readthedocs.io/en/latest/sambah_readme/
- 8: https://pypi.org/project/batchee/
🌐 Web query:
fetch https://raw.githubusercontent.com/nasa/batchee/v1.5.2/batchee/harmony/util.py
💡 Result:
The file batchee/harmony/util.py is part of the nasa/batchee Python package [1][2], a service within the NASA Harmony ecosystem used for grouping filenames together to facilitate batch operations like concatenation [1][3]. In version 1.5.2 of the package, the batchee/harmony/util.py module was updated to contain refactored, reusable grouping logic [4][5]. Specifically, this module was modified as part of a refactoring effort (referenced in pull request #220) to centralize grouping functionality that was previously duplicated in other parts of the codebase, such as service_adapter.py and tempo_filename_parser.py [4][5][6]. You can view the source code for this file directly in the official NASA Harmony Batchee GitHub repository at https://github.com/nasa/batchee/blob/v1.5.2/batchee/harmony/util.py.
Citations:
- 1: https://github.com/nasa/batchee
- 2: https://pypi.org/project/batchee/
- 3: https://harmony.earthdata.nasa.gov/docs
- 4: nasa/batchee@e34228f
- 5: TRADE-1167: Refactored grouping logic into separate function for reuse batchee#220
- 6: Separate grouping logic into reusable function batchee#215
🌐 Web query:
site:raw.githubusercontent.com/nasa/batchee/v1.5.2 "def _group_batch_indices"
💡 Result:
There is no standard or widely recognized Python function named _group_batch_indices. The term appears to be a specific internal naming convention used within the PyTorch ecosystem or custom implementations for data handling. In the provided search results, variations of batch indexing and grouping logic appear in the following contexts: 1. PyTorch Inductor (torch/_inductor/fx_passes/bucketing.py): This module contains logic for parsing flat lists of grouped indices that use -1 as a delimiter to separate different groups [1]. This is used for grouping operations during graph compilation [1]. 2. Detectron2 (detectron2/data/samplers/grouped_batch_sampler.py): This library implements a GroupedBatchSampler class, which buffers indices based on group IDs and yields batches containing elements from only one group [2]. 3. PyTorch Geometric (torch_geometric/data/batch.py): This module uses an index_select method to handle batch indexing for graph data, but it does not use a function named _group_batch_indices [3]. If you encountered _group_batch_indices in a specific codebase, it is likely a private helper function used internally to partition or organize indices for batch processing.
Citations:
- 1: https://github.com/pytorch/pytorch/blob/ec673ecd/torch/_inductor/fx_passes/bucketing.py
- 2: https://github.com/facebookresearch/detectron2/blob/main/detectron2/data/samplers/grouped%5Fbatch%5Fsampler.py
- 3: https://github.com/pyg-team/pytorch_geometric/blob/76ff9c2c/torch_geometric/data/batch.py
🌐 Web query:
github.com nasa/batchee main batchee harmony/util.py
💡 Result:
The nasa/batchee repository is a component of the NASA Harmony project, designed to intelligently group filenames together for efficient batch operations, such as concatenation [1][2]. Regarding the file path batchee/harmony/util.py within the nasa/batchee repository: - This file exists as a module within the project [3]. - It is a source code file utilized within the project's logic, and recent development activity (such as commit e34228f) has involved refactoring and updating the grouping logic contained within this utility file to promote code reuse across the package [3][4]. Note that there are other projects in the ecosystem with similarly named files (e.g., in home-assistant-core or older test suites), but within the context of the nasa/batchee repository, batchee/harmony/util.py is an internal utility script supporting the service's core file-grouping functionality [3][5][6].
Citations:
- 1: https://github.com/nasa/batchee
- 2: https://github.com/nasa/batchee/blob/develop/README.md
- 3: nasa/batchee@e34228f
- 4: Separate grouping logic into reusable function batchee#215
- 5: scop/home-assistant-core@7a2820d
- 6: https://github.com/vanto/home-assistant-core/blob/1f425b19427a782da9ac52f18c903e6717fb44ca/.coveragerc
Preserve granule dictionaries when grouping.
_group_batch_indices(batch_indices, granule_names) returns grouped filename strings, but the test later treats each selected item as a granule dict at line 52 and passes selected items to generate_partial_spatial_box. Pass granules into _group_batch_indices, or remap grouped filenames back to granules before selection.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/sambah/test_sambah.py` at line 40, Update the grouping setup around
_group_batch_indices so grouped results retain the original granule dictionaries
rather than only filename strings. Pass granules into _group_batch_indices if
supported, or remap each grouped filename back to its corresponding granule
before the later selection and generate_partial_spatial_box calls.
|
This PR is on hold awaiting the release of the latest batchee in pypi. |
1 participant
Description
Replaced manual batchee grouping in the sambah test with batchee grouping utility function
Jira Issue ID
TRADE-1227
Local Test Steps
Run the sambah test
PR Acceptance Checklist
CHANGELOG.mdupdated to include high level summary of PR changes.Summary by CodeRabbit