Skip to content

Batches overcounted when max_id divisible by query_batch_size #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pkrawczun wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Batches overcounted when max_id divisible by query_batch_size #1

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion simbad_query_hipsaohdhr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import pathlib
import tqdm
import math
from astropy.table import Table, vstack

from py.utils import custom_simbad
Expand Down Expand Up @@ -30,7 +31,7 @@
if combined_path.exists(): # if the combined file already exists, skip
print(f"Skipping {subdir.name} as the combined file already exists. If you want to re-query, delete the combined file.")
continue
for batch in tqdm.tqdm(range(max_id // query_batch_size + 1), desc=f"Querying {subdir.name}"):
for batch in tqdm.tqdm(range(math.ceil(max_id / query_batch_size)), desc=f"Querying {subdir.name}"):
max_id_clipped = min(max_id, batch * query_batch_size + query_batch_size)
ids = [f"{subdir.name.upper()} {str(i)}".strip() for i in range(1 + batch * query_batch_size, 1 + max_id_clipped)]
result = custom_simbad.query_objects(ids)
Expand Down