Skip to content

Add --dry-run to check how much data would be processed, but don't actually run query #168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hugovk merged 2 commits into from
Oct 15, 2025
Merged

Add --dry-run to check how much data would be processed, but don't actually run query #168

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6c39ade
Add --dry-run to check how many bytes would be billed, but don't actu…
hugovk Apr 28, 2025
2eb947e
Merge branch 'master' into dry-run
ofek Oct 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 28 additions & 27 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,36 +23,37 @@ pypinfo is a simple CLI to access [PyPI](https://pypi.org/) download statistics
```console
$ pypinfo
Usage: pypinfo [OPTIONS] [PROJECT] [FIELDS]... COMMAND [ARGS]...

Valid fields are:

project | version | file | pyversion | percent3 | percent2 | impl | impl-version |

openssl | date | month | year | country | installer | installer-version |

setuptools-version | system | system-release | distro | distro-version | cpu |
Valid fields are:

project | version | file | pyversion | percent3 | percent2 | impl | impl-version |

openssl | date | month | year | country | installer | installer-version |

setuptools-version | system | system-release | distro | distro-version | ci | cpu |

libc | libc-version

libc | libc-version

Options:
-a, --auth TEXT Path to Google credentials JSON file.
--run / --test --test simply prints the query.
-j, --json Print data as JSON, with keys `rows` and `query`.
-i, --indent INTEGER JSON indentation level.
-t, --timeout INTEGER Milliseconds. Default: 120000 (2 minutes)
-l, --limit INTEGER Maximum number of query results. Default: 10
-d, --days INTEGER Number of days in the past to include. Default: 30
-sd, --start-date TEXT Must be negative or YYYY-MM[-DD]. Default: -31
-ed, --end-date TEXT Must be negative or YYYY-MM[-DD]. Default: -1
-m, --month TEXT Shortcut for -sd & -ed for a single YYYY-MM month.
-w, --where TEXT WHERE conditional. Default: file.project = "project"
-o, --order TEXT Field to order by. Default: download_count
--all Show downloads by all installers, not only pip.
-pc, --percent Print percentages.
-md, --markdown Output as Markdown.
-v, --verbose Print debug messages to stderr.
--version Show the version and exit.
-h, --help Show this message and exit.
-a, --auth TEXT Path to Google credentials JSON file.
--run / --test --test simply prints the query.
-n, --dry-run Don't run query but display how much data would be processed.
-j, --json Print data as JSON, with keys `rows` and `query`.
-i, --indent INTEGER JSON indentation level.
-t, --timeout INTEGER Milliseconds. Default: 120000 (2 minutes)
-l, --limit INTEGER Maximum number of query results. Default: 10
-d, --days INTEGER Number of days in the past to include. Default: 30
-sd, --start-date TEXT Must be negative or YYYY-MM[-DD]. Default: -31
-ed, --end-date TEXT Must be negative or YYYY-MM[-DD]. Default: -1
-m, --month TEXT Shortcut for -sd & -ed for a single YYYY-MM month.
-w, --where TEXT WHERE conditional. Default: file.project = "project"
-o, --order TEXT Field to order by. Default: download_count
--all Show downloads by all installers, not only pip.
-pc, --percent Print percentages.
-md, --markdown Output as Markdown.
-v, --verbose Print debug messages to stderr.
--version Show the version and exit.
-h, --help Show this message and exit.
```

pypinfo accepts 0 or more options, followed by exactly 1 project, followed by
Expand Down
13 changes: 8 additions & 5 deletions pypinfo/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@
@click.argument('fields', nargs=-1, required=False)
@click.option('--auth', '-a', help='Path to Google credentials JSON file.')
@click.option('--run/--test', default=True, help='--test simply prints the query.')
@click.option('--dry-run', '-n', is_flag=True, help="Don't run query but display how much data would be processed.")
@click.option('--json', '-j', is_flag=True, help='Print data as JSON, with keys `rows` and `query`.')
@click.option('--indent', '-i', type=int, help='JSON indentation level.')
@click.option('--timeout', '-t', type=int, default=120000, help='Milliseconds. Default: 120000 (2 minutes)')
Expand All @@ -104,6 +105,7 @@ def pypinfo(
fields: list[str],
auth: str,
run: bool,
dry_run: bool,
json: bool,
indent: int,
timeout: int,
Expand Down Expand Up @@ -167,7 +169,7 @@ def pypinfo(

if run:
with create_client(get_credentials()) as client:
query_job = client.query(built_query, job_config=create_config())
query_job = client.query(built_query, job_config=create_config(dry_run))

@cursor cursor Bot Oct 8, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Dry Run Metadata Parsing Error

When dry_run=True, BigQuery queries return only metadata, not data rows. The code incorrectly attempts to retrieve and parse query results via query_job.result() and parse_query_result(), which causes errors or invalid data in subsequent processing.

Fix in Cursor Fix in Web

@ofek ofek Oct 9, 2025

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hugovk Legitimate?

@hugovk hugovk Oct 10, 2025

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get no errors running this:

❯ pypinfo --dry-run --all --days 1 --percent pillow pyversion
Served from cache: False
Data processed: 344.62 MiB
Data billed: 0.00 B
Estimated cost: $0.00

Adding some debug print:

            query_job = client.query(built_query, job_config=create_config(dry_run))
            print(query_job)
            query_rows = query_job.result(timeout=timeout // 1000)
            print(query_rows)
            rows = parse_query_result(query_rows)
            print(rows)

Gives:

❯ pypinfo --dry-run --all --days 1 --percent pillow pyversion
QueryJob<project=pypinfo-hugovk, location=US, id=None>
<google.cloud.bigquery.table._EmptyRowIterator object at 0x10b0e2f90>
[['python_version', 'download_count']]
Served from cache: False
Data processed: 344.62 MiB
Data billed: 0.00 B
Estimated cost: $0.00

Indeed, rows does not contain download data.

But this PR also adds not dry_run guards around subsequent processing, so we don't get errors.

query_rows = query_job.result(timeout=timeout // 1000)
rows = parse_query_result(query_rows)
Comment thread
ofek marked this conversation as resolved.

Expand All @@ -187,12 +189,12 @@ def pypinfo(
estimated_cost = Decimal(TIER_COST * billing_tier) / TB * Decimal(bytes_billed)
estimated_cost_str = str(estimated_cost.quantize(TO_CENTS, rounding=ROUND_UP))

if len(rows) == 1 and not json:
if len(rows) == 1 and not json and not dry_run:
# Only headers returned
click.echo("No data returned, check project name")
return

if percent:
if percent and not dry_run:
rows = add_percentages(rows, include_sign=not json)

# Only for tables, and if more than the header row + a single data row
Expand All @@ -205,8 +207,9 @@ def pypinfo(
click.echo(f'Data billed: {billed_amount:.2f} {billed_unit}')
click.echo(f'Estimated cost: ${estimated_cost_str}')

click.echo()
click.echo(tabulate(rows, markdown))
if not dry_run:
click.echo()
click.echo(tabulate(rows, markdown))
else:
query_info = {
'cached': from_cache,
Expand Down
5 changes: 4 additions & 1 deletion pypinfo/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,12 @@
Rows = list[list[str]]


def create_config() -> QueryJobConfig:
def create_config(dry_run: bool = False) -> QueryJobConfig:
config = QueryJobConfig()
config.use_legacy_sql = False
if dry_run:
config.dry_run = True
config.use_query_cache = False
return config


Expand Down
10 changes: 10 additions & 0 deletions tests/test_core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,19 @@ def test_create_config() -> None:
config = core.create_config()

# Assert
assert not config.dry_run
assert not config.use_legacy_sql


def test_create_config_dry_run() -> None:
# Act
config = core.create_config(dry_run=True)

# Assert
assert config.dry_run
assert not config.use_query_cache


def test_normalize_dates_yyy_mm() -> None:
# Arrange
start_date = "2019-03"
Expand Down
3 changes: 2 additions & 1 deletion tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@ deps =
commands =
coverage run --parallel-mode -m pytest -W all {posargs}
coverage combine --append
coverage report -m
coverage report --show-missing
coverage html
coverage xml

[testenv:lint]
Expand Down