Dataset Access Problems

[ryangn2]

Overview

• download_raw_dataset.py raises a FileURLRetrievalError upon being run. It seems to be a permission issue, or the dataset seems to have been modified.
• There seems to be a mistake with the SQL file on line 30. The description_content_type was not selected in the latest_metadata CTE.

[ryangn2]

Proposed SQL File Fix:

WITH recent_downloads AS (
  SELECT
    LOWER(project) AS project_lower,
    project,
    COUNT(*) AS download_count
  FROM
    `bigquery-public-data.pypi.file_downloads`
  WHERE
    DATE(timestamp) BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY) AND CURRENT_DATE()
  GROUP BY
    LOWER(project), project
  HAVING
    COUNT(*) >= 100
),
latest_metadata AS (
  SELECT
    LOWER(name) AS name_lower,
    name,
    description,
    description_content_type,  -- Include field
    summary,
    version,
    upload_time,
    ROW_NUMBER() OVER (PARTITION BY LOWER(name) ORDER BY upload_time DESC) AS rn
  FROM
    `bigquery-public-data.pypi.distribution_metadata`
)
SELECT
  lm.name AS name,
  lm.description AS description,
  lm.description_content_type AS description_content_type, -- Retrieve field correctly here
  lm.summary AS summary,
  lm.version AS latest_version,
  rd.download_count AS number_of_downloads
FROM
  recent_downloads rd
JOIN
  latest_metadata lm
ON
  rd.project_lower = lm.name_lower
WHERE
  lm.rn = 1
ORDER BY
  rd.download_count DESC;