Skip to content

Not getting the full list of datasets when there are more than 50 datasets in the project #145

Open
@Bobain

Description

@Bobain

I think the code for get_datasets should be fixed with something like:

def get_datasets(self, project_id=None):
    project_id = self._get_project_id(project_id)

    try:            
        datasets = self.bigquery.datasets()
        request = datasets.list(projectId=project_id)
        result = request.execute(num_retries=self.num_retries)
        page_token = result.get('nextPageToken')
        while page_token:
            res = datasets.list(projectId=project_id, pageToken=page_token).execute(num_retries=self.num_retries)
            page_token = res.get('nextPageToken')
            result['datasets'] += res.get('datasets', [])
        return result.get('datasets', [])
    except HttpError as e:
        logger.error("Cannot list datasets: {0}".format(e))
        return None

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions