Skip to content

Designing vendoring feature command on BigQuery #85

Open
@takegue

Description

@takegue

Vendoring is a feature to move all third-party resources, such as public datasets and UDFs, into a specified dataset.
This has several benefits that can help make your data modeling more efficient and reliable.

Background

In BigQuery, some useful UDF is published vid public dataset such as bqutil.fn, bqmake.v0 and so on.
However, these UDFs maintained by community doesn't gurantee permanent availability.
Additionaly, BigQuery resctrict to use UDF only in same-region and users often meet this restriction.

By using vendoring, you can ensure that the third-party resources you rely on are always available and that any updates or changes made to them do not affect your analysis. This can be especially important if you are working with sensitive or business-critical data.
Additionally, vendoring allows you to use third-party resources in different regions.

Usecase

# Vendoring all resources 
bqport import project.user_dataset [external_dataset]
# Update dependencies
bqport update-dependencies project.user_dataset

Support BigQuery Resources

  • Routine (UDF/Procedure)
  • (Optional) Table

Scope levels

  • (Optional) Control to exclude sources
    • Intra-project resource
    • White-list project

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions