Skip to content

[Epic] Add Support for Source Versioning / Tables from Sources #646

@bobbyiliev

Description

@bobbyiliev

Feature request

As per the discussion here: MaterializeInc/materialize#27907 and based on the design doc here: #645 we need to introduce the support for tables from sources on the Terraform provider side as well.

Extra details to be discussed in the design doc pull request, but the main things to keep in mind are:

Initial tasks

  • Deprecated the related table attributes on the source resources, eg. MySQL, Postgres, Loadgens
  • Introduce new resource_source_table_{mysql|postgres|load_generator} resources
  • Add tests including unit tests and acceptance tests
  • Handle source-specific attributes like the MySQL ignore_columns in the resource_table_from_source resource
  • Update all of the related documentation
  • Write a migrations guide for users using the old implementation
  • Add data sources resource_source_table resources
  • Add import and documentation examples
  • Once ready add upstream table name to read query and read funcs + acceptance tests
  • Add integration tests

Kafka Tasks

  • Deprecated the related table attributes on the source resources for the Kafka source resource
  • Introduce a new resource_source_table_kafka resource
  • Add tests including unit tests
  • Update all of the related Kafka source documentation
  • Add import and documentation examples
  • Update the migrations guide for users using the old implementation to include Kafka sources
  • Add acceptance tests
  • Add integration tests
  • Add the envelope, key_format and value_format columns to the read func

SQL server Tasks

  • Deprecated the related table attributes on the source resources for the SQL Server source resource
  • Introduce a new resource_source_table_sqlserver resource
  • Add tests including unit tests
  • Update all of the related SQL Server source documentation
  • Add import and documentation examples
  • Update the migrations guide for users using the old implementation to include SQL server sources
  • Add acceptance tests
  • Add integration tests

Webhook Tasks

  • Deprecated the related table attributes on the source resources for the Webhook source resource
  • Introduce a new resource_source_table_webhook resource
  • Add tests including unit tests, integration tests and acceptance tests
  • Update all of the related Webhook source documentation
  • Add import and documentation examples
  • Update the migrations guide for users using the old implementation to include webhooks sources

Additional Tasks

  • Data source for mz_internal.available_source_references
  • Add unit tests for the source reference data source
  • Add acceptance tests for the source reference data source
  • Add ALTER SOURCE .. REFRESH REFERENCES

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions