Skip to content

Show the Interceptor Pattern for Streaming Tables #28

@newfront

Description

@newfront

Brought up by @AndreGodinho7 in the Delta User's slack channel.

When we are migrating from Table A (call it "last stable") and we need to move to the new "stable" table, while not "interrupting" the streams until we are "ready". If there is a new set of "columns" then those can get picked up by the streaming applications without problems. However, if there is a completely re-written table structure, and we need to use a new "uuid" - then the change is more complex. We want to shoot for less complicated.

  • Consider that there are live customers "streaming data customers"
  • You want to create a "new" table to "replace" the old "tableA"
  • You want to ensure there is a "backup plan" - so you intercept the "table" and "redirect"

This is not a simple established pattern. It is a full "table aliasing" concept.

  • talk about ways to get this to work (with Trino for example - you can "redirect" tables to new locations) - with managed tables and a table UUID that changes with "Delta" - we will need a more "seamless" means to handle the redirection.

  • Include an official writeup (blog post) to go along with the pattern.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions