Feature Request: include syntax in Snaplet config

[pwingo]

Hey there, thanks for writing this! We're finding this super useful both for local development and some stack data migration workflows particular to our application. One improvement request I have is as follows-

Our data workflow is to input a large amount of data to our staging instance, QA'ing it before promoting the data to production. In production, most of the tables can be happily overwritten with the up-to-date staging instance data, but some tables corresponding to users and user-generated content cannot be overwritten because staging does not have any of that data. We're using snaplet to accomplish the data transfer from staging to production, with the exclude syntax to specify tables that we don't want to overwrite.

This is totally fine for now, since we know our codebase well and wouldn't add new user-facing tables without adding a new exclusion for those tables, but I'm concerned about future editors of our codebase that add new tables without excluding them. The next time we do a data transfer, we would accidentally overwrite those tables.

In order to solve this issue, I'd like to have an include syntax where all tables are excluded by default, but we can include certain, specific tables that we'd like to make sure get copied over. This would make the potential worst case behavior much less damaging, since it would just mean certain data didn't get copied over as expected, and we could go back and fix that.

Thanks so much!
Patrick

[peterp]

We spoke on Discord, but I wanted to close the loop here. We're working on a feature that we're calling "partial restores." It'll allow you to only restore data, and related data, for certain tables.

But, more to this configuration. It's currently possible to do this programmatically because the transform export can also be a function which receives your current databases structure.

// .snaplet/transform.ts
export const transform = (database) => {
  console.log(database)  
  // ꜜꜜꜜ
  //  {
  //   structure: {
  //     '$schemas': [ 'public', 'pgboss' ],
  //     public: {
  //       '$tables': [Array],
  //       Organization: [Object],
  //       User: [Object]
  //     },
  //     pgboss: {
  //       '$tables': [Array],
  //       archive: [Object],
  //       job: [Object],
  //       schedule: [Object],
  //       version: [Object]
  //     }
  //   }
  // }
}

Using this object it's possible to programmatically exclude everything, except for the table you want to include:

// .snaplet/transform.ts
const config = (database) => {
  const config = {}
  for (const schema of database.structure.$schemas) {
    config[schema] = {}
    for (const table of database.structure[schema].$tables) {
      // use `null` or `false` to exclude a table.
      // add filtering logic to "include" specific tables.
      config[schema][table] = false
    }
  }
  return config
}