Create a page-based derivation create workflow

[travjenkins]

Last Updated: May 24, 2024

---

Create a page-based derivation create workflow that provides a more feature-rich experience for SQL derivations.

Below are the base requirements for this feature: (TBD)

• Create a page-based, derivation create workflow. The core page of this workflow is broken into three sections: the Database section, where the transforms and migration of the drafted derivation can be edited; the Schema section, where the derivation key and schema can be edited; and the Advanced Specification Editor section, where the derivation specification can be edited.

• Add basic derivation schema validation. Erroneous scenarios include: the Flow-templated key (i.e., your_key) is present in the schema.

• Add basic derivation database validation (e.g. transform and migration). Erroneous scenarios include: a transform lambda consisting of an empty string, a migration consisting of only an empty string.

• Model the form section containing the transform and migration configuration after the Target/Source Collections section of existing task workflows.

• Default the transform name to a versioned variant of the evaluated source collection table name when the evaluated table name is nonunique. For example, the name of a transform with the source collection melk/waffles/events would be events. A second transform with the source collection melk/pancakes/events would have the name events_v1.

• Display the transform name on each list item (as opposed to the faux filename).

• Enable the creation of multiple transforms using the same source collection. Currently, one instance of a transform using a specific source collection can be added at a time.

• Limit the number of migrations that can be created in the derivation create workflow to one.

• Remove individual transforms (and migration) via a CTA on the list item.

• Keep the specification editor in sync with the schema and SQL editors. Valid changes made in the schema and SQL editors are reflected in the specification editor. Valid changes made to the derivation schema in the specification editor are reflected by the contents of the Schema section.

Design

Last Updated: May 24, 2024

No design documents reflect the current state of this feature. The current design is an amalgamation of the initial, dialog-based redesign and existing task workflows.

Dependencies

Last Updated: May 24, 2024

• Derivation preview. The client needs a means to display a preview of the data transformed by the derivation (akin to the capabilities of flowctl).

Additional Comments

• The selection of derivation language is ambiguous. Users do not have enough context to determine which language is appropriate for their needs.

References

Design

• Implementation of page-based redesign | 2023-07-11. A video demonstration of the initial, page-based redesign of the derivation create workflow.

• Implementation of dialog-based redesign | 2023-06-01. A video demonstration of the initial, dialog-based redesign of the derivation create workflow.

• Dialog-based redesign | 2023-05-13. The initial, dialog-based redesign of the derivation create workflow. For reference, relevant frames of the design document are titled SQL or Transformation.