Web: Updates for Table and Column Lineage

[phixMe]

Problem

We currently have a feature gap for column lineage where we don't have a display for it. In order to correctly display this we needed to have something to handle nested graphs so that columns can be nested under tables.

This PR both creates a new page for column lineage and an updated view for lineage with a common set of shared principles.

Solution

Feature Set

• Redesigned lineage page
• New column lineage page
• Refresh button (lineage + column lineage)
• Compact mode for lineage display
• Zooming and framing capabilities and zoom controls
• Open view for graphs with a side panel instead of a draggable bar that limited graph navigation and visualization.
• Shared action menu for graph context and actions
• Tuning down the graph depth and preserving more in the url.

Technical

• New client side rendering engine with out of the box support for arbitrarily nested graphs (subgraphs) and hot swappable rendering strategies. There is also an easy to use and type safe framework to construct node types
• New custom React based d3 manipulation framework for zooming, panning, transitions, framing and focal points that uses React paradigms instead of direct DOM access like a conventional d3 approach
• A Minimap that allows for context awareness in a larger graph along with auto positioning extents

Marquez.Lineage.mov

One-line summary:
New views for both table and column lineage with shared new engine. Marks column lineage as beta for preview.

Checklist

• You've signed-off your work
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
• You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
• You've included a header in any source code files (if relevant)

[netlify]

✅ Deploy Preview for peppy-sprite-186812 ready!

Name	Link
🔨 Latest commit	23959b3
🔍 Latest deploy log	https://app.netlify.com/sites/peppy-sprite-186812/deploys/65c64fe4b4440b00082ae0d1
😎 Deploy Preview	https://deploy-preview-2725--peppy-sprite-186812.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (b111d64) 84.45% compared to head (23959b3) 84.45%.

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #2725   +/-   ##
=========================================
  Coverage     84.45%   84.45%           
  Complexity     1416     1416           
=========================================
  Files           251      251           
  Lines          6447     6447           
  Branches        291      291           
=========================================
  Hits           5445     5445           
  Misses          850      850           
  Partials        152      152           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[davidsharp7]

Hey have we got rid of the "full" switch to just show critical path? I know we have the minimap but that feature was massively helpful for large graphs.

[phixMe]

Hey have we got rid of the "full" switch to just show critical path? I know we have the minimap but that feature was massively helpful for large graphs.

I missed this one in the migration, but I added it back in. 😄

[davidsharp7]

Hey have we got rid of the "full" switch to just show critical path? I know we have the minimap but that feature was massively helpful for large graphs.

I missed this one in the migration, but I added it back in. 😄

Yeah that's looking good - I guess do we need to repeat this info?

Not overly bothered either way as its not impacting anything.

That other PR I have open should sort out this part and make it a nicer fit with the new UI.

[phixMe]

@davidsharp7 Yes, I'm just going to repeat the tags for now. I think it doesn't really need it's own tab in the layout in the longer term. I also added some cleanup to this in the subsequent PR #2727.

[JDarDagran]

Hey @phixMe , the update looks amazing 🚀 I gave it a test and sent some simple lineage events from Airflow DAG.
Attached example events.
events.json

Job page looks good:

but whenever I click dataset from the graph page goes blank:

Also, if I go to column lineage view I see this:

and clicking anything on the graph results in:

Happy to provide more details if needed 😄

[phixMe]

@pawel-big-lebowski Looks like some encoding issues on the namespaces as far as I can tell. I'll try out your events and hopefully get that resolved.

[phixMe]

@JDarDagran
Yes, it was an encoding issue with the : delimiter... I think think the larger issue is null fields like I have in my payload here.

I'm expecting these to be non-null at the moment.

I can remove the null data nodes, but that still looks wrong.

[JDarDagran]

Tested some cases and it works excellent! Great work @phixMe!

[wslulciuc]

New lineage layout rendering engine, column-level lineage support, and so much more to kick off 2024! Amazing work, @phixMe 💯 🥇