Look into dbt semantic layer

[jeancochrane]

Let's investigate dbt semantic layer to see if it might be something that would benefit our data lake.

[jeancochrane]

I did a little bit of reading today to try to get my head around the way that Semantic Layer works. Props to @ccao-jardine for surfacing this tool, I do think it's quite interesting and I could see it adding value to our data lake.

TL;DR: Semantic Layer seems promising and I think it could help us revive the Source of Truth project; however, it is not currently available to us given our current dbt stack, so I don't think it's worth pursuing right now. Read on for more details.

Here are the benefits that I think Semantic Layer could provide to our data lake:

• The central abstraction of Semantic Layer -- metrics that are independent of the SQL implementation, and can be aggregated across various dimensions without needing to write SQL -- strikes me as basically what we were trying to implement with our now-defunct Source of Truth project.
• The Tableau and Excel integrations seems like they would be convenient for UX.
  • Note however that we would likely need to work with IT to approve and enable the Excel add-on.

Note, however, that there are some serious downsides that are currently preventing us from deploying Semantic Layer resources:

• Semantic Layer currently does not support the Athena adapter. This means we could not actually use it unless we switched away from Athena and onto a new data lake provider, or if dbt decides to support the Athena adapter in the future.
• The Semantic Layer API is only available in dbt Cloud, and cannot be used in dbt Core (which is the open source package that we use).

[ccao-jardine]

Rad, thanks for investigating this more deeply! I agree that the downsides make this a no-go.