Feature: Allow Synapse-based Architecture

[MiguelElGallo]

Feature or Idea - What?

Include a paramater to make My-SQL optional

Feature or Idea - Why?

Modern workloads for data processing do not need a Azure My-Sql.

Code of Conduct

• I agree to follow this project's Code of Conduct

[marvinbuss]

Hi @MiguelElGallo, The MySQL Server is used as an external Hive metastore for the Databricks workspaces. Is Databricks something that you are planning to use? If so, I would propose to use that MySQL instance as an external Hive metastore.

[MiguelElGallo]

Hi,
Then my bad, make Databricks optional (which will make Mysql server optional).
We will use Synapse Spark with Delta Lakes files from moving from Raw to the next storage and so on.
This leads us to the other point I mentioned that the Synapse cannot access the storage account.
Thanks,

[marvinbuss]

Thanks @MiguelElGallo!
Good suggestion and I agree with you. For users that want to fully switch to Synapse, we should provide the option to make some of the resources optional.
So in summary you are proposing the following:

1. Replace ADF with Synapse
2. Make Databricks and all related resources optional

Can you confirm?

[MiguelElGallo]

Hi @marvinbuss ,
I confirm.
Yes, since we are aiming at a Synapsed based Architecture.
Thanks!

[marvinbuss]

Let me get this into the backlog and work on it. I will also discuss with the team and change the title of the issue to make this more clear.