data-landing-zone icon indicating copy to clipboard operation
data-landing-zone copied to clipboard
verified publisher icon Azure

Feature: Allow Synapse-based Architecture

Open MiguelElGallo opened this issue 4 years ago • 5 comments

Feature or Idea - What?

Include a paramater to make My-SQL optional

Feature or Idea - Why?

Modern workloads for data processing do not need a Azure My-Sql.

Code of Conduct

MiguelElGallo avatar Jan 27 '22 12:01 MiguelElGallo

Hi @MiguelElGallo, The MySQL Server is used as an external Hive metastore for the Databricks workspaces. Is Databricks something that you are planning to use? If so, I would propose to use that MySQL instance as an external Hive metastore.

marvinbuss avatar Jan 27 '22 15:01 marvinbuss

Hi, Then my bad, make Databricks optional (which will make Mysql server optional). We will use Synapse Spark with Delta Lakes files from moving from Raw to the next storage and so on. This leads us to the other point I mentioned that the Synapse cannot access the storage account. Thanks,

MiguelElGallo avatar Jan 27 '22 15:01 MiguelElGallo

Thanks @MiguelElGallo! Good suggestion and I agree with you. For users that want to fully switch to Synapse, we should provide the option to make some of the resources optional. So in summary you are proposing the following:

  1. Replace ADF with Synapse
  2. Make Databricks and all related resources optional

Can you confirm?

marvinbuss avatar Jan 28 '22 09:01 marvinbuss

Hi @marvinbuss , I confirm. Yes, since we are aiming at a Synapsed based Architecture. Thanks!

MiguelElGallo avatar Jan 28 '22 09:01 MiguelElGallo

Let me get this into the backlog and work on it. I will also discuss with the team and change the title of the issue to make this more clear.

marvinbuss avatar Jan 28 '22 14:01 marvinbuss