Adding Synapse Serverless single tech sample to work with Delta format and implement CLS

[ydaponte]

Problem Statement

• Synapse Serverless pools have some limitations when compared with SQL dedicated pools. The aim of this sample is to demonstrate how to work with Serverless pools and Delta files, how to automate the view creation of top of a Data Lake folder structure holding the Delta tables and finally how to apply CLS (Column-Level Security) on top of the Lake without the need of using third-party solutions.

Proposed Solution

Deploy a Synapse workspace with the Built-in Serverless pool and artifacts that will implement the following:

• Import an open-source parquet format dataset
• Convert parquet to delta format
• Dynamically create partitioned views on top of the delta lake tables folder structure using Synapse pipelines
• Apply CLS to the columns in the Delta table through the use of AD Groups using the Control plane in Azure and Synapse and Data Plane in Synapse.
• Document process, blockers and workarounds found.

Alternative Proposals

• Not applicable.

Additional Context

• Not applicable at the moment.

Next Steps

• [X] Team consensus to proceed
• [ ] Complete Design Review Template (if applicable)
• [ ] Schedule Design Session