SQL parser lineage for MS SQL stored procedures do not create the lineage
Affected module ui/Ingestion Framework (we dont fully understand)
Describe the bug OpenMetadata do not create lineage for stored procedures, even for simpliest ones. We created example:
CREATE PROCEDURE indicators.TestProcedure
AS
INSERT INTO indicators.TestTargetTable (
Column2
,Column1)
SELECT
Column1 AS Column2
,Column2 AS Column1
FROM indicators.TestSourceTable
We ran metadata and lineage ingestions. Metadata ingestion was successful, but lineage tab was empty.
To Reproduce
- Create stored procedure in MS SQL
- Run metadata ingestion
- Check stored procedure lineage
Expected behavior We expected to see lineage between TestSourceTable and TestTargetTable and stored procedure on edge.
Version:
- Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-102-generic x86_64)
- Python version: Python 3.10.13
- OpenMetadata version: 1.4.7
- OpenMetadata Ingestion package version: openmetadata-ingestion==1.4.7.
Additional context We tried to do:
- We erased lines of stored procedure definition like:
CREATE PROCEDURE indicators.TestProcedure
AS
by rewriting OpenMetadata/ingestion/src/metadata/ingestion/source/database/mssql /queries.py because we found out that commenting commands like "CREATE PROCEDURE", "BEGIN", "END", "TRUNCATE", "DECLARE", "AS" gives correct lineage using SQL LINEAGE tool (which OM use for parsing lineages as i got). Example stored procedure below in link. https://reata.github.io/sqllineage/ - sql lineage web demo 2) But the lineage was also empty inside stored procedure lineage tab 3) We have executed the "insert"-query standaloned from example stored procedure
INSERT INTO indicators.TestTargetTable (
Column2
,Column1)
SELECT
Column1 AS Column2
,Column2 AS Column1
FROM indicators.TestSourceTable
and lineage was successfully produced.
Our questions are: 1)"openmetadata-sqllineage 1.1.2.0" really used in OpenMetadata. How can we set it to be used in lineage parsing process? 2)What workarounds can you recommend us to get the full lineage of DWH which uses stored procedures for ETL? https://github.com/open-metadata/OpenMetadata/issues/16737#issuecomment-2304484190