OpenMetadata icon indicating copy to clipboard operation
OpenMetadata copied to clipboard

SQL parser lineage for MS SQL stored procedures do not create the lineage

Open RabochiyYan opened this issue 1 year ago • 3 comments

Affected module ui/Ingestion Framework (we dont fully understand)

Describe the bug OpenMetadata do not create lineage for stored procedures, even for simpliest ones. We created example:

CREATE PROCEDURE indicators.TestProcedure

AS

	INSERT INTO  indicators.TestTargetTable (
		Column2
		,Column1)
	SELECT  
		Column1						AS Column2
		,Column2					AS Column1
  FROM	indicators.TestSourceTable

We ran metadata and lineage ingestions. Metadata ingestion was successful, but lineage tab was empty.

To Reproduce

  1. Create stored procedure in MS SQL
  2. Run metadata ingestion
  3. Check stored procedure lineage

Expected behavior We expected to see lineage between TestSourceTable and TestTargetTable and stored procedure on edge.

Version:

  • Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-102-generic x86_64)
  • Python version: Python 3.10.13
  • OpenMetadata version: 1.4.7
  • OpenMetadata Ingestion package version: openmetadata-ingestion==1.4.7.

Additional context We tried to do:

  1. We erased lines of stored procedure definition like:
CREATE PROCEDURE indicators.TestProcedure

AS

by rewriting OpenMetadata/ingestion/src/metadata/ingestion/source/database/mssql /queries.py because we found out that commenting commands like "CREATE PROCEDURE", "BEGIN", "END", "TRUNCATE", "DECLARE", "AS" gives correct lineage using SQL LINEAGE tool (which OM use for parsing lineages as i got). Example stored procedure below in link. https://reata.github.io/sqllineage/ - sql lineage web demo 2) But the lineage was also empty inside stored procedure lineage tab 3) We have executed the "insert"-query standaloned from example stored procedure

	INSERT INTO  indicators.TestTargetTable (
		Column2
		,Column1)
	SELECT  
		Column1						AS Column2
		,Column2					AS Column1
  FROM	indicators.TestSourceTable

and lineage was successfully produced.

Our questions are: 1)"openmetadata-sqllineage 1.1.2.0" really used in OpenMetadata. How can we set it to be used in lineage parsing process? 2)What workarounds can you recommend us to get the full lineage of DWH which uses stored procedures for ETL? https://github.com/open-metadata/OpenMetadata/issues/16737#issuecomment-2304484190

RabochiyYan avatar Aug 28 '24 06:08 RabochiyYan