Spark agent lineage does not interact with multiple database services

[aimendenche-nw]

Hi team,

I tested the OMD Spark agent for lineage with a Spark job that interacts with multiple database services (Iceberg catalogs, BigQuery, etc.).

I followed the documentation and configured the Spark agent with:

spark.openmetadata.transport.databaseServiceNames = db1,db2

Observed behavior:

When I provide more than one database service name, the agent only creates lineage for the first service in the list. The remaining services are ignored.

Key finding When I completely remove the following configuration: spark.openmetadata.transport.databaseServiceNames and leave all other Spark and OMD configs unchanged, lineage works correctly for all databases involved in the job.

So could you confirm: Whether spark.openmetadata.transport.databaseServiceNames supports multiple values today? If this is a known limitation or a bug in the Spark agent? Thanks