migrate icon indicating copy to clipboard operation
migrate copied to clipboard

Fix Views dependant on other views migration

Open amralieg opened this issue 4 years ago • 1 comments

We have done most of the migration and it was quite smooth but the migration of the Metastore is giving us trouble. Though most of the tables got migrated properly, the views are not when the view has dependency on another view. You can find below an example error: Importing view XYZdatalakeproduction.last_moe_XYZ_assignment_submissions_current_academic_year_view - Not found ERROR:

<span class="ansi-red-fg">AnalysisException</span>: View `XYZdatalakeproduction`.`last_XYZ_assignments_submissions_view` already exists. If you want to update the view definition, please use ALTER VIEW AS or CREATE OR REPLACE VIEW AS
{'resultType': 'error', 'summary': '<span class="ansi-red-fg">AnalysisException</span>: View `XYZdatalakeproduction`.`last_XYZ_assignments_submissions_view` already exists. If you want to update the view definition, please use ALTER VIEW AS or CREATE OR REPLACE VIEW AS', 'cause': '---------------------------------------------------------------------------\nAnalysisException                         Traceback (most recent call last)\n<command--1> in <module>\n----> 1 with open("/dbfs/tmp/migration/tmp_import_ddl.txt", "r") as fp: tmp_ddl = fp.read(); spark.sql(tmp_ddl)\n\n/databricks/spark/python/pyspark/sql/session.py in sql(self, sqlQuery)\n    775         [Row(f1=1, f2=\'row1\'), Row(f1=2, f2=\'row2\'), Row(f1=3, f2=\'row3\')]\n    776         """\n--> 777         return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)\n    778 \n    779     def table(self, tableName):\n\n/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)\n   1302 \n   1303         answer = self.gateway_client.send_command(command)\n-> 1304         return_value = get_return_value(\n   1305             answer, self.gateway_client, self.target_id, self.name)\n   1306 \n\n/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)\n    121                 # Hide where the exception came from that shows a non-Pythonic\n    122                 # JVM exception message.\n--> 123                 raise converted from None\n    124             else:\n    125                 raise\n\nAnalysisException: View `XYZdatalakeproduction`.`last_XYZ_assignments_submissions_view` already exists. If you want to update the view definition, please use ALTER VIEW AS or CREATE OR REPLACE VIEW AS'}

amralieg avatar Oct 21 '21 17:10 amralieg

Thanks. As a temporary workaround we can re-run the import twice since it does a best effort to import tables / views.

mrchristine avatar Oct 21 '21 18:10 mrchristine