migrate
migrate copied to clipboard
Fix Views dependant on other views migration
We have done most of the migration and it was quite smooth but the migration of the Metastore is giving us trouble. Though most of the tables got migrated properly, the views are not when the view has dependency on another view. You can find below an example error: Importing view XYZdatalakeproduction.last_moe_XYZ_assignment_submissions_current_academic_year_view - Not found ERROR:
<span class="ansi-red-fg">AnalysisException</span>: View `XYZdatalakeproduction`.`last_XYZ_assignments_submissions_view` already exists. If you want to update the view definition, please use ALTER VIEW AS or CREATE OR REPLACE VIEW AS
{'resultType': 'error', 'summary': '<span class="ansi-red-fg">AnalysisException</span>: View `XYZdatalakeproduction`.`last_XYZ_assignments_submissions_view` already exists. If you want to update the view definition, please use ALTER VIEW AS or CREATE OR REPLACE VIEW AS', 'cause': '---------------------------------------------------------------------------\nAnalysisException Traceback (most recent call last)\n<command--1> in <module>\n----> 1 with open("/dbfs/tmp/migration/tmp_import_ddl.txt", "r") as fp: tmp_ddl = fp.read(); spark.sql(tmp_ddl)\n\n/databricks/spark/python/pyspark/sql/session.py in sql(self, sqlQuery)\n 775 [Row(f1=1, f2=\'row1\'), Row(f1=2, f2=\'row2\'), Row(f1=3, f2=\'row3\')]\n 776 """\n--> 777 return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)\n 778 \n 779 def table(self, tableName):\n\n/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)\n 1302 \n 1303 answer = self.gateway_client.send_command(command)\n-> 1304 return_value = get_return_value(\n 1305 answer, self.gateway_client, self.target_id, self.name)\n 1306 \n\n/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)\n 121 # Hide where the exception came from that shows a non-Pythonic\n 122 # JVM exception message.\n--> 123 raise converted from None\n 124 else:\n 125 raise\n\nAnalysisException: View `XYZdatalakeproduction`.`last_XYZ_assignments_submissions_view` already exists. If you want to update the view definition, please use ALTER VIEW AS or CREATE OR REPLACE VIEW AS'}
Thanks. As a temporary workaround we can re-run the import twice since it does a best effort to import tables / views.