migrate
migrate copied to clipboard
The import pipeline fails when there's a job with an outdated cron expression
When I run the migration pipeline on a workspace that contains a job with an outdated expression, that won't run anymore, the whole pipeline fails with an error:
2022-04-25,14:17:54;DEBUG;https://xxx-play-ground.cloud.databricks.com:443 "POST /api/2.0/jobs/create HTTP/1.1" 400 None
2022-04-25,14:17:54;WARNING;{"error_code": "INVALID_PARAMETER_VALUE", "message": "The provided cron expression: 0 1 0 8 Apr ? 2021 will never fire."}
2022-04-25,14:17:54;ERROR;{"error_code": "INVALID_PARAMETER_VALUE", "message": "The provided cron expression: 0 1 0 8 Apr ? 2021 will never fire.", "http_status_code": 400}
Traceback (most recent call last):
File "migration_pipeline.py", line 328, in <module>
main()
File "migration_pipeline.py", line 324, in main
pipeline.run()
File "/home/ubuntu/migrate/pipeline/pipeline.py", line 64, in run
future.result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/ubuntu/migrate/pipeline/pipeline.py", line 73, in _run_task
task.run()
File "/home/ubuntu/migrate/tasks/tasks.py", line 274, in run
jobs_c.import_job_configs()
File "/home/ubuntu/migrate/dbclient/JobsClient.py", line 182, in import_job_configs
raise RuntimeError("Import job has failed. Refer to the previous log messages to investigate.")
RuntimeError: Import job has failed. Refer to the previous log messages to investigate.
Not sure what the right behaviour is: just skipping the job and logging / importing the job without the cron/... However it IMO should fail the whole import pipeline.