migrate icon indicating copy to clipboard operation
migrate copied to clipboard

The import pipeline fails when there's a job with an outdated cron expression

Open petrcv-db opened this issue 3 years ago • 0 comments

When I run the migration pipeline on a workspace that contains a job with an outdated expression, that won't run anymore, the whole pipeline fails with an error:

2022-04-25,14:17:54;DEBUG;https://xxx-play-ground.cloud.databricks.com:443 "POST /api/2.0/jobs/create HTTP/1.1" 400 None
2022-04-25,14:17:54;WARNING;{"error_code": "INVALID_PARAMETER_VALUE", "message": "The provided cron expression: 0 1 0 8 Apr ? 2021 will never fire."}

2022-04-25,14:17:54;ERROR;{"error_code": "INVALID_PARAMETER_VALUE", "message": "The provided cron expression: 0 1 0 8 Apr ? 2021 will never fire.", "http_status_code": 400}
Traceback (most recent call last):
  File "migration_pipeline.py", line 328, in <module>
    main()
  File "migration_pipeline.py", line 324, in main
    pipeline.run()
  File "/home/ubuntu/migrate/pipeline/pipeline.py", line 64, in run
    future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/ubuntu/migrate/pipeline/pipeline.py", line 73, in _run_task
    task.run()
  File "/home/ubuntu/migrate/tasks/tasks.py", line 274, in run
    jobs_c.import_job_configs()
  File "/home/ubuntu/migrate/dbclient/JobsClient.py", line 182, in import_job_configs
    raise RuntimeError("Import job has failed. Refer to the previous log messages to investigate.")
RuntimeError: Import job has failed. Refer to the previous log messages to investigate.

Not sure what the right behaviour is: just skipping the job and logging / importing the job without the cron/... However it IMO should fail the whole import pipeline.

petrcv-db avatar Apr 27 '22 08:04 petrcv-db