Don't fail the migration pipeline for minor issues

[petrcv-db]

When exporting notebooks I got the following error:

{"error_code": "RESOURCE_DOES_NOT_EXIST", "message": "Path (/Development/Buy-Alongs_dev/sets_poc/siamese/0_make_features (1)) doesn't exist.", "http_status_code": 404, "path": "/Development/Buy-Alongs_dev/sets_poc/siamese/0_make_features (1)"}

Checking the source workspace there's no notebook with such name. It seems like the listing function listed a notebook that doesn't exist and the exporting function failed when trying to get the notebook.

This error caused the whole export pipeline to fail:

RuntimeError: export_notebooks has failures. Refer to /data/pcv-pipeline8-export/pcv-pipeline8//app_logs/failed_export_notebooks.log to see failures. Terminating pipeline.

As a quick fix, I commented out the line that threw the exception:

diff --git a/pipeline/pipeline.py b/pipeline/pipeline.py
index 9b21759..a9fc8a7 100644
--- a/pipeline/pipeline.py
+++ b/pipeline/pipeline.py
@@ -77,7 +77,7 @@ class Pipeline:
             if os.path.exists(failed_task_log) and os.path.getsize(failed_task_log) > 0:
                 msg = f'{task.name} has failures. Refer to {failed_task_log} to see failures. Terminating pipeline.'
                 logging.info(msg)
-                raise RuntimeError(msg)
+                # raise RuntimeError(msg)
             self._completed_steps.write(f'{task.name}')
         else:
             logging.info(f'{task.name} Skipped.')

Not really hoping to get the 404 issue fixed, as this is hard to reproduce. But hoping to continue the pipeline if such error occurs, or to have an option to let the pipeline continue even though some notebook exports failed.

[mrchristine]

Are you certain the notebook doesn't exist and not that there's a hidden character or another issue here?

While we could skip this error, adding this type of fix could cause lots of notebooks to be missed during a migration. The tool is trying to prevent areas that could cause data loss.

My suggestion is for you to understand how you got to this error as I rarely see notebooks get deleted. Give us more concrete details as to why this scenario was hit and we can find a better long term solution.