Wys api mobile fixes: check for dupes in mobile sign installation table
What this pull request accomplishes:
- checks for duplicate rows in table
wys.mobile_sign_installations_dupesafter data has been read in by taskread_google_sheets - if the table is not empty, the dag sends a Slack alert message
Issue(s) this solves:
- #367
What, in particular, needs to reviewed:
- I have currently put the modifications in a temporary dag called
pull_wys_new.py. Once the modifications are approved, the oldpull_wys.pywill be deleted andpull_wys_new.pywill be renamedpull_wys.py, and the symbolic link in/etc/airflow/dags/topull_wys_new.pywill be deleted - ~~I am wondering about the
task_idin the slack message. It is currently set atcontext.get('task_instance').task_id == 'check_wys_mobile_dupes_id', but should it bet2b, i.e. the name of the task?~~ I tested it (onSELECT COUNT(1) != 0because the table is currently empty) and it works (i.e. Slack message is sent) - You can verify that the tasks ran in the correct order (
read_google_sheets->check_wys_mobile_dupes_id) by looking at the tree view in airflow for thepull_wys_newdag
CodeFactor is spitting an error for "No exception type(s) specified". If strptime spits out a specific exception that should be caught and a separate generic one could also be logged.
https://github.com/CityofToronto/bdit_data-sources/blob/b082620bb7c298039101ef2c6cc8009b5afb03f1/wys/api/python/wys_google_sheet.py#L68-L75
A new PR (#443) to resolve the issue of failing to pull data older than 2 days (#424). As pulling older data requires longer time (full days take more than 29 seconds), wys_api-py was modified to pull 1-hour data chunks instead of full days
This branch was merged into #536 and then into master