bdit_data-sources icon indicating copy to clipboard operation
bdit_data-sources copied to clipboard

Wys api mobile fixes: check for dupes in mobile sign installation table

Open KatiRG opened this issue 4 years ago • 2 comments

What this pull request accomplishes:

  • checks for duplicate rows in table wys.mobile_sign_installations_dupes after data has been read in by task read_google_sheets
  • if the table is not empty, the dag sends a Slack alert message

Issue(s) this solves:

  • #367

What, in particular, needs to reviewed:

  • I have currently put the modifications in a temporary dag called pull_wys_new.py. Once the modifications are approved, the old pull_wys.py will be deleted and pull_wys_new.py will be renamed pull_wys.py, and the symbolic link in /etc/airflow/dags/ to pull_wys_new.py will be deleted
  • ~~I am wondering about the task_id in the slack message. It is currently set at context.get('task_instance').task_id == 'check_wys_mobile_dupes_id', but should it be t2b, i.e. the name of the task?~~ I tested it (on SELECT COUNT(1) != 0 because the table is currently empty) and it works (i.e. Slack message is sent)
  • You can verify that the tasks ran in the correct order (read_google_sheets -> check_wys_mobile_dupes_id) by looking at the tree view in airflow for the pull_wys_new dag

KatiRG avatar May 31 '21 18:05 KatiRG

CodeFactor is spitting an error for "No exception type(s) specified". If strptime spits out a specific exception that should be caught and a separate generic one could also be logged.

https://github.com/CityofToronto/bdit_data-sources/blob/b082620bb7c298039101ef2c6cc8009b5afb03f1/wys/api/python/wys_google_sheet.py#L68-L75

radumas avatar Jan 10 '22 22:01 radumas

A new PR (#443) to resolve the issue of failing to pull data older than 2 days (#424). As pulling older data requires longer time (full days take more than 29 seconds), wys_api-py was modified to pull 1-hour data chunks instead of full days

tahaislam avatar Feb 02 '22 16:02 tahaislam

This branch was merged into #536 and then into master

tahaislam avatar Apr 03 '23 15:04 tahaislam