dash-sample-apps icon indicating copy to clipboard operation
dash-sample-apps copied to clipboard
verified publisher icon plotly

Salesforce app down due to test account password expires

Open ycaokris opened this issue 6 years ago • 3 comments

App name

dash-salesforce-crm

Description of bug

Related with https://github.com/plotly/dash-salesforce-crm/issues/7. This app on test server and production is down this morning because

  • we are using self-created developer trial account for this app.
  • The password and token of this account expires every other month without notification.

I've reset password and received a new token from the same account. But still got login error from script:

simple_salesforce.exceptions.SalesforceAuthenticationFailed: INVALID_LOGIN: Invalid username, password, security token; or user locked out.

Trying fresh signup with my own account and still got this error.

Downtime log:

2019-08-30T15:10:01.086378604Z app[web.1]: 172.17.0.19 - - [30/Aug/2019:15:10:01 +0000] "POST /dash-salesforce-crm/_dash-update-component HTTP/1.1" 200 1594 "https://dash-playground.plotly.host/dash-salesforce-crm/opportunities" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36"
2019-08-30T15:10:01.108452259Z app[web.1]: 172.17.0.19 - - [30/Aug/2019:15:10:01 +0000] "POST /dash-salesforce-crm/_dash-update-component HTTP/1.1" 200 40 "https://dash-playground.plotly.host/dash-salesforce-crm/opportunities" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36"
2019-08-30T15:10:36.013107515Z app[web.1]: [2019-08-30 15:10:33 +0000] [12] [CRITICAL] WORKER TIMEOUT (pid:222)
2019-08-30T15:10:39.667857277Z app[web.1]: [2019-08-30 15:10:39 +0000] [232] [INFO] Booting worker with pid: 232
2019-08-30T15:11:09.719844808Z app[web.1]: [2019-08-30 15:11:09 +0000] [12] [CRITICAL] WORKER TIMEOUT (pid:232)
2019-08-30T15:11:09.721251037Z app[web.1]: [2019-08-30 15:11:09 +0000] [232] [INFO] Worker exiting (pid: 232)
2019-08-30T15:11:10.736525104Z app[web.1]: [2019-08-30 15:11:10 +0000] [242] [INFO] Booting worker with pid: 242
2019-08-30T15:11:14.874747276Z app[web.1]: [2019-08-30 15:11:14 +0000] [242] [ERROR] Exception in worker process
2019-08-30T15:11:14.874783060Z app[web.1]: Traceback (most recent call last):
2019-08-30T15:11:14.874788194Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
2019-08-30T15:11:14.874792473Z app[web.1]:     worker.init_process()
2019-08-30T15:11:14.874796542Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
2019-08-30T15:11:14.874802507Z app[web.1]:     self.load_wsgi()
2019-08-30T15:11:14.874807854Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2019-08-30T15:11:14.874856963Z app[web.1]:     self.wsgi = self.app.wsgi()
2019-08-30T15:11:14.874862595Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
2019-08-30T15:11:14.874868270Z app[web.1]:     self.callable = self.load()
2019-08-30T15:11:14.874872888Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
2019-08-30T15:11:14.874878111Z app[web.1]:     return self.load_wsgiapp()
2019-08-30T15:11:14.874882955Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2019-08-30T15:11:14.874888348Z app[web.1]:     return util.import_app(self.app_uri)
2019-08-30T15:11:14.874893306Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
2019-08-30T15:11:14.874899170Z app[web.1]:     __import__(module)
2019-08-30T15:11:14.874904088Z app[web.1]:   File "/app/apps/dash-salesforce-crm/index.py", line 6, in <module>
2019-08-30T15:11:14.874910318Z app[web.1]:     from panels import opportunities, cases, leads
2019-08-30T15:11:14.874915443Z app[web.1]:   File "/app/apps/dash-salesforce-crm/panels/cases.py", line 12, in <module>
2019-08-30T15:11:14.874921198Z app[web.1]:     accounts = sf_manager.get_accounts()
2019-08-30T15:11:14.874926304Z app[web.1]:   File "/app/apps/dash-salesforce-crm/sfManager.py", line 100, in get_accounts
2019-08-30T15:11:14.874933392Z app[web.1]:     query_result = self.sf.query(query_text)
2019-08-30T15:11:14.874937457Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/simple_salesforce/api.py", line 375, in query
2019-08-30T15:11:14.874954956Z app[web.1]:     params=params, **kwargs)
2019-08-30T15:11:14.874958229Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/simple_salesforce/api.py", line 478, in _call_salesforce
2019-08-30T15:11:14.874961573Z app[web.1]:     exception_handler(result, name=name)
2019-08-30T15:11:14.874964549Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/simple_salesforce/util.py", line 61, in exception_handler
2019-08-30T15:11:14.874967704Z app[web.1]:     raise exc_cls(result.url, result.status_code, name, response_content)
2019-08-30T15:11:14.874973010Z app[web.1]: simple_salesforce.exceptions.SalesforceRefusedRequest: Request refused for https://na85.salesforce.com/services/data/v38.0/query/?q=SELECT+Id%2C+Name+FROM+Account. Response content: [{'message': 'The users password has expired, you must call SetPassword before attempting any other API operations', 'errorCode': 'INVALID_OPERATION_WITH_EXPIRED_PASSWORD'}]
2019-08-30T15:11:14.875966560Z app[web.1]: [2019-08-30 15:11:14 +0000] [242] [INFO] Worker exiting (pid: 242)
2019-08-30T15:11:15.790529469Z app[web.1]: [2019-08-30 15:11:15 +0000] [12] [INFO] Shutting down: Master
2019-08-30T15:11:15.790566315Z app[web.1]: [2019-08-30 15:11:15 +0000] [12] [INFO] Reason: Worker failed to boot.
2019-08-30T15:11:33.108164935Z app[web.1]: [2019-08-30 15:11:33 +0000] [11] [INFO] Starting gunicorn 19.9.0
2019-08-30T15:11:33.108848391Z app[web.1]: [2019-08-30 15:11:33 +0000] [11] [INFO] Listening at: http://0.0.0.0:5000 (11)
2019-08-30T15:11:33.108888315Z app[web.1]: [2019-08-30 15:11:33 +0000] [11] [INFO] Using worker: sync
2019-08-30T15:11:33.113443871Z app[web.1]: [2019-08-30 15:11:33 +0000] [213] [INFO] Booting worker with pid: 213
2019-08-30T15:11:35.313577602Z app[web.1]: [2019-08-30 15:11:35 +0000] [213] [ERROR] Exception in worker process
2019-08-30T15:11:35.313623146Z app[web.1]: Traceback (most recent call last):
2019-08-30T15:11:35.313628083Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
2019-08-30T15:11:35.313631955Z app[web.1]:     worker.init_process()
2019-08-30T15:11:35.313635719Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
2019-08-30T15:11:35.313639437Z app[web.1]:     self.load_wsgi()
2019-08-30T15:11:35.313642745Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2019-08-30T15:11:35.313646228Z app[web.1]:     self.wsgi = self.app.wsgi()
2019-08-30T15:11:35.313649659Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
2019-08-30T15:11:35.313653500Z app[web.1]:     self.callable = self.load()
2019-08-30T15:11:35.313656987Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
2019-08-30T15:11:35.313660455Z app[web.1]:     return self.load_wsgiapp()
2019-08-30T15:11:35.313663919Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2019-08-30T15:11:35.313667466Z app[web.1]:     return util.import_app(self.app_uri)
2019-08-30T15:11:35.313698908Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
2019-08-30T15:11:35.313703139Z app[web.1]:     __import__(module)
2019-08-30T15:11:35.313706247Z app[web.1]:   File "/app/apps/dash-salesforce-crm/index.py", line 6, in <module>
2019-08-30T15:11:35.313710072Z app[web.1]:     from panels import opportunities, cases, leads
2019-08-30T15:11:35.313713041Z app[web.1]:   File "/app/apps/dash-salesforce-crm/panels/cases.py", line 12, in <module>
2019-08-30T15:11:35.313716283Z app[web.1]:     accounts = sf_manager.get_accounts()
2019-08-30T15:11:35.313719290Z app[web.1]:   File "/app/apps/dash-salesforce-crm/sfManager.py", line 100, in get_accounts
2019-08-30T15:11:35.313722448Z app[web.1]:     query_result = self.sf.query(query_text)
2019-08-30T15:11:35.313725536Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/simple_salesforce/api.py", line 375, in query
2019-08-30T15:11:35.313728685Z app[web.1]:     params=params, **kwargs)
2019-08-30T15:11:35.313731634Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/simple_salesforce/api.py", line 478, in _call_salesforce
2019-08-30T15:11:35.313735166Z app[web.1]:     exception_handler(result, name=name)
2019-08-30T15:11:35.313738108Z app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/simple_salesforce/util.py", line 61, in exception_handler
2019-08-30T15:11:35.313741256Z app[web.1]:     raise exc_cls(result.url, result.status_code, name, response_content)
2019-08-30T15:11:35.313745940Z app[web.1]: simple_salesforce.exceptions.SalesforceRefusedRequest: Request refused for https://na85.salesforce.com/services/data/v38.0/query/?q=SELECT+Id%2C+Name+FROM+Account. Response content: [{'message': 'The users password has expired, you must call SetPassword before attempting any other API operations', 'errorCode': 'INVALID_OPERATION_WITH_EXPIRED_PASSWORD'}]

cc @nicolaskruchten

ycaokris avatar Aug 30 '19 16:08 ycaokris

Updates: I re-deployed on playground and gallery using my own account and both of them worked, https://dash-gallery.plotly.host/dash-salesforce-crm/

The original token from previous account is invalid. To avoid further downtime We should consider either

  • remove API dependency completely or
  • auto-create account and update password/token if there's a way

ycaokris avatar Aug 30 '19 18:08 ycaokris

@ycaokris is this still an ongoing issue that needs fixing?

lundstrj avatar Jan 30 '20 15:01 lundstrj

Already fixed.

This is a recurring issue that this app still have a chance to be brought down at an interval of every other month, because the password and token from the developer account it uses become invalid, and temporary solution is that we have to manually receive a new pair of password/token from email, update the credentials(environment variable in the app setting), and restart this app. I've logged credentials in the 1Password.

I'm not sure about the exact downtime interval how often it happens(I logged previous reset time together with tokens). To avoid possible downtime in the future, it could be better to set up a maintenance schedule(like, refresh creds every 2 month), or if we have a way to alert from app failure at server side.

ycaokris avatar Jan 30 '20 15:01 ycaokris