Retry queue for failed decrypts

[alexcos20]

Given the following log:

2022-05-10 06:52:48,923 - aquarius.events.events_monitor - INFO - Finished searching for MetadataCreated events on chain 4 in blocks 10650574 to 10650575. 1 MetadataCreated events detected.
2022-05-10 06:52:49,061 - aquarius.events.decryptor - ERROR - Failed to get a response for decrypt DDO with provider=https://v4.provider.rinkeby.oceanprotocol.com, payload={'transactionId': '0xbcff08e7e8fd6842b6019db5cb9a1ac6adcfc8253fb9398dac3601f270658e0b', 'chainId': 4, 'decrypterAddress': '0xF41681F0BE4A914d447d7A94C91906E74A6561D0', 'dataNftAddress': '0x4f406c603678869445110A81Dd215F1809BeC03b', 'signature': '0xd69b0c4b9a2789712cec2a513dc0e3e58ef40618473ea88b0040452b521ee1c7377bebe60014b00b834343502a1a8fbbd36c2a63a9d150ea8d1021b26515b71800', 'nonce': '1652165568'}, response is b'{"context":{"chainId":4,"dataNftAddress":"0x4f406c603678869445110A81Dd215F1809BeC03b","decrypterAddress":"0xF41681F0BE4A914d447d7A94C91906E74A6561D0","nonce":"1652165568","signature":"0xd69b0c4b9a2789712cec2a513dc0e3e58ef40618473ea88b0040452b521ee1c7377bebe60014b00b834343502a1a8fbbd36c2a63a9d150ea8d1021b26515b71800","transactionId":"0xbcff08e7e8fd6842b6019db5cb9a1ac6adcfc8253fb9398dac3601f270658e0b"},"error":"No option \'address.file\' in section: \'eth-network\'"}\n'
2022-05-10 06:52:49,062 - aquarius.events.events_monitor - ERROR - Error processing MetadataCreated event: Failed to get a response for decrypt DDO with provider=https://v4.provider.rinkeby.oceanprotocol.com, payload={'transactionId': '0xbcff08e7e8fd6842b6019db5cb9a1ac6adcfc8253fb9398dac3601f270658e0b', 'chainId': 4, 'decrypterAddress': '0xF41681F0BE4A914d447d7A94C91906E74A6561D0', 'dataNftAddress': '0x4f406c603678869445110A81Dd215F1809BeC03b', 'signature': '0xd69b0c4b9a2789712cec2a513dc0e3e58ef40618473ea88b0040452b521ee1c7377bebe60014b00b834343502a1a8fbbd36c2a63a9d150ea8d1021b26515b71800', 'nonce': '1652165568'}, response is b'{"context":{"chainId":4,"dataNftAddress":"0x4f406c603678869445110A81Dd215F1809BeC03b","decrypterAddress":"0xF41681F0BE4A914d447d7A94C91906E74A6561D0","nonce":"1652165568","signature":"0xd69b0c4b9a2789712cec2a513dc0e3e58ef40618473ea88b0040452b521ee1c7377bebe60014b00b834343502a1a8fbbd36c2a63a9d150ea8d1021b26515b71800","transactionId":"0xbcff08e7e8fd6842b6019db5cb9a1ac6adcfc8253fb9398dac3601f270658e0b"},"error":"No option \'address.file\' in section: \'eth-network\'"}\n'
event=AttributeDict({'address': '0x4f406c603678869445110A81Dd215F1809BeC03b', 'blockHash': HexBytes('0x908a668348774aae5a1b50504c398510750423a5cb41ec4f886988d779b69ec8'), 'blockNumber': 10650575, 'data': '0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e000000000000000000000000000000000000000000000000000000000000001400000000000000000000000000000000000000000000000000000000000000180830b3cf3d2d85f1966b15922931350b8a51b9c7850415a0c3af9c95ba4148fe700000000000000000000000000000000000000000000000000000000627a0bb90000000000000000000000000000000000000000000000000000000000a283cf000000000000000000000000000000000000000000000000000000000000002d68747470733a2f2f76342e70726f76696465722e72696e6b6562792e6f6365616e70726f746f636f6c2e636f6d0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000048e043b36e87d005e67fe51a0208cf85a42785dcc9fc575ba992082e5b748b8c1589b25386f54a0d38080f5a5a037ab5947aef02318bd085a4e8b970abcfcdf12c7504c28b25e93b587a9c13b655d8079fc4407cf361183d6e5c824d86644c83244921327a6add9aa6e592a383d57ff829b6a396f3ccb1bb5b103868a8d0f1d9ebbf1326587c614e08a603ad09e0d0c16b2e823e8b917eb45509e797f49b048b4a80548c7a10d47627ce2a1fccb8975cf201dbb51a2f4757eddfd1ed04d55363e4f7a280379ae23e4484732499b41f2c7c4fe898813d2884949ff41282e255ce38f01c20bbf94d612f4d4f9f8c0d750ade1a425fb8bd1842b874c418efce1b48c2ef61a930a7fa533612726f94a0e070b1b81bcf35768bba2768f8481112c10885c2e4e412ab1c9915777334c9de6b1e2c27d8724547dd58bbcdc39b9021260ba91672b2b91aa14e63c27b8b342df1b97c166cba4fe17bafdfa6c0af783a91a26ecf70d29498b0de485217d32e67cf748e428f2cfa7e6a992d0851b926adf4634d57171c28d191e9579760ecd966bb9cb67fb6f436cabd76295cdeecd2fb14f74859fff9d16157cdcd17c2c503957896ee27926eb914724db603b67ace3bf161dacbec25448a2b35f0782ee0bc964d9d644aa7ebf68d5a562769f929ed912742c7014e0ff3400b382a05bcf7d402300a8d20fa147583d537255b58f3c6060218f9e1f10f6d4776db4d704ef9ea32c5cd1c93e8b4b02783b1f729e32a49b8c4f41ca4ea978fa5b392745c0bb179dd902a21edb30c8e1be09edcc1d572b16dcf65fd415cd199d61d46dc08570dbf0a62e9f4d4e2e9266dc1aef57eac9767c78cc49c83e4cf217674fa404d92250a33a297cd6ac0ed95a37cc53e0ca6ebe62b511e689f880d57aa9fd0daa0e962424b2976bb3bb0f70eec34b6d6254b1adb6b28a8274ca10113ed5ab2c08b429c042c1848a75131f99375373b17b5580b4bf0bc2c2f7c94f776a8bb09ae640c7984bf7e6700f2a0adddae861586f8a890ca0cd7141ca6c7a2fa2298968369ef757a25fb40b0927241a7f114cdcfce58322185d32b3cafa752b206135c9a65e75b64fb132037ad7f2a3c71b8836f743557059cd9936dfac75d1812c53ab42f29346188c68247f2270811d4b3c1cd361a6413bf0dbad7c3d6a2e9fdd049657fd2777d217bc3e7d4f4090ae6c18fcb8632e26ec1c539ffe37e5b7a30a03bdc220a61e377476b4c9f5992903a77801a05ded46999e4a8fa50b659fa34336aa0203144f743521de336207329119c291930cb090d55c7f4454cfb7e76990e2048f3f3f85aa8751ea7b172a7336b0ed29510359fd9e31b1659666d0af9dd96a6a7291734a95601d0f0927553ee32f509b74fd2cbaf4940a254571ef857efe55dc1ba8c307364ee1123f2d66136e4d4101adefcd2be0c1622ca04d2a36ffbc8bd31c85d2ebbd48b003af6c33a328ffd2177157d539e161d9551960a989a4719a2641bcebc0573d81973782fded4a4791fe3efa3ab9d00ae09b5569034d41114ff29f0d7eb73fd85412140cbc2ca3eb4a6c0a7e416ce5588470fb878ef4453a28f2b1473a9795c5565c2a1ef4afcd04a935ba90e1326f359d85677bce82b3d1a21100efe3c17c714f000000000000000000000000000000000000', 'logIndex': 89, 'removed': False, 'topics': [HexBytes('0x5463569dcc320958360074a9ab27e809e8a6942c394fb151d139b5f7b4ecb1bd'), HexBytes('0x0000000000000000000000003e81aa994f774ee914d57946ddf2486ea7d42d65')], 'transactionHash': HexBytes('0xbcff08e7e8fd6842b6019db5cb9a1ac6adcfc8253fb9398dac3601f270658e0b'), 'transactionIndex': 50})
Traceback (most recent call last):
  File "/aquarius/aquarius/events/events_monitor.py", line 223, in handle_regular_event_processor
    event_processor.process()
  File "/aquarius/aquarius/events/processors.py", line 204, in process
    asset = decrypt_ddo(
  File "/aquarius/aquarius/events/decryptor.py", line 38, in decrypt_ddo
    raise Exception(msg)
Exception: Failed to get a response for decrypt DDO with provider=https://v4.provider.rinkeby.oceanprotocol.com, payload={'transactionId': '0xbcff08e7e8fd6842b6019db5cb9a1ac6adcfc8253fb9398dac3601f270658e0b', 'chainId': 4, 'decrypterAddress': '0xF41681F0BE4A914d447d7A94C91906E74A6561D0', 'dataNftAddress': '0x4f406c603678869445110A81Dd215F1809BeC03b', 'signature': '0xd69b0c4b9a2789712cec2a513dc0e3e58ef40618473ea88b0040452b521ee1c7377bebe60014b00b834343502a1a8fbbd36c2a63a9d150ea8d1021b26515b71800', 'nonce': '1652165568'}, response is b'{"context":{"chainId":4,"dataNftAddress":"0x4f406c603678869445110A81Dd215F1809BeC03b","decrypterAddress":"0xF41681F0BE4A914d447d7A94C91906E74A6561D0","nonce":"1652165568","signature":"0xd69b0c4b9a2789712cec2a513dc0e3e58ef40618473ea88b0040452b521ee1c7377bebe60014b00b834343502a1a8fbbd36c2a63a9d150ea8d1021b26515b71800","transactionId":"0xbcff08e7e8fd6842b6019db5cb9a1ac6adcfc8253fb9398dac3601f270658e0b"},"error":"No option \'address.file\' in section: \'eth-network\'"}\n'

what happens when this provider will be fixed ?
Nothing, that asset is lost

Proposal:

• when a decrypt fails (or there is any other reason which prevents aqua to index a ddo), we should have a retry queue. That queue will be retried from time to time, until the asset is indexed. retry timeout should be configurable

[calina-c]

@md00ux @mihaisc I created the retry-queue image in dockerhub to help test this.