Asset publishing malfunctioning

[Baddooo14]

Issue description

Since this Friday/Saturday issues have started to appear when publishing assets. First I received a lot of RPC errors, even when switching between multiple public ones. Then I created my own Alchemy RPC, which at least resolved the 1020 errors.

However, since then publishes are very often not being published. I receive a lot of Error occurred provisioning asset messages when running my publishing script. I've also tried other scripts to no avail. I didn't change anything to my node, it just stopped functioning well.

Every now and then a publish will work fine, and then all other functions such as resolving, querying etc. will also work. The issue seems to be related to publishing only.

I've scoured my node for errors, but have not found the cause of the issue. See section error logs below.

At the same time looking at telemetry data, you can see a clear drop off in publishes ever since the RPC issues appeared (mumbai being down completely stopping data collection). I'm not sure if these events are correlated, but it's worth exploring.

Expected behavior

When trying to publish an asset with correct information and in correct format, the asset should actually be published to the dkg.

Actual behavior

Publish will work once every 20 tries on average.

Steps to reproduce the problem

1. Run any script which loops over publishes.
2. See that most of them fail.
3. Not profit.

Specifications

• Node version: 6.0.0-beta.1.36
• Platform: Ubuntu Server 20.04
• Node wallet: 0xa705c6A78d99001383C39946225B0E22630A0BdF
• Node libp2p identity: QmTsJNu4DqFwTrmuTmrfFva6PWFA8MNbRNRVEjkRvahPcx

Contact details

• Email: [email protected]

Error logs

This is the error that most often appears when running the publishing script.

The error above seems to always be tied to these errors. However, this shouldn't affect my node as these are receiving node errors.

I've also seen the following error sporadically, but I can't timestamp it to any specific event:

ERROR: Error while handling store: FetchError: request to http://localhost:9999/blazegraph/sparql failed, reason: socket hang up - FetchError: request to http://localhost:9999/blazegraph/sparql failed, reason: socket hang up at ClientRequest. (/root/ot-node/node_modules/cross-fetch/node_modules/node-fetch/lib/index.js:1491:11) at ClientRequest.emit (node:events:527:28) at Socket.socketOnEnd (node:_http_client:478:9) at Socket.emit (node:events:539:35) at endReadableNT (node:internal/streams/readable:1345:12) at processTicksAndRejections (node:internal/process/task_queues:83:21)

Disclaimer

Please be aware that the issue reported on a public repository allows everyone to see your node logs, node details, and contact details. If you have any sensitive information, feel free to share it by sending an email to [email protected].