Infrastructure improvements

[kompotkot]

Currently, the infrastructure of moonstream servers is at a level that allows it to ensure reliable operation of the service.

Version 2. Current status

• There are 2 servers for Ethereum blockchain with backups in S3 bucket. If one of the servers fails, this allows it to quickly restart crawlers on another server in manual mode.
• There is 1 server for Polygon node in synchronization.
• Configured Instance Group and Disk Snapshots to recover database server in case of Instance shutdown.
• Alerts are configured with Pagerduty for each infrastructure object: database server, api backend server and each node with crawlers.
• A status page has been deployed to display the current state of the servers and most valuable information as: block number difference, etc (work only in incognito mode).

Version 3. Next steps

• Modigy Go server for pg.moonstream.to and each node requires update to be able to respond correctly not only in incognito mode, but also with attached authentication token (I think there is a problem in CORS middleware).
• Synchronize heimdall at Polygon node -> backup to S3.
• Synchronize bor at Polygon node -> backup to S3.
• Create Nnginx in front of each node, setup access to geth and bor only for internal IP range, bypass stream for listening ports, and http status server with tls. Call Polygon node carabidae.
• Deploy script in user data updating moonstream.internal A record with internal subroute IP.

Version 4. Challenges

• Create load balancer server for load balancing between nodes, as well as proxying access to geth and bor.
• Deploy crawlers at different machine.

[kompotkot]