chaingraph icon indicating copy to clipboard operation
chaingraph copied to clipboard
verified publisher icon bitauth

BSV Support

Open bitjson opened this issue 4 years ago • 5 comments

Unless I missed some deeply breaking change made recently to BSV, Chaingraph should already support the BSV node software. So full support requires only that we:

  • build the docker image in this repo (using a deterministically reproducible build of the BSV bitcoind application)
  • add configuration options to the Helm chart, i.e. --set bsv.enabled=true.

bitjson avatar Nov 24 '21 03:11 bitjson

Hi Jason

I'd love to contribute to make BSV work. I've made an attempt at https://github.com/ledgitbe/chaingraph/tree/bitcoin-sv however this is still very very rough. I'm unsure how to properly integrate support for a new node, but here's a start anyway and would appreciate your feedback and thoughts. I tried to keep my changes additive without changing too much in-place to prove the concept but there are still a lot of open questions on how to properly integrate a different node implementation.

Chart templates

  • Added templates for Bitcoin SV and Bitcoin SV Testnet
  • I based it off the bitcoin-cash-node files and did a lot of replacing to turn it into a bitcoin-sv-node
  • This is probably the most copyable part of my branch
  • values.yaml has been updated to support Bitcoin SV nodes

Images

  • I've added a bitcoin-sv-node image which can be built via yarn image:build:bsvn
  • The source files are straight from upstream, with 1 minor change: include APPEND_TO_BITCOIN_CONF in docker-entrypoint.sh

Package.json scripts

  • I tried to make sensible alternative scripts for BSV
  • I did change existing scripts (such as cluster init) to make it work regardless of which containers were up (separating the commands with ; instead of &&)
  • Again , I'm unsure if this is the right approach to support additional nodes

What does not work

Bitcoin SV Node container issues

There is an issue with the /data directory of the bitcoin-sv-node - when using local developer volumes (cluster init), a directory called bitcoin-sv-node appears in the data directory. However, this directory remains empty. I think it has something to do with permissions of the /data directory inside the container. Its owner remains root:root instead of bitcoin:root as in your bitcoin-cash-node images. When SSHing inside the container (yarn dev-cluster:ssh:bsvn) I also notice that /home/bitcoin/.bitcoin is empty. I tried umount /data and bitcoin-cli getblockchaininfo and then sometimes the files start appearing in /home/bitcoin/.bitcoin but not inside /data (which is correctly symlinked)

After giving up on this, I proceeded to just sync from a local (external from cluster) Bitcoin SV node. See P2P library for how this went.

P2P library

I had to use a different P2P library to sync with an external Bitcoin SV node because @chaingraph/bitcore-p2p-cash does not support some P2P messages that are used by Bitcoin SV (such as protoconf and more recently authch) What I did to make this work:

  • I downloaded the @chaingraph/bitcore-p2p-cash package from npmjs and committed it into a new git repo
  • I augmented the library to support Bitcoin SV
  • See the results here: https://github.com/ledgitbe/bitcore-p2p-bsv/commit/f9dc52c3abf285ede12674b949acae81cd706760

After this I proceeded to make a separate version of the agent so I could use the augmented version of bitcore-p2p to sync with a Bitcoin SV node. To keep my changes mostly additive in nature, I created a index-bsv and agent-bsv

P2P Library considerations

I realise it is not optimal having 2 separate P2P libraries integrated in this way - I see a few different scenarios going forward

  1. Include BSV support inside @chaingraph/bitcore-p2p-cash (easy)
  • It would be nice if there was a separate repository that we could make these changes in. As of now I only found the npm package but not the git repo (which I noticed includes additional changes you made (eg SendHeaders ))
  1. Import both libraries and then switch between them based on the bitcoin network
  • Very straightforward but not sure where you would place this logic
  1. Create an abstraction layer around different P2P libraries
  • This could allow the usage of any P2P library as long as it's wrapped correctly
  1. Create a unified P2P library with support for all networks and including streaming blocks
  • I'd like to bring your attention to https://github.com/kevinejohn/bsv-p2p
  • I tried to integrate this library initially but the interfaces were too different it started getting ugly very quickly
  • This library has support for
    • streaming blocks (I noticed a TODO for this in the code)
    • merkle proof generation

Looking forward to your insights!

ledgitbe avatar Mar 20 '23 19:03 ledgitbe

Hi @ledgitbe – wow, it looks like you made it a long way on integration!

My changes to bitcore-p2p-cash are published under @chaingraph/bitcore-p2p-cash from this repo currently: https://github.com/bitjson/bitcore. I haven't made too much effort to polish up that project because I'd love to eventually throw it away and replace it with a zero-dependency or Libauth-only implementation. If you're interested in working on such a unified P2P library, I'd certainly consider switching! I'd also be happy to accept a PR adding BSV-specific messages to the existing @chaingraph/bitcore-p2p-cash until a better P2P library is available. To get things working, you could also just publish your own @ledgitbe/chaingraph-p2p too.

I won't personally have much time to focus on this in the next few months, but I'd consider BSV support to be good way to stress-test the rest of Chaingraph too; I'd love to see you get it working!

bitjson avatar Mar 27 '23 21:03 bitjson

Thanks for having a look. I can start with submitting a PR to the P2P library (https://github.com/bitjson/bitcore) so at least there is that. Then I can get rid of the duplicate agent-bsv.ts and index-bsv.ts files in chaingraph. I'd also be happy to submit a PR for the changes in chaingraph itself to support BSV. That way it will be easier to communicate about any changes. Also as an aside: Have you considered syncing via the JSONRPC API? What were your conclusions when comparing P2P and JSONRPC? I have a very fast JSONRPC client library that supports streaming blocks, it could also potentially be used for syncing in combination with either P2P or ZMQ for realtime events.

ledgitbe avatar Mar 28 '23 11:03 ledgitbe

Hi @bitjson,

We went a bit further with the experiment that @ledgitbe talks about in the previous comment, and started to index from the BSV node.

I'm mainly working on it at the moment, and we are noticing that the DB is the main bottleneck for the indexation. We're currently considering solutions to workaround this challenge.

I was wondering if you are able to share a little bit your experience indexing BC, how's your DB deployment? how long did it take to perform the indexation process?

We'll appreciate any help!

raymundo-mintblue avatar Sep 26 '23 09:09 raymundo-mintblue

@raymundo-mintblue I summarized my performance testing here: https://github.com/bitauth/chaingraph/blob/master/docs/architecture.md#performance--scalability

Related, this migration might really improve ingest speed and aggregation performance for chains as large as BSV: https://github.com/bitauth/chaingraph/issues/29

bitjson avatar Dec 08 '23 18:12 bitjson