node icon indicating copy to clipboard operation
node copied to clipboard
verified publisher icon DataHighway-DHX

feat: MMD-1309 / Update from Substrate 3 to commit dated 10th Feb 2022 (common branch 'polkadot-v0.9.17' for use with substrate-parachain-template)

Open ltfschoen opened this issue 4 years ago • 0 comments

A client is trying update new blocks on their local server of DataHighway mainnet at http://xxx:8585/blocks/head https://datahighway-dhx.github.io/substrate-api-sidecar/, but they are reporting the dhx.api HTTP status code is not 200-299 so they are experiencing delay and timeout like in this similar issue https://github.com/polkadot-js/api/issues/3365 every few mins, so as a result users are not able to topup or withdraw in time. The said their node has both http and ws connections. This PR is an attempt to update to the latest version of Substrate (since it was last updated to an old Substrate 3 commit dated 10th Feb 2021) incase that helps resolve the issue. Then we'll do a forkless runtime upgrade on Harbour testnet and then DataHighway Westlake mainnet

Note: To debug this branch run the following:

git fetch origin luke/MMD-1309/update-substrate-3:luke/MMD-1309/update-substrate-3
git checkout luke/MMD-1309/update-substrate-3
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
./scripts/init.sh
cargo build --release
rm -rf /tmp/polkadot-chains
./target/release/datahighway \                                                            
  --base-path /tmp/polkadot-chains/alice \
  --name "Data Highway Development Chain" \
  --dev \
  --telemetry-url "wss://telemetry.polkadot.io/submit/ 0" \
  -lruntime=debug

Checklist

  • [X] - question asked in Substrate Builders Program chat here https://matrix.to/#/!XOxcnpiHXHhAjJfRFP:matrix.parity.io/$NvyB8HmGBX_17VAOF0QPFoRx4ToXEYpF_jbbhxryUgY?via=matrix.parity.io&via=matrix.org&via=web3.foundation. Resolve why i can't update from Substrate 3 commit commit deac6324a16fc4128b94a7b4c3826eebcb86917f (dated 9th July 2021) to the next Substrate 4-dev commit efd54879ecbf8aac41fe809d31c7215ea3101241 (dated 11th Feb 2021) by just changing the dependencies in the Cargo.toml files like <DEPENDENCY_NAME> = { git = 'https://github.com/paritytech/substrate', rev = 'efd54879ecbf8aac41fe809d31c7215ea3101241', default-features = false } but when i try to build with cargo build --release, it gives the following error:
error: no matching package found
searched package name: `frame-benchmarking`
prerelease package needs to be specified explicitly
frame-benchmarking = { version = "4.0.0-dev" }
location searched: https://github.com/paritytech/substrate?rev=efd54879ecbf8aac41fe809d31c7215ea3101241#efd54879
required by package `datahighway v3.0.5 (/Users/me/code/DataHighway-DHX/node/node)`

and if i replace the rev value with any more recent Substrate commit and try building it gives the same error. if i specify the dependencies the way it wants me to like <DEPENDENCY_NAME> = { version = '4.0.0-dev' } as shown here https://github.com/paritytech/substrate/commit/efd54879ecbf8aac41fe809d31c7215ea3101241#diff-8b507853b325d5137f0252305eeef561aaddec506b75027217df634a243d8875R23, then i can't target specific commits between Substrate commit efd54879ecbf8aac41fe809d31c7215ea3101241 (dated 11th July 2021) and the latest Substrate commit 3478e0f6b80bd43803d7b821d887527d86104a41 (dated 29th July 2021). how can i specify the Git commit number that i want it to build so I can target these most recent Substrate commits since 9th July 2021?

i tried updating all the dependencies to pre-release versions 4.0.0-dev, 5.0.0-dev, and 0.10.0-dev in this commit https://github.com/DataHighway-DHX/node/commit/23c6941587815b46eeddf178f40dc5587771049d, but when i run cargo build --release i get error:

    Updating crates.io index
error: failed to select a version for the requirement `frame-benchmarking = "^4.0.0-dev"`
candidate versions found which didn't match: 3.1.0, 3.0.0, 2.0.1, ...
location searched: crates.io index
required by package `datahighway v3.0.5 (/Users/ls2/code/DataHighway-DHX/node/node)`

how do i locally test the prerelease versions (i.e. 4.0.0-dev, 5.0.0-dev, and 0.10.0-dev) of the dependencies? is it possible to publish this pre-release at crates.io? otherwise it doesn't seem possible to update to to a more recent version of Substrate than its commit deac6324a16fc4128b94a7b4c3826eebcb86917f (dated 9th July 2021) See latest commit with the changes in branch 'luke/MMD-1309/update-substrate-3-4.0.0-dev' Note: This is resolved, I think it just needed a cargo clean. If encounter similar problem consider https://doc.rust-lang.org/cargo/reference/overriding-dependencies.html?highlight=patch.#the-patch-section

  • [X] - finish updating to latest Substrate commit dated 29th July 2021
  • [X] - increment from 3.0.5 to 3.0.6, and increment spec_version and impl_version for an on-chain runtime upgrade (since Per convention: if the runtime behavior changes, increment spec_version and set impl_version to 0 (according to the docs)
  • [X] - check that unit and integration tests still pass [N/A] - try adding finality-grandpa-warp-sync support to see if that fixes the delay/timeout (extra protocol layer of warp sync support sc-finality-grandpa-warp-sync) https://matrix.to/#/!XOxcnpiHXHhAjJfRFP:matrix.parity.io/$cowEgy-s4n7OORfgm6DosPWVZF4hlUzLBfL4oAqEBLY?via=matrix.parity.io&via=matrix.org&via=web3.foundation. Note that it looks like in latest Substrate commits they are incorporating Grandpa Warp Sync for us (up to Warp Sync Part 1 in this commit https://github.com/paritytech/substrate/commit/c093de5d305cfa9871d03a2e651edfee5c28b5e3) **Waiting for Substrate commits to finish incorporating it). This isn't applicable, since we have found that the delay is caused by something wrong with DataHighway that is deviating from Substrate Node Template, which doesn't experience the delay. See https://github.com/DataHighway-DHX/open-polkadot-js-library/tree/main/block-delay#update-datahighway
  • [X] - Switch to match latest Substrate Node Template commit '4d28ebeb8b027ca0227fe7779c5beb70a7b56467'
  • [X] - generate chain spec and "raw" chain definition based on final changes in this PR for brickable, harbour, and westlake
./target/release/datahighway build-spec \
  --chain=<CHAIN> > ./res/chain_spec_<CHAIN>.json &&
./target/release/datahighway build-spec \
  --chain ./res/chain_spec_<CHAIN>.json \
  --raw > ./res/chain_def_<CHAIN>.json

./target/release/datahighway build-spec \
  --chain=brickable > ./res/chain_spec_brickable.json &&
./target/release/datahighway build-spec \
  --chain ./res/chain_spec_brickable.json \
  --raw > ./res/chain_def_brickable.json

./target/release/datahighway build-spec \
  --chain=harbour > ./res/chain_spec_harbour.json &&
./target/release/datahighway build-spec \
  --chain ./res/chain_spec_harbour.json \
  --raw > ./res/chain_def_harbour.json

./target/release/datahighway build-spec \
  --chain=westlake > ./res/chain_spec_westlake.json &&
./target/release/datahighway build-spec \
  --chain ./res/chain_spec_westlake.json \
  --raw > ./res/chain_def_westlake.json
  • [X] - unfortunately i've found that there are major issues in this PR and i'm not sure what's caused them https://github.com/DataHighway-DHX/node/pull/215. the issues are mentioned in the "Outstanding issues" section here https://github.com/DataHighway-DHX/node/pull/218 (which is a branch that adds further changes in addition to this PR) Update: These issues are resolved in that PR
  • [X] - consider using a fork of Substrate and manually using a more recent version of the crypto-mac Update: not necessary as crypto-mac 0.7 has been restored https://crates.io/crates/crypto-mac/versions
  • [X] - wait for response on how to resolve the missing required Substrate dependency crypto-mac 0.7.0 that we've asked about here https://github.com/paritytech/subport/issues/231 Update: see above update
  • [X] - merge PR #218 (refer to that commit description to see how to run 5x validator nodes on local brickable, add session keys, etc)
  • [X] - resolve error https://github.com/paritytech/subport/issues/232, we might overcome this issue if we merge https://github.com/DataHighway-DHX/node/pull/223, which updates to latest Substrate commit. Note: this appears to have somehow been resolved...
  • [X] - generate chain spec and "raw" chain definition again (repeat earlier step again after merged PR #215
  • [ ] - deploy these changes to DockerHub and run Brickable testnet on a remote server. Request client to check if timeout issue has been resolved now that we're using Aura instead of Babe with only one initial authority. Afterwards the built binary (built locally or on the server) should be copied and distributed to all validators and bootnodes so they have the latest non-runtime changes. Attach a copy to the Github releases as datahighway_runtime.compactv3.0.6.wasm (if latest version is 3.0.6)
    • Status
      • did not upgrade when running 'master' branch and forkless upgrade via drag-and-dropping from spec_version 8 impl_version 2 running, to:
        • spec_version 9 impl_version 0 running --dev
        • spec_version 9 impl_version 2 running --dev

the steps i taken were:

  • created a new branch called 'lukemmd1309' from branch 'luke/MMD-1309/update-substrate-3' so Docker would recognise the branch name
  • in docker-compose-dev.yml, i replaced all image: dhxdocker/datahighway:latest with image: dhxdocker/datahighway:lukemmd1309
  • created .env file cp .env.sample .env
  • in .env file i changed BOOTNODE_ENDPOINT_TESTNET=3.67.117.245 to BOOTNODE_ENDPOINT_TESTNET=127.0.0.1
  • remove old version of docker on server and install new
sudo apt-get remove docker-compose

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

sudo chmod +x /usr/local/bin/docker-compose
the server had docker-compose version 1.24.1, but i needed 1.29.0
  • i setup rust by running ./scripts/init.sh,
rustup update stable
rustup update nightly
rustup toolchain install nightly-2021-12-01
rustup target add wasm32-unknown-unknown --toolchain nightly-2021-12-01
rustup default nightly-2021-12-01
rustup override set nightly-2021-12-01
  • i tried to build the local Docker image by running
sudo chmod -R 777 /home/luke/node/
docker-compose --env-file=./.env --file docker-compose-dev.yml --verbose build --build-arg CHAIN_VERSION="brickable"
  • deploy to DockerHub https://hub.docker.com/repository/docker/dhxdocker/datahighway
docker login
docker tag dhxdocker/datahighway:lukemmd1309 lukemmd1309
docker push dhxdocker/datahighway:lukemmd1309
  • run the development nodes and view logs
docker-compose -f docker-compose-dev.yml up --detach
docker-compose -f docker-compose-dev.yml logs --follow
  • go to https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fbrickable.datahighway.com#/explorer, import dev accounts like Alice, Bob, etc that are endowed with testnet tokens to test with subkey inspect --network datahighway --scheme sr25519 "//Alice"
  • copy WASM file from server to local machine
scp -J luke@<IP_ADDR_1> luke@<IP_ADDR_2>:/home/luke/node/target/release/wbuild/datahighway-runtime/datahighway_runtime.compact.wasm .
  • generate chain spec and definition.
sudo ./target/release/datahighway build-spec \
  --chain=brickable > /home/luke/node/res/chain_spec_brickable.json &&
sudo ./target/release/datahighway build-spec \
  --chain /home/luke/node/res/chain_spec_brickable.json \
  --raw > /home/luke/node/res/chain_def_brickable.json
  • copy chain spec and definition from server to local machine to share with client to connect to Brickable with a node
scp -J luke@<IP_ADDR_1> luke@<IP_ADDR_2>:/home/luke/node/res/chain_spec_brickable.json .
scp -J luke@<IP_ADDR_1> luke@<IP_ADDR_2>:/home/luke/node/res/chain_def_brickable.json .
  • note: if we do all the above in future with docker-compose-custom.yml instead we can run a Local network without dev accounts like Alice, Bob, then we'd run cURL commands to inject Grandpa and Aura keys, and finally restart each container so it finalizes blocks with docker restart <DOCKER_CONTAINER_ID>

STATUS: Requested client to check if they experience the timeout issues with the updates we've made in this PR and deployed to Brickable

IMPORTANT NOTE: Feedback from Substrate Builders was that the timeout that we're experiencing with Babe might get smaller as more and more validators are added. So in future when we have more validators or we simulate that this is true then we might choose to switch back to using Babe instead of Aura.

  • [ ] - Release new version of github.com/DataHighway-DHX/node, after client checked if they still encounter the same delays/timeouts with API requests to /block/head

ltfschoen avatar Aug 01 '21 04:08 ltfschoen