keep-core icon indicating copy to clipboard operation
keep-core copied to clipboard
verified publisher icon keep-network

Rewards

Open dimpar opened this issue 3 years ago • 4 comments

Refs #3345

There are 5 requirements for a peer to receive rewards:

  • Random Beacon authorized
  • tBTC authorized
  • Node uptime 96% in a given rewards interval. Counted after a peer joins the network
  • Pre-generated params (avg daily of 500 for now).
  • Build version (yet to be implemented)

rewards-requirements.ts will :

  • call beacon and wallet registry contracts for authorization info
  • queries Prometheus service for bootstrap peers info
  • aggregate and fetch rewards requirements for each peer

Each requirement has assigned a binary factor: 0 - unsatisfied or 1 - satisfied. If any of the requirements is 0 it means no rewards for a peer in a given interval. Uptime requirement has an additional coefficient which denotes a portion of rewards in a given rewards interval. For example, if a peer joins mid-month, then only 50% of rewards will be distributed to this peer.

Example JSON output 
{
  "10.102.5.12:9601": {
    "address": "0x0954efEFeb970D317a51736201B4EB2De75FF5DE",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 0
  },
  "10.102.5.15:9601": {
    "address": "0x0954efEFeb970D317a51736201B4EB2De75FF5DE",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 1
  },
  "10.102.5.18:9601": {
    "address": "0x0954efEFeb970D317a51736201B4EB2De75FF5DE",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 1,
    "upTimeRewardsCoefficient": 0.1388888888888889,
    "preParams": 0
  },
  "bootstrap-0.test.keep.network:9601": {
    "address": "0x0eC14BC7cCA82c942Cf276F6BbD0413216dDB2bE",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 1
  },
  "10.102.5.16:9601": {
    "address": "0x1Aa7A9De6bd5A5802A98be50fF12F5A024a5aBE0",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 0
  },
  "10.102.5.17:9601": {
    "address": "0x1Aa7A9De6bd5A5802A98be50fF12F5A024a5aBE0",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 1,
    "upTimeRewardsCoefficient": 0.1388888888888889,
    "preParams": 0
  },
  "10.102.5.7:9601": {
    "address": "0x1Aa7A9De6bd5A5802A98be50fF12F5A024a5aBE0",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 0
  },
  "bootstrap-alpha.test.threshold.p2p.org:9601": {
    "address": "0x1F91c3e91745aB09cc8e619ceD8Fa79B602e3649",
    "isBeaconAuthorized": 0,
    "isTbtcAuthorized": 0,
    "upTime": 1,
    "upTimeRewardsCoefficient": 1,
    "preParams": 1
  },
  "3.137.162.218:9601": {
    "address": "0x37847B906612885d7172639947Cecf76036458bA",
    "isBeaconAuthorized": 0,
    "isTbtcAuthorized": 0,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 1
  },
  "10.102.4.11:9601": {
    "address": "0x3FF855895EF4aC833c32Ab6A0d6C7fBfA137E26E",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 1
  },
  "10.102.4.20:9601": {
    "address": "0x3FF855895EF4aC833c32Ab6A0d6C7fBfA137E26E",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 1,
    "upTimeRewardsCoefficient": 0.1388888888888889,
    "preParams": 0
  },
  "10.102.4.6:9601": {
    "address": "0x3FF855895EF4aC833c32Ab6A0d6C7fBfA137E26E",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 0
  },
  "bst-a01.test.keep.boar.network:9601": {
    "address": "0x5232E393602c498D216d680C4cAbb0Cd179d1d16",
    "isBeaconAuthorized": 0,
    "isTbtcAuthorized": 0,
    "upTime": 1,
    "upTimeRewardsCoefficient": 1,
    "preParams": 1
  },
  "10.102.4.13:9601": {
    "address": "0x5cD847903Bb7F29de77Eecc135628cA5b104A355",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 1
  },
  "10.102.4.21:9601": {
    "address": "0x5cD847903Bb7F29de77Eecc135628cA5b104A355",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 1,
    "upTimeRewardsCoefficient": 0.1388888888888889,
    "preParams": 0
  },
  "10.102.4.5:9601": {
    "address": "0x5cD847903Bb7F29de77Eecc135628cA5b104A355",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 0
  },
  "10.102.4.12:9601": {
    "address": "0x677753A3cb8F3575Be626f6A1F26e5C027C0aF29",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 0
  },
  "10.102.4.19:9601": {
    "address": "0x677753A3cb8F3575Be626f6A1F26e5C027C0aF29",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 1,
    "upTimeRewardsCoefficient": 0.1388888888888889,
    "preParams": 0
  },
  "10.102.4.8:9601": {
    "address": "0x677753A3cb8F3575Be626f6A1F26e5C027C0aF29",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 0
  },
  "10.102.2.21:9601": {
    "address": "0x76bc6bAD38728329FE1c0E57D2555726f26a0399",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 1
  },
  "10.102.2.22:9601": {
    "address": "0x76bc6bAD38728329FE1c0E57D2555726f26a0399",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 1,
    "upTimeRewardsCoefficient": 0.1388888888888889,
    "preParams": 0
  },
  "10.102.2.3:9601": {
    "address": "0x76bc6bAD38728329FE1c0E57D2555726f26a0399",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 0
  },
  "138.68.240.252:9601": {
    "address": "0xB8d1511aFab121f07cfDC7D1086371b1Fa758Fa2",
    "isBeaconAuthorized": 0,
    "isTbtcAuthorized": 0,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 0
  },
  "keep-validator-0.eks-ap-northeast-2-secure.staging.staked.cloud:9601": {
    "address": "0xCa8754f7060A0648824f274e3a4d897FA497139D",
    "isBeaconAuthorized": 0,
    "isTbtcAuthorized": 0,
    "upTime": 1,
    "upTimeRewardsCoefficient": 1,
    "preParams": 1
  },
  "10.102.2.19:9601": {
    "address": "0xD12A53056b74d96F89910aD3485da69A662F7930",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 1
  },
  "10.102.2.23:9601": {
    "address": "0xD12A53056b74d96F89910aD3485da69A662F7930",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 1,
    "upTimeRewardsCoefficient": 0.1388888888888889,
    "preParams": 0
  },
  "10.102.2.4:9601": {
    "address": "0xD12A53056b74d96F89910aD3485da69A662F7930",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 0
  },
  "10.102.2.20:9601": {
    "address": "0xaC049223397e2F25Ea9FE56D5ee0896F6d8E8CB7",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 1
  },
  "10.102.2.24:9601": {
    "address": "0xaC049223397e2F25Ea9FE56D5ee0896F6d8E8CB7",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 1,
    "upTimeRewardsCoefficient": 0.1388888888888889,
    "preParams": 0
  },
  "10.102.2.7:9601": {
    "address": "0xaC049223397e2F25Ea9FE56D5ee0896F6d8E8CB7",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 1
  },
  "bootstrap-1.test.keep.network:9601": {
    "address": "0xcAB2a402bAc470686d14956FB310D51BbEF9fA31",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 1
  }
}

If all of the requirements are satisfied, then the rewards can be calculated with the following formula:

Random Beacon authorized : 1
tBTC authorized : 1
Node up time 96% : 1
Pre-generated params : 1
Build version : 1

Peer rewards = (peer's authorized stake / total authorized stake) * rewardsForAGivenMonth * upTimeRewardsCoefficient

Example usage

./rewards.sh --rewards-start-date 2022-09-01 --rewards-end-date 2022-09-30

For more options run ./rewards.sh --help

TBD:

  • [ ] Output of this script. Is the rewards calculation approach correct? See 1) in https://github.com/keep-network/keep-core/pull/3346#issuecomment-1268505791
  • [ ] Do we want to upload the output to GCP? If so, the example script is here: https://github.com/keep-network/keep-core/pull/3290

TODO:

  • [ ] Validate the correctness of the build version requirements: Example:
- Rewards interval Sep 1 - Sep 30 
- Allowed delay 14days
- We take into account only 2 latest versions
- Checkpoints are always at the end of the interval
Scenario 1:
- v2 was released between Sep 16 - Sep 30.  At the checkpoint a peer can run v1 or v2, both are fine.
Scenario 2:
- v2 was released between Sep 1 - Sep 16. At the checkpoint a peer can run v2 only, because 14days have passed already
Scenario 3:
- no v2 was released in Sep, so we just check the latest version
  • [ ] Calculate rewardsWeight as described here 2) https://github.com/keep-network/keep-core/pull/3346#issuecomment-1268505791
  • [ ] Make sure NU team can process the output file for their rewards calculations.

dimpar avatar Oct 04 '22 14:10 dimpar

summarizing the broad approach to make sure I understand:

  1. we call rewards.sh from arguments somewhere (from a service, manually, etc). That parses arguments, and eventually...
  2. we call rewards-requirements.ts with the data. We parse all the arguments passed from rewards.sh, and interpolate prometheus queries.
  3. we run queryPrometheus(url: string, params: any) on those queries (uptime, pre-param count, build version)
  4. we fetch the other two (authorized random beacon, authorized tbtc) from querying the relevant contracts directly via ethers and having the ABIs
  5. we save all of those results into an object named peerData and return it

Seems like a direct, lightweight approach, and I don't see why we'd need to engineer it any more than this! I think so long as we make sure the file itself is well structured, it won't be a problem maintaining it going forward either 👍

We're not writing enough prometheus queries that it's worth any engineering effort to do anything but manually craft query strings. No prometheus abstraction rabbit holes here.

We're not importing enough ABIs that we need an abstraction layer on top of that either.

beaurancourt avatar Oct 05 '22 00:10 beaurancourt

I did not look at the code, leaving this part to @beaushinkle but I have two comments regarding the structure of the JSON output:

  1. The key is the IP address. I think this is wrong because the IP address may change (e.g. cluster relocation). I think we should use the operator address as a key instead. Alternatively, consider using staking provider address as a key. This needs to be agreed upon with the Nu team.
  2. We do have the upTimeRewardsCoefficient but this is not enough. Someone who has 1M T staked and only 100k T authorized for random beacon and tBTC should not receive rewards for 1M T. Instead of upTimeRewardsCoefficient I would consider exposing unsigned integer rewardWeight that would be calculated as rewardWeight = uptime_coefficient * AVG(beacon_authorization, ecdsa_auhtorization).

pdyraga avatar Oct 05 '22 14:10 pdyraga

The key is the IP address. I think this is wrong because the IP address may change (e.g. cluster relocation). I think we should use the operator address as a key instead. Alternatively, consider using staking provider address as a key. This needs to be agreed upon with the Nu team.

This can be easily changed and that was my original approach actually, but I noticed that we have cases where the same address is on two different IPs like here:

"10.102.4.11:9601": {
    "address": "0x3FF855895EF4aC833c32Ab6A0d6C7fBfA137E26E",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 0,
    "upTimeRewardsCoefficient": 0,
    "preParams": 1
  },
  "10.102.4.20:9601": {
    "address": "0x3FF855895EF4aC833c32Ab6A0d6C7fBfA137E26E",
    "isBeaconAuthorized": 1,
    "isTbtcAuthorized": 1,
    "upTime": 1,
    "upTimeRewardsCoefficient": 0.1388888888888889,
    "preParams": 0
  },
(...)

How do we deal with that? What if one peer goes down and the other stays up?

dimpar avatar Oct 05 '22 16:10 dimpar

How do we deal with that? What if one peer goes down and the other stays up?

It shouldn't be possible in the network to have two peers with the same peer ID. And peer ID is created from the operator address. The reason we see the same address under different IPs is most probably because of how Prometheus is aggregating data or because of how the query is constructed. It is not ideal, yet acceptable and quite common for a peer to change IP address several times a month. We need to find a way to aggregate the data properly.

pdyraga avatar Oct 11 '22 07:10 pdyraga

Here's a little audit on the prometheus queries:

1) present_over_time(up{chain_address="0x0954efEFeb970D317a51736201B4EB2De75FF5DE", job="keep-discovered-nodes"} [2591999s] offset 74124s)
2) up{chain_address="0x0954efEFeb970D317a51736201B4EB2De75FF5DE", job="keep-discovered-nodes"} [2591999s:120s] offset 74124s
3) sum_over_time(up{chain_address="0x0954efEFeb970D317a51736201B4EB2De75FF5DE", job="keep-discovered-nodes"} [364679s:120s] offset 74124s) * 120 / 364679
4) avg_over_time(tbtc_pre_params_count{chain_address="0x0954efEFeb970D317a51736201B4EB2De75FF5DE", job="keep-discovered-nodes"} [2591999s:120s] offset 74124s)
5) client_info{chain_address="0x0954efEFeb970D317a51736201B4EB2De75FF5DE", job="keep-discovered-nodes"}[2591999s:120s] offset 74124s

I switched out the chain_address to point at 0x0954efEFeb970D317a51736201B4EB2De75FF5DE because it's an operator with a lot going on and it's on testnet.

Running those above queries in the UI: https://monitoring.test.threshold.network/prometheus/graph

We get

image
image
image
image
image

Here's how to read these:

present_over_time(up{chain_address="0x0954efEFeb970D317a51736201B4EB2De75FF5DE", job="keep-discovered-nodes"} [2591999s] offset 74124s)

We can read these as an inner query and outer query. The inner query is up{chain_address="0x0954efEFeb970D317a51736201B4EB2De75FF5DE", job="keep-discovered-nodes"} [2591999s] offset 74124s, which has a couple things going on. The base is up{chain_address="0x0954efEFeb970D317a51736201B4EB2De75FF5DE", job="keep-discovered-nodes"}, which means "return the up data where the chain address is 0x0954efEFeb970D317a51736201B4EB2De75FF5DE and the job is keep-discovered-nodes"

Then, we specify a time range: 2591999s, which is ~30 days. So that part says "run this up query on the data for the last 30 days".

Finally, we supply an offset of 74124s which is 20.6 hours. This shifts our range backwards, and makes the query "run this up query on 30 days worth of data starting 20.6 hours ago".

Then, we aggregate that data via the present_over_time function which return a value of 1 if there is any data present in the time interval.

All of the other functions follow a similar format with one addition: the resolution param.

sum_over_time(up{chain_address="0x0954efEFeb970D317a51736201B4EB2De75FF5DE", job="keep-discovered-nodes"} [364679s:120s] offset 74124s) * 120 / 364679

In the above query, 364679s:120s says "retrieve 364679s worth of data, and grab a data point every 120s".

Since these data points are either 1s or 0s, we can sum up how many data points we found, multiply that by 120 (the number of seconds in the interval), and divide by the total seconds in the range to get the total uptime percentage.

Everything appears to be doing it's job :100:

In a follow up PR, I would investigate whether we need all of the data we're fetching from query 2 (paramsOperatorUptime) and query 5 (buildVersionInstancesParams). I think for paramsOperatorUptime, we're only using that to figure out what the first registered uptime is, and there has to be a better way than making prometheus ship us all of the data points. The version instance params I think only need the latest timestamp.

beaurancourt avatar Oct 31 '22 21:10 beaurancourt

Okay! left some misc review but it's all non-blocking. The code works as-is and is ready for use, so I'm going to ship it! Totally down to handle other concerns in follow-up PRs

beaurancourt avatar Oct 31 '22 21:10 beaurancourt