CCF icon indicating copy to clipboard operation
CCF copied to clipboard

Add sample of value-based indexing strategy

Open eddyashton opened this issue 3 years ago • 1 comments

Expanding samples of how historical queries/indexes can be used, with a strategy defined entirely in app-space. There's a few more things to do here, though they might not all make this PR:

  • [ ] Add tests for this new endpoint
  • [ ] Add docs referencing this app-space strategy.
  • [ ] Factor out the common bits of historical endpoints. Now that we have several different range/sparse historical queries, we have a lot of duplicated code. We should factor this out, and see if it makes sense to create another adapter for historical queries.
  • [ ] See if we can build an intermediate data structure for dumping data from an index like this to LFS. I don't think they should all access it directly, but I don't yet see the API for doing so generically.

eddyashton avatar May 20 '22 15:05 eddyashton

index_by_value@45717 aka 20220520.21 vs main ewma over 20 builds from 45353 to 45706

Click to see table

main

build_id build_number tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_v8_sgx_cft^ ls_v8_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_full_v8_sgx_cft^ ls_full_v8_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
45353 20220513.6 6226.72 8.55942e+07 19068.5 1.69124e+07 5605.86 1.50774e+07 2588.55 1.00967e+07 1594.51 1.63189e+08 2111.69 9.04813e+06 1410.43 9.81771e+07 1934.28 8.52384e+06 21951.9 882333 1.37652e+06 9.20859e+06 3.58669e+07
45363 20220516.3 6352.8 8.3497e+07 19490.6 1.69124e+07 5636.86 1.53396e+07 2558.6 1.00967e+07 1655.54 1.61092e+08 2131.19 9.04813e+06 1465.37 9.84392e+07 1936.53 8.78598e+06 17780.9 909207 1.35908e+06 9.22518e+06 3.56794e+07
45431 20220516.16 6363.59 8.45456e+07 19479.9 1.71746e+07 5559.39 1.56017e+07 2544.15 1.00967e+07 1602.12 1.63189e+08 2070.54 9.04813e+06 1423.73 9.81771e+07 1931.99 8.78598e+06 19313.4 911184 1.36678e+06 9.45513e+06 3.58036e+07
45448 20220516.20 6341.2 8.40213e+07 19370.9 1.6126e+07 5640.37 1.56017e+07 2497.31 1.00967e+07 1674.41 1.63189e+08 2120.59 9.04813e+06 1445.92 9.84392e+07 1969.79 8.78598e+06 21880 900570 1.36305e+06 9.30905e+06 3.22008e+07
45458 20220517.1 6190.26 8.40213e+07 19545.8 1.6126e+07 5524.26 1.50774e+07 2534.03 1.00967e+07 1618.19 1.63189e+08 2112.88 9.04813e+06 1432.27 9.89635e+07 1928.43 8.78598e+06 19620.1 906266 1.35961e+06 9.39868e+06 3.45363e+07
45482 20220517.8 5676.11 8.32349e+07 19387.5 1.56017e+07 5183.07 1.53396e+07 2450.77 1.00967e+07 1585.83 1.62665e+08 2033.81 1.2456e+07 1414.18 9.81771e+07 1923.27 8.78598e+06 17589 917804 1.38378e+06 9.27948e+06 3.58042e+07
45498 20220517.13 6339.89 8.37592e+07 19648.8 1.66503e+07 5543.33 1.56017e+07 2545.04 1.00967e+07 1678.65 1.62665e+08 2114.29 9.04813e+06 1443.63 9.84392e+07 1963.58 8.78598e+06 17743.3 887382 1.36889e+06 9.19619e+06 3.64413e+07
45520 20220517.20 6255.57 8.3497e+07 19511.2 1.6126e+07 5593.54 1.53396e+07 2537.46 9.83456e+06 1610.13 1.63189e+08 2150.14 9.04813e+06 1419.41 9.81771e+07 1926.63 8.78598e+06 19207.5 903309 1.38088e+06 9.17144e+06 3.58042e+07
45534 20220517.26 6333.45 8.42835e+07 19603.9 1.58639e+07 5546.34 1.53396e+07 2544.54 1.00967e+07 1635.27 1.63713e+08 2083.01 1.27181e+07 1446.06 9.87014e+07 1977.96 8.78598e+06 19352.7 904749 1.37458e+06 9.21688e+06 3.58669e+07
45541 20220517.29 6119.89 8.3497e+07 19296.4 1.69124e+07 5208.35 1.53396e+07 2538.41 1.00967e+07 1593.08 1.62927e+08 2110.37 1.14074e+07 1418.89 9.81771e+07 1923.05 8.78598e+06 19658.7 904588 1.34887e+06 9.21269e+06 3.56174e+07
45560 20220518.3 6225 8.40213e+07 19683.5 1.56017e+07 5622.3 1.53396e+07 2543.73 1.00967e+07 1651.58 1.62927e+08 2114.2 9.04813e+06 1435.43 9.89635e+07 1976.19 8.52384e+06 20095 889385 1.35925e+06 9.2335e+06 3.58669e+07
45562 20220518.4 6357.87 8.42835e+07 19426.1 1.58639e+07 5626.1 1.50774e+07 2538.07 1.00967e+07 1631.42 1.63975e+08 2081.33 9.04813e+06 1420.73 9.76528e+07 2014.41 1.03588e+07 21147.9 876819 1.36505e+06 9.42472e+06 3.56168e+07
45582 20220518.7 6129.08 8.42835e+07 19433.2 1.63882e+07 5616.88 1.50774e+07 2522.2 1.00967e+07 1589.27 1.63451e+08 2116.46 9.04813e+06 1425.92 9.84392e+07 1915.46 8.78598e+06 21832.5 905628 1.36962e+06 9.29633e+06 3.58669e+07
45594 20220518.9 6041.02 8.32349e+07 20000.4 1.58639e+07 5570.11 1.53396e+07 2533.31 1.00967e+07 1601.56 1.62927e+08 2113.38 9.04813e+06 1418.8 9.84392e+07 1972.07 8.52384e+06 19393.5 903230 1.35065e+06 9.42034e+06 3.57411e+07
45604 20220519.3 6222.87 8.42835e+07 19402.4 1.58639e+07 5481.67 1.6126e+07 2532 1.00967e+07 1621.86 1.62927e+08 2147.49 9.04813e+06 1397.25 9.84392e+07 1915.53 8.78598e+06 17324.1 873230 1.36797e+06 9.48144e+06 3.58669e+07
45614 20220519.6 6192.12 8.37592e+07 19433 1.6126e+07 5571.3 1.53396e+07 2522.34 1.00967e+07 1637.84 1.61092e+08 2115.5 9.04813e+06 1459.62 9.84392e+07 2015.66 8.78598e+06 18324.9 910051 1.37236e+06 9.22099e+06 3.58663e+07
45630 20220519.12 6368.6 8.37592e+07 19537.1 1.6126e+07 5643.55 1.53396e+07 2550.37 1.00967e+07 1644.48 1.63451e+08 2117.59 9.04813e+06 1445.35 9.87014e+07 1976.21 8.78598e+06 17521.4 908561 1.36532e+06 9.22518e+06 3.55549e+07
45667 20220520.2 6364.31 8.37592e+07 19429.6 1.6126e+07 5654.34 1.56017e+07 2546.73 1.00967e+07 1643.53 1.63189e+08 2127.17 1.27181e+07 1416.82 9.81771e+07 1969.98 8.52384e+06 20347.6 868490 1.3253e+06 8.89657e+06 3.58042e+07
45686 20220520.10 6218.85 8.40213e+07 19023.6 1.63882e+07 5607.52 1.56017e+07 2542.14 1.00967e+07 1614 1.62927e+08 2105.53 9.04813e+06 1432.3 9.84392e+07 1970.11 8.78598e+06 17648.9 906746 1.36468e+06 9.22095e+06 3.57411e+07
45706 20220520.17 6181.63 8.29727e+07 19600.7 1.56017e+07 5540.12 1.53396e+07 2441.55 9.83456e+06 1608.85 1.63713e+08 2075.58 1.19317e+07 1435.75 9.81771e+07 1921.45 8.78598e+06 19876.7 886116 1.37329e+06 9.22518e+06 3.58042e+07

index_by_value

build_id build_number tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_v8_sgx_cft^ ls_v8_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_full_v8_sgx_cft^ ls_full_v8_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
45717 20220520.21 6194.14 8.27106e+07 16455.7 1.76989e+07 5090.31 1.6126e+07 2545.82 1.00967e+07 1609.73 1.62927e+08 2118.01 9.04813e+06 1436.85 9.76528e+07 1890.28 1.11453e+07 21416.2 907033 1.37977e+06 9.22518e+06 3.66369e+07

images

ccf-bot avatar May 20 '22 15:05 ccf-bot

Closing this PR as not an immediate priority. Will revisit when we add more samples.

eddyashton avatar Sep 16 '22 08:09 eddyashton