Test time to data

[harbu]

How to run

npm run bootstrap
cd packages/sdk
./loop-run.sh

Measurements (2024-08-16)

• Each run is 30 trials
• Websocket server has been disabled to mimic browser environment more closely
• "Pre-populate" getStream cache trick employed
• Full proxy behavior with accelerated entry point discovery (* we have to do retries until we hit an operator running latest version, only successful one is counted in time, retries are not counted towards time)

Normal vanilla run

Received 1st message in 1659 ms (runMode=normal)
Received 1st message in 1419 ms (runMode=normal)
Received 1st message in 3169 ms (runMode=normal)
Received 1st message in 2229 ms (runMode=normal)
Received 1st message in 5483 ms (runMode=normal)
Received 1st message in 4497 ms (runMode=normal)
Received 1st message in 2237 ms (runMode=normal)
Received 1st message in 2191 ms (runMode=normal)
Received 1st message in 2367 ms (runMode=normal)
Received 1st message in 16933 ms (runMode=normal)
Received 1st message in 2060 ms (runMode=normal)
Received 1st message in 1348 ms (runMode=normal)
Received 1st message in 1090 ms (runMode=normal)
Received 1st message in 1607 ms (runMode=normal)
Received 1st message in 1926 ms (runMode=normal)
Received 1st message in 1716 ms (runMode=normal)
Received 1st message in 1908 ms (runMode=normal)
Received 1st message in 1856 ms (runMode=normal)
Received 1st message in 995 ms (runMode=normal)
Received 1st message in 1687 ms (runMode=normal)
Received 1st message in 2068 ms (runMode=normal)
Received 1st message in 5383 ms (runMode=normal)
Received 1st message in 2446 ms (runMode=normal)
Received 1st message in 1209 ms (runMode=normal)
Received 1st message in 4540 ms (runMode=normal)
Received 1st message in 2022 ms (runMode=normal)
Received 1st message in 1655 ms (runMode=normal)
Received 1st message in 2597 ms (runMode=normal)
Received 1st message in 3461 ms (runMode=normal)
Received 1st message in 1257 ms (runMode=normal)

Mean: 2833.83 ms Median: 2041.00 ms Standard Deviation: 2870.11 ms Minimum Value: 995 ms Maximum Value: 16933 ms 25th Percentile (Q1): 1656.00 ms 75th Percentile (Q3): 2559.25 ms

Using proxy subscriber pattern (see asterix above)

Received 1st message in 2711 ms (runMode=proxy)
Received 1st message in 1017 ms (runMode=proxy)
Received 1st message in 3468 ms (runMode=proxy)
Received 1st message in 2858 ms (runMode=proxy)
Received 1st message in 1338 ms (runMode=proxy)
Received 1st message in 1212 ms (runMode=proxy)
Received 1st message in 2266 ms (runMode=proxy)
Received 1st message in 1993 ms (runMode=proxy)
Received 1st message in 1442 ms (runMode=proxy)
Received 1st message in 2871 ms (runMode=proxy)
Received 1st message in 2502 ms (runMode=proxy)
Received 1st message in 3069 ms (runMode=proxy)
Received 1st message in 2039 ms (runMode=proxy)
Received 1st message in 1610 ms (runMode=proxy)
Received 1st message in 6801 ms (runMode=proxy)
Received 1st message in 1646 ms (runMode=proxy)
Received 1st message in 2294 ms (runMode=proxy)
Received 1st message in 2294 ms (runMode=proxy)
Received 1st message in 1541 ms (runMode=proxy)
Received 1st message in 1775 ms (runMode=proxy)
Received 1st message in 1494 ms (runMode=proxy)
Received 1st message in 2320 ms (runMode=proxy)
Received 1st message in 1817 ms (runMode=proxy)
Received 1st message in 1845 ms (runMode=proxy)
Received 1st message in 909 ms (runMode=proxy)
Received 1st message in 1100 ms (runMode=proxy)
Received 1st message in 1376 ms (runMode=proxy)
Received 1st message in 1104 ms (runMode=proxy)
Received 1st message in 1740 ms (runMode=proxy)
Received 1st message in 1125 ms (runMode=proxy)

Mean: 2052.57 ms Median: 1796.00 ms Standard Deviation: 1096.65 ms Minimum Value: 909 ms Maximum Value: 6801 ms 25th Percentile (Q1): 1392.50 ms 75th Percentile (Q3): 2313.50 ms

Old Measurements

Each run is 30 trials. Websocket server has been disabled to mimic browser environment more closely.

Normal vanilla run

Received 1st message in 1902 ms
Received 1st message in 2456 ms
Received 1st message in 2020 ms
Received 1st message in 2641 ms
Received 1st message in 17260 ms
Received 1st message in 5395 ms
Received 1st message in 2150 ms
Received 1st message in 2501 ms
Received 1st message in 2251 ms
Received 1st message in 2185 ms
Received 1st message in 1925 ms
Received 1st message in 2455 ms
Received 1st message in 3295 ms
Received 1st message in 6353 ms
Received 1st message in 2039 ms
Received 1st message in 17300 ms
Received 1st message in 2051 ms
Received 1st message in 3526 ms
Received 1st message in 1639 ms
Received 1st message in 3421 ms
Received 1st message in 3076 ms
Received 1st message in 3345 ms
Received 1st message in 2220 ms
Received 1st message in 2610 ms
Received 1st message in 1981 ms
Received 1st message in 1934 ms
Received 1st message in 1963 ms
Received 1st message in 1837 ms
Received 1st message in 2593 ms
Received 1st message in 4877 ms

Mean: 3706.7 ms Standard Deviation: 3783.60 ms Median: 2455.5 ms Min: 1639 ms Max: 17300 ms Q1 (25th percentile): 2024.75 ms Q3 (75th percentile): 3332.5 ms

Hardcoded layer0 entrypoint + streamEntrypoint

Received 1st message in 1853 ms
Received 1st message in 16395 ms
Received 1st message in 18365 ms
Received 1st message in 1143 ms
Received 1st message in 1552 ms
Received 1st message in 2369 ms
Received 1st message in 1008 ms
Received 1st message in 16946 ms
Received 1st message in 1497 ms
Received 1st message in 1266 ms
Received 1st message in 1212 ms
Received 1st message in 1759 ms
Received 1st message in 2377 ms
Received 1st message in 1037 ms
Received 1st message in 1959 ms
Received 1st message in 1133 ms
Received 1st message in 1689 ms
Received 1st message in 2996 ms
Received 1st message in 2203 ms
Received 1st message in 1766 ms
Received 1st message in 1775 ms
Received 1st message in 1117 ms
Received 1st message in 1215 ms
Received 1st message in 2439 ms
Received 1st message in 1686 ms
Received 1st message in 1253 ms
Received 1st message in 1942 ms
Received 1st message in 1491 ms
Received 1st message in 1459 ms
Received 1st message in 2527 ms

Mean: 3247.63 ms Standard Deviation: 4695.38 ms Median: 1724.0 ms Min: 1008 ms Max: 18365 ms Q1 (25th percentile): 1256.25 ms Q3 (75th percentile): 2327.5 ms

Using proxy subscriber pattern (default entrypoints, hard-coded proxy peer descriptor)

Received 1st message in 1462 ms
Received 1st message in 929 ms
Received 1st message in 809 ms
Received 1st message in 1621 ms
Received 1st message in 2191 ms
Received 1st message in 867 ms
Received 1st message in 974 ms
Received 1st message in 1373 ms
Received 1st message in 2118 ms
Received 1st message in 1802 ms
Received 1st message in 1645 ms
Received 1st message in 1703 ms
Received 1st message in 865 ms
Received 1st message in 2227 ms
Received 1st message in 1709 ms
Received 1st message in 1002 ms
Received 1st message in 980 ms
Received 1st message in 1751 ms
Received 1st message in 917 ms
Received 1st message in 790 ms
Received 1st message in 1906 ms
Received 1st message in 1019 ms
Received 1st message in 1089 ms
Received 1st message in 1744 ms
Received 1st message in 870 ms
Received 1st message in 952 ms
Received 1st message in 1786 ms
Received 1st message in 1304 ms
Received 1st message in 1329 ms
Received 1st message in 1503 ms

Mean: 1374.57 ms Standard Deviation: 444.45 ms Median: 1351.0 ms Min: 790 ms Max: 2227 ms Q1 (25th percentile): 957.5 ms Q3 (75th percentile): 1735.25 ms