C2 large table

Open griddis opened this issue 4 years ago • 1 comments

Hello, we ran into a problem that on some servers, in particular, it is precisely confirmed that the dell m630 raft is generated with a large table C2, and since it is completely captured by the harvester in memory, there is a strong consumption of RAM and CPU, with a large number of disks it simply dies under load

2021/12/26 22:27:32 Generating plot 1 / 1: bcc43f3f56d8897df2e9a437e5ec92f5247b789a3234d0201decf5d410f1a464
2021/12/26 22:27:32 Running Phase 1
2021/12/26 22:27:32 Generating F1...
2021/12/26 22:27:34 Finished F1 generation in 1.90 seconds.
2021/12/26 22:27:34 Sorting F1...
2021/12/26 22:27:44 Finished F1 sort in 9.90 seconds.
2021/12/26 22:27:44 Forward propagating to table 2...
2021/12/26 22:27:44   Pairing L/R groups...
2021/12/26 22:27:57   Finished pairing L/R groups in 13.0130 seconds. Created 4294913632 pairs.
2021/12/26 22:27:57   Average of 236.1377 pairs per group.
2021/12/26 22:27:57   Computing Fx...
2021/12/26 22:28:14   Finished computing Fx in 16.8320 seconds.
2021/12/26 22:28:14   Sorting entries...
2021/12/26 22:28:40   Finished sorting in 25.75 seconds.
2021/12/26 22:28:40 Finished forward propagating table 2 in 56.08 seconds.
2021/12/26 22:28:40 Forward propagating to table 3...
2021/12/26 22:28:41   Pairing L/R groups...
2021/12/26 22:28:54   Finished pairing L/R groups in 13.0520 seconds. Created 4294813016 pairs.
2021/12/26 22:28:54   Average of 236.1321 pairs per group.
2021/12/26 22:28:54   Computing Fx...
2021/12/26 22:29:12   Finished computing Fx in 18.4310 seconds.
2021/12/26 22:29:12   Sorting entries...
2021/12/26 22:29:39   Finished sorting in 26.76 seconds.
2021/12/26 22:29:39 Finished forward propagating table 3 in 58.73 seconds.
2021/12/26 22:29:39 Forward propagating to table 4...
2021/12/26 22:29:39   Pairing L/R groups...
2021/12/26 22:29:52   Finished pairing L/R groups in 13.0350 seconds. Created 4294633976 pairs.
2021/12/26 22:29:52   Average of 236.1223 pairs per group.
2021/12/26 22:29:52   Computing Fx...
2021/12/26 22:30:11   Finished computing Fx in 18.9310 seconds.
2021/12/26 22:30:11   Sorting entries...
2021/12/26 22:30:38   Finished sorting in 26.90 seconds.
2021/12/26 22:30:38 Finished forward propagating table 4 in 59.36 seconds.
2021/12/26 22:30:38 Forward propagating to table 5...
2021/12/26 22:30:39   Pairing L/R groups...
2021/12/26 22:30:52   Finished pairing L/R groups in 13.0830 seconds. Created 4294426557 pairs.
2021/12/26 22:30:52   Average of 236.1109 pairs per group.
2021/12/26 22:30:52   Computing Fx...
2021/12/26 22:31:10   Finished computing Fx in 18.4930 seconds.
2021/12/26 22:31:10   Sorting entries...
2021/12/26 22:31:37   Finished sorting in 26.75 seconds.
2021/12/26 22:31:37 Finished forward propagating table 5 in 58.82 seconds.
2021/12/26 22:31:37 Forward propagating to table 6...
2021/12/26 22:31:37   Pairing L/R groups...
2021/12/26 22:31:50   Finished pairing L/R groups in 13.0270 seconds. Created 4293863604 pairs.
2021/12/26 22:31:50   Average of 236.0799 pairs per group.
2021/12/26 22:31:50   Computing Fx...
2021/12/26 22:32:08   Finished computing Fx in 17.9250 seconds.
2021/12/26 22:32:08   Sorting entries...
2021/12/26 22:32:35   Finished sorting in 26.22 seconds.
2021/12/26 22:32:35 Finished forward propagating table 6 in 57.66 seconds.
2021/12/26 22:32:35 Forward propagating to table 7...
2021/12/26 22:32:35   Pairing L/R groups...
2021/12/26 22:32:48   Finished pairing L/R groups in 13.0910 seconds. Created 4292825712 pairs.
2021/12/26 22:32:48   Average of 236.0229 pairs per group.
2021/12/26 22:32:48   Computing Fx...
2021/12/26 22:33:04   Finished computing Fx in 15.8520 seconds.
2021/12/26 22:33:04 Finished forward propagating table 7 in 29.43 seconds.
2021/12/26 22:33:04 Finished Phase 1 in 331.89 seconds.
2021/12/26 22:33:04 Running Phase 2
2021/12/26 22:33:04   Prunning table 6...
2021/12/26 22:33:05   Finished prunning table 6 in 0.56 seconds.
2021/12/26 22:33:05   Prunning table 5...
2021/12/26 22:33:16   Finished prunning table 5 in 10.68 seconds.
2021/12/26 22:33:16   Prunning table 4...
2021/12/26 22:33:26   Finished prunning table 4 in 9.94 seconds.
2021/12/26 22:33:26   Prunning table 3...
2021/12/26 22:33:35   Finished prunning table 3 in 9.77 seconds.
2021/12/26 22:33:35   Prunning table 2...
2021/12/26 22:33:45   Finished prunning table 2 in 9.72 seconds.
2021/12/26 22:33:45 Finished Phase 2 in 40.99 seconds.
2021/12/26 22:33:45 Running Phase 3
2021/12/26 22:33:45   Compressing tables 1 and 2...
2021/12/26 22:34:19   Finished compressing tables 1 and 2 in 34.02 seconds
2021/12/26 22:34:19   Table 1 now has 3429286888 / 4294913632 entries ( 79.85% ).
2021/12/26 22:34:19   Compressing tables 2 and 3...
2021/12/26 22:34:56   Finished compressing tables 2 and 3 in 36.74 seconds
2021/12/26 22:34:56   Table 2 now has 3439693817 / 4294813016 entries ( 80.09% ).
2021/12/26 22:34:56   Compressing tables 3 and 4...
2021/12/26 22:35:36   Finished compressing tables 3 and 4 in 39.91 seconds
2021/12/26 22:35:36   Table 3 now has 3465721255 / 4294633976 entries ( 80.70% ).
2021/12/26 22:35:36   Compressing tables 4 and 5...
2021/12/26 22:36:14   Finished compressing tables 4 and 5 in 37.79 seconds
2021/12/26 22:36:14   Table 4 now has 3532291334 / 4294426557 entries ( 82.25% ).
2021/12/26 22:36:14   Compressing tables 5 and 6...
2021/12/26 22:36:55   Finished compressing tables 5 and 6 in 41.95 seconds
2021/12/26 22:36:55   Table 5 now has 3712450130 / 4293863604 entries ( 86.46% ).
2021/12/26 22:36:55   Compressing tables 6 and 7...
2021/12/26 22:37:36   Finished compressing tables 6 and 7 in 40.16 seconds
2021/12/26 22:37:36   Table 6 now has 4292825710 / 4292825712 entries ( 100.00% ).
2021/12/26 22:37:36 Finished Phase 3 in 230.57 seconds.
2021/12/26 22:37:36 Running Phase 4
2021/12/26 22:37:36   Writing P7.
2021/12/26 22:37:36   Finished writing P7 in 0.68 seconds.
2021/12/26 22:37:36   Writing C1 table.
2021/12/26 22:37:36   Finished writing C1 table in 0.00 seconds.
2021/12/26 22:37:36   Writing C2 table.
2021/12/26 22:37:36   Finished writing C2 table in 0.00 seconds.
2021/12/26 22:37:36   Writing C3 table.
2021/12/26 22:37:37   Finished writing C3 table in 0.66 seconds.
2021/12/26 22:37:37 Finished Phase 4 in 1.34 seconds.
2021/12/26 22:37:37 Writing final plot tables to disk
2021/12/26 22:38:11 Plot /chiatmp/plot-k32-2021-12-26-22-27-bcc43f3f56d8897df2e9a437e5ec92f5247b789a3234d0201decf5d410f1a464.plot finished writing to disk:
2021/12/26 22:38:11   Table 1 pointer  :          1048576 ( 0x0000000000100000 )
2021/12/26 22:38:11   Table 2 pointer  :      14840496128 ( 0x0000000374900000 )
2021/12/26 22:38:11   Table 3 pointer  :      28823257088 ( 0x00000006b6000000 )
2021/12/26 22:38:11   Table 4 pointer  :      42911924224 ( 0x00000009fdc00000 )
2021/12/26 22:38:11   Table 5 pointer  :      57271123968 ( 0x0000000d55a00000 )
2021/12/26 22:38:11   Table 6 pointer  :      72362229760 ( 0x00000010d9200000 )
2021/12/26 22:38:11   Table 7 pointer  :      89812631552 ( 0x00000014e9400000 )
2021/12/26 22:38:11   C1 table pointer :     107520983040 ( 0x0000001908c00000 )
2021/12/26 22:38:11   C2 table pointer :     107523080192 ( 0x0000001908e00000 )
2021/12/26 22:38:11   C3 table pointer :     107524128768 ( 0x0000001908f00000 )
2021/12/26 22:38:11 Finished writing tables to disk in 34.30 seconds.
2021/12/26 22:38:11 Finished plotting in 639.09 seconds (10.65 minutes).

Dec 26 '21 19:12 griddis

That's quite the large C2 table. In reality they should only have a maximum if 43 entries for bladebit, which is tiny. But the table pointers get aligned to the filesystem block size, which by the looks of it you are using a block size of 1 MiB.

The table content itself is small, the rest is empty data use for alignment since we do unbuffered writes to disk.

Unfortunately, the prover disk right now loads C2 tables by substracting table pointers, I believe, which will cause 1 MiB to be allocated per C2; much more than what it actually needs.

I will bring this up internally to see what we can do with the disk prover. Thanks for bringing this up.

Jan 07 '22 22:01 harold-b