Arjun Sunil Kumar
Arjun Sunil Kumar
The ideal cache size is 6GB. Meta - very small Centrois - 500 x VECF32(960) = 500 x 4KB = 2MB Entry - 1000,000 x VECF32(960) = 1000,000 x 4KB...
## Steps ```shell curl -o trace.out http://localhost:9876/debug/pprof/trace?seconds=10 go tool trace trace.out ``` [trace.out.zip](https://github.com/matrixorigin/matrixone/files/15386629/trace.out.zip) ## Result 1. Find the tasks with time in `ms`.   2. Trace from `LocalFs.Read` 15ms...
### Use L2DistanceFn directly in SorkKey (current implementation) -- 15QPS  [cpu_1.pprof.zip](https://github.com/user-attachments/files/15523089/cpu_1.pprof.zip) ```sql mysql> explain analyze SELECT a FROM t5 ORDER BY l2_distance(b,'[0.0326,0.0248,0.0067,0.0173,0.0216,0.0097,0.015,0.0246,0.0071,0.0469,0.0939,0.0408,0.0207,0.0125,0.0112,0.0311,0.036,0.0323,0.0075,0.0138,0.0235,0.0139,0.0149,0.0231,0.0066,0.0203,0.0627,0.0445,0.0142,0.0078,0.0078,0.0147,0.0402,0.0594,0.0086,0.0153,0.0399,0.0129,0.0138,0.0254,0.0102,0.0092,0.0434,0.0321,0.0174,0.0138,0.0113,0.0274,0.0569,0.0648,0.0079,0.0187,0.0726,0.0181,0.0153,0.024,0.0116,0.0254,0.0511,0.0462,0.0158,0.0174,0.0116,0.0396,0.0659,0.0465,0.0088,0.0178,0.0917,0.0199,0.0189,0.0266,0.0153,0.0397,0.0484,0.0432,0.024,0.0174,0.0088,0.0333,0.072,0.045,0.0119,0.0333,0.0644,0.0253,0.0383,0.0443,0.012,0.0286,0.0709,0.041,0.0178,0.0252,0.0118,0.025,0.0723,0.0421,0.0153,0.0239,0.0339,0.0205,0.0313,0.062,0.0086,0.0184,0.0755,0.0639,0.016,0.0167,0.0152,0.0232,0.0456,0.0305,0.0082,0.0251,0.0279,0.0108,0.0196,0.0444,0.0068,0.0361,0.0704,0.0601,0.0201,0.0116,0.0088,0.0364,0.0454,0.0341,0.0063,0.0222,0.0363,0.0272,0.0291,0.0238,0.0078,0.0439,0.0923,0.0352,0.0211,0.0185,0.0191,0.0158,0.0596,0.052,0.0091,0.0194,0.0334,0.0262,0.0243,0.026,0.0068,0.024,0.0652,0.0546,0.0152,0.0128,0.011,0.0206,0.038,0.0518,0.0076,0.0193,0.0208,0.0172,0.0132,0.0229,0.01,0.015,0.0222,0.0344,0.0212,0.0208,0.0187,0.0207,0.0713,0.0622,0.0121,0.0252,0.0453,0.0322,0.0235,0.0276,0.0135,0.0409,0.0494,0.0407,0.0193,0.0228,0.0253,0.0343,0.0624,0.0513,0.0134,0.0156,0.0306,0.0248,0.025,0.0314,0.0195,0.0614,0.0688,0.0432,0.0251,0.0307,0.0183,0.0118,0.061,0.0557,0.0171,0.0327,0.0305,0.0228,0.0359,0.0276,0.0158,0.0446,0.08,0.0273,0.0194,0.0321,0.0192,0.0104,0.0635,0.0594,0.0155,0.023,0.0255,0.0285,0.0331,0.0342,0.0123,0.0162,0.0806,0.0382,0.015,0.021,0.0159,0.0112,0.0287,0.0395,0.0113,0.0249,0.0326,0.0281,0.0196,0.0301,0.009,0.0242,0.0671,0.0277,0.0089,0.0126,0.0108,0.012,0.0129,0.0201,0.0074,0.0226,0.0217,0.0138,0.0359,0.0144,0.0134,0.0252,0.0637,0.0185,0.0109,0.0248,0.0251,0.0287,0.0266,0.0209,0.0082,0.0132,0.019,0.0092,0.0174,0.0142,0.0133,0.022,0.0393,0.0211,0.0202,0.0192,0.0162,0.0246,0.0548,0.034,0.022,0.0304,0.0188,0.0145,0.0154,0.0172,0.0241,0.0239,0.0351,0.0191,0.0253,0.0233,0.0149,0.0191,0.0331,0.0277,0.0201,0.0332,0.0332,0.0179,0.0331,0.0339,0.0214,0.0226,0.0566,0.0295,0.0187,0.0173,0.0186,0.0178,0.0315,0.0242,0.0059,0.0194,0.0172,0.0085,0.0152,0.0218,0.0055,0.042,0.0844,0.0366,0.0162,0.0116,0.0102,0.03,0.0258,0.0287,0.0064,0.0112,0.0153,0.0099,0.0141,0.0211,0.0073,0.0194,0.0502,0.0407,0.0127,0.0117,0.0069,0.0138,0.035,0.0542,0.0071,0.0101,0.0338,0.0116,0.0131,0.023,0.0096,0.009,0.0358,0.0286,0.0171,0.0104,0.0093,0.0216,0.0495,0.0584,0.0062,0.0139,0.0639,0.0137,0.0126,0.0215,0.011,0.0188,0.0411,0.0401,0.0146,0.0118,0.0098,0.033,0.0661,0.0427,0.0097,0.0172,0.0815,0.0125,0.0168,0.0277,0.0126,0.0266,0.038,0.0371,0.0201,0.0113,0.0083,0.0295,0.0618,0.0391,0.0108,0.0293,0.0628,0.0219,0.0348,0.0402,0.0095,0.0231,0.0595,0.0391,0.0141,0.0178,0.011,0.0215,0.0584,0.0333,0.0136,0.0216,0.0309,0.0164,0.0286,0.057,0.0072,0.0174,0.0598,0.0551,0.015,0.0133,0.013,0.0183,0.0406,0.0257,0.0077,0.0228,0.0189,0.0078,0.0167,0.0405,0.0062,0.0343,0.0621,0.0545,0.0146,0.0085,0.0074,0.0286,0.033,0.0287,0.0047,0.0225,0.0319,0.0185,0.0224,0.0225,0.007,0.0431,0.0899,0.0361,0.0155,0.0156,0.0169,0.0168,0.0408,0.0427,0.0089,0.0155,0.025,0.0198,0.0197,0.0221,0.007,0.0278,0.0577,0.0488,0.0126,0.016,0.0122,0.019,0.0383,0.0483,0.0057,0.0149,0.0202,0.0153,0.0108,0.0197,0.0086,0.0151,0.0182,0.0293,0.0208,0.0144,0.0174,0.0179,0.0714,0.0606,0.0087,0.0204,0.0383,0.026,0.019,0.0214,0.0091,0.0294,0.0429,0.0323,0.0166,0.0171,0.0233,0.0276,0.0669,0.0538,0.0145,0.015,0.0308,0.0252,0.0253,0.0328,0.0176,0.041,0.0587,0.0351,0.0222,0.0195,0.0175,0.0092,0.0543,0.0491,0.0148,0.0299,0.0259,0.0216,0.0313,0.0279,0.0129,0.0332,0.0629,0.0253,0.015,0.0205,0.0155,0.0097,0.0494,0.0462,0.0144,0.0206,0.0228,0.0193,0.0293,0.0317,0.009,0.0122,0.062,0.0314,0.0147,0.016,0.0146,0.0101,0.0189,0.0327,0.0074,0.0262,0.0264,0.0198,0.0168,0.0282,0.01,0.026,0.0641,0.025,0.008,0.0094,0.0084,0.0097,0.0094,0.0101,0.0079,0.0206,0.0236,0.0125,0.0337,0.0137,0.0131,0.0269,0.0635,0.0187,0.0081,0.0169,0.0272,0.0275,0.0252,0.0171,0.0068,0.0119,0.0175,0.0088,0.0176,0.0141,0.0126,0.0221,0.0417,0.0193,0.0147,0.0146,0.0168,0.0229,0.0511,0.0291,0.017,0.0234,0.0188,0.0148,0.0154,0.0183,0.02,0.0159,0.0272,0.0167,0.0209,0.014,0.0146,0.0178,0.03,0.0189,0.0156,0.0279,0.0335,0.0166,0.0293,0.032,0.0154,0.0149,0.0457,0.0246,0.0145,0.0093,0.0143,0.014,0.0306,0.0245,0.0059,0.0202,0.0152,0.0081,0.013,0.0206,0.0055,0.0416,0.078,0.0326,0.0162,0.0114,0.0098,0.0324,0.027,0.0229,0.0075,0.0114,0.0147,0.0088,0.0139,0.0207,0.0073,0.0188,0.0486,0.0398,0.0131,0.0114,0.0055,0.0145,0.0376,0.048,0.009,0.0121,0.0357,0.0111,0.0111,0.0217,0.0076,0.0075,0.0314,0.0269,0.0155,0.009,0.0082,0.0238,0.0532,0.0576,0.0062,0.017,0.0643,0.0142,0.0139,0.0206,0.0083,0.019,0.0377,0.0423,0.0138,0.013,0.0082,0.0316,0.0705,0.0411,0.0082,0.0212,0.0869,0.0133,0.0196,0.0255,0.0123,0.0258,0.0346,0.0363,0.0213,0.0143,0.0064,0.0253,0.065,0.0378,0.0097,0.0288,0.0634,0.021,0.0317,0.0372,0.0097,0.0231,0.0509,0.0332,0.0114,0.0147,0.01,0.0146,0.0618,0.0334,0.0128,0.0213,0.0324,0.016,0.0257,0.0557,0.0068,0.0149,0.0536,0.0519,0.0123,0.0117,0.0129,0.0181,0.0405,0.0303,0.0081,0.023,0.0188,0.0085,0.0174,0.0378,0.0062,0.0321,0.0557,0.0512,0.0152,0.0087,0.0062,0.0281,0.0274,0.0269,0.0053,0.0225,0.0248,0.0197,0.0251,0.0216,0.0072,0.0422,0.0856,0.0329,0.0126,0.013,0.0155,0.0151,0.0372,0.0292,0.0075,0.0127,0.0222,0.0166,0.0228,0.0216,0.0062,0.0225,0.0594,0.0453,0.0105,0.0113,0.0091,0.0149,0.0424,0.0387,0.0051,0.0167,0.0192,0.018,0.0095,0.0184,0.0056,0.0139,0.0185,0.0259,0.0207,0.0124,0.014,0.015,0.0783,0.0542,0.0082,0.0243,0.0363,0.0197,0.0167,0.0193,0.0085,0.0294,0.04,0.0302,0.0178,0.0163,0.0192,0.025,0.0755,0.045,0.0105,0.0181,0.0332,0.0221,0.0222,0.0279,0.0174,0.0375,0.052,0.0342,0.0283,0.0187,0.0152,0.0133,0.0643,0.0483,0.0113,0.0256,0.0277,0.0196,0.0244,0.0244,0.0133,0.0314,0.0543,0.0214,0.0114,0.0161,0.0136,0.0129,0.0565,0.0484,0.0119,0.0193,0.0268,0.0178,0.0221,0.029,0.0068,0.0092,0.0547,0.029,0.0115,0.0138,0.0134,0.0132,0.0202,0.0387,0.0079,0.0239,0.0277,0.0211,0.0168,0.0291,0.009,0.0232,0.0582,0.0223,0.0074,0.0108,0.0093,0.0113,0.0103,0.0086,0.0067,0.0193,0.023,0.0135,0.0301,0.0144,0.0125,0.0242,0.0587,0.0197,0.0071,0.015,0.0243,0.0242,0.0289,0.0173,0.0067,0.0111,0.0209,0.0088,0.0159,0.0145,0.0115,0.0191,0.0362,0.0148,0.014,0.0116,0.0115,0.0167,0.0567,0.0293,0.0126,0.0217,0.0208,0.0157,0.0105,0.0164,0.0153,0.0115,0.0212,0.0135,0.0157,0.0099,0.0081,0.0102,0.0392,0.0162,0.0123,0.029,0.0365,0.0197,0.0241,0.0272,0.0122,0.0117,0.0402,0.0231,0.0099,0.0064,0.0107,0.0161]') ASC LIMIT 100; +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | QUERY PLAN...
This QPS can be improved by this issue: https://github.com/matrixorigin/matrixone/issues/16387
The CGO Call (18QPS) is more expensive than the Golang Native L2Distance (26 QPS) Changes: https://github.com/arjunsk/matrixone/pull/94 
## Latest results QPS has improved from 25 to 65 (concurrent clients 5)  As you can see, the `mpool.Grow()` impact is fixed in both `TableScan` and `SEMI Join`. Some...
Trace file [trace.out.zip](https://github.com/user-attachments/files/16229313/trace.out.zip) ```shell go tool trace trace.out ``` Duration in `ms` is mostly dominant in `pipeline.Run`  Read More - Sample Trace for 100ms task  - Sample Trace...
### Steps to reproduce 1. Load t5 table with VECF32(960) 2. Run the below query ```sql drop table if exists meta; drop table if exists centroids; drop table if exists...
### EXPLAIN ANALYZE ```sql +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | QUERY PLAN | +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Plan 0: | | Sink | | Analyze: timeConsumed=4218206ms waitTime=4194757ms inputRows=2000000 outputRows=0 InputSize=7431mb OutputSize=0bytes MemorySize=64mb | | -> Lock...
### Proposal Implement `L2 INNER JOIN` select centroids.version, centroids.id, centroids.centroid, tbl.embedding from centroids l2 inner join tbl where centroids.centroid = tbl.embedding ### NOTES - `where centroids.centroid = tbl.embedding` is only...