columnar
columnar copied to clipboard
manticoresoftware
crash on select when too many ft fields
If you run this script:
#!/usr/bin/php
<?php
if (count($argv) < 5) die("Usage: ".__FILE__." <batch size> <concurrency> <docs> <number of fields>\n");
// This function waits for an idle mysql connection for the $query, runs it and exits
function process($query) {
global $all_links;
global $requests;
foreach ($all_links as $k=>$link) {
if (@$requests[$k]) continue;
mysqli_query($link, $query, MYSQLI_ASYNC);
@$requests[$k] = microtime(true);
return true;
}
do {
$links = $errors = $reject = array();
foreach ($all_links as $link) {
$links[] = $errors[] = $reject[] = $link;
}
$count = @mysqli_poll($links, $errors, $reject, 0, 1000);
if ($count > 0) {
foreach ($links as $j=>$link) {
$res = @mysqli_reap_async_query($links[$j]);
foreach ($all_links as $i=>$link_orig) if ($all_links[$i] === $links[$j]) break;
if ($link->error) {
echo "ERROR: {$link->error}\n";
if (!mysqli_ping($link)) {
echo "ERROR: mysql connection is down, removing it from the pool\n";
unset($all_links[$i]); // remove the original link from the pool
unset($requests[$i]); // and from the $requests too
}
return false;
}
if ($res === false and !$link->error) continue;
if (is_object($res)) {
mysqli_free_result($res);
}
$requests[$i] = microtime(true);
mysqli_query($link, $query, MYSQLI_ASYNC); // making next query
return true;
}
};
} while (true);
return true;
}
$all_links = [];
$requests = [];
$c = 0;
for ($i=0;$i<$argv[2];$i ) {
$m = @mysqli_connect('127.0.0.1', '', '', '', 9306);
if (mysqli_connect_error()) die("Cannot connect to Manticore\n");
$all_links[] = $m;
}
// init
mysqli_query($all_links[0], "drop table if exists t2");
$create = "create table t2(";
$ar=[]; for ($n=1;$n<=$argv[4];$n ) $ar[] = "f$n text"; $create .= implode(',', $ar);
$create .= ", s string) engine='columnar'";
mysqli_query($all_links[0], $create);
$batch = [];
$query_start = "insert into t2(";
$ar=[]; for ($n=1;$n<=$argv[4];$n ) $ar[] = "f$n"; $query_start .= implode(',', $ar);
$query_start .= ",s) values ";
srand(1);
echo "preparing...\n";
$error = false;
$cache_file_name = '/tmp/dmitry'.md5($query_start).'_'.$argv[1].'_'.$argv[3];
$c = 0;
if (!file_exists($cache_file_name)) {
$batches = [];
while ($c < $argv[3]) {
$ar = [];
for ($n=0;$n<$argv[4];$n ) $ar[] = "'".substr(md5(rand()), 0, 4)."'";
$ar[] = "'".uniqid()."'";
$batch[] = "(".implode(',', $ar).")";
$c ;
if (floor($c/1000) == $c/1000) echo "\r".($c/$argv[3]*100)."% ";
if (count($batch) == $argv[1]) {
$batches[] = $query_start.implode(',', $batch);
$batch = [];
}
}
if ($batch) $batches[] = $query_start.implode(',', $batch);
file_put_contents($cache_file_name, serialize($batches));
} else {
echo "found in cache\n";
$batches = unserialize(file_get_contents($cache_file_name));
}
echo "querying...\n";
$t = microtime(true);
foreach ($batches as $batch) {
if (!process($batch)) die("ERROR\n");
}
// wait until all the workers finish
do {
$links = $errors = $reject = array();
foreach ($all_links as $link) $links[] = $errors[] = $reject[] = $link;
$count = @mysqli_poll($links, $errors, $reject, 0, 100);
} while (count($all_links) != count($links) count($errors) count($reject));
echo "finished inserting\n";
echo "Total time: ".(microtime(true) - $t)."\n";
echo round($argv[3] / (microtime(true) - $t))." docs per sec\n";
like this:
rm /tmp/dmitry*
php load_sql_columnar_crash.php 1000 4 50000 47
against version Server version: 5.0.3 aea4e54ff@220727 dev (columnar 1.16.1 637ddc6@220727) git branch HEAD (no branch)
then the following will cause a crash:
snikolaev@dev:~$ mysql -P9306 -h0 -e "select * from t2 where s='abc'"
ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query
The config is :
snikolaev@dev:~$ cat /etc/manticoresearch/manticore.conf
common {
plugin_dir = /usr/local/lib/manticore
lemmatizer_base = /var/lib/manticore/
}
searchd {
listen = 127.0.0.1:9312
listen = 127.0.0.1:9306:mysql
listen = 127.0.0.1:9308:http
log = /var/log/manticore/searchd.log
query_log = /var/log/manticore/query.log
pid_file = /var/run/manticore/searchd.pid
data_dir = /var/lib/manticore
query_log_format = sphinxql
binlog_max_log_size = 1M
max_open_files = max
max_packet_size = 128M
}
Notes:
- it crashes in the latest release as well
- if I remove
engine='columnar'it doesn't crash - if I lower the number of ft fields to 40 (
php load_sql_columnar_crash.php 1000 4 50000 40) it doesn't crash - when it crashes you have to remove the index dir manually, otherwise you can't create the table again:
nikolaev@dev:~$ mysql -P9306 -h0 -e "select * from t2 where s='abc'" RROR 2013 (HY000) at line 1: Lost connection to MySQL server during query nikolaev@dev:~$ sudo cat /var/lib/manticore/manticore.json "clusters": { }, "indexes": { "chinese_tokenization": { "type": "rt", "path": "chinese_tokenization" }, "t1": { "type": "rt", "path": "t1" }, "t2": { "type": "rt", "path": "t2" } } snikolaev@dev:~$mysql -P9306 -h0 -e "show tables" ---------------------- ------ Index | Type | ---------------------- ------ chinese_tokenization | rt | t1 | rt | ---------------------- ------ - archive of the index is t2.tgz
- the crash looks really bad (no crashlog):
Wed Jul 27 18:14:16.110 2022] [334175] prereading 2 indexes Wed Jul 27 18:14:16.110 2022] [334171] accepting connections Wed Jul 27 18:14:16.120 2022] [334175] preread 2 indexes in 0.010 sec Wed Jul 27 18:19:12.955 2022] [334174] rt: index t2: diskchunk 0(1), segments 29 saved in 0.443840 (0.535320) sec, RAM saved/new 48616545/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728) Wed Jul 27 18:21:04.950 2022] [327687] watchdog: main process 334171 killed dirtily with signal 11, core dumped, will be restarted Wed Jul 27 18:21:04.950 2022] [327687] watchdog: main process 337621 forked ok Wed Jul 27 18:21:04.951 2022] [337621] starting daemon version '5.0.3 aea4e54ff@220727 dev (columnar 1.16.1 637ddc6@220727)' ... Wed Jul 27 18:21:04.951 2022] [337621] listening on 127.0.0.1:9312 for sphinx and http(s) Wed Jul 27 18:21:04.951 2022] [337621] listening on 127.0.0.1:9306 for mysql
