docker
docker copied to clipboard
manticoresoftware
Incorrect UTF-8 character display in stdout logs
Russian search results are correct.
When logging to a file, the display of Russian letters is also correct.
But when redirecting query.log to /dev/stdout characters looks like on image:
English letters displayed also correctly.
Please provide more details on how to reproduce the issue and make sure there's no locale issue in the environment. So far I can't reproduce it like this:
msnikolaev@dev:~/issue_20$ cat docker-compose.yml
version: '2.2'
services:
manticore:
container_name: manticore
image: manticoresearch/manticore
restart: always
ports:
- 127.0.0.1:9315:9306
ulimits:
nproc: 65535
nofile:
soft: 65535
hard: 65535
memlock:
soft: -1
hard: -1
environment:
- QUERY_LOG_TO_STDOUT=true
- searchd_query_log_format=plain
volumes:
- ./data:/var/lib/manticore
mysql> create table t(f text);
Query OK, 0 rows affected (0.01 sec)
mysql> select * from t where match('абв');
Empty set (0.00 sec)
snikolaev@dev:~/issue_20$ docker-compose up
Recreating manticore ... done
Attaching to manticore
manticore | [Wed Jun 29 14:59:23.670 2022] [1] using config file '/etc/manticoresearch/manticore.conf' (9168 chars)...
manticore | starting daemon version '5.0.2 348514c86@220530 dev' ...
manticore | listening on all interfaces for mysql, port=9306
manticore | listening on UNIX socket /var/run/mysqld/mysqld.sock
manticore | listening on 192.168.240.2:9312 for sphinx and http(s)
manticore | listening on all interfaces for sphinx and http(s), port=9308
manticore | Manticore 5.0.2 348514c86@220530 dev
manticore | Copyright (c) 2001-2016, Andrew Aksyonoff
manticore | Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
manticore | Copyright (c) 2017-2022, Manticore Software LTD (https://manticoresearch.com)
manticore |
manticore | precaching index 't'
manticore | accepting connections
manticore | prereading 1 indexes
manticore | prereaded 1 indexes in 0.000 sec
manticore | [Wed Jun 29 14:59:25.970 2022] 0.000 sec 0.000 sec [ext2/0/ext 0 (0,20)] [t] абв
docker-compose
# Search Service
tamaranga-manticore:
container_name: tamaranga-manticore
image: tamaranga-manticore:${TAG:-latest}
build:
context: ./docker/manticore/
dockerfile: Dockerfile
restart: unless-stopped
tty: true
ulimits:
nproc: 65535
nofile:
soft: 65535
hard: 65535
memlock:
soft: -1
hard: -1
volumes:
- tamaranga-manticore:/var/lib/manticoresearch
networks:
- internal-network
env_file:
- .env
Dockerfile
FROM manticoresearch/manticore
ARG RU_PACK_URL=https://repo.manticoresearch.com/repository/morphology/ru.pak.tgz
ARG EN_PACK_URL=https://repo.manticoresearch.com/repository/morphology/en.pak.tgz
# Locales installation
RUN apt-get update
RUN apt-get install -y locales-all locales curl xz-utils manticore-columnar-lib
RUN locale-gen ru_RU.UTF-8
# Copy conf file
COPY manticore.conf /etc/manticoresearch/manticore.conf
# Prepare folder for Index files
RUN mkdir -pv /var/lib/manticoresearch
RUN chmod -R 777 /var/lib/manticoresearch
# Install ru.pack
RUN curl -LJO ${RU_PACK_URL}
RUN tar -xvzf ru.pak.tgz -C /usr/share/manticore
RUN rm ru.pak.tgz
# Install en.pack
RUN curl -LJO ${EN_PACK_URL}
RUN tar -xvzf en.pak.tgz -C /usr/share/manticore
RUN rm en.pak.tgz
CMD ["searchd", "--nodetach", "--config", "/etc/manticoresearch/manticore.conf"]
environment: - QUERY_LOG_TO_STDOUT=true - searchd_query_log_format=plain
with your parameters i have no query logs in stdout
I still can't reproduce the issue based on your docker-compose and your Dockerfile. Unfortunately you didn't provide a fully reproducible example, so I had to guess. Please modify this so the problem starts manifesting. Otherwise it's hard/impossible to fix it:
snikolaev@dev:~/issue_20/repro$ cat docker-compose.yml
version: '2.2'
services:
tamaranga-manticore:
container_name: tamaranga-manticore
image: tamaranga-manticore:${TAG:-latest}
build:
context: ./docker/manticore/
dockerfile: Dockerfile
restart: unless-stopped
tty: true
privileged: true
ulimits:
nproc: 65535
nofile:
soft: 65535
hard: 65535
memlock:
soft: -1
hard: -1
volumes:
- tamaranga-manticore:/var/lib/manticoresearch
networks:
- internal-network
env_file:
- .env
volumes:
tamaranga-manticore:
networks:
internal-network:
snikolaev@dev:~/issue_20/repro$ cat .env
TAG=a
snikolaev@dev:~/issue_20/repro$ ls -la docker
total 12
drwxrwxr-x 3 snikolaev snikolaev 4096 Jul 1 06:00 .
drwxrwxr-x 3 snikolaev snikolaev 4096 Jul 1 06:14 ..
drwxrwxr-x 2 snikolaev snikolaev 4096 Jul 1 06:19 manticore
snikolaev@dev:~/issue_20/repro$ ls -la docker/manticore/
total 16
drwxrwxr-x 2 snikolaev snikolaev 4096 Jul 1 06:19 .
drwxrwxr-x 3 snikolaev snikolaev 4096 Jul 1 06:00 ..
-rw-rw-r-- 1 snikolaev snikolaev 899 Jul 1 06:19 Dockerfile
-rw-rw-r-- 1 snikolaev snikolaev 453 Jul 1 06:18 manticore.conf
snikolaev@dev:~/issue_20/repro$ cat docker/manticore/Dockerfile
FROM manticoresearch/manticore
ARG RU_PACK_URL=https://repo.manticoresearch.com/repository/morphology/ru.pak.tgz
ARG EN_PACK_URL=https://repo.manticoresearch.com/repository/morphology/en.pak.tgz
# Locales installation
RUN apt-get update
RUN apt-get install -y locales-all locales curl xz-utils manticore-columnar-lib
RUN locale-gen ru_RU.UTF-8
# Copy conf file
COPY manticore.conf /etc/manticoresearch/manticore.conf
# Prepare folder for Index files
RUN mkdir -pv /var/lib/manticoresearch
RUN chmod -R 777 /var/lib/manticoresearch
# Install ru.pack
RUN curl -LJO ${RU_PACK_URL}
RUN tar -xvzf ru.pak.tgz -C /usr/share/manticore
RUN rm ru.pak.tgz
# Install en.pack
RUN curl -LJO ${EN_PACK_URL}
RUN tar -xvzf en.pak.tgz -C /usr/share/manticore
RUN rm en.pak.tgz
RUN ln -sf /dev/stdout /var/log/manticore/query.log
CMD ["searchd", "--nodetach", "--config", "/etc/manticoresearch/manticore.conf"]
snikolaev@dev:~/issue_20/repro$ cat docker/manticore/manticore.conf
common {
plugin_dir = /usr/local/lib/manticore
lemmatizer_base = /var/lib/manticore/
}
searchd {
listen = 127.0.0.1:9312
listen = 127.0.0.1:9306:mysql
listen = 127.0.0.1:9308:http
log = /var/log/manticore/searchd.log
query_log = /var/log/manticore/query.log
pid_file = /var/run/manticore/searchd.pid
data_dir = /var/lib/manticore
query_log_format = plain
binlog_max_log_size = 1M
max_open_files = max
}
build:
snikolaev@dev:~/issue_20/repro$ docker-compose build tamaranga-manticore
Building tamaranga-manticore
Step 1/17 : FROM manticoresearch/manticore
---> dda3734281a2
Step 2/17 : ARG RU_PACK_URL=https://repo.manticoresearch.com/repository/morphology/ru.pak.tgz
---> Using cache
---> 47bad5f977b1
Step 3/17 : ARG EN_PACK_URL=https://repo.manticoresearch.com/repository/morphology/en.pak.tgz
---> Using cache
---> 8d37a8b799f0
Step 4/17 : RUN apt-get update
---> Using cache
---> 61a7fb504a52
Step 5/17 : RUN apt-get install -y locales-all locales curl xz-utils manticore-columnar-lib
---> Using cache
---> 5bdbfc630d5b
Step 6/17 : RUN locale-gen ru_RU.UTF-8
---> Using cache
---> c5b4fc5f2973
Step 7/17 : COPY manticore.conf /etc/manticoresearch/manticore.conf
---> Using cache
---> 62911604dd72
Step 8/17 : RUN mkdir -pv /var/lib/manticoresearch
---> Using cache
---> 2cda4ae78be2
Step 9/17 : RUN chmod -R 777 /var/lib/manticoresearch
---> Using cache
---> e3aa5d3fecbc
Step 10/17 : RUN curl -LJO ${RU_PACK_URL}
---> Using cache
---> e2a6ab644e09
Step 11/17 : RUN tar -xvzf ru.pak.tgz -C /usr/share/manticore
---> Using cache
---> 29fb544c7c0e
Step 12/17 : RUN rm ru.pak.tgz
---> Using cache
---> 726b4d2dcf06
Step 13/17 : RUN curl -LJO ${EN_PACK_URL}
---> Using cache
---> 0caec4307a6c
Step 14/17 : RUN tar -xvzf en.pak.tgz -C /usr/share/manticore
---> Using cache
---> 0110a174c015
Step 15/17 : RUN rm en.pak.tgz
---> Using cache
---> abe86d6a656c
Step 16/17 : RUN ln -sf /dev/stdout /var/log/manticore/query.log
---> Using cache
---> 94f0405d7b65
Step 17/17 : CMD ["searchd", "--nodetach", "--config", "/etc/manticoresearch/manticore.conf"]
---> Using cache
---> bdc1b4250ff3
Successfully built bdc1b4250ff3
Successfully tagged tamaranga-manticore:a
up:
snikolaev@dev:~/issue_20/repro$ docker-compose up
Starting tamaranga-manticore ... done
Attaching to tamaranga-manticore
tamaranga-manticore | Manticore 5.0.2 348514c86@220530 dev (columnar 1.15.4 2fef34e@220522) (secondary 1.15.4 2fef34e@220522)
tamaranga-manticore | Copyright (c) 2001-2016, Andrew Aksyonoff
tamaranga-manticore | Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
tamaranga-manticore | Copyright (c) 2017-2022, Manticore Software LTD (https://manticoresearch.com)
tamaranga-manticore |
tamaranga-manticore | [24:12.139] [1] using config file '/etc/manticoresearch/manticore.conf' (454 chars)...
tamaranga-manticore | [24:12.141] [1] Set max_open_files to 65535 (previous was 65535), hardlimit is 65535.
tamaranga-manticore | starting daemon version '5.0.2 348514c86@220530 dev (columnar 1.15.4 2fef34e@220522) (secondary 1.15.4 2fef34e@220522)' ...
tamaranga-manticore | listening on 127.0.0.1:9312 for sphinx and http(s)
tamaranga-manticore | listening on 127.0.0.1:9306 for mysql
tamaranga-manticore | listening on 127.0.0.1:9308 for sphinx and http(s)
tamaranga-manticore | precaching index 't'
tamaranga-manticore | precached 1 indexes in 0.001 sec
tamaranga-manticore | accepting connections
tamaranga-manticore | prereading 1 indexes
tamaranga-manticore | prereaded 1 indexes in 0.000 sec
tamaranga-manticore | [Fri Jul 1 04:24:48.887 2022] 0.000 sec 0.000 sec [ext2/0/ext 0 (0,20)] [t] абв
query:
snikolaev@dev:~/issue_20/repro$ docker-compose exec tamaranga-manticore bash
root@6dbb6590beb8:/var/lib/manticore# mysql -P9306 -h0
mysql> create table t(f text);
ERROR 1064 (42000): index 't': CREATE TABLE failed: index 't' already exists
mysql> select * from t where match('абв');
host's locale:
snikolaev@dev:~/issue_20/repro$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8