proycon
Dubious ownership of annotations
Our FLAT server goes down from time to time and we are trying to establish the reasons for that.
One such case happened yesterday evening.
I looked into the logs of the docker (sudo docker logs -n 1000 flat).
An annotator was working on FLAT, then she seems to have logged out and logged back in. Then this error occurred:
fatal: detected dubious ownership in repository at '/data/annotations'
To add an exception for this directory, call:
git config --global --add safe.directory /data/annotations
This is likely due to our versioning of the annotations via an external git repository. It seems that the error is not that problematic as discussed here.
But I checked the access permissions in our annotation directory and they do not look very homogeneous:
parseme@parseme:~/annotations$ cd ..
parseme@parseme:~$ cd annotations/
parseme@parseme:~/annotations$ ls -lia
total 1752
5505187 drwxrwsr-x 351 parseme parseme 20480 nov. 13 18:49 .
5505059 drwxr-x--- 9 parseme parseme 4096 août 7 11:43 ..
5637212 drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 Abbas
5636773 drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 abdelati.hawwari
5637259 drwxr-xr-x 3 parseme parseme 4096 janv. 10 2024 abigail.walsh
5637425 drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 adela.tocaru
5767548 drwxr-sr-x 2 root parseme 4096 juil. 10 2024 adelina.cerpja
5767563 drwxr-sr-x 2 root parseme 4096 juil. 10 2024 adina.duca
5505488 drwxr-xr-x 4 parseme parseme 4096 janv. 10 2024 agata.savary
5637082 drwxr-xr-x 16 parseme parseme 4096 oct. 8 10:26 agata.savary.annotator
5506634 drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 agata.savary.test
5767594 drwxr-sr-x 2 root parseme 4096 juil. 5 2024 agata.savary.unidive
5636803 drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 aggelfoto123
5767576 drwxr-sr-x 2 root parseme 4096 juil. 17 10:38 agute.klints
5636807 drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 ainara.estarrona
5637002 drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 aixiu.an.zh
...
parseme@parseme:~/annotations$ ls -l ./yalda.yarandi/final
total 22000
-rw-r--r-- 1 root root 2373753 août 12 16:01 dev.udpipe-2.10-xpos-to-deprel.folia.xml
-rw-r--r-- 1 root root 2001046 août 12 16:01 test.udpipe-2.10-xpos-to-deprel.folia.xml
-rw-r--r-- 1 root root 12274118 août 12 16:01 train.udpipe-2.10-xpos-to-deprel.folia.xml
-rw-r--r-- 1 root root 1073626 août 12 16:01 tree_bank_without_VMWE.folia.xml
-rw-r--r-- 1 root root 4792400 août 12 16:01 tree_bank_with_VMWE.folia.xml
parseme@parseme:~/annotations$ ls -l ./jaka.cibej/
total 328
-rw-r--r-- 1 parseme parseme 333884 janv. 10 2024 parseme_sl_ssj500k_13412_13511_noIDs.parsemetsv.folia.xml
Some directories have root as owner, another have parseme. In the former case the permissions are drwxr-sr-x, in the latter they are drwxr-xr-x.
Similarly, some files have root as owner, some others have parseme.
Given that users are always added via DJANGO interface, what causes the difference? And are the correct owners and permissions? Could this be a reason for the unstability of the server?
Such inconsistent ownership occurs when the git repository is shared amongst multiple (unix) users (in this case root and parseme). Note that this does not refer to FLAT users but unix users.
Inside the FLAT container foliadocserve runs under UID/GID 100. How this maps
on the actual host depends a bit on your configuration. You can check this on the host
with ps aux | grep foliadocserve.
Ownership can get messed up if the FLAT container wasn't always consistently started in the same way but as different users, or if the git repository at the document root is pulled by some external process rather than foliadocserve and that runs under another user. If you do a git pull, take care to use the same user foliadocserve runs under.
You'll want to reset the ownership to make it consistent again using chown -R on the document root, setting it to the UID/GID foliadocserve runs under (which I guess is
parseme in your case, but make sure to check). Running as root, even within the
container, is not recommended.
This definitely can be a cause of errors, but that would be permission denied errors where foliadocserve can't write a file to disk. Actually I'm thinking back and it's quite likely this is the cause of https://github.com/proycon/flat/issues/191 .
Could this be a reason for the unstability of the server?
If you say the FLAT server goes down from time to time do you really mean the actual physical server goes down?? Or just FLAT/foliadocserve process. In any case, none of those should really occur because of this.
Such inconsistent ownership occurs when the git repository is shared amongst multiple (unix) users (in this case root and parseme). Note that this does not refer to FLAT users but unix users.
Right.
Inside the FLAT container foliadocserve runs under UID/GID 100. How this maps on the actual host depends a bit on your configuration. You can check this on the host with
ps aux | grep foliadocserve.
I'm not sure how to find it from this command?
parseme@parseme:~$ ps aux | grep foliadocserve
root 3849 0.0 0.0 820 0 ? Ss janv.14 0:00 runsv foliadocserve
root 4119 0.0 0.0 2224 516 ? S janv.14 0:01 tee /data/BKP/2025-01-14_13:19:47/foliadocserve.stdout
root 4120 0.0 0.0 2224 544 ? S janv.14 0:00 tee /data/BKP/2025-01-14_13:19:47/foliadocserve.stderr
root 4121 1.3 0.8 178152 134876 ? Sl janv.14 141:18 /usr/bin/python3 /usr/bin/foliadocserve -d /data/annotations --log /data/BKP/2025-01-14_13:19:47/foliadocserve.log -p 8080 --git
parseme 4144594 0.0 0.0 6612 2424 pts/0 S+ 13:57 0:00 grep --color=auto foliadocserve
Note that the parseme user has an UID which does not appear above:
parseme@parseme:~$ id -u parseme
1004
Ownership can get messed up if the FLAT container wasn't always consistently started in the same way but as different users, or if the git repository at the document root is pulled by some external process rather than foliadocserve and that runs under another user. If you do a git pull, take care to use the same user foliadocserve runs under.
Sorry, it is not quite clear to me. We indeed use git to version the annotations folder:
parseme@parseme:~$ cd annotations/
parseme@parseme:~/annotations$ git config --get remote.origin.url
[email protected]:parseme/annotations.git
I think it is launched by cron, since in the logs I see:
sudo grep CRON /var/log/syslog
...
Jan 23 11:00:01 parseme CRON[3871233]: (parseme) CMD (/home/parseme/annotations/countMWEs.py \ >/home/parseme/annotations/.mwe-count.json)
Jan 23 11:00:01 parseme CRON[3871234]: (tuanbui) CMD (/home/parseme/parseme/updateGitlabWithAnnotations.sh)
Jan 23 11:00:01 parseme CRON[3871235]: (parseme) CMD \ (/home/parseme/annotations/.settingsCommitHack/updateGitlabWithSettings.sh)
Jan 23 11:00:01 parseme CRON[3871226]: (CRON) info (No MTA installed, discarding output)
Jan 23 11:00:01 parseme CRON[3871238]: (parseme) CMD (/home/parseme/annotations/updateGitlabWithAnnotations.sh)
And here is what the script contains:
parseme@parseme:~/annotations$ cat updateGitlabWithAnnotations.sh
#! /bin/sh
export LANGUAGE=C
export LANG=C
export LC_ALL=C
HERE="$(cd "$(dirname "$0")" && pwd)"
DATE="$(date --rfc-3339=s)"
set -o nounset # Using "$UNDEF" var raises error
set -o errexit # Exit on error, do not continue quietly
cd "$HERE"
exec 1>.git-commit-log.stdout
exec 2>.git-commit-log.stderr
echo "$DATE" >&2
git pull
modified_userdirs="$(git status | grep modified: | awk '{print $2}' | grep '/' | sed 's@/.*@@g' | sort -u | awk 'BEGIN{ORS=" "} 1' | sed 's@ *$@@g')"
git add * # commit all files in $HERE that do not start with "."
git commit -am "Auto-commit: $DATE" -m "Modified: [$modified_userdirs]" || true
git push
How do I know out of this which user launches git pull and git push?
In the logs it looks like it is tuanbui, right (line 3871234 above)? Which is probably not what is expected.
But even so, the folders and files in annotations\ belong either to parseme or to root, not to tuanbui.
You'll want to reset the ownership to make it consistent again using
chown -Ron the document root, setting it to the UID/GID foliadocserve runs under (which I guess is parseme in your case, but make sure to check). Running as root, even within the container, is not recommended.
I understand, but again I'm not quite sure. The docker image was set up by a colleague who moved somewhere else. He left a good documentation though and there I see that to restart the server I have to run:
sudo docker ps
Does it mean that the annotation directories and files will have root as owner (I don't think so).
This definitely can be a cause of errors, but that would be permission denied errors where foliadocserve can't write a file to disk. Actually I'm thinking back and it's quite likely this is the cause of #191
OK, so I have now given ownership to parseme to all the folders and files in annotations? parseme@parseme:~/annotations$ sudo chown -R parseme:parseme * ... parseme@parseme:~/annotations$ ll | head -10 total 1776 drwxrwsr-x 357 parseme parseme 20480 janv. 14 22:32 ./ drwxr-x--- 9 parseme parseme 4096 janv. 14 15:49 ../ drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 Abbas/ drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 abdelati.hawwari/ drwxr-xr-x 3 parseme parseme 4096 janv. 10 2024 abigail.walsh/ drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 adela.tocaru/ drwxr-sr-x 2 parseme parseme 4096 juil. 10 2024 adelina.cerpja/ drwxr-sr-x 2 parseme parseme 4096 juil. 10 2024 adina.duca/ drwxr-xr-x 4 parseme parseme 4096 janv. 10 2024 agata.savary/`
And what would be the right permissions? Note that currently we still have drwxr-xr-x and drwxr-sr-x in folders. And for files we have:
parseme@parseme:~/annotations$ ll */* | cut -d' ' -f1 | grep -E '^\-' | sort -u
-rw-------
-rw-r--r--
-rw-rw-r--
What does FLAT require?
Could this be a reason for the unstability of the server?
If you say the FLAT server goes down from time to time do you really mean the actual physical server goes down?? Or just FLAT/foliadocserve process. In any case, none of those should really occur because of this.
I'm actually not quite sure. Maybe we also have issues with the physical server from time to time. I will go on following this issue.
Thank you very much, this is enlightening.
I'm not sure how to find it from this command?
@.***:~$ ps aux | grep foliadocserve
root 3849 0.0 0.0 820 0 ? Ss janv.14 0:00 runsv foliadocserveroot 4119 0.0 0.0 2224 516 ? S janv.14 0:01 tee /data/BKP/2025-01-14_13:19:47/foliadocserve.stdoutroot 4120 0.0 0.0 2224 544 ? S janv.14 0:00 tee /data/BKP/2025-01-14_13:19:47/foliadocserve.stderrroot 4121 1.3 0.8 178152 134876 ? Sl janv.14 141:18 /usr/bin/python3 /usr/bin/foliadocserve -d /data/annotations --log /data/BKP/2025-01-14_13:19:47/foliadocserve.log -p 8080 --gitparseme 4144594 0.0 0.0 6612 2424 pts/0 S+ 13:57 0:00 grep --color=auto foliadocserve`
The last one above the grep is the one I meant. It shows foliadocserve running as root, which is definitely not recommended.
Ownership can get messed up if the FLAT container wasn't always consistently started in the same way but as different users, or if the git repository at the document root is pulled by some external process rather than foliadocserve and that runs under another user. If you do a git pull, take care to use the same user foliadocserve runs under.
Sorry, it is not quite clear to me. We indeed use git to version the annotations folder:
@.:~$ cd annotations/
***@***.***:~/annotations$ git config --get remote.origin.url@.:parseme/annotations.git`I think it is launched by cron, since in the logs I see:
sudo grep CRON /var/log/syslog...Jan 23 11:00:01 parseme CRON[3871233]: (parseme) CMD (/home/parseme/annotations/countMWEs.py \ >/home/parseme/annotations/.mwe-count.json)Jan 23 11:00:01 parseme CRON[3871234]: (tuanbui) CMD (/home/parseme/parseme/updateGitlabWithAnnotations.sh)Jan 23 11:00:01 parseme CRON[3871235]: (parseme) CMD \ (/home/parseme/annotations/.settingsCommitHack/updateGitlabWithSettings.sh)Jan 23 11:00:01 parseme CRON[3871226]: (CRON) info (No MTA installed, discarding output)Jan 23 11:00:01 parseme CRON[3871238]: (parseme) CMD (/home/parseme/annotations/updateGitlabWithAnnotations.sh)
Yes, indeed
And here is what the script contains:
@.***:~/annotations$ cat updateGitlabWithAnnotations.sh
#! /bin/shexport LANGUAGE=Cexport LANG=Cexport LC_ALL=CHERE="$(cd "$(dirname "$0")" && pwd)"DATE="$(date --rfc-3339=s)"`
set -o nounset # Using "$UNDEF" var raises errorset -o errexit # Exit on error, do not continue quietly
cd "$HERE"exec 1>.git-commit-log.stdoutexec 2>.git-commit-log.stderr
echo "$DATE" >&2git pullmodified_userdirs="$(git status | grep modified: | awk '{print $2}' | grep '/' | sed 's@/.*@@g' | sort -u | awk 'BEGIN{ORS=" "} 1' | sed 's@ *$@@g')"git add * # commit all files in $HERE that do not start with "."git commit -am "Auto-commit: $DATE" -m "Modified: [$modified_userdirs]" || truegit pushHow do I know out of this which user launches
git pullandgit push?
Your cron log showed that, there were two, one run by parseme and one tuanbui.
In the logs it looks like it is
tuanbui, right (line 3871234 above)? Which is probably not what is expected.
Yes, the one in /home/parseme/annotations is run by 'parseme', which looks good. tuanbui runs on a different subdirectory.
But even so, the folders and files in
annotations\belong either toparsemeor toroot, not totuanbui.
Yes, the ones by root are caused by foliadocserve.
You'll want to reset the ownership to make it consistent again using
chown -Ron the document root, setting it to the UID/GID foliadocserve runs under (which I guess is parseme in your case, but make sure to check). Running as root, even within the container, is not recommended.I understand, but again I'm not quite sure. The docker image was set up by a colleague who moved somewhere else. He left a good documentation though and there I see that to restart the server I have to run:
sudo docker psDoes it mean that the annotation directories and files will haverootas owner (I don't think so).
docker ps will only show the running containers, not restart any, but
that suggests to me that the container is likely also started via sudo docker run and runs as root.
I'm not entirely sure why foliadocservice is not running as UID 100 but as 0 (root). But this is likely a problem in the docker setup on your server. One if the most cumbersome things of docker management is mapping UIDs between host and containers and running rootless containers. You can find the relevant documentation here:
- https://docs.docker.com/engine/security/userns-remap/
- https://docs.docker.com/engine/security/rootless/
The most secure way is to setup the parseme user to run docker containers without requiring root privileges. The UID inside the container can be controlled by the UWSGI_UID environment variable you pass into the container (I just committed one extra fix for that and republished the docker image). The UID on the host should map to a UID in the container (a so-called subuid). If you have a system administrator around with docker experience, he/she might be able to help navigate this maze.
OK, so I have now given ownership to
parsemeto all the folders and files inannotations? ***@***.***:~/annotations$ sudo chown -R parseme:parseme *...@.***:~/annotations$ ll | head -10total 1776drwxrwsr-x 357 parseme parseme 20480 janv. 14 22:32 ./drwxr-x--- 9 parseme parseme 4096 janv. 14 15:49 ../drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 Abbas/drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 abdelati.hawwari/drwxr-xr-x 3 parseme parseme 4096 janv. 10 2024 abigail.walsh/drwxr-xr-x 2 parseme parseme 4096 janv. 10 2024 adela.tocaru/drwxr-sr-x 2 parseme parseme 4096 juil. 10 2024 adelina.cerpja/drwxr-sr-x 2 parseme parseme 4096 juil. 10 2024 adina.duca/drwxr-xr-x 4 parseme parseme 4096 janv. 10 2024 agata.savary/`And what would be the right permissions? Note that currently we still have
drwxr-xr-xanddrwxr-sr-xin folders. And for files we have: @.***:~/annotations$ ll / | cut -d' ' -f1 | grep -E '^-' | sort -u-rw--------rw-r--r---rw-rw-r--` What does FLAT require?
Things should be readable and writable for the user foliadocserve runs under (which is now root, so that works with the current permissions, but is a bad idea).
Hi @proycon Martin. It took us some time but we did delve into the two links you sent above. Obada (@obadas) from my IT team tried to install a test FLAT server in rootless mode, following your link. The installations went well but he didn’t succeed in launching the server. The error was 500 and the logs show below that the connection with the database fails. When the rootless mode is disabled, all seems to work fine.:
Traceback (most recent call last):
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection self.connect()
File "/usr/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/mysql/base.py", line 247, in get_new_connection
connection = Database.connect(**conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/MySQLdb/__init__.py", line 121, in Connect
return Connection(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/MySQLdb/connections.py", line 200, in __init__
super().__init__(*args, **kwargs2)
MySQLdb.OperationalError: (2002, "Can't connect to local server through socket '/run/mysqld/mysqld.sock' (111)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapper_view
if test_func(request.user):
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/contrib/auth/decorators.py", line 51, in <lambda>
lambda u: u.is_authenticated,
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/utils/functional.py", line 266, in inner self._setup()
File "/usr/lib/python3.12/site-packages/django/utils/functional.py", line 419, in _setup
self._wrapped = self._setupfunc()
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/contrib/auth/middleware.py", line 25, in <lambda>
request.user = SimpleLazyObject(lambda: get_user(request))
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/contrib/auth/middleware.py", line 11, in get_user
request._cached_user = auth.get_user(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/contrib/auth/__init__.py", line 191, in get_user
user_id = _get_user_session_key(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/contrib/auth/__init__.py", line 60, in _get_user_session_key
return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/contrib/sessions/backends/base.py", line 53, in __getitem__
return self._session[key]
^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/contrib/sessions/backends/base.py", line 192, in _get_session
self._session_cache = self.load()
^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/contrib/sessions/backends/db.py", line 42, in load
s = self._get_session_from_db()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/contrib/sessions/backends/db.py", line 32, in _get_session_from_db
return self.model.objects.get(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/models/query.py", line 633, in get
num = len(clone)
^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/models/query.py", line 380, in __len__
self._fetch_all()
File "/usr/lib/python3.12/site-packages/django/db/models/query.py", line 1881, in _fetch_all
self._result_cache = list(self._iterable_class(self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/models/query.py", line 91, in __iter__
results = compiler.execute_sql(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1560, in execute_sql
cursor = self.connection.cursor()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 330, in cursor
return self._cursor()
^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 306, in _cursor
self.ensure_connection()
File "/usr/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
with self.wrap_database_errors:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/usr/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/mysql/base.py", line 247, in get_new_connection
connection = Database.connect(**conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/MySQLdb/__init__.py", line 121, in Connect
return Connection(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/MySQLdb/connections.py", line 200, in __init__
super().__init__(*args, **kwargs2)
django.db.utils.OperationalError: (2002, "Can't connect to local server through socket '/run/mysqld/mysqld.sock' (111)")
As to usermap, @ObadaS followed your second link to launch the server as parseme user. Here, we face the problem of unauthorised access to files in the /home/parseme/BKP folder. Here are out logs:
Starting foliadocserve...
mkdir: cannot create directory ‘/data’: Permission denied
chown: cannot access '/data/annotations': Permission denied
chmod: cannot access '/data/annotations': Permission denied
rm: cannot remove '/data/BKP/latest': Permission denied
mkdir: cannot create directory ‘/data/BKP/2025-02-26_10:27:42’: Permission denied
./run: cd: line 24: can't cd to /data/BKP/2025-02-26_10:27:42: Permission denied
coreutils: failed to create symbolic link '../latest': File exists
tee: '/data/BKP/2025-02-26_10:27:42/foliadocserve.stdout': Permission denied
tee: '/data/BKP/2025-02-26_10:27:42/foliadocserve.stderr': Permission denied
Traceback (most recent call last):
File "/usr/bin/foliadocserve", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/lib/python3.12/site-packages/foliadocserve/foliadocserve.py", line 1033, in main
logfile = open(args.logfile,'a',encoding='utf-8')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/data/BKP/2025-02-26_10:27:42/foliadocserve.log'
In each case we used the latest version of the docker image.
Any further hints would be most welcome. Many thanks!
On Fri Feb 28, 2025 at 10:02 AM CET, Agata Savary wrote:
savary left a comment (proycon/flat#193)
Hi @proycon Martin. It took us some time but we did delve into the two links you sent above. Obada @.***) from my IT team tried to install a test FLAT server in rootless mode, following your link. The installations went well but he didn’t succeed in launching the server. The error was 500 and the logs show below that the connection with the database fails. When the rootless mode is disabled, all seems to work fine.:
django.db.utils.OperationalError: (2002, "Can't connect to local server through socket '/run/mysqld/mysqld.sock' (111)")
What did you set the environment variables FLAT_DATABASE, FLAT_DATABASE_ENGINE and FLAT_DATABASE_HOST to? The logs say it's trying to access a local database via a socket, so that means the db is local to the container, but there is no MySQL in the FLAT container. Are you sure the configuration is the same between the rootless and root versions you've been trying? Make sure your MySQl/Mariadb server is accessible via TCP from the container.
As to usermap, @ObadaS followed your second link to launch the server as
parsemeuser. Here, we face the problem of unauthorised access to files in the /home/parseme/BKP folder. Here are out logs:
Starting foliadocserve...mkdir: cannot create directory ‘/data’: Permission deniedchown: cannot access '/data/annotations': Permission deniedchmod: cannot access '/data/annotations': Permission deniedrm: cannot remove '/data/BKP/latest': Permission deniedmkdir: cannot create directory ‘/data/BKP/2025-02-26_10:27:42’: Permission denied./run: cd: line 24: can't cd to /data/BKP/2025-02-26_10:27:42: Permission deniedcoreutils: failed to create symbolic link '../latest': File existstee: '/data/BKP/2025-02-26_10:27:42/foliadocserve.stdout': Permission deniedtee: '/data/BKP/2025-02-26_10:27:42/foliadocserve.stderr': Permission deniedTraceback (most recent call last):File "/usr/bin/foliadocserve", line 8, in <module>sys.exit(main())^^^^^^File "/usr/lib/python3.12/site-packages/foliadocserve/foliadocserve.py", line 1033, in mainlogfile = open(args.logfile,'a',encoding='utf-8')^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^PermissionError: [Errno 13] Permission denied: '/data/BKP/2025-02-26_10:27:42/foliadocserve.log'
That indicates that the mapping wasn't correct. I can't see from the log what UIDs/GIDs are used on both ends. What you can do is, when the container is running, try to see what UID the foliadocserve process is using. If you chown the files to that UID they should be readable/writable from the container.
I hope this helps a bit, UID mappings can indeed be complicated.
Hello,
What did you set the environment variables FLAT_DATABASE, FLAT_DATABASE_ENGINE and FLAT_DATABASE_HOST to? The logs say it's trying to access a local database via a socket, so that means the db is local to the container, but there is no MySQL in the FLAT container. Are you sure the configuration is the same between the rootless and root versions you've been trying? Make sure your MySQl/Mariadb server is accessible via TCP from the container.
For the tests I made, I copied all the files from the production server so that I can have the closest possible configuration. Since it works on the production server, the TCP connection should also work correctly, especially since it works fine when I disable rootless Docker.
That indicates that the mapping wasn't correct. I can't see from the log what UIDs/GIDs are used on both ends. What you can do is, when the container is running, try to see what UID the foliadocserve process is using. If you chown the files to that UID they should be readable/writable from the container.
I hope this helps a bit, UID mappings can indeed be complicated.
I used the guide to remap docker to be used by parseme, but foliadocserve is still being run by root. From what I understand from the previous comments, foliadocserve should be launched by uwsgi instead of root ? I am not sure why it's not working as intended on our end.
On Mon Mar 3, 2025 at 2:35 PM CET, Obada Haddad-Soussac wrote:
For the tests I made, I copied all the files from the production server so that I can have the closest possible configuration.
Yes, that's a good approach indeed.
Since it works on the production server, the TCP connection should also work correctly, especially since it works fine when I disable rootless Docker.
It's a bit hard for me to judge your configuration from here. It's indeed strange that the database connection is a problem when switching between rootless and non-rootless.
That indicates that the mapping wasn't correct. I can't see from the log what UIDs/GIDs are used on both ends. What you can do is, when the container is running, try to see what UID the foliadocserve process is using. If you chown the files to that UID they should be readable/writable from the container.
I hope this helps a bit, UID mappings can indeed be complicated.
I used the guide to remap docker to be used by parseme, but foliadocserve is still being run by root. From what I understand from the previous comments, foliadocserve should be launched by uwsgi instead of root ? I am not sure why it's not working as intended on our end.
The foliadocserve python process should run as a a non-root user yes (both from the host perspective as from the container perspective). If it runs as root then something is wrong with the rootless container setup/user map. Foliadocserve is not launched by uwsgi technically, FLAT is, but both run under the same UID though (configurable via environment variable UWSGI_UID).
Here's an example of how the process list can look from the host's perspective, but note that I always use podman instead of docker (podman was designed a bit better with rootless containers in mind, and is mostly docker compatibly):
$ ps aux | grep flat
proycon 328300 0.0 0.0 2672192 44220 pts/4 Sl+ 17:38 0:00 /usr/bin/podman run -t -i -p 8080:80 --env FLAT_DEBUG=1 proycon/flat:dev
proycon 328317 0.0 0.1 3328124 69400 pts/4 Sl+ 17:38 0:00 /usr/bin/podman run -t -i -p 8080:80 --env FLAT_DEBUG=1 proycon/flat:dev
proycon 328389 0.0 0.0 1776 1160 ? S 17:38 0:00 sudo -u nginx foliadocserve -d /data/flat.docroot --log /dev/stdout --git --gitshare false --expirationtime 120 -p 8080
100099 328390 0.2 0.0 76332 37728 ? Sl 17:38 0:01 /usr/bin/python3 /usr/bin/foliadocserve -d /data/flat.docroot --log /dev/stdout --git --gitshare false --expirationtime 120 -p 8080
100099 328451 0.0 0.0 51328 40392 ? S 17:38 0:00 uwsgi --plugin python3 --uid 100 --gid 100 --master --socket 127.0.0.1:8888 --wsgi-file /etc/flat.wsgi --processes 2 --single-interpreter --threads 2 --manage-script-name
100099 328458 0.0 0.0 51484 33772 ? Sl 17:38 0:00 uwsgi --plugin python3 --uid 100 --gid 100 --master --socket 127.0.0.1:8888 --wsgi-file /etc/flat.wsgi --processes 2 --single-interpreter --threads 2 --manage-script-name
100099 328460 0.0 0.0 51484 33772 ? Sl 17:38 0:00 uwsgi --plugin python3 --uid 100 --gid 100 --master --socket 127.0.0.1:8888 --wsgi-file /etc/flat.wsgi --processes 2 --single-interpreter --threads 2 --manage-script-name
In /etc/subuid I have:
proycon:100000:65536
UID 100 in the container then maps to 100000 + 100 - 1 on the host.
I have some updates on this issue. I'm using the docker usermap service. I managed to make foliadocserve run using the user that I want (I changed /etc/subuid so that Root in the container is mapped to the parseme account on the host). Which means that foliadocserve now runs as root in the container which is fine for the host too.
However, when I try to launch the server, it doesn't successfully get past these steps :
Creating FLAT superuser and (re)setting password...
Updating existing password for superuser.
Starting FLAT via uwsgi...
*** Starting uWSGI 2.0.27 (64bit) on [Tue Mar 25 09:19:14 2025] ***
compiled with version: 14.2.0 on 22 October 2024 13:39:35
os: Linux-5.10.0-20-cloud-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13)
nodename: a1dd36fec89e
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 6
current working directory: /etc/service/uwsgi
detected binary path: /usr/sbin/uwsgi
group 0 not found.
Applying migrations...
System check identified some issues:
WARNINGS:
metadata.MetadataIndex: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
users.ReadPermissions: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
users.WritePermissions: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
Operations to perform:
Synchronize unmigrated apps: editor, messages, metadata, staticfiles, structureeditor, viewer
Apply all migrations: admin, auth, contenttypes, sessions, sites, users
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Running migrations:
No migrations to apply.
It keeps repeating this over and over again.
I tried adding --env FLAT_DEBUG=1 but it changed nothing in the logs.
I'm not sure if this is a permission problem again or if it's something else entirely.
On Tue Mar 25, 2025 at 11:14 AM CET, Obada Haddad-Soussac wrote:
I have some updates on this issue. I'm using the docker usermap service. I managed to make foliadocserve run using the user that I want (I changed /etc/subuid so that Root in the container is mapped to the parseme account on the host). Which means that foliadocserve now runs as root in the container which is fine for the host too.
Great progress, that sounds good.
However, when I try to launch the server, it doesn't successfully get past these steps :
Creating FLAT superuser and (re)setting password... Updating existing password for superuser. Starting FLAT via uwsgi... *** Starting uWSGI 2.0.27 (64bit) on [Tue Mar 25 09:19:14 2025] *** compiled with version: 14.2.0 on 22 October 2024 13:39:35 os: Linux-5.10.0-20-cloud-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) nodename: a1dd36fec89e machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 6 current working directory: /etc/service/uwsgi detected binary path: /usr/sbin/uwsgi group 0 not found. Applying migrations... System check identified some issues: WARNINGS: metadata.MetadataIndex: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'. HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'. users.ReadPermissions: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'. HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'. users.WritePermissions: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'. HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
Those warnings are okay, you can ignore them...
Operations to perform: Synchronize unmigrated apps: editor, messages, metadata, staticfiles, structureeditor, viewer Apply all migrations: admin, auth, contenttypes, sessions, sites, users Synchronizing apps without migrations: Creating tables... Running deferred SQL... Running migrations: No migrations to apply.
This looks all good...
It keeps repeating this over and over again. I tried adding
--env FLAT_DEBUG=1but it changed nothing in the logs.
If you get this over and over again then uwsgi/flat is being restarted over and over again (because it keeps failing). There really aren't any more cues in the logs? Nothing else before it restarts?
I'm guessing (but not sure) it still has something to do with the UID and the docker setup. Maybe it has trouble switching to the uwsgi UID/GID 100 which is used by default. Try try setting UWSGI_UID and UWSGI_GID to 0 (root) in the environment variables passed to the containers and see if that helps (not ideal but okay). If it's not that then I wonder if it somehow fails to open a socket for 127.0.0.1:8888 due to the docker setup (is the loopback network device available inside the container?).
There really aren't any more cues in the logs? Nothing else before it restarts? It complains about not being able to connect to the local database for a bit, but it eventually connects and proceeds. Here are the logs of it:
Starting FLAT via uwsgi...
*** Starting uWSGI 2.0.27 (64bit) on [Thu Mar 27 13:23:38 2025] ***
compiled with version: 14.2.0 on 22 October 2024 13:39:35
os: Linux-5.10.0-20-cloud-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13)
nodename: a0deb07eced8
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 6
current working directory: /etc/service/uwsgi
detected binary path: /usr/sbin/uwsgi
group 0 not found.
Applying migrations...
Traceback (most recent call last):
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/usr/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/mysql/base.py", line 247, in get_new_connection
connection = Database.connect(**conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/MySQLdb/__init__.py", line 121, in Connect
return Connection(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/MySQLdb/connections.py", line 200, in __init__
super().__init__(*args, **kwargs2)
MySQLdb.OperationalError: (2002, "Can't connect to local server through socket '/run/mysqld/mysqld.sock' (111)")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/bin/django-admin", line 8, in <module>
sys.exit(execute_from_command_line())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
utility.execute()
File "/usr/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python3.12/site-packages/django/core/management/base.py", line 412, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/lib/python3.12/site-packages/django/core/management/base.py", line 458, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/core/management/base.py", line 106, in wrapper
res = handle_func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/core/management/commands/migrate.py", line 100, in handle
self.check(databases=[database])
File "/usr/lib/python3.12/site-packages/django/core/management/base.py", line 485, in check
all_issues = checks.run_checks(
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/core/checks/registry.py", line 88, in run_checks
new_errors = check(app_configs=app_configs, databases=databases)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/core/checks/model_checks.py", line 36, in check_all_models
errors.extend(model.check(**kwargs))
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/models/base.py", line 1558, in check
*cls._check_indexes(databases),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/models/base.py", line 2002, in _check_indexes
connection.features.supports_expression_indexes
File "/usr/lib/python3.12/site-packages/django/utils/functional.py", line 57, in __get__
res = instance.__dict__[self.name] = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/mysql/features.py", line 331, in supports_expression_indexes
not self.connection.mysql_is_mariadb
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/utils/functional.py", line 57, in __get__
res = instance.__dict__[self.name] = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/mysql/base.py", line 439, in mysql_is_mariadb
return "mariadb" in self.mysql_server_info.lower()
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/utils/functional.py", line 57, in __get__
res = instance.__dict__[self.name] = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/mysql/base.py", line 425, in mysql_server_info
return self.mysql_server_data["version"]
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/utils/functional.py", line 57, in __get__
res = instance.__dict__[self.name] = self.func(instance)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/mysql/base.py", line 399, in mysql_server_data
with self.temporary_connection() as cursor:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 705, in temporary_connection
with self.cursor() as cursor:
^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 330, in cursor
return self._cursor()
^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 306, in _cursor
self.ensure_connection()
File "/usr/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 288, in ensure_connection
with self.wrap_database_errors:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/usr/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/django/db/backends/mysql/base.py", line 247, in get_new_connection
connection = Database.connect(**conn_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/MySQLdb/__init__.py", line 121, in Connect
return Connection(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/MySQLdb/connections.py", line 200, in __init__
super().__init__(*args, **kwargs2)
django.db.utils.OperationalError: (2002, "Can't connect to local server through socket '/run/mysqld/mysqld.sock' (111)")
However, after repeating this a few times, it eventually manages to connect to the database and the logs changes a bit. It stops crashing after trying to apply migration and becomes what I shared in my previous comment. It just keeps repeating it over and over again.
Try try setting UWSGI_UID and UWSGI_GID to 0 (root) in the environment variables passed to the containers and see if that helps (not ideal but okay)
This is already how I configured it, it's set up through the Dockerfile to rebuild the images with some other changes. I also made some changes to the /etc/subuid file so that Root inside the container becomes Parseme on the host:
parseme:1004:10000
Which seems to work fine because when I do ps aux | grep folia on the host I see :
And inside the container :
I thought it might be a permission issue somewhere, but wouldn't the logs show a permission denied in that case ?
Thank you for taking the time to help us with this.