poetry env use fails with: ModuleNotFoundError: No module named 'encodings'

[dataspun]

• Poetry version: 1.2.2

• Python version: 3.10.8

• OS version and name: Windows 11

• pyproject.toml: n/a

• [x ] I am on the latest stable Poetry version, installed using a recommended method.

• [ x] I have searched the issues of this repo and believe that this is not a duplicate.

• [ x] I have consulted the FAQ and blog for any relevant entries or release notes.

• [x ] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Having installed a standard, Python.org Python as well as poetry using official Powershell script, I attempt to use a separate python environment:

poetry env use /full/path/to/python

Here is the output:

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Attempting poetry debug info generates the same error.

[Ericson2314]

I wonder if the problem https://github.com/NixOS/nix/pull/11257 is fixing is at all related.

[Ericson2314]

CC @L-as

[L-as]

I wouldn't think so, that was introduced in 2.24. I can still try to investigate this since I'm so familiar with the build/worker code now.

[roberth]

This may be improved by

• https://github.com/NixOS/nix/pull/11171

However, it should never hang for 2 hours. Do we have a timeout that applies during the handling of the connection? Something akin to max-silent-time but for libcurl?

[nebez]

I have a retry mechanism on my side during my build phase, but unfortunately i'm just realizing now that it's silent. I can't be 100% sure that the connection isn't timing out something like ~30 minutes and then trying the nix-env command again.

I'll add some more verbosity to it and report back, as well as try the fix applied in #11171.

[nebez]

Just tested 2.24.2 which includes https://github.com/NixOS/nix/commit/6e3bba5e2686e9e7fbdf6dc1dd4e9f2d7f3c561e – result is the same.

Takes 226.3s to run the nix-env command that took 34.6s on 2.22.3. Both just ran on the same host machine. output of nix-info:

docker run --rm nixos/nix:2.24.2 nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 6.1.90-99.173.amzn2023.aarch64`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.24.2`
 - channels(root): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

The Dockerfile to test:

FROM nixos/nix:2.24.2-arm64
RUN nix-env --file https://github.com/NixOS/nixpkgs/archive/47b604b07d1e8146d5398b42d3306fdebd343986.tar.gz --install --attr sops

[roberth]

I've changed the title because it is likely to only be slow now, since #11171

[mischavandenburg]

I'm having this issue as well. I'm running dev containers using Nix as my package manager. nix is installed as a dev container feature.

It always gets stuck on this specific package:

copying path '/nix/store/6a7qxqcm33y3lwr1v55hpp5hg7akja92-stdenv-linux' from 'https://cache.nixos.org'...

.devcontainer.json

{
  "image": "mcr.microsoft.com/devcontainers/base:debian",
  "features": {
    "ghcr.io/devcontainers/features/nix:1": {}
  },
}

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-08-12-nix-team-meeting-minutes-168/50561/1

[roberth]

DIscussed during the aforementioned Nix maintainer team meeting:

~Possibly same cause: https://github.com/NixOS/nix/issues/11249,~ but @edolstra could not reproduce.

Could this be libgit2 overhead of the new tarball-cache? In a Docker container we may be dealing with an overhead of many fs overlays, amplifying the overhead (normally on the order of 2× at most?). This might also make reproducing dependent on the layer complexity of the image?

[kjeremy]

I'm seeing this in a docker container running 2.18.5 as well.

[nebez]

I can donate an aws linux machine for you to reproduce this with. it only has docker installed and the issue is consistently reproducible. send me your public key, i'll get you on it.

[mischavandenburg]

Tested just now, and the issue still persists. I tried hacking the dev container feature to bring it back to an older Nix version, but unfortunately that hasn't worked.

[PkmX]

We hit a similar problem with extremely slow builds in docker, which turns to be caused by a huge default OPEN_MAX value in docker:

$ docker run -it --rm alpine:latest getconf OPEN_MAX
1073741816

This causes closeMostFDs() to try to loop over and close 1 billion fds:

https://github.com/NixOS/nix/blob/9d8669b14a402a8fd440fdce0ab3d874319a6984/src/libutil/unix/file-descriptor.cc#L141-L147

For reasons I didn't investigate, the fast path above that iterates over /proc/self/fd on Linux throws an exception inside the docker environment, so this fallback method gets called with a massive iteration count.

Maybe check if running the container with --ulimit nofile=1024:1024 helps?

[nebez]

This does indeed fix it, @PkmX ! Thank you!

Using this Dockerfile:

FROM nixos/nix:2.23.0-arm64
RUN nix-env --file https://github.com/NixOS/nixpkgs/archive/47b604b07d1e8146d5398b42d3306fdebd343986.tar.gz --install --attr sops

Running docker build --ulimit nofile=65534:65534 . finishes in a very reasonable 41.7 seconds. Incredible.