salt
salt copied to clipboard
saltstack
[BUG] archive.extracted fails with UnicodeEncodeError when applied from master, works locally
Description
The archive.extracted state fails with UnicodeEncodeError when being applied from master to a minion, but works fine when applied from the minion via calling salt-call.
Setup
I'm observing this behavior with a setup of following components:
- a Debian 11 "bullseye" amd64 VM acting as a salt master (v3005 from onedir installation),
- an Amazon Linux 2 amd64 VM acting as a minion (v3005 from onedir installation).
Steps to Reproduce the behavior
Example SLS:
{% set osarch = salt['grains.get']('osarch') %}
{% set goversion = '1.19.8' %}
{% if osarch in ('x86_64', 'amd64') %}
{% set platform = 'linux-amd64' %}
{% set sha256_hash = 'e1a0bf0ab18c8218805a1003fd702a41e2e807710b770e787e5979d1cf947aba' %}
{% elif osarch in ('aarch64', 'arm64') %}
{% set platform = 'linux-arm64' %}
{% set sha256_hash = 'f89e7c0ba63782143bd1f896e4b96ea09e4baf39e8bc2f2ddf27339f9e433dd3' %}
{% endif %}
Remove Preinstalled Golang Packages:
pkg.removed:
- pkgs:
- golang
GOROOT Cleanup:
cmd.run:
- name: rm -rf /usr/local/go
- unless: if [ -d /usr/local/go ]; then /usr/local/go/bin/go version | grep -- "{{ goversion }}"; fi
Golang {{ goversion }}:
archive.extracted:
- name: /usr/local
- source: https://go.dev/dl/go{{ goversion }}.{{ platform }}.tar.gz
- source_hash: sha256={{ sha256_hash }}
- if_missing: /usr/local/go/bin/go
- trim_output: 10
- require:
- GOROOT Cleanup
{% for binary in ('go', 'gofmt') %}
Golang Symlink For {{ binary }}:
file.symlink:
- name: /usr/bin/{{ binary }}
- target: /usr/local/go/bin/{{ binary }}
- require:
- Remove Preinstalled Golang Packages
- Golang {{ goversion }}
{% endfor %}
Applying the state:
# salt builder-amzn2-amd64 state.apply packages.golang
builder-amzn2-amd64:
----------
ID: Golang 1.19.8
Function: archive.extracted
Name: /usr/local
Result: False
Comment: An exception occurred in this state: Traceback (most recent call last):
File "salt/state.py", line 2276, in call
ret = self.states[cdata["full"]](
File "salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "salt/loader/lazy.py", line 1228, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "salt/loader/lazy.py", line 1243, in _run_as
return _func_or_method(*args, **kwargs)
File "salt/loader/lazy.py", line 1276, in wrapper
return f(*args, **kwargs)
File "/opt/saltstack/salt/run/salt/states/archive.py", line 1228, in extracted
path_mode = os.lstat(full_path.rstrip(os.sep)).st_mode
UnicodeEncodeError: 'ascii' codec can't encode character '\xc4' in position 44: ordinal not in range(128)
Started: 08:49:33.908934
Duration: 6558.326 ms
Changes:
----------
ID: Golang Symlink For go
Function: file.symlink
Name: /usr/bin/go
Result: False
Comment: One or more requisite failed: packages.golang.Golang 1.19.8
Started: 08:49:40.468050
Duration: 0.006 ms
Changes:
----------
ID: Golang Symlink For gofmt
Function: file.symlink
Name: /usr/bin/gofmt
Result: False
Comment: One or more requisite failed: packages.golang.Golang 1.19.8
Started: 08:49:40.468222
Duration: 0.004 ms
Changes:
Summary for builder-amzn2-amd64
------------
Succeeded: 2
Failed: 3
------------
Total states run: 5
Total run time: 7.974 s
ERROR: Minions returned with non-zero exit code
Minion log (I've added simple debug logging full_path after https://github.com/saltstack/salt/blob/b34f84c/salt/states/archive.py#L1225):
2023-04-24 08:49:40,462 [salt.loaded.int.states.archive:1226][DEBUG ][9362] L1227: "/usr/local/go/test/fixedbugs/issue27557.go"
2023-04-24 08:49:40,462 [salt.loaded.int.states.archive:1226][DEBUG ][9362] L1227: "/usr/local/go/test/fixedbugs/issue27595.go"
2023-04-24 08:49:40,462 [salt.loaded.int.states.archive:1226][DEBUG ][9362] L1227: "/usr/local/go/test/fixedbugs/issue27695.go"
2023-04-24 08:49:40,463 [salt.loaded.int.states.archive:1226][DEBUG ][9362] L1227: "/usr/local/go/test/fixedbugs/issue27695b.go"
2023-04-24 08:49:40,463 [salt.loaded.int.states.archive:1226][DEBUG ][9362] L1227: "/usr/local/go/test/fixedbugs/issue27695c.go"
2023-04-24 08:49:40,463 [salt.loaded.int.states.archive:1226][DEBUG ][9362] L1227: "/usr/local/go/test/fixedbugs/issue27718.go"
2023-04-24 08:49:40,463 [salt.loaded.int.states.archive:1226][DEBUG ][9362] L1227: "/usr/local/go/test/fixedbugs/issue27732a.go"
2023-04-24 08:49:40,463 [salt.loaded.int.states.archive:1226][DEBUG ][9362] L1227: "/usr/local/go/test/fixedbugs/issue27829.go"
2023-04-24 08:49:40,463 [salt.loaded.int.states.archive:1226][DEBUG ][9362] L1227: "/usr/local/go/test/fixedbugs/issue27836.dir/Äfoo.go"
2023-04-24 08:49:40,463 [salt.state :2286][DEBUG ][9362] An exception occurred in this state: 'ascii' codec can't encode character '\xc4' in position 44: ordinal not in range(128)
Traceback (most recent call last):
File "salt/state.py", line 2276, in call
ret = self.states[cdata["full"]](
File "salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "salt/loader/lazy.py", line 1228, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "salt/loader/lazy.py", line 1243, in _run_as
return _func_or_method(*args, **kwargs)
File "salt/loader/lazy.py", line 1276, in wrapper
return f(*args, **kwargs)
File "/opt/saltstack/salt/run/salt/states/archive.py", line 1228, in extracted
path_mode = os.lstat(full_path.rstrip(os.sep)).st_mode
UnicodeEncodeError: 'ascii' codec can't encode character '\xc4' in position 44: ordinal not in range(128)
2023-04-24 08:49:40,467 [salt.state :321 ][ERROR ][9362] An exception occurred in this state: Traceback (most recent call last):
File "salt/state.py", line 2276, in call
ret = self.states[cdata["full"]](
File "salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "salt/loader/lazy.py", line 1228, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "salt/loader/lazy.py", line 1243, in _run_as
return _func_or_method(*args, **kwargs)
File "salt/loader/lazy.py", line 1276, in wrapper
return f(*args, **kwargs)
File "/opt/saltstack/salt/run/salt/states/archive.py", line 1228, in extracted
path_mode = os.lstat(full_path.rstrip(os.sep)).st_mode
UnicodeEncodeError: 'ascii' codec can't encode character '\xc4' in position 44: ordinal not in range(128)
2023-04-24 08:49:40,467 [salt.state :2345][INFO ][9362] Completed state [/usr/local] at time 08:49:40.467260 (duration_in_ms=6558.326)
2023-04-24 08:49:40,468 [salt.state :3401][DEBUG ][9362] File /var/cache/salt/minion/accumulator/139796154265216 does not exist, no need to cleanup
2023-04-24 08:49:40,468 [salt.channel.client:313 ][DEBUG ][9362] Closing AsyncReqChannel instance
2023-04-24 08:49:40,471 [salt.loader.lazy :960 ][DEBUG ][9362] The functions from module 'state' are being loaded by dir() on the loaded module
2023-04-24 08:49:40,471 [salt.utils.lazy :99 ][DEBUG ][9362] LazyLoaded state.check_result
2023-04-24 08:49:40,472 [salt.minion :2192][INFO ][9362] Returning information for job: 20230424084928348103
Expected behavior
State applied without errors.
Versions Report
master - salt --versions-report
# salt --versions-report
Salt Version:
Salt: 3005.1
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.0
libgit2: 1.5.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: 1.11.1
Python: 3.9.16 (main, Jan 6 2023, 22:49:58)
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 23.2.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: debian 11 bullseye
locale: utf-8
machine: x86_64
release: 5.10.0-21-cloud-amd64
system: Linux
version: Debian GNU/Linux 11 bullseye
master - packages details
# dpkg -s salt-common salt-master
Package: salt-common
Status: install ok installed
Priority: optional
Section: admin
Installed-Size: 192667
Maintainer: Debian Salt Team <[email protected]>
Architecture: amd64
Source: salt
Version: 3005.1+ds-4
Recommends: lsb-release
Suggests: ifupdown
Conffiles:
/etc/logrotate.d/salt-common 928be90e45382067b08c53a702189e55
Description: shared libraries that salt requires for all packages
salt is a powerful remote execution manager that can be used to
administer servers in a fast and efficient way.
.
It allows commands to be executed across large groups of
servers. This means systems can be easily managed, but data can
also be easily gathered. Quick introspection into running
systems becomes a reality.
.
Remote execution is usually used to set up a certain state on a
remote system. Salt addresses this problem as well, the salt
state system uses salt state files to define the state a server
needs to be in.
.
Between the remote execution system, and state management Salt
addresses the backbone of cloud and data center management.
.
This particular package provides shared libraries that
salt-master, salt-minion, and salt-syndic require to function.
Homepage: http://saltstack.org/
Package: salt-master
Status: install ok installed
Priority: optional
Section: admin
Installed-Size: 123
Maintainer: Debian Salt Team <[email protected]>
Architecture: all
Source: salt
Version: 3005.1+ds-4
Depends: salt-common (= 3005.1+ds-4)
Conffiles:
/etc/init.d/salt-master ad5097989467e5153f0275beabd382a8
/etc/salt/master 5f52d8958a54a32f0c3785603751e065
Description: remote manager to administer servers via salt
salt is a powerful remote execution manager that can be used to
administer servers in a fast and efficient way.
.
It allows commands to be executed across large groups of
servers. This means systems can be easily managed, but data can
also be easily gathered. Quick introspection into running
systems becomes a reality.
.
Remote execution is usually used to set up a certain state on a
remote system. Salt addresses this problem as well, the salt
state system uses salt state files to define the state a server
needs to be in.
.
Between the remote execution system, and state management Salt
addresses the backbone of cloud and data center management.
.
This particular package provides the salt controller.
Homepage: http://saltstack.org/
minion - salt --versions-report
# /usr/bin/salt-call --versions-report
Salt Version:
Salt: 3005.1
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.0
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
Python: 3.9.16 (main, Jan 6 2023, 22:54:07)
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 23.2.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: amzn 2
locale: utf-8
machine: x86_64
release: 4.14.311-233.529.amzn2.x86_64
system: Linux
version: Amazon Linux 2
minion - packages details
[root@builder-amzn2-amd64 salt]# rpm -qi salt salt-minion
Name : salt
Version : 3005.1
Release : 4.amzn2
Architecture: x86_64
Install Date: Mon 24 Apr 2023 07:58:06 AM UTC
Group : System Environment/Daemons
Size : 171933179
License : ASL 2.0
Signature : RSA/SHA512, Fri 06 Jan 2023 11:50:20 PM UTC, Key ID 0e08a149de57bfbe
Source RPM : salt-3005.1-4.amzn2.src.rpm
Build Date : Fri 06 Jan 2023 10:59:21 PM UTC
Build Host : runner-rck4ycfx-project-35076987-concurrent-0
Relocations : (not relocatable)
URL : http://saltstack.org/
Summary : A parallel remote execution system
Description :
Salt is a distributed remote execution system used to execute commands and
query data. It was developed in order to bring the best solutions found in
the world of remote execution together and make them better, faster and more
malleable. Salt accomplishes this via its ability to handle larger loads of
information, and not just dozens, but hundreds or even thousands of individual
servers, handle them quickly and through a simple and manageable interface.
Name : salt-minion
Version : 3005.1
Release : 4.amzn2
Architecture: x86_64
Install Date: Mon 24 Apr 2023 07:58:07 AM UTC
Group : System Environment/Daemons
Size : 73144
License : ASL 2.0
Signature : RSA/SHA512, Fri 06 Jan 2023 11:50:35 PM UTC, Key ID 0e08a149de57bfbe
Source RPM : salt-3005.1-4.amzn2.src.rpm
Build Date : Fri 06 Jan 2023 10:59:21 PM UTC
Build Host : runner-rck4ycfx-project-35076987-concurrent-0
Relocations : (not relocatable)
URL : http://saltstack.org/
Summary : Client component for Salt, a parallel remote execution system
Description :
The Salt minion is the agent component of Salt. It listens for instructions
from the master, runs jobs, and returns results back to the master.
Additional context
Applying the same state from the same minion with salt-call state.apply path.to.state works just fine (log entries show that no errors/exceptions are triggered by unicode chars in file names from the archive).
Applying the same state to other minions with different platforms and the same salt versions (3005/3005.1) works fine. Moreover, the arm64 version of Amazon Linux 2 works just fine with salt v3005 installed from pip.
The only visible difference in environment between those 2 minions (Amazon Linux 2 amd64 vs arm64) is extra LANG:
# salt builder-amzn2-amd64 cmd.run 'env|sort'
builder-amzn2-amd64:
LANGUAGE=C
LC_ADDRESS=C
LC_COLLATE=C
LC_CTYPE=C
LC_IDENTIFICATION=C
LC_MEASUREMENT=C
LC_MESSAGES=C
LC_MONETARY=C
LC_NAME=C
LC_NUMERIC=C
LC_PAPER=C
LC_TELEPHONE=C
LC_TIME=C
LD_LIBRARY_PATH=
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
PWD=/root
SHLVL=1
SSL_CERT_DIR=/etc/pki/tls/certs
SSL_CERT_FILE=/etc/pki/tls/cert.pem
_=/usr/bin/env
# salt builder-amzn2-arm64 cmd.run 'env|sort'
builder-amzn2-arm64:
LANG=en_US.UTF-8
LANGUAGE=C
LC_ADDRESS=C
LC_COLLATE=C
LC_CTYPE=C
LC_IDENTIFICATION=C
LC_MEASUREMENT=C
LC_MESSAGES=C
LC_MONETARY=C
LC_NAME=C
LC_NUMERIC=C
LC_PAPER=C
LC_TELEPHONE=C
LC_TIME=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
PWD=/root
SHLVL=1
_=/usr/bin/env
May be related to #56224.
UPDATE: unsetting LANG on the minion in ssh session and running salt-call state.apply ... ends up with the same error.
JFTR, systemd service override like this solves the issue:
[root@builder-amzn2-amd64 defan]# cat /etc/systemd/system/salt-minion.service.d/override.conf
[Service]
Environment="LANG=en_US.UTF-8"