salt icon indicating copy to clipboard operation
salt copied to clipboard
verified publisher icon saltstack

[BUG] Tornado UnicodeDecodeError behind proxy

Open nathanmetcalf opened this issue 2 years ago • 3 comments

Description Downloading some files behind a proxy causes the below error:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 2: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 0: invalid continuation byte

---=== TRIM ===---

This is only for some files. This URL (On the same server) does not error.

http://archive.ubuntu.com/ubuntu/dists/focal-updates/Release

Setup Salt minion config:

proxy_port: 3128
proxy_host: <proxy server>

State file

download_proxy_test:
  file.managed:
    - name: /tmp/test.download
     - source: http://archive.ubuntu.com/ubuntu/dists/focal-updates/main/dep11/Components-amd64.yml.xz
    - skip_verify: true
    - keep_source: false

Squid proxy config

http_port 3128
visible_hostname proxy1.local
access_log syslog:daemon.info
cache_log syslog:daemon.info
http_access allow all

Please be as specific as possible and give set-up details.

  • [ ] on-prem machine
  • [X] VM - Virtualbox
  • [X] VM running on a cloud service, AWS
  • [ ] container (Kubernetes, Docker, containerd, etc. please specify)
  • [ ] or a combination, please be explicit
  • [ ] jails if it is FreeBSD
  • [X] classic packaging
  • [ ] onedir packaging
  • [ ] used bootstrap to install

Steps to Reproduce the behavior

Debug Logs

Debug

[DEBUG   ] In saltenv 'base', looking at rel_path 'default/proxy.sls' to resolve 'salt://default/proxy.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/default/proxy.sls' to resolve 'salt://default/proxy.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/base/default/proxy.sls
[DEBUG   ] Jinja search path: ['/var/cache/salt/minion/files/base']
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/default/proxy.sls' using 'jinja' renderer: 0.04261612892150879
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/default/proxy.sls:
download_proxy_test:
  file.managed:
    - name: /tmp/test.download
    - source: http://archive.ubuntu.com/ubuntu/dists/focal-updates/main/dep11/Components-amd64.yml.xz
    - skip_verify: true
    - keep_source: false

[DEBUG   ] Results of YAML rendering:
OrderedDict([('download_proxy_test', OrderedDict([('file.managed', [OrderedDict([('name', '/tmp/test.download')]), OrderedDict([('source', 'http://archive.ubuntu.com/ubuntu/dists/focal-updates/main/dep11/Components-amd64.yml.xz')]), OrderedDict([('skip_verify', True)]), OrderedDict([('keep_source', False)])])]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/default/proxy.sls' using 'yaml' renderer: 0.00148773193359375
[DEBUG   ] LazyLoaded config.option
[DEBUG   ] LazyLoaded file.managed
[INFO    ] Running state [/tmp/test.download] at time 13:46:12.824547
[INFO    ] Executing state file.managed for [/tmp/test.download]
[DEBUG   ] LazyLoaded file.source_list
[DEBUG   ] LazyLoaded cp.is_cached
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Requesting URL http://archive.ubuntu.com/ubuntu/dists/focal-updates/main/dep11/Components-amd64.yml.xz using GET method
[DEBUG   ] Using backend: tornado
[DEBUG   ] GET http://archive.ubuntu.com/ubuntu/dists/focal-updates/main/dep11/Components-amd64.yml.xz
[DEBUG   ] Trying 10.0.14.11:3128...
[DEBUG   ] TCP_NODELAY set
[DEBUG   ] Connected to proxy1.local (10.0.14.11) port 3128 (#0)
[DEBUG   ] > GET http://archive.ubuntu.com/ubuntu/dists/focal-updates/main/dep11/Components-amd64.yml.xz HTTP/1.1
[DEBUG   ] > Host: archive.ubuntu.com
[DEBUG   ] > Accept: */*
[DEBUG   ] > Accept-Encoding: none
[DEBUG   ] > Proxy-Connection: Keep-Alive
[DEBUG   ] > User-Agent: Salt/3004.2 http.query()
[DEBUG   ] >
[DEBUG   ] Mark bundle as not supporting multiuse
[DEBUG   ] < HTTP/1.1 200 OK
[DEBUG   ] < Date: Tue, 18 Apr 2023 13:44:06 GMT
[DEBUG   ] < Server: Apache/2.4.29 (Ubuntu)
[DEBUG   ] < Last-Modified: Tue, 18 Apr 2023 08:24:02 GMT
[DEBUG   ] < ETag: "43018-5f99807af942d"
[DEBUG   ] < Accept-Ranges: bytes
[DEBUG   ] < Content-Length: 274456
[DEBUG   ] < Content-Type: application/x-xz
[DEBUG   ] < Age: 131
[DEBUG   ] < X-Cache: HIT from proxy1.local
[DEBUG   ] < X-Cache-Lookup: HIT from proxy1.local:3128
[DEBUG   ] < Via: 1.1 proxy1.local (squid/4.10)
[DEBUG   ] < Connection: keep-alive
[DEBUG   ] <
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9e in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9e in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9e in position 0: invalid start byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 0: invalid start byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa4 in position 1: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa4 in position 1: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa4 in position 1: invalid start byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 2: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 2: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 2: invalid continuation byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfe in position 0: invalid start byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9e in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9e in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9e in position 0: invalid start byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 7: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 7: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 7: invalid continuation byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc1 in position 1: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc1 in position 1: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc1 in position 1: invalid start byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 3: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 3: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 3: invalid continuation byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 2-3: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 2-3: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 2-3: invalid continuation byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 0: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 0: invalid start byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd9 in position 1: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd9 in position 1: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd9 in position 1: invalid continuation byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x88 in position 8: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x88 in position 8: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x88 in position 8: invalid start byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 1-2: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 1-2: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 1-2: invalid continuation byte
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xca in position 2: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xca in position 2: invalid continuation byte
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xca in position 2: invalid continuation byte
[DEBUG   ] Connection #0 to host proxy1.local left intact
[INFO    ] File changed:

[INFO    ] Completed state [/tmp/test.download] at time 13:46:12.900403 (duration_in_ms=75.855)
[DEBUG   ] File /var/cache/salt/minion/accumulator/139663626267328 does not exist, no need to cleanup
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] LazyLoaded highstate.output
[DEBUG   ] LazyLoaded nested.output
local:
----------
          ID: download_proxy_test
    Function: file.managed
        Name: /tmp/test.download
      Result: True
     Comment: File /tmp/test.download updated
     Started: 13:46:12.824548
    Duration: 75.855 ms
     Changes:
              ----------
              diff:

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  75.855 ms

Expected behavior The file should download without errors.

Screenshots N/A

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
Salt Version:
          Salt: 3004.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.8.10 (default, Nov 14 2022, 12:59:47)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-77-generic
        system: Linux
       version: Ubuntu 20.04 focal

Additional context Add any other context about the problem here.

nathanmetcalf avatar Apr 18 '23 13:04 nathanmetcalf

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

welcome[bot] avatar Apr 18 '23 13:04 welcome[bot]

Can you intercept and show the response headers that the proxy is returning? If the proxy is claiming that it's UTF-8 when it's not then that would be where a fix is needed.

OrangeDog avatar Apr 20 '23 10:04 OrangeDog

Intercepting any traffic and checking wont help here.

A tornado update will. If you go and patch the file /opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/curl_httpclient.py manually to the latest version from tornado 6.4.0 like this:

Old code:

    def _curl_debug(self, debug_type, debug_msg):
        debug_types = ('I', '<', '>', '<', '>')
        debug_msg = native_str(debug_msg)
        if debug_type == 0:
            curl_log.debug('%s', debug_msg.strip())
        elif debug_type in (1, 2):
            for line in debug_msg.splitlines():
                curl_log.debug('%s %s', debug_types[debug_type], line)
        elif debug_type == 4:
            curl_log.debug('%s %r', debug_types[debug_type], debug_msg)

New code (latest tornado lib):

    def _curl_debug(self, debug_type: int, debug_msg: str) -> None:
        debug_types = ("I", "<", ">", "<", ">")
        if debug_type == 0:
            debug_msg = native_str(debug_msg)
            curl_log.debug("%s", debug_msg.strip())
        elif debug_type in (1, 2):
            debug_msg = native_str(debug_msg)
            for line in debug_msg.splitlines():
                curl_log.debug("%s %s", debug_types[debug_type], line)
        elif debug_type == 4:
            curl_log.debug("%s %r", debug_types[debug_type], debug_msg)

The error disappears after replacing the block. I suppose this is because of different handling on the DEBUGMESSAGE callback from pycurl since version 7.19.5.2: The second argument to a DEBUGFUNCTION callback is now of type bytes on Python 3. Previously the argument was of type str. (Source)

However, in my use-cases, the message did not prevent the function. It was merely an annoying bunch of tracebacks on each saltstate-call.

Tornado lib in salt is scheduled to be updated in 3007.x as I have read in #64305. So from that release on the error should not appear anymore.

maltris avatar May 15 '24 10:05 maltris