Issue with plugin upload functionality

[samwmarsh]

Describe the bug Currently, the offline pluginset uploaded by tenable to https://plugins.nessus.org/get.php?f=sc-plugins-diff.tar.gz (passing username/password strings) is too large for ssl.py to handle (larger than 32-bit limit). This causes issues when using the following code to upload plugins:

    with open('sc-plugins-diff.tar.gz', 'rb') as pluginfile:
        print(sc.feeds.process('active', pluginfile))

To Reproduce Steps to reproduce the behavior:

1. Download sc-plugins-diff.tar.gz
2. Run the above python code, having authenticated to an sc instance
3. See error

    print(sc.feeds.process('active', pluginfile))
  File "/var/lib/jenkins/workspace/Nessus/Plugin-Uploader/venv/lib64/python3.6/site-packages/tenable/sc/feeds.py", line 100, in process
    filename = self._api.files.upload(fobj)
  File "/var/lib/jenkins/workspace/Nessus/Plugin-Uploader/venv/lib64/python3.6/site-packages/tenable/sc/files.py", line 33, in upload
    'Filedata': fobj}).json()['response']['filename']
  File "/var/lib/jenkins/workspace/Nessus/Plugin-Uploader/venv/lib64/python3.6/site-packages/restfly/session.py", line 631, in post
    return self._req('POST', path, **kwargs)
  File "/var/lib/jenkins/workspace/Nessus/Plugin-Uploader/venv/lib64/python3.6/site-packages/restfly/session.py", line 516, in _req
    timeout=self._timeout, **kwargs)
  File "/var/lib/jenkins/workspace/Nessus/Plugin-Uploader/venv/lib64/python3.6/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/lib/jenkins/workspace/Nessus/Plugin-Uploader/venv/lib64/python3.6/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/var/lib/jenkins/workspace/Nessus/Plugin-Uploader/venv/lib64/python3.6/site-packages/requests/adapters.py", line 450, in send
    timeout=timeout
  File "/var/lib/jenkins/workspace/Nessus/Plugin-Uploader/venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 722, in urlopen
    chunked=chunked,
  File "/var/lib/jenkins/workspace/Nessus/Plugin-Uploader/venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/var/lib/jenkins/workspace/Nessus/Plugin-Uploader/venv/lib64/python3.6/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib64/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/usr/lib64/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
  File "/usr/lib64/python3.6/ssl.py", line 968, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib64/python3.6/ssl.py", line 937, in send
    return self._sslobj.write(data)
  File "/usr/lib64/python3.6/ssl.py", line 601, in write
    return self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes

Expected behavior Plugin to upload and process without failing out.

Screenshots edit: This seems to cause an error when uploading through the UI too, I've raised as a case with Tenable directly.

System Information (please complete the following information):

• OS: tenable.sc running on CentOS 7.9, pytenable running within Jenkins
• Architecture 64-bit
• Version latest

[kintaroju]

Hi,

I too am having the EXACT same issue, as I was trying to upload the plugin file manually provided by Tenable and unfortunately all the Tenable.SC plugins have exceed 2GB now. Last month it was just under the 2GB limit. I've sent a ticket to Tenable support and see if they have a hotfix or something to apply to Tenable.SC to fix this issue. I will try to update this github ticket accordingly.

For the methods to update the plugins I did exactly what you did which was via webgui and pyTenable script, both failed.

[aseemsavio]

The Security Center team is aware and is working on this issue, and they're planning to fix it in SC v6.4.0, which is expected to come out in the first half of this year.

[kintaroju]

That is great to hear so the bigger question is this affecting all versions of tenable.sc or just 6.3.0?

[aseemsavio]

@kintaroju this seems to affect all SC versions. In the mean time, the SC team has put out a workaround for this issue. You may have to reach out to Customer Support for the same.

[kintaroju]

Good to know thanks for the info. I will remind the tech about not requiring the legendary diagnostics file to deal with this issue lol

[kintaroju]

@aseemsavio I finally got a response from Tenable and they provided me a workaround (can't post the workaround as they indicated they will post a KB on this), except it only worked for the WebGUI and not via pyTenable. So the question is for a pyTenable perspective how can I upload the plugins, or do we have to wait till there is an official patch before pyTenable can be fixed as well?

[samwmarsh]

@kintaroju they gave me 2 KBs, neither worked, they seem to have managed to minimize the plugin-set size, though this doesn't work with pyTenable.

[samwmarsh]

Hey, I'm now able to upload manually, however I'm getting the following error from pytenable:

Requests Library Error: ('Connection aborted.', timeout('The write operation timed out',))

[samwmarsh]

We're being hit by the 32-bit limit again for this.

OverflowError: string longer than 2147483647 bytes

[SteveMcGrath]

looks like the error is related to how requests is handling files differently than raw encoded data. Took aq little digging but this seems to be related:

https://stackoverflow.com/questions/22915295/python-requests-post-and-big-content/22915488#22915488

[SteveMcGrath]

After some testing the memory usage using the file upload went from 2G to to under 50M. I think this should long-term solve this issue on the library-front.

[samwmarsh]

Thanks for resolving this @SteveMcGrath - do we have an estimated release date? I notice the last release is >1mo ago so presumably its soon-ish?

[SteveMcGrath]

yeah, just trying to work through the backlog of PRs before releasing. If you're in a rush, you can always test with a pip install from git:

pip install git+https://github.com/tenable/pytenable.git

[samwmarsh]

All good, we're in an airgapped network so happy to wait. Soon-ish is good enough for me!