xmlrpc y2038 problem

[mcepl]

(originally filed in the openSUSE Bugzilla](https://bugzilla.suse.com/show_bug.cgi?id=1203311)):

While working on reproducible builds for openSUSE, I found that our python-softlayer package failed tests in 2038 with

 self = <xmlrpc.client.Marshaller object at 0x7fa5770e91f0>, value = 2167459529
 write = <built-in method append of list object at 0x7fa577194740>
 
     def dump_long(self, value, write):
         if value > MAXINT or value < MININT:
 >           raise OverflowError("int exceeds XML-RPC limits")
 E           OverflowError: int exceeds XML-RPC limits
 
 /usr/lib64/python3.8/xmlrpc/client.py:539: OverflowError
 =========================== short test summary info ============================
 FAILED tests/CLI/modules/cdn_tests.py::CdnTests::test_detail_account - Overfl...
 FAILED tests/managers/cdn_tests.py::CDNTests::test_detail_usage_metric - Over...
 =========== 2 failed, 1831 passed, 6 skipped, 5 deselected in 38.01s ===========

To Reproduce:

osc co openSUSE:Factory/python-softlayer && cd $_
osc build --noservice --trust-all-projects --vm-type=kvm \
  --alternative-project=home:bmwiedemann:reproducible \
  --build-opt=--vm-custom-opt="-rtc base=2038-01-20T00:00:00" \
  openSUSE_Tumbleweed x86_64

(basically run with the machine hardware clock set to post-2038)

Expected behavior Test suite should pass.

Version 6.1.2 from tarball.

[allmightyspiff]

I don't know what osc is... so I'm not really able to test this as you mentioned.

However I suspect the problem is some of the fixture data has a large id number, so I'll reduce that below 2167459529 and see if that helps you.

If that fix still doesn't work, the output of the following would be helpful:

pytest -v tests/CLI/modules/cdn_tests.py
pytest -v tests/managers/cdn_tests.py

Really weird this test would only fail in this specific situation though.

[mcepl]

@bmwiedemann Could you take over the conversation here, please?

[bmwiedemann]

The current #1765 does not help as it fixes nothing that would only break in 2038.

Here is a full backtrace:

 /usr/lib64/python3.8/xmlrpc/client.py:539: OverflowError
 ----------------------------- Captured stdout call -----------------------------
 int exceeds XML-RPC limits
 ______________________ CDNTests.test_detail_usage_metric _______________________

 self = <tests.managers.cdn_tests.CDNTests testMethod=test_detail_usage_metric>

     def test_detail_usage_metric(self):
 >       self.cdn_client.get_usage_metrics(12345, history=30, frequency="aggregate")

 tests/managers/cdn_tests.py:32:
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 SoftLayer/managers/cdn.py:163: in get_usage_metrics
     usage = self.cdn_metrics.getMappingUsageMetrics(unique_id, self._start_date, self._end_date, frequency)
 SoftLayer/API.py:610: in call_handler
     return self(name, *args, **kwargs)
 SoftLayer/API.py:578: in call
     return self.client.call(self.name, name, *args, **kwargs)
 SoftLayer/API.py:302: in call
     return self.transport(request)
 SoftLayer/transports/timing.py:22: in __call__
     result = self.transport(call)
 SoftLayer/transports/xmlrpc.py:81: in __call__
     request.payload = xmlrpc.client.dumps(tuple(largs),
 /usr/lib64/python3.8/xmlrpc/client.py:968: in dumps
     data = m.dumps(params)
 /usr/lib64/python3.8/xmlrpc/client.py:501: in dumps
     dump(v, write)
 /usr/lib64/python3.8/xmlrpc/client.py:523: in __dump
     f(self, value, write)
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

 self = <xmlrpc.client.Marshaller object at 0x7f696fc6c520>, value = 2170227406
 write = <built-in method append of list object at 0x7f697658da00>
 
     def dump_long(self, value, write):
         if value > MAXINT or value < MININT:
 >           raise OverflowError("int exceeds XML-RPC limits")
 E           OverflowError: int exceeds XML-RPC limits

 /usr/lib64/python3.8/xmlrpc/client.py:539: OverflowError

[allmightyspiff]

Sorry, it was not clear to me on reading your initial comment that you were running these tests in the YEAR 2038.

The problem is this section of code uses a timestamp from datetime.now(), which if your running this test in 2038, is going to be > MAXINT for xmlrpc, hence the error.

Since the xmlrpc doesn't support 64 bit ints, I'm not really sure I want to force these into strings just to fix a problem that won't exist for another 16 years. Anyway I will fix the tests so they don't fail.

[bmwiedemann]

btw: You can emulate it on any Linux with

date --set=2038-01-20

[bmwiedemann]

https://stackoverflow.com/questions/107616/xml-rpc-best-way-to-handle-64-bit-values says many xml-rpc implementations support an "i8" data type for 64-bit integers (so if python's xmlrpc.py doesn't it could be time to add it there), but OTOH XML transmits everything as string anyway, so you might just as well convert on the client side.