google.cloud
google.cloud copied to clipboard
ansible-collections
gcp_storage_object Python3 UnicodeDecodeError: 'utf-8' codec can't decode
From @bucklander on Jul 31, 2020 22:41
SUMMARY
While using Python3, gcp_storage_object module is unable to upload binary files (and perhaps other file types) without throwing:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x98 in position 522: invalid start byte
This is possibly related to Ansible python3's explicit string conversion (or not). When using other file types (E.g. simple text files) or just using Python2 with any file type, the problem does not exist.
Below are two example playbooks; One which uploads a gzip tarball file, and another which uploads a simple text file. Each playbook is executed with Python2 and again with Python3.
I'm certainly no expert in file compression or file types, however in my particular case the gzip'd file format seems to cause the issue. I'm also seeing this problem with binary files within regular tarball (no gzip compression). You could also replace the compressed file examples below with seemingly any randomly generated binary file, and the result appears to be the same.
ISSUE TYPE
- Bug Report
COMPONENT NAME
gcp_storage_object
ANSIBLE VERSION
$ ansible --version
ansible 2.9.11
config file = None
configured module search path = ['${HOME}/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible
executable location = /Library/Frameworks/Python.framework/Versions/3.8/bin/ansible
python version = 3.8.5 (v3.8.5:580fbb018f, Jul 20 2020, 12:11:27) [Clang 6.0 (clang-600.0.57)]
CONFIGURATION
$ ansible-config dump --only-changed
$
OS / ENVIRONMENT
This can be reproduced on a few different systems. Here are a few specific examples:
macOS Mojave 10.14.6
Python 2.7.15
Python 3.8.5
CentOS Linux release 8.1.1911 (Core)
Python 3.6.8
STEPS TO REPRODUCE
Execute the following example playbooks with a simple, flat text file as well as a tarball.
echo "hello" > foobar.txt
tar -zcf binary.tar foobar.txt
Example playbook to upload binary file to GCS:
# upload-bin-file.yml
---
- name: Upload a binary file to Google Cloud Storage bucket using gcp_storage_object module
hosts: all
connection: local
tasks:
- name: Upload Latest Backup(s) to GCS Bucket
gcp_storage_object:
action: upload
overwrite: yes
bucket: gcs-bucket-name
src: binary.tar
dest: "binary.tar"
project: google-cloud-project-name
auth_kind: serviceaccount
service_account_file: "jwt.json"
state: present
Example playbook to upload text file to GCS:
# upload-text-file.yml
---
- name: Upload a text file to Google Cloud Storage bucket using gcp_storage_object module
hosts: all
connection: local
tasks:
- name: Upload Latest Backup(s) to GCS Bucket
gcp_storage_object:
action: upload
overwrite: yes
bucket: gcs-bucket-name
src: foobar.txt
dest: "foobar.txt"
project: google-cloud-project-name
auth_kind: serviceaccount
service_account_file: "jwt.json"
state: present
Run upload-text-file.yml using Python 2 (modify Python envs per your system):
ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=`which python2`
Run upload-text-file.yml using Python 3:
ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=`which python3`
Run upload-bin-file.yml using Python 2:
ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=`which python2`
Run upload-bin-file.yml using Python 3:
ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=`which python3`
EXPECTED RESULTS
It's expected that all files upload successfully to GCS, regardless of using Python 2, or Python 3 or file type.
ACTUAL RESULTS
Python 2 Text File Upload (successful):
$ ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2
PLAY [Upload a file to GCS using gcp_storage_object module] ********************************************************************************
TASK [Gathering Facts] *********************************************************************************************************************
ok: [localhost]
TASK [Upload Latest Backup(s) to GCS Bucket] ***********************************************************************************************
changed: [localhost]
PLAY RECAP *********************************************************************************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Python 3 Text File Upload (successful):
$ ansible-playbook --connection=local --inventory localhost, upload-text-file.yml -e ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3
PLAY [Upload a file to GCS using gcp_storage_object module] *****************************************************************************************************************
TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [localhost]
TASK [Upload Latest Backup(s) to GCS Bucket] ********************************************************************************************************************************
changed: [localhost]
PLAY RECAP ******************************************************************************************************************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Python 2 Binary File Upload (successful):
$ ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2
PLAY [Upload a file to GCS using gcp_storage_object module] *****************************************************************************************************************
TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [localhost]
TASK [Upload Latest Backup(s) to GCS Bucket] ********************************************************************************************************************************
changed: [localhost]
PLAY RECAP ******************************************************************************************************************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Python 3 Binary File Upload (fails):
$ ansible-playbook --connection=local --inventory localhost, upload-bin-file.yml -e ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3 -vvvv
ansible-playbook 2.9.11
config file = None
configured module search path = ['${HOME}/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible
executable location = /Library/Frameworks/Python.framework/Versions/3.8/bin/ansible-playbook
python version = 3.8.5 (v3.8.5:580fbb018f, Jul 20 2020, 12:11:27) [Clang 6.0 (clang-600.0.57)]
No config file found; using defaults
setting up inventory plugins
Set default localhost to localhost
Parsed localhost, inventory source with host_list plugin
Loading callback plugin default of type stdout, v2.0 from /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible/plugins/callback/default.py
PLAYBOOK: upload-bin-file.yml ***********************************************************************************************************************************************************************************
Positional arguments: upload-bin-file.yml
verbosity: 4
connection: local
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('localhost,',)
extra_vars: ('ansible_python_interpreter=/Library/Frameworks/Python.framework/Versions/3.8/bin/python3',)
forks: 5
1 plays in upload-bin-file.yml
PLAY [Upload a file to GCS using gcp_storage_object module] *****************************************************************************************************************************************************
TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
task path: ${HOME}/Desktop/upload-bin-file.yml:3
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: bwallander
<localhost> EXEC /bin/sh -c 'echo ~bwallander && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ${HOME}/.ansible/tmp `"&& mkdir ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500 && echo ansible-tmp-1596236353.566118-46540-87177398296500="` echo ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500 `" ) && sleep 0'
Using module file /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible/modules/system/setup.py
<localhost> PUT ${HOME}/.ansible/tmp/ansible-local-46536i64et516/tmpj1i25bxy TO ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/AnsiballZ_setup.py
<localhost> EXEC /bin/sh -c 'chmod u+x ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/ ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/AnsiballZ_setup.py && sleep 0'
<localhost> EXEC /bin/sh -c '/Library/Frameworks/Python.framework/Versions/3.8/bin/python3 ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/AnsiballZ_setup.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r ${HOME}/.ansible/tmp/ansible-tmp-1596236353.566118-46540-87177398296500/ > /dev/null 2>&1 && sleep 0'
ok: [localhost]
META: ran handlers
TASK [Upload Latest Backup(s) to GCS Bucket] ********************************************************************************************************************************************************************
task path: ${HOME}/Desktop/upload-bin-file.yml:7
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: bwallander
<localhost> EXEC /bin/sh -c 'echo ~bwallander && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ${HOME}/.ansible/tmp `"&& mkdir ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804 && echo ansible-tmp-1596236354.939475-46572-195465935008804="` echo ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804 `" ) && sleep 0'
Using module file /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/ansible/modules/cloud/google/gcp_storage_object.py
<localhost> PUT ${HOME}/.ansible/tmp/ansible-local-46536i64et516/tmp1a31n78a TO ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py
<localhost> EXEC /bin/sh -c 'chmod u+x ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/ ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py && sleep 0'
<localhost> EXEC /bin/sh -c '/Library/Frameworks/Python.framework/Versions/3.8/bin/python3 ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r ${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
File "${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py", line 102, in <module>
_ansiballz_main()
File "${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py", line 94, in _ansiballz_main
invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
File "${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py", line 40, in invoke_module
runpy.run_module(mod_name='ansible.modules.cloud.google.gcp_storage_object', init_globals=None, run_name='__main__', alter_sys=True)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 207, in run_module
return _run_module_code(code, init_globals, run_name, mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py", line 286, in <module>
File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py", line 190, in main
File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py", line 206, in upload_file
File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/module_utils/gcp_utils.py", line 99, in post_contents
File "/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/module_utils/gcp_utils.py", line 159, in full_post
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py", line 578, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/auth/transport/requests.py", line 448, in request
response = super(AuthorizedSession, self).request(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 1039, in _send_output
for chunk in chunks:
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 994, in _read_readable
datablock = readable.read(self.blocksize)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x98 in position 522: invalid start byte
fatal: [localhost]: FAILED! => {
"changed": false,
"module_stderr": "Traceback (most recent call last):\n File \"${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py\", line 102, in <module>\n _ansiballz_main()\n File \"${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py\", line 94, in _ansiballz_main\n invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n File \"${HOME}/.ansible/tmp/ansible-tmp-1596236354.939475-46572-195465935008804/AnsiballZ_gcp_storage_object.py\", line 40, in invoke_module\n runpy.run_module(mod_name='ansible.modules.cloud.google.gcp_storage_object', init_globals=None, run_name='__main__', alter_sys=True)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py\", line 207, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py\", line 97, in _run_module_code\n _run_code(code, mod_globals, init_globals,\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py\", line 87, in _run_code\n exec(code, run_globals)\n File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py\", line 286, in <module>\n File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py\", line 190, in main\n File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/modules/cloud/google/gcp_storage_object.py\", line 206, in upload_file\n File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/module_utils/gcp_utils.py\", line 99, in post_contents\n File \"/var/folders/t4/mfbl431x7hx6_w64l1p01yjm0000gn/T/ansible_gcp_storage_object_payload_9pk0efwx/ansible_gcp_storage_object_payload.zip/ansible/module_utils/gcp_utils.py\", line 159, in full_post\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py\", line 578, in post\n return self.request('POST', url, data=data, json=json, **kwargs)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/auth/transport/requests.py\", line 448, in request\n response = super(AuthorizedSession, self).request(\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py\", line 530, in request\n resp = self.send(prep, **send_kwargs)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/sessions.py\", line 643, in send\n r = adapter.send(request, **kwargs)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/requests/adapters.py\", line 439, in send\n resp = conn.urlopen(\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 670, in urlopen\n httplib_response = self._make_request(\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 392, in _make_request\n conn.request(method, url, **httplib_request_kw)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 1255, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 1301, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 1250, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 1039, in _send_output\n for chunk in chunks:\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py\", line 994, in _read_readable\n datablock = readable.read(self.blocksize)\n File \"/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/codecs.py\", line 322, in decode\n (result, consumed) = self._buffer_decode(data, self.errors, final)\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0x98 in position 522: invalid start byte\n",
"module_stdout": "",
"msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
"rc": 1
}
PLAY RECAP ******************************************************************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
Copied from original issue: ansible/ansible#71034
From @bucklander on Jul 31, 2020 22:55
FWIW, besides my local macbook, this is also repeatable on an AWX server container:
bash-4.4# cat /etc/redhat-release
CentOS Linux release 8.1.1911 (Core)
bash-4.4# python3 -V
Python 3.6.8
bash-4.4# ansible --version
ansible 2.9.7
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.6/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
From @bucklander on Jul 31, 2020 23:23
cc @erjohnso @GoogleCloudPlatform @rambleraptor
FWIW this bug can be resolved by simply changing gcp_storage_object.py#L254 from...
with open(src, "r") as file_obj: to with open(src, "rb") as file_obj:
This ensures there's no decoding attempt (or subsequent error).
This approach appears similar to download_file's use of the open func at gcp_storage_object.py#L243, where it specifically is configured with the b (binary) mode specifier.
Not only does this resolve the issue with uploading binary files, but the change doesn't appear to harm text uploads. Unsure though if this is a safe strategy overall.
Have opened https://github.com/ansible-collections/google.cloud/pull/268 with a proposed fix, but have no real way to regression test it.
Until it fixed I stay with Python 2 ansible.cfg:
[defaults]
interpreter_python = /usr/bin/python2
An alternative approach is to use uri module and handle authentication / error checking manually: https://github.com/ansible-collections/google.cloud/pull/268#issuecomment-706752987
This error is still persisting in January 2022 more than a year later.
Safe to say at this point the Google Cloud ansible collection should be considered legacy
Safe to say at this point the Google Cloud ansible collection should be considered legacy
There is an incentive to create new products inside Google (to get the promotion). There is no incentive to keep it working.
Still I see new commits in the repository...
I use gcloud cli + Bash for automation, GCP Ansible modules were unreliable for automation: I need work done, not to waste time debugging Python code of google.collection (most problems with lack of idempotency).
I think this collection is only a downstream output of https://github.com/GoogleCloudPlatform/magic-modules
downstream output
It explains lack of idempotency, idiomatic for Ansible. They robo-generate code around REST API instead of providing convenient tools. That explains the wide coverage of API & difficulty to fix bugs or add extra functionality beside dumb translation to HTTP call.
Here is on the bug tracker another person recommended to use Google API directly using JSON API key and uri module. You can list & update resources, building idempotency themselves.