InseeFrLab
Customising storage connection scripts
It seems like scripts provided here may depend to some extent on the network configuration of the Onyxia instance (e.g. behind a proxy or not), which may not be entirely adressed through environnement variables. I recently bumped into an issue when trying to use the s3fs connecting script, not working for an instance behind a proxy and for which the NO_PROXY variable does not apply (more precisely, it seems to work for http but not for https requests).
One way to adress this, add an extra argument:
fs = s3fs.S3FileSystem(client_kwargs={'endpoint_url': 'https://'+'${c.AWS_S3_ENDPOINT}'},key ='${c.AWS_ACCESS_KEY_ID}', secret = '${c.AWS_SECRET_ACCESS_KEY}', token = '${c.AWS_SESSION_TOKEN}', config_kwargs={'proxies': {'https://'+'${c.AWS_S3_ENDPOINT}': None}})
But not working for an instance not behind a proxy...
After a little reflection, I would advocate for having those scripts passed as parameters, providing the admins of an instance with the possibility of deciding for which tools/software they want to provide the users with connecting scripts.
Which form it should take? On this I am not clear; but I would definitively like to see scripts stored on a repo internal to the instance.
