pyfilesystem
pyfilesystem copied to clipboard
PyFilesystem
S3 expose module
I have altered an MIT licensed implementation of S3 called mock-s3 to function
as a pyfs expose module. I copied http.py's format to create the __init__.py.
You can find my work here:
https://github.com/ClashTheBunny/mock-s3/tree/pyfs_expose_mocks3
(mock-s3 branch pyfs_expose_mocks3).
do:
cd pyfs/fs/expose
git clone https://github.com/ClashTheBunny/mock-s3 -b pyfs_expose_mocks3 mocks3
and now you can run something like __init__.py.
It requires:
redis (and a redis-server) or mockredispy
and
jinja2.
It needs some cleaning up and a much better readme, but as it is, it works
fairly well on my localhost with the default parameters and the test scripts if
I add mocking.localhost to my /etc/hosts file and fake AWS credentials.
TODO would include things like real authentication and such, but I'm not sure
where this will go, so we'll see about what would be priority.
What would need to be done to get this in the next release?
Original issue reported on code.google.com by [email protected] on 21 Jul 2014 at 6:19
Here is the output from svn diff for the mocks3 changes for people who don't
use much git.
Original comment by [email protected] on 21 Jul 2014 at 8:44
Attachments:
Also, I updated it so that it has a __main__.py so that if you install pyfs in
your path (or add it via environment variables) it is good enough just to run:
python -mfs.expose.mocks3
and set up dns and authentication.
Original comment by [email protected] on 21 Jul 2014 at 8:52
So this serves an fs with the S3 protocol? I guess it's for unit testing
purposes?
Original comment by willmcgugan on 21 Jul 2014 at 8:54
It was originally designed for that, yes. And I think it would still be very
useful for that, specifically if you use the in-memory file system provided by
pyfs with mockredis so that no other services were required and everything was
stateless.
One other possibility is to use TahoeLAFS as a back end for s3 services. Not
that it would be fast, but it could be resilient and cheap in a way that other
s3 services wouldn't be.
Specifically if a camlistore user wanted to distribute his or her data across a
friend net, this may be eventually useful once authentication is included.
Original comment by [email protected] on 21 Jul 2014 at 9:06
It's a cool project, but I don't think it's appropriate to get it in at PyFS
release. We want to keep it as general purpose as possible, and this is quite a
specialised use case. Best you host it in your own repos. I would like to tweet
about it though.
Original comment by willmcgugan on 21 Jul 2014 at 9:33
I'll try to get into the upstream project and then I'll let you know.
Original comment by [email protected] on 21 Jul 2014 at 2:06