pfio
pfio copied to clipboard
pfnet
IO library to access various filesystems with unified API
Experimental implementation for https://github.com/pfnet/pfio/issues/272 .
One of requirements in our systems is to be suitable with other tools. As PPE has now Profiling feature, it would be nice if PFIO rides on its use cases,...
Currently, FileCache checks the underlying filesystem with `stat` command, depending on GNU stat, to ensure it is not the remote filesystem. On the other hand, BSD stat does not support...
I found out that when reading the entire content of a file of 2+αGiB from S3 fails by `OverflowError: signed integer is greater than maximum` exception raised from Python SSL...
May supersede https://github.com/pfnet/pfio/issues/29 . - https://opentelemetry.io/ - Report span to Jaeger on each IO call Pros - Might suit well with other systems in OpenTelemetry ecosystem Cons - Not sure...
Memo for the requirements 1. Plug-in like => can switch between different profilers, or customize profilers, e.g. record fields 2. Separate log writers to switch formats 3. Being able to...
Hi, the document says > PFIO supports various container file formats to aggregate many small files into single large file with metadata mapping, e.g. HDF5, ZIP and Tar (and more...
Deep learning frameworks support multi-process data loading, such as `num_worker` option of `DataLoader` in PyTorch, `MultiprocessIterator` in Chainer, etc. They use multiprocessing module to launch worker processes using `fork` by...
```py import _hashlib import mysql.connector import pfio dest_path = 'hdfs:///my/hdfs/file' with pfio.open(dest_path, 'wb') as file_out: file_out.write(b'data') ``` In my environment, the above code results in a segmentation fault: ``` python:...
