ts-odd icon indicating copy to clipboard operation
ts-odd copied to clipboard

Add information about encoding/be explicit about encoding

Open matheus23 opened this issue 4 years ago • 0 comments

Summary

Don't handle encoding in webnative very well. What is the data you can put into webnative? Are they utf8-encoded strings? Arrays of numbers? Uint8Arrays? And what do we get when we take it back out?

We need to get more explicit about this. This will also help us fix inconsistencies between the public and private filesystem.

Problem

There are multiple problems:

  • In the public filesystem you don't get back what you put into the filesystem, which can be very confusing:
    await fs.write(publicPath, "asdf") // input: string
    await fs.read(publicPath) // output: Uint8Array
    
    In the private filesystem this works.
  • In the private filesystem, everything is encoded as cbor, which stores some metadata about what type you're storing. E.g. int, boolean, string, number, list or record of things. On the public side, everything is just bytes.

Impact

  • It's very confusing to users.
  • There's a small overhead to store data on the private filesystem
  • It's impossible to make precise types for the UnixFS interface
  • It's hard to get strings from the public filesystem

Solution

We should provide an api just like node's fs.writeFile(path, data, { encoding: X }).

matheus23 avatar Jul 22 '21 17:07 matheus23