koute
FileList API
I noticed that the files property of InputElement was removed some time ago. Are there any thoughts on how this should be reimplemented? In the meantime, are there any quick workarounds to get access to a FileList?
Are there any thoughts on how this should be reimplemented?
Yes - create subtypes for each kind of an input element, so, e.g. FileInputElement for InputElements whose type is set to file, and expose the files property only on that type.
A custom InstanceOf implementation can be used to enforce that the type is correctly set for a given subtype.
In the meantime, are there any quick workarounds to get access to a FileList?
Yep. Something like this:
let files: FileList = js!( return @{input}.files; ).try_into().unwrap();
I originally attempted that workaround but FileList doesn't appear to be publicly exposed. Perhaps I was missing something?
I may take a stab at the suggested implementation when I have some free time.
Did you ever get this working?
Right now I'm running into a hairy issue where I'm trying to access a file from an input element, subdivide it with the slice method, then send updates to Yew which writes them to an Emscripten filesystem mounted via IDBFS. This is a 3.2G file, so I can't just read the whole thing into memory at once. At the moment I have this hackery:
let write_to_map_upload = |bytes: Vec<u8>| self.update(Message::WriteToMapUpload(bytes));
js! {
var writeToMapUpload = @{write_to_map_upload};
var index = 0;
var offset = 1000000;
var file = document.querySelector("#import").files[0];
while(index < file.size) {
var slice = file.slice(index, offset);
var reader = new FileReader();
reader.addEventListener("loadend", () => writeToMapUpload(reader.result));
reader.readAsArrayBuffer(slice);
index += offset
}
writeToMapUpload.drop();
};
But that throws:
error[E0477]: the type `[closure@src/web/mod.rs:90:43: 90:105 self:&mut &mut web::Model]` does not fulfill the required lifetime
...
= note: type must satisfy the static lifetime
I feel like I could potentially bypass this by staying in Rust for longer, rather than dropping to JS, then maybe implementing slice on Blob if it isn't already there. But I'm kind of at a loss as to how I'd wrangle this large file's bytes into something I can read from JS and write to a Rust file handle.
Sorry for using the issue tracker in this way, Gitter seems pretty quiet. I'd love to contribute more, I'm just having a hard time wrapping my head around the different levels of interop going on here. :)
Thanks.
@ndarilek Converting your snippet to mostly Rust should be doable; you can grab the FileList though js! (I've just pushed a commit which exposes the FileList), and the only other thing for which there are no bindings is the File::slice method (should be trivial to add to stdweb or just simply called through js!).
You will still have to deal with needing a closure which is 'static though; as far as I can see you're passing &self into it (to call update), which is why it doesn't compile. I'm not super familiar with yew, so I'm not sure what you can do here, but you need to replace that with something that has a 'static lifetime (e.g. a reference counted handle).
Also, you probably don't want to have a busy loop in there - what you could do here is to slice something like a megabyte (or a few), wait for LoadEndEvent, process that megabyte chunk, and then queue it up again.
I implemented a specialized FileInput type in a branch which works, but still needs additional work based on feedback I received. I’d like to address the feedback and submit a PR, but am going to be pretty busy for a few weeks as I move. On Wed, Jun 20, 2018 at 4:09 PM Koute [email protected] wrote:
@ndarilek https://github.com/ndarilek Converting your snippet to mostly Rust should be doable; you can grab the FileList though js! (I've just pushed a commit which exposes the FileList), and the only other thing for which there are no bindings is the File::slice method (should be trivial to add to stdweb or just simply called through js!).
You will still have to deal with needing a closure which is 'static though; as far as I can see you're passing &self into it (to call update), which is why it doesn't compile. I'm not super familiar with yew, so I'm not sure what you can do here, but you need to replace that with something that has a 'static lifetime (e.g. a reference counted handle).
Also, you probably don't want to have a busy loop in there - what you could do here is to slice something like a megabyte (or a few), wait for LoadEndEvent, process that megabyte chunk, and then queue it up again.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/koute/stdweb/issues/226#issuecomment-398897713, or mute the thread https://github.com/notifications/unsubscribe-auth/AA80zF0LlszXpybj27qsElD6abC3cbLwks5t-roIgaJpZM4UQ4Zp .
Thanks. Where would I track LoadEndEvent in this instance? Would it be on the element where the file was added, the FileReader, etc.?
Also, how did you expose the FileList? At the moment I'm getting it via:
let element: InputElement = document().query_selector(
"#import" ).unwrap().unwrap().try_into().unwrap();
let files: FileList = js!(return
@{element}.files;).try_into().unwrap();
That seems to be working, but it sounds like you may have made the files
accessible without having to call .files in JS. Did I misread? I'm
using git master as of this morning.
Thanks.
Oh, NM on loadend, I'm looking at my old JavaScript code and see that I'm hooking it on FileReader. Thanks, still curious if it's possible to access the FileList without dropping to JS, but no big deal if I can't.
Hitting an odd issue with this. I have code that pulls out a file,
performs a slice, and uses LoadEndEvent to iteratively slice about
1000000 bytes, checkpointing it to IDBFS after 100000000 bytes are
processed. The results are written to a File stored using IDBFS.
First, the load of about 3.2 GB of data happens quickly, in a second or
less. This alone is suspect. Second, when I reload my app, I have a
function that lists a directory and outputs the filename/size. It looks
like only ~973K is written--at least, based on the file size. I have a
variable, offset, which I use to control the number of bytes read at a
time. If I drop offset from 1000000 to 100000, the number of bytes
written to the file drops to ~97.3K.
I inserted lots of logging throughout my function, and it appears that
my slice is 0-length after the first trip through the event-handler,
when it correctly has a len() of 1000000. It's as if the reference is
somehow getting lost after the first slice, and subsequent calls to
slice return a 0-length Blob.
Does anything about the following code look wrong? I'm doing some funny
things with Arc so I can use two copies of the same FileReader, but
I assume they're both referring to the same underlying object, so I
should be able to initiate reads and intercept events on both. I'm also
creating a separate Blob to see if that made a difference, but it
appears not to have. Initially I had a single mutable Blob that I just
reassigned, but I'm still getting 0-length slices.
I'd like to upstream some of this code once I figure it out--a
Blob.slice() implementation in particular. Thanks for any help.
Message::StartImport => {
let element: InputElement = document().query_selector(
"#import" ).unwrap().unwrap().try_into().unwrap();
let files: FileList = js!(return
@{element}.files;).try_into().unwrap();
let filename = match files.len() {
0 => None,
_ => files.iter().nth(0).map(|f| f.name()),
};
let mut fh = self.map_storage_directory.clone();
fh.push(filename.unwrap());
let mut upload = File::create(fh).expect("Failed to
create file for upload");
let file = files.iter().nth(0).unwrap();
let mut index = 0;
let size = file.len();
info!("Starting map import");
let mut count = 0;
let offset = 1000000;
let reader = Arc::new(FileReader::new());
let slice: Blob = js!(return @{&file}.slice(@{index as
u32}, @{offset as u32});).try_into().unwrap();
reader.read_as_array_buffer(&slice).expect("Failed to
read file");
let reader1 = reader.clone();
reader.add_event_listener(move |_: LoadEndEvent| {
match reader1.result().unwrap() {
FileReaderResult::ArrayBuffer(v) => {
let bytes: Vec<u8> = v.into();
info!("Writing bytes: {}", bytes.len());
upload.write_all(&bytes).expect("Failed to
write uploaded data");
upload.flush().expect("Failed to flush
uploaded data");
},
_ => info!("Should not happen"),
}
index += offset;
count += 1;
if count % 100 == 0 {
info!("Imported {} bytes, checkpointing", index);
upload.flush().expect("Failed to flush upload");
js! {
FS.syncfs(false, (err) => {
if(err)
console.error(err);
})
};
}
if index >= size {
upload.sync_all().expect("Failed to sync upload");
js!{
FS.syncfs(false, (err) => {
if(err)
console.error(err);
})
};
info!("Import complete");
} else {
let slice2: Blob = js!(return
@{&file}.slice(@{index as u32}, @{offset as u32});).try_into().unwrap();
reader1.read_as_array_buffer(&slice2).expect("Failed to read file");
}
});
},
@ndarilek I'm not seeing anything obviously wrong with it.
Some things you might try:
- Try to create a new
FileReaderfor every read. (I have no idea if that will change anything.) You could do that pretty easily by factoring out most of this code to a separate function and calling it recursively from inside the event handler. - Write a minimal reproduction crate which would just print the amount of bytes instead of saving it anywhere; that should be easier to debug.
I created a minimal reproduction using stdweb and yew if anyone is interested in helping me debug this:
https://github.com/ndarilek/file_upload_error
cargo web start, then visit localhost:8000. Upload a large file, preferably over 100 MB.
What's currently happening is that no data is being persisted to IDBFS, but the code outputs debugging data indicating that most slice operations are returning 0-length arrays, which they shouldn't be.
I don't know if something bad is happening with my use of Arc and RefCell, or if something else entirely is happening. It might be helpful, though, if stdweb both made the FileList accessible without dropping to JS, and maybe added some method to stream its contents to a fs::File.
Anyhow, I've spent hours banging my head on this one and am stuck. I'm trying to write a 3.3 GB Spatialite database file into a browser's local IDBFS storage, at which point I'd like to write and contribute a geolocation API that interacts with that data.
Thanks for any help.