Chunk Dictionary Grows with Failed uploads

[agostnello]

uploadChunkDictionary.Remove(chunkMeta.FlowIdentifier);

Is never run for failed uploads. Is there a way to have it remove ones after some period of time.

[samhowes]

There is actually, I have changed my production version to use a MemoryCache in System.Runtime.Caching. You can use a memory cache just like a dictionary, but when you add an item to the cache, you specify a "CacheItemPolicy"

// Use this instead of uploadChunkDictionary
public static readonly MemoryCache Cache = MemoryCache.Default;
...

if (addNewEntry)
{
     var cachePolicy = new CacheItemPolicy()
     {
            SlidingExpiration = new TimeSpan(0, uploadConfig.UploadTimeoutMinutes, 0, 0)
      };
      Cache.Add(flowIdentifier, fileMetaData, cachePolicy);
 }

//To retrieve an item: 
var fileMetadata = Cache[flowIdentifier] as FileMetaData;
if (fileMetadata == null) {

This way, the memory will get discarded after a period of time.

There are a couple other cache classes that you can use, but MemoryCache is the recommended one by Microsoft. Though I can't find the article right now.

I'll make a commit later today to put these changes in this project as well.

Hope this makes sense!

[agostnello]

Thank you very much. It works really well. Are there any other improvements?

I was wondering about 2 thing public override async Task ExecutePostProcessingAsync() { } This empty function override for the stream provider does that need to be there?

I am actually uploading 8 and 10 gig files with it and it works great I had change the size variables to long and int64. I am actually saving the file and then fowarding it on to a couchdb attachment.

Should I do that from RegisterSuccessfulChunk when this if is true if (fileMeta.IsComplete) { uploadChunkDictionary.Remove(chunkMeta.FlowIdentifier); right here ---- }

Thanks again for your help.

[agostnello]

The one other issue does your other version have support for the flow query paramter where you can get other form data posted?

[samhowes]

I'm glad to hear it is working well for you! There haven't been any core improvements related to the mechanics of Flowjs. In my production version I use authentication to track uploads for each user in FlowUploadProcessor, that's really the only modifications that I've made to the base implementation.

One general change that I made: I named an in progress upload with the extension '.inprogressupload' and then rename the file when it is completed. Just to know at the file level if it has completed.

I'm open to any suggestions though!

Hopefully commit 1c72484 will answer your question about the empty function override. It is hard to remember, but I think if I didn't override that method when I was first developing it, then an exception would get thrown because of something the base method was trying to do.

I hadn't actually worked with an upload of > 2GB until very recently, at which point I also found out that Int32 just wasn't enough (duh!). I have made that change here too.

---

For forwarding the file to a you can essentially do it right there where you were putting it with that type of logic:

if (fileMeta.Iscomplete)
{
    // Persist information about the upload completion
}

However, since we are holding a lock (uploadCacheLock) at that exact point in the code, executing an asynchronous I/O task there would might result in a performance decrease/unexpected results. What I have chosen to do in my own usage is to instead do this with a property "IsComplete" and execute the task in the Controller. In my opinion, this is better separation of concerns. I have added code to support this mechanism in commit 03badaf

I haven't spent any time on the Flow Query parameter for additional data. However, I'm sure it won't be hard to do with FlowMultipartFormDataStreamProvider. I created Issue #4 for that and I'll get back to you there!