async/await support and other changes for meteor@3

[dr-dimitru]

[dr-dimitru]

@jankapunkt created this PR to track changes inherited from #884

[jankapunkt]

@dr-dimitru so you think this is ready for an alpha release? We have two medium sized apps (100k loc) with lots of files involving which we can use to test

[dr-dimitru]

@jankapunkt I'm working on couple of more changes, planning to pre-release today. Testing from your end would be very helpful

[Friction-less-development]

@dr-dimitru Where is this pre-release? The latest I see is 3.0.0-beta.6 which does not seem to be up to date.

[dr-dimitru]

@Friction-less-development @jankapunkt v3.0.0-beta.7 is out, please give it a try

[Friction-less-development]

Thank you, I was also happy to see it kept legacy cursor methods on the front-end, alleviating some churn on that front.

I so far have seen one issue.

Bug: When calling removeAsync, the file is not being deleted from the server.

Background: I am running the new function, this.removeAsync in the onAfterUpload hook. (Basically doing some server-side checks, if they fail, I remove the file.)

Cause: Through debugging, I can see when it reaches this code, the files cursor is empty (And resolves to an empty array.) Presumably because the collection entries have been removed earlier in the code. So to fix, should either keep a copy of the collection for use when unlinking, or unlink earlier.

[dr-dimitru]

@Friction-less-development thanks a lot for reporting this one, ostrio:[email protected] is out with fix for removeAsync included, please give it a try

[dr-dimitru]

@jankapunkt @bratelefant @harryadel @StorytellerCZ here's draft changelog for ostrio:[email protected], going to refactor and update docs now, please report issues if any.

Summary

• ✨ Refactor: Hook options: protected, onBeforeRemove, onAfterRemove, onInitiateUpload, onAfterUpload, namingFunction are now async
• 🤝 Refactor: Compatibility with meteor@3 and other modern packages
• ☄️ Refactor: Match FilesCollection APIs with new *Async methods of Mongo.Collection; Deprecate callback APIs on the Server
• 👨‍💻 Refactor: Utilize node's async APIs where suitable
• 👨‍💻 Refactor: Improve pause/resume logic on connection interruption/reconnect
• 📔 Docs: Updated and refactored docs with better examples
• 📔 Docs: Refactored JSDoc matching definitions in TypeScript
• 🤓 Dev: Improved TypeScript support
• 👷‍♂️ Dev: Improved debugging logs
• 👨‍🔬 Tests: Improved test-suite
• 👷‍♂️ Git: CI GitHub Action Workflows for lint and build tests

Major changes

FilesCollection:

• ⚠️ FilesCollection#remove() — deprecated on server, use FilesCollection#removeAsync instead
• ⚠️ FilesCollection#findOne() — deprecated on server, use FilesCollection#findOneAsync instead
• ⚠️ FilesCollection#unlink() — deprecated on server, use FilesCollection#unlinkAsync instead
• ⚠️ FilesCollection#write() — deprecated on server, use FilesCollection#writeAsync instead
• ⚠️ FilesCollection#load() — deprecated on server, use FilesCollection#loadAsync instead

FileCursor:

• ⚠️ FileCursor#remove() — deprecated on server, use FileCursor#removeAsync instead

FilesCursor:

• ⚠️ FilesCursor#remove() — deprecated on server, use FilesCursor#removeAsync instead
• ⚠️ FilesCursor#hasNext() - deprecated, use FilesCursor#hasNextAsync instead
• ⚠️ FilesCursor#count() - deprecated, use FilesCursor#countDocuments instead
• ⚠️ FilesCursor#countAsync() - deprecated, use FilesCursor#countDocuments instead

FileUpload:

• ⚠️ FileUpload#start() is now async!

Callbacks and hooks:

• ⚠️ Anywhere: this.user() is deprecated, use this.userAsync() instead
• ⚠️ Client: FileUpload now always triggers end even in the case of successful and failed uploads; Before: end event wasn't called under certain conditions
• ⚠️ Client: All errors appeared during upload in all hooks and events of FileUpload are now instance of Meteor.Error; Before: Errors had mixed type or were simply text
• ⚠️ Client: Errors are the same now (type, code, text, reason, details) within DDP and HTTP protocols; Before: DDP and HTTP protocols had different errors
• ⚠️ Client: The next private events were removed from UploadInstance Class: upload, sendEOF, prepare, sendChunk, proceedChunk

New methods

FilesCollection:

• ✨ Client: FilesCollection#insertAsync()
• ✨ Anywhere: FilesCollection#updateAsync()
• ✨ Anywhere: FilesCollection#removeAsync()
• ✨ Anywhere: FilesCollection#findOneAsync()
• ✨ Anywhere: FilesCollection#countDocuments()
• ✨ Anywhere: FilesCollection#estimatedDocumentCount()
• ✨ Server: FilesCollection#unlinkAsync()
• ✨ Server: FilesCollection#writeAsync()
• ✨ Server: FilesCollection#loadAsync()

FileCursor:

• ✨ Anywhere: FileCursor#removeAsync()
• ✨ Anywhere: FileCursor#fetchAsync()

FilesCursor:

• ✨ Anywhere: FilesCursor#getAsync()
• ✨ Anywhere: FilesCursor#hasNextAsync()
• ✨ Anywhere: FilesCursor#nextAsync()
• ✨ Anywhere: FilesCursor#hasPreviousAsync()
• ✨ Anywhere: FilesCursor#previousAsync()
• ✨ Anywhere: FilesCursor#removeAsync()
• ✨ Anywhere: FilesCursor#fetchAsync()
• ✨ Anywhere: FilesCursor#firstAsync()
• ✨ Anywhere: FilesCursor#lastAsync()
• ✨ Anywhere: FilesCursor#countDocuments()
• ✨ Anywhere: FilesCursor#forEachAsync()
• ✨ Anywhere: FilesCursor#eachAsync()
• ✨ Anywhere: FilesCursor#mapAsync()
• ✨ Anywhere: FilesCursor#currentAsync()
• ✨ Anywhere: FilesCursor#observeAsync()
• ✨ Anywhere: FilesCursor#observeChangesAsync()

New features

• ✨ Client: FileUpload#remainingTime ReactiveVar — returns remaining upload time in hh:mm:ss format;

Other changes

• 🔧 Security: Fixed #894 — now x_mtok cookie is set with secure and httpOnly flags
• 🔧 Refactor: Fixed FileCursor#with implementation
• ✨ Refactor: FileUpload#abort is now async
• ✨ Server: FilesCollection#addFile is now async
• ✨ Server: FilesCollection#download is now async
• ✨ Server: WriteStream Class is now available for import
• ⚙️ Client: disableUpload option processed on the client and returns error in the end/error events, and onError hooks. Before — throws an error upon calling .insert() method

Dependencies

release:

• [email protected], was 4.0.7
• removed: abort-controller, now using native AbortController
• removed: fs-extra, now using native fs

dev:

• added: [email protected]
• added: [email protected]

[harryadel]

I've just used the latest RC in our project. The only thing we had to change really was unlink to unlinkAsync which was about 3 calls. So far so good, I'll report back if I find any issues in the upcoming days. Thank you all for pushing this through.

[dr-dimitru]

@harryadel awesome, thank you for feedback. Do you use protected other this.user() in any callback's contexts? Have some reports that this one is broken

[harryadel]

@harryadel awesome, thank you for feedback. Do you use protected other this.user() in any callback's contexts? Have some reports that this one is broken

No, I don't think we really do.

[senad-zaimovic]

Any update on this? Is there an ETA for official release?

[fidelsam1992]

@harryadel awesome, thank you for feedback. Do you use protected other this.user() in any callback's contexts? Have some reports that this one is broken

@dr-dimitru We have been testing the protected flag to only allow authenticated users to login, but it is not working properly. (userId is always null)

Reason:

• x_mtok cookie is not received on the request headers on the server side
• None of those variables has a trace of the x_mtok cookie

http.request.headers
http.request.cookies
http.request.Cookies.get("x_mtok"));

• after removing the httpOnly flag on the setTokenCookie method, it works fine

[jankapunkt]

I am running a few tests right now with rc 3 and give you a summary tomorrow

[jankapunkt]

Why is onBeforeUpload not async as well? This prevents me from doing db calls, such as checking for Roles with the involved user or other npm tools that are async by default.

Edit: it seems the function is async but it's not documented anywhere.

[make-github-pseudonymous-again]

Another issue with the x_mtok cookie: it does not seem to get reset on logout. The following test which passes on Meteor 2.16, fails with Meteor 3.0: https://github.com/infoderm/patients/blob/d99cb6e35b8b91deb5d9c1fda81da69189b018f5/imports/api/uploads.app-tests.ts#L20-L31.

EDIT: The root cause might actually be in how Meteor 3.x invalidates session ids, because in Meteor 2.16 the cookie is also left dangling but it does not allow to access the files.

EDIT (bis): Confirmed. Adding a 1 second delay between the logout action success and attempting to fetch the file solves the issue for that test. Root cause likely Meteor confirming the logout action on the client without having first properly destroyed the corresponding sessions on the server.

[harryadel]

@dr-dimitru I'm so glad you're back in the mix. Hopefully we can have a full release soon and get rid of the betas :clap: