PlayForm
[performance] Compress in parallel
Love this package!
But not the performance, especially when using it with lots of image files :(
It doesn't seem to be compressing in parallel.
A "make parallel" improvement to Astro's build command was made here. Perhaps something similar could be done here to improve the performance.
From a quick look at the code, perhaps the current bottleneck somewhere here?
Happy to take a stab at making some improvements on this once I can free-up some time.
For context, I'm seeing astro-compress take around 30 minutes to compress ~1,000 images (on an M1 Max).
1,000 images might seem like a lot, but current responsive/retina best practices result in lots of image variants (the site in reference here only has 36 source images).
Thank you, I've been wondering about how to make it compress in parallel!
So I did a benchmark with three different methods https://github.com/nikolaxhristov/bench-piscina I think piscina might be the best for this.
@nikolaxhristov I would love to help out with this if it's something you want help with.
@johnzanussi TY! ❤️ I would love some help. I wanted to make the files-pipe component itself parallelized, so whatever you use in callbacks gets called parallelly. This is used by astro-compress, astro-rome, astro-critters and a few other extensions, so it would greatly benefit the whole ecosystem.
Ideally, this would be parallelized https://github.com/lightrix/files-pipe/blob/main/src/lib/files/pipe.ts
Of all my benchmarks I stopped at piscina https://github.com/nikolaxhristov/bench-piscina, but if you have other suggestions, I'm open.
I have a repo that's a bit unmaintained at the moment to test those exact extensions https://github.com/nikolaxhristov/bench and the files-pipe component itself.
Is there a way to cache previously compressed files? It would help out rather than making things parallel, as I think that is much harder
@kyeshmz See: https://github.com/astro-community/AstroCompress/issues/49
It would be nice to be able to run the compressor in parallel. I have a lot of images in my project too and the application build currently takes 26 minutes
