Docusaurus bundle size heavily impacted by number of routes & source path length

[NEnochK]

Have you read the Contributing Guidelines on issues?

• [X] I have read the Contributing Guidelines on issues.

Prerequisites

• [X] I'm using the latest version of Docusaurus.
• [ ] I have tried the npm run clear or yarn clear command.
• [ ] I have tried rm -rf node_modules yarn.lock package-lock.json and re-installing packages.
• [ ] I have tried creating a repro with https://new.docusaurus.io.
• [ ] I have read the console error message carefully (if applicable).

Description

Discussion from the Docusaurus Discord server

tl;dr The number of routes/pages is directly impacting the bundle size of a Docusaurus build and it seems there are opportunities to further optimize this.

I unfortunately can't share my project/output here, but I am running into an issue where my site with ~4k pages is very slow on initial site load. After initial investigation, I found that the bundled main.xxxx.js file was (~10mb) and was littered with strings of the file paths for all the pages in our project (e.g. /docs/path/to/file.md). One optimization we made was to dump all markdown files into /docs and rely on slugs in order to shorten the length of the path string (e.g. /docs/path/to/file.md to /docs/file.md). This reduced it down to ~4mb but this is still too large. Ideally the number of routes/paths wouldn't be a bottleneck to the framework.

@Josh-Cena mentioned a possible optimization to split up registry.js/routesChunkNames.json into multiple JSON files and lazy load each only when on the path

Reproducible demo

No response

Steps to reproduce

1. Create a project with a large amount of routes/pages in the order of thousands.
2. Do a production build.
3. Look at the built main.xxxx.js under build/assets/js directory
4. Note the binary size (for me was a couple mb's) and that the file is littered with the file paths of the markdown files
5. Serve the build and note that it takes a long time to load the page initially due to this large main.xxxx.js

Expected behavior

Expect the site to load quickly regardless of the number of pages

Actual behavior

The initial load of the site is hindered by this large main.xxxx.js

Your environment

No response

Self-service

• [ ] I'd be willing to fix this bug myself.

[Josh-Cena]

TBF, the number routes would always impact the entry point size, because the router has to know all available routes upfront. However, including the file paths in the bundle seems suspicious—maybe we can find a way to prune that.

[slorber]

yes, there are a few things we can improve to make the bundle size smaller.

we'll improve that over time

[Glup3]

any updates or new ideas? i stumbled upon this issue - i'm facing the same problem with 1000 routes and huge registry.js file (1 json file for each route)

[richard-lopes-ifood]

Lazy loading the pages isn't an option?

[Josh-Cena]

@richard-lopes-ifood Depends on what you mean by "lazy load". The router has to know certain metadata about the routes available to be able to load them at all, but all substantial logic of each page (its React components, for example) is only loaded when you are about to navigate to it.

[fgatti675]

Thanks to this issue I discovered why my performance was so bad. I autogenerate 2k pages for our API, and have 3 versions. So 6k+ routes. It was not even building anymore :(