speed up compilation

[jeremieamstutz]

Hello,

At my company, we manage a dataform-based data warehouse with over 5,000 tables. At this scale, dataform compilation takes 30–50 seconds per change, leading to frequent timeouts and a really poor developer experience.

To work around this, we split our monolithic repo into separate ones (e.g., raw vs. business), and now by team. While this helped marginally, it has made orchestration and navigation a lot harder. We ended up building our own orchestrator, dependency graph, and pooling logic — complex and redundant work.

My wish would therefore be that the compilation time is extensively reduced so that we can go back to a monorepo.

Suggestions

Profile and Optimize Compilation: investigate performance hotspots during compilation to identify optimizations.

Incremental Compilation Support: cache compiled outputs with hashes and only recompile changed files. dbt is already exploring a similar approach (issue #3121).

Thanks for your help!