Running tests uses 50.1GB on Ubuntu

[samuelcolvin]

Describe the bug

I just cloned datafusion and tried cargo t on my ubuntu desktop, to check things were working properly.

It crashed.

I restarted, and it seems datafusion is using 50.1GB to just run tests.

The examples directory seems to be the biggest culpret.

To Reproduce

Just clone and run cargo t on Ubuntu (no idea if this is limited to linux).

Expected behavior

Humm, I guess in principle this isn't a show stopper, but seems somewhat unfortunate.

If there's an easy/low impact way to reduce disk usage, it might be useful.

Additional context

No response

[jayzhan211]

I got 28GB after cargo clean and cargo t on macbook

[jcsherin]

On Ubuntu,

$ lsb_release -rc
Release:	22.04
Codename:	jammy

After cargo clean and running cargo t:

$ du -h -d2 target
4.0K	target/tmp
6.1G	target/debug/incremental
12M	target/debug/.fingerprint
337M	target/debug/build
16G	target/debug/deps
28G	target/debug/examples
50G	target/debug
50G	target

[samuelcolvin]

Not sure what I've done (except run tests more times):

Update, not using 103GB.

Seeing this nice error:

...
   Compiling datafusion v39.0.0 (/home/samuel/code/datafusion/datafusion/core)
error: failed to write to `/home/samuel/code/datafusion/target/debug/deps/rmetalvEs2G/lib.rmeta`: No space left on device (os error 28)

error: could not compile `datafusion` (lib) due to 1 previous error

[jayzhan211]

The more you run, the more artifacts in /target is. cargo clean is all you need.

[alamb]

Yeah, I think this obscene use of temp space is an artifact of rust (rather than an artifact of datafusion specifically)

[alamb]

It would also help if we consolidated some of the example binaries: https://github.com/apache/datafusion/issues/11172

[alamb]

I filed a few tickets to start consolidating the examples which i think will help:

• https://github.com/apache/datafusion/issues/13842
• https://github.com/apache/datafusion/issues/13844
• https://github.com/apache/datafusion/issues/13843