Does using include_bytes! for static files impact incremental compile times?

[9876691]

I like the idea of using build.rs to implement cache busting for static files.

I've used a hand coded technique similar to Ructe but instead of including the file in the final build executable they get loaded by the web server instead.

Is it possible to setup Ructe to work this way. So for example

statics::favicon_png.name would still give me the name with a hash

But when accessing the file later on

data = templates::statics::StaticFile::get(&name)

I would want data.file_name instead of data.content

And with the original file name I can let actix or whatever load the file and return the contents.

So the executable won't have all the static files embedded.

My reasoning for this is that I suspect it will have an impact on incremental build times, but I'm not sure how to measure that.

Thanks

[kaj]

I don't think static file data has any big impact on compile times. The files has to be read at compile-time anyway to implement the hash in the file name (and if the file is read from disk rather than included in the binary, there is no guarantee that the hash is updated correctly).

In an example project https://github.com/kaj/r4s , the build time is about 3.5 seconds when there is a change in a static file, and about 6.5 seconds for a whitespace change in a rust source file. I believe most of the time in both cases are actually from the warp routing, which is known to cause long build times due to complex use of implied typing.

Any experiment and measurments is of course welcome!

[9876691]

I don't think static file data has any big impact on compile times.

I've done some testing on this and it doesn't appear to affect compile times.

To test this I generate the Ructe statics.rs and then make the following changes in my build.rs to do a sed on the code and twist it into not using include_bytes!.

    // s/pub content: \&\x27static \[u8\]/pub file_name: \&\x27static str/g
    let output = std::process::Command::new("sed")
        .arg("-i")
        .arg(r"s/pub content: \&\x27static \[u8\]/pub file_name: \&\x27static str/g")
        .arg(file_path.clone())
        .output()?;
    if !output.status.success() {
        panic!("{}", &std::str::from_utf8(&output.stderr).unwrap());
    }

    // s/content: include_bytes!(/file_name: /g
    let output = std::process::Command::new("sed")
        .arg("-i")
        .arg(r"s/content: include_bytes!(/file_name: /g")
        .arg(file_path.clone())
        .output()?;
    if !output.status.success() {
        panic!("{}", &std::str::from_utf8(&output.stderr).unwrap());
    }

    // s/),/,/g
    let output = std::process::Command::new("sed")
        .arg("-i")
        .arg(r"s/),/,/g")
        .arg(file_path)
        .output()?;
    if !output.status.success() {
        panic!("{}", &std::str::from_utf8(&output.stderr).unwrap());
    }

So my statics.rs looks something like

...
#[allow(dead_code)]
pub struct StaticFile {
    pub file_name: &'static str,
    pub name: &'static str,
    pub mime: &'static Mime,
}
...
/// From "/workspace/app/dist/index.css"
#[allow(non_upper_case_globals)]
pub static index_css: StaticFile = StaticFile {
  file_name: "/workspace/app/dist/index.css",
  name: "index-HWcrt1Fx.css",
  mime: &mime::TEXT_CSS,
};

So it would be relatively easy to get Ructe to create both styles of statics.rs based on a config paramter.

I think the above style is useful if people have large files in their statics and don't want those loaded into memory.

I could raise a PR for the above. What do you think?

[kaj]

Could be useful. But will the full path to the file be the same on the deployed server as in the build environment? Or would some kind of dist-dir parameter somewhere be necessary?

A similar possibility may be to not include the file_name (or content) in the StaticFile struct at all, but instead copy all static files to their name (with hash) in a dist directory (maybe target/dist when building). Then either the application web server or a front proxy (such as nginx, h2, or traefik) can get a copy of that directory in a known location and serve the static files from there.

[9876691]

Could be useful. But will the full path to the file be the same on the deployed server as in the build environment? Or would some kind of dist-dir parameter somewhere be necessary?

So currently adding files in ructe looks like the following for me.

    let out_dir = env::var_os("OUT_DIR").unwrap();
    let path_buf = PathBuf::from(format!("{}/ructe", &out_dir.into_string().unwrap()));
    let mut ructe = Ructe::new(path_buf).unwrap();
    let mut statics = ructe.statics().unwrap();
    statics.add_files("dist").unwrap();
    statics.add_files("asset-pipeline/images").unwrap();

add_files is relative to the build directory. so if file_name was also relative rather than absolute it might help.

In either case when packaging a rust web server and assets into a docker container one has to be careful about where the files are placed.

A similar possibility may be to not include the file_name (or content) in the StaticFile struct at all, but instead copy all static files to their name (with hash) in a dist directory (maybe target/dist when building). Then either the application web server or a front proxy (such as nginx, h2, or traefik) can get a copy of that directory in a known location and serve the static files from there.

That would also work.