stdlib Suggestion: Open File for Appending

[jcalabro]

Hey there! Apologies if this has been discussed before; I did a search through Zig's issues but wasn't able to find anything on the topic.

I'd like to be able to call std.fs.openFileAbsolute (and similar functions) and by some mechanism supply O_APPEND or FILE_APPEND_DATA. For instance, I've got a log file on disk that I want to append to during restarts of my program; I don't want to truncate the file and start from scratch every time (truncating is the current behavior).

Currently, I think the only way to do this in the fs package. I can call os.open directly with these flags, but then I just get back an os.fd_t rather than a fs.File, which is not quite as nice to use. I'm pretty new to Zig, so apologies if there is a way to easily achieve what I want and I'm missing it.

Assuming that's not the case, I think there may be a minimally invasive way to get this done, though I'm definitely open to feedback: I propose adding a new enum field to fs.File.OpenFlags called TruncateMode or similar that has:

pub const TruncateMode = enum {
    truncate, // default
    append,
};

If we open a file with write capabilites (write_only or read_write) and append, then we'd also want to OR that with O.APPEND in fs.openFileZ (and its windows counterpart).

Thoughts? If this seems like a good approach, I'm happy to take a shot at implementing it. Thanks!

[Vexu]

fs.createFile does take fs.File.CreateFlags with truncate but setting it to false is currently a no-op so it defaults to truncating either way.

[jcalabro]

Ah good point, so perhaps adding a truncate: bool to fs.File.OpenFlags would be a better approach than an enum for consistency?

[Vexu]

.truncate = false should be fixed to not truncate the file.

[jcalabro]

I do see the issue on createFile, and I suppose I could repeatedly call that to get a File back, but do you think it is valuable to also add this behavior to the openFile calls?

I can submit a quick patch to fix the bug you mentioned, looks like a very quick fix.

[Vexu]

I don't have too strong of an opinion about it, I guess there could be value to asserting a file exists before appending to it?

[nektro]

var file = try std.fs.cwd().openFile(path, .{ .mode = .read_write });
var stat = try file.stat();
try file.seekTo(stat.size);

try file.writer().writeAll(); // this will happen at the end of the file

[jcalabro]

@nektro thanks for that example! It definitely does what I want.

The suggestion of this issue is that, in addition, since O_APPEND exists in the Linux open system call, it would be a nice feature in the zig std library to also have an append option on std.fs.File.OpenFlags.

If there's agreement that it would be a valuable addition, I'd be happy to take a shot at adding it! :smile:

[jcalabro]

Hi again @Vexu! I realized while implementing a test case for #14376 that we should be careful here.

I think this statement is incorrect: .truncate = false should be fixed to not truncate the file. Currently, .truncate = false does not truncate the file; it just opens a fp at the start of the file, and you're able to write over the data at the start of the file from there (or seek to a location and keep writing). So it's not truncating, it's just opening the file and pointing at the first byte. I think this is the intended behavior?

More generally though, I don't want to touch createFile. I'm not quite sure what O_APPEND would even mean in the context of createFile :smile:

Instead, I still wish to complete the work I outlined in the original post of this issue. I want to add the ability to open a file in "truncate mode", "append mode", or "start of file" mode. Basically, I want more similarity to what is supplied by the open system call.

[praschke]

@nektro Unix filesystem attributes/flags (including Mac) and NT, POSIX, and NFSv4 ACLs all have specific append permissions, where opening in write mode can fail.

[kuon]

I agree with @jcalabro in what we need to have separate truncate or append.

Maybe we can take some inspiration from the rust approach? https://doc.rust-lang.org/std/fs/struct.OpenOptions.html

Basically, the rust fs open options has the following booleans:

read
write
append
truncate
create
create_new

Also, as @praschke said, it is possible to have only append permission on a file, (without read and write), which needs to be exposed because it is a common use for writing to log files. With the above, it would be append = true with all the other flags to false.

[kuon]

While speaking of this, there is also an append permission in NFSv4 ACL for directories that give permission to only create sub-directories and not files. This is not directly related to this issue, but I thought I'd mention it.

[andrewrk]

I'm not convinced that the standard library cross-platform abstractions should support the use case of append mode, and so I have changed this from bug to proposal and deprioritized it. Note that I still expect std.posix (#5019) to have this functionality.

[karlseguin]

Until this is decided, can the truncate field of File.CreateFlags be removed if it's being ignored?

[Calder-Ty]

var file = try std.fs.cwd().openFile(path, .{ .mode = .read_write });
var stat = try file.stat();
try file.seekTo(stat.size);

try file.writer().writeAll(); // this will happen at the end of the file

I want to note that this solution is susceptible to race conditions. If you want to avoid that you will have to go to they underlying system calls, as I don't see a way to atomically append to a file using the openFile API.

[mnemnion]

I don't see why this would not be supported cross-platform. fopen in Windows supports append mode, WASI has append-via-stream, and of course this is standard on any POSIX system.

I would be most surprised to find a platform which a) has a concept of a file and b) does not support appending to it. So I don't understand why std.io.File would not support it also, I personally find that baffling and spent an hour or more trying to figure out how to do it before searching for the question and finding this issue.

It's the standard thing to do for logs, so people are going to be barking their shins on this lack more or less indefinitely until the feature is added.

[faustinoribeiro]

Like others here, I am surprised that there is no "append" option when opening a file. I am learning zig and I spent about an hour reading the standard library documentation looking for that append option that exists in pretty much every language I used in my career. I am even sure it was available in the early 90's when I wrote my first - and unfortunately last - large C++ application! That doesn't give me a lot of confidence in the language. I enjoyed the language so far, but I barely started on the learning path, and this makes me wonder what other useful (should I say expected/standard?) features are missing.

[ViktorTrojan]

I find it more of a problem due to lack of good documentation. It just makes it difficult to find the right information. I don't think we really need 10 ways to do the same thing; instead, it should be documented easily and made findable.

There is a seekFromEnd function that does what you want, but nobody mentioned that.

For example:

const file = try std.fs.cwd().openFile(path, .{ .mode = .read_write });
try file.seekFromEnd(0); // now it's "appending"
try file.writer().print("Appending\n", .{});
try file.writer().print("Still Appending ...\n", .{});

[GalaxySnail]

There is a seekFromEnd function that does what you want, but nobody mentioned that.

No, this isn't what O_APPEND does. According to the manual open(2):

The modification of the file offset and the write operation are performed as a single atomic step.

[secrary]

Both seekTo and seekFromEnd have a race and another writer can overwrite the data. Isn't better to have an append mode that guarantees that writes will be placed at the end of file, even when there are other threads appending to the same file? In general, what's recommended way to append files?

[matklad]

In general, what's recommended way to append files?

Use platform specific API. Here's an example for POSIX:

const std = @import("std");

test "smoke" {
    const fd = try std.posix.open("log", .{
        .CREAT = true,
        .APPEND = true,
        .ACCMODE = .WRONLY,
    }, 0o666);
    defer std.posix.close(fd);
    _ = try std.posix.write(fd, "line\n");
}

[gegogi]

Apps like Mercurial SCM is dependent on atomicity of file append semantics when a local repo is share by multiple users. https://wiki.mercurial-scm.org/Presentations?action=AttachFile&do=get&target=ols-mercurial-paper.pdf And std.posix.open() is not supported on Windows. So there is no way to do this on Windows. I hope std.fs supports a platform independent way of doing this.

[Calder-Ty]

Apps like Mercurial SCM is dependent on atomicity of file append semantics when a local repo is share by multiple users. https://wiki.mercurial-scm.org/Presentations?action=AttachFile&do=get&target=ols-mercurial-paper.pdf And std.posix.open() is not supported on Windows. So there is no way to do this on Windows. I hope std.fs supports a platform independent way of doing this.

I haven't tested this out, as I don't have a machine to do so, but you should be able to open for append on windows using std.os.windows.OpenFile. Have the access_mask have FILE_APPEND_DATA value.

https://ziglang.org/documentation/0.14.0/std/#std.os.windows.OpenFileOptions