ziglang
fs.path: Fix Windows path component comparison being ASCII-only
We can use RtlEqualUnicodeString via eqlIgnoreCaseWTF16 to get Unicode-aware case insensitive path component comparison (but need to convert the components to UTF-16 first)
Addresses some TODOs I noticed while looking at the path.zig code. For some more context about RtlEqualUnicodeString, see https://github.com/ziglang/zig/pull/10595.
Is std.fs.path.relativeWindows meant to be callable from non-Windows platforms? Currently that seems to be the case since it's marked pub and the test case runs both relativeWindows and relativePosix on all platforms.
If we want to keep that being true, then this particular fix can't be applied since it means relativeWindows will depend on ntdll.
Yes it is meant to be. Relying on RtlEqualUnicodeString seems problematic in general because:
- If it depends on global OS state, then it should not be used in most cases that
std.fs.path.relativeWindowsis used. - If it does not depend on global OS state, then there is no reason for it to be in a separate DLL since that pure functional logic could be implemented in zig std lib and provide better insight as to what is possible, be callable at compile-time, be callable from other host operating systems, and in general avoid a DLL call for simple string manipulation.
I understand that in practice we may need to use it regardless, and I would value your recommendation in particular.
Are you sure that eqlIgnoreCaseWTF16 cannot be implemented without a call to RtlEqualUnicodeString?
Are you sure that
eqlIgnoreCaseWTF16cannot be implemented without a call toRtlEqualUnicodeString?
It's definitely possible to implement on its own, but AFAIK that would involve pulling in Unicode data into the standard library which is something I've seen you mention a few times that you want to avoid. How much Unicode data it would need I'm unsure about (I think RtlEqualUnicodeString only does direct UTF-16 code unit comparison, so it might not do any normalization which may make the amount of Unicode data needed smaller than would be needed for full Unicode-aware case insensitive comparison; AFAIK the relevant Unicode data used by RtlEqualUnicodeString is within PEB.UnicodeCaseTableData but I'm unsure how large that data is in memory)
From https://github.com/ziglang/zig/commit/8bf7cffe29b520cda094756309bca1d354af0d6b:
It is not planned to support full Unicode case-insensitivity on Windows, and in fact relying on non-ASCII case-insensitive environment variables is fundamentally problematic.
From https://github.com/ziglang/zig/pull/5772#issuecomment-710737298:
This looks like it adds a dependency on Unicode data, which needs to go through the approval process before having a PR accepted for it. So far zig std lib does not have a dependency on any of the non-static components of unicode data (e.g. stuff that can change in future unicode updates). There have been several proposals rejected to add a dependency of the language and standard library on this.
See https://github.com/ziglang/zig/pull/10796#issuecomment-1124276074 for where this came up last time and it was decided that RtlEqualUnicodeString was acceptable (but in that instance it's used by an API that's Windows-only).
If it's only comparing UTF-16 codepoints, that's super easy to implement and does not require any unicode data, right? I think I'm missing something.
It needs a table of lowercase <-> uppercase conversions (only one direction is strictly necessary though).
For the ASCII range only? Or do they keep their unicode data up-to-date with standards changes? Wouldn't it be a problem if the unicode consortium changed the rule of whether something is uppercase or lowercase and caused files to now alias that didn't before?
For the ASCII range only?
For the range of U+0000 to U+FFFF.
I haven't investigated it too much, but I do know that codepoints that need to be encoded as a UTF-16 surrogate pair (>= U+010000) do not actually get compared case-insensitively. See ReactOS's implementation of RtlCompareUnicodeString which just compares the uppercased version u16-to-u16 (rather than codepoint-to-codepoint):
https://github.com/reactos/reactos/blob/3fa57b8ff7fcee47b8e2ed869aecaf4515603f3f/sdk/lib/rtl/unicode.c#L2197-L2200
My (total, with no basis) guess is that:
- The data in
PEB.UnicodeCaseTableDatachanges rarely, probably only on major Windows version updates if at all - The data in
PEB.UnicodeCaseTableDatais probably perennially out-of-date and maybe locked to an older version of the Unicode standard
If my guess is correct, then this might add some extra challenge if the goal is to match Windows behavior since that may be a moving target separate from the Unicode standard (but would need to introspect into the UnicodeCaseTableData on different Windows versions and then compare them with eachother and to the Unicode standard to know for sure)
Just to make things clearer, the use of RtlEqualUnicodeString allows these two UNC paths to be seen as the same from the perspective of std.fs.path.relativeWindows:
\\кириллица\ελληνικά\português
\\КИРИЛЛИЦА\ΕΛΛΗΝΙΚΆ\PORTUGUÊS
AFAICT the NLS Unicode tables are determined as follows (may be wrong, though):
- Query the
HKLM\SYSTEM\CurrentControlSet\Control\Nls\Languagekey for the value namedDefault. - Query the same key for the string data of
Default. If it exists, the filename is the newly retrieved value, otherwise the filename isl_intl.nls. - Read the NLS file from
%SystemRoot%\System32\{filename}. This is then stashed in the loader block and eventually plumbed through to the PEB on process creation.
For reference, l_intl.nls is 9926 bytes on Windows 11 22H2. I'm not sure how it changes across different Windows versions or in what situation the non-default (that is, Default - as in not l_intl.nls) file would be chosen.
Either way, it seems the exact Windows behavior would be pretty difficult or even impossible to replicate in a crossplatform manner. There's probably a good enough compromise solution, but in any case a few kB would still have to be spared for the Unicode data, that's just unavoidable.
Also it should be noted the actual table used for filesystem operations is the special NTFS file $UpCase, which is just a copy of the system-wide NLS data at the time the volume is created. This can have some interesting consequences (see also the comments there). How should Zig handle this?
Apparently the $UpCase/l_intl.nls data hasn't changed since Windows 8.1 (see the comments of the article posted previously by @e4m2; unsure about Windows 11, though), so it might be okay to use the current version of the data for now since Zig only officially supports Windows 8.1+.
I went ahead and wrote some code that generates a switch case for all lowercase -> uppercase u16 conversions using the data from PEB.UnicodeCaseTableData (should be the same as $UpCase/l_intl.nls for most setups AFAICT, I used PEB instead of reading l_intl.nls or just using RtlUpcaseUnicodeChar because I was curious about the PEB data):
Code to generate the function with the switch
const std = @import("std");
pub fn main() !void {
var stdout = std.io.getStdOut();
var buffered_stdout = std.io.bufferedWriter(stdout.writer());
var stdout_writer = buffered_stdout.writer();
var peb = std.os.windows.peb();
const base_case_table_data = @ptrCast([*]u16, @alignCast(@alignOf(u16), peb.UnicodeCaseTableData));
// See https://github.com/reactos/reactos/blob/3fa57b8ff7fcee47b8e2ed869aecaf4515603f3f/sdk/lib/rtl/nls.c#L222
const uppercase_table_data = @ptrCast([*]u16, &base_case_table_data[2]);
var i: u16 = 0;
try stdout_writer.writeAll("pub fn upcase(c: u16) u16 {\n return switch (c) {\n");
while (true) : (i += 1) {
const upper = upcase(i, uppercase_table_data);
if (upper != i) {
try stdout_writer.print(" 0x{X} => 0x{X},\n", .{ i, upper });
}
if (i == std.math.maxInt(u16)) break;
}
try stdout_writer.writeAll(" else => c,\n };\n}\n");
try buffered_stdout.flush();
}
/// Based on the ReactOS implementation of RtlpUpcaseUnicodeChar
/// https://github.com/reactos/reactos/blob/3fa57b8ff7fcee47b8e2ed869aecaf4515603f3f/sdk/lib/rtl/nls.c#L714-L735
fn upcase(c: u16, upper_case_table: [*]u16) u16 {
if (c < 'a') return c;
if (c <= 'z') return c - ('a' - 'A');
var offset: u16 = (c >> 8) & 0xFF;
offset = upper_case_table[offset];
offset += (c >> 4) & 0xF;
offset = upper_case_table[offset];
offset += c & 0xF;
offset = upper_case_table[offset];
// relies on wrapping overflow, e.g. the final offset could be 0xFFFF
return c +% offset;
}
The generated function with the switch
pub fn upcase(c: u16) u16 {
return switch (c) {
0x61 => 0x41,
0x62 => 0x42,
0x63 => 0x43,
0x64 => 0x44,
0x65 => 0x45,
0x66 => 0x46,
0x67 => 0x47,
0x68 => 0x48,
0x69 => 0x49,
0x6A => 0x4A,
0x6B => 0x4B,
0x6C => 0x4C,
0x6D => 0x4D,
0x6E => 0x4E,
0x6F => 0x4F,
0x70 => 0x50,
0x71 => 0x51,
0x72 => 0x52,
0x73 => 0x53,
0x74 => 0x54,
0x75 => 0x55,
0x76 => 0x56,
0x77 => 0x57,
0x78 => 0x58,
0x79 => 0x59,
0x7A => 0x5A,
0xE0 => 0xC0,
0xE1 => 0xC1,
0xE2 => 0xC2,
0xE3 => 0xC3,
0xE4 => 0xC4,
0xE5 => 0xC5,
0xE6 => 0xC6,
0xE7 => 0xC7,
0xE8 => 0xC8,
0xE9 => 0xC9,
0xEA => 0xCA,
0xEB => 0xCB,
0xEC => 0xCC,
0xED => 0xCD,
0xEE => 0xCE,
0xEF => 0xCF,
0xF0 => 0xD0,
0xF1 => 0xD1,
0xF2 => 0xD2,
0xF3 => 0xD3,
0xF4 => 0xD4,
0xF5 => 0xD5,
0xF6 => 0xD6,
0xF8 => 0xD8,
0xF9 => 0xD9,
0xFA => 0xDA,
0xFB => 0xDB,
0xFC => 0xDC,
0xFD => 0xDD,
0xFE => 0xDE,
0xFF => 0x178,
0x101 => 0x100,
0x103 => 0x102,
0x105 => 0x104,
0x107 => 0x106,
0x109 => 0x108,
0x10B => 0x10A,
0x10D => 0x10C,
0x10F => 0x10E,
0x111 => 0x110,
0x113 => 0x112,
0x115 => 0x114,
0x117 => 0x116,
0x119 => 0x118,
0x11B => 0x11A,
0x11D => 0x11C,
0x11F => 0x11E,
0x121 => 0x120,
0x123 => 0x122,
0x125 => 0x124,
0x127 => 0x126,
0x129 => 0x128,
0x12B => 0x12A,
0x12D => 0x12C,
0x12F => 0x12E,
0x133 => 0x132,
0x135 => 0x134,
0x137 => 0x136,
0x13A => 0x139,
0x13C => 0x13B,
0x13E => 0x13D,
0x140 => 0x13F,
0x142 => 0x141,
0x144 => 0x143,
0x146 => 0x145,
0x148 => 0x147,
0x14B => 0x14A,
0x14D => 0x14C,
0x14F => 0x14E,
0x151 => 0x150,
0x153 => 0x152,
0x155 => 0x154,
0x157 => 0x156,
0x159 => 0x158,
0x15B => 0x15A,
0x15D => 0x15C,
0x15F => 0x15E,
0x161 => 0x160,
0x163 => 0x162,
0x165 => 0x164,
0x167 => 0x166,
0x169 => 0x168,
0x16B => 0x16A,
0x16D => 0x16C,
0x16F => 0x16E,
0x171 => 0x170,
0x173 => 0x172,
0x175 => 0x174,
0x177 => 0x176,
0x17A => 0x179,
0x17C => 0x17B,
0x17E => 0x17D,
0x180 => 0x243,
0x183 => 0x182,
0x185 => 0x184,
0x188 => 0x187,
0x18C => 0x18B,
0x192 => 0x191,
0x195 => 0x1F6,
0x199 => 0x198,
0x19A => 0x23D,
0x19E => 0x220,
0x1A1 => 0x1A0,
0x1A3 => 0x1A2,
0x1A5 => 0x1A4,
0x1A8 => 0x1A7,
0x1AD => 0x1AC,
0x1B0 => 0x1AF,
0x1B4 => 0x1B3,
0x1B6 => 0x1B5,
0x1B9 => 0x1B8,
0x1BD => 0x1BC,
0x1BF => 0x1F7,
0x1C6 => 0x1C4,
0x1C9 => 0x1C7,
0x1CC => 0x1CA,
0x1CE => 0x1CD,
0x1D0 => 0x1CF,
0x1D2 => 0x1D1,
0x1D4 => 0x1D3,
0x1D6 => 0x1D5,
0x1D8 => 0x1D7,
0x1DA => 0x1D9,
0x1DC => 0x1DB,
0x1DD => 0x18E,
0x1DF => 0x1DE,
0x1E1 => 0x1E0,
0x1E3 => 0x1E2,
0x1E5 => 0x1E4,
0x1E7 => 0x1E6,
0x1E9 => 0x1E8,
0x1EB => 0x1EA,
0x1ED => 0x1EC,
0x1EF => 0x1EE,
0x1F3 => 0x1F1,
0x1F5 => 0x1F4,
0x1F9 => 0x1F8,
0x1FB => 0x1FA,
0x1FD => 0x1FC,
0x1FF => 0x1FE,
0x201 => 0x200,
0x203 => 0x202,
0x205 => 0x204,
0x207 => 0x206,
0x209 => 0x208,
0x20B => 0x20A,
0x20D => 0x20C,
0x20F => 0x20E,
0x211 => 0x210,
0x213 => 0x212,
0x215 => 0x214,
0x217 => 0x216,
0x219 => 0x218,
0x21B => 0x21A,
0x21D => 0x21C,
0x21F => 0x21E,
0x223 => 0x222,
0x225 => 0x224,
0x227 => 0x226,
0x229 => 0x228,
0x22B => 0x22A,
0x22D => 0x22C,
0x22F => 0x22E,
0x231 => 0x230,
0x233 => 0x232,
0x23C => 0x23B,
0x242 => 0x241,
0x247 => 0x246,
0x249 => 0x248,
0x24B => 0x24A,
0x24D => 0x24C,
0x24F => 0x24E,
0x250 => 0x2C6F,
0x251 => 0x2C6D,
0x253 => 0x181,
0x254 => 0x186,
0x256 => 0x189,
0x257 => 0x18A,
0x259 => 0x18F,
0x25B => 0x190,
0x260 => 0x193,
0x263 => 0x194,
0x268 => 0x197,
0x269 => 0x196,
0x26B => 0x2C62,
0x26F => 0x19C,
0x271 => 0x2C6E,
0x272 => 0x19D,
0x275 => 0x19F,
0x27D => 0x2C64,
0x280 => 0x1A6,
0x283 => 0x1A9,
0x288 => 0x1AE,
0x289 => 0x244,
0x28A => 0x1B1,
0x28B => 0x1B2,
0x28C => 0x245,
0x292 => 0x1B7,
0x371 => 0x370,
0x373 => 0x372,
0x377 => 0x376,
0x37B => 0x3FD,
0x37C => 0x3FE,
0x37D => 0x3FF,
0x3AC => 0x386,
0x3AD => 0x388,
0x3AE => 0x389,
0x3AF => 0x38A,
0x3B1 => 0x391,
0x3B2 => 0x392,
0x3B3 => 0x393,
0x3B4 => 0x394,
0x3B5 => 0x395,
0x3B6 => 0x396,
0x3B7 => 0x397,
0x3B8 => 0x398,
0x3B9 => 0x399,
0x3BA => 0x39A,
0x3BB => 0x39B,
0x3BC => 0x39C,
0x3BD => 0x39D,
0x3BE => 0x39E,
0x3BF => 0x39F,
0x3C0 => 0x3A0,
0x3C1 => 0x3A1,
0x3C3 => 0x3A3,
0x3C4 => 0x3A4,
0x3C5 => 0x3A5,
0x3C6 => 0x3A6,
0x3C7 => 0x3A7,
0x3C8 => 0x3A8,
0x3C9 => 0x3A9,
0x3CA => 0x3AA,
0x3CB => 0x3AB,
0x3CC => 0x38C,
0x3CD => 0x38E,
0x3CE => 0x38F,
0x3D7 => 0x3CF,
0x3D9 => 0x3D8,
0x3DB => 0x3DA,
0x3DD => 0x3DC,
0x3DF => 0x3DE,
0x3E1 => 0x3E0,
0x3E3 => 0x3E2,
0x3E5 => 0x3E4,
0x3E7 => 0x3E6,
0x3E9 => 0x3E8,
0x3EB => 0x3EA,
0x3ED => 0x3EC,
0x3EF => 0x3EE,
0x3F2 => 0x3F9,
0x3F8 => 0x3F7,
0x3FB => 0x3FA,
0x430 => 0x410,
0x431 => 0x411,
0x432 => 0x412,
0x433 => 0x413,
0x434 => 0x414,
0x435 => 0x415,
0x436 => 0x416,
0x437 => 0x417,
0x438 => 0x418,
0x439 => 0x419,
0x43A => 0x41A,
0x43B => 0x41B,
0x43C => 0x41C,
0x43D => 0x41D,
0x43E => 0x41E,
0x43F => 0x41F,
0x440 => 0x420,
0x441 => 0x421,
0x442 => 0x422,
0x443 => 0x423,
0x444 => 0x424,
0x445 => 0x425,
0x446 => 0x426,
0x447 => 0x427,
0x448 => 0x428,
0x449 => 0x429,
0x44A => 0x42A,
0x44B => 0x42B,
0x44C => 0x42C,
0x44D => 0x42D,
0x44E => 0x42E,
0x44F => 0x42F,
0x450 => 0x400,
0x451 => 0x401,
0x452 => 0x402,
0x453 => 0x403,
0x454 => 0x404,
0x455 => 0x405,
0x456 => 0x406,
0x457 => 0x407,
0x458 => 0x408,
0x459 => 0x409,
0x45A => 0x40A,
0x45B => 0x40B,
0x45C => 0x40C,
0x45D => 0x40D,
0x45E => 0x40E,
0x45F => 0x40F,
0x461 => 0x460,
0x463 => 0x462,
0x465 => 0x464,
0x467 => 0x466,
0x469 => 0x468,
0x46B => 0x46A,
0x46D => 0x46C,
0x46F => 0x46E,
0x471 => 0x470,
0x473 => 0x472,
0x475 => 0x474,
0x477 => 0x476,
0x479 => 0x478,
0x47B => 0x47A,
0x47D => 0x47C,
0x47F => 0x47E,
0x481 => 0x480,
0x48B => 0x48A,
0x48D => 0x48C,
0x48F => 0x48E,
0x491 => 0x490,
0x493 => 0x492,
0x495 => 0x494,
0x497 => 0x496,
0x499 => 0x498,
0x49B => 0x49A,
0x49D => 0x49C,
0x49F => 0x49E,
0x4A1 => 0x4A0,
0x4A3 => 0x4A2,
0x4A5 => 0x4A4,
0x4A7 => 0x4A6,
0x4A9 => 0x4A8,
0x4AB => 0x4AA,
0x4AD => 0x4AC,
0x4AF => 0x4AE,
0x4B1 => 0x4B0,
0x4B3 => 0x4B2,
0x4B5 => 0x4B4,
0x4B7 => 0x4B6,
0x4B9 => 0x4B8,
0x4BB => 0x4BA,
0x4BD => 0x4BC,
0x4BF => 0x4BE,
0x4C2 => 0x4C1,
0x4C4 => 0x4C3,
0x4C6 => 0x4C5,
0x4C8 => 0x4C7,
0x4CA => 0x4C9,
0x4CC => 0x4CB,
0x4CE => 0x4CD,
0x4CF => 0x4C0,
0x4D1 => 0x4D0,
0x4D3 => 0x4D2,
0x4D5 => 0x4D4,
0x4D7 => 0x4D6,
0x4D9 => 0x4D8,
0x4DB => 0x4DA,
0x4DD => 0x4DC,
0x4DF => 0x4DE,
0x4E1 => 0x4E0,
0x4E3 => 0x4E2,
0x4E5 => 0x4E4,
0x4E7 => 0x4E6,
0x4E9 => 0x4E8,
0x4EB => 0x4EA,
0x4ED => 0x4EC,
0x4EF => 0x4EE,
0x4F1 => 0x4F0,
0x4F3 => 0x4F2,
0x4F5 => 0x4F4,
0x4F7 => 0x4F6,
0x4F9 => 0x4F8,
0x4FB => 0x4FA,
0x4FD => 0x4FC,
0x4FF => 0x4FE,
0x501 => 0x500,
0x503 => 0x502,
0x505 => 0x504,
0x507 => 0x506,
0x509 => 0x508,
0x50B => 0x50A,
0x50D => 0x50C,
0x50F => 0x50E,
0x511 => 0x510,
0x513 => 0x512,
0x515 => 0x514,
0x517 => 0x516,
0x519 => 0x518,
0x51B => 0x51A,
0x51D => 0x51C,
0x51F => 0x51E,
0x521 => 0x520,
0x523 => 0x522,
0x561 => 0x531,
0x562 => 0x532,
0x563 => 0x533,
0x564 => 0x534,
0x565 => 0x535,
0x566 => 0x536,
0x567 => 0x537,
0x568 => 0x538,
0x569 => 0x539,
0x56A => 0x53A,
0x56B => 0x53B,
0x56C => 0x53C,
0x56D => 0x53D,
0x56E => 0x53E,
0x56F => 0x53F,
0x570 => 0x540,
0x571 => 0x541,
0x572 => 0x542,
0x573 => 0x543,
0x574 => 0x544,
0x575 => 0x545,
0x576 => 0x546,
0x577 => 0x547,
0x578 => 0x548,
0x579 => 0x549,
0x57A => 0x54A,
0x57B => 0x54B,
0x57C => 0x54C,
0x57D => 0x54D,
0x57E => 0x54E,
0x57F => 0x54F,
0x580 => 0x550,
0x581 => 0x551,
0x582 => 0x552,
0x583 => 0x553,
0x584 => 0x554,
0x585 => 0x555,
0x586 => 0x556,
0x1D79 => 0xA77D,
0x1D7D => 0x2C63,
0x1E01 => 0x1E00,
0x1E03 => 0x1E02,
0x1E05 => 0x1E04,
0x1E07 => 0x1E06,
0x1E09 => 0x1E08,
0x1E0B => 0x1E0A,
0x1E0D => 0x1E0C,
0x1E0F => 0x1E0E,
0x1E11 => 0x1E10,
0x1E13 => 0x1E12,
0x1E15 => 0x1E14,
0x1E17 => 0x1E16,
0x1E19 => 0x1E18,
0x1E1B => 0x1E1A,
0x1E1D => 0x1E1C,
0x1E1F => 0x1E1E,
0x1E21 => 0x1E20,
0x1E23 => 0x1E22,
0x1E25 => 0x1E24,
0x1E27 => 0x1E26,
0x1E29 => 0x1E28,
0x1E2B => 0x1E2A,
0x1E2D => 0x1E2C,
0x1E2F => 0x1E2E,
0x1E31 => 0x1E30,
0x1E33 => 0x1E32,
0x1E35 => 0x1E34,
0x1E37 => 0x1E36,
0x1E39 => 0x1E38,
0x1E3B => 0x1E3A,
0x1E3D => 0x1E3C,
0x1E3F => 0x1E3E,
0x1E41 => 0x1E40,
0x1E43 => 0x1E42,
0x1E45 => 0x1E44,
0x1E47 => 0x1E46,
0x1E49 => 0x1E48,
0x1E4B => 0x1E4A,
0x1E4D => 0x1E4C,
0x1E4F => 0x1E4E,
0x1E51 => 0x1E50,
0x1E53 => 0x1E52,
0x1E55 => 0x1E54,
0x1E57 => 0x1E56,
0x1E59 => 0x1E58,
0x1E5B => 0x1E5A,
0x1E5D => 0x1E5C,
0x1E5F => 0x1E5E,
0x1E61 => 0x1E60,
0x1E63 => 0x1E62,
0x1E65 => 0x1E64,
0x1E67 => 0x1E66,
0x1E69 => 0x1E68,
0x1E6B => 0x1E6A,
0x1E6D => 0x1E6C,
0x1E6F => 0x1E6E,
0x1E71 => 0x1E70,
0x1E73 => 0x1E72,
0x1E75 => 0x1E74,
0x1E77 => 0x1E76,
0x1E79 => 0x1E78,
0x1E7B => 0x1E7A,
0x1E7D => 0x1E7C,
0x1E7F => 0x1E7E,
0x1E81 => 0x1E80,
0x1E83 => 0x1E82,
0x1E85 => 0x1E84,
0x1E87 => 0x1E86,
0x1E89 => 0x1E88,
0x1E8B => 0x1E8A,
0x1E8D => 0x1E8C,
0x1E8F => 0x1E8E,
0x1E91 => 0x1E90,
0x1E93 => 0x1E92,
0x1E95 => 0x1E94,
0x1EA1 => 0x1EA0,
0x1EA3 => 0x1EA2,
0x1EA5 => 0x1EA4,
0x1EA7 => 0x1EA6,
0x1EA9 => 0x1EA8,
0x1EAB => 0x1EAA,
0x1EAD => 0x1EAC,
0x1EAF => 0x1EAE,
0x1EB1 => 0x1EB0,
0x1EB3 => 0x1EB2,
0x1EB5 => 0x1EB4,
0x1EB7 => 0x1EB6,
0x1EB9 => 0x1EB8,
0x1EBB => 0x1EBA,
0x1EBD => 0x1EBC,
0x1EBF => 0x1EBE,
0x1EC1 => 0x1EC0,
0x1EC3 => 0x1EC2,
0x1EC5 => 0x1EC4,
0x1EC7 => 0x1EC6,
0x1EC9 => 0x1EC8,
0x1ECB => 0x1ECA,
0x1ECD => 0x1ECC,
0x1ECF => 0x1ECE,
0x1ED1 => 0x1ED0,
0x1ED3 => 0x1ED2,
0x1ED5 => 0x1ED4,
0x1ED7 => 0x1ED6,
0x1ED9 => 0x1ED8,
0x1EDB => 0x1EDA,
0x1EDD => 0x1EDC,
0x1EDF => 0x1EDE,
0x1EE1 => 0x1EE0,
0x1EE3 => 0x1EE2,
0x1EE5 => 0x1EE4,
0x1EE7 => 0x1EE6,
0x1EE9 => 0x1EE8,
0x1EEB => 0x1EEA,
0x1EED => 0x1EEC,
0x1EEF => 0x1EEE,
0x1EF1 => 0x1EF0,
0x1EF3 => 0x1EF2,
0x1EF5 => 0x1EF4,
0x1EF7 => 0x1EF6,
0x1EF9 => 0x1EF8,
0x1EFB => 0x1EFA,
0x1EFD => 0x1EFC,
0x1EFF => 0x1EFE,
0x1F00 => 0x1F08,
0x1F01 => 0x1F09,
0x1F02 => 0x1F0A,
0x1F03 => 0x1F0B,
0x1F04 => 0x1F0C,
0x1F05 => 0x1F0D,
0x1F06 => 0x1F0E,
0x1F07 => 0x1F0F,
0x1F10 => 0x1F18,
0x1F11 => 0x1F19,
0x1F12 => 0x1F1A,
0x1F13 => 0x1F1B,
0x1F14 => 0x1F1C,
0x1F15 => 0x1F1D,
0x1F20 => 0x1F28,
0x1F21 => 0x1F29,
0x1F22 => 0x1F2A,
0x1F23 => 0x1F2B,
0x1F24 => 0x1F2C,
0x1F25 => 0x1F2D,
0x1F26 => 0x1F2E,
0x1F27 => 0x1F2F,
0x1F30 => 0x1F38,
0x1F31 => 0x1F39,
0x1F32 => 0x1F3A,
0x1F33 => 0x1F3B,
0x1F34 => 0x1F3C,
0x1F35 => 0x1F3D,
0x1F36 => 0x1F3E,
0x1F37 => 0x1F3F,
0x1F40 => 0x1F48,
0x1F41 => 0x1F49,
0x1F42 => 0x1F4A,
0x1F43 => 0x1F4B,
0x1F44 => 0x1F4C,
0x1F45 => 0x1F4D,
0x1F51 => 0x1F59,
0x1F53 => 0x1F5B,
0x1F55 => 0x1F5D,
0x1F57 => 0x1F5F,
0x1F60 => 0x1F68,
0x1F61 => 0x1F69,
0x1F62 => 0x1F6A,
0x1F63 => 0x1F6B,
0x1F64 => 0x1F6C,
0x1F65 => 0x1F6D,
0x1F66 => 0x1F6E,
0x1F67 => 0x1F6F,
0x1F70 => 0x1FBA,
0x1F71 => 0x1FBB,
0x1F72 => 0x1FC8,
0x1F73 => 0x1FC9,
0x1F74 => 0x1FCA,
0x1F75 => 0x1FCB,
0x1F76 => 0x1FDA,
0x1F77 => 0x1FDB,
0x1F78 => 0x1FF8,
0x1F79 => 0x1FF9,
0x1F7A => 0x1FEA,
0x1F7B => 0x1FEB,
0x1F7C => 0x1FFA,
0x1F7D => 0x1FFB,
0x1F80 => 0x1F88,
0x1F81 => 0x1F89,
0x1F82 => 0x1F8A,
0x1F83 => 0x1F8B,
0x1F84 => 0x1F8C,
0x1F85 => 0x1F8D,
0x1F86 => 0x1F8E,
0x1F87 => 0x1F8F,
0x1F90 => 0x1F98,
0x1F91 => 0x1F99,
0x1F92 => 0x1F9A,
0x1F93 => 0x1F9B,
0x1F94 => 0x1F9C,
0x1F95 => 0x1F9D,
0x1F96 => 0x1F9E,
0x1F97 => 0x1F9F,
0x1FA0 => 0x1FA8,
0x1FA1 => 0x1FA9,
0x1FA2 => 0x1FAA,
0x1FA3 => 0x1FAB,
0x1FA4 => 0x1FAC,
0x1FA5 => 0x1FAD,
0x1FA6 => 0x1FAE,
0x1FA7 => 0x1FAF,
0x1FB0 => 0x1FB8,
0x1FB1 => 0x1FB9,
0x1FB3 => 0x1FBC,
0x1FC3 => 0x1FCC,
0x1FD0 => 0x1FD8,
0x1FD1 => 0x1FD9,
0x1FE0 => 0x1FE8,
0x1FE1 => 0x1FE9,
0x1FE5 => 0x1FEC,
0x1FF3 => 0x1FFC,
0x214E => 0x2132,
0x2170 => 0x2160,
0x2171 => 0x2161,
0x2172 => 0x2162,
0x2173 => 0x2163,
0x2174 => 0x2164,
0x2175 => 0x2165,
0x2176 => 0x2166,
0x2177 => 0x2167,
0x2178 => 0x2168,
0x2179 => 0x2169,
0x217A => 0x216A,
0x217B => 0x216B,
0x217C => 0x216C,
0x217D => 0x216D,
0x217E => 0x216E,
0x217F => 0x216F,
0x2184 => 0x2183,
0x24D0 => 0x24B6,
0x24D1 => 0x24B7,
0x24D2 => 0x24B8,
0x24D3 => 0x24B9,
0x24D4 => 0x24BA,
0x24D5 => 0x24BB,
0x24D6 => 0x24BC,
0x24D7 => 0x24BD,
0x24D8 => 0x24BE,
0x24D9 => 0x24BF,
0x24DA => 0x24C0,
0x24DB => 0x24C1,
0x24DC => 0x24C2,
0x24DD => 0x24C3,
0x24DE => 0x24C4,
0x24DF => 0x24C5,
0x24E0 => 0x24C6,
0x24E1 => 0x24C7,
0x24E2 => 0x24C8,
0x24E3 => 0x24C9,
0x24E4 => 0x24CA,
0x24E5 => 0x24CB,
0x24E6 => 0x24CC,
0x24E7 => 0x24CD,
0x24E8 => 0x24CE,
0x24E9 => 0x24CF,
0x2C30 => 0x2C00,
0x2C31 => 0x2C01,
0x2C32 => 0x2C02,
0x2C33 => 0x2C03,
0x2C34 => 0x2C04,
0x2C35 => 0x2C05,
0x2C36 => 0x2C06,
0x2C37 => 0x2C07,
0x2C38 => 0x2C08,
0x2C39 => 0x2C09,
0x2C3A => 0x2C0A,
0x2C3B => 0x2C0B,
0x2C3C => 0x2C0C,
0x2C3D => 0x2C0D,
0x2C3E => 0x2C0E,
0x2C3F => 0x2C0F,
0x2C40 => 0x2C10,
0x2C41 => 0x2C11,
0x2C42 => 0x2C12,
0x2C43 => 0x2C13,
0x2C44 => 0x2C14,
0x2C45 => 0x2C15,
0x2C46 => 0x2C16,
0x2C47 => 0x2C17,
0x2C48 => 0x2C18,
0x2C49 => 0x2C19,
0x2C4A => 0x2C1A,
0x2C4B => 0x2C1B,
0x2C4C => 0x2C1C,
0x2C4D => 0x2C1D,
0x2C4E => 0x2C1E,
0x2C4F => 0x2C1F,
0x2C50 => 0x2C20,
0x2C51 => 0x2C21,
0x2C52 => 0x2C22,
0x2C53 => 0x2C23,
0x2C54 => 0x2C24,
0x2C55 => 0x2C25,
0x2C56 => 0x2C26,
0x2C57 => 0x2C27,
0x2C58 => 0x2C28,
0x2C59 => 0x2C29,
0x2C5A => 0x2C2A,
0x2C5B => 0x2C2B,
0x2C5C => 0x2C2C,
0x2C5D => 0x2C2D,
0x2C5E => 0x2C2E,
0x2C61 => 0x2C60,
0x2C65 => 0x23A,
0x2C66 => 0x23E,
0x2C68 => 0x2C67,
0x2C6A => 0x2C69,
0x2C6C => 0x2C6B,
0x2C73 => 0x2C72,
0x2C76 => 0x2C75,
0x2C81 => 0x2C80,
0x2C83 => 0x2C82,
0x2C85 => 0x2C84,
0x2C87 => 0x2C86,
0x2C89 => 0x2C88,
0x2C8B => 0x2C8A,
0x2C8D => 0x2C8C,
0x2C8F => 0x2C8E,
0x2C91 => 0x2C90,
0x2C93 => 0x2C92,
0x2C95 => 0x2C94,
0x2C97 => 0x2C96,
0x2C99 => 0x2C98,
0x2C9B => 0x2C9A,
0x2C9D => 0x2C9C,
0x2C9F => 0x2C9E,
0x2CA1 => 0x2CA0,
0x2CA3 => 0x2CA2,
0x2CA5 => 0x2CA4,
0x2CA7 => 0x2CA6,
0x2CA9 => 0x2CA8,
0x2CAB => 0x2CAA,
0x2CAD => 0x2CAC,
0x2CAF => 0x2CAE,
0x2CB1 => 0x2CB0,
0x2CB3 => 0x2CB2,
0x2CB5 => 0x2CB4,
0x2CB7 => 0x2CB6,
0x2CB9 => 0x2CB8,
0x2CBB => 0x2CBA,
0x2CBD => 0x2CBC,
0x2CBF => 0x2CBE,
0x2CC1 => 0x2CC0,
0x2CC3 => 0x2CC2,
0x2CC5 => 0x2CC4,
0x2CC7 => 0x2CC6,
0x2CC9 => 0x2CC8,
0x2CCB => 0x2CCA,
0x2CCD => 0x2CCC,
0x2CCF => 0x2CCE,
0x2CD1 => 0x2CD0,
0x2CD3 => 0x2CD2,
0x2CD5 => 0x2CD4,
0x2CD7 => 0x2CD6,
0x2CD9 => 0x2CD8,
0x2CDB => 0x2CDA,
0x2CDD => 0x2CDC,
0x2CDF => 0x2CDE,
0x2CE1 => 0x2CE0,
0x2CE3 => 0x2CE2,
0x2D00 => 0x10A0,
0x2D01 => 0x10A1,
0x2D02 => 0x10A2,
0x2D03 => 0x10A3,
0x2D04 => 0x10A4,
0x2D05 => 0x10A5,
0x2D06 => 0x10A6,
0x2D07 => 0x10A7,
0x2D08 => 0x10A8,
0x2D09 => 0x10A9,
0x2D0A => 0x10AA,
0x2D0B => 0x10AB,
0x2D0C => 0x10AC,
0x2D0D => 0x10AD,
0x2D0E => 0x10AE,
0x2D0F => 0x10AF,
0x2D10 => 0x10B0,
0x2D11 => 0x10B1,
0x2D12 => 0x10B2,
0x2D13 => 0x10B3,
0x2D14 => 0x10B4,
0x2D15 => 0x10B5,
0x2D16 => 0x10B6,
0x2D17 => 0x10B7,
0x2D18 => 0x10B8,
0x2D19 => 0x10B9,
0x2D1A => 0x10BA,
0x2D1B => 0x10BB,
0x2D1C => 0x10BC,
0x2D1D => 0x10BD,
0x2D1E => 0x10BE,
0x2D1F => 0x10BF,
0x2D20 => 0x10C0,
0x2D21 => 0x10C1,
0x2D22 => 0x10C2,
0x2D23 => 0x10C3,
0x2D24 => 0x10C4,
0x2D25 => 0x10C5,
0xA641 => 0xA640,
0xA643 => 0xA642,
0xA645 => 0xA644,
0xA647 => 0xA646,
0xA649 => 0xA648,
0xA64B => 0xA64A,
0xA64D => 0xA64C,
0xA64F => 0xA64E,
0xA651 => 0xA650,
0xA653 => 0xA652,
0xA655 => 0xA654,
0xA657 => 0xA656,
0xA659 => 0xA658,
0xA65B => 0xA65A,
0xA65D => 0xA65C,
0xA65F => 0xA65E,
0xA663 => 0xA662,
0xA665 => 0xA664,
0xA667 => 0xA666,
0xA669 => 0xA668,
0xA66B => 0xA66A,
0xA66D => 0xA66C,
0xA681 => 0xA680,
0xA683 => 0xA682,
0xA685 => 0xA684,
0xA687 => 0xA686,
0xA689 => 0xA688,
0xA68B => 0xA68A,
0xA68D => 0xA68C,
0xA68F => 0xA68E,
0xA691 => 0xA690,
0xA693 => 0xA692,
0xA695 => 0xA694,
0xA697 => 0xA696,
0xA723 => 0xA722,
0xA725 => 0xA724,
0xA727 => 0xA726,
0xA729 => 0xA728,
0xA72B => 0xA72A,
0xA72D => 0xA72C,
0xA72F => 0xA72E,
0xA733 => 0xA732,
0xA735 => 0xA734,
0xA737 => 0xA736,
0xA739 => 0xA738,
0xA73B => 0xA73A,
0xA73D => 0xA73C,
0xA73F => 0xA73E,
0xA741 => 0xA740,
0xA743 => 0xA742,
0xA745 => 0xA744,
0xA747 => 0xA746,
0xA749 => 0xA748,
0xA74B => 0xA74A,
0xA74D => 0xA74C,
0xA74F => 0xA74E,
0xA751 => 0xA750,
0xA753 => 0xA752,
0xA755 => 0xA754,
0xA757 => 0xA756,
0xA759 => 0xA758,
0xA75B => 0xA75A,
0xA75D => 0xA75C,
0xA75F => 0xA75E,
0xA761 => 0xA760,
0xA763 => 0xA762,
0xA765 => 0xA764,
0xA767 => 0xA766,
0xA769 => 0xA768,
0xA76B => 0xA76A,
0xA76D => 0xA76C,
0xA76F => 0xA76E,
0xA77A => 0xA779,
0xA77C => 0xA77B,
0xA77F => 0xA77E,
0xA781 => 0xA780,
0xA783 => 0xA782,
0xA785 => 0xA784,
0xA787 => 0xA786,
0xA78C => 0xA78B,
0xFF41 => 0xFF21,
0xFF42 => 0xFF22,
0xFF43 => 0xFF23,
0xFF44 => 0xFF24,
0xFF45 => 0xFF25,
0xFF46 => 0xFF26,
0xFF47 => 0xFF27,
0xFF48 => 0xFF28,
0xFF49 => 0xFF29,
0xFF4A => 0xFF2A,
0xFF4B => 0xFF2B,
0xFF4C => 0xFF2C,
0xFF4D => 0xFF2D,
0xFF4E => 0xFF2E,
0xFF4F => 0xFF2F,
0xFF50 => 0xFF30,
0xFF51 => 0xFF31,
0xFF52 => 0xFF32,
0xFF53 => 0xFF33,
0xFF54 => 0xFF34,
0xFF55 => 0xFF35,
0xFF56 => 0xFF36,
0xFF57 => 0xFF37,
0xFF58 => 0xFF38,
0xFF59 => 0xFF39,
0xFF5A => 0xFF3A,
else => c,
};
}
Test case to verify that the generated switch matches the behavior of `RtlUpcaseUnicodeChar` for all possible `u16` values
const std = @import("std");
test {
var c: u16 = 0;
while (true) {
var win32_upper = std.os.windows.ntdll.RtlUpcaseUnicodeChar(c);
var zig_upper = upcase(c);
std.testing.expectEqual(win32_upper, zig_upper) catch |err| {
std.debug.print("c=0x{X}\n", .{c});
return err;
};
if (c == 0xFFFF) break;
c += 1;
}
}
Some notes:
- There are exactly 973 lowercase -> uppercase mappings in the data when including the ASCII range, or 947 when excluding the ASCII range
- This would be enough to implement
eqlIgnoreCaseWTF16in a cross-platform way if that's the way we end up wanting to go.- When targeting Windows, it would still probably make more sense to call the
ntdllfunctions instead, though; the cross-platform implementation would therefore only get used on non-Windows targets (EDIT: Thinking about this more, there may be no benefit to using thentdllstuff for something likestd.fs.path.relativeWindowssince we're comparing the path component-by-component so even if the underlying filesystem uses different case conversion data, usingRtlEqualUnicodeStringwouldn't be able to provide us with the 'filesystem-correct' comparison anyway).
- When targeting Windows, it would still probably make more sense to call the
- As mentioned, unsure about how we'd go about dealing with changes to this uppercasing data in Windows if it ends up changing (or is already different on Windows 11)
- There might be a better way to encode the data than a giant switch case, maybe some sort of minimal perfect hashing or something like that
- I believe a theoretically perfect MPH solution would be able to use an array of 947
u16s or 1,894 bytes total (since the ASCII range wouldn't need to be part of the table)
- I believe a theoretically perfect MPH solution would be able to use an array of 947
On newer Windows, only the l_intl.nls file is ever read, so that's good news. On the other hand, in Windows 11 the function was modified to look like this (RE'd):
WCHAR __stdcall RtlUpcaseUnicodeCharWin11(WCHAR Source) {
if (Source < 'a') {
return Source;
}
if (Source <= 'z') {
return Source - ' ';
}
if (Table && Source >= 0xC0) {
Source += Table[(Source & 0xF) + Table[((Source >> 4) & 0xF) + Table[(Source >> 8) & 0xFF]]];
}
return Source;
}
The table lookup is the same, but the table itself might not exist now and an additional condition is added on the source argument. The tables are initialized in the aptly named RtlpInitUppercaseTables.
RE'd here
void __stdcall RtlpInitUppercaseTables(USHORT UseCaseMapping) {
if (UseCaseMapping == 0) {
NlsUppercaseTables = NULL;
NlsLowercaseTables = NULL;
NlsSectionPtr = NULL;
return;
}
USHORT *NlsPtr = NULL;
NTSTATUS Result = NtGetNlsSectionPtr(14, 0, NULL, (PVOID *)&NlsPtr, NULL);
if (Result < 0) {
NlsPtr = NULL;
}
if (NlsPtr) {
_InterlockedExchange64((LONG64 volatile *)&NlsUppercaseTables, (LONG64)(NlsPtr + 2));
_InterlockedExchange64((LONG64 volatile *)&NlsLowercaseTables, (LONG64)(NlsPtr + NlsPtr[1] + 2));
NlsSectionPtr = NlsPtr;
} else {
NlsUppercaseTables = NULL;
NlsLowercaseTables = NULL;
NlsSectionPtr = NULL;
}
}
The tables are initialized only if UseCaseMapping is non-zero. If they stay uninitialized, only the ASCII range will be converted in RtlUpcaseUnicodeChar. UseCaseMapping is a new field in the PEB (UnicodeCaseTableData is now null as well) passed down to RtlpInitUppercaseTables from the loader. MmCreatePeb sets UseCaseMapping if the equivalent ESERVERSILO_GLOBALS.RtlNlsState.UnicodeUpcaseTable844 (not sure what 844 stands for) table is also initialized, it's uninitialized only when the kernel fails to open the \NLS object directory (makes sense because the case mapping table is later created and read from this namespace) - I honestly have no clue when or how it would fail, but clearly the error condition somehow exists if Microsoft bothers to check against it even in user-space.
TL;DR
The upcasing function is different in Windows 11 and may somehow fallback to the ASCII range if a very specific condition arises, but it's probably nothing to worry about. Especially since, as @squeek502 said, there really is no advantage to calling the Windows function directly (even on Windows) as there may be an inherent mismatch with the filesystem anyway.
So I think a reimplementation is the way to go. I prefer the table-based version over the switch-based, but it might indeed be worth investigating whether a better solution exists.
LUT-based upcase
const table = [2544]u16{
272, 288, 304, 320, 336, 352, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256,
256, 256, 256, 256, 256, 368, 384, 400, 256, 416, 256, 256, 432, 256, 256, 256, 256, 256, 256, 256, 448, 464, 256, 256,
256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256,
256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256,
256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256,
256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256,
256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 480, 496,
256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256,
256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256,
256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256,
256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 512, 528, 528, 528, 528, 528, 528, 528, 528,
528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 544, 560, 528, 528, 528, 576, 528, 528, 592, 608,
624, 640, 656, 672, 688, 704, 720, 736, 752, 768, 784, 800, 816, 832, 848, 864, 880, 896, 912, 928, 944, 960, 976, 992,
1008, 1024, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 1040, 528, 528, 1056, 528, 528, 1072, 1088, 1104, 1120, 1136, 1152,
528, 528, 528, 1168, 1184, 1200, 1216, 1232, 1248, 1264, 1280, 1296, 1312, 1328, 1344, 1360, 1376, 1392, 1408, 528, 528, 528, 1424, 1440,
1456, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 1472, 528, 528, 528, 528, 528, 528, 528, 528,
1488, 1504, 1520, 1536, 1552, 1568, 1584, 1600, 1616, 1632, 1648, 1664, 1680, 1696, 1712, 1728, 1744, 1760, 1776, 1792, 1808, 1824, 1840, 1856,
1872, 1888, 1904, 1920, 1936, 1952, 1968, 1984, 528, 528, 528, 528, 2000, 528, 528, 2016, 2032, 528, 528, 528, 528, 528, 528, 528,
528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 2048, 2064, 528, 528, 528, 528, 2080, 2096, 2112, 2128, 2144,
2160, 2176, 2192, 2208, 2224, 2240, 2256, 528, 2272, 2288, 2304, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528,
528, 528, 528, 528, 2320, 2336, 2352, 528, 2368, 2384, 528, 528, 528, 528, 528, 528, 528, 528, 2400, 2416, 2432, 2448, 2464, 2480,
2496, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 2512, 2528, 528, 528, 528, 528, 528, 528, 528, 528, 528, 528,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 65504, 65504, 65504, 65504, 65504, 65504, 65504,
65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504,
65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 0, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 121,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 0, 0, 65535, 0, 65535, 0, 65535, 0, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0,
65535, 0, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 0, 65535, 0, 65535, 0, 65535, 0, 195, 0, 0, 65535, 0, 65535, 0, 0, 65535, 0, 0, 0, 65535, 0, 0, 0,
0, 0, 65535, 0, 0, 97, 0, 0, 0, 65535, 163, 0, 0, 0, 130, 0, 0, 65535, 0, 65535, 0, 65535, 0, 0,
65535, 0, 0, 0, 0, 65535, 0, 0, 65535, 0, 0, 0, 65535, 0, 65535, 0, 0, 65535, 0, 0, 0, 65535, 0, 56,
0, 0, 0, 0, 0, 0, 65534, 0, 0, 65534, 0, 0, 65534, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0,
65535, 0, 65535, 0, 65535, 65457, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 0, 0, 65534, 0, 65535, 0, 0, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 0, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 0, 0, 0,
0, 0, 0, 0, 65535, 0, 0, 0, 0, 0, 65535, 0, 0, 0, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
10783, 10780, 0, 65326, 65330, 0, 65331, 65331, 0, 65334, 0, 65333, 0, 0, 0, 0, 65331, 0, 0, 65329, 0, 0, 0, 0,
65327, 65325, 0, 10743, 0, 0, 0, 65325, 0, 10749, 65323, 0, 0, 65322, 0, 0, 0, 0, 0, 0, 0, 10727, 0, 0,
65318, 0, 0, 65318, 0, 0, 0, 0, 65318, 65467, 65319, 65319, 65465, 0, 0, 0, 0, 0, 65317, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 65535, 0, 65535, 0, 0, 0, 65535, 0, 0, 0, 130, 130, 130, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 65498, 65499, 65499, 65499, 0, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504,
65504, 65504, 0, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65472, 65473, 65473, 0, 0, 0, 0, 0, 0, 0, 0, 65528,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 0, 7, 0, 0, 0, 0, 0, 65535, 0, 0, 65535, 0, 0, 0, 0, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504,
65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504,
65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 65456, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 0, 0, 0, 0, 0, 0, 0, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 0, 65535, 0, 65535, 0, 65535, 0,
65535, 0, 65535, 0, 65535, 0, 65535, 65521, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488,
65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 35332, 0, 0, 0, 3814, 0, 0,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 8, 8, 8, 8, 8, 8, 8, 8,
0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 8, 8, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
8, 8, 8, 8, 8, 8, 8, 8, 0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 8, 8, 8, 8, 8,
0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 8, 8, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 8, 0, 8, 0, 8, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 8, 8, 8, 8, 8,
0, 0, 0, 0, 0, 0, 0, 0, 74, 74, 86, 86, 86, 86, 100, 100, 128, 128, 112, 112, 126, 126, 0, 0,
8, 8, 8, 8, 8, 8, 8, 8, 0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 8, 8, 8, 8, 8,
0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 8, 8, 8, 8, 8, 0, 0, 0, 0, 0, 0, 0, 0,
8, 8, 0, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 9, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
8, 8, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 9, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 65508, 0,
65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 65520, 0, 0, 0, 0, 65535, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510,
65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 65510, 0, 0, 0, 0, 0, 0, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488,
65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488,
65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 65488, 0, 0, 65535, 0, 0, 0, 54741, 54744, 0,
65535, 0, 65535, 0, 65535, 0, 0, 0, 0, 0, 0, 65535, 0, 0, 65535, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272,
58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272, 58272,
58272, 58272, 58272, 58272, 58272, 58272, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 0, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 0, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 0, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 65535, 0, 65535, 0, 0, 65535,
0, 65535, 0, 65535, 0, 65535, 0, 65535, 0, 0, 0, 0, 65535, 0, 0, 0, 0, 65504, 65504, 65504, 65504, 65504, 65504, 65504,
65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 65504, 0, 0, 0, 0, 0,
};
fn upcase(c: u16) u16 {
if (c < 'a') {
return c;
}
if (c <= 'z') {
return c - ('a' - 'A');
}
if (c >= 0xC0) {
var offset: u16 = 0;
offset += @truncate(u8, c >> 8);
offset = table[offset];
offset += @truncate(u4, c >> 4);
offset = table[offset];
offset += @truncate(u4, c);
offset = table[offset];
return c +% offset;
}
return c;
}
I've updated the PR with something of a proof-of-concept of a cross-platform eqlIgnoreCaseWTF16 implementation. The details are a bit messier than I'd like, but I'm not really sure how to avoid the messy parts (the biggest one being when to use the ntdll implementation and when to use our own, since our own implementation is faster but increases the binary size if its used).
The Windows NLS format lookup table version was both faster than a switch-based version and contributed less to the final executable size, so that's the implementation that was used here. I benchmarked using the code here:
https://gist.github.com/squeek502/2d5d6a60448e856bd352511911d69401
where news-week-17aug24.txt is the data from here but with everything but the titles removed (and with one title per line) and news-week-17aug24-upper.txt is the same file but converted to uppercase using Sublime Text (so not the same as the Windows uppercasing; not all lines will compare equally). Here are the results on Windows 10 for me:
Benchmark 1: zig-out\bin\upcase-table.exe
Time (mean ± σ): 1.384 s ± 0.006 s [User: 1.258 s, System: 0.126 s]
Range (min … max): 1.378 s … 1.396 s 10 runs
Benchmark 2: zig-out\bin\upcase-switch.exe
Time (mean ± σ): 1.528 s ± 0.006 s [User: 1.400 s, System: 0.129 s]
Range (min … max): 1.519 s … 1.538 s 10 runs
Benchmark 3: zig-out\bin\upcase-switch2.exe
Time (mean ± σ): 1.519 s ± 0.009 s [User: 1.375 s, System: 0.141 s]
Range (min … max): 1.509 s … 1.540 s 10 runs
Benchmark 4: zig-out\bin\upcase-rtl.exe
Time (mean ± σ): 1.551 s ± 0.002 s [User: 1.396 s, System: 0.149 s]
Range (min … max): 1.546 s … 1.554 s 10 runs
Summary
'zig-out\bin\upcase-table.exe' ran
1.10 ± 0.01 times faster than 'zig-out\bin\upcase-switch2.exe'
1.10 ± 0.01 times faster than 'zig-out\bin\upcase-switch.exe'
1.12 ± 0.00 times faster than 'zig-out\bin\upcase-rtl.exe'
(similar results on Linux but I don't have those handy anymore, would have to re-run them)
When the added std.os.windows.nls.upcaseW is referenced, it pulls in 5,088 bytes that will be included in the final executable (in practice, it seems to only add 4,608 bytes but presumably that's for unrelated reasons like padding bytes in the exe format). It seems possible that this size could be compressed further if necessary.
More details from the commit message:
The cross-platform Zig implementation is a bit faster than RtlUpcaseUnicodeChar on my system (1.12 times faster in a benchmark using 100mb of text in many different languages), so it is generally preferred. However, when targeting windows and with the mode set to ReleaseSmall,
RtlUpcaseUnicodeCharfunction is used to avoid having to include the 5,088 bytes of case conversion data in the executable.Users can also opt to use an ntdll-dependent version of
os.windows.eqlIgnoreCaseWTF16calledeqlIgnoreCaseWTF16Ntdllif they know they are targeting windows and want to avoid the uppercase data being included.eqlIgnoreCaseWTF16Ntdllis used instd.os.getenvWfor this reason.
An alternate path would be to just to always use the ntdll version when targeting windows to avoid this speed vs binary size tradeoff and favor the smaller binary size generally.
Thanks for the review @e4m2, I think you hit on every reason I wasn't happy with the previous implementation. Updated with one that's much more straightforward:
-
std.os.windows.nls.upcaseWalways pulls in the uppercase table when referenced. As of now, it is never referenced when targeting Windows unless the user calls it directly. -
std.os.windows.eqlIgnoreCaseWTF16/eqlIgnoreCaseUtf8always use ntdll calls when targeting Windows to avoid pulling in a redundant copy of the uppercase data- caveat:
nls.upcaseWis used on Windows when@inComptime()to alloweqlIgnoreCaseWTF16/eqlIgnoreCaseUtf8to work during comptime on Windows
- caveat:
The NtGetNlsSectionPtr approach may be an improvement but seems like it might introduce more complication than its worth; I'll leave that for a future improvement if there's an indication that it would be beneficial.
EDIT: The below is wrong, the comptime try error was a distraction for a real failure on big endian architectures.
Not sure what's going on with the CI failures. Big endian architectures (mips, powerpc) seem to be failing with:
/home/ci/actions-runner11/_work/zig/zig/lib/std/os/windows.zig:1996:9: error: function called at runtime cannot return value at comptime
which is the second testing.expect call in this block:
comptime {
try std.testing.expect(expect_eql == eqlIgnoreCaseUtf8(a, b));
try std.testing.expect(expect_eql == eqlIgnoreCaseWTF16(
std.unicode.utf8ToUtf16LeStringLiteral(a),
std.unicode.utf8ToUtf16LeStringLiteral(b),
));
}
EDIT: Looks like it might be the same issue that was hit by https://github.com/ziglang/zig/pull/15957 that led to comptime try being changed to try comptime: https://github.com/ziglang/zig/pull/15957/commits/d884d7050e061c620324aaaabfba507e08cb40f4
Seems like it might deserve an issue though since the above comptime block should probably work?
EDIT#2: Reported it as a bug: https://github.com/ziglang/zig/issues/16176