hydrusnetwork
Don't arbitrarily limit the maximum path length when exporting files
Currently, Hydrus limits the maximum path length to 240 characters: https://github.com/hydrusnetwork/hydrus/blob/82fc1c9b3ba079c07392d9d5cd3292705480d0bc/hydrus/client/ClientExporting.py#L17
Ideally, it should instead use the limits of the respective OS/FS.
You can catch whether a file failed creation for any weirdness pretty easily by catching an OSError, but getting PATH_MAX is OS specific. Obviously you could just keep getting substrings and retrying until its short enough, but it would be easier to just use MAX_PATH_LENGTH as a known acceptable value, and fallback to it in the exception handler if you fail once.
>>> f=open ('x'*512)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 63] File name too long: 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
>>> os.makedirs( (('x'*10)+'/')*100 )
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
#...
OSError: [Errno 63] File name too long: 'xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx/xxxxxxxxxx'
Its current value is good because a FAT-16 FS can support a max path length of 255, but maybe calling it pathlength if it is a number for the path without extension is suboptimal.
Windows has removed the 260 character path length limit a few years ago, but it's an opt-in. There is still a 255 character filename limit (using UTF-16) with NTFS, but that path can be much, much longer (apparently 32,767 Unicode characters, but I couldn't find information about the encoding; assuming UTF-16 again).
On Linux/*nix, it seems to be quite convoluted as well: https://eklitzke.org/path-max-is-tricky
So the main issue here is about the current limit applying to the full path and not just the filename, which seems far below the limits of all the supported operating systems. As for file systems, this might be more tricky if some obscure system supports only short paths. But it seems like the major ones all either have a 32,767 bytes limit or none at all:
The on-disk structures have no inherent limit. Particular Installable File System drivers and operating systems may impose limits of their own, however. Limited by its Current Directory Structure (CDS), DOS does not support more than 32 directory levels (except for DR DOS 3.31-6.0) or full pathnames longer than 66 bytes for FAT, or 255 characters for LFNs. Windows NT does not support full pathnames longer than 32,767 bytes for NTFS. Older POSIX APIs which rely on the PATH_MAX constant have a limit of 4,096 bytes on Linux but this can be worked around. Linux itself has no hard path length limits.
See https://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits
But my assumption here was mainly that Python might provide an API to determine this OS/FS-agnostic; if it doesn't then this obviously becomes more difficult and might not be worth it since this limit mostly seems to come up in edge cases. I'll flag this as not-actionable for now and maybe we can discuss this on Discord in more detail.
Edit: There is https://docs.python.org/3/library/os.html#os.pathconf which can determine path and filename limits on *nix (with PC_PATH_MAX and PC_NAME_MAX as second param).
Example on Linux with ext4:
➜ python
Python 3.8.5 (default, Jul 27 2020, 08:42:51)
[GCC 10.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.pathconf('/', 'PC_PATH_MAX')
4096
>>> os.pathconf('/', 'PC_NAME_MAX')
255
I'm currently experiencing Hydrus cutting my export pathnames off at 160 characters right now. I have several example exports that were all cut off at this exact limit. Was the limit reduced from 240 down to 160? Or was a setting added for this recently? That is definitely not acceptable in my opinion, it should at least be 240, and ideally be an optional limit that I can enable in the settings.
I am using Hydrus version 566 and Windows 11, for reference.
