fsspec
`find` includes the parent folder name
At least since version 2023.9 s3fs.S3FileSystem().find(...) includes the folder that it searches in. This is a breaking change with previous behavior and also possibly introduces bugs in the completely valid case of having a subfolder with the same name as the folder to search in, for example for exhaustive iteration.
This was introduced after 2023.5.0, it still behaves as expected (not including the additional folder name) on that version.
I believe we have been striving for consistency with posix commands, and find in a shell deos indeed also include the path matching the argument as an output.
@ianthomas23 , any comment?
Understood, and if consistency with posix is the goal that makes sense - it just broke some of our workflows because we relied on the old behavior. It'd be easy enough to filter for this case with a separate query, however if there was a flag that allowed to use either behavior within s3fs that would save the additional request and make it backwards compatible if required.
I think we have to stick with consistency with posix. I know we have made many backwards-incompatible changes in the last 6 months or so but they have all been to fix previously incorrect behaviour, and we have to accept that this will have annoyed users but it is best to correct problems as soon as possible.
We can't add flags to support previous behaviour. We would need CI to support both current and previous behaviour, and we don't have spare maintenance time to spend on this. If a given maintainer had an extra n hours to spend on fsspec then I would rather they spend it fixing current problems rather than supporting previous behaviour.
