Andrew Otto

[edited comment] Sorry, spoke way too soon! The copy/pasted parse regex was not matching my -dev release name (and my previous iterations had the wrong optional value specified.) I have...

Oh! Didn't realize you were using your own websockets broadcast. Hm. In that case you can probably decline this issue. We are probably stuck with irc.wikimedia.org for the time being,...

Also, `$id` is not required to be a fully qualified URL. At the Wikimedia Foundation [we use relative path URIs](https://techblog.wikimedia.org/2020/09/17/wikimedias-event-data-platform-json-event-schemas/) for `$id`s and provide base URIs when resolving them. I...

What's the status here? We are waiting for this at the Wikimedia Foundation! :)

> It seems python-poetry/poetry-core#40 is implemented but, from the master docs here, this change is not being implemented in v1.2? Also curious about this!

I'm working on this at Wikimedia, and was just wondering the same thing! I was just considering [adding an entry_point](https://filesystem-spec.readthedocs.io/en/latest/developer.html#implementing-a-backend) to override the existing hdfs registry to use arrow_hdfs. However...

Ah great, good to know. Perhaps we will just wait for that then.

Hm okay, then perhaps I will do it now. My python project if installed will change the way fsspec hdfs:// urls work if it is installed (or imported? tbd). Thanks.

> you simply need to use the arrow_hdfs:// prefix instead of hdfs:// Yes, but I'm writing a library that is trying to abstract that kind of detail away. Folks are...

So far my lib uses a [only a few fsspec API calls](https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/blob/main/workflow_utils/artifact/cache.py#L102-122). Mostly check if exists, open, read, write. I've tried with `arrow_hdfs://` and it seems to work just fine,...