skyjake
Iapetus uploads (and general discussion about upload protocols, Titan, editing pages)
Any plans to add support for iapetus?
Phoebe now supports it, and I'm considering writing a wiki server implementing it.
IMO, it's "cleaner" and more Gemini like than Titan.
IMO, it's "cleaner" and more Gemini like than Titan.
I think I actually agree with this also. I just found out about this from this issue. At first glance I feel like I might prefer it over Titan.
I did consider Iapetus, but chose to implement Titan first because 1) it's equivalent to a Gemini request followed by a payload, and 2) the response that the client receives is a standard Gemini response, meaning you can for example redirect the client somewhere after finishing the upload.
In comparison, Iapetus seems to have a mandatory two-step communication pattern where the client needs to read the initial response and only then continue, and then get a second response for final confirmation. The client gets no indication what to do after the upload is finished. As far as being Gemini-like, I think this two-step pattern is a major difference; Gemini does things in one request.
From the iapetus readme:
Whereas Titan is meant to be used mainly without client certificates
I don't think this statement is based on fact. Titan can be used similarly well with or without client certs.
Maybe I haven't fully understood the Iapetus spec, but what is it about it that you prefer over Titan?
Maybe I haven't fully understood the Iapetus spec, but what is it about it that you prefer over Titan?
On first glance what I liked about iapetus is that it didn't use a weird password + optional certificate thing, it just uses certificates. I find the password thing in Titan to be odd. The other thing is Titan has in its specs, last time I checked, that request uris are supposed to be idempotent.
The client gets no indication what to do after the upload is finished.
This is definitely a problem, actually. I wonder if oppenlab would consider adding this. I do think that Titan is better in this regard.
Iapetus seems to have a mandatory two-step communication pattern where the client needs to read the initial response and only then continue, and then get a second response for final confirmation.
Hm... this is a valid point. I do see why it might have been done this way though, as it relieves the problem of overloading the server by sending giant files, since otherwise it can't reject them before the full file is sent.
I find the password thing in Titan to be odd.
Titan's "token" parameter is optional. You can just leave it out if it isn't needed. When not using a certificate, this could be a password, but it can also be any kind of auxiliary instruction(s) or a comment related to the request.
request uris are supposed to be idempotent
I don't recall seeing that... Have to check again.
the problem of overloading the server by sending giant files, since otherwise it can't reject them before the full file is sent.
Well, a Titan server can see the size of the incoming data from the URL parameters and immediately respond with an error message and close the connection before the client has finished sending all the data. Similarly, if the client sends more than it pre-announced, the server can do the same. I don't remember how this is written in the spec, but I'm pretty sure the server isn't obliged to just politely sit there receiving whatever amounts of data the client is sending. 🙂
Well, I take it back :) The two-step approach is not Gemini-like.
What bugs me most about Titan is the token idea. At least it's optional. But it is always present in the Titan dialog in Lagrange, even if the server doesn't require a token. I'm thinking of UX/usability ("What's a token? Why do I need a token? Where do I get a token?"), it's complicating something that should be very simple (in that way, Titan is not very Gemini-like). If you want tokens, just use a query param in the URL and have your server check it?
I also don't like the unusual MIME style parameters in the URL. It feels like a hack to me, but that's of course subjective.
What I like about Iapetus' two-step process is precisely that you have to check for permission before you upload. But I seem to have missed the part about including the file size in the intent data. I think it would be better without. That is, without yet knowing what content to upload.
The reason is again UX. If the file intent was only for checking write permissions to the file, a client like Lagrange could warn the user before opening the dialog, before writing a whole lot of text only to get an error in return because computer says no. It would also serve as a check to see if the server supports uploads before opening the dialog. And if the response to the intent is 60, Lagrange could prompt the user for a certificate first. It would almost be like logging in.
Yes, it would be more back and forth, but that's not always a bad thing.
The client gets no indication what to do after the upload is finished.
I interpreted the spec as using standard Gemini response headers for confirmation as well?
My vision for editing Gemini pages would be to visit some wiki, hit ⌘E, make edits to the existing text, click "Save", and see the edit that I just made. If the wiki didn't allow anonymous edits, the browser might prompt me for a certificate first. Ideally (for me), editing wouldn't even be in a dialog, but within the browser tab itself. But that's maybe another browser.
It's clear though that both Titan and Iapetus focus on simple file uploads, editing existing resources seems only like an afterthought.
Well, a Titan server can see the size of the incoming data from the URL parameters and immediately respond with an error message and close the connection before the client has finished sending all the data. Similarly, if the client sends more than it pre-announced, the server can do the same. I don't remember how this is written in the spec, but I'm pretty sure the server isn't obliged to just politely sit there receiving whatever amounts of data the client is sending. 🙂
Actually, yeah, you're right. The size is part of the Titan URL.
I did read somewhere about the idempotence of URLs but I'm not sure where. I can't find it in any of the documentation though.
I just seen the Tokens were actually optional according to the documentation. I don't remember seeing this before (and it seems the docs had slight changes in regards to this, afaik), but that's good. That I guess covers all of my concerns with Titan.
To be clear (for anyone else reading), Iapetus does not use two transactions, but it does use two requests within one transaction.
An Iapetus transaction:
- C: Opens connection
- S: Accepts connection
- C/S: Complete TLS handshake
- C: Validates server certificate
- C: Sends upload intent request (one CRLF terminated line)
- S: Sends initial response header (one CRLF terminated line), closes connection under non-success conditions
- C: Sends upload data (text or binary data)
- S: Sends final response header (one CRLF terminated line)
- S: Closes connection (including TLS close_notify)
- C: Handles response
Steps 7 and 8 are the ones that are different from a Gemini transaction.
Step 5 has a different request structure, and step 6 a different interpretation of the status code 10.
Titan's transaction dance is actually identical except for this step, which also differs from Gemini:
- S: If an error is detected, sends response header (one CRLF terminated line) and closes connection
That, and the structure of the intent request (including tokens) are the main differences, as far as I can tell.
From an implementation perspective, I can (and did) implement clientside Titan support simply by tacking on the payload data and URL parameters to the existing Gemini request code. The response handling is the same as Gemini.
Iapetus would need to be implemented as a completely separate request type, with a little state machine for tracking the steps. That's not a problem in itself, but the practical implementation cost and complexity are higher than Titan.
One more implementation note: Titan's URL parameters for token, file size, and MIME type comply with RFC 3986, making it parseable with off-the-shelf URL parsers, while Iapetus's intent syntax has a non-standard space separator. Also not a huge problem, but it's another little addition to the implementation cost.
When it comes to Titan's shortcomings, I'm not so convinced about the mime parameter. The Titan spec states that this needs to be there for the server to know what media type to serve the data as, in case it's a run-of-the-mill file upload. The assumption is that the client or the user, who own the original file, know its media type. In practice, the client may be less equipped to figure out the media type than the server: Lagrange does this naively by file extension, while a server (running a flavor of Unix) probably has available more sophisticated tools to autodetect the type based on content.
My own Gemini clients use different approaches. The Elpher extension Gemini Write uses mailcap-extension-to-mime in Emacs which is based on file name extension where as the titan script either accepts a MIME type argument or auto-detects it using file --mime-type --brief. And if all that fails, or if you want to write a lean client, just use something like this:
sub mime_type {
$_ = shift;
return 'text/gemini' if /\.gmi$/i;
return 'text/plain' if /\.te?xt$/i;
return 'text/markdown' if /\.md$/i;
return 'text/html' if /\.html?$/i;
return 'image/png' if /\.png$/i;
return 'image/jpeg' if /\.jpe?g$/i;
return 'image/gif' if /\.gif$/i;
return 'audio/mpeg' if /\.mp3$/i;
return 'application/octet-stream';
}
That is probably going to cover 98% of all our needs. :laughing:
As for servers, sure, a Titan-enabled server can check the target URL, extract a filename with extension, and determine the MIME type if it was not provided. Making this change would be simple. If that's what it takes to have more implementations, then by all means. I'm just going to be disappointed if we make the change and people still find reasons not to implement it. Both the Titan spec change and and a client implementation seem to be trivial.
I would hate to lose the option to override the MIME types, though. Forcing people to use file name extensions in their URLs is a return to a past that is unnecessary. Not all servers serve files. If I have a server that generates map, the URL to download the map should not have to end with the right extension, since the MIME type is part of the response. Therefore it would be strange to mandate that the MIME is always derived from the file name extension in URLs for uploads. In the same vein, Gemini already mandates a MIME type when downloading from the server instead of telling clients to depend on filename extensions. It seems strange to me to have uploads be different.
I think the simplest solution for people implementing Titan clients that don't want to bother with MIME types is to hardcode text/gemini for text uploads. And if they want to upload binary files, then hardcode application/octet-stream for file uploads.
But as I said, if making MIME type optional is what it takes for the Titan breakthrough, then I'm all for it. I'd say server behaviour is unspecified, however: servers are free to do as much or as little MIME type auto-detection as they want. Treating a missing MIME type as application/octet-stream is a valid option, and rejecting application/octet-stream uploads is a valid option.
Iapetus would need to be implemented as a completely separate request type, with a little state machine for tracking the steps. That's not a problem in itself, but the practical implementation cost and complexity are higher than Titan.
Yea, Iapetus is more elaborate. I also didn't know Titan URLs comply to RFC 3986, that's pretty clever.
To summarize my main gripe with Titan (and Iapetus too, it seems): It handles file uploads from your hard drive just fine, but uploading text input is problematic. You have to begin uploading to find out if you're allowed to upload. If you're editing a page, you can spend a long time editing that text, believing all is well, only to have it rejected by the server. That's horrible UX. But that's another issue.
DX is important. But I'd argue that UX is even more important. A technology is used a whole lot more by end users than by developers.
If you're editing a page, you can spend a long time editing that text, believing all is well, only to have it rejected by the server. That's horrible UX.
Lagrange deals with this by keeping the text to be uploaded saved until the upload has successfully finished. If a client simply throws away user input without regard for success or failure, that's just a poor client implementation...
@kensanata 👍 Making the MIME parameter optional sounds great to me. That means the protocol will not have an implicit use case assumption — not all data to be uploaded is coming from a file. It depends on the server implementation whether declaring a MIME type is necessary and/or relevant. IMO this will make Titan the simplest possible (i.e., ideal) upload protocol to use with Gemini: it's literally just an extended Gemini request with a size parameter and payload. The MIME type and the token are nice extras for specific use cases, like file uploads and certificate-less uploads.
A Best Practices document could/should describe the recommended server and client behaviors when it comes to the parameter usage, for example when and if it's a good idea for a server to try to autodetect content types. Maybe we need a small capsule dedicated to the Titan protocol, similar to gemini.circumlunar.space? Purely from a readability perspective transjovian.org/titan is a little fragmented/noisy with multiple small pages and all the internal wiki links.
Lagrange deals with this by keeping the text to be uploaded saved until the upload has successfully finished. If a client simply throws away user input without regard for success or failure, that's just a poor client implementation...
That's true. But isn't that complicating things on the client end? I noticed that the same text gets remembered even if I close the dialog (meaning to cancel), go to another page and try to upload there. That has confused me a few times.
I don't mean to criticise Lagrange, it's a wonderful browser, and the best Gemini client by a long shot :)
But isn't that complicating things on the client end?
Requests may fail for other reasons in addition to the server denying access, for example if network connectivity is suddenly lost or the app crashes due to a bug. A client needs to have this kind of safeguards if it cares about UX.
I noticed that the same text gets remembered even if I close the dialog (meaning to cancel), go to another page and try to upload there. That has confused me a few times.
I can see why that would be confusing if one doesn't expect the text to be persistent. The current approach is a little simplistic for sure. As an improvement, the text could be saved on a per-site basis or there could be a separate list to manually look up past contents.
Requests may fail for other reasons in addition to the server denying access, for example if network connectivity is suddenly lost or the app crashes due to a bug. A client needs to have this kind of safeguards if it cares about UX.
I'm sorry, I didn't think straight there. A good client should of course keep the text until a successful response. @todo ☕️
BTW, this discussion reminds me that simple != easy (ease of use). And often, more easy == more complex. I'm all for balance. Too simple can become too hard to use, and too easy to use can become too complex.
And in general, I'm not so sure that multiple competing upload protocols is a good thing. I'm not a pythonist, but I do think "there should be one – and preferably only one – obvious way to do it" applies well to protocols, including at a meta level ("there should be only one obvious protocol"). For the task pushing data to servers, I think that protocol should be general enough to support the most common use cases elegantly, from simple file uploads to resource edits. But if it's left up to the market to decide, so to speak, I have a hunch that end users will favour the protocol that's easiest to use.
Maybe we need a small capsule dedicated to the Titan protocol, similar to gemini.circumlunar.space? Purely from a readability perspective transjovian.org/titan is a little fragmented/noisy with multiple small pages and all the internal wiki links.
I did a rewrite and pulled it all together into one document again.
Perhaps this discussion is hijacking the actual issue at hand. Feel free to contact me some other way, perhaps? Or catch me on IRC. :laughing:
@joakim Good points. Given the common availability of other means of uploading (like SCP and HTTPS), the uptake and popularity of a Gemini-first upload protocol still remains to be seen. I can definitely see the value here, but server/service authors need to also figure things out on their end.
@kensanata Nice, I think that looks much more readable! I may have futher comments, but yeah let's continue this elsewhere. 🙂
Getting back on topic:
I am not fundamentally opposed to implementing Iapetus, but it would be nice to see a few servers supporting it before investing the time to add it to Lagrange. The UX will be slightly different than with Titan, but a similar upload dialog should be doable.
To summarize my main gripe with Titan (and Iapetus too, it seems): It handles file uploads from your hard drive just fine, but uploading text input is problematic. You have to begin uploading to find out if you're allowed to upload. If you're editing a page, you can spend a long time editing that text, believing all is well, only to have it rejected by the server. That's horrible UX. But that's another issue.
The way I interpreted this was that you need to be able to know whether you can edit a file before you start editing the file. Because if you edited a file just to find out it was not allowed, then you wasted your time editing that file. In this case, the client saving the editing input won't do anything, you still wasted time editing. If a request was made to see if you have permissions, and then you edit the file, you are delaying the work of editing until after you know that you are allowed to upload an edit.
Instead of complicating the protocol though, perhaps this is something that can be added to the gemtext document itself, idk.
Instead of complicating the protocol though, perhaps this is something that can be added to the gemtext document itself, idk.
Yes, that's what I'm thinking now, after sleeping on it. To solve the use case of editing a resource, you really need two requests. One to check if you have permission to write, and one to write.
This is getting very off-topic again, but I wonder if this is actually best achieved with Titan. It doesn't even have to change how it works today. I have some ideas I'll send over to Alex.
Then again, if we all agree that this is what we want to talk about, we might as well just edit the topic title and talk about how to improve the upload UI. :laughing:
As for the UI problem with Titan: my take has been that it is up to the server to tell clients: "this is a URL that accepts uploads". After all, you can send PUT and POST requests to any URL on the web, too. Rarely will it do what you think it should do, unless you've seen some HTML from the server telling you what to do. I don't see why Titan should be conceptually different from a PUT request in this regard.
Phoebe has has a WebDAV extension. There you will see files and folders, some of them editable, some of them not, just like on a file system. And maybe on a file system you can discover which ones are editable by looking at file attributes, but most of people are like me: they know that files outside their home are often not editable; for example. I don't think that this is something Titan has to fix. This is UI the server has to provide.
Continuing the discussion here works for me, this has been the topic from the start anyway :)
I think we're talking about slightly different things though. Editing involves more than just uploading. I (now) think Titan does uploading really well. But editing a Gemini resource is convoluted, and I think hard to solve in a good way by clients alone.
I don't have any data, but I have a hunch that editing resources is the most common use of uploading in a hypertext protocol. Not just wikis, but personal sites and other content. In any case, I believe it's a use case that's important to do well.
my take has been that it is up to the server to tell clients: "this is a URL that accepts uploads"
Do you mean just by linking in-band to titan://, or the server actually communicating this in some form?
I think it's worth considering an out-of-band signal about the server's capabilities for the served page, such as a MIME parameter in the response header (#415). If the current page can be edited, the client would know so up-front and show the option to edit in the UI. The server could even do this on a per-user basis (client certificate).
Maybe that's enough to make a decent flow of user interactions when editing a page:
- Visit a page (with correct client certificate)
- See the
Editoption is available, click that - Get a form with the source of the page pre-filled, edit it and click
Save - Client uploads edited source to the
titan://equivalent of the URL - 30 Redirect to see the now edited page
No need for two requests, just one standard Titan upload and good cooperation between client and server.
For wikis and such, you'd do this on the raw/source view of the page.
Without this hint from the server, clients would have to show the Edit option on all pages you visit. You the user would have to figure out if you're actually allowed to edit or not.
Here's how I use my wiki using Emacs: https://alexschroeder.ch/videos/gemini_write_demo.mp4
The server has a link at the bottom saying "Raw text". Imagine it said "Raw text for editing".
The server shows the raw text. The user can edit it using a command or menu, and submit it back, without seeing the titan:// link you mentioned. An out of band signal telling the client that this is not just text/plain but editable text/gemini or editable text/plain would be very nice indeed, if unaware clients would render it correctly. Perhaps text/plain; charset=UTF-8; type=editable would work? There seem to be all sorts of MIME subtypes in circulation, not just charset. https://www.iana.org/assignments/media-type-sub-parameters/media-type-sub-parameters.xhtml
One example would be the format parameter for application/mbox.
https://www.rfc-editor.org/rfc/rfc4155.html
Using Emacs isn't fair :)
With all respect (and I do have respect for people who use Emacs), not everyone lives inside Emacs. I know it's a nice place to live, though!
I was mainly thinking about GUI clients, but think it could still be useful for terminal clients. It's very much opt-in anyway, more like a hint than a directive, clients can deal with it as they wish.
A media type parameter like this is within the requirements of RFC6838, AFAICT.
I'd be interested to know how clients handle unknown parameters to MIME types. I'll set something up so we can test existing clients.
If you want to see a document with many screenshots explaining how to maintain a capsule using Lagrange (without using the token!), see Capsule update using Lagrange on the phone.
Here's a document explaining how to do it using WebDAV.
I don't just live inside Emacs. But looking at Emacs sometimes shows that different user interfaces are possible. Same with WebDAV and other file system layers (fuse etc). Sometimes our own expectations are very strong, preventing us from thinking about novel ways of interacting with programs.
OK, I've got a test URL: gemini://transjovian.org/do/mime/test
AV-98 handles it. :heavy_check_mark:
Amfora handles it. :heavy_check_mark:
Agunua handles it (but complains about the missing close_notify). :heavy_check_mark:
Lagrange handles it. :heavy_check_mark:
My own command line client suspects it to be binary data because it doesn't parse the MIME type correctly. Oops!
$ gemini gemini://transjovian.org/do/mime/test
20 text/plain; charset=UTF-8; type=editable
Better not to print binary data to a terminal (use --force to do it anyway)
Or even better, redirect it to a file:
gemini gemini://transjovian.org:1965/do/mime/test > data.txt
Therefore:
$ gemini gemini://transjovian.org:1965/do/mime/test | cat
20 text/plain; charset=UTF-8; type=editable
This page claims to be editable plain text.
=> gemini://transjovian.org:1965/do/mime/test Again
That's good to see! I think a client should be expected to parse a valid MIME string correctly.
I've also tested it with Lagrange MIME hooks, and my hook received it as an argument just fine. I like type=editable 👍
I don't just live inside Emacs. But looking at Emacs sometimes shows that different user interfaces are possible. Same with WebDAV and other file system layers (fuse etc). Sometimes our own expectations are very strong, preventing us from thinking about novel ways of interacting with programs.
I meant it as very tounge-in-cheek, I hope I didn't offend! I do think it's amazing that you can do almost everything with Emacs, and I know someone who does live inside it :)
You make a very good point, though. The UX you demonstrated in Elpher is actually really close to what I'd like to see in GUI clients like Lagrange. Hit ⌘E (or click Edit) and immediately edit the current page in-place (no dialog window). Hit #⏎ (or click Save) and the text is uploaded. There could be a bar at the bottom for selecting certificate and entering a token, with Save and Cancel buttons. There could also be an Edit button in the address line, similar to the Toggle reader view button in Firefox.
Well, the problem with lagrange having an editor inside is that skyjake would then have to implement a full multiline editor (although, I think he already has a basic one because you can now do multiline for the input gemini requests). He could do just a simple editor, I suppose, but some people might want more complex features (syntax highlighting, line numbers, moving lines up and down, etc.)
There should definitely be a feature to just auto-open the file in the OSs default editor, not sure if this is already a thing.
Oh, I stand corrected, I didn't realize the Titan upload already has an edit box now, lol. I guess he just needs to populate it with the current page source then.
Also, a link to open in OS's default editor would be useful too.
and immediately edit the current page in-place (no dialog window).
I reread this again, I guess what you mean is something more like a wysiwyg? That could actually be doable with gemtext, since gemtext is much simpler than other document formats. It would be more complicated than what Lagrange currently has though.
Also @skyjake I think when you go to the File tab when clicking to upload with Titan on the current page, the MIME type should auto-populate with what the MIME is for the current page. Currently it is blank:
P.S. Sorry I'm making so many separate comments, lol