ShareLaTeX API

[jpallen]

We're starting to think about a proper API for ShareLaTeX, and need your feedback on what you would need from something like this. So please can you comment here and let us know:

1. What is your use case for the ShareLaTeX API (an app you're developing, a product integration, a personal back up script, etc)?
2. Which data you would need access to do be able to do this (in as much detail as possible please). If you can speculate on the end points you would need that would be useful too (e.g. "I'd need access to be able to list projects owned by a user, see their names and owners, and then get the email addresses of the owners").

The API will likely use OAuth as an authentication mechanism, but comments on this are welcome too. What would you need from an auth flow?

Thanks!

[johan12345]

What I would like and am already trying to create (see also #149) is a client application for Android tablets (and phones, but it will probably be much more useful on tablets) which provides functionality for editing ShareLaTeX projects, similar to the web version. The advantage of a native app would be a much more responsive UI and avoiding bugs of the Ace editor on mobile devices (see sharelatex/sharelatex#13). The following APIs would be needed for that (the list is probably applicable to any mobile or desktop client application):

• Non-real-time API (using HTTP requests and JSON/XML or similar data format)
  • Authenticate to the ShareLaTeX server with a user account (OAuth would be fine)
  • Get the list of projects the user has created, including their ID, owner, folders and sharing configuration (private or public, list of collaborators). This would be equivalent to the JSON data contained on the {{/project}} page ({{window.data = { ...}})
  • Add a new project (using a specified name, template, ...)
  • Get and edit metadata of a specific project (list of files, compiler, main document, spell checking language, ...)
  • Get and edit sharing configuration of a specific project (public/private, list of collaborators)
  • Download and upload files from/to the project (.tex and e.g. image files)
  • recompile a project (could also be moved to the real-time API)
  • SyncTeX (convert a code position to a PDF position and the other way round, could also be moved to the real-time API)
  • get the revisions of a .tex file in a project and restore previous versions (History feature)
  • get the output file (PDF) of a project
    • Optional: Get single pages of the PDF (as PDF/other vector graphics format or as PNG in a specified resolution) to display the current page more quickly in large documents and on slow networks
  • Get the LaTeX log file(s) of a project in raw and parsed (using jpallen/latex-log-parser) form
• Real-time API (e.g. using Socket.IO, like the webapp already does)
  • Subscribe (join) and unsubscribe (leave) a specific project and document (.tex file) for receiving updates about any changes that are made by other users
  • See which users are currently editing the file
  • Make changes to the .tex fle that you are subscribed to
  • Send the current position of the cursor and receive the cursor position of other users that are currently editing the file
  • Send and receive chat messages
• Optional: Subscribe to updates about new chat messages in your projects (all or specific ones) and other events (e.g. new project shared with the user) via Google Cloud Messaging (or APNS etc. for other operating systems), so that the app can receive messages when it is not in the foreground without staying connected to the Real-time API.

It would also be great if the Real-time API used a current version (>1.0) of Socket.IO, because there are more and better client libraries for those versions (or client libraries for <1.0 aren't maintained anymore).

This is everything that I can think of right now, I will update this post when I have more ideas.

[danielneis]

Hello,

I am working with three more people in an integration of Open Science Framework (https://osf.io/) with ShareLatex and have found this is sue. We have read the other issues and took a look at the ShareLatex code.

We are aware that OAuth2 should be a good protocol to work with, but we have a simpler solution to start with.

ShareLatex already have an "auth_token" (https://github.com/sharelatex/web-sharelatex/blob/master/app/coffee/Features/Authentication/AuthenticationManager.coffee#L39) that is tied to user records and is used to access some routes like user/personal_info (https://github.com/sharelatex/web-sharelatex/blob/master/app/coffee/router.coffee#L86) and collaborators (https://github.com/sharelatex/web-sharelatex/blob/master/app/coffee/Features/Collaborators/CollaboratorsRouter.coffee#L8).

If we just change the other routes to allow the use of auth_token and show this token to the user on profile page (like https://packagist.org or amazon), then user can copy this code to the other application (in our case OSF) and they will be able to communicate.

Another "pro" of this approach is that we can use the existing routes instead of creating new ones.

The main difference between this approach and OAuth2 is that the later uses one token per application instead of only one user token, and the process of generating the token on one app and add it to the another is automatically done by the applications. This way, this approach can be seen as a step towards a full OAuth2 implementation.

We also have found other applications that would like to integrate with ShareLatex that would benefit from this approach: https://apps.owncloud.com/kb/ShareLatex+integration/?kbid=2523 http://sourceforge.net/p/texstudio/feature-requests/890/

We are willing to do the changes if you agree. =)

Kind regards, Daniel

[henryoswald]

Hey Daniel,

thanks for getting in touch, Open Science Framework looks really cool.

Your plan does sound good but we are not really in the right place with the company to start offering an API yet. An api will need a lot of management from us and ideally it will be part of a large effort to clean up the auth schema. The auth token is something we are not in love with and would like to improve. With our new product getdatajoy.com, which uses the same code as ShareLaTeX, we are hopeful there will be more of a need for an api which will make it easy to justify putting a lot of development time into it.

[danielneis]

Hello, Henry

thanks for your promptly response.

Let me tell you a little more why we are willing to do this. =)

Recently, the Center for Open Science (https://cos.io/) have made available what they call "Integration Grants" (https://cos.io/integrationgrants/) that is, they are giving money to free software developers to integrate OSF with other software like ownCloud, BitBucket, Zenodo, OneDrive, and others.

We first applied to integrate BitBucket, but there were some other people willing to do that too, so COS people suggested we developt integration with ShareLatex. We liked it better, because ShareLatex is a great (free and) open source tool while bitbucket is not open.

We took a brief look at the ShareLatex code and at a first glance thought that the filestore and docstore were the APIs we would need. But now with a closer look, we've found that is not the case.

Our scenario is much simpler than the full android version of the app, so we will only need the following endpoints to allow_auth_token:

/project /project/new /Project/:Project_id/file/:File_id /Project/:Project_id/output/output.pdf /Project/:Project_id/download/zip

I've not find an endpoint to get all files of a project (or folders or files of a folder), it would be useful for our work too.

It seems not that much work to get something we could start playing with. We have three months to develop this integration and have two people to work on ShareLatex and more two people already working on OSF/Waterbutler.

Our requirements is to alter this 5 routes, plus their respective controllers changing lines like:

res.render 'project/list', viewModel

https://github.com/sharelatex/web-sharelatex/blob/master/app/coffee/Features/Project/ProjectController.coffee#L161

Wraping then with an "if req.query.returnData?" return just data and doing the rendering otherwise.

The main site is already using httpS so this schema using the auth_key is pretty secure. We can also implement a way for users to revoke and generate new auth token.

This is just a first step towards an API and of course it can be done much better, but we can start with this first, so people will have something to work with and think about, give feedback and then we increment and improve.

The other tool you mentioned can benefit from these improvements too, because once we get ShareLatex integrated with OSF, almost the same API would be valid for GetDataJoy (maybe we will not have a PDF output) and we can also implement an integration of it with OSF with almost no effort (jut copy the plugins, use sed to replace names, and its done).

As I said, this is just a first step and we are willing to work collaboratively with you to get this done.

Kind regards, Daniel

[jpallen]

Hi Daniel,

I just saw the grant announcement, congrats!

Can we set up a call sometime on Google Hangouts to discuss this? There are so many possibilities, options, thing to consider etc, that I think we should talk in person, especially if you're about to do 3 months of development on this. We'd like to make sure that it's as useful and long-term as possible.

An API has been on our roadmap for a while now, but the main thing we're concerned about is doing it properly. Because once an API is public, you can never take it back!

Can you drop us an email at [email protected] and [email protected] please so that we can arrange a time? Thanks.

James

[henryoswald]

Hey Guys,

So you know we are planning the next few months of development early next week so if we could chat before then it would be great. If we can get a rough plan for the API before then it will be easier for us to allocate developer resources to the work.

[jpallen]

Pinging @danielneis just in case he hasn't seen this.

[danielneis]

Hello,

I've just sent you an email about this.

Kind regards, Daniel

[kofronpi]

It would be great to be able to generate copies of a document with some (authorized) vars injected into it.

Use case : on-demand generation of invoices, for instance :+1:

[danielneis]

A patch for this issue was proposed on https://github.com/sharelatex/web-sharelatex/pull/212

[kofronpi]

@danielneis I don't see any discussion/commits about this, what do you mean? :)

[danielneis]

@kofronpi a patch for this issue, not for your issue ;) Me and two other co-workers developed a first version of a public api on sharelatex/web-sharelatex#212 I think it would be better to open another issue to document and discuss your needs. =)

[liori]

Hi, I am often using complex machine-generated images (graphs, diagrams, etc.) or data files for pgfplots, generated in Python or R, in my LaTeX documents. So far I was just manually reuploading them whenever I generated new ones or redid old ones on new data. Today I decided to look for a way to automate this process, and found this ticket. For this use case, I'd just need a single non-realtime "POST this content as new contents of file X in project Y" and (maybe, would be a small time-saver too) "Recompile the project and refresh the PDF previews in my opened tabs".

Today I'll probably just hack my way around the current code/inofficial API.

[Siedlerchr]

Hi, I wanted to know if there is way to programmatically upload (bib) files to a project in sharelatex? I am currently writing an extension to JabRef that will allow the synchronization of a local bib database with the one in the sharelatex project.

I already tried to do a POST-Request to the /upload endpoint, with all parameters, but I always get a {success:false} as response. So any help would be appreciated.

Edit// I switched to WebSockets and got it to work

Thanks in advance. Regards Siedlerchr

[pedosb]

@Siedlerchr could you share how you did it? Maybe you could share a snippet?

[Siedlerchr]

@pedosb Sure, you can take a look at my PR I created for JabRef. I use websockets to receive the json encoded messages and to send my changes.

The connection is starting here https://github.com/JabRef/jabref/pull/2866/files#diff-ee115bac52c0a87fc5b356d3f3be08efR27

The Websocket stuff is here: https://github.com/JabRef/jabref/pull/2866/files#diff-1ff667f43a1980b99f9620c5ba802e50R37

Edit// You can also take a look at this project, that's where I got my ideas from: https://github.com/thomashn/Vim-ShareLaTeX-Plugin

[pedosb]

@Siedlerchr Great!

Just so you know. I will try to implement the "push to ShareLaTeX" function in https://github.com/Jorl17/sharelatex-git-integration-unofficial

[retorquere]

Can someone get me a walk-through of the websocket protocol? I'm specifically looking for a way to push a file to sharelatex, probably in much the same as the jabref PR does.

[retorquere]

I see the web frontend just uses regular HTTP POST/PUT, but it uses a query param qquuid during the upload. Where does this param come from?

[retorquere]

I'm not seeing any websocket traffic when I do stuff in ShareLaTeX through the browser, so it's really hard to reverse engineer, and I really have no idea what the JabRef PR and the VIM plugin are doing.

I do see HTTP POSTs/PUTs but they require a folder_id and I have no idea where to find that. I hav everything in place to connect Zotero to ShareLaTeX, I just need a way to (re)place (preferably) or edit (if need be) a bib file through either xhr (preferably) or websocket (if need be).

[Siedlerchr]

@retorquere For Websocket logging in your browser you need websocket monitor in Firefox https://addons.mozilla.org/de/firefox/addon/websocket-monitor/ (does not work with quantum or newer, so grab an older version of firefox)

the quuid is coming from the upload form control internally somewhere (Was my first idea to use this way, but I could not get it work)

If you want to try out how this works with JabRef you can use this built to test it: https://builds.jabref.org/sharelatex/ And you can git clone JabRef https://github.com/JabRef/jabref.git and checkout the branch sharelatex to see the code and as JabRef is licensed under MIT you can freely reuse the code.

To start websocket you need first to establish a login session via http post:

1. Get the website, ectract csrf token, store cookies
2. Do a Json post request to the login, send cookies from prev request, store the new cookies from the repsonse
3. Make a call to /projects/ to get all projects and ids
4. Start Websocket connection: https://github.com/JabRef/jabref/pull/2866/files#diff-1ff667f43a1980b99f9620c5ba802e506
5. Wait for connection accepteced
6. Send join project over websocket
7. Wait for response. Response contains a list of docs and their ids
8. Send join doc
9. Wait for response. You will get the content
10. Sending changes: They must be in the diff/match/patch format https://github.com/google/diff-match-patch Whenever someone types sth you will receive the content in the same diff/match/patch format The diff format is always column, line and operation (e.g. inserted/deleted)

[retorquere]

Crap, that's a lot more work than I'd hoped it would be. It looks like I can grab the folder ID from the websocket conversation though and then I could try to use file upload through regular xhr. Less efficient perhaps, but I'm really not interested in the changes a user would make in their end -- if my Zotero plugin detects relevant changes on the Zotero end, I just want overwrite what's in ShareLaTeX. Manual changes made to the bib file on ShareLaTeX would be lost, and I'm OK with that.

In the websocket chatter I see this coming by. Can you tell me what the numbers, colons and pluses mean at the start of the message?

<= 1::

<= 5:::{"name":"connectionAccepted"}

=> 5:1+::{"name":"joinProject","args":[{"project_id":"54f..."}]}

<= 6:::1+[null,{"_id"...

=> 5:2+::{"name":"joinDoc","args":["56...",{"encodeRanges":true}]}

=> 5:3+::{"name":"clientTracking.getConnectedUsers"}

<= 6:::3+[null,[{"last_updated_at": ...

<= 6:::2+[null,["\documentclass{article}...

<= 5:::{"name":"broadcastDocMeta",...

[Siedlerchr]

The numbers in front are opcodes. Some of them have to be increased. It's a websocket proto thing. If I remember correctly when you receive a repsonse and send a new one, you have to +1 the original code. I have that somewhere in my code.
http://lucumr.pocoo.org/2012/9/24/websockets-101/

[retorquere]

If the numbers at the front are opcodes, how come I see 6? According to that link, opcodes can only be

0x00: this frame continues the payload from the last.
0x01: this frame includes utf-8 text data.
0x02: this frame includes binary data.
0x08: this frame terminates the connection.
0x09: this frame is a ping.
0x0a: this frame is a pong.

[retorquere]

I also can't find the part in that doc that talks about the colons and the pluses.

[retorquere]

I've opened a WS connection from my code, and SL sends me 7:::1+0. What should I make of that?

[retorquere]

I'm not seeing anything else, even if I do

websocket.send(`5:1+::${JSON.stringify({name: 'joinProject', args: [ { project_id: project.id } ] })}`)

after the 7:::1+0. I do see in the close event that I'm getting a close code of 1006, which apparently means "connection was closed abnormally", but I'm seeing no activity in the onerror event.

[Siedlerchr]

When you receive the connectionAccepted you have to wait some time, I used a Thread.sleep(200), otherwise the conn will be closed. Here are the relevant cases I check for: https://github.com/JabRef/jabref/pull/2866/files#diff-1ff667f43a1980b99f9620c5ba802e50R211

Here are the messages I sent, you have to increment the number after the colons: https://github.com/JabRef/jabref/pull/2866/files#diff-1ff667f43a1980b99f9620c5ba802e50R150

[retorquere]

I have a 2 second wait, same thing.

[retorquere]

Oh wait -- you mean wait 200 msecs after the 7:::1+0?