Synchronization backends should be plugable

[SpotlightKid]

In my opinion watson should not support only one (relatively unknown) service for synchronizing frames with a remote storage, but give the means to users to choose alternatives and to developers to implement support for them. For example, there could be synchronization Backends for Dropbox, Google Drive, iCloud, etc.

Please excuse the long text below, but I think this issue warrants some thorough discussion and I would be glad to hear your thoughts on it.

To not bloat the watson distribution with too many sync backends (and their dependencies), I propose to use a plugin framework to load backend implementations and to specify the API that they have to support.

As a fist step, the existing backend implementation for artich.io should be refactored as a plugin and a framework for loading it implemented in watson. For the latter I propose to use setuptools entry points since it allows to implement backend plugins in separate distributions and it can be supported in watson without adding additional dependencies and very little code. A good brief introduction to using entry points can be found here.

As a Proof of Concept I have started a new branch called syncplugins on my fork of watson. I plan to add a backend plugin using Dropbox to it soon.

The branch has the following changes:

• Moved pull, push, _get_request_info and _get_remote_projects methods of watson.watson.Watson into a new class ArtichIOSync in the new module watson.artichio.
• Added an entry point group watson.sync in setup.py
• Added the entry point artichio with the implementation in watson.artichio:ArtichIOSync to it
• Added a new method _load_backend to the Watson class to load the sync backend specified in the configuration option backend.name (defaults to artichio)
• Re-implemented Watson.pull and Watson.push to call _load_backend and then delegate the actual pushing and puling of frames to the backend plugin.
• I've kept the logic to decide which frames to add and upload in those methods, since this will probably be the same for all plugins (but see discussion below about using several backends at the same time). This means that push and pull just return a list of frames, they do not modify Watson.frames themselves.
• To support this better, I've added a __contains__ method to frames.Frames to test for existing frames by ID.
• Added some tests for Watson._load_backend and fixed others to account for moved methods.

Since I don't have an artich.io account, I can't really test wether the changes broke the synchronization, but the tests still pass. The way the artich.io responses are mocked, leads to few questions, though:

• Do the frames returned by artich.io not contain an updated_at timestamp? IMHO this should be always required to enable watson to recognise editing conflicts.
• Is there a way to pass information about deleted frames (in both directions)?
• Currently watson keeps a global timestamps when the last sync was performed and this is passed to artich.io. If we want to allow the user to synchronize with several services at the same time, this timestamp must either be saved per backend or it should be the backend's responsibility to save and retrieve it.

The current API for plugins looks like this:

class BaseSyncBackend:
    def __init__(self, config):
         self.config = config  # a configparser.ConfigParser instance

   def pull(self, last_sync):
       return received_frames  # a list of frames.Frame instances

   def push(self, frames):
       return uploaded_frames  # a list of frames.Frame instances

Does this need to be extended / changed somehow? If backends should become responsible for keeping their state (last_sync), do we need to pass a storage path?

[willdurand]

Hi @SpotlightKid! Thanks for your work and thoughts on this point. We are actually working on it, and it is semi-related to #29. We need more time to come up with a good plan I guess, but what you did so far makes perfectly sense!

My (very own) advice: don't get stuck because of artichio, we'll be able to change things on our side.

[willdurand]

Note that my comment still apply too: https://github.com/TailorDev/Watson/issues/29#issuecomment-148961321.

[SpotlightKid]

@willdurand regarding the integration with a Watson dashboard mentioned #29, I think the same would apply: it should be implemented as as plugin and the API be specified, so third-party implementation of both clients and servers can be created.

[marcaube]

I'd :heart: to be able to write a bridge to use watson with Jira. I name my Watson projects after Jira tasks, so I could call their api and log my time.

[shulard]

Hello !

Any news about that issue ? Is this kind of feature still on the roadmap ?

[SpotlightKid]

Well, since the backend (artichio), which the current code is written for, never materialized, there is no real need to follow the synchronization protocol it implements. Especially, since it has some conceptional flaws, IMHO.

So what's needed is, that somebody writes a synchronization backend or an adapter for an existing system. The integration into Watson could still be done within the framework laid down by this PR, though. But it doesn't make much sense to merge it, before there is a matching backend.

[gustawdaniel]

I tested connection with

https://crick.io/

and works perfectly. This project is described here:

https://tailordev.fr/blog/2017/06/07/le-lab-5-crick-a-backend-for-watson-time-tracker/

@SpotlightKid - what do you mean by conceptional flaws?

[SpotlightKid]

See the last third of the decription of the PR above. It may be that the interface has changed in the meantime, though.

[gustawdaniel]

Lets update. Idea of synchronization and integration with toggle is worth the commitment of my time.

You wrote:

For example, there could be synchronization Backends for Dropbox, Google Drive, iCloud, etc.

I love this concept. I want help to code it.

But I have some questions to this paragraph:

As a fist step, the existing backend implementation for artich.io should be refactored as a plugin and a framework for loading it implemented in watson.

1. What is artich.io, this domain is not working. I found only article

https://tailordev.fr/blog/2016/02/23/say-hello-to-artichio-first-leaf/

but I do not understood what it really is. Sentence "artich.io is an ambitious project for a small company." tells me nothing.

---

For the latter I propose to use setuptools entry points since it allows to implement backend plugins in separate distributions and it can be supported in watson without adding additional dependencies and very little code.

2. Link to enty points redirect to not existing webpage

https://pythonhosted.org/setuptools/pkg_resources.html

---

A good brief introduction to using entry points can be found here.

3. Word here is connected with link

https://pylons-webframework.readthedocs.io/en/latest/advanced_pylons/entry_points_and_plugins.html

but this packege is in maintained-only stage. I asked about their status in issue that it appended above.

---

4. You attached also link to your branch:

https://github.com/SpotlightKid/watson/tree/feature/syncplugins

This repository has been archived by the owner. It is now read-only.

5. Finally I found three your open pull requests from 2016:

• https://github.com/TailorDev/Watson/pull/115
• https://github.com/TailorDev/Watson/pull/119
• https://github.com/TailorDev/Watson/pull/140

in 115 there was braking cahnges and discussion about release 2.0, but I see that now newest version is 1.6.0. Does anyone know if this pull request will be included or closed?

https://pypi.org/project/td-watson/#history

in 119 @jnsebgosselin told that he want to complete is during holidays. This is about messages in time frames, feature described in #93 and it is "nice to have" during integration with toggl #29. @jnsebgosselin how can I help with this?

in 140 there is styles. Probably it is not connected with subject of synchronization.

---

Questions:

• is there any schema of architecture in watson project? If I know there is cli, but in repository

https://github.com/TailorDev/crick

there is backend and frontend. But nothing about synchronization and external data storages like dropbox or google drive.

• should be synchronization considered as blocking feature for integration with toggle #29?

• there are endpoints for synchronization

https://crickapi.docs.apiary.io/#reference/crick-api-for-watson/frames/push-(not-yet-synchronized)-user's-frames https://crickapi.docs.apiary.io/#reference/crick-api-for-watson/frames/retrieve-all-user's-frames

But I afraid about POST method because of POST is not idempotent. It means that client sending data to synchronization (watson cli?) should have logic about rules what should be synchronized, and what should not. Has watson cli these logic? Where are related commits / issues?

[gustawdaniel]

Which component should be considered as responsible for synchronization?

a) watson cli - this repository b) crick backend - i mean api written in go https://github.com/TailorDev/crick/tree/master/api

if a) then what exactly should be done (taking into account that artichio never materialized and crick exists)

if b) I propose to close this issue and create new one referring to this in crick repo. Then I will learn go, maybe react... and will try to implement this.

anyway if I have to help somehow, I need clear graph presenting architecture of this software and propositions about where logic of synchronization should be placed.

[medwig]

I'd heart to be able to write a bridge to use watson with Jira. I name my Watson projects after Jira tasks, so I could call their api and log my time.

If anyone is still waiting for this feature specifically for Jira syncing, I wrote a CLI layer on top of Watson to tide me over until then:

https://github.com/medwig/watson-jira

[jmaupetit]