Summary

The goal is that users should be able to sync data over a network of any size without having to do lots of manual configuration (e.g. setting up shares and user accounts).

Rationale

Users can have data on any number of desktops, laptops and servers. While it is relatively easy to work off a document on a file server, it becomes less easy when you have to work offline for a while and you end up with multiple copies of documents scattered around or you want to keep data from Evolution, Tomboy or F-Spot in sync across multiple machines.

Use cases

  • Tom wants to a copy of his notes on both his laptop laptop and his desktop. They change often.
  • Fred wants to be able to get his latest appointments onto his laptop from the office server whilst out on the road
  • John wants to keep an up to date copy of his photos and music collection on his server in case anything happens to his laptop

Scope

The initial development will concentrate on Tomboy and Evolution sync, but should scale to other dp's with minimal effort.

Design

The basic routine would be as follows:

  • Jc2k (User A) wants to share Tomboy notes. He drags a Tomboy DP and a NetworkShare DP onto the canvas.
  • Conduit A detects a new NetworkShare DP and advertises it on the network. (NetworkShare is really a dummy DP, Twisted will provide a REST proxy for the Tomboy DP and no syncing will take place on Jc2k's machine).
  • Conduit B detects the share on Conduit A and adds it to a list of available dataproviders (Category=Machine Name, DP Name=Tomboy or Notes...)
  • Nzjrs sees a new DP on his list and drags it on to his canvas. He also drags a Tomboy DP and then chooses sync all
  • Conduit B pulls data from Conduit A over REST protocol and syncs it with data from the Tomboy DP

Implementation

The bulk of the code for network syncing can be kept in in NetworkModule.py without needing to tightly integrate with any of the core Conduit. The code will be split into a server and client portion. The server will "share" a dataprovider. A client consumes the data from this as though it was any other web service.

Discovery

Avahi will be used to perform Conduit discovery on the local network. It's purpose will be to collect Conduit REST URL's. URL's can also be manually added (initially through gconf or a .conduit file?) for when you are syncing devices across the interweb. At this point, the URL list could easily be filtered against a blacklist or whitelist.

For each URL it discovers, Conduit will connect to it and obtain a list of dataproviders and information about them (icon, name, description). Again, this list could easily be filtered to ignore, for example, certain datatypes.

The Server

To share a dataprovider the user will see it as though they are creating a conduit with a Tomboy dp and a Network dp. The network dp doesn't necessarily need to implement anything - the user just needs to be able to add the dp to the canvas. It is similar to a Pending DP (will block attempts to sync on the server side) but will have an icon and a name to highlight this data is available on the network. By connecting a networked dp to tomboy the user is making it clear that they want to share tomboy data.

When Conduit starts a Twisted Web XML-RPC server is instanced (as part of NetworkServerFactory). This will map XML-RPC calls to actual dataproviders that reside on the server.

Avahi will be used to discover Conduit URL's. These will be held in a list in Conduit.

The root URL of the Web2 server will list available dataproviders. Combined with avahi, this will form the base of the discovery features. By moving the bulk of discovery into the HTTP code we are prepared for situations where we are syncing outside the realm of avahi (interweb syncing).

The Client

NetworkClientFactory will use avahi to detect new dataproviders on the local network. When a new dataprovider is detected a ClientDataProvider will be added to the tree. This will contain information such as the URL to sync to and info such as the name, description and icon of a dataprovider, which was retrieved from the base URL. This dataprovider can be added to the canvas like any other. Other than being listed under a category for the remote machine it would look just like a local dp (e.g. a Tomboy dp).

When you sync against a ClientDataProvider it will in reality be syncing to the remote Conduit DataProvider. The only way this is any different from syncing to any other web service is that one set of code will look like multiple dataproviders by changing the icon, name, description and category.

Unresolved Issues

  • Authentication
    • Shouldn't be a problem in Twisted. Initially have a blanket username and password that covers the whole "server".
  • Configuration
    • Implement like the plugins in Rhythmbox/Banshee
    • Implement a hook that allows factories to hook in to the preferences and add a tab.
  • File sync will be a little tricky - especially large files. Need to consider ways of splitting the data from the meta-data so that the bulk of the file doesn't have to be pickled and can be downloaded *only* when needed.

Task List

  • Implement dummy Server and Client dp's
  • Each Server dp should be advertised
  • Client should add a Client dp for each detected Server dp
  • Server should provide an XML resource describing itself
  • Client should be able to grab XML resource and make itself look like the dp it is proxying
  • Implement put (on both sides)
  • Implement delete (on both sides)

At this point we have a "Sink" server.

  • Solve HTTP sessions problem
  • Implement get (on both sides)
  • Make it threaded (needs checking in, blocked till spoken to Nzjrs)
  • Make it obvious on canvas which dp's are shared. Color code?
  • Test, test, test!

Pie in the sky

  • Refactoring for a GTKless sync-server mode
  • Then can run Conduit on an x-less Rimuhosting account
  • Have a dumb database store DP (or just a folder dp) and let anything be stored in there
  • Voila, enterprise ready "Tomboy Server edition"