Planned enhancements to ZSyncer
===============================

Cleanup / Refactoring
---------------------

Look at factoring out all the conditionals on use_relative_paths.
(Since the ZSyncer itself has persistent configuration, this might
require moving much of its functionality into a new class that does
the actual syncing and which would be instantiated on the fly.)

Need to audit all remote methods and ensure that they consistently
trap the right errors, return meaningful response codes, etc.  And
check the local callers for consistency too.  I think it's kind of
a hodge-podge now.

Also clean up the UI to more consistently use StatusMsg and
response streaming.  Probably some refactoring to do there.

Our separate logfile should be reimplemented using the python logging
module.  (It would also be nice if there were a config option to
mix these in to zope's standard event log.) 

Testing
-------

It would be good to refactor the tests so that:

- A dummy connection type is implemented and used for all zsyncer tests,
  so none of them make any network requests.

- The connection type API is explicitly spelled out somewhere
  e.g. write a formal interface.

- Tests are written for each connection type, ensuring that they
  implement the interface properly. These tests *would* make
  network connections.  (Currently, only ConnectionMgr has any
  tests and those could use work too.)

Bugs
----

These should be moved to the Sourceforge collector.

* Many operations that should be transactional are not.
  e.g. manage_touch makes a separate request for each touched object.
  If some of them fail, you end up with inconsistent state.  In
  general, all should succeed or all should fail.  This can be done
  by making more use of callManyRemote.

* manage_approvedAction is useless in undo logs since you can't tell
  what it really did. See if we can improve that.

* A report of failure on Windows that I have not reproduced: "illegal
  data at start of file", from Kamal Gill to zope list, from 29 Oct
  2003.

Known Plone Issues
~~~~~~~~~~~~~~~~~~

* ATContentTypes don't work with the "diff" feature.
  You can fix this by adding appropriate
  method names to the diff_methods dictionary in Config.py.

* Order of content objects in folders is not preserved
  when you sync in Plone 2.1.  Patches welcome!

* Sync Status doesn't work in Plone 2.1 when viewing a page
  that's set as the default view for a folder.
  See http://dev.plone.org/plone/ticket/5082
  and https://sourceforge.net/tracker/index.php?func=detail&aid=1397852&group_id=28073&atid=392340

CMF Timezone bug
~~~~~~~~~~~~~~~~

If you sync across multiple timezones, everything should "just
work".  There was a problem prior to CMF 1.5.1 in that the
folder_contents view defined by CMFDefault uses the Date()
method of each object, which displayed the DublinCore modification
time according to the timezone in which the object originated, but
does not display the timezone. In CMF 1.5.1 this was fixed to
convert the date into the local timezone before display.

Other Known Issues
~~~~~~~~~~~~~~~~~~

The "status" display can easily be wrong. We rely on
bobobase_modification_time for objects that don't provide DublinCore
timestamps.  This is not good. Among other things, it means that you
frequently get "out of date" for no good reason.  Be suspicious of
the status. Again, we recommend that you keep the clocks on your
servers in sync. This will greatly reduce the likelihood of false
status reports.

"status" display for FilesystemDirectoryViews is meaningless.
In debug mode, zope updates their bobobase_modification_time 
periodically.

Performance issues
------------------

* Big syncs: Transferring a lot of stuff fails because ZSyncer does
  everything in memory.  E.g. if I sync a folder containing ~70 MB of
  stuff, the destination server's ram useage shoots up by 300 MB!!!
  Note that import of large .zexp's seems to succeed in cases when
  zsyncer fails.

  This may be somewhat mitigated since we got rid of XML-RPC; should
  test it again.

  Things are also somewhat better since we switched to using a
  tempfile instead of StringIO when syncing something larger than
  Config.upload_threshold_kbytes.  **Still haven't done this on the
  client side for uploads**

  Further possible optimization: for pull only: Use a separate thread
  (or process?)  to build the pickle to a temp file, and send it back
  in chunks with a content-range with an empty total size.
  Or - probably better - _p_jar.exportFile() can write to
  a TemporaryFile; we could use a zpublisher IStreamIterator
  implementation, maybe the default one would just work.

  For push, there doesn't seem to be anything like content-range for
  POST.  So I probably have to start uploading after building the
  whole file.  (One possibility to investigate: connection:
  keep-alive might help? or not.)  But I can still be nice to the
  client by using httplib's connection.send(chunk) in a loop and
  never have to load the whole thing into memory.  (But this has to
  be implemented per each back-end.) OR - suggested by chris
  mcdonough - use a PUT request instead, which afaik means the
  httplib client code doesn't have to fiddle w/ the request much at
  all - just hand it a file and it behaves sensibly.  (The server is
  another question... apparently it currently makes multiple copies
  of the tempfile, see
  http://mail.zope.org/pipermail/zope-dev/2005-April/024616.html)

UI enhancements
---------------

* Switching syncers while working is clunky. You either have to edit
  the syncer, which is error-prone, or go out of the current syncer
  and into another one.  Instead of having a bunch of syncers
  configured for different destination servers, maybe have one syncer
  configured with multiple destinations, and in Folders.dtml allow
  the user to toggle which of the configured destinations are used
  for comparisons and syncing.

* Currently with multiple destination servers, only the first one is
  used for comparisons. This is not very informative.

  Quick solution: If multiple destination servers, allow user to
  toggle which is used for comparison (without changing the syncer
  config)

  More extensive solution: provide a combined comparison
  mode in which each object's status is shown on all destination
  servers.  The status could then be any of::

       ok
       missing
       out-of-date
       ok/missing
       ok/out-of-date
       missing/out-of-date
       ok/missing/out-of-date

  alternatively, have a status column for each destination.
  Or maybe rows within each object's row?
  This a lot of information - especially with some of the other
  recent enhancements like showing the mod. time for the destination -
  now we have to decide WHICH destination to show, or how to show all
  of them, or whether to show the latest only, etc.
  Hard to keep the UI from getting cluttered with all this info.
  The next item might help wit this.

* status column could contain links to more detailed information,
  e.g. in the multi-destination case, show the usual comparison
  for each server in turn. This could also be used to do more
  expensive comparisons as described below.

Functional enhancements
-----------------------

* Config.py should be replaced by extensions to zope.conf.
  Look into how that's done.

* Log dir should be configurable in the config file.

* Comparison enhancements

  Motivation:

  - Timestamps give wrong results if the servers do not have closely
    synced clocks (destination = source +/- network latency), and even
    then can be misleading.

  - Prevent false out-of-date results: wastes time for the user
    investigating and/or needlessly syncing.

  - Prevent false OK results: if something is changed directly on the
    destination server, it will compare "OK", which makes it very
    likely that emergency patches to a production server will get
    clobbered unless great care is taken to back-sync them.

  Goals:

  - Comparison code should be pluggable somehow.
    Default comparison should be enhanced as follows:

  - any difference in DC Metadata -> not equal.

  - any difference in Properties -> not equal.

  - any difference in security settings (permission/role mapping, local
    roles) -> not equal.

  - for ObjectManagers: comparison should continue to be atomic rather than
    recursive by default.

    Recursive means that if foo/bar/bat/baz is considered different,
    then foo, bar, and bat would be considered different even if
    there are no other differences.  Atomic means that we ignore
    foo._objects.  Atomic is both easier and cheaper.  Also,
    recursive would be confusing since you don't know how much of the
    tree you need to sync to fix the problem.

    A better solution to recursive comparison is already implemented
    in the UI.

  Ideas for additional comparisons:

  There are many potential comparisons but many are likely to be
  too expensive for routine recursive use. this should be investigated.
  One option would be to give the user control over which comparisons
  are performed.

  - improve timestamp comparison:
    I don't think this is possible for non-CMF objects,
    for the reason given below.

    this was my strategy::

     dest_time = remote call to DateTime()
     local_time = DateTime()
     fudge_factor = dest_time - local_time

    when comparing times::

     if dest_modtime <= source_modtime + fudge_factor:
        status is OK

    PROBLEM: if the clocks on the two systems have changed
    since dest_modtime or source_modtime, this doesn't work
    at all. And for hypothetical two-way syncs, it's useless
    even if the clocks are right.

    Somebody on zope@zope.org suggested that zsyncer should
    forcefully set the mod time of the source & destination
    objects so that they match. This is not possible
    since bobobase_modification_time is database-level and zope
    can't control it.

  - CMF content is easier since it has real metadata: we can ignore
    bobobase_modification_time and use the DublinCore modification
    date as our timestamp. When comparing, we're OK only if they are
    identical. This should work very reliably. DONE!

  - add a new status ("Changed"? "Newer"?) if the timestamps look ok
    but other comparisons indicate a difference.  (should this be
    red, orange, or other color?)

  - compare size - easy and cheap, will catch many differences, but
    ObjectManagers and many other objects don't support it or might
    have a different interface for getting it. (get_size vs getSize,
    maybe others)

  - compare metatype: this always works.  unfortunately it's unlikely
    that we have 2 different types, but it does happen.  so this is
    probably only rarely useful as a way to avoid more expensive
    comparisons.

  - compare all DublinCore metadata if available. Again, mostly
    useful as a short-circuit; even so, comparing DC modification
    time should prevent the need for this.

  - simple comparison of the object's "content", whatever that is -
    e.g. body of a DTML Method. Not useful for all types.
    Potentially too expensive on the network and not useable for all
    types - would need to be toggleable. Should only be done after a
    meta_type check.

* recursive sync: a new (toggleable) sync mode, in which zsyncer
  tries to walk down the tree and automagically do the right thing
  depending on status: sync only missing or out-of-date things (don't
  try to descend into missing folders, just sync them).  This could
  leave some out-of-sync objects if the comparisons are wrong, but it
  could also be a big win when syncing a few scattered changes in a
  huge folder tree.  Result should report only the items that were
  actually synced.

  There should always be a confirmation page in this mode, that shows
  you what would be synced and allows deselecting items.

  Hmm, this is not really necessary if I just fix the filtering in
  recursive comparison mode.

* Aspect-based syncing:
  Currently, sync is all-or-nothing.  What if you have a 200 MB file
  and change its permissions, or its properties, or its CMF metadata?
  You have to sync the whole thing.  Worse, what if you have a huge
  portal_skins folder, with some stuff you don't want to sync yet,
  but there are changes to the skin paths (properties) that you need
  to sync ASAP?  Another use case: Access Rule settings on a folder.

  This is not a YAGNI - I have needed it for months :-)

  one problem is making this extensible to handle more features, e.g.
  custom products.
  maybe just register the attributes that are considered "content".

  The new _callRemote() method is intended to be useful here.

* "Unsyncable": it should be possible to flag an object (not just a
   meta_type) as unsyncable. On the source, this would disallow
   syncing it; on the destination, this would disallow
   manage_deleteRemote.  (OK, that's really 2 flags.) The UI should
   tell who marked it, when, & why.  The rationale is similar to the
   "immutable" bit in unix. The syncer itself could store these as an
   OOBTree whose items are path: flag. This could get stale if objects
   move. Or, set properties on objects, like zsyncer_unsyncable
   (boolean); but that might annoy people, and requires the object to
   implement PropertyManager.  TANSTAAFL.
