Changes between Initial Version and Version 1 of DistributingChanges

02/22/10 17:44:52 (4 years ago)

initial considerations for distribution of CKAN model changes


  • DistributingChanges

    v1 v1  
     1Abstracting and continuing considerations from: 
     2 - SyncingInstances 
     4Apparent wants and needs: 
     5 - to distribute local CKAN changes to foreign CKAN instances 
     6 - to preserve history (perhaps so that normally each instance's Recent Changes resembles the others) 
     7 - to support selective distribution (either by selecting for some logical subset of CKAN records, or by manual approval of changes on a case-by-case basic, or a combination, or otherwise) 
     9Inferred wants and needs: 
     10 - to support use through firewall (client-server: where client pushes changes on client to server, and client polls and pulls changes on server to client)  
     11 - to support use across internet (peer-peer: each accepts listeners, each can register to listen for changes, each notifies its listeners of any new changes, each listener pulls changes it doesn't have, etc) 
     14Highlighted core models of Mercurial and of CKAN: 
     15(because it is thought that CKAN is like a DVCS) 
     17of Mercurial: 
     18 - Repository (create with: hg init; hg clone) 
     19 - File (create with: editor) 
     20 - Repository History (create with: hg log; hg glog) 
     21 - Changeset (create with: hg commit; hg merge) 
     22 - Working Directory (create with: hg update) 
     23 - Changeset Patch (create with: hg export) 
     24 - Branch (create with hg: pull) 
     26of CKAN: 
     27 - Repository (create with: paste setup-app) 
     28 - Package 
     29 - Tag 
     30 - Group 
     31 - Recent Changes 
     32 - Revision 
     34Concepts that don't easily carry over from Mercurial to CKAN: 
     35 - Mercurial Branch (CKAN just has a single change history) 
     36 - Mercurial Merge (CKAN doesn't have any branches to merge) 
     37 - Mercurial Push/Pull (CKAN can't send or receive foreign branches) 
     38 - Working Directory (CKAN presents its repository directly) 
     40Concepts that do easily carry over from Mercurial to CKAN: 
     41 - Changeset Patch 
     42 - Export/Import (which BTW causes changes to be applied to the working directory before committing, that is different from Pull which doesn't affect the working directory - also worth noting that import normally aborts if there are outstanding changes in the working directory, which would carry over to CKAN should received changeset patches be queued and progressively applied unless there is a conflict, which [could] normally cause the queue to be held up, and notification to be sent to the site admins to intervene, after the conflicting patch is resolved the queue would continue until the next conflict -- other behaviours could include: automatic merging; automatic skipping; intervention each time) 
     44Actions needed in CKAN to distribute changes (functional requirements): 
     45 - Changeset patch, creation (on new revision: create serialised diff) 
     46 - Changeset patch, publication (handle register-get and entity-get, searchable, publish-subscribe) 
     47 - Changeset patch, retrieval (get and add new changeset patches to local queue entity-get, add to queue) 
     48 - Changeset patch, conflict detection (possibly by asserting either that new values of changed attributes match current values of same attributes in model - so the patch would leave the local model in it's current state, or that old values of changed attributes match current values of same attributes in model?) 
     49 - Changeset patch, resolution (human response to conflict notification, decide new state, continue the queue) 
     50 - Changeset patch, application (model merge, record changeset patch has been applied) 
     52 - Model Merge (to include add/remove aggregated children e.g. packages) 
     53 - Model Package Merge (to include add/remove child associations e.g. taggings) 
     54 - Model Tag Merge (if there are any editable attributes ) 
     55 - Model Group Merge (if there are any editable attributes) 
     57 - CKAN merge (merge queued changeset patches into the model: FIFO, for each: if changeset patch conflicts, request resolution and stop; otherwise apply changeset patch and continue with next in queue) 
     58 - CKAN pull (retrieve new patches) 
     59 - CKAN push (send new patches) 
     61Sub-domain models needed in CKAN: 
     62 - changeset patch model (need uuids for changeset patches, need each to record which patches have already been received, need to record what state of application they are in, need to arrange things so that new change numbers are created only in the case where the change does not arise from applying a patch) 
     63 - changeset patch notification publish-subscribe model (for event-driven changeset patch distribution) 
     66 - the more divergent two instances the more likely it is that a changeset will conflict, so there is a very good reason to make the changeset distribution loop as tight as possible (in order to minimise the need to conflicts to be resolved) -- hence the event-driven peer-peer considerations 
     67 - the frequency of human intervention will also depend on the strictness of the changeset patch conflict detection and the forcefulness of the patch application mechanism