There are a number of extra properties we want to store in a resource:

• resource size (in bytes, no. triples etc.) - ​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/673
• resource hash is 'static' (default) or 'changeable' - ​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/674
• alternative link (e.g. TNA Web Archive) to use the link is broken - ​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/173

i.e. this is a much needed enhancement

Need to be able to read/write these values in web interface and API.

(At some point we may wish to have custom form widgets for these new fields in the resource table, but let's leave this for the future.)

Implementation

I suggest these are arbitrary key/value pairs, just like package extras, reusing PackageExtra? code as much as possible. (Other much more radical alternative is to move model to RDF.)

The ckan queuing system should provide the option to subscribe to timed re-submissions of specific resources. This could look as follows:

• routing_key: Package
• operation: daily
• payload: pkg.as_dict()

Where operation is one of daily, weekly, monthly or any other interval.

Why?

• Will make it easier to give resources their own form, independent of packages.
• Makes uploading, harvesting and previewing data cleaner as its separate from the packages.
• Gives the option of sharing a resource between packages.
• Leads the way to supporting storing richer semantic information against them.

Child tickets:

• #1107 Move package autocomplete from package controller and move to API
• #1087 version and contact info api call

Move to a format that has a separate responseHeader and response.

A standard package response

{
  responseHeader: {
    status: 0,
  }    
  response: {package-dict}
}

On error:

{
  responseHeader: {
    status: {err-code},
    error: 'message'
  }    
  response: none
}

A search query

Based directly on solr.

{
  responseHeader: {
    status: 0,
  }    
  response: {
    numFound: 5,
    start: 0
    docs: [
    ]
  }
}

Issues

This is a breaking change for clients

References

• ​http://wiki.apache.org/solr/SolJSON
• ​http://www.flickr.com/services/api/response.json.html

The new field was outlined in

​http://lists.okfn.org/pipermail/ckan-discuss/2011-February/000887.html

it says

Can be "data", "metadata", "service", NULL. There is a need to do this for UKLP where some of the data associated with a package is metadata, some is data and some could be a service.

In order to test migrations properly, we need to test the upgraded database against *all* the tests.

To do this a special flag to nosetests should be added. nosetests --migrations

This flag will make sure that the database is recreated from scratch each time using the migrate repository, bypassing init_db.

This was missed out the migrate script. It should be to be added as a conditional script at the end of the repository so that databases can catch up.

We need to support extending the model from plugins. This could involve:

• Adding a plugin hook to extend the mapper
• Adding an upgrade hook for plugin schema migrations
• Documenting how this is to be done
• Find a way to avoid conflicts

Try and minimise the amount of different caching methodologies, especially for api calls.

When doing create on the cli upgrade should be run instead of create to make sure that every database is initiated in the same way.

Not used and extra overhead. Very useful to strip this out and low cost.

Places where it seems to be:

• lib/base.py: c.time_call_started
• ....

Revision history information is not accessible, dgu want this.

The stages involved with doing this.

• Convert model objects to standard dict format (DONE)
• Convert standard dicts to current api formats (DONE)
• Make standard dicts savable (DONE)
• Validate standard dict format. (DONE)
• Authorize actions

Option 1: 'Changeset' Model

See ticket:1135 for vdm ticket. This would involve a) moving to changeset in vdm b) doing the migration in ckan to support this.

Have developed a new "changeset" based model for revisioning in vdm.

Implementation

• The main challenge with this change is schema and data migration

Every revisioned object has a revision_id and revision attribute.

Approximate algorithm:

Revision -> Changeset

for revtype in [PackageRevision, ...]:
    for pkgrev in package_revision:
        changeset = lookupchangeset(package_revision)
        ChangeObject(cset, (table, id), dictize(pkgrev))

Question:

• does pkg include tags attributes or not? or we have to dictize, pkgrev, pkg2tagrev, and tag. Probably the latter.

Option 2: Simplify Revision Object Model

Just use a simpler vdm, see ticket:1136 (move to SessionExtension) and ticket:1137 (remove need for statefulness in vdm).

Discussion

Advantage of Option 1 versus 2:

• Easier support for pending state and similar behaviour
• No need to introduce new tables (and hence migrations) when making something revisioned (or not).

Disadvantages

• Migration is required
• More difficult to query revision history.
  • Could be addressed by having ChangeObject have separate cols for table name and id but would likely be more difficult.
• Performance (?)
  • Have one big ChangeObject table to query when looking at changed objects rather than many revision tables.
    • Not sure this is a biggie as even with Revision model biggest revision object tables are probably on the order of the ChangeObject table

Conclusion

Implement Option 2 and leave Option 1 for present.

Option 1 includes Option 2 so it seems that that is required in either case (so we may as well with Option 2).

Option 1 requires significant effort (esp migration) so leave for present and then review the situation at some later date.

• Convert current api saves to the new standard dict format.

We need a way to pass in schemas to the logic layer to deal with edge cases.

We need to see what areas of ckan are slow.

Move to Changeset model for vdm.

A changeset model is like an Audit-Log model in which we just record Changesets with Change-Objects rather than have Revision-Objects for each Object that is revisioned.

This change would also incorporate significant simplication of vdm.

Statefulness, especially statefulness for relation (esp m2m) is cause of most of the complexity in vdm. It is required because, atm, revision objects have FKs to continuity objects.

This ticket proposes the following changes:

NB: this could be limited just to case of join tables (leaving state stuff on other tables)

• Remove FKs from revision to continuity (or allow for them to be nullable).
  • We could just limit this to m2m stuff
• Delete of an object leads to:
  • Deletion of continuity object
  • Adding an entry in revision table with state set to deleted (we retain state on revision table)

If this is done we will no longer need to worry about filtering on state on relationships as join table will only contain "active" relationships.

If we do this on all tables we remove need for any state awareness in client (e.g. no need to filter tables on active state).

The only disadvantage of this change is that undeletion becomes more problematic (we have to recreate some continuity objects).

Revision tables need expired_id to make querying history AND pending changes more efficient.

This involves making a session extension and a large table migration.

This should make it more efficient as it currently does a lot of repeating work. i.e if you change a package and a resource in the same commit it sends out 2 notifications and should only really send out 1.

When upgrading database for cz.ckan.net (psql.okfn.org/cz.okfn.org) an error occurred (see below).

cz.ckan.net has been running on eu3 and this updrade was part of moving it to eu19, with a more recent version of ckan.

In the meantime I've reverted the db to the old version (pre-upgrade).

cz.ckan.net is not working properly until this is fixed.

okfn@eu19:~$ paster --plugin ckan db upgrade --config=/etc/ckan/cz/cz.ini
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/url.py:105: SADeprecationWarning: The SQLAlchemy PostgreSQL dialect has been renamed from 'postgres' to 'postgresql'. The new URL format is postgresql[+driver]://<user>:<pass>@<host>/<dbname>
  module = __import__('sqlalchemy.dialects.%s' % (dialect, )).dialects
2011-06-29 09:07:41,294 INFO  [migrate.versioning.api] 28 -> 29... 
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_lname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_slname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_suname
  ret = fn(self, con, *args, **kw)
/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/reflection.py:46: SAWarning: Skipped unsupported reflection of expression-based index idx_pkg_uname
  ret = fn(self, con, *args, **kw)
Traceback (most recent call last):
  File "/usr/bin/paster", line 18, in <module>
    command.run()
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/pymodules/python2.6/paste/script/command.py", line 218, in run
    result = self.command()
  File "/usr/lib/pymodules/python2.6/ckan/lib/cli.py", line 84, in command
    model.repo.upgrade_db()
  File "/usr/lib/pymodules/python2.6/ckan/model/__init__.py", line 163, in upgrade_db
    mig.upgrade(self.metadata.bind, self.migrate_repository, version=version)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 185, in upgrade
    return _migrate(url, repository, version, upgrade=True, err=err, **opts)
  File "<string>", line 2, in _migrate
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/util/__init__.py", line 160, in with_engine
    return f(*a, **kw)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/api.py", line 364, in _migrate
    schema.runchange(ver, change, changeset.step)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/schema.py", line 83, in runchange
    change.run(self.engine, step)
  File "/usr/lib/pymodules/python2.6/ckan_deps/migrate/versioning/script/py.py", line 140, in run
    script_func(engine)
  File "/usr/lib/pymodules/python2.6/ckan/migration/versions/029_version_groups.py", line 152, in upgrade
    migrate_engine.execute(q)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1788, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1191, in execute
    params)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1271, in _execute_clauseelement
    return self.__execute_context(context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1302, in __execute_context
    context.parameters[0], context=context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1401, in _cursor_execute
    context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/base.py", line 1394, in _cursor_execute
    context)
  File "/usr/lib/pymodules/python2.6/ckan_deps/sqlalchemy/engine/default.py", line 299, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) insert or update on table "package_group_revision" violates foreign key constraint "package_group_revision_continuity_id_fkey"
DETAIL:  Key (continuity_id)=(7f373978-40a0-4848-bdc9-018ad213aa0b) is not present in table "package_group".
 'INSERT INTO package_group_revision (id, package_id, group_id, state, revision_id, continuity_id) VALUES (%(id)s, %(package_id)s, %(group_id)s, %(state)s, %(revision_id)s, %(continuity_id)s)' {'state': u'active', 'package_id': u'14d76e9e-028c-49c1-955b-ef30d4abb970', 'continuity_id': u'7f373978-40a0-4848-bdc9-018ad213aa0b', 'revision_id': u'24a11f58-2ff9-46f3-b113-0bf7f0baf254', 'group_id': u'4b832199-b8e3-4989-9b5d-de8dd6322612', 'id': u'7a3863bd-3054-40c8-bb3d-68b73096c108'}

Child tickets:

• #1101 Integrate stats and googlanalytics into site nav

We have a spreadsheet from UKLP of statistics we'd like to generate

(Just creating this ticket as somewhere to keep notes)

• Decide on REST api versus action API
  • Do we want to support both?
• Tidying
  • Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...)
  • extras merged into normal field list in package
  • Get rid of /rest/ so just have api/v1/package
  • Get rid of separation of search api from 'rest' api
    • Propose that GET on REST index is search e.g. /package/?q=...
      • This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned)
• Resource read/write in API (separate from package)
  • Does this need authorization work?
• user/account API - read/write
• Remove autocomplete -- can just use search
  • Do not worry about backwards compat as should only be used in our js (if others using it too bad!)

Solr is not working for moderation, pending changes are not being indexed.

At the moment this appears to be slow (kindly suggests because missing index in db). Page is /user on a ckan instance.

Would this also speed up user page where we recent changes?

This should be done from the logic layer or automatically from a session extension.

curl http://ckan.net/api/action/package_show -d '{"name": "uk-quango-data"}'

This gives 500 error due to KeyError? (it looked for the 'id' param), rather than 400.

Provide an API to the logic layer at /api/action.

Spam users where added to thedatahub and we need to clean them.

There is some evidence pointing to CKAN handling memory inefficiently or even leaking under certain conditions:

When we migrated ckan.net/thedatahub.org from eu7.okfn.org (32bit) to s053.okserver.org (64bit) (​ticket) we experienced extraordinary memory usage peaks (​ticket). Here are the observed value with Apache default settings:

• eu7, mpm-prefork: base level ~0.6GB, peaks up to 2GB
• s055, mpm-prefork: base level ~1GB, peaks up to 4GB
• s055, mpm-worker: base level ~1.5GB, peaks up to 6GB

William reduced the life-time of a WSGI CKAN process from 500 requests down to 25 requests (​changeset). This (together with two other tweaks) changed the situation drastically:

• s055, mpm-event: base level ~1.4GB, no peaks

This suggests that the more requests a CKAN processes serves over time, the more memory it consumes, aka bad memory management or a leak.

To prove this theory, one could reduce the total number of WSGI CKAN processes as much as possible without killing the performance (e.g. down to processes=3), and then observing the relation between maximum-requests=25...500 and memory consumption.

On 14/09/11 17:49, David Read wrote:

Someone to do a bit of top-down memory-use profiling would be very useful. Also useful would be something in the tests that reported what test cases use lots of memory - this could be in the nose plugin.

+1

We should ignore these but log the packages that are causing this to investigate.

Need to add way for extensions to trigger based on resource change and just resource url change.

• Possible sibling / requirement: #1397 (resource archiving)

• Setup test environment. 1d
• Make task to upload files to webstorer. 3d
  • This includes writing back webstore_url and updated date. 0.5d
  • Type guessing of file format. 1d
  • Make cron to upload all un webstored resources. 0.5d
• Testing for errors by running cron 2d
• Make nice way to deploy celery 1d.
• Documentation. 1d
• Deployment. 1d

Give site user system administrative rights.

Periodically we see some CKAN servers fall over because they run out of disk space. We need to find out if there is a common cause and fix it.

One problem in the past has been file handles running out when creating lots of tiny files in the data directory.

Another problem has been several enourmous backups being created every day - pdeu on eu25.

This will comprise of the following steps.

• model migration.
• simplify revisioning
• Remove any sqlalchemy relations from groups.
• change logic functions

estimate 2d

From Jan Kučera <elquenor@…>

I have also tried to use extras fields to store information about relationships between datasets. I found out that if you first add relationship using CKAN API into dataset relationships attribute and than you add extras field for the same relationship with key in format "parent_of:dataset_name" or "child_of:dataset_name", storing this extras field will clear all relationships in relationships attribute stored earlier.

I might be some kind of bug but I suppose you will know better. Is it possible to use relationships attribute and extras field for relationships at the same time?

Add a function to logic.action.get that returns a user's public activity stream as a list of dicts.

The new members table needs a way so you can add arbitrary domain objects against them.

We need to extend the group schema to accept types, and instead of just being able to add packages to groups add members with their capacities that associated with different table rows.

4d

Multi-language:

• dataset and resource metadata (and other objects such as groups?)
  • Options for implementation in ​http://ckan.okfnpad.org/multilingual
• Field values in taxonomy (e.g. country names - Eurovoc)
  • #1665 Research into Eurovoc
  • Display Taxonomy in different languages
• Field values not in taxonomy (e.g. title & description)
  • Use extra fields e.g. _i18n_title_fr = Le data.
  • (If lots of fields would need translating then would consider having a new package for each language, linked together with PackageRelationships?. But I think it is just title and description (resource description etc. are so minor, not worth translating?), so using extra field better.)
• EC extension (templates, form)
  • Currently the pot file is just for CKAN core. New pot file for CKAN core and this extension? Or separate ones for extensions?

Its a pain to see the difference between branches as there are normally a lot of po file transaction. Make the default be see that they have changed without actually show the diffs themselves.

The admin extension allows deleted datasets to be viewed, but there is no equivalent in the API. The package_list API call filters to just 'state=active' datasets.

'state' could be a parameter on the package_list call

• Easy to call and implement
• But getting permissions right for the different state values may not be easy - maybe a separate logic function is better?

Translate all areas of metatdata to selected language. This includes finding datasets in all languages.

Multilingal Metadata: Tasks:

• Add translation table. 2d
• Api to add translation 3d
• Add extention point to dataset view. 3d
• Add extension point to search index to add translations. 2d.
• Add all eurovoc data to translations. 1d.
• Make multilingual solr schema 2d #1715

Test failures in master due to tom and rufus not running tests. 0.3d.

• Upgrade to new DataStore? system
• Better support for unicode etc
• ? Rename to DataStorer?
• Deploy

A new table with 3 columns should be added. term, term_tranlastion, language_code. This table will be used for all translations, including tags. The table should have indexs on both the term and (term, language_code) combination.

Add api to translation to the term_translation table.

• add to logic 2d
• test 1d

All translated fields and vocabularies need to be added to search index in the correct fields.