This is a meta-ticket to pull together all the work on SOLR as a backend for CKAN search. (Work on SOLR search has been going on since March of 2010).

3 key aspects of this:

• Index package information in SOLR - ticket:353 - SOLR search indexing (index CKAN)
  • Synchronously (generify Search Index update method to support SOLR or postgres?) - #317: Make search pluggable
  • Asynchroously: via worker with queue
    • #324 Search indexing using notifications
    • #669: Refactor Solr (indexer) to become a generic worker
    • #874: Extract Solr search backend into an extension
• In core support using SOLR backend as well as postgres for queries
  • #317: Make search pluggable (?)
• Integrate into WUI (if changes needed)
  • #672 - Facets in CKAN search results

Extras:

• Solr index testing tool #431

This is a regression that happened after refactoring the api.

It was shown by

​http://pastebin.com/2v7QasZy

This is a reworking of the existing apps extension.

Initial proposal at ​http://wiki.ckan.org/Proposals#Apps_in_CKAN and ​http://wiki.ckan.org/Proposals#References.2FLinks_in_CKAN

Naming

What do we call this extension?

• related
  • RP best IMO but perhaps too close to separate relationships concept?
• relatedstuff
• links
• references

Proposal

"Related Stuff": Apps as in an application (website/service/tool) that uses this dataset (as in Apps and Ideas extension)

• This case seems a general example of wanting to link some external thing to a dataset (e.g. this policy paper refs this dataset, this application uses this dataset, this visualization is built on this dataset cf ​http://lists.okfn.org/pipermail/ckan-discuss/2011-August/001607.html)

Implementation

New table named Related with following structure

|| id (int) || type || title || description (markdown) || image_url || owner_id || url || created (timestamp)

• type = Idea | App | API | Visualization | Post | Paper | News Article
  • Suggest we make this a ckan.ini config option (comma separated ...?)
    • Do we want the possibility of different templates for different types of Related objects?

• image: ?? Depends where we store images. Simplest option would be to change to image_url and leave it to users to have already uploaded an image somewhere. If not we need to support image uploading and storage. See #1692 (add image attribute to datasets and groups) for more discussion, once implemented the URL here can be an internal url.

• owner_id = user_id or creating user (see authorization below)

Related2Dataset (note that related_id, dataset_id tuple should be unique). This allows for m2m connections. If a given related item is only with one dataset this could be simplified. May contain status so dataset owner can turn this on/off.

|| id || dataset_id || related_id || status

status should be used to allow for a dataset owner (for dataset_id) to de-activate the relationship between the dataset and the related.

Url

• /dataset/{dataset-name}/related/{related-item-id}/{related-item-title-stringified}
  • If a reference item could exist in its own right (and perhaps refer to multiple datasets then it should get its own url at e.g. /related/{id}
• /dataset/{dataset-name}/related/add => Modal dialog on related tab so we can use API to create them.

/dataset/{id}/related <- list

• use image_url for small icon etc, title description (shortened?)
• Click through to full related item (optional)
• dataset owner is shown show / hide button ... (or on /dataset/{id}/related/{id} )
• related owner sees an edit button / icon (pops up modal)

/dataset/{id}/related/{id} (optional)

/dataset/{id} will have a Related tab (with bubble with count).

• Drop down with Add Related -> Pop-up modal and save via API

(Not used: /related/add with dataset prefilled ... )

/related/{id}/edit

Authorization

Addition of related item be considered orthogonal to datasets (and hence with separate authorization i.e. i can add the info that site X uses dataset Y without needing permission to edit dataset Y).

Thus any logged in user could add a Related item. We set the owner of the related item to creating user and going forward only that user or a sysadmin can update or delete.

NB: we could have a system where datasets owners have to approve related items before they show up next to their dataset. This would add substantial complexity so I propose we leave out of phase 1.

Tasks and estimates {7.5d}

[x] Model + Migration for Related table. {0.75d}

[x] Controller for Related (or relevant sections in Package controller). {0.75d}

[x] Routing setup. {0.25d}

[x] Schema for related. {0.5d}

[x] Logic layer actions. {1.0d}

[x] Auth (default + publisher). {0.5d}

[x] Templates + Dataset changes (new tab etc). {1.0d}

[x] JS Application for adding Related objects in a modal. {1.0d}

[x] Testing. {0.75d}

[x] Dataset owner disabling of Related (via M2M table). {0.5d}

• Updated to allow author of related to delete as well

[x] Documentation. {0.25d}

This is a very slow call it would benefit from the sort of speed-ups that package-search received

for me locally this is taking 6 seconds for rufus using the datahub data I have. I think a lot of this is the dataset retrival/dictization

can we just grab json blobs from solr?

also is it possible to specify a sort order/paging?

I've put this as a demo-theme ticket as it is an big issue on the demo we are at 25 second page loads - which i can get down to about 8.8 secs so this is the main pain point now

This is caused by editing a package with resources over the REST interface.

Exception:

URL: http://de.ckan.net/api/rest/package/destatis
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module repoze.who.middleware:107 in __call__
<<          wrapper = StartResponseWrapper(start_response)
               app_iter = app(environ, wrapper.wrap_start_response)
       
               # The challenge decider almost(?) always needs information from the
>>  app_iter = app(environ, wrapper.wrap_start_response)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:51 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.rest:145 in update
<<                  fs = ckan.forms.group_fs_combined
                   fs = fs.bind(entity, data=request_fa_dict)
                   validation = fs.validate_on_edit(entity.name, entity.id)
                   if not validation:
                       response.status_int = 409
>>  validation = fs.validate_on_edit(entity.name, entity.id)
Module ckan.forms.package:385 in validate_on_edit
<<              temp_name = orig_pkg_name
                   self.data['Package-%s-name' % record_id] = u'something_unique'
               validation = self.validate()
               if temp_name:
                   # restore it
>>  validation = self.validate()
Module formalchemy.forms:175 in validate
<<          if self.readonly:
                   raise Exception('Cannot validate a read-only FieldSet')
               return AbstractFieldSet.validate(self)
       
           def sync(self):
>>  return AbstractFieldSet.validate(self)
Module formalchemy.forms:104 in validate
<<          success = True
               for field in self.render_fields.itervalues():
                   success = field._validate() and success
               # run this _after_ the field validators, since each field validator
               # resets its error list. we want to allow the global validator to add
>>  success = field._validate() and success
Module formalchemy.fields:763 in _validate
<<          try:
                   value = self._deserialize()
               except validators.ValidationError, e:
                   self.errors.append(e)
>>  value = self._deserialize()
Module formalchemy.fields:1354 in _deserialize
<<          if self.is_composite_foreign_key:
                   return self.query(self.relation_type()).get(python_pk(self.renderer.deserialize()))
               return self.renderer.deserialize()
>>  return self.renderer.deserialize()
Module formalchemy.fields:183 in deserialize
<<          if self.field.is_collection:
                   return [self._deserialize(subdata) for subdata in self._serialized_value()]
               return self._deserialize(self._serialized_value())
       
           def _deserialize(self, data):
>>  return self._deserialize(self._serialized_value())
Module formalchemy.fields:167 in _serialized_value
<<          if self.field.is_collection:
                   return self._params.getall(self.name)
               return self._params.getone(self.name)
       
           def deserialize(self):
>>  return self._params.getone(self.name)
Module formalchemy.base:63 in getone
<<          if v is None or isinstance(v, basestring) or isinstance(v, cgi.FieldStorage):
                   return v
               return v[0]
           def getall(self, key):
               v = dict.get(self, key)
>>  return v[0]
TypeError: 'int' object is unsubscriptable

Thanks to Friedrich for spotting this.

This is happening whenever people try to run paster commands. polib should only be needed for the check-po-files command don't import otherwise.

This is not trivial, requires a database upgrade. But 0.12.3 has some nice new features (multiple vcs repository support, ticket comment editing and nicer preview) and the latest spam filter plugin requires 0.12.

​http://trac.edgewall.org/wiki/ChangeLog

This is so you can easily look up whether a given package is already listed - otherwise as lists get bigger becomes difficult to see what is already there.

Suggested by Jonathan Gray

This is what caused the exception:

curl http://localhost:5000/api/action/package_search -d='{"q": "osm"}'

The problem is the content-type is formencoding, the '=' is the first character of the request, and so internally the data_dict is set to ' ', which needs to be a dictionary.

This keeps the intended order.

This needs more analysis, but the GEMINI2 attribute "metadata point of contact" must be reconciled with the registered publisher (or agent).

This might also be used to filter records harvested from a CSW source, but filtering also needs more analysis, as does distinction between agent and provider.

This needs to be fixed in the CKAN templates ("no-sidebar" should not be marked for translation) but for 1.7 just fixed it in the translation files.

This occurs when CKAN is run with mod_wsgi (not under paster).

This is because of a unicode header being creating in rest.py.

This overlaps with Sean's follow support (for activity streams) and so will be dealt with there.

This page ​http://thedatahub.org/feeds/custom.atom?q=wombat has 0 results and contains a link to ​http://thedatahub.org/feeds/custom.atom?q=wombat&amp;page=0 which the page=0 causes this exception:

ckan.lib.search.common.SearchError'>: SOLR returned an error running query
Error: "'start' parameter cannot be negative"

This query caused ckan to except because ckan/controllers/api.py doesn't catch SearchError? and SearchQueryError?:

curl http://localhost:5000/api/action/package_search -d '{"sort": "metadata_modified"}'

This seems to require doing something else with such a group's 'delete' button, such as showing it greyed, or perhaps not showing it at all.

This should cover formatting and best practices for HTML, CSS and JavaScript?

Perhaps this should sit in it's own repository (ckan-styleguides?) or gh-pages branch.

This stems from a feature to allow you to prefill fields in the authorizationgroup creation form.

e.g. ​http://ckan.net/authorizationgroup/new?name=science

It's not tested, not supported in the new logic layer and therefore broken. I don't think anyone uses it - can remove it.

This ticket depends on #2423, #2424, #2425, #2426 and #2427.

• Try to remove as many of CKAN's dependencies as possible

• Try to update as many of the remaining ones as possible

• Try to get rid of some dependencies that are fetched from random hg or git servers instead of pypi

• Then try to get the order of deps in the requirements files right so that you end up with the right versions of everything

• Each module in the dependencies file should specify the exact version, to protect us from changes due to them upgrading versions under us and breaking things or changing their own deps

• Consider shipping dependencies with ckan instead of pip installing them?

• This is related to moving CKAN to Ubuntu 12.04

• Add a test case that runs pip freeze and tests that all the right versions of deps are installed?

This ticket refers to the form shown in Drupal. This doesn't have a preview function, but we don't have a requirement for it right now, so leaving ticket closed.

This was missed out the migrate script. It should be to be added as a conditional script at the end of the repository so that databases can catch up.

This was removed during the auth refactoring, but it's needed in the old controllers

This will comprise of the following steps.

• model migration.
• simplify revisioning
• Remove any sqlalchemy relations from groups.
• change logic functions

estimate 2d

This will solve e.g. issues in formalchemy admin interface (which is due to lack of renderer for Json type)

Cost: 0.5h

from ckan.model.types import JsonType

forms.FieldSet.default_renderers[JsonType] = formalchemy.fields.TextFieldRenderer

This would allow for a community-driven definition of field values, conventions, etc.

Plus: create wiki.ckan.net

Thread: ​http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html

Scenario 1

1. User installs Refine and CKAN extension for refine
2. On booting refine and asked to load data they can choose from any data package on CKAN.net (or any other CKAN instance)
3. They edit the dataset on Refine
4. On save (or perhaps as a separate option) they are prompted as to whether they wish to sync the dataset back to CKAN (either as a new package or as a new resource on the existing package)

NB: for the dataset sync back some form of "CKAN" storage would be required (we already have storage.ckan.net running but a closer integration would be required)

Scenario 2

1. User visits a package on CKAN.net (or another CKAN instance)
2. There is a button on the page "View and edit this dataset in Google Refine"
3. Click button -- ask them if they have Google refine installed
   • Yes: instructions for loading dataset into refine
   • No: load dataset in hosted version of google refine (we could run this)
4. User edits dataset and hits save. As in previous scenario they are prompted to sync the dataset.

Three small changes:

• Set Resource type to None instead of 'Undefined'
• Set extra 'UKLP' = True instead of 'INSPIRE' = True

These two changes will require manual update of the database.

• Allow datasets to be updated if the old source is inactive, even if the metadata date hasn't changed:

    if last_harvested_object.metadata_modified_date is None \
       or last_harvested_object.metadata_modified_date < self.obj.metadata_modified_date \
       or self.force_import \
       or last_harvested_object.metadata_modified_date == self.obj.metadata_modified_date and last_harvested_object.source.active is False:

Ticket #787 described join auth between CKAN and Drupal. The authentication part is live and implemented. This ticket is a placeholder for work that will be needed in the new auth system to link authorization functions to Drupal. It is dependent on the groups refactor.

Time estimate: 2d

• Fix logs (apache, ckan, harvest): rotate, set suitable levels
• Fix harvesting jobs: supervisord for gather consumer, cron job
• Fix backups

Also it may be worth setting up a test instance ( on s023 ?)

To allow navigation between sections to still work, also potentially rephrase to avoid confusion that you have to save every time to add a tag or resource and can't say add description, tag and resource in one go

To counter Cross Site Request Forgery attacks, each form generated is assigned a random number in its url, which must be passed when you submit the form.

(Something to consider for the new form mechanism)

To enable us to show DGU work in progress, for feedback.

To maintain backwards compatibility, the package controller (new/edit) should check to see if the controller has a package_form variable defined, and if so render the form pointed to by this variable before calling the new self._package_form() function.

This behaviour is now deprecated however.

To reproduce:

1. paster db init
2. paster create-test-data
3. In Web UI: create new group 'tilo', which includes package 'annakarenina'
4. curl http://localhost:5000/api/search/package?groups=tilo results in 0 results (WRONG)
5. paster search-index rebuild
6. curl http://localhost:5000/api/search/package?groups=tilo results in 1 result

To reproduce:

1. Create a package with two tags the same: "bulk bulk"
2. Click 'save'
3. 500 ERROR - 'Server Error'

To support multiple languages we should have an easy way to specify the language as part of the URL, so that URLs are both specific and we also reduce the dependency on the session.

• Analysis [1d] - Find the best way of implementing this and how everyone else does their language URLs.

• Write Middleware + update url_for to take account of the language. [2d]

• Document the language setup, and how to replicate it. [1d]

Tool that checks which packages have not been indexed.

Required for DGU: ​https://trac.dataco.coi.gov.uk/projects/datagov/ticket/940

Tried to refresh the datagm harvest source to get error message:

"An error occurred: [There already is an unrun job for this source]"

This is unclear and not sure why it created an error. We should have something informative such as "A harvest has already been scheduled for this source"

Do we have a date/time for when this has been scheduled that we can display?

Try and minimise the amount of different caching methodologies, especially for api calls.

Try i18n for one language (German) for ckan (de.ckan.net).

Basic approach to i18n is well-covered in Pylons book: ​http://pylonsbook.com/en/1.1/internationalization-and-localization.html

However, we need to integrate with our Genshi templates. Details on how to do this in comment: ​http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization

Steps

1. i18n templates
   • Extract content using Babel
   • Get genshi translation working
   • Do the translation!
2. i18n all strings in controllers (if any)
3. i18n forms

Try to login to CKAN using Opera 12, get "Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

Trying to run the tests in test_authz.py with

$ nosetests --ckan ckan/tests/functional/test_authz.py results in no tests being run:

Ran 0 tests in 0.840s

OK (SKIP=3)

US-0a

Do not show failed preview with red-bar (esp when just empty datastore ...)

• Data Viewer checks if datastore is empty nad falls back to ther options if emtpy
• DataStore? enabled is off by default and only turned on either explicitly by user or by DataStorer? on successful store ...

Loading/showing error reports

Unfortunately ES is a little bit complex / opaque to newcomers. Need to write a simple tutorial walking through this.

This is documentation / example only (no code!)

Unsure what needs to happen here. Need to list outstanding tasks and implement.

Update report queries with more recent versions (below are report queries shipped with trac 0.11.7):

NB: Report 1 has already been done.

id | query

1 |

: SELECT p.value AS color, : id AS ticket, summary, component, version, milestone, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY CAST(p.value AS integer), milestone, t.type, time :

2 |

: SELECT p.value AS color, : version AS group, : id AS ticket, summary, component, version, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (version IS NULL),version, CAST(p.value AS integer), t.type, time :

3 |

: SELECT p.value AS color,

: 'Milestone '

milestone AS group,

: id AS ticket, summary, component, version, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (milestone IS NULL),milestone, CAST(p.value AS integer), t.type, time :

4 |

: : SELECT p.value AS color, : owner AS group, : id AS ticket, summary, component, milestone, t.type AS type, time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status = 'accepted' : ORDER BY owner, CAST(p.value AS integer), t.type, time :

5 |

: SELECT p.value AS color, : owner AS group, : id AS ticket, summary, component, milestone, t.type AS type, time AS created, : description AS _description_, : changetime AS _changetime, reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status = 'accepted' : ORDER BY owner, CAST(p.value AS integer), t.type, time :

6 |

: SELECT p.value AS color, : t.milestone AS group, : (CASE status : WHEN 'closed' THEN 'color: #777; background: #ddd; border-color: #ccc;' : ELSE : (CASE owner WHEN $USER THEN 'font-weight: bold' END) : END) AS style, : id AS ticket, summary, component, status, : resolution,version, t.type AS type, priority, owner, : changetime AS modified, : time AS _time,reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : ORDER BY (milestone IS NULL), milestone DESC, (status = 'closed'), : (CASE status WHEN 'closed' THEN changetime ELSE (-1) * CAST(p.value AS integer) END) DESC :

7 |

: SELECT p.value AS color, : (CASE status WHEN 'accepted' THEN 'Accepted' ELSE 'Owned' END) AS group, : id AS ticket, summary, component, version, milestone, : t.type AS type, priority, time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE t.status <> 'closed' AND owner = $USER : ORDER BY (status = 'accepted') DESC, CAST(p.value AS integer), milestone, t.type, time :

8 |

: SELECT p.value AS color, : (CASE owner : WHEN $USER THEN 'My Tickets' : ELSE 'Active Tickets' : END) AS group, : id AS ticket, summary, component, version, milestone, t.type AS type, : owner, status, : time AS created, : changetime AS _changetime, description AS _description, : reporter AS _reporter : FROM ticket t : LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority' : WHERE status <> 'closed' : ORDER BY (COALESCE(owner, ) = $USER) DESC, CAST(p.value AS integer), milestone, t.type, time

Upload the new pot file/resource on transifex, put out the call for translations. Some recent commits on master need to be merged into 1.8 first

Use 'db dump' and 'db load' for 'pg_dump' and 'psql -f' of a database. Use pylons config to find out database options.

Use an IGroupForm plugin to add a custom metadata field to groups using convert_to_extras() and convert_from_extras(), when calling group show the value comes back quoted, e.g. '"my_value"'

Should add tests to example_igroupform and others that setting and getting the custom fields works through the action API.