{22} Trac tickets (2647 matches)

CKAN 1.5.1 will introduce changes in the SOLR schema and functions to support multiple schema versions. Unfortunately the changes in the schema will be backwards incompatible so either both CKAN and SOLR are upgraded, or SOLR is configured as multicore, with one core for each schema. The solr cores would look like:

http://<solr-server>/solr/schema-<version>

Different scenarios:

• Single SOLR instance used by only one CKAN site (e.g. SOLR running on the same machine):
  • Update CKAN
  • Update SOLR schema (symlink to suitable version in CKAN source)
  • Rebuild search index

• Single SOLR instance used by multiple CKAN sites.
  • Update CKAN source
  • Configure SOLR as multicore, one core for version 1.2 and another for 1.3 of the schema (symlink to appropriate version in CKAN source)
  • Update solr_url on each of the CKAN sites with the suitable core
  • Rebuild search index

These are the SOLR and CKAN instances that need to be updated (Please add any missing ones):

• Same machine as the CKAN site:
  • test.ckan.net
  • iati.test.ckan.net
  • data.gov.uk/ catalogue.data.gov.uk (confirm)

• s046.okserver.org
  • iatiregistry.org

• s052.okserver.org
  • datacatalogs.org

• s004.okserver.org / eu4.okfn.org / solr.okfn.org
  • thedatahub.org (=www.ckan.net)
  • datagm.org.uk
  • publicdata.eu
  • hri.fi (dev.fvh.fi?)
  • it.ckan.net
  • ie.ckan.net
  • cz.ckan.net
  • register.data.overheid.nl (nl.ckan.net?)
  • no.ckan.net / datakilder.no
  • br.ckan.net
  • colorado.ckan.net
  • at.ckan.net

TODO: which SOLR server are using these instances?

• data.norge.no
• nederland.ckan.net
• lt.ckan.net
• pl.ckan.net
• datadotmontreal.ca/
• ca.ckan.net / datadotgc.com

Suggested fields:

• id, name (username/profile name), openid, email, password, extras (JsonType?)
  • + current items: apikey, about, created)
  • current contents of name is usually openid and should migrate to openid field (could match based on starting with ​http://)
• May also want a fullname field (or perhaps that can

just go in extras - may be better to have its own field as useful to search on this )

Related changes (probably separate tickets):

• Improve user home page to include this additional information (and have it be editable)
• Use user 'name' (username) for home page links (rather than current user/<id>)
• Where we show lists of revisions, each revision author is now a link to the author's user page.

Related to ticket:136 and ticket:138.

As a

Authenticated user

I want to

Update a release

So that

The release details are amended. Attributes of a release are:

• name (may include '.' and '-'
• nickname
• date
• description

Notes

• permissions: as for updating a package (ticket:10)

Make the same instructions work for Ubuntu Lucid and Precise and make them simpler

Also see #1471

[Child of #1526]

Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa.

[Child of #1526]

Provide the interface for the themes and subthemes hierarchy.

Currently cache usage in home controller is inconsistent with elsewhere and, in particular, does not respect cache_disabled and is not configurable using path notation (e.g. ckan.controllers.home. ...).

This is especially frustrating in a development environment where is currently running into fact front page is being cached.

Fix this.

Some recent changes caused things to break in ckanext-ecportal:

• change from blueprint to bootstrap
• other style / css changes (eg: tables, resource view notes field ID, etc. Forms are also broken, but that will be addressed in #2262)
• change to going through db_to_form_schema on package_show

• create importer for geo tags
• add translations to database
• translate tags in WUI
• test
• deploy
• document how to add a geo coverage field in ​http://docs.ckan.org/en/latest/forms.html

They've become inconsistent as the design has changed.

• Remove white stripe
• Make text sizes consistent
• Make white space around text boxes consistent

As a

Authenticated user

I want to

Update the attributes of a package and add/remove releases

So that

The package attributes are changed. Package attributes are:

• A title
• A description (short or long)
• Date added
• Project url
• package license(s)
• download url
• A list of releases with their dates and a summary line
• a list of owners (referencing authenticated users)

I would also like to be able to add comments

Notes

• Only an authenticated user is one of owners of a package may update it
• Any authenticated user may add comments (but visitors may not)
• can break down into:
  • Update a package by changing simple package attributes (value: 8)
  • Update a package by adding or removing an owner (value: 5)
  • Update a package by adding or removing comments (value: 7)

Depends on #2639

Server setup scripts need to have two distinct steps:

1) Download all files that aren't in RHEL repositories, create RPMs where necessary (python virtualenv).

2) Install from local files.

See the etherpad for notes: ​http://ckan.okfnpad.org/22

There's quite a lot of custom stuff going on, with all the custom jinja extensions, and the use of Fanstatic plus custom fanstatic extensions. People who are new to CKAN wanting to write CKAN themes are likely to be confused. I think the whole lot could do with documentation, maybe in the Theming and Customizing chapter in the sphinx docs.

I suggest completely replacing the contents of doc/theming.rst, and putting a 'legacy theming' link at the bottom linking to a doc/legacy_theming.rst file with the old contents.

There are some new translations already on the CKAN 1.8 resource on Transifex already (e.g. 100% Norwegian translation I think). We should at least pull these from Transifex for 1.8.1, and may as well do a call for translations too.

• (Recline) Better graphing (multiple graph types etc)
• (Recline) Bootstrapify
• Integrate

Finalised metadata model requires several changes to our existing model:

• new extra: alternative title
• new extra: identifier
• new extra: interop level
• new extra: version description
• new extra: dataset status
• dataset type: change to fixed vocab
• resource types: prefix with documentation/distribution/etc for clarity, add new resource types
• rename update frequency to accrual periodicity
• add new fixed vocab field: language coverage
• resources: missing issued date - add as new core field called 'created'

Also add functional tests for full metadata model.

ckanext-example needs updating for CKAN 1.5:

• theme changes
• new forms

About: 'ckanext-exampletheme' was created in Spring 2011 as an example CKAN extension that showed how to customise the look & operation of CKAN. This moved to github and renamed 'ckanext-example'.

Pylons 1.0 has been out for a while and appears to be running stable on a few production sites. It also introduces a few new features, among them the ability to specifiy controllers using the entrypoint mechanism.

​http://bitbucket.org/pudo/ckanextdeliverance/src/tip/ckanext/deliverance/pylonspatch.py

This might be a bit ugly since the default behaviour of some pylons objects, such as tmpl_context, app_globals and redirect has changed.

• Move code base, fix any breakage.
• Deploy to eu8
• Test
• Deploy to eu14

Why

Later CKAN codebase has a bunch of improvements / fixes that it would be good to have.

Estimate

• 8h

• Add Ascii folding filter to text fields.
• Add capacity field for public, private access.
• Add title_string so you can sort alphabetically on title.
• Fields related to analytics, access and view counts.
• Add data_dict field for the whole package_dict.

Actual implementation of data_dict adding will be added later.

At the moment most of the harvesting functionality is only available to sysadmins. This is fine for most cases but going forward we will need to grant each publisher access to their own sources, harvest objects and harvest object errors. This will probably be done based on the groups refactor.

One more pressing concern is giving DGU publishers the ability to make calls to /harvest/object/id. We want them to be able to see documents that faild validation and also to see *all* errors in each document.

Cost: 2h

I minified bootstrap and apparently downgraded the version. This breaks things in Recline, and creates weird discrepancies (DGU uses 2.0.2 and had some problems importing HTML from TheDataHub? core, for example.)

• Switch to the latest minifed CSS + JS
• QA-sweep the site, especially dataset/edit

Upload to ckan.net: ​https://sites.google.com/site/scotlandsdata/

I'm not seeing any javascript or python error, the upload just hangs.

Because people might try and do "tags:gov sweden" when "tag:gov sweden" makes more sense (sweden is searched for in all fields in these cases).

But in REST i/f search allow "tag"="gov" OR "tags"="gov sweden" in parameters.

The JavaScript? email encryption used is not very reassuring. Google's MailHide? is a much better solution that is easily implemented.

​http://www.google.com/recaptcha/mailhide/

Check on the Mailhide API where there are even some Python libraries already built.

The current search implementation uses the output of the the as_dict method of the domain Package object to update the index

​https://bitbucket.org/okfn/ckan/src/56c79e3fc44c/ckan/lib/search/index.py#cl-48

It also uses package_to_api1 in the SynchronousSearch? plugin:

​https://bitbucket.org/okfn/ckan/src/f9dfb0506594/ckan/lib/search/__init__.py#cl-93

This prevents extensions from being able to index custom properties (e.g. faceting by custom extras not included in the model).

The search should use the logic function to get the package properties:

get_action('package_show')(context,data_dict)

If a resource has both description and name the name is used. Descriptions are generally more, well, descriptive, so let's use those.

A user reported wanting to install ckan at a suburl. Apparently this does not work well because some urls are hard-coded: ​http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000726.html

Instead we should use the site_url config option where necessary.

Cost: 1h

Details of how to go about this here <​http://www.rufuspollock.org/2009/07/27/sqlalchemy-migrate-with-pylons/>

Cost: 2h

Only happens if the user does not update the password. Most probably introduced during #1229.

User story #035 here: ​http://ckan.okfnpad.org/feature-1515-activity-streams-user-stories

Technical analysis here: ​http://ckan.okfnpad.org/27

• Generic text box for markdown about the user 'About'

Model's user table reflects these:

• 'about' attribute
• 'created' attribute is a default=datetime.now() - so when user was created

Need edit page for this.

Only user should be able to edit it.

Follows on from ticket:136 and ticket:142

Cost: 2h

Use case:

• Login using OpenID
• Click on 'My account' - results in 404

Solutions:

• User user.id instead of their name
• Escape the URL properly.

Native login scenario: Complete login form, require email address. Any session is resumed.

Mandated technical requirement: use Repoze (see plugin:form of repoze.who); borrow from BlastOff?.

See CKAN or Microfacts for usage of repose with openid.

The user management in organizations ( ​http://localhost:5000/organization/users/NAME ) should autocomplete names in the dropdown box so that only valid usernames are added to the table.

They seem to be not exactly well thought out. Needs some thought on better display of the information that they contain.

Requirements

In the data hub plugin we require the ability to differentiate users between those that have paid for a service, and those that haven't. The distinction isn't boolean as there may be levels of service for paid users, so it may be that we need a 'type' of user where there are various grades of 'paid' which are likely to be strings (specific to the data hub).

Required interface

Once changes have been made to the user schema, for a given user we want to be able to:

• determine if they have a paid or a free account, and
• get a string name of the type of paid account.

Care should be taken to ensure that the 'paid' status of the user cannot be set through the API and only by the datahub plugin.

User Stories

User stories related to the management, setting and changing of a user's payment level, as well as historical information on payments should be done as part of the work that includes actually allowing purchases. For now it is adequate that we can manually control these things through paster commands.

Payments types should be linear as I don't believe for this type of service a pick-and-mix modular model would work well. Organizations will inherit the payment level of their owner, so currently there is no requirement for it to affect organizations at all.

• As a sysadmin I would like to be able to use a paster command to manually set a user's payment level, or remove it entirely.

• As a sysadmin I would like to be able to run a paster command to view a list of users who have a payment plan, grouped by the plan that they have.

• As a sysadmin I would like to be able to use the API to change the payment status of a specific user through user_create and user_update. This shouldn't be available to anybody else.

• As a user, and only if I have one, I'd like to see my current payment level on my user profile page.

Tasks

[x] Tests

[x] Plugin based migration

[x] Code

[x] Model

[x] API

[x] Documentation

Estimates

With the organizations and organizations_dataset plugins enabled login and create a new dataset, adding it to a group that you are a member of. Edit the dataset and set the group to none. Now you can no longer edit your own dataset. (You can also specify None for the organization when first creating the dataset, same problem.)