Ability to add multiple packages to a group in one go (e.g. with 'add' link which makes drop down menu appear - so can add one after another - then submit simultaneously)

Use a bit of javascript to add more dropdowns.

Suggested by Jonathan Gray

API to provide credentials to allow authorized 3rd parties (ie. one's with CKAN api keys) to make uploads to storage.

Implement as a CKAN extension.

A user's 'home page' is at: user/<user.id> e.g. user/28394723982-03849472

Step 1:

• Current user home page with recent edits

Step 2: Readonly

• Number of edits
• Number of Packages they are admin of

How do we do a nicer URL for the home page - can we use their openid login?

Follow on tickets: ticket:138 and ticket:142

A url such as: ​http://someurl.com/xyz?x=1&VERSION=1.1&Service=WFS when set as url or download_url breaks the rendering of the package with an error like:

There was an error rendering the package: not well-formed (invalid token): line 1, column 181

Have checked that removing the & stuff makes the error go away so this looks like an issue with escaping urls when displaying them ...

A test case is currently failing for me on master:

ERROR: ckan.tests.functional.test_user.TestUserController.test_user_create_unicode
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/seanh/pyenv/lib/python2.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/seanh/pyenv/src/ckan/ckan/tests/functional/test_user.py", line 342, in test_user_create_unicode
    res = res.follow()
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 603, in follow
    return self.test_app.get(location, **kw)
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 208, in get
    return self.do_request(req, status=status)
  File "/usr/lib/pymodules/python2.6/paste/fixture.py", line 389, in do_request
    **req.environ)
  File "/usr/lib/pymodules/python2.6/paste/wsgilib.py", line 343, in raw_interactive
    app_iter = application(basic_environ, start_response)
  File "/usr/lib/pymodules/python2.6/paste/lint.py", line 170, in lint_app
    iterator = application(environ, start_response_wrapper)
  File "/usr/lib/pymodules/python2.6/paste/cascade.py", line 130, in __call__
    return self.apps[-1](environ, start_response)
  File "/usr/lib/pymodules/python2.6/paste/registry.py", line 350, in __call__
    app_iter = self.application(environ, start_response)
  File "/usr/lib/pymodules/python2.6/repoze/who/middleware.py", line 69, in __call__
    auth_ids = self.authenticate(environ, classification, ids)
  File "/usr/lib/pymodules/python2.6/repoze/who/middleware.py", line 201, in authenticate
    userid = plugin.authenticate(environ, identity)
  File "/home/seanh/pyenv/src/ckan/ckan/lib/authenticator.py", line 29, in authenticate
    if user.validate_password(identity.get('password')):
  File "/home/seanh/pyenv/src/ckan/ckan/model/user.py", line 113, in validate_password
    hashed_pass = sha1(password_8bit + self.password[:40])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 12: ordinal not in range(128)

A summary/user story doc of how organizations and groups are expected to work.

A small but useful ui improvement would be to have a search box at top right on every page.

As an example see the one here on trac or on github.com or bitbucket.org.

• It would be particularly good to include a small advanced search link that took you to the full search page. Need to keep it small because screen real-estate here is limited (see how github.com does this for inspiration).

A possible bug in CKAN when I tried deleting users using "paster --plugin=ckan user delete" command.

To reproduce the bug do the following:

1. Create a user with an ID (which in my case was a user's full name)

that contains non-unicode caracters like Norwegian "æ", "ø", or "å".

2. Make sure that you can see something like the example below:

(pyenv) rm@mycomputer:$ paster --plugin=ckan user Users: name=Rustæm

3. Then try deleting the user with following command:

(pyenv) rm@mycomputer:$ paster --plugin=ckan user delete "Rustæm"

You should now get a python encoding error. I know that this is quite rare case, but in our case it caused some trouble. Could you guys have a look at this bug?

CKAN ver. 1.3.3.

A place to list crucial fixes for v1.7 release:

Related extension:

All not complete now moved to #2347

• Related Item has comments on index page (they should not show up here -- why is this happening!)
• Related item should be to left of history (history could in fact move to RHS)
  • What about losing icons since we duplicate for related item
• Related item plus sign is very poorly aligned within box
  • Add related items should be in drop down like for resources (?) to make it easy to add items
  • Hover should show a title that gives some information about what related are (not at all clear)
• Related item only includes App, Idea, Visualization but not API | Post | Paper | News Article options as in original ticket #2204

Search results:

• #2331 - search should add not or
• Search results should be similar across group pages and normal search results (suggest adopting group page style)
  • Search results resource link should link to resource page rather than url for that resource
  • Suggest we could tidy this a bit (e.g. have resources in mini-sidebar of search result ...?)

Data viewer:

• Merge updates to Recline (that did go in before deadline but do not seem to have made v1.7 branch) ​https://github.com/okfn/ckan/commit/146a5c1130cc77cb2ad2c6ef6b3a53074908bf07
  • Ensure still using read-only (this seems to have been lost in upgrade)
• Do not show failed preview with red-bar (esp when just empty datastore ...)

a) Data Viewer checks if datastore is empty nad falls back to ther options if emtpy b) DataStore? enabled is off by default and only turned on either explicitly by user or by DataStorer? on successful store ...

• (? not really a fix) Give out a link as well as embed instructions (I often want to link to the explorer in a given state)
  • Should this link to the resource page or to a dedicated viewer page - cf # 2322 (resource sub-urls like /download, /api ...)
  • Do #2322 ...
• Decent powered by icon for the Embed for the DataHub? (make this text customizable and allow html ...)
• Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
• Data viewer embed url includes all the resource info

Dataset page:

• Put home page link somewhere more prominent (e.g. the top of the sidebar)
• Put author more prominent (at top right)
• (? - new work) List CKAN owners at top right in sidebar
• Make resources stand out more
• Rename resources tab and heading to Data & Resources

Dataset edit and create:

• Corrections to dataset creation form for v1.7 - #2317
• Better auto-complete for groups on dataset edit/new page - #2319
  • Remove groups box and replace with tags on the new page (or is this needed for organizations stuff?)
• Description / notes text area is not wide enough.
  • Should also be larger on edit page
• Problem with date instructions on add/edit resource: "Dates are in ISO Format — eg. 2012-12-25 or %c2010-05-31T14:30%d."
• Add/edit resource: new code to auto-check urls being added is great but from UX point of view has draw back that (esp on low speed connections) page just seems to hang for a long time with nothing happening. I wonder if either
  • A) this could happen after resource created on form (e.g. it creates, and then an overlay comes up saying: checking url and gathering url information or similar)
  • Or B) we still do this before creating new resource form on page but we still should show something like a spinner and some info like "checking url and gathering url info ...)

Resource view:

• Do not show no description in no description (just show dataset info)

Theme:

• Fix spacing above page title (i.e. dataset title etc)
• Bootstrap 2.0.2 (@zephod was *supposed* to have done this) - should be trivial and gives important bugfixes for data viewer
• Fix subnav width on resources pages (does not extend all the way to the right like it should)
• Fix spacing of top menu and search box (do not look right)
• Group pages:
  • view page: image / logo is really poorly locate and sized, see e.g. ​http://datahub.io/group/wikimedia
  • view page: admin heading on RHS sidebar is wrongly indented

Miscellaneous:

• Ensure that we out-of-the-box config for CKAN has (#2388):
  • Only logged in users able to create
  • Creation permissions have only creators able to update a dataset

Deployments (without deployment we cannot know these are working):

• Social extension
• Analytics

A package discussion page is like a wikipedia discussion page: an editable free text page for people to have discussion/post comments about a given package.

It provides a way for people to make suggestions about a package without needing access to main package.

A nice map in the homepage showing the availability of data across Europe

A new library, ​guiders.js, has been open sourced that seems to be great at unobtrusively introducing new users to features of websites.

The library drives Optimizely's website. The GitHub repo is here: ​https://github.com/jeff-optimizely/Guiders-JS

Some examples:

A link “About CKAN” pointing to ​http://ckan.net/about should be visible on every page. This is important because people often land on subpages and may have trouble figuring out what CKAN is.

I would put this link into the main navigation bar, on the very right next to “Revision History”. Then I would also remove the “Home” item from the main navigation bar because it is redundant. The CKAN logo is already a link to the homepage. The only other subpage in the “Home” section is “Statistics” and that's already linked from the homepage sidebar.

But anywhere else would be fine as well.

A large number of XXX.ckan.net sites give the following message:

"This Site is Down for Maintenance We apologize for the inconvenience. ~ The Open Knowledge Foundation sysadmins."

The message is unhelpful and patently false - the sites do not exist. Some of them were supposed to have been redirected to a relevant group at thedatahub.org in this ticket (now closed):

​http://trac.okfn.org/ticket/933

However, the redirection only seems to have worked in one case, ​http://si.ckan.net.

The problem still affects the following sites - the first 4 of which have supposedly been merged:

• ​http://lt.ckan.net
• ​http://hu.ckan.net
• ​http://be.ckan.net
• ​http://gr.ckan.net

• ​http://nz.ckan.net
• ​http://bg.ckan.net
• ​http://fi.ckan.net
• ​http://katalogdanych.centrumcyfrowe.pl
• ​http://pl.ckan.net
• ​http://southampton.ckan.net
• ​http://sl.ckan.net

Please sort this out by redirecting, removing the sites, giving a more helpful (and accurate) failure message, etc, as appropriate.

A function that takes a vocabulary name or ID as argument and returns a nice select box for selecting items from that given vocabulary. Meant to be used by form templates, to make it easy for them to integrate custom vocabularies. Could use ​http://harvesthq.github.com/chosen/

A bad action name: e.g. curl ​http://test.ckan.net/api/action/user_showaa -d '{}' causes 500 response.

Occurs only on 1.4.3 beta.

A 'Derived' relationship can be applied from one package to another.

e.g. sussex-demography is derived from census-2001

'Derived' relationship is:

• directional
• many:many
• stateful

'derived' table columns:

• id (primary key)
• source_package (foreign key)
• result_package (foreign key)
• description (markdown text)

Further tickets:

• WUI - package view - shows 'derives from package x' and 'derived package y' with UML-like diagram of x -> this package -> y
• WUI - package edit form - new option to say it 'derives from' or 'has derivation' and you select the appropriate
• REST if - expose reading and writing this property

@aron

We should be able to have longish tag lines that look ok currently they wrap and stop the logo text being clickable

Can we have white-space:nowrap; for the tagline so long ones don't wrap and maybe look at aligning long ones

Which events to notify on

Listed by domain object, these are the notification message 'change types' that will be sent:

• Package
• PackageResource

Also it is clear that it could be useful to know when db-wide maintenance is carried out:

• db - 'clean', 'rebuild' (db is wiped and replaced with new data), 'upgrade' (migration)

Ignored domain objects

These parts of the domain model will not carry notifications as no use case has been identified for them:

• Revision
• Group
• Tag
• Rating
• User - list of users is sensitive info
• Relationships - complicated
• Authz - complicated and sensitive info
• License - change of a license's metadata is a question for the 'license service'

Message format

A notification message's header contains the routing key, identifying the object type. The client is probably interested in the object (all use cases so far), so it makes sense to send the object in the payload. This should be the JSON-encoded dictionary exactly as provided for the object's REST Entity.

For the 'db' notifications there shall be no payload.

As a

CKAN client program

I want to

be notified when changes to the CKAN metadata occur.

Examples of use

1. An external search engine needing to (re)index a package. (interest: Package)
2. A front-end system that caches package info and wants to know when it changes, to keep in step. (interest: Package or Revision) See further details here: ticket:352 and previous iteration here: ticket:333.
3. A system for automatically checking package URLs and resource URLs as they are put on the system. This could alert to bad URLs and automatically email feedback to (meta)data owners. (interest: PackageResource)
4. Do some processing on resource (e.g. extract sample data for display) (interest: PackageResource)

Context

The current state of CKAN can be queried through the REST API, you can keep track of changes by reviewing the feeds, but there is no way to find out the instant something is changed, without costly polling.

Design

Split-off into two tickets:

• Notification message - ticket:323
  • Which events to notify on
  • Message format
• Interface for Notifier Service - ticket:322

Testing

To test notifications, Carrot / AMQP will be configured to use a native-Python Queue, instead of requiring RabbitMQ to be running on the machine.

Related

• Run CKAN search indexing using this system - ticket:324
• Run SOLR indexing using this system - ticket:353
• Web hooks for notifications - ticket:327

Where does the data go?

In a postgres database configured by the ckan.datastore_write_url config option which is a sqlalchemy url.

The user should have rights to create tables.

Whats the api like?

We will just implement it as logic functions like the rest of CKAN and will part of core. After that we may add some nicer api functions that use these but that is a secondary concern.

What are the initial logic functions?

• datastore_create
• datastore_delete
• datastore_show

What is the JSON input format for datastore_create

To begin with it can have the following keys. It is fairly consistent with Max Ogdens' gut servers. Except adds resource_id.

{
resource_id: resource_id # the data is going to be stored against.
fields: a list of dictionaries of fields/columns and their extra metadata.
records: a list of dictionaries of the data eg  [{"dob": "2005", "some_stuff": ['a', b']}, ..]
}

• The first row will be used to guess types not in the fields and the guessed types will be added to the headers permanently. Consecutive rows have to conform to the field definitions.
• rows: can be empty so that you can just set the fields
• fields are optional but needed if you want to do type hinting or add extra information for certain columns or to explicitly define ordering.

eg: [{"id": "dob", "type": "timestamp" }, {"id": "some_stuff", "type": "text"}, ...]. A header items values can not be changed after it has been defined nor can the ordering of them be changed. They can be extended though.

• Any error results in total failure!! For now pass back the actual error.
• Should be transactional

What json does datastore_delete take?

{
resource_id: resource_id # the data is going to be deleted.
filters: dictionary of matching conditions to delete
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "delete from table where key1 = 'a' and key2 = 'b' ".
    No filters (either not present or not defined) then delete the table. If we want truncate then add truncate: true to truncate the table.
}

What json does datastore_search take?

{
resource_id: resource_id # the data is going to be selected.
filters : dictionary of matching conditions to select
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "select * from table where key1 = 'a' and key2 = 'b' "
q: full text query
limit: limit the amount of rows to size default 100
offset: offset the amount of rows
fields:  list of fields return in that order, defaults (empty or not present) to all fields in fields order.
sort: comma separated field names with ordering e.g "fieldname1, fieldname2 desc"
}

Some free code: ​https://gist.github.com/3163864

What json does datastore_search return?

{
fields: same type as datastore_create accepts (i.e. with metadata)
offset: The same offset that was supplied in datastore_show
limit: The original limit
filters: The filters that were applied in data_show
total: # total matching records without size or offset
records: [same as data_create] # list of matching results
}

On error will return:

{
__error__: … sql error …
}

What types are allowed?

Aim to support as many postgres/postgis types that have string representations.

​http://www.postgresql.org/docs/9.1/static/datatype.html

​http://www.postgresql.org/docs/9.1/static/sql-createdomain.html

IDs

Each row in a table will be given an _id column which has an id generated by us which you can use in queries.

Other Features

Each row will store the _full_text index of all the data in the row. At some later point there will most likely be a way to index fields add constraints etc.

Use case

As a user I want to stay abreast of package changes, such as a new package being created, newer data is available for a package or a new download is available for a package.

Implementation

Add into the Atom feed:

1. a package is a new one
2. the resources have been changed
3. the last_updated field has been changed

Use case

As a ckan administrator I want to easily change options about the CKAN install.

Implementation

Settings to be in DB

Suggested:

## Title of site (using in several places including templates and <title> tag
ckan.site_title = CKAN

## Logo image to use (replaces site_title string on front page if defined)
ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_box.png

## Site tagline / description (used on front page)
ckan.site_description = 

## Used in creating some absolute urls (such as rss feeds, css files) and 
## dump filenames
ckan.site_url =

## Favicon (default is the CKAN software favicon)
ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico


## An 'id' for the site (using, for example, when creating entries in a common search index) 
## If not specified derived from the site_url
# ckan.site_id = ckan.net

## API url to use (e.g. in AJAX callbacks)
## Enable if the API is at a different domain
# ckan.api_url = http://www.ckan.net

## html content to be inserted just before </body> tag (e.g. google analytics code)
## NB: can use html e.g. <strong>blah</strong>
## NB: can have multiline strings just indent following lines
# ckan.template_footer_end = 

NB: these will still need to be stored somewhere for loading on initialization. do this in db init function ...

Settings / Options / KeyValues? Table

Columns:

• [namespace]: ? only if KeyValues? (for settings this would then always be settings)
• key
• label
• value (json)
• type (e.g. date and to specify in advance what type should be)
• description
• tags: ?? (for grouping ...)

Loading settings from DB

Do this in ckan/config/environment.py

WUI

• /ckan-admin/settings
• Show label, plus description plus text field

Depends

• Would be part of ckan-admin section and hence build on ticket:833 (Administrative dashboard)

Requirements

To provide a freemium service it is necessary to be able to provide differing levels of functionality based on the type of user (see #2550). These levels can be specific to the data hub but may require overriding functionality from core to provide these checks.

Initial implementation should focus on limiting access to datastore disk space.

Interface

These changes are currently only for the data hub and should be kept as much as possible within the data hub extension.

User Stories

• As a system component I want to find out if the current user has access to a feature (i.e. storage) and if so to what extent (xMb, xGb or unlimited).

• As a system administrator I don't expect to need to manage the levels of users or the features that this applies to.

Tasks

[ ] Clarification of requirements/analysis

[ ] Tests

[ ] Code

[ ] Model

[ ] API

[ ] UI

[ ] Documentation

Estimates

Reproduction

Using the government form, create a new package with name 'test' and date released of '23/5/0210'. The result is a 500 error and 'Server Error' message.

Affects all versions of CKAN.

Why it's happening

The dates module is raising an exception on the invalid date when saving the date, which is not being caught. The exception should have been raised only during the earlier 'validation' step and that would be caught.

Purging Revisions

• Delete button displayed on:
  • /revision/list
  • (/package/history)
    • /package/history is problematic because html does not allow nested forms and we already have form for doing diff/comparison.
  • /revision/{id}
• Delete button submits to delete action on revision and changes revision state to 'deleted'.
  • undelete button now displayed and revisions are marked as deleted in some way (e.g. greyed out?)
• Sysadmins then visit /ckan-admin/trash which lists all revisions with deleted state. There is a large button: "Empty trash" (irreversible). Click button purges all revisions with deleted state.

Purging Packages

• Put into deleted state.
• Listed on /ckan-admin/trash
• Separate Empty trash button which deletes all associated revisions.
  • Should be separate from Empty trash for revisions

Current system

• Single purge link on revision listing if a sysadmin which permanently purges the revision and all associated changes (without confirmation atm!)

Option 1: 'Changeset' Model

See ticket:1135 for vdm ticket. This would involve a) moving to changeset in vdm b) doing the migration in ckan to support this.

Have developed a new "changeset" based model for revisioning in vdm.

Implementation

• The main challenge with this change is schema and data migration

Every revisioned object has a revision_id and revision attribute.

Approximate algorithm:

Revision -> Changeset

for revtype in [PackageRevision, ...]:
    for pkgrev in package_revision:
        changeset = lookupchangeset(package_revision)
        ChangeObject(cset, (table, id), dictize(pkgrev))

Question:

• does pkg include tags attributes or not? or we have to dictize, pkgrev, pkg2tagrev, and tag. Probably the latter.

Option 2: Simplify Revision Object Model

Just use a simpler vdm, see ticket:1136 (move to SessionExtension) and ticket:1137 (remove need for statefulness in vdm).

Discussion

Advantage of Option 1 versus 2:

• Easier support for pending state and similar behaviour
• No need to introduce new tables (and hence migrations) when making something revisioned (or not).

Disadvantages

• Migration is required
• More difficult to query revision history.
  • Could be addressed by having ChangeObject have separate cols for table name and id but would likely be more difficult.
• Performance (?)
  • Have one big ChangeObject table to query when looking at changed objects rather than many revision tables.
    • Not sure this is a biggie as even with Revision model biggest revision object tables are probably on the order of the ChangeObject table

Conclusion

Implement Option 2 and leave Option 1 for present.

Option 1 includes Option 2 so it seems that that is required in either case (so we may as well with Option 2).

Option 1 requires significant effort (esp migration) so leave for present and then review the situation at some later date.

Historical Parent Tickets

• Parent-ticket: #1450
• Parent-ticket: #1032

Step Zero

• COMPLETELY REDESIGN resource edit process

Dataset Edit-Resources

• Force format field to be lowercase
  • Modify backend to serve it up in that format
• Subticket #978: Add extra fields to resources in Web UI
• Fix display of large datasets (​http://test.ckan.net/dataset/europeana-lod)
• When submitting forms with errors, open the correct resource pane.
• No hint is given that the time format for a resource is in ISO, eg 2005-01-01T10:10:10
• Resource description field to become the markdown editor we use everywhere else.

Dataset Settings

• Subticket #1648: Clarify edit terminology
• Merge Groups and Tags into main Settings section (no point having them on their own and most people will miss this).
• Fix (i.e. allow / show) name editing on dataset edit.
  • Fix the slugify so it does not overwrite if i change title after fixing name.
  • Make slugify encourage people to use shorter names. Either by cutting off (not so good) or showing a warning and saying something like 'hey you may want to cut this down. Hit edit to customize the dataset name now'
• Autocomplete the author/maintainer box (requires API extension?) [WONTFIX what? it's not like that field is linked to users of the site or anything]

Dataset View

• Fix animation glitch in expanding description
• Can we make the list of files look more like a table? Lose space between files? Lose rounded corners?
• Verify that there are no more (none) placeholder <td> objects (i believe they've been refactored out)

Dataset Create

• "Upload or link data files, APIs and other materials related to your dataset."
• Include the description markdown editor again (keep it small)
• Licence should maybe be restored
  • "(Don't worry if you don't know what your licence is)"
• Show the log message field
  • Beautify
• Pre-populate the log message field?
• ? Show Home Page field when creating dataset [WONTFIX don't push it]
• ? Do some Guerilla UX to see whether we need to Redirect to Resources directly after dataset creation? [WONTFIX doesnt make sense with new resource editor awesomeness]

User pages

• Improvements to user page UX - 1d - #1396
• Broken links for non-Gravatar use icons - 0.25d - #1719

Etc

• Take before and after screenshots

Background

In the Web UI, when you rate a package it simply links to something like:

​http://ckan.net/package/rate/mke-liquor-licenses?rating=3

This creates a GET request.

This is bad because:

• Search engine crawlers follow links to find pages, and in this case end up creating a rating (although we've got a robots.txt to try and avoid this)
• There are occasions when we want to make a CKAN instance read-only, so we put a <LimitExcept? GET> Apache instruction in. But the database may still get written for these ratings.
• Best practise for web requests is for GET to be a read-only request.

As a

user browsing a CKAN package which has a resource in a bad format

I want to

be alerted to the possibility of scraping it on scraperwiki

Implementation

This should be done in a CKAN extension. It should appear in a side-bar on the package read page.

If the package does not have a resource with scraperwiki.com in the url, then it displays this message:

Title: Scrape it
Image: (Scraperwiki dumper truck)
Body text: Could this data be in a better format? Why not use Scraperwiki to extract data from tables, Excel or PDF for everyone's benefit? You can ​request this data is scraped or even ​write the scraper yourself.

As a

admin / user

I want to

Admin: export all HEAD data to JSON / CSV / RDF format dump.

User: wants to get hold of this dump.

Details

• Just do JSON & CSV for now.
• Put this in CLI.
• Just do list of packages - as_dict info. CSV needs a bit of flattening.

Cost: 2h

As a

Visitor

I want to

View the files associated with a package.

Notes

• See also ticket:16 (updating a file).
• Files should either be listed on the same page as a release or there should be a link from the release page to a page listing all the files associated with the release. See also ticket:14 (update a release)

As a

Visitor

I want to

View a tag's page which should list all packages associated with that tag (or link to a page containing that list).

Notes

• The tags should have a nice primary url such as /tag/<tag-name> (could also use the plural i.e. /tag/<tag-name>)

As a

Visitor

I want to

Login to the service

So that

I am now an authenticated user and may perform the actions that only authenticated users may perform

Notes

• Login by providing username and password
• Login screen should also include a link or information about registration
• If username do not match (or the username does not exist) a login failure is noted and the visitor is re-presented with the login screen as well as information on how to register

As a

Visitor

I want to

Browse a list of packages resulting from a search or browse request (see other use cases)

So that

I can select one of the packages to view in more detail (-> viewing an individual package ticket:6)

Notes

• When browsing a list of packages you should be able to see summary information about the package such as title (though this may be shortened in order to conveniently fit the list
• The list should be broken up into pages so that the number of packages per page should be kept to a reasonable number (<= 50). Response time should be kept reasonable

As a

User

I want to

Create a new group via the WUI

Details

• locate at /group/new/
• You must be logged in to create a group
• Group creator is automatically given role of group admin
• Edit screen is same as edit screen ...

As a

Registered user

I want to

Add a comment to a package

Notes

• The comment should be tagged with the username of the person making the comment
• Should we allow comments per release as well as per package? No for the time being.
• Comments will be 'flat' and not threaded (at least for the present)
• Comments should be plain text and so html should be escaped (in the future we might look to use some kind of formatter such as markdown)

As a

CKAN client such as ScraperWiki

I want to

search for Package Resources, either by URL or other field, or just get them all. I want to be able to get all the resource's fields, such as URL.

Proposed implementation

Add resource search API at:

/api/search/resource

AND resource added to model API at:

api/rest/resource

(see ticket:358)

Functional differences from the ScraperWiki suggested patch:

• URL is not normalised

• URLs are not grouped

• All fields of the resource object are returned, not just the URL

• Package is identified by its ID, not name or full URL. (This is for consistency in the API - you can simple prepend '​http://ckan.net/package/' to the package ID)

This is to make our API more general, simple and consistent. It means that the ScraperWiki client has to do a bit more processing to get exactly what it needs. Is this ok?

Example search

POST

{"url": "scraperwiki.com/", "all_fields": 1}

to: /api/2/search/resource

returns JSON:

 [{"id": "a3dd8f64-9078-4f04-845c-e3f047125028",
   "package_id": "b8a325c8-af2a-43f3-8245-9db7d73dfbfe",
   "URL": "http://scraperwiki.com/lincolnshire-councillors", 
   "format": "CSV", 
   "Description": "Scrape of www.lincs.gov/councillors.pdf by ScraperWiki.",
   "hash": "", 
   "position": 2
 }]

Note use of package_id instead of package_name is something we're moving towards in the API, since names can change. When we've done ticket:341 then ckan.net/package/lincs-councillors will be a synonym of ckan.net/package/b8a325c8-af2a-43f3-8245-9db7d73dfbfe

Search Parameters

Key:  q
Description: Search all resource fields for the value

Key: url / description / format / 
Description: Search particular field for the value

Key: all_fields
Value: 0 or 1 (0 is default)
Description: If 1 (true), the full record of the package resource
(and it's package reference) are returned, rather than just the
PackageResource ID.

May also choose to introduce 'offset' and 'limit' to page through a large number of results.

JSONP achieved through API-wide parameter - see ticket:342

Search is case insensitive.

Original request

Hi, have attached a patch for adding a resource list api call. Have also added a JSONP compatible callback section, along the lines of #388.

Could also add a search version. Not sure what the best url would be for that though.

Haven't written a test as the structure seems to follow a functional spec. Is that document around somewhere?

Donovan

As a

Authenticated user

I want to

Update a release

So that

The release details are amended. Attributes of a release are:

• name (may include '.' and '-'
• nickname
• date
• description

Notes

• permissions: as for updating a package (ticket:10)

As a

Authenticated user

I want to

Delete a package

So that

So that the package is no longer displayed for visitors of the site

Notes

• The permissions for this action are the same as for updating the package details (see ticket:10)
• When a package is deleted it is put in an inactive state and is invisible to users and vistors of the service. However its name remains reserved and it may be undeleted or purged by the service administrator

As a

Authenticated user

I want to

Create a file associated with a release

So that

The release has a new file associated with it

Notes

• Permissions are as for package updating (see ticket:10)
• file attributes are described in file updating: ticket:16)
  • there are no attributes that are required to be supplied to create a file

As A

User or Visitor

I Want To

See simple icons (e.g. tick/cross) next to packages (in lists or on main page) indicating openness status of package.

So That

It is easy to see the openness status of a package.

Details

• Suggested by Liz Turner
• Openness = open license at present (no easy way to determine whether accessible)
• Have separate tick for downloadability (i.e. existence of download url)
• Could do more stuff in future (e.g. money where tag price- exists, etc etc)

As A

Administrator

I want to

Purge a deleted package

Notes

• this need only be possible via the admin interface

Rationale

The package edit form is restricted to three extra fields. To enter more than three fields, one has to save the package and hit edit again (or hit preview).

Implementation

A mechanism similar to the one for resources (where you can add lines as you go) would solve this. So, have a button that adds more extra field rows via JS. (Extra fields don't need up/down buttons that the Resource table has)

Nice to have: a blank field is added when you tab from the last filled-in field in the table.

As a User or Visitor

I wish to rate a package in some way. On the package view, below Openness, there is a 'Rating' heading with the star rating which is selectable.

Rating out of 5. One user or IP, one vote.

package_rating table:

package | user | rating annakarenina | joe.bloggs.openid.com | 4 annakarenina | 154.2.5.47 | 1

As A

sysadmin

I Want To

Have revisions to the CKAN data filtered in order to reduce the spam in the system.

Details

In the long run this is a quite a generic problem common across several OKF systems and probably can become a general component in the okfmisc repo. For time being focus on a well-factored CKAN-specific solution.

Suggest we follow path of trac: ​http://trac.edgewall.org/wiki/SpamFilter

Could have a general engine that aggregates spam scores from many different 'plugins' and then marks spam appropriately (actions should be configurable depending on spam level from 'purge' to 'delete' (mark revision as inactive) to 'flag' to 'do nothing').

Main initial plugins would be:

• regex filter (this would seem very useful here, e.g. do not allow urls in commit messages ...)
  • could augment using the badcontent list approach (can find list on e.g. moinmoin)
• spambayes and/or akismet

As A

Visitor

I Want To

View lists of items but not have too many on a page (as this leads to slow page loads and difficulty navigating the list). Where there are more items in the results of my action that can be fit on a single page the results should be 'paged' in some manner.

Details

Main place this issue arises:

• List of tags
• List of packages
• Repository history (/revision)

Should probably have no more than 50 (or so) items on a page. Obvious solution is for actions that return lists to have some kind of pageno parameter and then use this in controller to select what to display.

As A

User

I want To

Create/register and update a package via a web api.

4 failing tests, including 2 in misc/package_saver and 2 in functional/test_package.

3 (4) use cases: (With and without access to auth keys.)

1. upload form on ckan (by default local system has auth key for storage)
2. datapkg upload (client) (by default without key) (plumbing done)
3. background job archiving downloads (by queue client)

Most important are:

1. Upload form on ckan.net (with access to keys)
2. API for getting credentials for upload from ckan.net api (without access to keys).

Aside: Once 2 was done we could deprecate storage.ckan.net

Sub-tickets

Required infrastructure

• #826 - Resource 'extra' fields
• #358 - Resources in REST API
• #235 - Resource format normalization and detection

User upload via Web UI

• #877 - File upload in WUI
• #878 - Integrate file upload with workflow around package resources

User upload via CLI/datapkg

• #853 - plumbing for credential-less rest upload
• #879 - Storage auth API

Also related:

• #405 - Retrieval options for package resources

More Info

• Upload/storage discussion: ​http://ckan.okfnpad.org/storage
  • For api to generate upload key (for e.g. datapkg usage) suggest locating at ckan.net/api/utils/