CKAN: Ticket Query

#1562: Finish Geo Spatial

jilly mathews — Mon, 19 Dec 2011 10:56:33 GMT

Estimated 4 weeks of Adria's time. I guess this will need to be broken down into more tickets. This feature is being requested by a number of potential customers and we have some ideas of requirements between Rufus and Jilly for this. This is the most popular new feature we talk about to new clients.

#1557: Complete Webstore Preview Extension

jilly mathews — Mon, 19 Dec 2011 10:40:53 GMT

Finish any work out standing on web store preview extension to be able to package and release.

Ref James and I going through existing features and trying to mention any polishing that needed doing to get exiting features ready for release with projects such as CKAN hosted.

#1558: Publisher Tools

jilly mathews — Mon, 19 Dec 2011 10:46:13 GMT

Summarise final set of requirements for this and finish development and test. Estimated 10 working days.

#1560: Follow extension

jilly mathews — Mon, 19 Dec 2011 10:51:19 GMT

Estimate 2 days to finish dev and test.

David can you add any info needed here?

#1561: To do extension

jilly mathews — Mon, 19 Dec 2011 10:52:52 GMT

Can we finish this ready for release on data hub and CKAN Hosted.

#1564: Structured Data (Data API)

jilly mathews — Mon, 19 Dec 2011 11:07:14 GMT

Basic websotre exists but this may be not what is described yet.

CKAN provides a rich API for the data itself, allowing users to query retrieve and use data instantly from datasets in CKAN without needing to download or process it first.

#1567: Finish QA extension

jilly mathews — Mon, 19 Dec 2011 11:19:59 GMT

Requires change to celeryd. Estimated 4 weeks.

#1569: Wordpressser

jilly mathews — Mon, 19 Dec 2011 11:27:36 GMT

How much effort will this be to be ready to use?

#1572: Meta data Harvester

jilly mathews — Mon, 19 Dec 2011 11:35:09 GMT

Need to write custom harvesters for each client. Is it worth having one for data hub?

#1573: Apps and Ideas

jilly mathews — Mon, 19 Dec 2011 11:36:33 GMT

Estimate 2 weeks for someone to finish and test.

#1565: Admin dashboard finished?

jilly mathews — Mon, 19 Dec 2011 11:11:32 GMT

Is testing complete and ready for release?

#2938: Explain bundling in doc/resources.rst

dominik — Mon, 24 Sep 2012 12:13:55 GMT

Since it makes sense to bundle resources into one js file that are commonly used together, there should be some explanations on how separate js scripts are merged into one file for production.

Explanations could include how groups are handled or different resource files.

#728: CSW Harvesting shall be optimised in respect of reharvesting only records that have changed

johnbywater — Thu, 21 Oct 2010 15:35:40 GMT

Hi Will, this is important again because some CSW servers we use have over 300 documents in. Could you take a look at modifying the filter please?

#794: Investigate reconciling UKLP Publisher and Provider with DGU

johnbywater — Mon, 08 Nov 2010 14:50:11 GMT

This needs more analysis, but the GEMINI2 attribute "metadata point of contact" must be reconciled with the registered publisher (or agent).

This might also be used to filter records harvested from a CSW source, but filtering also needs more analysis, as does distinction between agent and provider.

#1815: Reenable Sparql endpoint on publicdata.eu

amercader — Mon, 20 Feb 2012 17:08:43 GMT

Estimate: ?

The 4store Sparql endpoint has been down for a long time. It needs to be reenabled and we need to check that the necessary informtion is pushed when creating /editing a dataset.

#1818: Spatial metadata editor

amercader — Mon, 20 Feb 2012 18:52:21 GMT

Estimate 2.5d

Right now users need to manually enter a geojson value in the 'spatial' extra field in order to define a geometry for a dataset.

We need a map widget on the edit form that allows users to draw the geometry on top of a map. This will be based on the draw functions offered by OpenLayers? [1], [2].

Things to take into account:

Integration with the form sections.
Allow some degree of configuration (default extent, limit geometry types...)

[1] http://openlayers.org/dev/examples/draw-feature.html [2] http://openlayers.org/dev/examples/modify-feature.html

#2382: Investigate options for basic geocoding

amercader — Mon, 14 May 2012 17:39:20 GMT

A simple way for geocoding place names would be very useful, e.g in the spatial search, defining a geometry for a dataset (on the form or bulk)

#2407: 'Access denied' message is unhelpful

markw — Mon, 21 May 2012 15:01:40 GMT

This chap tried to use the API to add a dataset to a group and got the uninformative message 'Access denied':

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002229.html

The problem was he didn't have access to the group. A message like 'You don't have access to the group "LODcloud"' would have been clearer.

#2788: Speed improvements on creating/updating and indexing

amercader — Wed, 01 Aug 2012 14:56:32 GMT

Specially needed when importing large numbers of datasets.

Profiling the import command from the harvesting extension has shown some areas where improvements could be made.

#2949: Reenable Data API button on the new theme

amercader — Mon, 01 Oct 2012 16:04:24 GMT

The checks to show or not the button need to be updated for the latest datastore version

#3023: New methods on IPackageController to provide access to the data_dict

amercader — Thu, 22 Nov 2012 17:00:57 GMT

Extension hooking into the edit and create methods of the IPackageController interface receive the package object. This may not include all the fields that came from the form. The new extension points will pass the validated data_dict so extensions can have access to it

#3025: Add requests to core requirements

amercader — Tue, 27 Nov 2012 21:35:24 GMT

Because yes please

#3026: Support icons on nav_named_link

amercader — Fri, 30 Nov 2012 18:51:56 GMT

nav_named_link won't include the nice icon even when passing the icon keyword param.

#2357: Create build script for front-end resources

aron.carroll — Tue, 01 May 2012 15:29:41 GMT

Should minify and concatenate JavaScript? and CSS as part of #2354

#2493: Demo site 404 needs some love

toby — Wed, 06 Jun 2012 12:34:41 GMT

templates/error_document_template.html

I've converted the 404 page to the new style but we probably want to serve this as a 'full page' ie content is 100% page width not sidebar and primary content

just go to a non-existant url to see

Discusion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63374042/

Implementation:

http://s031.okserver.org:2375/404

#2961: Preview plugin endpoint

dominik — Mon, 08 Oct 2012 11:39:27 GMT

Users could write ckan extensions that offer previews for a certain datatype.

Advantages:

We can say that you can add your own previews
Every preview would have to define it's own dependencies
External development (community creates new extensions?)

Questions to answer:

Specification of the interface
Precedence of previews for the same data types
Plugins only define the datatype that they are responsible for or do we call a function that returns whether the extension is responsible.

As far as I can see, this should be fairly easy to implement by following these instructions: http://docs.ckan.org/en/latest/writing-extensions.html#writing-a-plugin-interface

#3013: common-error-messages is unreadable

dominik — Sat, 10 Nov 2012 13:18:25 GMT

Since the update of the doc theme, the page became unreadable.

http://docs.ckan.org/en/latest/common-error-messages.html

#250: RDF link in Atom feed

dread — Thu, 18 Feb 2010 15:41:35 GMT

Add link to RDF representation of a package in our Atom feed.

#2298: Add sort-by controls to the search results page

icmurray — Mon, 16 Apr 2012 09:39:22 GMT

Sort-by functionality was exposed through the package controller in [1]. But no controls were added to the search-page.

What should the sortBy controls/widget look like?
Which fields should be exposed?

The above commit is in the release-1.6.1 branch, so this work is dependant upon release-1.6.1 from being merged into master.

[1] https://github.com/okfn/ckan/commit/8685c6000d1cb211928b4dbc63990fb72d884f8c

#2310: Refactor the search-query construction in feeds.py

icmurray — Mon, 16 Apr 2012 16:01:31 GMT

The feeds controller, used to construct atom feeds, duplicates code found in the package controller's search action (in order to construct the custom feed).

Refactor this to remove duplication.

#2350: Hooks in package controller for validating form data

icmurray — Tue, 01 May 2012 14:16:01 GMT

When the validation schema was moved from the package controller into the logic layer, the schema's role was changed from that of processing a form and validating data; to just validating data. Whilst is makes sense to have a schema for validating data as it comes in and leaves the logic layer; there's no longer the hook available to PackageController? subclasses to do form validation/processing.

Add a hook into _save_new() and save_edit() prior to the logic-layer call which subclasses can then override.

NB - form processing/validation *is* different to validating data. For example, many data can be inferred from a user's input; or there may be aspects of the form that need to be filled in, but do not make sense to pass as data into the logic-layer, eg. agreeing to T&Cs.

[ ] Add hooks
[ ] Use in ckanext-example

#2405: Improvements to data viewer embed

rgrp — Mon, 21 May 2012 06:59:25 GMT

Decent powered by icon for the Embed for the DataHub?? (make this text customizable and allow html ...)
- Current icon and text looks really bad
Embed link uses uuid rather than dataset name (this goes to perma-url question ...)
Data viewer embed url includes all the resource info (not needed -- just url and backend info ...)
Embed extension urls do not respect original domain but instead use site_url (I'm guessing). E.g. datahub.io domain ends up with links to embed on thedatahub.org

#2485: Encourage leak containment by limiting the number of requests a CKAN process serves

nils.toedtmann — Fri, 01 Jun 2012 11:47:09 GMT

CKAN has memory leaks. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced.

One should ...

research ways to limit requests-per-process with the different web servers (see below for a start);
explain these safeguards in the CKAN documentation and encourage users to apply them;
consider the helper script ckan-create-instance to generate Apache configs with WSGIDaemonProcess ... maximum-requests=XY being active instead of commented out. Or at least let it warn the user to use MaxRequestsPerChild;
Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them.

How to limit requests-per-process

Apache:
- Use WSGIDaemonProcess with maximum-requests=50 or whatever limit is appropriate. We did this sucessfully on datahub.io and the ckan farm
- (This need to be verified) Without WSGIDaemonProcess, MaxRequestsPerChild 50 should achieve the same.
nginx/supervisord: to be researched.

#2550: User types

ross — Mon, 18 Jun 2012 11:06:30 GMT

Requirements

In the data hub plugin we require the ability to differentiate users between those that have paid for a service, and those that haven't. The distinction isn't boolean as there may be levels of service for paid users, so it may be that we need a 'type' of user where there are various grades of 'paid' which are likely to be strings (specific to the data hub).

Required interface

Once changes have been made to the user schema, for a given user we want to be able to:

determine if they have a paid or a free account, and
get a string name of the type of paid account.

Care should be taken to ensure that the 'paid' status of the user cannot be set through the API and only by the datahub plugin.

User Stories

User stories related to the management, setting and changing of a user's payment level, as well as historical information on payments should be done as part of the work that includes actually allowing purchases. For now it is adequate that we can manually control these things through paster commands.

Payments types should be linear as I don't believe for this type of service a pick-and-mix modular model would work well. Organizations will inherit the payment level of their owner, so currently there is no requirement for it to affect organizations at all.

As a sysadmin I would like to be able to use a paster command to manually set a user's payment level, or remove it entirely.

As a sysadmin I would like to be able to run a paster command to view a list of users who have a payment plan, grouped by the plan that they have.

As a sysadmin I would like to be able to use the API to change the payment status of a specific user through user_create and user_update. This shouldn't be available to anybody else.

As a user, and only if I have one, I'd like to see my current payment level on my user profile page.

Tasks

[x] Tests

[x] Plugin based migration

[x] Code

[x] Model

[x] API

[x] Documentation

Estimates

#2603: Remove deprecated 'fields' parameter from resource_search

icmurray — Tue, 26 Jun 2012 17:10:01 GMT

The fields parameter of resource_search was deprecated when fixing #2438. It can be removed in release 1.9, and the action tidied up as a result.

#2621: Remove the deprecated 'fields' parameter from tag_search and tag_autocomplete

icmurray — Thu, 28 Jun 2012 16:22:49 GMT

This was deprecated in 1.8 as it wasn't accessible via GET requests due to being a dict. See #2439

In a future release of CKAN (probably 1.9) it can be removed.

Internal uses of it were removed in #2439, but there are tests that still use it.

#2679: Change default behaviour of TemplateController.view to 404.

icmurray — Mon, 16 Jul 2012 10:55:33 GMT

The current behaviour of TemplateController?.view() (which is the fallback controller should all others fail) is to attempt to render (as a genshi template) the requested file.

Although this may be a feature that some instances want. In general, it leads to:

500s when attempting to access a normal template (eg - http://datahub.io/importer/preview)
A way of inadvertantly serving things you may not want to serve. (Small risk, as it needs to be renderable as a genshi template).

Solution:

Change the controller to 404
Ensure there's a way for existing ckan instances to override that behaviour should they need it.

#2709: Atom feeds are undocumented

markw — Wed, 18 Jul 2012 15:11:50 GMT

There doesn't seem to be any documentation yet for Atom feeds.

#2784: model dictize sensitive data

icmurray — Wed, 01 Aug 2012 09:51:25 GMT

The model dictize layer doesn't consistently remove sensitive data from the dictized models. It should use the current context to decide whether to include sensitive data or not.

#2859: Fix the build

icmurray — Thu, 16 Aug 2012 09:26:57 GMT

The requires files have been removed. These were required by buildkit.

We've lost the information regarding which packages were already in ubuntu repos, which conflicted and which were missing. But perhaps we can just package everything up.

#2959: Changing a Group's name through the action api disassociates it from its datasets in the index

icmurray — Thu, 04 Oct 2012 15:18:55 GMT

Repro:

Create a new Group, named "test-group".
Add a dataset to it.
Verify the dataset belongs to the group by visiting the Group's read-page
Update the Group through the action api (group_update), using the uid in the "id" field, and a new name in the "name" field.
Visit the group's read-page. The list of datasets will be empty.

This was an issue when editing a Group through the web interface, which was fixed in [1]. However it only fixes the issue in the group controller.

[1] https://github.com/okfn/ckan/commit/dbe25d8b8d7fabfc40c5d794a920b91cec349335

#1062: Data preview encoding error

sebbacon — Tue, 29 Mar 2011 10:55:43 GMT

The preview of "Species Misc Turtle Download" at http://ckan.net/package/taxonconcept results in the following error:

Unable to Preview - Had an error from dataproxy: Data Transformation Error (Data transformation failed. Reason: 'utf8' codec can't decode byte 0x8b in position 1: unexpected code byte

#1259: "Add a row" for Extras on Package form

pudo — Tue, 02 Aug 2011 16:31:33 GMT

The default package form offers 4 empty extras fields. Like the resource section, it should have an "add more" button to add another row.

#1336: License fudge

dread — Tue, 13 Sep 2011 11:07:37 GMT

cset:4b59ab34137d ckan/logic/action/get.py:

-            isopen = model.Package.get_license_register()[license_id].isopen()
-            result_dict['isopen'] = isopen
+            try:
+                isopen = model.Package.get_license_register()[license_id].isopen()
+                result_dict['isopen'] = isopen
+            except KeyError:
+                # TODO: create a log message this error?
+                result_dict['isopen'] = False

This change hides problems with the license server and returns potentially incorrect values for openness.

This has been noted as 'temporary fix' but seems to be forgotten about, since it has been merged to default and gone into release 1.4.3.

I suggest the licenses are cached (I thought this was already the case when CKAN first requests them after start-up?). I suggest failure would return 503.

#1393: Don't skip search tests

dread — Thu, 13 Oct 2011 11:30:53 GMT

Now we don't use postgres search, all the tests involving search now don't need to be skipped when running on sqlite. Should help coders spot earlier if these tests break.

#1584: QA report improvements - 2.5d

johnglover — Wed, 21 Dec 2011 09:23:53 GMT

Super: #1594

qa/{username}
qa/{groupname}
paginate QA results
search / filter QA results
~~list organisation report by default, but can disable via config option~~ (done)
UX tidy up of report pages - hide border if no sidebar, etc

#1588: QA - Give SPARQL endpoints a 4 star rating

johnglover — Wed, 21 Dec 2011 15:13:25 GMT

Super: #1594

From Richard Cyganiak on the CKAN Discuss list:

Besides considering the media type of resources, it would also make sense to check for the presence of a SPARQL endpoint. SPARQL endpoints are recorded for more than 300 datasets on the Data Hub using the pseudo-type "api/sparql". A few more are recorded with the format "SPARQL". I suggest that datasets with such resources should also be considered for the fourth star.

#1589: QA - Give 5 star rating to datasets with link metadata

johnglover — Wed, 21 Dec 2011 15:16:40 GMT

Super: #1594

From Richard Cyganiak on the CKAN Discuss list:

Regarding the fifth star (is the dataset linked to others?). This cannot be automatically determined just by looking at the format. It either requires inspection of the actual data, or information about links in the metadata. As you're probably aware, we've established conventions for recording information on data links in CKAN [1], as part of the work of the lodcloud group on the Data Hub. Link information is captured for hundreds of datasets. I would claim that we have the majority of four-star datasets covered there, and hence you can determine if they should get the fifth star by checking for the presence of a links:xxx field.

#2733: Datastore logic functions

johnglover — Mon, 23 Jul 2012 15:54:46 GMT

Where does the data go?

In a postgres database configured by the ckan.datastore_write_url config option which is a sqlalchemy url.

The user should have rights to create tables.

Whats the api like?

We will just implement it as logic functions like the rest of CKAN and will part of core. After that we may add some nicer api functions that use these but that is a secondary concern.

What are the initial logic functions?

datastore_create
datastore_delete
datastore_show

What is the JSON input format for datastore_create

To begin with it can have the following keys. It is fairly consistent with Max Ogdens' gut servers. Except adds resource_id.

{
resource_id: resource_id # the data is going to be stored against.
fields: a list of dictionaries of fields/columns and their extra metadata.
records: a list of dictionaries of the data eg  [{"dob": "2005", "some_stuff": ['a', b']}, ..]
}

The first row will be used to guess types not in the fields and the guessed types will be added to the headers permanently. Consecutive rows have to conform to the field definitions.
rows: can be empty so that you can just set the fields
fields are optional but needed if you want to do type hinting or add extra information for certain columns or to explicitly define ordering.

eg: [{"id": "dob", "type": "timestamp" }, {"id": "some_stuff", "type": "text"}, ...]. A header items values can not be changed after it has been defined nor can the ordering of them be changed. They can be extended though.

Any error results in total failure!! For now pass back the actual error.
Should be transactional

What json does datastore_delete take?

{
resource_id: resource_id # the data is going to be deleted.
filters: dictionary of matching conditions to delete
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "delete from table where key1 = 'a' and key2 = 'b' ".
    No filters (either not present or not defined) then delete the table. If we want truncate then add truncate: true to truncate the table.
}

What json does datastore_search take?

{
resource_id: resource_id # the data is going to be selected.
filters : dictionary of matching conditions to select
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "select * from table where key1 = 'a' and key2 = 'b' "
q: full text query
limit: limit the amount of rows to size default 100
offset: offset the amount of rows
fields:  list of fields return in that order, defaults (empty or not present) to all fields in fields order.
sort: comma separated field names with ordering e.g "fieldname1, fieldname2 desc"
}

Some free code: https://gist.github.com/3163864

What json does datastore_search return?

{
fields: same type as datastore_create accepts (i.e. with metadata)
offset: The same offset that was supplied in datastore_show
limit: The original limit
filters: The filters that were applied in data_show
total: # total matching records without size or offset
records: [same as data_create] # list of matching results
}

On error will return:

{
__error__: … sql error …
}

What types are allowed?

Aim to support as many postgres/postgis types that have string representations.

http://www.postgresql.org/docs/9.1/static/datatype.html

http://www.postgresql.org/docs/9.1/static/sql-createdomain.html

IDs

Each row in a table will be given an _id column which has an id generated by us which you can use in queries.

Other Features

Each row will store the _full_text index of all the data in the row. At some later point there will most likely be a way to index fields add constraints etc.

#3011: Recline fixes and updates for CKAN 2.0

johnglover — Wed, 07 Nov 2012 11:40:55 GMT

Fixes and updates to Recline / Datapreview for CKAN 2.0.

#3017: New stable branch for the demo server

johnglover — Tue, 13 Nov 2012 15:04:48 GMT

#2446: Create demo dataset history/comparison page

aron.carroll — Mon, 28 May 2012 13:12:18 GMT

This is a low priority for the demo site.

Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62414120/comments

Implementation:

http://s031.okserver.org:2375/dataset/history/adur_district_spending

#2451: Create demo user list page

aron.carroll — Mon, 28 May 2012 13:20:13 GMT

Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments

Implementation:

http://s031.okserver.org:2375/user

#2457: Create demo tags list page

aron.carroll — Mon, 28 May 2012 13:28:55 GMT

This includes the tag page as well for now.

Discussion:

https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998445/comments

Implementation:

http://s031.okserver.org:2375/en/tag

#2641: Adapt spatial widgets to new theme

amercader — Mon, 09 Jul 2012 15:02:27 GMT

Dataset extent map and spatial filter need to adapted to the new theme, as they are not showing up now

#2697: create dataset validation

shevski — Wed, 18 Jul 2012 14:00:35 GMT

Includes: missing fields, existing field checks (i.e. whether a name/dataset already exists with that name) during input (i.e. no need to submit form to check)

#2785: Allow resources to be re-ordered

aron.carroll — Wed, 01 Aug 2012 10:22:03 GMT

Not sure where this functionality should be added, possibly in one of the sidebar widgets when editing a resource?

Ira, what are your thoughts?

#2917: Organization admins can delete themselves

ross — Wed, 12 Sep 2012 12:10:43 GMT

Organization administrators can delete themselves from the user management pages. We should disable this on the front-end as well as the back-end.

#3016: CKAN 2.0 template tweaks

johnmartin — Tue, 13 Nov 2012 13:30:17 GMT

Just a ticket to keep track of a few suggested template changes.

#3018: Load more in activity streams

johnmartin — Wed, 14 Nov 2012 13:34:11 GMT

Activity streams should be able to load more than 15 items within them. Suggest the default amount of loading is around 30 and then click to load more.

#3021: Logout doesn't work without JS

johnmartin — Tue, 20 Nov 2012 11:19:10 GMT

Essentially, the functionality should be as follows:

Add logout link that has .js-hide attached to it within the header that isn't hidden within a dropdown

See http://plus.google.com/ (when logged in) with and without JS to see an example of the actual sign-out working without JS

#84: Revert support on versioned objects

rgrp — Thu, 23 Jul 2009 08:59:03 GMT

Basic revert in the classic wiki form is already support by purging a Revision. However may wish to support:

Cases where multiple objects changed in a revision but only want to revert 1 (low priority)
Want to revert but have reversion as a new revision of that object.

Seems low priority at present.

Cost: ?

#979: Edit Resource extras in the API

dread — Fri, 11 Feb 2011 13:09:37 GMT

Follows on from #826. We can now edit resource extras in the WUI (to some extent - see #978 for remaining issues) and we can view resource extras in the API, but we can't yet edit them in the API.

#989: Extending the model from plugins

pudo — Mon, 14 Feb 2011 13:22:04 GMT

We need to support extending the model from plugins. This could involve:

Adding a plugin hook to extend the mapper
Adding an upgrade hook for plugin schema migrations
Documenting how this is to be done
Find a way to avoid conflicts

#1077: Move to simpler vdm system

rgrp — Fri, 08 Apr 2011 23:14:24 GMT

Option 1: 'Changeset' Model

See ticket:1135 for vdm ticket. This would involve a) moving to changeset in vdm b) doing the migration in ckan to support this.

Have developed a new "changeset" based model for revisioning in vdm.

Implementation

The main challenge with this change is schema and data migration

Every revisioned object has a revision_id and revision attribute.

Approximate algorithm:

Revision -> Changeset
for revtype in [PackageRevision, ...]:
    for pkgrev in package_revision:
        changeset = lookupchangeset(package_revision)
        ChangeObject(cset, (table, id), dictize(pkgrev))

Question:

does pkg include tags attributes or not? or we have to dictize, pkgrev, pkg2tagrev, and tag. Probably the latter.

Option 2: Simplify Revision Object Model

Just use a simpler vdm, see ticket:1136 (move to SessionExtension) and ticket:1137 (remove need for statefulness in vdm).

Discussion

Advantage of Option 1 versus 2:

Easier support for pending state and similar behaviour
No need to introduce new tables (and hence migrations) when making something revisioned (or not).

Disadvantages

Migration is required
More difficult to query revision history.
- Could be addressed by having ChangeObject have separate cols for table name and id but would likely be more difficult.
Performance (?)
- Have one big ChangeObject table to query when looking at changed objects rather than many revision tables.
  - Not sure this is a biggie as even with Revision model biggest revision object tables are probably on the order of the ChangeObject table

Conclusion

Implement Option 2 and leave Option 1 for present.

Option 1 includes Option 2 so it seems that that is required in either case (so we may as well with Option 2).

Option 1 requires significant effort (esp migration) so leave for present and then review the situation at some later date.

#1240: [super] API v4

rgrp — Sun, 24 Jul 2011 16:41:00 GMT

(Just creating this ticket as somewhere to keep notes)

Decide on REST api versus action API
- Do we want to support both?
Tidying
- Unify on /api/v{version num}/... structure (do we want a default option that points to current default? e.g. /api/default/ ...)
- extras merged into normal field list in package
- Get rid of /rest/ so just have api/v1/package
- Get rid of separation of search api from 'rest' api
  - Propose that GET on REST index is search e.g. /package/?q=...
    - This is also resolves issue whereby GET at root returns whole package set (a *bad* idea) as this would now become the matchall search query (with a default limit on items returned)
Resource read/write in API (separate from package)
- Does this need authorization work?
user/account API - read/write
Remove autocomplete -- can just use search
- Do not worry about backwards compat as should only be used in our js (if others using it too bad!)

#1255: Drupal consistancy checks.

kindly — Mon, 01 Aug 2011 17:32:48 GMT

Make a robust way to make sure the drupal database is consistent with the ckan data.

#1341: Delete spam users from ckan

kindly — Wed, 14 Sep 2011 10:10:34 GMT

Spam users where added to thedatahub and we need to clean them.

#1447: disk space leakage

dread — Mon, 07 Nov 2011 11:54:03 GMT

Periodically we see some CKAN servers fall over because they run out of disk space. We need to find out if there is a common cause and fix it.

One problem in the past has been file handles running out when creating lots of tiny files in the data directory.

Another problem has been several enourmous backups being created every day - pdeu on eu25.

#1652: How we intergrate with Drupal Multiligual?

ross — Mon, 16 Jan 2012 10:31:34 GMT

Analysis of how/where we can integrate with Drupal Multilingual

Is it likely to access CKAN via /en/data/... or /data/en/... (not sure, probably former) and we need to make sure that it ends up at /en/...

Check with Ian for other issues.

#1689: List deleted datasets in API

dread — Fri, 20 Jan 2012 12:23:34 GMT

The admin extension allows deleted datasets to be viewed, but there is no equivalent in the API. The package_list API call filters to just 'state=active' datasets.

'state' could be a parameter on the package_list call

Easy to call and implement
But getting permissions right for the different state values may not be easy - maybe a separate logic function is better?

#2203: Data Viewer support for PDF

rgrp — Sun, 04 Mar 2012 11:24:53 GMT

Boot viewer for format/mimetype: application/pdf, application/x-pdf, application/acrobat, applications/vnd.pdf, text/pdf, text/x-pdf
propose we use pdf.js - https://github.com/mozilla/pdf.js
We could also try iframing for browsers with native support but this seems more complex

#2291: DataStorer incorrectly attempts to store JSON in DataStore

rgrp — Sat, 14 Apr 2012 19:19:07 GMT

See http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670

This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do).

Assigning to David for triage and review.

#2294: Unauthorized action on Data API gets 302 rather than 401 or similar

rgrp — Sat, 14 Apr 2012 19:35:04 GMT

Well known issue that 401 *in API* end up with 302 redirect to login page (which makes no sense and is very confusing for clients).

Fix this by ensuring that error messsages on API get passed through correctly.

Minor because affect is annoying but not terminal

#2331: Search should AND terms not OR terms

rgrp — Sat, 28 Apr 2012 18:24:45 GMT

Appears current default search in CKAN ORs terms rather than ANDing them (i.e. adding more terms increasing number of items found rather than reducing it).

Not sure when this crept in or if it has been there for a long time.

#2334: Improved data import

rgrp — Sat, 28 Apr 2012 22:21:02 GMT

Placeholder for thinking about this ...

Allow users to configure info for CSV import (e.g. separators, quotes etc)

#2406: Errors in source install doc

markw — Mon, 21 May 2012 10:11:33 GMT

There are a few errors or unclarities in the source install doc - see

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html

#2415: Confusing message on attempting to register new account with upper case

markw — Wed, 23 May 2012 13:19:49 GMT

On registering a new account, if you try to use a capital letter in your 'Login', you get the unhelpful error message:

"The form contains invalid entries: Name: Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_"

What does 'Name' or 'Url' have to do with anything? It means 'Login must be ...'

But even better would be to make the login case-insensitive and allow capitals to be entered.

#2426: Get rid of CKAN's formalchemy dependency

seanh — Fri, 25 May 2012 11:47:42 GMT

#2427: Reduce cruft installed into CKAN's virtualenv due to Pylons

seanh — Fri, 25 May 2012 11:48:27 GMT

#2473: Make datstorer store field ordering in _meta field

kindly — Tue, 29 May 2012 12:11:32 GMT

Recline views should have a default table order, being the same as the csv that was imported.

#2524: If there are no translation files for selected language fall back to default lang.

kindly — Wed, 13 Jun 2012 17:37:28 GMT

If a user selects a language there are no mo files for then an error is raised. Revert to default language instead.

#2530: DataHub purge fails on some revisions

rgrp — Fri, 15 Jun 2012 08:54:58 GMT

See http://datahub.io/ckan-admin/trash and try to purge revisions (*not* datasets). It will fail on some of the revisions with errors like:

Problem purging revision 391db9e8-df57-4e0e-8fe6-d4e0c2318344: (IntegrityError?) update or delete on table "revision" violates foreign key constraint "group_extra_revision_revision_id_fkey" on table "group_extra_revision" DETAIL: Key (id)=(391db9e8-df57-4e0e-8fe6-d4e0c2318344) is still referenced from table "group_extra_revision". 'DELETE FROM revision WHERE revision.id = %(id)s' {'id': u'391db9e8-df57-4e0e-8fe6-d4e0c2318344'}

#2548: Object ownership for groups/package

ross — Mon, 18 Jun 2012 11:02:11 GMT

Requirements

We need to be able to easily determine who the owner of a dataset or group is. Datasets and Groups should have an Owner, who may change over time but is a specific user within the CKAN instance. It should be easy for CKAN components to determine the user and for the initial version we should ignore the can of worms labelled 'ownership transfer'.

At this point migration is likely to be the biggest issue, and would suggest that it is acceptable that the last user to edit a dataset be set as the current owner.

More tickets should arise as a result of this work where we may be able to optimise some queries to use the new feature.

Interface

None

User Stories

None

Tasks

[ ] Analysis/Clarification?

[ ] Tests

[ ] Migration

[ ] Code/Schema? changes

[ ] Documentation

Estimates

#2674: Data preview not loading on s031

shevski — Mon, 16 Jul 2012 10:38:22 GMT

Not loading for all resources as far as I can tell; e.g. http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360

Fine on demo.ckan.org

#2708: limit extra data for package/group show

toby — Wed, 18 Jul 2012 14:40:20 GMT

contextpackage_limits? = { 'tags': 5, <- get first 5

'extras': 0, <- get all

}

only get what you ask for have to be explicit

contextgroup_limits? = {} only main item

start with datasets/groups expand if we like it

#2729: searching for tags:[tag] works but tag:[tag] doesn't

shevski — Sun, 22 Jul 2012 09:58:29 GMT

which is confusing since you can only search for one tag like this at a time. I.e. tags:economics,cvs or tags:economics, csv or tags:economics+CSV doesn't work for example; therefore tag:economics, should also work!

http://s031.okserver.org:2375/dataset?q=tags%3Aeconomics&sort=relevance+asc

#2790: logic.action.user_show is slow

toby — Wed, 01 Aug 2012 20:21:23 GMT

This is a very slow call it would benefit from the sort of speed-ups that package-search received

for me locally this is taking 6 seconds for rufus using the datahub data I have. I think a lot of this is the dataset retrival/dictization

can we just grab json blobs from solr?

also is it possible to specify a sort order/paging?

I've put this as a demo-theme ticket as it is an big issue on the demo we are at 25 second page loads - which i can get down to about 8.8 secs so this is the main pain point now

#2877: Bugs with datastore v2

rgrp — Sat, 18 Aug 2012 00:33:22 GMT

In progress

~~[major] q does not seem to work reliably. e.g. using the setup from this gist https://gist.github.com/1930806 and doing a ?q=DE yields no results (does not work with "q=de" either)~~
- ~~q=second does work ...~~
[major] q does not work with 2 values (see below)
Query on search with limit 0 results in total of 0 (should either be null or correct total). Queries with other limits yield correct total AFAICT
- Also weird fact that limit is returned but as as as string - should it not be an integer
Types on fields: could these not be canonical and as per recline (or is it important to allow exact sql types ...)

Multiple query values

Try a query such as: "second UK" and you will get 500 error:

http://localhost:5000/api/3/action/datastore_search?resource_id=4f1299ab-a100-4e5f-ba81-e6d234a2f3bd&q=second%20UK

ProgrammingError: (ProgrammingError) syntax error in tsquery: "second UK" 'select "_id", "id", "date", "x", "y", "z", "country", "title", "lat", "lon", count(*) over() as "_full_count"\n from "4f1299ab-a100-4e5f-ba81-e6d234a2f3bd" where _full_text @@ to_tsquery(%s) limit 100 offset 0' (u'second UK',)

Suggestions

Filter support: should think in more detail about this (may want to follow recline style)

Simple filters in query parameters would be nice too ...

#3027: solr for 2.0

kindly — Tue, 04 Dec 2012 02:22:09 GMT

change mm support solr 3 and 4 add *_date field

#1171: Citation instructions on dataset and resource view pages

dread — Wed, 01 Jun 2011 09:33:19 GMT

Some sort of citation helper. Something small on the dataset and resource page that would show how to cite.

wwaites: Some related thoughts on this from opb: http://homepages.inf.ed.ac.uk/opb/papers/ssdbm2006.pdf

timclicks: I'm looking at Dataverse for the first time[0]. It seems very popular in the social sciences. I noticed that there is a recommended citation for each dataset. For example, [1] is has this one: "Targeted Input Programme (TIP) 2000-01", http://hdl.handle.net/1902.1/SSC-MWI-TIP2000-01-M1 V1 [Version]"

Implementation

Add a small box at bottom of dataset / resource page (or in sidebar on dataset page) with title "Cite this" with contents like:

%title. %author. Retrieved %date. %site_title.

For resource: %title = %dataset_title. %resource_name.

Could also add export to ref managers (e.g. to bibtex) but that is for later.

#2796: Need a datahub one-pager

ross — Thu, 02 Aug 2012 16:28:36 GMT

A one-pager explaining what the datahub is and with howto/examples for new users. This would make it much easier to explain the value in using the datahub for storing data.

#1581: Blog post about Google Analytics extension for CKAN

johnglover — Tue, 20 Dec 2011 17:40:00 GMT

The CKAN Google Analytics extension has been updated to work with the latest version of CKAN, could make for a nice blog post.

Can ping John Glover in January for any details required.

Key link is: http://thedatahub.org/analytics/dataset/top though this should probably move to be under stats (e.g. http://thedatahub.org/stats/usage)

#2773: About page needs improving

markw — Mon, 30 Jul 2012 11:13:15 GMT

The about page for the DataHub? (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh.

#1009: Improvements to user accounts sytem

rgrp — Fri, 25 Feb 2011 12:13:11 GMT

Forgot password (email a new password)
Confirm email
Do not show register page if you are logged in (redirect to home page)
ticket:1010 - listing of users.
- Do not use /user for general user account home page (either user normal user page /user/{id} or /user/myaccount)

#1311: Modal user register and login form

rgrp — Tue, 06 Sep 2011 08:20:27 GMT

Subticket of: #1294

Rather than having to visit a dedicated page it would be good if registration and login could be done from a modal form (separate or combined ...).

Why

It could be used from dataset creation page in situations where user needs to be registered / logged in to create a dataset (so we could allow someone to start creating a dataset and only get them to login at the end ...)
It allows for quicker and easier logging in

Implementation

See Friedrich's work on the datahub

#1343: [super] User related improvements (login, user pages etc)

rgrp — Wed, 14 Sep 2011 16:18:18 GMT

Disallow account creation via openid - #1386
Require email field - #1319
- Require email confirmation to be activated (?)
Improvements to user page (e.g. show activity and more info about user) - #1396
Modal user login - #1311

#1384: CKAN wiki needs updating to refer to thedatahub.org instead of ckan.net and datasets instead of packages

shevski — Wed, 12 Oct 2011 10:07:57 GMT

Most articles still refer and link to ckan.net, wiki.ckan.net and to packages

#1429: Provide DOIs for datasets in a CKAN instance

rgrp — Sun, 30 Oct 2011 12:21:45 GMT

DOI = digital object identifier = http://www.doi.org/

As a Publisher I want a DOI for my dataset so that it can be cited by and linked to by others in a standard and easy way.

Details

Probably implement as extension rather than core

#1458: Support previewing kml files in data viewer

rgrp — Thu, 10 Nov 2011 14:48:08 GMT

Super ticket: #1151 (viewing geo data)

E.g. preview of http://thedatahub.org/dataset/louisvillecrime should bring up a map