{22} Trac tickets (2647 matches)

Results (2401 - 2500 of 2647)

Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#2247 enhancement zephod ckan-backlog new Resource preview glitch in some browsers

From Ira: Preview for google spreadsheets are not displaying correctly for me in Firefox v.10.0.02, fine in Chrome. http://i.imgur.com/KJaqz.png

1332246614000000 1332246614000000
#2257 enhancement toby toby ckan-v1.9 new cleanup template vars

look at reducing what is pulled into templates eg ckan.lib.helpers ensure that these changes don't break existing extensions etc

1332513307000000 1340097071000000
#2258 enhancement rgrp ckan-future new Customizable contributor agreement
  • Customize text at bottom of forms
  • Also need to make clear that this does not apply to the data itself (that is covered by the license you choose on your dataset ...)
1332751549000000 1338204747000000
#2265 enhancement dread ckan-future new 'More Like This' for a dataset

When viewing a dataset, it would be nice to show a couple of 'Related Datasets'. i.e. ones that are similar.

SOLR has a feature for finding documents similar to a particular document, called 'More Like This'.

We would like this for DGU.

1332865220000000 1339771350000000
#2268 enhancement thejimmyg amercader ckan-backlog new UKLP Use same schema as form for harvested records

Datasets created via the harvesting (UKLP ones) and the ones created via the form use a different schema. That could lead to problems. We need to discuss the best approach.

1333108606000000 1333108606000000
#2277 refactor icmurray icmurray ckan-backlog new Use the new atom feeds in IATI

Atom feeds have been implemented in core in #1593. The IATI-specific implementation can now be removed when IATI move to CKAN >= 1.6.1

1333386628000000 1333386628000000
#2279 enhancement rgrp rgrp ckan-v1.8 new Write tutorial for our DataStore API / Data Explorer

Unfortunately ES is a little bit complex / opaque to newcomers. Need to write a simple tutorial walking through this.

This is documentation / example only (no code!)

1333482263000000 1338205426000000
#2282 enhancement toby toby ckan-future new log api calls for users

This was part of #2251 but is now it's own ticket.

we need clarity on this. The use cases so far have been possible limiting api calls for users.

so we want to monitor the user or ip address

what about the api calls?

do we want to record all calls or just some? do we want to record the type of call and to what level of detail?

recording the url just seems to specific to me.

1334074725000000 1338210829000000
#2287 enhancement seanh ckan-future new Consistent datetime and time period strings localization

Find all places where date, times and time periods are rendered as strings in CKAN. Make them all go through the same code path (e.g. the helper function in helpers.py) with localization support.

For example, on the user index page right now there are time period strings like "less than 1 month" that currently do not get translated (and the way they're implemented doesn't look i18n-friendly).

ISO date format everywhere might be a good idea.

Need to consider local timezone issues.

Python standard library and Babel may have useful helpers for this.

Should the local timezone displayed be a global setting per CKAN instance? Or should it change depending on the location of the user who is viewing the page?

1334254123000000 1339411445000000
#2291 enhancement kindly rgrp ckan-v1.9 new DataStorer incorrectly attempts to store JSON in DataStore

See http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670

This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do).

Assigning to David for triage and review.

1334431147000000 1340636737000000
#2298 enhancement icmurray icmurray ckan-v1.9 new Add sort-by controls to the search results page

Sort-by functionality was exposed through the package controller in [1]. But no controls were added to the search-page.

  • What should the sortBy controls/widget look like?
  • Which fields should be exposed?

The above commit is in the release-1.6.1 branch, so this work is dependant upon release-1.6.1 from being merged into master.

[1] https://github.com/okfn/ckan/commit/8685c6000d1cb211928b4dbc63990fb72d884f8c

1334569162000000 1340635947000000
#2301 enhancement seanh seanh ckan-future new Replace old-style string formatting with format() method (at least in strings marked for translation)

Some strings in CKAN are passed to gettext with more than one %s in them, this is no good as translators may need to change the order of substituted words.

Strings should not use the old style % substitution, they should use the new format method in Python 2.6+, e.g.:

'This {food} is {adjective}.'.format(food='spam', adjective='absolutely horrible')

http://docs.python.org/release/2.6.8/tutorial/inputoutput.html#fancier-output-formatting

We should probably only change strings that are marked for translation and that contain multiple %s, don't want to make too much work or cause too many strings to change and need to be re-translated.

1334579264000000 1340635812000000
#2310 enhancement icmurray icmurray ckan-backlog new Refactor the search-query construction in feeds.py

The feeds controller, used to construct atom feeds, duplicates code found in the package controller's search action (in order to construct the custom feed).

Refactor this to remove duplication.

1334592091000000 1337159386000000
#2320 enhancement rgrp ckan-v1.9 new Introduce new state: archived

Some datasets become obsolete: e.g. http://datahub.io/dataset/uncdb Do not want to delete them but want them marked as archived or deprecated or something.

1335211949000000 1340624119000000
#2321 enhancement rgrp ckan-v1.9 new Use resource names/titles in URLs

Would make them much nicer but requires:

  • Ensure uniqueness (or deal with ambiguity)
  • (?) Slug/names versus titles
    • Aside: what is currently called name should probably be title
1335236412000000 1340624174000000
#2324 enhancement amercader ckan-v1.9 new Link API docs to relevant version not just latest

The link to the API docs in the footer always points to the latests docs, which may not be correct for the particular CKAN version.

Also, remove the 'API' link (to /api/1) as pointless and misleading

1335351532000000 1345199411000000
#2326 enhancement toby toby ckan-future new dataset permalinks

create permalinks

eg \dataset\~a5-DQ\nice-name

~XXXXX defines the permalink the nice-name is ignored

  • packages table needs a new field for this data
  • existing datasets need updating
  • new datasets need the identifier creating
  • routing needs updating
  • needs to work for all urls like edit

Once this is implemented we can decide when we will use permalinks vs the normal urls.

1335371946000000 1338212799000000
#2328 enhancement rgrp ckan-v1.9 new Link to Catalog API for each Dataset, Group and Tag

At the moment no way for a normal user to find JSON versions of this material. Would it be worth adding back a link somewhere on the page?

1335442501000000 1340624331000000
#2333 enhancement rgrp ckan-v1.9 new Introduce new resource type datastore

This is a resource that is *just* in the datastore. At the moment there are hacky ways to do this (create resource with meaningless URL etc) but we should do this properly:

  • New Option: DataStore? Table in add a resource form
  • URL field is disabled (or points at /api/data or something rather pointless)
  • Download button is disabled
    • Better would be to find a way to serialize an ES table or stream it ...
1335651360000000 1340624473000000
#2350 enhancement icmurray icmurray ckan-v1.9 new Hooks in package controller for validating form data

When the validation schema was moved from the package controller into the logic layer, the schema's role was changed from that of processing a form and validating data; to just validating data. Whilst is makes sense to have a schema for validating data as it comes in and leaves the logic layer; there's no longer the hook available to PackageController? subclasses to do form validation/processing.

Add a hook into _save_new() and save_edit() prior to the logic-layer call which subclasses can then override.

NB - form processing/validation *is* different to validating data. For example, many data can be inferred from a user's input; or there may be aspects of the form that need to be filled in, but do not make sense to pass as data into the logic-layer, eg. agreeing to T&Cs.

  • [ ] Add hooks
  • [ ] Use in ckanext-example
1335881761000000 1340635966000000
#2363 enhancement toby kindly ckan-v1.8 new Documentation of best caching practice.

Need better documentation on best practices in making page loads faster for non logged in users.

1335889017000000 1340099794000000
#2382 task amercader amercader ckan-future new Investigate options for basic geocoding

A simple way for geocoding place names would be very useful, e.g in the spatial search, defining a geometry for a dataset (on the form or bulk)

1337017160000000 1338205325000000
#2395 defect dread ckan-backlog new paster db clean/init don't work when spatial extension enabled

If you have a spatial enabled database then if you don't disable the spatial extension in the CKAN config temporarily then you get errors when you run paster db clean and paster db init.

Can't you just modify the clean and init functions to run without extensions enabled?

The wider problem is that extensions do their own inits every time you do load_environment, which seems crazy and inefficient to me, since this occurs every time a request comes into CKAN. But that is another problem/ticket.

1337159793000000 1339771313000000
#2406 defect kindly markw ckan-v1.9 new Errors in source install doc

There are a few errors or unclarities in the source install doc - see

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002203.html http://lists.okfn.org/pipermail/ckan-dev/2012-May/002204.html

1337595093000000 1340636804000000
#2407 enhancement amercader markw ckan-v1.9 new 'Access denied' message is unhelpful

This chap tried to use the API to add a dataset to a group and got the uninformative message 'Access denied':

http://lists.okfn.org/pipermail/ckan-dev/2012-May/002229.html

The problem was he didn't have access to the group. A message like 'You don't have access to the group "LODcloud"' would have been clearer.

1337612500000000 1340633440000000
#2411 enhancement toby markw ckan-future new .ini file is a mess

The .ini file needs cleaning up in assorted ways which Toby will expand upon.

1337686900000000 1338210872000000
#2415 defect kindly markw ckan-v1.9 new Confusing message on attempting to register new account with upper case

On registering a new account, if you try to use a capital letter in your 'Login', you get the unhelpful error message:

"The form contains invalid entries: Name: Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_"

What does 'Name' or 'Url' have to do with anything? It means 'Login must be ...'

But even better would be to make the login case-insensitive and allow capitals to be entered.

1337779189000000 1341268304000000
#2426 enhancement kindly seanh ckan-v1.9 new Get rid of CKAN's formalchemy dependency 1337946462000000 1340616095000000
#2427 enhancement kindly seanh ckan-v1.9 new Reduce cruft installed into CKAN's virtualenv due to Pylons 1337946507000000 1341268323000000
#2433 defect toby toby ckan-v1.8 new API uses name not id for some version 3 calls

see https://github.com/okfn/ckan/pull/5

1337957648000000 1340099820000000
#2479 enhancement markw ckan-v1.9 new Meanings of Author and Maintainer fields are unclear

CKAN's default schema has fields for Author ('The name of the main contact, for enquiries about this particular dataset') and 'Maintainer ('another important contact person'). The distinction is not clear. Also the fields are often left blank.

The roles that seems most important are those of the original owner/publisher of the data, and the person maintaining the CKAN record/copy of it. So I suggest:

(1) Rename the fields 'Owner' and 'Maintainer'.

(2) Change the explanatory text for the relevant fields:

Owner: 'The person or organisation who create/collect/publish the data in this dataset.' Owner e-mail: 'E-mail address for enquiries to the Owner named above.' Maintainer: 'The person maintaining this dataset on [name of CKAN instance], if different from the above.' Maintainer e-mail: 'E-mail address for enquiries to the Maintainer named above.'

(3) When a logged-in user creates a new dataset, the main form should have a checkbox, checked by default, marked 'I am the maintainer of this dataset'. If checked, the Maintainer name and e-mail fields are populated from the user's profile.

1338452898000000 1339771115000000
#2480 enhancement markw ckan-v1.9 new Better message when dataset has no resources

If a dataset has no resources the resources list currently says '(none)'.

Here is a suggested improvement, provided that a maintainer is named: 'There are no data resources here yet. For information about this data, contact the dataset maintainer.'

1338453093000000 1339771086000000
#2483 enhancement markw ckan-v1.9 new Non-local resources should not have Download links

At present, a resource which is just a URL link to an external resource has a 'Download' button on the resource page. This gives the misleading impression that the resource is stored locally.

This is related to another small UI issue: I think the URL of a resource should be much more prominent, not buried in the 'Additional Information' table.

Suggested fix:

  • Put the URL prominently at the top of the resource page (above the preview) (and make it link to the resource)
  • Disable the Download button unless the resource is stored locally.
1338468734000000 1339771043000000
#2485 defect icmurray nils.toedtmann ckan-v1.9 new Encourage leak containment by limiting the number of requests a CKAN process serves

CKAN has memory leaks. They can be contained by limiting the time-to-live of a ckan process. An easy way to achieve this is to limit the number of requests a ckan server process can serve before it gets killed and replaced.

One should ...

  • research ways to limit requests-per-process with the different web servers (see below for a start);
  • explain these safeguards in the CKAN documentation and encourage users to apply them;
  • consider the helper script ckan-create-instance to generate Apache configs with WSGIDaemonProcess ... maximum-requests=XY being active instead of commented out. Or at least let it warn the user to use MaxRequestsPerChild;
  • Investigate current CKAN deployments whether they suffer from mem leaks, and if so contain them.

How to limit requests-per-process

  • Apache:
    • Use WSGIDaemonProcess with maximum-requests=50 or whatever limit is appropriate. We did this sucessfully on datahub.io and the ckan farm
    • (This need to be verified) Without WSGIDaemonProcess, MaxRequestsPerChild 50 should achieve the same.
  • nginx/supervisord: to be researched.
1338551229000000 1340636028000000
#2486 defect seanh ckan-backlog new Should be able to use . in dataset names 1338655583000000 1339770978000000
#2499 enhancement toby icmurray ckan-v1.8 new Documentation for the internal analytics

There's no docs for setting up the internal analytics, nor what it provides.

Setup

Add: ckan.tracking_enabled = true to .ini file

Run: paster tracking -c dev.ini

http://trac.ckan.org/ticket/2251

1339070297000000 1339070297000000
#2508 enhancement seanh ckan-backlog new Make it possible to run CKAN tests for each language

Mistakes in translated strings can cause CKAN to crash or otherwise not work, but it's not practical to manually test every page and function of CKAN in every language that we have new translations for before a CKAN release. It'd be great if the tests could automatically be run for each language.

This is probably a big job, we would have to get the tests to respect a language setting in the ini file, check for any individual test cases that specify the language (e.g. in the URL), and also fix test cases that look for specific English words in HTML output, etc.

In the meantime, a good stop-gap solution might be a script that tests for common mistakes in the po files.

1339411335000000 1339770771000000
#2524 enhancement kindly kindly ckan-ecportal new If there are no translation files for selected language fall back to default lang.

If a user selects a language there are no mo files for then an error is raised. Revert to default language instead.

1339609048000000 1340117608000000
#2529 enhancement rgrp ckan-v1.9 new DataHub (or CKAN) widgets

Simple widgets in pure JS. For example:

  • Count of datasets in a group (could generalise to a query but not sure how useful that is ...)
  • Embeddable list of top X (5) datasets for a given query
  • Embeddable list of *my* datasets

Either these live at: {site}/widgets and we have some kind of generator (form where I choose my group, or my query).

Or: we have this attached to areas of site where relevant.

Can combine the 2 so that the latter links to the former. Think first will be easier to do and possibly more useful long-term (e.g. can just link people to that page).

Cf. http://okfnlabs.org/ckanjs/

1339750049000000 1340624917000000
#2530 enhancement kindly rgrp ckan-v1.9 new DataHub purge fails on some revisions

See http://datahub.io/ckan-admin/trash and try to purge revisions (*not* datasets). It will fail on some of the revisions with errors like:

Problem purging revision 391db9e8-df57-4e0e-8fe6-d4e0c2318344: (IntegrityError?) update or delete on table "revision" violates foreign key constraint "group_extra_revision_revision_id_fkey" on table "group_extra_revision" DETAIL: Key (id)=(391db9e8-df57-4e0e-8fe6-d4e0c2318344) is still referenced from table "group_extra_revision". 'DELETE FROM revision WHERE revision.id = %(id)s' {'id': u'391db9e8-df57-4e0e-8fe6-d4e0c2318344'}

1339750498000000 1341268280000000
#2531 enhancement rgrp ckan-backlog new New state option: archived / deprecated

Deleted means things will get purged at some point.

Archived means they stay around but get hidden from search results and a big warning notice gets displayed saying this is archived / deprecated.

@richard cyganiak ...

1339750787000000 1339770649000000
#2543 enhancement icmurray ckan-v1.9 new facet.sort is not available in the package_search action

Not all solr facet parameters are available through the pcakage_search action. In particular, facet.sort has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too.

See: http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort

1340013335000000 1340633091000000
#2555 enhancement toby aron.carroll demo phase 5 new Demo site needs a breadcrumb helper

Something to make building breadcrumbs a bit nicer

1340026983000000 1342618384000000
#2572 enhancement toby toby ckan-v1.9 new clean up stats plugin

attempt to disengage the stats plugin from core as much as possible

1340105054000000 1340899524000000
#2573 enhancement icmurray ckan-future new package_search does not allow solr's per-field facet parameters

Solr allows its facet parameters to be specified on a per-field basis, eg. facet.limit applies to all facet fields, but solr allows it to be overriden for a specific field, eg. facet.tags.limit.

We don't support this at the moment because we have a whitelist of valid solr query parameters that we accept. See ckan.lib.search.query.VALID_SOLR_PARAMETERS.

1340112439000000 1340633101000000
#2579 enhancement toby toby ckan-v1.9 new move sort_by functions into lib.helpers

make these more available but keep existing functionality so not to break any users

remove_field()

drill_down_url()

etc

1340281798000000 1340899337000000
#2582 enhancement rgrp ckan-v1.9 new Do not hide notes / readme on dataset pages

Current we hide most of readme and then let users reveal it. Stop doing this and if necessary add a quick link down to resources section. (Maybe also rename resources to Data and Resources ...?)

Aside: believe I have mentioned this somewhere a month + ago but could not find the ticket.

1340312340000000 1340625009000000
#2583 enhancement toby toby demo phase 5 new make sure that we implement authentication where needed

in development many auth checks may have been lost we need to check they are still working etc

1340359478000000 1342086274000000
#2585 enhancement seanh ckan-v1.9 new Escape solr control characters in search queries, add advanced search screen

Suggestion from David Read:

We noticed that some search queries produce unexpected search results in CKAN, due to them containing special characters. For example if you were to search for "Spend over £25,000 - NHS Leeds" then it would not come up with the dataset with that exact name. It was excluding datasets with the word "NHS" due to the dash/minus sign. It works fine if you escape the minus sign: "Spend over £25,000 \- NHS Leeds".

So in data.gov.uk I've added escaping of such control characters in our plugin and this useful routine:

http://fragmentsofcode.wordpress.com/2010/03/10/escape-special-characters-for-solrlucene-query/

Perhaps you would consider providing this in CKAN core in future?

I think there is an occasional case when power users would want to use the special characters - brackets, +, -, boolean operators etc. but maybe these could be reserved for an 'advanced search' screen?

1340360773000000 1340625078000000
#2603 refactor icmurray icmurray ckan-v1.9 new Remove deprecated 'fields' parameter from resource_search

The fields parameter of resource_search was deprecated when fixing #2438. It can be removed in release 1.9, and the action tidied up as a result.

1340730601000000 1340730601000000
#2621 refactor icmurray icmurray ckan-v1.9 new Remove the deprecated 'fields' parameter from tag_search and tag_autocomplete

This was deprecated in 1.8 as it wasn't accessible via GET requests due to being a dict. See #2439

In a future release of CKAN (probably 1.9) it can be removed.

Internal uses of it were removed in #2439, but there are tests that still use it.

1340900569000000 1340900569000000
#2622 defect seanh new Login fails in Opera 12

Try to login to CKAN using Opera 12, get "Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

1340902602000000 1340902602000000
#2625 enhancement seanh ckan-v1.9 new Add i18n strings from non-core but supported extensions to ckan.pot file

Have to decide which non-core extensions are going to be supported first.

1341236903000000 1341236903000000
#2635 enhancement dread new Non-destructive SOLR reindex

You can't run the search-index reindex on a live server because it will give us bad results for 2 to 3 hours while it runs. Can there be an option that doesn't delete the entire index at the start?

Instead it could just delete any items that don't exist any more, then delete them and regenerate them one by one. So the total number of datasets doesn't change much.

1341829394000000 1341829394000000
#2656 defect seanh dread new Feed with few results has bad paging link, causing exception

This page http://thedatahub.org/feeds/custom.atom?q=wombat has 0 results and contains a link to http://thedatahub.org/feeds/custom.atom?q=wombat&page=0 which the page=0 causes this exception:

ckan.lib.search.common.SearchError'>: SOLR returned an error running query
Error: "'start' parameter cannot be negative"
1342001112000000 1342001112000000
#2663 enhancement toby toby ckan-v1.9 new h.resource_display_name needs love

This function is shit and needs cleaning up and a doc string

description is markdown and should be treated properly

either we should truncate all or leave it to the templates but work universally

url if no name / desc this is in demo-theme branch

1342017746000000 1342017746000000
#2673 enhancement rgrp new simplify set of options for resources

Far too many resource options. Lets restrict back to data file and API. Visualizations etc can either get linked in description or in the Related items.

1342300559000000 1342300559000000
#2679 enhancement icmurray icmurray ckan-v1.9 new Change default behaviour of TemplateController.view to 404.

The current behaviour of TemplateController?.view() (which is the fallback controller should all others fail) is to attempt to render (as a genshi template) the requested file.

Although this may be a feature that some instances want. In general, it leads to:

  • 500s when attempting to access a normal template (eg - http://datahub.io/importer/preview)
  • A way of inadvertantly serving things you may not want to serve. (Small risk, as it needs to be renderable as a genshi template).

Solution:

  • Change the controller to 404
  • Ensure there's a way for existing ckan instances to override that behaviour should they need it.
1342436133000000 1342436133000000
#2683 enhancement seanh new Add no-cache header to _tracking API call's response to make sure it doesn't get cached 1342446577000000 1342446577000000
#2688 enhancement ross new Allow ordering of groups in WUI

Currently the group_index page just shows the entire list of groups, forcing the ordering to be by name. It would be better if it could be sortable by name (or reversed) or by package_count (or reversed)

1342520875000000 1342520875000000
#2708 enhancement kindly toby ckan-v1.9 new limit extra data for package/group show

contextpackage_limits? = { 'tags': 5, <- get first 5

'extras': 0, <- get all

}

only get what you ask for have to be explicit

contextgroup_limits? = {} only main item

start with datasets/groups expand if we like it

1342622420000000 1342622420000000
#2709 enhancement icmurray markw new Atom feeds are undocumented

There doesn't seem to be any documentation yet for Atom feeds.

1342624310000000 1342626212000000
#2718 enhancement toby shevski demo phase 4 new can't add dataset to more than one group

trying to add a dataset to another group means it's no longer part of the first group

http://demo.ckan.org/dataset/edit/afterfibre

1342780550000000 1344544203000000
#2719 defect dread new Feeds controller does not catch NotAuthorized exception

Results in bad user experience and WebApp? errors emailed out. Seen in 1.7.1

1342872863000000 1342872863000000
#2721 defect toby shevski demo phase 4 new deleted groups should not show on 'Add to Groups' dropdown

Groups previously deleted still show up in the add dataset process in step 3 'Additional info'

http://s031.okserver.org:2375/dataset/new_metadata/ff

1342948632000000 1344544214000000
#2725 enhancement toby shevski demo phase 5 new Case sensitivity on tags

My feeling is that 'country-US' and 'country-us' should be the same tag. However currently tags with caps are treated differently

see http://s031.okserver.org:2375/en/dataset/test-dataset

with TEST and test - there also get indexed twice in the search page

1342949667000000 1343030773000000
#2726 enhancement toby shevski demo phase 5 new confusing logic on data preview formats
  1. If a user enters the wrong format on a file that can be previewed - it simply won't be previewed (e.g. a CSV or XML file that can be filled in with JSON in format will just not work or check this
  1. If I incorrectly edit format to one that data preview will try to preview it will work even for a format that it doesn't accept (sometimes) e.g. this PDF file I changed the metadata to HTML http://s031.okserver.org:2375/en/dataset/test-dataset/resource/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28
1342949927000000 1343030906000000
#2728 defect toby shevski demo phase 4 new deleted group shows on search index - for admins

'test-group', which has been deleted,shows up on main search page under groups - and can be filtered by - see http://s031.okserver.org:2375/dataset?groups=test-group

1342950784000000 1345023944000000
#2729 enhancement kindly shevski ckan-backlog new searching for tags:[tag] works but tag:[tag] doesn't

which is confusing since you can only search for one tag like this at a time. I.e. tags:economics,cvs or tags:economics, csv or tags:economics+CSV doesn't work for example; therefore tag:economics, should also work!

http://s031.okserver.org:2375/dataset?q=tags%3Aeconomics&sort=relevance+asc

1342951109000000 1342951176000000
#2731 enhancement markw new Some sites permanently 'down for maintenance'

A large number of XXX.ckan.net sites give the following message:

"This Site is Down for Maintenance We apologize for the inconvenience. ~ The Open Knowledge Foundation sysadmins."

The message is unhelpful and patently false - the sites do not exist. Some of them were supposed to have been redirected to a relevant group at thedatahub.org in this ticket (now closed):

http://trac.okfn.org/ticket/933

However, the redirection only seems to have worked in one case, http://si.ckan.net.

The problem still affects the following sites - the first 4 of which have supposedly been merged:

Please sort this out by redirecting, removing the sites, giving a more helpful (and accurate) failure message, etc, as appropriate.

1343045168000000 1343051608000000
#2733 enhancement johnglover johnglover ckan-v1.9 new Datastore logic functions

Where does the data go?

In a postgres database configured by the ckan.datastore_write_url config option which is a sqlalchemy url.

The user should have rights to create tables.

Whats the api like?

We will just implement it as logic functions like the rest of CKAN and will part of core. After that we may add some nicer api functions that use these but that is a secondary concern.

What are the initial logic functions?

  • datastore_create
  • datastore_delete
  • datastore_show

What is the JSON input format for datastore_create

To begin with it can have the following keys. It is fairly consistent with Max Ogdens' gut servers. Except adds resource_id.

{
resource_id: resource_id # the data is going to be stored against.
fields: a list of dictionaries of fields/columns and their extra metadata.
records: a list of dictionaries of the data eg  [{"dob": "2005", "some_stuff": ['a', b']}, ..]
}
  • The first row will be used to guess types not in the fields and the guessed types will be added to the headers permanently. Consecutive rows have to conform to the field definitions.
  • rows: can be empty so that you can just set the fields
  • fields are optional but needed if you want to do type hinting or add extra information for certain columns or to explicitly define ordering.

eg: [{"id": "dob", "type": "timestamp" }, {"id": "some_stuff", "type": "text"}, ...]. A header items values can not be changed after it has been defined nor can the ordering of them be changed. They can be extended though.

  • Any error results in total failure!! For now pass back the actual error.
  • Should be transactional

What json does datastore_delete take?

{
resource_id: resource_id # the data is going to be deleted.
filters: dictionary of matching conditions to delete
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "delete from table where key1 = 'a' and key2 = 'b' ".
    No filters (either not present or not defined) then delete the table. If we want truncate then add truncate: true to truncate the table.
}

What json does datastore_search take?

{
resource_id: resource_id # the data is going to be selected.
filters : dictionary of matching conditions to select
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "select * from table where key1 = 'a' and key2 = 'b' "
q: full text query
limit: limit the amount of rows to size default 100
offset: offset the amount of rows
fields:  list of fields return in that order, defaults (empty or not present) to all fields in fields order.
sort: comma separated field names with ordering e.g "fieldname1, fieldname2 desc"
}

Some free code: https://gist.github.com/3163864

What json does datastore_search return?

{
fields: same type as datastore_create accepts (i.e. with metadata)
offset: The same offset that was supplied in datastore_show
limit: The original limit
filters: The filters that were applied in data_show
total: # total matching records without size or offset
records: [same as data_create] # list of matching results
}

On error will return:

{
__error__: … sql error …
}

What types are allowed?

Aim to support as many postgres/postgis types that have string representations.

http://www.postgresql.org/docs/9.1/static/datatype.html

http://www.postgresql.org/docs/9.1/static/sql-createdomain.html

IDs

Each row in a table will be given an _id column which has an id generated by us which you can use in queries.

Other Features

Each row will store the _full_text index of all the data in the row. At some later point there will most likely be a way to index fields add constraints etc.

1343058886000000 1343656105000000
#2745 defect amercader ckan-v1.9 new Password reset returns an exception if the key parameter is missing

Instead of showing a notice, the password reset page throws an exception if the key parameter is missing:

Module ckan.controllers.user:329 in perform_reset
         c.reset_key = request.params.get('key')
               if not mailer.verify_reset_link(user_obj, c.reset_key):
                   h.flash_error(_('Invalid reset key. Please try again.'))
                   abort(403)
 if not mailer.verify_reset_link(user_obj, c.reset_key):
Module ckan.lib.mailer:100 in verify_reset_link
     if not user.reset_key or len(user.reset_key) < 5:
               return False
           return key.strip() == user.reset_key
 return key.strip() == user.reset_key
AttributeError: 'NoneType' object has no attribute 'strip'

Apart from the obvious fix of checking for the 'key' parameter, it seems like is quite common to get these reset urls without the key parameter, so I suspect some email clients might strip the query params when building the links. We could avoid this problem by making the key part of the url instead of a param:

http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d/b4c2d03fa8

instead of:

http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key=b4c2d03fa8

1343145931000000 1343145931000000
#2748 enhancement shevski demo phase 5 new add 'add new resource' button to sidebar

When editing a resource you see the current and any other existing resources in sidebar

(e.g. see http://s031.okserver.org:2375/en/dataset/format-error-test/resource_edit/d1eac556-c16f-44af-8148-5e3467b57cf8?inner_span=True)

Would be good to have a pretty 'add new' slightly transparent resource folder/pointer undearneath - letting you add resources from the end resource page

1343212878000000 1344503744000000
#2751 enhancement toby toby demo phase 5 new check translations for full demo site

need to check everything gets translated - sean did this before so will have info

1343216443000000 1344243046000000
#2758 enhancement toby toby ckan-v1.9 new file storage gives error if config not available but no useful user information

We get an error which should be improved the actual problem is this but not passed to user

KeyError?: 'ofs.impl'

View as: Interactive (full) | Text (full) | XML (full) Module ckan.controllers.storage:2 in auth_form view Module ckan.lib.jsonp:26 in jsonpify view

data = func(*args, kwargs)

Module ckan.controllers.storage:407 in auth_form view

authorize(method, bucket, label, c.userobj, self.ofs)

Module ckan.controllers.storage:200 in ofs view

StorageAPIController._ofs_impl = get_ofs()

Module ckan.controllers.storage:71 in get_ofs view

storage_backend = configofs.impl?

Module paste.registry:146 in getitem view

return self._current_obj()[key]

KeyError?: 'ofs.impl'

1343287709000000 1343287709000000
#2761 enhancement seanh ckan-v1.8 new Document all the errors you can get when setting up filestore, and how to fix them

Add it to a 'Troubleshooting' section on the filestore page: http://docs.ckan.org/en/ckan-1.7.1/filestore.html

For the error messages and their solutions, see various threads on ckan-dev

1343302566000000 1343302566000000
#2762 defect seanh ckan-v1.8 new test_related.py crashes

/home/seanh/Projects/ckan/ckan/ckan/tests/functional/test_related.py

ImportError? (cannot import name assert_regexp_matches)

1343303753000000 1343303753000000
#2763 defect seanh ckan-v1.8.1 new Multilingual tests failing

test_multilingual_plugin.TestDatasetTermTranslation?.test_dataset_index_translation, test_multilingual_plugin.TestDatasetTermTranslation?.test_group_read_translation both failing for me on master

1343303819000000 1350303864000000
#2768 enhancement toby shevski demo phase 5 new normalise excel to xls

so that data proxy works

1343319382000000 1344351663000000
#2771 enhancement seanh seanh ckan-v1.8 new Documentation and examples for IDatasetForm and IGroupForm

Add minimal, working IDatasetForm and IGroupForm example extensions to core, with tests.

The IDatasetForm example should use tag vocabularies (two birds with one stone)

The IDatasetForm and IGroupForm docs are not very good (and are somewhat spread around different doc chapters), fix them up, and reference the new working examples.

Tab Vocabularies docs should reference IDatasetForm example.

When using convert_to/from_extras() you have to remove any free extras from the form or it won't work, this needs to be documented (in the docstring maybe)

There have been recent changes to the schemas that IDatasetForm and IGroupForm use, make sure the docs are up to date.

1343392238000000 1350303564000000
#2773 enhancement markw markw ckan-v1.9 new About page needs improving

The about page for the DataHub? (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh.

1343646795000000 1343646795000000
#2775 enhancement toby aron.carroll demo phase 4 new Add bin/less to paster serve command

Ideally the ./bin/less command would be run when the server is started.

  • Also it would be good to have a paster command to build the production CSS with {{{ ./bin/less --production }}
  • The command could also detect missing node binaries and redirect to the documentation.
1343685686000000 1344543962000000
#2777 enhancement icmurray new bug: user attributes 1343726363000000 1343726363000000
#2780 enhancement toby shevski demo phase 4 new way for admins to undelete datasets

Since admins can see deleted datasets - there should be a way for them a) to know they are currently deleted & not viewable by normal users - ticket: #2779 b) way to undelete such datasets - this ticket

I suggest a button on the edit form instead of the delete button i.e. remove normal delete button with 'deleted dataset, only admins can view' with undelete button next to the message?

1343737248000000 1345023811000000
#2784 defect icmurray icmurray new model dictize sensitive data

The model dictize layer doesn't consistently remove sensitive data from the dictized models. It should use the current context to decide whether to include sensitive data or not.

1343814685000000 1343814685000000
#2786 enhancement shevski demo phase 5 new target blank HTML downloads

e.g. if I click on download here: http://s031.okserver.org:2375/dataset/example-dataset/resource/d8797e51-b497-46ca-a274-8675533d110b can it take me to a new tab instead of navigating away from ckan?

1343819814000000 1343819814000000
#2788 enhancement amercader amercader ckan-v1.9 new Speed improvements on creating/updating and indexing

Specially needed when importing large numbers of datasets.

Profiling the import command from the harvesting extension has shown some areas where improvements could be made.

1343832992000000 1343832992000000
#2790 enhancement kindly toby demo phase 4 new logic.action.user_show is slow

This is a very slow call it would benefit from the sort of speed-ups that package-search received

for me locally this is taking 6 seconds for rufus using the datahub data I have. I think a lot of this is the dataset retrival/dictization

can we just grab json blobs from solr?

also is it possible to specify a sort order/paging?

I've put this as a demo-theme ticket as it is an big issue on the demo we are at 25 second page loads - which i can get down to about 8.8 secs so this is the main pain point now

1343852483000000 1345023734000000
#2795 enhancement toby demo phase 5 new Check validation of HTML, CSS, JS

Ensure that we are being standards compliant

1343903128000000 1343903128000000
#2796 enhancement mark.wainwright ross new Need a datahub one-pager

A one-pager explaining what the datahub is and with howto/examples for new users. This would make it much easier to explain the value in using the datahub for storing data.

1343924916000000 1345129495000000
#2810 enhancement kindly ckan-future new heroku ckan support

Get ckan working on heroku

1344364858000000 1344364858000000
#2813 enhancement toby markw demo phase 5 new Confusing sidebar on demo dataset page

On a dataset page on demo.ckan.org, the left sidebar is confusing.

  • It starts with some random links. Actually they are links to groups which the dataset is in, but this isn't clear.
  • The sidebar elements that are actually part of the dataset are 'Datset extent' and 'License', so these should be right at the top (if they belong in the sidebar at all), instead of which they are right at the bottom in the junk part of the page (i.e. probably lower than the bottom of the main page, and hence lower than anyone will scroll).
1344420206000000 1344445419000000
#2815 defect seanh seanh ckan-v1.8.1 new db_to_form_package_schema() strips tracking summary, isopen

If an IDatasetForm plugin with a db_to_form_schema() based on db_to_form_package_schema() (which is in turn based on default_package_schema()) is in use then the 'tracking_summary' dict and the 'isopen' bool get stripped from package dicts during validation, e.g. during package_show(), and these values are then not available to templates.

1344444427000000 1350303821000000
#2820 defect danieljohnlewis demo phase 5 new English Language: Visualization -> Visualisation

Problem: In the English version (which has a UK flag, indicating British English), the word "Visualization" is used. For an example see the "Filter by type" drop down on the /apps page. Expected: This should be "Visualisation" in British English. Any instances of "Visualize" should be changed to "Visualise" too.

1344504455000000 1344504455000000
#2821 enhancement danieljohnlewis demo phase 5 new Featured Items on Filter

Problem: On /apps page in the Filter Results box there is a "Only show featured items" checkbox, on selection it comes up with 0 solutions. Expected: Presumably an admin can create "featured items" so that they can be randomly selected on front page (is this correct)? If there are no "featured items" in the whole database can this check box be hidden? Bug is: no UI or obvious way to create featured items. Also the checkbox looks un-styled

1344504504000000 1344505492000000
#2822 enhancement toby toby demo phase 4 new Resource additional info titles format/i18n

the title for additional info should be translated

capitalised etc

1344504620000000 1344543985000000
#2823 enhancement toby toby demo phase 5 new resource additional info title order

Order the items so that none user fields are first from ticket #2707

1344504773000000 1344504773000000
#2828 enhancement toby shevski demo phase 4 new Draft datasets are confusing - tickets need creating

reported as editing datasets incorrect

e.g. clicking on edit here http://s031.okserver.org:2375/dataset/ff takes you to create dataset page http://s031.okserver.org:2375/dataset/edit/ff

but this is the correct behaviour of a draft dataset

We need to show draft datasets correctly

proper tickets need making for the different issues after review of issues with them - who can see, where, admins and viewing, orgs too etc

1344506178000000 1344547324000000
#2829 enhancement johnglover toby ckan-v1.9 new Archiver fails on 403 http response

Had this issue with the archiver on my local machine need to be logged in (I am admin) to see via web front end

$ paster archiver update -c ../ckan/development.ini 


2012-08-09 11:01:37,636 INFO  [ckanext.archiver.commands] Archival of dataset resource data added to celery queue: opencontext-chogha-mish-fauna (1 resources)
2012-08-09 11:01:37,671 INFO  [ckanext.archiver.commands] Getting dataset metadata: south-african-national-gov-budget-2012-13
2012-08-09 11:01:37,900 INFO  [ckan.lib.base]  /api/action/package_show render time 0.043 seconds
Traceback (most recent call last):
  File "/home/toby/okfn/pyenv/bin/paster", line 8, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/home/toby/okfn/pyenv/src/ckanext-archiver/ckanext/archiver/commands.py", line 98, in command
    response = app.post(api_url + '/package_show', data)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 262, in post
    expect_errors=expect_errors)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 243, in _gen_request
    return self.do_request(req, status=status)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 406, in do_request
    self._check_status(status, res)
  File "/home/toby/okfn/pyenv/lib/python2.7/site-packages/paste/fixture.py", line 439, in _check_status
    res.body))
paste.fixture.AppError: Bad response: 403 Forbidden (not 200 OK or 3xx redirect for /api/action/package_show)
{"help": "Return the metadata of a dataset (package) and its resources.\n\n    :param id: the id or name of the dataset\n    :type id: string\n\n    :rtype: dictionary\n\n    ", "success": false, "error": {"message": "Access denied", "__type": "Authorization Error"}}
1344508484000000 1344508484000000
#2830 enhancement toby toby demo phase 4 new Need method to undelete groups

need controller action and front-end method

1344509408000000 1344547341000000
Note: See TracReports for help on using and creating reports.