{22} Trac tickets (2647 matches)

Spec - see attached, which is a copy of the Pirate Pad from Lauren: ​http://piratepad.net/2C2iwiLDhd

Spam entries have started to occur on ckan.net in the last few months. It would be useful to be able to purge these revisions -- that is permanently delete all changes to domain objects associated with those revision (one might also permanently delete that revision -- or alternatively simply mark it as purged).

In addition to providing this facility from the shell it would also be useful to be able to do this from the web interface (with associated restrictions on usage via an authorization controller of some sort).

Sort out how we deal with dashes versus underscores in package names.

Sometimes editing a package via api on dgu results in some countries being lost in geo coverage.

Something to make building breadcrumbs a bit nicer

Some strings in CKAN are passed to gettext with more than one %s in them, this is no good as translators may need to change the order of substituted words.

Strings should not use the old style % substitution, they should use the new format method in Python 2.6+, e.g.:

'This {food} is {adjective}.'.format(food='spam', adjective='absolutely horrible')

​http://docs.python.org/release/2.6.8/tutorial/inputoutput.html#fancier-output-formatting

We should probably only change strings that are marked for translation and that contain multiple %s, don't want to make too much work or cause too many strings to change and need to be re-translated.

Some request exceptions are currently not being caught (see the celery log on thedatahub for examples)

Some files that might be uploaded as resources on a dataset, such as some image formats, already contain metadata. For example some jpeg files might contain Exif ( ​http://en.wikipedia.org/wiki/Exchangeable_image_file_format ) data.

[ ] Obtain list of embedded metadata 'standards'

[ ] Where the metadata is available it should be added as extra fields on the resource that is uploaded.

[ ] HTML?

[ ] Image formats (exif/gif etc)

[ ] PDF

[ ] Eventually consider ...

[ ] MS Office documents

Some datasets become obsolete: e.g. ​http://datahub.io/dataset/uncdb Do not want to delete them but want them marked as archived or deprecated or something.

Solutions: ResourceSeriesLoader? allows synonyms for values.

Solr install instructions install the wrong version

So users can rename a tag and/or move it between vocabularies using the API.

Currently we have create_tag() and delete_tag(), but if you were to 'update' a tag by deleting it and then recreating it all the datasets that had that tag will have lost it and you'll have to re-add it to them all.

What should happen to datasets that have the tag, if the tag gets moved between vocabularies? All the datasets just keep the tag with the new vocabulary? This will become a problem if/when we support 'radio button'-style vocabularies (where each dataset can only have one tag from the vocabulary).

So that Drupal doesn't need to make any adjustments to the form, and risk data being lost on submission.

Since we upgraded to new vdm (v0.4) which has built in purge support existing purge support looks to be broken. Should be reasonably easy to fix this.

Cost: 1h

Since we now require email addresses, it's sensible to request users' gravatars to add a little flavour to their user profile (and, potentially, other places eg. comments threads?)

Since the main menu items have changed (part of the UI redesign) the selected items are not functioning correctly.

Specifically if Home / About or Add Package / Search are selected both menu items are highlighted.

This is because the code assumes only one top-level menu item per controller.

Since my last pull I'm getting a strange new error which is presumably to do with my local config or assumptions about the storage extension.

Test output here: ​https://gist.github.com/2007985

Since admins can see deleted datasets - there should be a way for them a) to know they are currently deleted & not viewable by normal users - ticket: #2779 b) way to undelete such datasets - this ticket

I suggest a button on the edit form instead of the delete button i.e. remove normal delete button with 'deleted dataset, only admins can view' with undelete button next to the message?

Simple widgets in pure JS. For example:

• Count of datasets in a group (could generalise to a query but not sure how useful that is ...)
• Embeddable list of top X (5) datasets for a given query
• Embeddable list of *my* datasets

Either these live at: {site}/widgets and we have some kind of generator (form where I choose my group, or my query).

Or: we have this attached to areas of site where relevant.

Can combine the 2 so that the latter links to the former. Think first will be easier to do and possibly more useful long-term (e.g. can just link people to that page).

Cf. ​http://okfnlabs.org/ckanjs/

Simon has spotted 31 packages whose resources have disappeared in CKAN, but still appear in the Drupal front-end.

Here are the details:

• in CKAN's package view (in web interface, API and dump) no resources display
• but they WERE created in CKAN, as shown by the revision diffs.
• they are in the Drupal front-end

So these resources must have been in the CKAN API at some point and then disappeared without trace/revision to alert Drupal.

Packages affected: anti-social-behaviour-orders-1999-2007 asylum-applications-jan-mar-2009 control-of-immigration-quarterly-statistical-summary-united-kingdom-2009-october-december coroners-statistics-england-and-wales courts-statistics-user-survey-england-and-wales court-statistics-company-insolvency-and-bankruptcy-england-and-wales court-statistics-england-and-wales court-statistics-mortages-and-landlord-possession-england-and-wales crime-in-england-and-wales crime-statistics-local-reoffending-england-and-wales crime-statistics-prison-and-probation-england-and-wales crime-statistics-reoffending-of-adults-england-and-wales crime-statistics-reoffending-of-juvenilles-england-and-wales data_gov_uk-datasets digest-uk-energy-statistics-2008 directgov-central-hottest-pages-monthly directgov-central-internal-search-terms-monthly directgov-section-visits-monthly electricity-consumption-2007 electricity-gas-consumption-2007 energy-consumption-uk-2008 final-energy-consumption-2007 foi-statistics-uk-central-government fuel-poverty-statistics-2007 gas-consumption-2007 gb-reported-bicycling-accidents gb-road-traffic-counts gb-traffic-matrix greenhouse-gas-emissions-2008 high-level-indicators-energy-use-2006 judicial-and-court-statistics-england-and-wales laboratory-tests-and-prices local-authority-carbon-dioxide-emissions-2007 magistrates-courts-statistics-survey-england-and-wales monthly-energy-prices monthly-energy-trends ni_012_refused_and_deferred_houses_in_multiple_occupation_hmos_licence_applications_leading_to_immig ni_013_migrants_english_language_skills_and_knowledge ni_023_perceptions_that_people_in_the_area_treat_one_another_with_respect_and_consideration ni_024_satisfaction_with_the_way_the_police_and_local_council_dealt_with_anti-social_behaviour ni_025_satisfaction_of_different_groups_with_the_way_the_police_and_local_council_dealt_with_anti-so ni_026_specialist_support_to_victims_of_a_serious_sexual_offence ni_029_gun_crime_rate ni_031_re-offending_rate_of_registered_sex_offenders ni_032_repeat_incidents_of_domestic_violence ni_034_domestic_violence_-_murder ni_036_protection_against_terrorist_attack ni_038_drug_related_class_a_offending_rate ni_078_reduction_in_number_of_schools_where_fewer_than_30_of_pupils_achieve_5_or_more_a-_c_grades_at ni_101_looked_after_children_achieving_5_a-c_gcses_or_equivalent_at_key_stage_4_including_english_an ni_109_delivery_of_sure_start_childrens_centres ni_126_early_access_for_women_to_maternity_services ni_127_self_reported_experience_of_social_care_users ni_128_user_reported_measure_of_respect_and_dignity_in_their_treatment ni_181_time_taken_to_process_housing_benefit-council_tax_benefit_new_claims_and_change_events ni_184_food_establishments_in_the_area_which_are_broadly_compliant_with_food_hygiene_law ni_185_co2_reduction_from_local_authority_operations ni_190_achievement_in_meeting_standards_for_the_control_system_for_animal_health ni_194_air_quality_-_reduction_in_nox_and_primary_pm10_emissions_through_local_authorities_estate_an other-fuels-consumption-2006 police-use-firearms-england-wales-2007-2008 prison-end-of-custody-licence-releases-and-recalls-england-and-wales prison-population-england-and-wales probation-offender-management-caseload-statistics-england-and-wales probation-statistics-quarterly-brief-england-and-wales quality-indicators-energy-data-2007 quarterly-energy-prices quarterly-energy-trends road-transport-energy-consumption-2007 sentencing-statistics-england-and-wales statistics-terrorism-arrests-outcomes-2001-2008 ukba-control-of-immigration-statistics-2008 ukba-control-of-immigration-statistics-2008-supplementary-tables uk-energy-in-brief-2008 uk-energy-sector-indicators-background-2008 uk-energy-sector-indicators-key-supporting-2008 uk-exportcontrollists uk-exportcontrol-sanctions uk-export-control-statistics uk-glossary-exportcontrol uk-ipo-offences weekly-fuel-prices

Similar to publicdata.eu, want to have themed areas such as finance, environment, census, etc and country tags

Should return help for the harvester (currently crashes).

Should not be hard to do (lots of existing libraries) but not sure that this is very important.

Should list resources in the form of search results rather than as a table.

Should have the following features:

• A list of all packages with broken links on the resource
• Under each packages a description of any resources with broken links to include:
  • Link
  • Description
  • Format
  • First found to be broken date

[If you can't do the last one yet, don't worry]

Should have all supported extensions enabled, those that don't work should be disabled and ticketed. Also needs a customised homepage.

Should do this sooner rather than later to avoid 'clear water' opening up and need to deploy then asap onto production machine to avoid 'clear water' between trunk and production (which then prevents pushing from trunk to production).

Should be group_form() I think

Should be able to limit ability to a user's ability to do things with packages (read, edit etc). This is a big ticket (it may required splitting) and full details are in separate wiki page: AccessControl.

Cost: 8d

Several things:

• Default package search (used e.g. on front page) should deal well with multiple items: "abc xyz" means search for both abc and xyz (as separate searches)
• Should automatically display a list of matching tags in a line at top with number of associated packages
• have an option to only search for fully open material (or openly licensed material ...)

Since we should be moving to a more sophisticated search solution anyway (to allow prioritisation etc) maybe we should cautious as to what we do now.

Set-up publicdata.eu to harvest datasets at rs.ckan.net (Serbian community CKAN).

Set up a Mediawiki to contain info on CKAN.net (community) conventions etc.

Set limits in /etc/security/limits.conf so that we can always ssh in at least. Requested by DGU.

Set a dataset's license to 'Other (Not Open)', the dataset will have an Open Data icon next to it

Sending emails is failing in some cases, and the error message just says 'Could not send reset link'.

Seems some package have ended up with same tag twice: ​http://www.ckan.net/package/fred2dot0

Check whether this is still possible and if it is fix it ...

Cost: 2h

See trace below.

As in #2231, the schema passed with the context is no longer used on the logic layer (in this case the one used on the DGU form is used). This can be fixed allowing a schema to be passed with the context.

  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 139, in import_stage
    self.import_gemini_object(harvest_object.content)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 163, in import_gemini_object
    package = self.write_package_from_gemini_string(unicode_gemini_string)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 386, in write_package_from_gemin
    package = self._create_package_from_data(package_dict, package = package)
  File "/var/lib/ckan/std/pyenv/src/ckanext-inspire/ckanext/inspire/harvesters.py", line 486, in _create_package_from_dat
    package_dict = action_function(context, package_dict)
  File "/usr/lib/pymodules/python2.6/ckan/logic/action/update.py", line 169, in package_update
    data, errors = validate(data_dict, schema, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 216, in validate
    converted_data, errors = _validate(flattened, schema, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 289, in _validate
    convert(converter, key, converted_data, errors, context)
  File "/usr/lib/pymodules/python2.6/ckan/lib/navl/dictization_functions.py", line 179, in convert
    converter(key, converted_data, errors, context)
  File "/var/lib/ckan/std/pyenv/src/ckanext-dgu/ckanext/dgu/validators.py", line 81, in validate_license
    if data[('license_id',)]== '__extra__':
KeyError: ('access_constraints',)
2012-03-19 10:47:41,736 ERROR [ckanext.inspire.harvesters] Error importing Gemini document: ('access_constraints',)

See the related sidebar box

​http://s031.okserver.org:2375/dataset/adur_district_spending

If a package has related items the first (or a random) related item should be passed into the snippet.

See reports on list at:

• ​http://lists.okfn.org/pipermail/ckan-dev/2011-July/001103.html
• ​http://lists.okfn.org/pipermail/ckan-dev/2011-July/001107.html

Tracked this to issue in dictization/model_save whereby was ignoring case where Package Tag already there but in deleted state. Rather than describe at length see fix.

See ​http://wiki.ckan.org/Dataset_and_Resource_Pages_Proposal

• Create a collapsible section for the README
• Reduce the number of things on the page
• Tidy up groups and tags in the sidebar
• Resource view becomes a list of search results

(additional)

• Resource navbar item links to first resource
• Resource navbar has a dropdown menu of all resources in the dataset
• Edit and Authz navbar items should be in a separate logical group to the right

Very rough summary: create more info section and move stuff out (relayout page e.g. README, tags etc)

See ​http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670

This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do).

Assigning to David for triage and review.

See ​http://datahub.io/ckan-admin/trash and try to purge revisions (*not* datasets). It will fail on some of the revisions with errors like:

Problem purging revision 391db9e8-df57-4e0e-8fe6-d4e0c2318344: (IntegrityError?) update or delete on table "revision" violates foreign key constraint "group_extra_revision_revision_id_fkey" on table "group_extra_revision" DETAIL: Key (id)=(391db9e8-df57-4e0e-8fe6-d4e0c2318344) is still referenced from table "group_extra_revision". 'DELETE FROM revision WHERE revision.id = %(id)s' {'id': u'391db9e8-df57-4e0e-8fe6-d4e0c2318344'}

See how we did it already for other things.

Note: on ckan.net older PackageRevision?.id might not be identical to Package.id but this may need sorting at this point.

See attachment. Just need to amke sure that there's enough spacing around the edit button and that it lines up so it doesn't look too wierd

See Update a release (ticket 14) for more information on what should be shown.

See '### Task 2' in this etherpad: ​http://ckan.okfnpad.org/notifications

I have no idea yet about how to do this or how long it might take me.

Search results are slow as we do many package_dictize. Store the package_dict in the search index so that we do not have to hit the database for each of them.

Search of form: tags: ... behaves differently depending on whether there is a leading space:

• tags: postcode - tags found correctly but no packages found
• tags:postcode - tags incorrectly found but correct packages found

Let's fix this.

Cost: 0.5h

Seanh and icmurray both touched on the html snippets in facets.html. So there's a merge issue. icmurray to merge changes in the multi-lingual branch prior to merging into master.

Sean Burlington wrote:

I recently came across a documentation issue

​http://knowledgeforge.net/ckan/doc/ckan/api.html

---

The key should be passed in the API request header: Header Example value HTTP_AUTHORIZATION fde34a3c-b716-4c39-8dc4-881ba115c6d4

---

But the header name is actually just 'AUTHORIZATION'

For example to set it in PHP

curl_setopt($ch, CURLOPT_HTTPHEADER, array('AUTHORIZATION: xxxxx-xxxx-xxxx-xxxx-xxxxxxx'));