{22} Trac tickets (2647 matches)

I had a package descriptions with URLs that contain "group:foo". This produces garbled output as the system tries to generate two sets of links: the outer link and an inner link.

Need to fix the parser.

Text:

Webdienst basierende Bereitstellung von Geobasisdaten der Freien und Hansestadt Hamburg. Folgende Geobasisdaten werden als WebMapTileService? (WMT-S) für die Dauer des Wettbewerbs netzbasiert unter der Creative Commons Lizenz zur Verfügung gestellt: Digitale Orthophotos 40 cm Auflösung (Layer: apps4d_DOP40), Digitale Stadtkarte (Layer: apps4d_DISK), Digitale Regionalkarte (Layer: apps4d_DIRK), Digitale Karte 1:5000 (Layer: apps4d_DK5).

Metadateneinträge zu den Daten im PortalU:

• [apps4d_DOP40](​http://www.portalu.de/trefferanzeige?docuuid=D6082285-6CAC-49C5-B7ED-B5A72BC17DEC&plugid=/kug-group:kug-iplug-udk-db_hh)
• [apps4d_DISK](​http://www.portalu.de/trefferanzeige?docuuid=D2DA3359-3DC9-4CA4-AE9B-7C63D53B220F&plugid=/kug-group:kug-iplug-udk-db_hh)
• [apps4d_DIRK](​http://www.portalu.de/trefferanzeige?docuuid=B7DDD599-F677-4C80-BED5-F11D82D5D1DD&plugid=/kug-group:kug-iplug-udk-db_hh)
• [apps4d_DK5](​http://www.portalu.de/trefferanzeige?docuuid=2AE6D23E-48A5-4D85-BC0A-160737E0C8D2&plugid=/kug-group:kug-iplug-udk-db_hh)

One fix is quoting the URLs

Instead of alphabetically as we do currently, alternatively by most viewed datasets

for ​http://thedatahub.org/dataset

Super ticket: #1506

Need to decide which term to use and then have the same for editing as well as viewing a dataset.

In creating/editing a dataset, want more explanation about adding extra fields (probably as a tooltip or similar).. i.e. that this let's you add extra custom metadata such as 'location: uk' which is then searchable etc

Jaakko Louhio reported that the wrong Routes version got installed during CKAN's package install.

He is using Ubuntu 10.04 and he believes it install Routes 1.12.3 instead of what we use which is 1.11.

There's a discrepency in repoze.who versions between the source and package installs:

• repoze.who - package 1.0.18 vs source 1.0.19
• repoze.who-friendlyform - package 1.0b3 vs source 1.0.8

We get a test failure [1] with the 1.0b3 version (from the ubuntu 10.04 python-repoze.who-plugins package). But we've not noticed any problems on s057 instances (br, no, ie etc) which have the package versions of repoze.who.

The reason the package install uses the earlier packaged versions rather than the ones we'd like is that repoze uses all sorts of horrendous import hacks, making it too difficult to put into our 'ckan-conflict' source package.

James suggests we 'do something horrible like dynamically patch repoze on CKAN import'.

[1] ​http://buildbot.okfn.org/builders/builder-ckan/builds/1371/steps/shell/logs/stdio ERROR: ckan.tests.functional.test_user.TestUserController?.test_user_create_unicode

The 'editor', 'anon_editor' and 'reader' roles are intended to have immutable actions. This was designed to prevent their names being subverted - e.g. an editor should always be able to edit! It also meant that when we add Actions (e.g. DELETE-PACKAGE) then it can be added sensibly to these roles in an upgrade just by changing the defaults table (ckan/model/authz.py).

The problem is that this immutability is only enforced on 'db upgrade'. So you can happily change the editor role using the paster command and it works, right up until you do an upgrade and realise permissions are different.

We should stop the paster commands being able to edit these roles. Or get rid of the immutability completely. Views?

Add image attribute for Dataset and Group which would be shown in prominent place.

For dataset this could be a relevant image and for Group this would be more of a logo.

I think this has very high value from a UX point of view.

Issues

• Do we need to support uploading and storing the image (or do users store elsewhere)
  • Could use something like transloadit service
  • Use our own storage facility with a reserved directory
• Do we need to do resizing etc?
  • Maybe. Perhaps can prefer this in reasonably small size and then just size down in css for thumbnails etc.

1. Make it possible to search by tag (e.g. by typing tag:csv into the search bar and clicking enter, it should add the 'csv' tag facet to the search)

2. Rename and standardise the list of format tags, on search page this should also be called 'Format' instead of 'res_format' (in the right hand side bar on search page).

3. Make it possible to view full list of tags, formats and groups by clicking on the name. From here you should be table to click on a classification and go back to a search page faceted by that classification. E.g. from search page, click on 'tags', on tag page click on 'london' or whatever, and be navigated back to search page with search within 'london' tag only. Or y'know, a better way of doing it.

4. More standard classifications, such as 'Location' and 'Theme' - like on publicdata.eu

5. Blue search button should be displayed in line with the search bar, not underneath

6. Datasets should be displayed in order of most viewed or downloaded instead of alphabetically. For alphabetic search we could consider adding a way to facet by first letter of dataset name

Changes to make search both more exploratory and more satisfying to use

• Search query build - #1603
  • Ability to add new facet fields "live"
  • That is add fields which then contain faceted options (a bit like data.hri.fi)
• (??) Autocomplete / drop down on search (i.e. search while you type)
  • Dubious about value / cost ratio here

Probably would involve to pure JS and HTML implementation.

Implementation

Probably require

• API changes to expose solr style API directly #1737

Currently the activity streams database tables just get longer and longer over time. Do we want to eventually delete the oldest activities, to keep the length of the table within limits?

Move ckan/lib/activity.py moved to into the model - say ckan/model/activity_extension.py, because it's so tightly knit with the model code, whereas most of the lib code is used in the controllers.

So users can rename a tag and/or move it between vocabularies using the API.

Currently we have create_tag() and delete_tag(), but if you were to 'update' a tag by deleting it and then recreating it all the datasets that had that tag will have lost it and you'll have to re-add it to them all.

What should happen to datasets that have the tag, if the tag gets moved between vocabularies? All the datasets just keep the tag with the new vocabulary? This will become a problem if/when we support 'radio button'-style vocabularies (where each dataset can only have one tag from the vocabulary).

You get an exception if you use the API to search packages and specify a non-ascii character in a field other than q.

For example:

http://catalogue.data.gov.uk/api/search/package?title=%E2%80%93

This "N-dash" (Unicode character 2013) causes this exception:

Module ckan.controllers.api:460 in search
<<                      if ver in u'12':
                               # Otherwise, put all unrecognised ones into the q parameter
                               params = convert_legacy_parameters_to_solr(params)
                           query = query_for(model.Package)
                           results = query.run(params)
>>  params = convert_legacy_parameters_to_solr(params)
Module ckan.lib.search.query:38 in convert_legacy_parameters_to_solr
<<      for search_key in non_solr_params:
               value_obj = legacy_params[search_key]
               value = str(value_obj).replace('+', ' ')
               if search_key == 'all_fields':
                   if value:
>>  value = str(value_obj).replace('+', ' ')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 30: ordinal not in range(128)

This problem affects CKAN 1.5 and 1.5.1 only.

You can reset your password (#1186) but you have to know the URI (/user/reset) - there is no link to this page!

Huge number of accumulated (and likely unnecessary) scripts in /bin directory.

I have just deny visitors the create-user permission:
sudo -u ckanstd paster --plugin=ckan roles deny reader create-user -c /etc/ckan/std/std.ini
sudo -u ckanstd paster --plugin=ckan roles deny anon_editor create-user -c /etc/ckan/std/std.ini

and after restarting, the register link is *not* hidden, but now when you access the register page, it shows you this message "Unauthorized to create a user" (when not logged in). But anyway that is an improvement.

Purging datasets (deleting them fully, not just changing the state to 'deleted') is important for users testing dataset creation over the API on a test CKAN instance.

Without this, they need to resort to more difficult methods such as:

• cleaning and reloading the database
• setting the test datasets to state 'deleted' and also appending a suffix '_00' and incrementing the number until there is no clash of names.

Requested for NHSIC.

Implementation

• This could slot into the Action API.
• Of course we would need to ensure the user's had been given the specific right to purge.
• I suggest we log the full details of the dataset being purged.

Now that ckanext-storage is back into the core (v1.6), CKAN documentation should probably contain info on ​Storate Metadata API.

Just like we display download counts we should display view counts.

You configure the extension with some RSS and/or Atom feeds, it automatically reads items from these feeds and makes them available in the template context, you write a custom template to e.g. display 'news' items from a Wordpress blog on your front page.

This extension might be simpler and less fragile than ckanext-wordpresser, and also more generally useful.

Details

• Mark Pilgrim's Universal Feed Parser might be useful for reading the feeds

• Feed items should probably be cached somewhere

Suggested

• The news item 'widget' should be wrapped in a known class so that it can be styled easily regardless of the format of any HTML entry.

• For non-HTML formatted items (Atom should tell you the content type of the entry) maybe we should have a template for rendering each item along with any enclosures that it might reference

• Caching is pretty crucial and should probably obey the ttl of the feed.

From Ira: Preview for google spreadsheets are not displaying correctly for me in Firefox v.10.0.02, fine in Chrome. ​http://i.imgur.com/KJaqz.png

When viewing a dataset, it would be nice to show a couple of 'Related Datasets'. i.e. ones that are similar.

SOLR has a feature for finding documents similar to a particular document, called 'More Like This'.

We would like this for DGU.

From #2204

not sure how much we have thought through using this for storing queries / views / visualizations coming from our data viewer. In particular, wonder if this necessitates some kind of support for arbitrary json data ...

### Comments

In the simplest form, right now all I need to store is store the url to the resource plus hashbang config for recline. That would be super easy ...

There are a couple of strings on the authorization group add/edit form that are not being translated:

"Name: Please enter a value" (error message when you try to add an auth group with no name)

"User name"

"Save" button

The form seems to be built using FormBuilder?. See ckan/forms/authorizationgroup.py, builder.set_displayed_fields()

add resource_status_show action function to see the state of celery tasks.

Need to get information from both out task status table and the celery_taskmeta table.

Use jquery chosen?

Would make them much nicer but requires:

• Ensure uniqueness (or deal with ambiguity)
• (?) Slug/names versus titles
  • Aside: what is currently called name should probably be title

At the moment no way for a normal user to find JSON versions of this material. Would it be worth adding back a link somewhere on the page?

This is a resource that is *just* in the datastore. At the moment there are hacky ways to do this (create resource with meaningless URL etc) but we should do this properly:

• New Option: DataStore? Table in add a resource form
• URL field is disabled (or points at /api/data or something rather pointless)
• Download button is disabled
  • Better would be to find a way to serialize an ES table or stream it ...

Some files that might be uploaded as resources on a dataset, such as some image formats, already contain metadata. For example some jpeg files might contain Exif ( ​http://en.wikipedia.org/wiki/Exchangeable_image_file_format ) data.

[ ] Obtain list of embedded metadata 'standards'

[ ] Where the metadata is available it should be added as extra fields on the resource that is uploaded.

[ ] HTML?

[ ] Image formats (exif/gif etc)

[ ] PDF

[ ] Eventually consider ...

[ ] MS Office documents

One user reports that after resetting the password via the reset form he was unable to log in again. He did have an email address associated and received the reset email. After updating his password, login failed

"Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

CKAN's default schema has fields for Author ('The name of the main contact, for enquiries about this particular dataset') and 'Maintainer ('another important contact person'). The distinction is not clear. Also the fields are often left blank.

The roles that seems most important are those of the original owner/publisher of the data, and the person maintaining the CKAN record/copy of it. So I suggest:

(1) Rename the fields 'Owner' and 'Maintainer'.

(2) Change the explanatory text for the relevant fields:

Owner: 'The person or organisation who create/collect/publish the data in this dataset.' Owner e-mail: 'E-mail address for enquiries to the Owner named above.' Maintainer: 'The person maintaining this dataset on [name of CKAN instance], if different from the above.' Maintainer e-mail: 'E-mail address for enquiries to the Maintainer named above.'

(3) When a logged-in user creates a new dataset, the main form should have a checkbox, checked by default, marked 'I am the maintainer of this dataset'. If checked, the Maintainer name and e-mail fields are populated from the user's profile.

At present, a resource which is just a URL link to an external resource has a 'Download' button on the resource page. This gives the misleading impression that the resource is stored locally.

This is related to another small UI issue: I think the URL of a resource should be much more prominent, not buried in the 'Additional Information' table.

Suggested fix:

• Put the URL prominently at the top of the resource page (above the preview) (and make it link to the resource)
• Disable the Download button unless the resource is stored locally.

​http://datahub.io/stats

Unless explicitly told by the source web server the dataproxy should not assume that the content it has can be encoded as UTF-8. Even though the chars from 128 - 255 overlap an attempt to decode some byte array as utf8 will fail whenever a latin1 char whose bitpattern has the MSB set.

This will mean that the UTF8Recoder can be more rigid in its acceptance of data, Postel aside.

Deleted means things will get purged at some point.

Archived means they stay around but get hidden from search results and a big warning notice gets displayed saying this is archived / deprecated.

@richard cyganiak ...

Not all solr facet parameters are available through the pcakage_search action. In particular, facet.sort has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too.

See: ​http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort

Requirements

Require the ability for users to control some level of content that is visible on the home page of their ODS installation. This may be through RSS/Atom feeds (see #2234) or another mechanism but should result in admins being able to change blocks of text on their homepage.

This should not be configuration, but accessible through WUI.

Interface

None

User Stories

• As a system administrator I want to have control over content displayed on the front page beyond featured/popular items.

• As a system administrator I don't want to manage content through having to write an extension.

Tasks

[ ] Analysis

Estimates

Look into writing a cartridge for importing CKAN data into a Virtuoso quadstore

​http://virtuoso.openlinksw.com/dataspace/dav/wiki/Main/VirtSponger#How Does It Work?

Current we hide most of readme and then let users reveal it. Stop doing this and if necessary add a quick link down to resources section. (Maybe also rename resources to Data and Resources ...?)

Aside: believe I have mentioned this somewhere a month + ago but could not find the ticket.

With the organizations and organizations_dataset plugins enabled login and create a new dataset, adding it to a group that you are a member of. Edit the dataset and set the group to none. Now you can no longer edit your own dataset. (You can also specify None for the organization when first creating the dataset, same problem.)

I'm not seeing any javascript or python error, the upload just hangs.

Have to decide which non-core extensions are going to be supported first.

The group_index page has no support in WUI for ordering the groups displayed. Should allow sorting by name

Add support for this for datahub now, and discuss for new 1.9 UI

Currently the group_index page just shows the entire list of groups, forcing the ordering to be by name. It would be better if it could be sortable by name (or reversed) or by package_count (or reversed)

A large number of XXX.ckan.net sites give the following message:

"This Site is Down for Maintenance We apologize for the inconvenience. ~ The Open Knowledge Foundation sysadmins."

The message is unhelpful and patently false - the sites do not exist. Some of them were supposed to have been redirected to a relevant group at thedatahub.org in this ticket (now closed):

​http://trac.okfn.org/ticket/933

However, the redirection only seems to have worked in one case, ​http://si.ckan.net.

The problem still affects the following sites - the first 4 of which have supposedly been merged:

• ​http://lt.ckan.net
• ​http://hu.ckan.net
• ​http://be.ckan.net
• ​http://gr.ckan.net

• ​http://nz.ckan.net
• ​http://bg.ckan.net
• ​http://fi.ckan.net
• ​http://katalogdanych.centrumcyfrowe.pl
• ​http://pl.ckan.net
• ​http://southampton.ckan.net
• ​http://sl.ckan.net

Please sort this out by redirecting, removing the sites, giving a more helpful (and accurate) failure message, etc, as appropriate.

Instead of showing a notice, the password reset page throws an exception if the key parameter is missing:

Module ckan.controllers.user:329 in perform_reset
         c.reset_key = request.params.get('key')
               if not mailer.verify_reset_link(user_obj, c.reset_key):
                   h.flash_error(_('Invalid reset key. Please try again.'))
                   abort(403)
 if not mailer.verify_reset_link(user_obj, c.reset_key):
Module ckan.lib.mailer:100 in verify_reset_link
     if not user.reset_key or len(user.reset_key) < 5:
               return False
           return key.strip() == user.reset_key
 return key.strip() == user.reset_key
AttributeError: 'NoneType' object has no attribute 'strip'

Apart from the obvious fix of checking for the 'key' parameter, it seems like is quite common to get these reset urls without the key parameter, so I suspect some email clients might strip the query params when building the links. We could avoid this problem by making the key part of the url instead of a param:

​http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d/b4c2d03fa8

instead of:

​http://thedatahub.org/en/user/reset/3086e91c-fe09-4a98-92e1-19de67a9ac9d?key=b4c2d03fa8

Add it to a 'Troubleshooting' section on the filestore page: ​http://docs.ckan.org/en/ckan-1.7.1/filestore.html

For the error messages and their solutions, see various threads on ckan-dev

test_multilingual_plugin.TestDatasetTermTranslation?.test_dataset_index_translation, test_multilingual_plugin.TestDatasetTermTranslation?.test_group_read_translation both failing for me on master