{22} Trac tickets (2647 matches)

Currently there's a nice legend titled "Package Listing Key" that appears in right side of "Browse Packages" results. The same key should show on other search results like when searching for a tag.

When people link to a package, a track-back link is auto-created. (Similar system as for blogs).

As suggested by Tim Davies:

Allowing some form of ‘track back’ against datasets When a non-technical user comes to look at a dataset it would be really useful for them to be able to see if anyone has created an interface interpretation of it already.

I found quite a few cases in research of end-users struggling to make sense of a dataset when good interfaces to that data had already been built and blogged about, but without there being any link from the dataset listing to those data uses. Accepting track backs could also make it easier for technical users to find blog posts / shared code etc. relating to a given dataset.

Thread: ​http://lists.okfn.org/pipermail/ckan-discuss/2010-November/000718.html

Scenario 1

1. User installs Refine and CKAN extension for refine
2. On booting refine and asked to load data they can choose from any data package on CKAN.net (or any other CKAN instance)
3. They edit the dataset on Refine
4. On save (or perhaps as a separate option) they are prompted as to whether they wish to sync the dataset back to CKAN (either as a new package or as a new resource on the existing package)

NB: for the dataset sync back some form of "CKAN" storage would be required (we already have storage.ckan.net running but a closer integration would be required)

Scenario 2

1. User visits a package on CKAN.net (or another CKAN instance)
2. There is a button on the page "View and edit this dataset in Google Refine"
3. Click button -- ask them if they have Google refine installed
   • Yes: instructions for loading dataset into refine
   • No: load dataset in hosted version of google refine (we could run this)
4. User edits dataset and hits save. As in previous scenario they are prompted to sync the dataset.

There's a stats plugin (e.g. at ​http://trac.ckan.org/ticket/832).

Output from the googleanalytics plugin should append to that page, if the stats plugin is present.

Possibly the stats plugin and the googleanalytics plugin should be merged?

Finally, if the stats plugin is active, then a link to the stats page should be added to the main site footer.

Proposer: Adrià Mercader

Abstract

The new harvester interface allows to create harvesters for different sources, but right now harvesters don't have many ways to describe and configure themselves. We need a way of allowing them to:

• Expose their type and other details so they can be used internally and on the UI.
• Define configuration settings for particular harvester instances.

The Problem

Harvester description

The current UI for adding and editing harvest sources is the same used in ckanext-dgu, and thus the 3 harvester types used in DGU to harvest various GEMINI realted sources are hardcoded in the form. The form will be migrated to a DGU-independent one, so we need the harvesters to provide all the necessary data. There is a current get_type method that returns the harvester type, but for make it compatible with the DGU forms, it returns a machine-readable string (e.g. "CSW Server"), making it error prone.

Arbitrary configuration

In the current implementation, when the harvest process is started, ckanext-harvest looks for all the available plugins that implement the IHarvester interface and calls the appropiate methods for the current stage (gather_stage,fetch_stage,import_stage). At these stages, harvesters have no way of applying arbitrary configuration options, so all harvesters of the same type behave on the same way. For instance, the CKAN harvester needs a way to define the API version to use when harvesting remote instances (Right now, the version 2 is hardcoded on the code).

Specification

Harvester description

Harvesters will need to provide the following information so the UI form can be built:

• name: machine-readable name (e.g. "waf"). This will be the value stored in the database, and the one used by ckanext-harvest to call the appropiate harvester.
• title: human-readable name (e.g. "Web Accessible Folder (WAF)"). This will appear in the form's select box.
• description: a description of what the harvester does (e.g. "A Web Accessible Folder (WAF) displaying a list of GEMINI 2.1 documents"). This will appear on the form as a guidance to the user.

The way to provide it will be an info method that all harvesters must implement, which will return a dictionary with the previous elements:

    {
        'name': 'csw',
        'title': 'CSW Server',
        'description': 'A server that implements OGC's Catalog Service 
                        for the Web (CSW) standard'
    }

Arbitrary configuration

As different harvesters will have very different needs, we need to provide a way to persist arbitrary configuration flags for each harvest source. The more flexible way given the current architecture in my opinion would be to store the configuration options as a JSON encoded object as a property of the harvest source (There already is an unused DB field called config in the database) (Maybe using JsonType??).

This will mean adding an extra field in the harvest source form to allow entering the configuration. This could be just a simple text field where users enter the JSON encoded object or a more clever mechanism (i.e an "Add a configuration flag" link that adds two new text fields for the key and value for each flag, and a mechanism to later build the JSON object). In any case, this should probably be hidden in an "Advance options" section.

Why do it this way

Harvester description

The info method would provide a single point to get all the information related to the harvester, and future properties could be added to the dictionary returned without having to modify the interface.

Arbitrary configuration

There is an already existing config field in the database, so we won't need to change the model. Harvesters could access the config object at any of the stages. Of course they could provide default values in their implementations so users don't need to enter them everytime.

Implementation plan

Deliverables

Risks and mitigations

The highest risk on the harvesters info method side is that harvester implementation don't offer one of the necessary properties (namely name and title). This could fire a warning when showing the UI form or using the CLI.

Participants

Adrià Mercader to do it.

Progress

None yet.

The Handle System is an initiative to provide persistent references for resources. That is, it's basically a proxy system for preventing link rot.

Its documentation is here: ​http://www.handle.net/. Servers running CKAN could host a "Local Handle Service", which redirects a hash of a resource to an actual URL.

Some suggested use cases:

• Researcher would like to cite where data came from
• Agencies would like to have a way to prevent vendor lock-in from CKAN if they decide to move to another platform

Currently, each ckan site needs its own ckan wsgi process. That eats a lot of resources where many ckan sites are served from one machine (e.g. eu3).

That would dramatically change if a ckan process could behave like multiple ckans (e.g. like Apache's "<VirtualHost?>", or tracd). Depending on the "Host:" header in the HTTP1.1 request, it would choose which local ckan ini file to obey.

I see two ways to constitute the map hostname-to-ini-file map:

• ckan reads a set of ini files, and each ini file declares which servers names it is responsible for
• In a global ini file, there are directives mapping servernames to ini files.

In either case there should be a global ckan ini having the default settings for all local ckan sites. Each site ini could be very short then, just having e.g. title, name, database credentials, active plugins etc.

When a package has been deleted, say for spam moderation, comments still appear in the recent comments section.

This is a problem because non-admin users will be shown a warning that they're not authorised to view the package if they click on the link.

At CKAN.net currently, this affects the most recent comment.

Administrators have "View", "Edit" and "History" tabs. However, I can't see a way to delete a package from the web UI.

Version: CKAN.net as of today

'Publisher' entities in the model. They are hierarchical.

'User-Publisher' connections with one or more roles (e.g. drafter, moderator).

Authorization settings can control who can set what values in a 'published by' type field.

Publishers and User-Publishers available to read in the API.

Future tickets will provide:

• API to write Publishers and User-Publishers
• UI to edit Publishers and User-Publishers

(This feature deprecates authorization groups)

We are getting more and more spam on ckan.net and we need to improve our strategy of combating it. It is bad because google ranks who we link to (which we do want for legitimate links), and our front page contains the latest package edits, so spam is immediately very visible.

Spam:

• Package creation
• Package edit
• User creation
• Group creation

Systems to consider:

• Automatic
  • captcha
  • bayesian scoring
• Sysadmin
  • a tool to quickly analyse and remove packages, edits and users
• Helpful users
  • button to click indicating spam, leading to any/all of:
    • add TODO item (which emails package admin and sysadmins)
    • quarantine the package/user so it doesn't show up on the front page and a warning is shown when you view it

General thoughts:

• We should learn from the wikipedia tools, as they are the experts
• This is a serious problem that is only going to get worse and is a threat to the whole site.

The current search implementation uses the output of the the as_dict method of the domain Package object to update the index

​https://bitbucket.org/okfn/ckan/src/56c79e3fc44c/ckan/lib/search/index.py#cl-48

It also uses package_to_api1 in the SynchronousSearch? plugin:

​https://bitbucket.org/okfn/ckan/src/f9dfb0506594/ckan/lib/search/__init__.py#cl-93

This prevents extensions from being able to index custom properties (e.g. faceting by custom extras not included in the model).

The search should use the logic function to get the package properties:

get_action('package_show')(context,data_dict)

Super ticket: #1190

A data processing system which utilizes the Webstore. One could get a long way with simple javascript running in the browser for development with this javascript then run offline using something like nodejs. Alternatively one could allow one to specify a url to e.g. a python file which would then be run in a sandbox (with access to some specified set of python modules)

A sysadmin should be able to easily examine a suspect group or package, determine if it was created by a spammer (as opposed to being a legitimate object that has been graffitied by a spammer) and purge it.

The existing two-stage revision delete is currently unreliable and perhaps too laborious.

Olav and Richard have needs along this line.

I had a package descriptions with URLs that contain "group:foo". This produces garbled output as the system tries to generate two sets of links: the outer link and an inner link.

Need to fix the parser.

Text:

Webdienst basierende Bereitstellung von Geobasisdaten der Freien und Hansestadt Hamburg. Folgende Geobasisdaten werden als WebMapTileService? (WMT-S) für die Dauer des Wettbewerbs netzbasiert unter der Creative Commons Lizenz zur Verfügung gestellt: Digitale Orthophotos 40 cm Auflösung (Layer: apps4d_DOP40), Digitale Stadtkarte (Layer: apps4d_DISK), Digitale Regionalkarte (Layer: apps4d_DIRK), Digitale Karte 1:5000 (Layer: apps4d_DK5).

Metadateneinträge zu den Daten im PortalU:

• [apps4d_DOP40](​http://www.portalu.de/trefferanzeige?docuuid=D6082285-6CAC-49C5-B7ED-B5A72BC17DEC&plugid=/kug-group:kug-iplug-udk-db_hh)
• [apps4d_DISK](​http://www.portalu.de/trefferanzeige?docuuid=D2DA3359-3DC9-4CA4-AE9B-7C63D53B220F&plugid=/kug-group:kug-iplug-udk-db_hh)
• [apps4d_DIRK](​http://www.portalu.de/trefferanzeige?docuuid=B7DDD599-F677-4C80-BED5-F11D82D5D1DD&plugid=/kug-group:kug-iplug-udk-db_hh)
• [apps4d_DK5](​http://www.portalu.de/trefferanzeige?docuuid=2AE6D23E-48A5-4D85-BC0A-160737E0C8D2&plugid=/kug-group:kug-iplug-udk-db_hh)

One fix is quoting the URLs

The 'editor', 'anon_editor' and 'reader' roles are intended to have immutable actions. This was designed to prevent their names being subverted - e.g. an editor should always be able to edit! It also meant that when we add Actions (e.g. DELETE-PACKAGE) then it can be added sensibly to these roles in an upgrade just by changing the defaults table (ckan/model/authz.py).

The problem is that this immutability is only enforced on 'db upgrade'. So you can happily change the editor role using the paster command and it works, right up until you do an upgrade and realise permissions are different.

We should stop the paster commands being able to edit these roles. Or get rid of the immutability completely. Views?

Move ckan/lib/activity.py moved to into the model - say ckan/model/activity_extension.py, because it's so tightly knit with the model code, whereas most of the lib code is used in the controllers.

I have just deny visitors the create-user permission:
sudo -u ckanstd paster --plugin=ckan roles deny reader create-user -c /etc/ckan/std/std.ini
sudo -u ckanstd paster --plugin=ckan roles deny anon_editor create-user -c /etc/ckan/std/std.ini

and after restarting, the register link is *not* hidden, but now when you access the register page, it shows you this message "Unauthorized to create a user" (when not logged in). But anyway that is an improvement.

Now that ckanext-storage is back into the core (v1.6), CKAN documentation should probably contain info on ​Storate Metadata API.

If you have a spatial enabled database then if you don't disable the spatial extension in the CKAN config temporarily then you get errors when you run paster db clean and paster db init.

Can't you just modify the clean and init functions to run without extensions enabled?

The wider problem is that extensions do their own inits every time you do load_environment, which seems crazy and inefficient to me, since this occurs every time a request comes into CKAN. But that is another problem/ticket.

If a dataset has no resources the resources list currently says '(none)'.

Here is a suggested improvement, provided that a maintainer is named: 'There are no data resources here yet. For information about this data, contact the dataset maintainer.'

Deleted means things will get purged at some point.

Archived means they stay around but get hidden from search results and a big warning notice gets displayed saying this is archived / deprecated.

@richard cyganiak ...

Looks like recent CKAN cleanups broke some imports:

​https://gist.github.com/2935639

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments

Implementation:

​http://s031.okserver.org:2375/group/new

Instead of pointing to /tags/{tagname} all tag links should just point to a dataset search filtered by that tag.

Examples are on the homepage and the dataset pages.

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments

Implementation:

​http://s031.okserver.org:2375/dataset/

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments

Implementation:

​http://s031.okserver.org:2375/group/test-group

All strings from the ckanext-ecportal extension (and from CKAN core) need to be exported to a pot file to send to tenforce.

• branch from 1.7.1
• apply CKAN patches that are current part of installation instructions
• create i18n patch so custom i18n directories can be specified via a config option

For example:

Creating a template in ckanext-demo/templates/base.html containing something like

{% ckan_extends "base.html" %}

Will extend ckan/templates/base.html

Super: #1594

From Richard Cyganiak on the CKAN discuss list:

According to (most of) the definitions of the five-star scheme, the first star already requires an open license, so a dataset shouldn't get any stars at all if it's not under an open license.

Looks like some of the style has been updated/changed on the group read page but the changes haven't been applied to the organizations read page.

Reported on IRC by 'floapps'.

• when posting a group update with {packages: [package-1, package-2, etc.]}, getting ValidationError?: {'Packages': {'name': Missing value?}}, although package objects did contain a name field
• same API call worked with organization and publisher profile disabled
• also worked if added a form_to_db_schema_options function which used default_update_group_schema for api updates instead of default_group_schema
• tried on both 1.7 and 1.7.1b

Check to see that all the strings are getting translated, i.e. that all user-visible strings are being passed to gettext. Fix any strings that are not getting translated.

There's no docs for setting up the internal analytics, nor what it provides.

Setup

Add: ckan.tracking_enabled = true to .ini file

Run: paster tracking -c dev.ini

Related links

​http://trac.ckan.org/ticket/2251

Example

​http://s031.okserver.org:2375/not-here

Should show the 404 template.

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62821386/comments

Implementation:

​http://s031.okserver.org:2375/dataset/adur_district_spending/related/new

Needs a new endpoint at /dataset/DATASET/related/new this should have a form containing the following fields.

• Title (required)
• Type of item (Application|Visualisation)
• Description
• URL (required)
• Image URL

When submitted if an item is created it should redirect (303) back to /dataset/DATASET/related with a flash message saying "Related item was successfully created".

If failed to create it should redirect (303) back to the form and populate the error messages, see add dataset for examples (​https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/package/new_package_form.html)

Super: #1594

• qa/{username}
• qa/{groupname}
• paginate QA results
• search / filter QA results
• list organisation report by default, but can disable via config option (done)
• UX tidy up of report pages - hide border if no sidebar, etc

• Already done for basic publisher forms (index, read, edit), but needs to be done for additional forms (in particular, form paths like 'organization_apply' that are hardcoded into the organizations controller).
• change organization mapping to /publisher

When clicking 'add dataset' on a publisher/group page, the new dataset form should select the appropriate group.