{22} Trac tickets (2647 matches)

Add Vocabulary domain class, add logic functions for creating, updating, listing, getting, deleting vocabularies, add tests.

Part of #1698

See the etherpad for notes: ​http://ckan.okfnpad.org/22

ckan/logic/action/get.py:tag_list() Currently (I think) just returns all tags in the CKAN instance that the user has access to. Needs an optional argument to get only tags from a particular vocabulary? When called with no argument, should only return free tags? Also add an optional dataset argument, so you can get tags for a given dataset and/or vocabulary?

ckan/logic/action/get.py:tag_autocomplete() Add optional argument to get only tags for a given vocab? When called with no argument, should get only free tags?

ckan/logic/action/get.py:tag_search() Again, optional arguments for restricting by vocabulary and/or dataset.

Add a helper function to make it easy for extensions to add new vocabularies to default_package_schema(). The helper function should take the name or ID of a vocabulary (which should already exist in the db) and add the necessary key: [schema] entry to default_package_schema(), with the necessary validation, authorisation, transformation.

Right now harvest sources can be inactivated (only via the cli), but we need to make sure:

• Expose this on the web interface (only for sysadmins, see #1731)
• Inactive sources are not shown in the web interface

Also for deleted packages:

a) Deleted packages get state="deleted" (not actually purged) so they are hidden on front end ACTION: need a button on the front end for "Delete" for sysamdins

Done in #1425. Not sure if the button is only shown to sysadmins.

b) No undelete: If you reharvest a deleted package, it remains deleted

• Publisher should create a new record with new identifier and same/different content as applicable.
• If publisher reharvests, a deleted document they get a error and are told to create new identifier for their updated.
• You won't get the state where a package is updated in its deleted state
• If they change the ID and reharvest, a new record appears on DGU.

c) Update CSW interface so that export doesn't export deleted packages d) Check geospatial search doesn't search deleted pacakges e) Check deleted packages don't appear in harvest source details, or in the counts or anywhere in the harvest interface or API.

Need to add extension point to change the data_dict that passed to the dataset view template. This extensions modify what data gets shown.

Tasks include:

• Fix up data_dict and templates so that the pkg objects do not get used in the templates. This is to assure that all data passed is modifiable. (2d) (complete)
• Add extension point to Ipackage controller (1d)
• Test (1d)

• [x] hide black stars on the data-view page
• [x] Edit on view form (if you have the right permission)
• [x] RES_FORMAT ~> Resource Format
• [x] Map Based Search -> UK Location Map Based Search
• [x] Note at the top right expalining that these are seach filters (if they are) -> this requires "add a dataset" and "other access" to be moved to a different place (just some whitespace is fine)
• [x] Display publisher and provider (UKLP records only).
• [x] analysis of filter by OGL / non-OGL
• [x] provide /dump on the build server

The publisher index page should show a treeview of the publishers within the system. Probably behind a tab.

Only sysadmin can access create publisher form.

History on user profile should only be visible to a user who is a member of at least one group.

Add a vocabulary column to the tags database table, change tags to have unique (tag_name, vocabulary_id) instrad of unique tag name, update methods in the Tag and Package classes to deal with the fact that tags may belong to vocabularies and that tag names are no longer unique.

Need to test:

• API call to create new vocab
• API call to add tags with a given vocab
• API call to update tags
• Dummy plugin to test that updating tags works via WUI

Add API calls for adding one or more tags to and removing one or more tags from a vocabulary, without affecting the other tags in that vocabulary and without having to pass the full list of the vocabulary's tags.

A new table with 3 columns should be added. term, term_tranlastion, language_code. This table will be used for all translations, including tags. The table should have indexs on both the term and (term, language_code) combination.

If the current page URL has unicode in it (e.g. u'/tag/biocombust\xedveis') then I'm not sure how why it isn't more encoded, but we are seeing links such as http://beta.dados.gov.br/dados/locale?locale=hu&return_to=/dados/tag/biocombust%C3%ADveis&hash=7fd941b5ba1a28bc84406c4191bf07b45a42a251 which cause an exception when the foreign character is passed to ckan.lib.hash.get_message_hash().

When you are on the /dataset/new page and you try and change locale then you get a 500 error.

This is because it adds the 'cache' parameter, to ensure any proxy cache in the chain does not just send the cached page.

e.g. ​http://127.0.0.1:5000/dataset/new?__cache=37713707

Super ticket: #1506

• Show member since
• Display email (in obfuscated form)
• List all owned datasets (in standard manner)
  • Will need change to logic layer to show datasets on user dict (this is useful for the API as well ...)
• (Re)move list of changes
  • Should have activity (once activity available)

WONTFIX:

• ?? Allow users to provide a home page, organization (requires migration)

Richard Cyganiak: I often see a problem where half the system thinks that I'm logged in, while the other half insists I'm not. Logout then login again may fix it.

David Read: I think I've seen it several times that the "My Account" link is given (suggesting you're logged in) but when you click it it asks you to log in, which means you weren't.

[Child of #1526]

Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa.

Super ticket: #1602

Steps:

• Include (or git submodule) data explorer into vendor
• Update existing js integration code
• Fix css and any bugs

Est: 1d

Actual: 1d

This CSV resource used to preview but now the format appears to be unsupported: "We are unable to preview this type of resource: x-osdata-csv"

​http://thedatahub.org/dataset/lbhf-spending-2010/resource/9661abbd-2816-4d58-8b20-3cb0eb770c69

This is used as an example by the OpenSpending? team all the time.

...and show them on the tag pages.

This means adding tag_activity_list() and tag_activity_list_html() logic functions, adding the templating to render tags as HTML, and adding the tag HTML into the tag pages.

What exactly should go into a tag's activity stream? As with datasets we may want a more complex SQL query, where events from objects related to the tag (datasets, users, etc.) appear in the tag's stream.

Super #1669

Authz for normal users in publisher profile as per sysadmin

Super #1669

Test all of the new auth logic hooks for publisher profile to make sure they work as expected. Also make sure that we have a test to compare the naming of the core auth funcs with the profile hooks (so we can see what may have changed in one but not the other).

The activity streams feature is already thoroughly tested via the logic layer and API calls, but there needs to be a quick test of the rendering code, e.g. fetch a user's page (thus testing that ckan doesn't crash while rendering it) and then assert that the expected activity streams HTML is in the fetched HTML.

Need to setup server in tenforce test server.

Make sure accented characters are normalized when indexed and when searched for.

Eurostat datasets mainly.

From Mark on the ckan-dev list:

In the table of broken links at <​http://thedatahub.org/qa/dataset/broken_resource_links>, the links as printed should be abridged to a fixed length (but link to the full URL). Otherwise the 'reason' column vanishes off the screen. Also a table would probably be better since as things stand the 'reasons' are not lined up

We want to make email announcements to users of thedatahub.org.

The most simple way to do this is to get a dump of the email addresses and subscribe/invite them to a mailman list.

Different ways to do it:

• 'paster dump-users-csv' creates a CSV file. Apache serves up the directory using password protection. Is this secure enough?
• CKAN admin pages provides a button to produce the CSV and download it. A config option may be necessary to enable this (disabled by default).

Something more advanced in the future could be:

• CKAN tells mailman about new subscribers using the mailman web interface. Could be brittle.
• Mailman doesn't have an Internet API, but it does have a python interface. So CKAN could tell mailman about new subscribers using a python proxy. There may be one out there, or maybe easy to do. But annoying to have to maintain another process running on machine.
• CKAN sends out the emails itself. A sysadmin is given a form. Users have an opt in/out option in their account. But receiving emails is a sensitive thing so it would need to be right first time, and there are several risks to this approach.

Search results page of DGU.

Breakdown:

[X] ensure faceting by group (publisher) works [X] allow faceting by UKLP dataset type [X] allow faceting by INSPIRE

To allow navigation between sections to still work, also potentially rephrase to avoid confusion that you have to save every time to add a tag or resource and can't say add description, tag and resource in one go

If you use tag autocompletion, it automatically adds a comma after the tag. This is fine if you are going to add another tag, but not fine if it is the last one. If you don't manually remove the final comma than you get form validation error "Tag string: Tag "" length is less than minimum 2"

Affects only version 1.5.2a.

Right now is quite difficult to get the most current version of a package via the harvest objects, and the queries involved are prone to errors. A new "current" field will be added which will identify unequivocally the most recent harvested object. Unfortunately this will involve some major changes:

• Model update (See #1726)
• Update queries for getting harvest sources' datasets
• Update ckanext-csw queries for getting the objects
• Update inspire and ckan harvesters
• Document new field

Test failures in master due to tom and rufus not running tests. 0.3d.

if you want your ckan to reside not at ​http://yourhost.org/ but rather e.g. at ​http://yourhost.org/ckan/ you can use this hint ​http://wiki.pylonshq.com/display/pylonsdocs/Configuration+Files and set ckan.site_url accordingly in your .ini. However, there are some instances where internal links are created without the required prefix.

An API call that returns a user's public activity stream in JSON format.

Various UX and UI improvements to groups.

Etherpad: ​http://ckan.okfnpad.org/feature-1521-groups-ux

Improved groups

From Richard Cyganiak.

This is a set of proposals to make the groups feature more powerful. As a result, currently separate CKAN instances such as ​http://ie.ckan.net/ could be turned into a simple group on the ​Data Hub without losing their separate identity.

• Group homepages: Make the group pages more attractive and welcoming as homepages. The main Data Hub homepage does a great job at this now – can the group pages be made more like that? They're quite uninformative and bland right now.
• Subdomains for groups: Enable subdomains such as ie.thedatahub.org as aliases for thedatahub.org/group/ireland
• Search within the group: Have a big search box that searches just within the group on the group page, with a look like on ​http://thedatahub.org/dataset
• Have an Add a dataset to this group link on the group page, perhaps in the sidebar.
• Navigation within the group: Have some more navigation starting points on the group page, e.g., the Tags + Resource Format sidebar from ​http://thedatahub.org/dataset
• Group documentation: Add the ability to add more “About this group” text, or pages like “FAQ” and “Extra field conventions”.
  • The only place where one can currently put group-specific text is before the dataset list. Putting much text there is bad as it moves the datasets below the fold.
  • Can the top-level sections of a group's description be automatically turned into separate pages that are linked from the sidebar? With headlines About, Frequently Asked Questions and How to get involved, we'd then get three links in the sidebar that point to datahub.org/group/ireland/about, datahub.org/group/ireland/frequently-asked-questions, datahub.org/group/ireland/how-to-get-involved
    • RP: WONTFIX this is being considered for future (generally useful) but cost/benefit here was low particularly as it wasn't clear what optimal implementation was.

Additional

• Display members of a group as well as admins (if less than 10? If more than 10 display count only ...?)
• Better group index page (sort by number of dataset, allow searching ...) - #1403
  • Add search to groups: ​http://thedatahub.org/group
  • Way to filter by activity, number of datasets
• #1597 Tag search, filtered by group
  • RP: not worth it in terms of benefit/cost IMO

Steps

• "Add Dataset" button on action bar
  • Make it a dropdown? -> New Dataset / Existing Dataset
  • Redirect Add Existing -> Edit
  • Check permissions
• Expandable "about this group" box
• Implement simplest search-this-group box
  • turn the list of datasets into a dynamic list of search results
    • pagination
  • Matching set of facets in the sidebar (matches 'real' search page)
    • Deleteable, displayed facets

Nice-To-Have

• Get rid of the old group_package_show action (confirm with team)
• Tabbed "about this group" box (extract headers?)
• Scrollable group members widget in the sidebar

To enable us to show DGU work in progress, for feedback.

Last week I imported ca.ckan.net datasets into thedatahub.org, but the tags seem to have lost their dashes, underscores and dots.

I want to browse tags, but filtered for a particular group. Currently our tag API doesn't allow for filtering by group.

This is important for improving groups as communities within a site #1521. It would be easy to do this by adding an option to filter by a group. BUT are there any other use cases that would warrant a more complete faceted tag search?

--

BTW I can currently draw a tag cloud for a group - I can get the top tags used in a group like this:

curl ​http://thedatahub.org/api/action/package_search -d '{"q":"groups:country-ca", "facet.field":"tags", "rows":"0"}'

but it only contains the top 20 tags.

From super Storage changes - #1574 - and ​http://ckan.okfnpad.org/newstorage we determined that ckanext-archiver should have a celery task for grabbing local file uploads and writing to webstore

Analysis

When I upload a file to CKAN:

• End up with file in permanent storage
• IF file is ot type ... csv,xls,xlsx,sqlite,.sql
  • End up with new db in webstore
    • Where? {username}/{resource-id}/...
      • If a single table: name it after the file name (appropriately slugified)
    • A resource *always* corresponds to a 'database' in webstore ...
    • In Data Explorer have "Sheets" tab ...
• Resource url = /dataset/{x}/resource/{y}/link -> cached_url ...

• Update CKAN on DGU UAT to 1.5.1
• Deploy Celery
• Deploy QA extension

Futher Information -> Further Information

thedatahub.org/about contains info that is very general to the CKAN and really quite technical. The text should be changed to be both specific to thedatahub.org and provide the context in a non-technical way. It should be easy to customise the About page to be appropriate for say new-york.ckan.net - a bit of info about who runs it, plus the general stuff about CKAN powering it and it was written by OKF to further open data.