{22} Trac tickets (2647 matches)

changing a tag from "country-uk" to "country:uk" and updating gives server error

otherwise updating metadata / adding tags seem fine

either we should allow colons or form should return a message indicating what characters are allowed

cf. Tariqs mail

Updated version: ​https://spreadsheets5.google.com/ccc?key=tuOtQjD0Psoqr1pWTS8EXZQ&hl=en#gid=0

Original Version

Publishing Entity:
Publishing Entity Type: (Donor, Recipient, Community Data..)
Donor
Country
Activity period:
Verification status: enumeration of statuses (checked, not checked etc)
Resource links: to the actual IATI record
Number of activities: ...
Date record updated:
Date data updated:
License: Need this field even if it may be a standard license

So naively mashing these together, we get something like:

Identifier
Title
Abstract
Donor
Country
Publisher
Publisher Type
Verification Status
Department
Contact
Contact e-mail
Licence
Resource format
Resource URL
Resource ID
Activity period
Number of activities
Date record updated
Date data updated

can we get the link going to an explanation of the 5 star system on page ​http://s031.okserver.org:2375/qa/dataset/five_stars to open in new tab? it's leaving demo all together so better to open in new tab than navigate away

be good to do this in phase 3

attempt to disengage the stats plugin from core as much as possible

as title.

as per http://ckan.org/wiki/UIRedesignPackage

apps dashboard available at ​http://localhost:5000/apps in main dev branch

I have got this to work in the demo however

a) it is a bit ugly

b) the template could do with some sorting eg the dropdowns should be being created in a nicer way

allow group-level authorization on all system entities

add to groups is a drop down menu where you can only choose one needs a new UI & logic allowing user to add new groups & potentially remove from other lists

add resource_status_show action function to see the state of celery tasks.

Need to get information from both out task status table and the celery_taskmeta table.

a simple api call that returns data like this:

{ "version": ckan_software_version,
  "contact": { "name": "Some Admin", "mbox": "[email protected]" },
  "description": "Site Description",
  "url": "http://canonical.name.ckan.net/"

[x] The publisher_form extension and the 'publisher' component of the system will be renamed to organizations to both make its functionality clear, and to differentiate it from the publish auth profile.

Ownership of datasets is important and this ticket will enforce membership for a dataset to a single individual organization. The workflow for adding a dataset to a publisher/organization will stay the same but there will be no mechanism for adding existing datasets to an organization from within the organization forms.

[x] New datasets can be created from the organization form, in which case the organization will be set in the dataset form dropdown (with privacy set to private - see below)

[x] Datasets created from direct access to /dataset/new will have a drop down showing all of the organizations available to the current user.

[x] When editing a dataset the organization can only be changed if the current user is the admin of the current organization, and only then to one of the organizations available to the current user.

Only Organization admin can see the edit Organization form.

[x] Privacy will be enforced by changing the dataset form to have a public/private switch next to the organization drop-down which will be set as the capacity with which the dataset is added to an Organization. Private means visible only within the Organization, and should be the default.

[x] Search will be affected as the public/private status of a dataset will need to be added to the schema

[x] Only public datasets showing in site search, and private showing in organization dataset search (if user is within the organization).

[x] Make sure resources are restricted in visibility if the parent dataset is private

[x] Documentation should be updated to reflect these changes.

[x] Verify, then add, then re-add to documentation.

• A User joins or creates an Organization

• If creator of the Organization User becomes administrator of the Organization otherwise they become a Member.

• Creation of new Organization must be approved by a System Administrator

• New Members must be approved by the Organization Administrator

[Refactored] :: Follows on from #1506 UX changes.

• Declutter the sidebar. No yellow box.
• Facets to go on the left, rather than the right. More logical flow.
• Did you know you can search by author? Probably not. Find a nice way of presenting extended search options.
• Make Datasets in the search page look more like Datasets on the groups pages (ie. like awesome sexy search results).
  • Update that look-and-feel to include the new resource icons created in #1506

[Child of #1526]

Create the interface used to retrieve publisher and authz information from the publisher hierarchy.

[Child of #1526]

Can't change a dataset from containing time-series resources to one containing individual resources or visa-versa.

You get an exception if you use the API to search packages and specify a non-ascii character in a field other than q.

For example:

http://catalogue.data.gov.uk/api/search/package?title=%E2%80%93

This "N-dash" (Unicode character 2013) causes this exception:

Module ckan.controllers.api:460 in search
<<                      if ver in u'12':
                               # Otherwise, put all unrecognised ones into the q parameter
                               params = convert_legacy_parameters_to_solr(params)
                           query = query_for(model.Package)
                           results = query.run(params)
>>  params = convert_legacy_parameters_to_solr(params)
Module ckan.lib.search.query:38 in convert_legacy_parameters_to_solr
<<      for search_key in non_solr_params:
               value_obj = legacy_params[search_key]
               value = str(value_obj).replace('+', ' ')
               if search_key == 'all_fields':
                   if value:
>>  value = str(value_obj).replace('+', ' ')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 30: ordinal not in range(128)

This problem affects CKAN 1.5 and 1.5.1 only.

You configure the extension with some RSS and/or Atom feeds, it automatically reads items from these feeds and makes them available in the template context, you write a custom template to e.g. display 'news' items from a Wordpress blog on your front page.

This extension might be simpler and less fragile than ckanext-wordpresser, and also more generally useful.

Details

• Mark Pilgrim's Universal Feed Parser might be useful for reading the feeds

• Feed items should probably be cached somewhere

Suggested

• The news item 'widget' should be wrapped in a known class so that it can be styled easily regardless of the format of any HTML entry.

• For non-HTML formatted items (Atom should tell you the content type of the entry) maybe we should have a template for rendering each item along with any enclosures that it might reference

• Caching is pretty crucial and should probably obey the ttl of the feed.

You can't delete things from the SOLR search index if the ckan.site_id and ckan.site_url are blank.

Should assert that one of these are set up.

You can register a user with password and (optional) email address. But if you forget the password you can't then log in again. We need a password reset facility that sends and email with a new password.

Write an importer that supports most well known variants of DCat in importing the data as CKAN packages. In particular, the following sources should be supported:

• CKANrdf generated exports
• opengov.se RDF (not really DCat)
• Sunlight Nationdatacatalog as harvested by dcat-tools.

Write a worker that scans all packages in a ckan instance and uploads the data to storage.ckan.net or another suitable storage system.

• Naming scheme?
  • Bucket: {ckan-instance-id}-{package-name}? {ckan-instance-id}-{package-id}?
    • What happens if names change
  • File: filename? hash?
• Store hash back on ckan instance?

The caching worker will consumer update notifications and fetch packages.

Extra points for:

• Properly checking for source file modification (Last-modified, Etag)
• Using PIP VCS Backends for retrieval
• OFS/S3 Storage

Write CKAN FAQ (Basis can be: ​http://wiki.ckan.net/FAQ)

Post preliminary questions on:

​http://notebook.okfn.org/

Would be nice if CKAN could work out of the box without the need for SOLR (solr is great but complex and heavyweight to install).

Propose:

• ckan.simple_search config option
• If set:
  • query via simple query to database backend
  • do no specialized indexing

Extras

Remove

• TagSearchQuery? from lib/search (just do a search directly)?
• #1360: filter by downloadable and filter by open search options

Possible Extras

(Probably future improvements)

• Re-introduce full-text search indexing where supported in e.g. postgres and use for querying

With up to date statistics such as:

• total packages
• total tags
• total users
• most active users
• last update
• etc.

Could also be nice to have a more dedicated stats page - with information such as graphs showing rate of packages being added and such like. See ticket:184

With the organizations and organizations_dataset plugins enabled login and create a new dataset, adding it to a group that you are a member of. Edit the dataset and set the group to none. Now you can no longer edit your own dataset. (You can also specify None for the organization when first creating the dataset, same problem.)

With the latest template, css and js changes in 2.0, there are a number of things that need preparation prior to a production deployment. One of these is:

• the fanstatic_resources defined in ckan/lib/fanstatic_resources.py (bottom of module) should be prepared, and minified.

This ticket is to:

• provide a paster command which when run will generate the minified javascript as css files.
• stop the auto-minification of files when CKAN starts up.
• remove reference to the minified files in the .gitignore file.
• add instruction to the release process to run this command, and check-in the minified files to the repo.

## Background

Currently, minification works seamlessly without the need for any preparation when CKAN is started in a development setup. But on a production site, the webserver will (almost certainly) not have write-access to the directories that will contain the minified files. And so the minification will fail, and the site will end up serving the un-minified media, or even *old* minified media.

One way around this would be to allow webserver write access to the directory its serving out of. But this is not generally considered good practice.

Another method would be to distribute the minified files with CKAN. This ticket describes how to do this without causing a lot of noise in the commit history of the repo.

The auto-minifcation occurs when importing ckan/lib/fanstatic_resources.py.

## Changing the process slightly

The reason for moving away from auto-minifying files at start-up, to minifying files when running a paster command is:

• we should be distributing the minified files when we make a release (as we do with translation files). This makes it easier to install CKAN from source.

• if we distribute the minified files, then they need to be checked-in to the repo.

• if they are auto-minified, whenever small changes are made, then this will create a lot of noise in the repo. It will be clearer to have the minified-files generated manually as part of the release process. (Or whever we make a deployment internally).

With the default test data created by

paster db clean paster db init paster create-test-data

going to the front page shows two recently changed packages A Wonderful Story A Novel by Tolstoy

But none of those words "Wonderful", etc produce search hits. In fact as far as I can tell, nothing produces any search hits.

That isn't true on ckan.net, where searching seems to work.

With the 1.5 package release, CKAN instances over-write each other's indexes. Adria has a fix.

I'd like to see a new CKAN 1.5.1 release with the Solr changes merged and the Solr site ID set up as part of the install ckan.site_id = dgu-os.okfn.org. I'd also like the ability to search on ID and to have the ID directly visible on the package page.

We need to decide how to do this upgrade without affecting existing CKAN solr indexes.

With an IDatasetForm plugin active with a form_to_db_package_schema() method that returns a schema based on ckan/logic/schema.py:form_to_db_package_schema() I'm getting validation errors like this:

[ckan.controllers.api] Validation error: "{'type': 'Validation Error', 'junk': [u'The input field junk was not expected.']}"

They seem to be caused when a data dict with key 'tags' is passed to e.g. the package_update API, for example in ckan/tests/functional/api/model/test_vocabulary.py there are various tests that get this error when they call package_update (if you run the tests with a suitable IDatasetForm plugin active).

In ckan/logic/schema.py:form_to_db_package_schema() 'tags' is for some reason deliberately removed from the schema.

Wish to fold service records into package resources. Trouble is that service records don't necessarily refer to a data record, so should this be a new package?

Why?

• We already use quite a bit of boostrap
• We already plan to use bootstrap JS
• Bootstrap is excellent and pretty light

Why is this here? Is it up to date? Is anyone keeping it up to date? Does this belong in some sort of CKAN planning or roadmap document rather than in the documentation?

​http://docs.ckan.org/en/latest/user-stories.html

Whitelist any GETable api actions, and optionally construct the query from url params rather than body.

Whilse using CKAN web interface, you are not tempted to edit stuff:

• You know at all times this CKAN is read-only
• All editing facilities are still seen but greyed-out with an indication why it is.

While integrating the new theme and making a large number of UX tweaks, I've allowed the CSS file to bloat out of control with some very specific rules and subtle hacks, as well as a couple of clearfix <div> objects in the markup which isn't really good practice.

In general we want a logical CSS structure which is as easy to modify as possible. You should not find yourself being overwritten by highly specific rules all the time!

Actions

1. Clean up css
2. Add a config option ckan.template_head_end - arbitrary string inserted at end of <head> tag so you can add custom css etc (may also want to update theming docs to reflect possibility of using this)
3. (While we're at it): Add ckan.template_footer_end config option to deployment.ini_template. (This already exists, we just haven't told anybody)
4. The /users index page isn't linked anywhere. Clean it up and link it in the footer next to Groups etc.
5. package/new, package/edit, group/new and group/edit all include their forms differently. Use h.literal in all cases.
6. Fix the padding on minornavigation.

Background

David Read sent me the following as background:

22. basic theming is described here: ​http://docs.ckan.org/en/latest/theming.html

Deep customisation of the templates is done by creating an extension, with this as a template: ​https://bitbucket.org/okfn/ckanext-exampletheme/overview

That's what Augusto and the Brazilians have done ​http://alpha.dados.gov.br/dados/ and their code is here: hg clone ​http://dev.dados.gov.br/codigo/dev/tema-ckan <snip> Another thing of interest was us becoming compatible with Wordpress themes last December. Richard Pope worked with Rufus on this. I believe the base theme is 'twentyten' and I'm guessing that one would swap that css file with another to change theme, but I don't know. I don't know if this feature has been dropped since then. ​http://lists.okfn.org/pipermail/ckan-dev/2010-December/000073.html

Dave

Where said "v0.4" would be the result of converging four lists mentioned in requirement #711.

When you reset a password, the flash message saying "check your email" sometimes doesn't display on the next page. You then click on the link in your email and it displays on that page (i.e. too late).

Seen this on thedatahub.org. To do with the proxy caching?

When you register a user, it doesn't seem to be logged in properly straight away. Top-right it invites you to "log in" and "register". Also the "create package" menu option doesn't appear. Going to /user/me sends you to login page. If you then login, then all these things are corrected.

When you install ckanext-store (Storage extension) then CKAN displays 'Upload a file' button in the dataset/resource edit tab.

If you are not logged in, then this feature doesn't work, but there is nothing to indicate this. The button should be disabled.

Furthermore, it lets you think you are starting the upload process, the spinner spins, it says 'Uploading...' and stays like that for a long while.

This is a problem CKAN 1.4.3(ish) - 1.5.1 (current version) when you install the current ckanext-storage.

When you edit a package, all of its resources are deleted and a fresh set created. This causes all history of them to be lost.

When you do a search like this:

http://thedatahub.org/api/search/package?q=tauberer+census&all_fields=1

the "extra" fields (e.g. "triples", "shortname") get missed off the results. The docs say it should be a "full record" and I don't see any reason why this is missed off.

This is a problem because search all_fields is the only way for clients and front-ends to get packages in bulk. They end up (like lodcloud) doing thousands of requests to get packages individually.

The full record is:

http://thedatahub.org/api/rest/dataset/2000-us-census-rdf
{"count": 1, "results": [{"res_description": ["Download", "XML Sitemap", "SPARQL enpdoint", "Example (RDF/XML)"], "name": "2000-us-census-rdf", "license": "Non-OKD Compliant::Creative Commons Non-Commercial (Any)", "author": "Joshua Tauberer", "author_email": "http://razor.occams.info/", "ckan_url": "http://thedatahub.org/dataset/2000-us-census-rdf", "notes": "2000 U.S. Census converted into over a billion RDF triples.\n\nPopulation statistics at various geographic levels, from the U.S. as a whole, down through states, counties, sub-counties (roughly, cities and incorporated towns)\n\nNotes: also found in the of SPARQL Endpoints.\n\nFrom home page:\n\n> * For the detailed Census statistics, you'll have to download the raw Census data files from the Census Bureau, my Perl script and the patch file below and run it yourself because the files are too big for me to offer as a download!\n> \n> * The data and scripts can be reused under Creative Commons Attribution-NonCommercial-ShareAlike.\n", "entity_type": "package", "site_id": "www.ckan.net", "download_url": "http://www.rdfabout.com/demo/census/", "indexed_ts": "2011-11-01T12:52:36.034Z", "url": "http://www.rdfabout.com/demo/census/", "state": "active", "title": "2000 U.S. Census in RDF (rdfabout.com)", "groups": ["lod", "lodcloud"], "res_format": ["", "meta/sitemap", "api/sparql", "example/rdf+xml"], "license_id": "cc-nc", "revision_id": "fcbad0de-79ea-41bd-8e01-eb832a05b732", "res_url": ["http://www.rdfabout.com/demo/census/", "http://www.rdfabout.com/sitemap.xml", "http://www.rdfabout.com/sparql", "http://www.rdfabout.com/rdf/usgov/geo/us/ny"], "id": "551ec435-f198-4d52-9b56-ec0b0be6aec9", "tags": ["census", "data", "demographics", "deref-vocab", "format-dc", "format-geonames", "format-politico", "format-rdf", "geographic", "linkeddata", "lod", "lodcloud.nolinks", "no-license-metadata", "no-provenance-metadata", "no-vocab-mappings", "population", "published-by-third-party", "rdf", "statistics", "us"]}]}

When you create or edit a package (via the API), you aren't able to specify the relationships it has. (If you do you get 409 {"__junk": ["The input field __junk was not expected."]} )

The normal way to create relationships is via /api/rest/relationships/ and this works. But when you GET a package, the dictionary lists all relationship details. So this bug creates a problem for editing a package that has relationships - you want to GET it, make any edits and then PUT it back. The work-around is to delete the 'relationships' key from the dict before you PUT it back.

Options

Ideally, CKAN would read the 'relationships' key and make the necessary changes. This is a chunk of work.

Another good option is to allow an unchanged 'relationships' value, but barf it is edited. This is also a chunk of work.

A bad option would be to just ignore the 'relationships' value, since users will get frustrated changing this value and wonder why it never saves, not understanding it is different to all the rest, without error message.

A final option is to get rid of relationships altogether.

When you create a package using the API, you might get a core field name wrong e.g. 'title' mispelt as 'titel' or 'groups' (which can't be specified in package creation). Currently that field will be ignored by the API. It should warn or give an error, rather than give 200 OK.

When you create a package then the 'location' header gets set. This doesn't happen for any other domain objects. I think this should be consistent - either none or all.

I've removed the info about the header in the docs in the meantime.

When you are on the /dataset/new page and you try and change locale then you get a 500 error.

This is because it adds the 'cache' parameter, to ensure any proxy cache in the chain does not just send the cached page.

e.g. ​http://127.0.0.1:5000/dataset/new?__cache=37713707

When viewing a user and a package, the about/notes fields contain Markdown, which may have links. These should have rel="nofollow" to discourage link spam.

When viewing a dataset, the "Resources" navigation button contained the Resource titles on the Resource navigator button, instead of in a drop-down mouse-hover menu.

​http://thedatahub.org/dataset/realtime-birth-data-in-bulgaria/resource/66fc5831-ce01-4954-9beb-e2889ef8a20f

Chrome/Linux?

When vdm was created there was no SessionExtension so we use MapperExtension for doing revisioning. Now that SessionExtension? exists we should use it. We can also follow the existing SQLAlchemy recipe: <​http://www.sqlalchemy.org/docs/orm/examples.html?highlight=versioning#versioned-objects>

When using the API 'locally' i.e. from the CKAN instance (as would be the case with an ajax interface) the API, especially that allowing READ requests should use the normal user credentials if they are available prior to looking for an API key.

The key change appears to be to change _get_user_for_apikey method in lib/base.py BaseController? to check the c.user attribute (may wish to rename as the name may now be a bit misleading ...).

This is critical to incorporating any ajax editing into the frontend.

As part of this ticket we should do a general consolidation of the identification system in lib/base.py so that both api_key and normal user auth lead to the same set of auth-related objects being available (suggest c.user and c.userobj and c.author).