{22} Trac tickets (2647 matches)

Dismax query across languages. Titles and selected languages, weighted more highly.

• Boot viewer for format/mimetype: application/pdf, application/x-pdf, application/acrobat, applications/vnd.pdf, text/pdf, text/x-pdf
• propose we use pdf.js - ​https://github.com/mozilla/pdf.js
• We could also try iframing for browsers with native support but this seems more complex

There is a disrepency between what we put in and get out when using a custom form/api. Narrow this gap by making sure package_show shows the correct serilization.

Need google spreadsheet app to work with the new datastore. This should simplify it a lot.

See ​http://thedatahub.org/dataset/ecb13156-cf11-4e91-b51c-f287d154276e/resource/5db67915-40cb-467e-9d72-194ea4973670

This was marked as json format (and is JSON) but appears to have been imported as a quasi-CSV. Either this hsould not be imported at all or it should be imported as JSON ... (which is straightforward to do).

Assigning to David for triage and review.

• Add Ascii folding filter to text fields.
• Add capacity field for public, private access.
• Add title_string so you can sort alphabetically on title.
• Fields related to analytics, access and view counts.
• Add data_dict field for the whole package_dict.

Actual implementation of data_dict adding will be added later.

Placeholder for thinking about this ...

• Allow users to configure info for CSV import (e.g. separators, quotes etc)

Home page loads *all* groups and dictizes them, it should only dictize the top 10 or so from the search facets.

On registering a new account, if you try to use a capital letter in your 'Login', you get the unhelpful error message:

"The form contains invalid entries: Name: Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_"

What does 'Name' or 'Url' have to do with anything? It means 'Login must be ...'

But even better would be to make the login case-insensitive and allow capitals to be entered.

If a user selects a language there are no mo files for then an error is raised. Revert to default language instead.

Requirements

We need to be able to easily determine who the owner of a dataset or group is. Datasets and Groups should have an Owner, who may change over time but is a specific user within the CKAN instance. It should be easy for CKAN components to determine the user and for the initial version we should ignore the can of worms labelled 'ownership transfer'.

At this point migration is likely to be the biggest issue, and would suggest that it is acceptable that the last user to edit a dataset be set as the current owner.

More tickets should arise as a result of this work where we may be able to optimise some queries to use the new feature.

Interface

None

User Stories

None

Tasks

[ ] Analysis/Clarification?

[ ] Tests

[ ] Migration

[ ] Code/Schema? changes

[ ] Documentation

Estimates

we want to be able to add new datasets but not have them indexed or seem by people

is the vdm.state the place to do this

Not loading for all resources as far as I can tell; e.g. ​http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360

Fine on demo.ckan.org

which is confusing since you can only search for one tag like this at a time. I.e. tags:economics,cvs or tags:economics, csv or tags:economics+CSV doesn't work for example; therefore tag:economics, should also work!

​http://s031.okserver.org:2375/dataset?q=tags%3Aeconomics&sort=relevance+asc

In progress

1. [major] q does not seem to work reliably. e.g. using the setup from this gist ​https://gist.github.com/1930806 and doing a ?q=DE yields no results (does not work with "q=de" either)
   • q=second does work ...
2. [major] q does not work with 2 values (see below)
3. Query on search with limit 0 results in total of 0 (should either be null or correct total). Queries with other limits yield correct total AFAICT
   • Also weird fact that limit is returned but as as as string - should it not be an integer
4. Types on fields: could these not be canonical and as per recline (or is it important to allow exact sql types ...)

Multiple query values

Try a query such as: "second UK" and you will get 500 error:

​http://localhost:5000/api/3/action/datastore_search?resource_id=4f1299ab-a100-4e5f-ba81-e6d234a2f3bd&q=second%20UK

ProgrammingError: (ProgrammingError) syntax error in tsquery: "second UK" 'select "_id", "id", "date", "x", "y", "z", "country", "title", "lat", "lon", count(*) over() as "_full_count"\n from "4f1299ab-a100-4e5f-ba81-e6d234a2f3bd" where _full_text @@ to_tsquery(%s) limit 100 offset 0' (u'second UK',)

Suggestions

Filter support: should think in more detail about this (may want to follow recline style)

Simple filters in query parameters would be nice too ...

change mm support solr 3 and 4 add *_date field

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments

Implementation:

​http://s031.okserver.org:2375/user

If you go to http://ckan.org/features/geospatial/ there is a problem with the pop up tag shown (and zoomed in) - missing status and description.

can we promote this to phase 3?

Can we do this as a localisation?

Includes: missing fields, existing field checks (i.e. whether a name/dataset already exists with that name) during input (i.e. no need to submit form to check)

e.g. on ​http://s031.okserver.org:2375/en/dataset/afghanistan-election-data/resource_edit/f6331f99-51f6-44d9-95b9-b20f3b74f360

& ensure consistant with sidebar of edit dataset page

see ​http://demo.ckan.org/dataset/vif-2012-population-totals/resource/e7b8401a-c410-4902-9e5a-6dbd78834873

what XLS previews do we support?

​http://localhost:5000/ckan-admin/authz

you need to be an admin

paster sysadmin add <username> -c dev.ini

like #2741 we need other admin links

you may want to make the inputs use your form macros - I've just done a basic port

also we will need to get all the tables done but looks like we need to get auth groups working for that

e.g. currently no way to read full description for the 'data explorer examples' group here:http://s031.okserver.org:2375/group/data-explorer

Can we add a 'read more' link that expands and contracts the sidebar box or similar?

The user login / avatar with in the header really should be within another topbar, as it goes against convention to have the user information within the normal header.

Organization administrators can delete themselves from the user management pages. We should disable this on the front-end as well as the back-end.

We need a nice way to be able to get to editing the different resources that are associated to a dataset.

From the google doc: When I move between grid, graph and map view mode (or click on filters) the whole data explorer flickers (jumps up & down slightly) for a few moments before settling. This is happening on all resources for me and when you initially open launch the page: ​http://demo.ckan.org/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360. Only in Chrome, working fine in Firefox.

Just a ticket to keep track of a few suggested template changes.

Essentially, the functionality should be as follows:

• Add logout link that has .js-hide attached to it within the header that isn't hidden within a dropdown

See ​http://plus.google.com/ (when logged in) with and without JS to see an example of the actual sign-out working without JS

The coverage tool (run by buildbot in the ckan build) reports that only 24% of lines of ckan.controllers.authorization_group are run in tests and 38% of ckan.forms.authorization_group. This suggests a need for more tests.

create a new test file ckan/model/test_user.py

add tests for the following three functions in ckan/model/user.py

number_of_edits, number_administered_packages, search

merged in in changeset 0046f83aedcf

Purging Revisions

• Delete button displayed on:
  • /revision/list
  • (/package/history)
    • /package/history is problematic because html does not allow nested forms and we already have form for doing diff/comparison.
  • /revision/{id}
• Delete button submits to delete action on revision and changes revision state to 'deleted'.
  • undelete button now displayed and revisions are marked as deleted in some way (e.g. greyed out?)
• Sysadmins then visit /ckan-admin/trash which lists all revisions with deleted state. There is a large button: "Empty trash" (irreversible). Click button purges all revisions with deleted state.

Purging Packages

• Put into deleted state.
• Listed on /ckan-admin/trash
• Separate Empty trash button which deletes all associated revisions.
  • Should be separate from Empty trash for revisions

Current system

• Single purge link on revision listing if a sysadmin which permanently purges the revision and all associated changes (without confirmation atm!)

the add_user_to_role/remove_user_from_role functions are asymmetrical in that the add function is happy to add the same role twice but the remove asserts that it's only in the table once and crashes if that's not true.

an attempt has been made to guard against this, but fails, I think because the add functions rely on the caller committing the change to the db.

same problem affects corresponding authorization_group functions

I'll try to sort this out. Making a note here.

The CKAN instance on data.gov.uk serves invalid URIs out of its API.

For example the following can be found,

​http://uk.sitestat.com/homeoffice/rds/s?rds.hosb0509tabsxls&ns_type=pdf&ns_url=[http://www.homeoffice.gov.uk/rds/pdfs09/hosb0509tabs.xls]

In this URI, the : and / characters after the ? in the query part are invalid according to section 3.4 of RFC2396

Also URIs are not stripped of whitespace at the end.

This causes problems when other software with a more correct interpretation of what a valid URI is attempts to consume data from CKAN. In this instance the Talis triplestore complains about such URIs.

"Be liberal in what you accept and conservative in what you send" would seem apt.

Actions

• Validation of urls as part of form entry or data loading
  • Need to consider situation where this should happen out-of-band (i.e. we allow load even with invalid data and then flag bad dates in separate validation process). In general doubtful that we should do this here because url invalidity is such a big deal
• This code should support analysis of existing data so we can go through existing database and find invalid urls
  • Also useful to have this so we can do out of band validation

An extension that provides a todo list feature on CKAN so that people can register and find things to do.

Extension name: ckanext-todo

User Story

Package page

As a user I come to a package:

• Have a todo count at that top that takes you down to the todo list (which may say nothing todo)
• At the bottom is a section of the package display titled "ToDo?" where I see a list of all toDos for the package most recent at the top
  • If I am logged in
    • See a form for "Add to do" at the top of the todo section and can add one straight away
    • I see a "now resolved" button next to each which goes green when you hover.

When clicked the todo fades away.

• Not logged in: I see a button that says "login to add todo"

Todo list page

When a user comes to todo overview at /todo

At top list all todo categories with counts (or a progress bar). Click on category name or bar takes you down page to list for that category.

Category list has a list of todo items (ul with li items with class todo) - link to package relevant to the todo.

Implementation

The Todo form

• One of the fields is category -> autocomplete the category (not constrained) (lowercase, no spaces, .-_ allowed)
• Add a description
• Submit, the todo gets added via AJAX to the list at the top as the most recent todo

Model:

todo table

  id (autoincrement integer)
  package_id
  todo_category_id (required)
  description (required)
  created=NOW()
  resolved=null (unresolved) or a datetime (datetime of resolution)
  creator=user
  resolver=user

todo_category table
  id
  name

Prepopulate with: broken-resource-link, no-author, bad-format, add-description

API at /api/2/todo

• GET / POST / PUT ...

/api/2/todo?package=package_id_or_name&category=...&resolved=0/1

• support limit (?)

/api/2/todo/category -> return list of todo categories

• No GET / PUT / POST (these are auto-created by creation of todo)

Optional Extras (Will not be done atm)

• Integrate todo tags (e.g. list packages tagged with a todo.{xxx} on Todo List page ...

Superticket: #1397

Write a worker daemon to download all resources from a CKAN instance to a local repository.

Questions

• Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues)
• Should we have clever ways to dump APIs? ANS: no.
• Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving)
• Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache.
  • Complete support for ETags
  • Expires, Max-Age etc.
• Check

Functionality

• Download files via HTTP, HTTPS (will not do FTP)

Process:

1. [Archiver.Update checks queue (automated as part of celery)]
2. Open url and get any info from resource on cache / content-length etc
   1. If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery
   2. Check headers for content-length and content-type ...
      • IF: content-length > max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?)
      • ELSE: check content-type.
        • IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource)
        • ELSE: archive it (compute md5 hash etc)
      • IF: get content-length and content-length unchanged GOTO step 4
3. Archive it: connect to storage system and store it. Bucket: from config, Key: /archive/{timestamp}/{resourceid}/filename.ext
   • Add cache url to resource and updated date
   • Add other relevant info to resource such as md5, content-type etc
4. Update task_status

Optional functionality

• If result object is HTML, search for references to "proper data" (CSV download pages etc.)
• Download from POST forms (accepting licenses or weird proprietary systems)
• Support running on Google Apps Engine to save traffic costs.

Existing work

• ​https://bitbucket.org/okfn/ckanext-qa/overview
• out of date: ​https://bitbucket.org/pudo/ckanextarchive - Old archiver extension, largely experimental.
• out of date: ​https://bitbucket.org/ollyc/ckan/changeset/1b16fbe9aa65 - Openness scores by ollyc

As a (logged-in) User I want to watch (follow) a package, that is register my interest about a package. (Similar to watch/follow features in github/bitbucket/wikis).

NB: this is as much (if not more) about showing what packages are interesting to people as giving info to 'watchers'.

Need to finalize terminology (github uses watch for repos and follow for users while bitbucket combines both in 'followers'). Decision: use follow

Implementation

Interface

Become a follower:

• Follow button on packages (if already watching say 'unfollow')

Package-related changes:

• Show number of followers on a package
• List followers of a package at /package/{name}/followers
  • On a separate page

a package User-related changes:

• List followed packages
  • Either on user's page on a separate 'following' page. (NB: called 'following')
• Does watching involve notifications (by email)
  • Probably not: you can already subscribe to RSS feed after all and email not that necessary (?)
• [Future - don't have activity stream yet] Show what packages a user has started/stopped followed on a user's public activity stream on their user page

Nitty-Gritty

• Want to do this in ajax-y manner
• API endpoint: /api/2/follower
• Store data in a new follower table

API

/api/2/follower

follow => PUT / POST
{
   user_id
   object_type
   object_id
}

If this is submitted by a user with user.id != user_id => error (401)

unfollow => DELETE

/api/2/follower/package/{id}
=> list of followers
[
    { safe dictized user }
]

NB: depends on access to a 'safe' dictized user object. Dictization is in nearly done, and current example of doing this by hand is in user API autocomplete method.

Table

Called 'follower'

user_id, table, object_id, created
xxx, package, yyy, ...
xxx, user, yyy, ... [future]

Random Extras

• What about following users as well

Characters outside of ASCII range are not supported within data previews.

Steps to reproduce:

1. Visit ​http://ckan.net/package/kivele2010
2. Click on [preview] for any of the resources

The "Preview" button is a nice idea, but it doesn't seem to actually "preview" anything if the file MIME type would ordinarily cause the browser to download the file. If so, the browser does indeed just download the file.

This is notable in the context of most hosted file services (including Google Storage) which will deliberately serve a MIME type of application/x-some-junk-here in order to force a download.

The default package form offers 4 empty extras fields. Like the resource section, it should have an "add more" button to add another row.

Copy ckan core postgres package search tests to the ckanext-solr extension and update them so that they use the solr search backend.

For example:

• name and title should have a higher weight than the notes field

Resource and Tag search is currently still using the old ckan query parser which is unnecessary. Refactor this out and move to the logic layer.

If there is a problem with the SOLR connection,the home page just shows a 500 Server error, instead of a notice (the package controller shows a notice when visiting ​http://localhost:5000/dataset).

Steps to reproduce:

• Change the solr_url parameter in the ini file to something non-existant
• Visit ​http://localhost:5000

table: task_status

id
entity_id
entity_type
task_type
key
value
last_updated
state

unique(entity_id, task_type, key)

example keys:
last_task_id