{22} Trac tickets (2647 matches)

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61752357/comments

Implementation:

​http://s031.okserver.org:2375/dataset/adur_district_spending

Implementation:

​http://s031.okserver.org:2375/dataset/edit/5b451cab-8671-4513-91c4-257784c1f484

Should have all supported extensions enabled, those that don't work should be disabled and ticketed. Also needs a customised homepage.

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61911712/comments

Implementation:

​http://s031.okserver.org:2375/group/test-group

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61696765/comments

Implementation:

​http://s031.okserver.org:2375

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998362/comments

Implementation:

​http://s031.okserver.org:2375/en/user/register

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments

Implementation:

​http://s031.okserver.org:2375/dataset/

This includes the tag page as well for now.

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62998445/comments

Implementation:

​http://s031.okserver.org:2375/en/tag

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/62268474/comments

Implementation:

​http://s031.okserver.org:2375/user/aron

Because CKAN's worth it!

As designed by Sam Smith -> ​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/ Also to be used as the demo theme although I think they will diverge at some point. At the moment they should be implemented as one and the same.

Object properties:

• id (uuid)
• apikey (uuid) -- migrated from the apikey table
• name (username = openid)

/account is being renamed to /user in the wui.

Related to: ticket:90 Link to RDF version of CKAN data and ticket:163 (regularly convert CKAN data to RDF).

Convert ckan data to RDF with suitable ontology and put on semantic.ckan.net.

This occurs when CKAN is run with mod_wsgi (not under paster).

This is because of a unicode header being creating in rest.py.

'created' attribute on User, Group and Rating

datetime object: default=datetime.datetime.now()

See example on revision

Requires migrate script

Importer function is exposed to logged in users.

Alongside specifying file to import, user chooses between 'form schemas' - 'basic' and 'government'.

• Basic is as it is currently
• Government form has pre-defined Extra fields, careful validation of lots of fields, code formats/processes some field data for storage. e.g. searching for tag keywords.

All fields should be in step with the government custom input form.

For fields which have suggested values but the user can input his own value, this is achieved in the spreadsheet with two columns - the first being a drop-down and the second free text.

Estimate:

Spec - see attached, which is a copy of the Pirate Pad from Lauren: ​http://piratepad.net/2C2iwiLDhd

Write an importer that supports most well known variants of DCat in importing the data as CKAN packages. In particular, the following sources should be supported:

• CKANrdf generated exports
• opengov.se RDF (not really DCat)
• Sunlight Nationdatacatalog as harvested by dcat-tools.

The publisher index page should show an alpha paged list of publishers.

Only sysadmin can access create publisher form.

On publisher view page - hide administrators from public view - only logged in admins of group should see them.

A new form for create publisher -> Add publisher access to your account' New wizard Autocomplete publisher name if linked from publisher view. Email address used to publish should be activated/validated on first use. Original email still used to log in.

A collection of improvements to the dataset creation/edition form (feedback form DGU).

• [x] validation error formatting
• [x] geograohic coverage: add local authority
• [x] resource format field
• [x] resource format auto-complete
• [x] renamed data tab -> files tab
• [x] integrate Adris'a INSPIRE changeset, and migration scripts as part of build
• [x] remove "discontinued" from the update frequency
• [x] disable the disabling of save
• [x] Remove ability to edit url on the edit-form (and remove "url not available")

Dataset read page for DGU

Breakdown:

• [X] Fix missing title
• [X] Links to the social media stuff
  • "Share your app"
  • "Share your idea"
  • "Request new data"
• [ ] Presentation of additional information should be broken into sections

Content is there. Don't do anything until theming/design work is undertaken.

• [ ] "More like this": links to similar datasets.

Dropping

• [N] Stars (I think these are being removed)
• [ ] Comments

Possible integration point with Drupal. Needs it own ticket.

• [X] Give feedback to department
• [X] Tidy up the QA stars

So that Drupal doesn't need to make any adjustments to the form, and risk data being lost on submission.

• [x] hide black stars on the data-view page
• [x] Edit on view form (if you have the right permission)
• [x] RES_FORMAT ~> Resource Format
• [x] Map Based Search -> UK Location Map Based Search
• [x] Note at the top right expalining that these are seach filters (if they are) -> this requires "add a dataset" and "other access" to be moved to a different place (just some whitespace is fine)
• [x] Display publisher and provider (UKLP records only).
• [x] analysis of filter by OGL / non-OGL
• [x] provide /dump on the build server

Sort out how we deal with dashes versus underscores in package names.

Super ticket: #1602 (Data Previewer v2)

We already have first pass of Data Explorer that was released as part of #1357.

• Merging ​https://github.com/okfn/dataexplorer with ​http://github.com/okfn/recline and developing it
  • ​https://github.com/okfn/recline/issues

Tickets include (* indicates improvement over current explorer)

• Core Backbone Models representing Dataset and Tabular data: ​https://github.com/okfn/recline/issues/10
• New theme: ​https://github.com/okfn/recline/issues/22
• Read-only mode: ​https://github.com/okfn/recline/issues/17
• Introduce hash navigation / state support (*): ​https://github.com/okfn/recline/issues/19
• Re-enable editing in DataTable? (*): ​https://github.com/okfn/recline/issues/13
• [super] DataTable? view (in Backbone) (*): ​https://github.com/okfn/recline/issues/14
• DataExplorer? parent view (*): ​https://github.com/okfn/recline/issues/12
• Simple graph widget using flot: ​https://github.com/okfn/recline/issues/11

Est: 10-15d (should be broken down -- partly is in recline issues)

• Boot viewer for format/mimetype: application/pdf, application/x-pdf, application/acrobat, applications/vnd.pdf, text/pdf, text/x-pdf
• propose we use pdf.js - ​https://github.com/mozilla/pdf.js
• We could also try iframing for browsers with native support but this seems more complex

e.g. ONS importer

The preview of "Species Misc Turtle Download" at ​http://ckan.net/package/taxonconcept results in the following error:

Unable to Preview - Had an error from dataproxy: Data Transformation Error (Data transformation failed. Reason: 'utf8' codec can't decode byte 0x8b in position 1: unexpected code byte

As a user I want to preview a dataset ('resource') before downloading it.

Implementation

This will be implemented as an Extension

NB: This ticket requires a way to get data out of a remote url (the dataset). We do not want to put this data load/streaming into ckan core due to a) complexity b) load problems (do not want to take ckan out by accidentally pulling in a 1GB file)

1. Need a 'data proxy' - see ticket:698 and ticket:888
2. Preview using javascript in WUI - do this in an extension ​http://bitbucket.org/okfn/ckanext-dataapi

• get up to speed with DataExplorer? code (and dependencies like Backbone)

Simple widgets in pure JS. For example:

• Count of datasets in a group (could generalise to a query but not sure how useful that is ...)
• Embeddable list of top X (5) datasets for a given query
• Embeddable list of *my* datasets

Either these live at: {site}/widgets and we have some kind of generator (form where I choose my group, or my query).

Or: we have this attached to areas of site where relevant.

Can combine the 2 so that the latter links to the former. Think first will be easier to do and possibly more useful long-term (e.g. can just link people to that page).

Cf. ​http://okfnlabs.org/ckanjs/

Major UX gotcha reported by several people is that DataStore? is disabled by default and must be enabled for DataStorer? to run and import data.

Suggest switch to enable datastore by default for file and file.upload types (disable for API). People can always disable if they want.

Try and put datapreview in Iframe.

When given a url parameter for a file it cannot fetch, the app raises a 500 instead of something useful.

Better error reporting in the JSON would be useful

From Jonathan gray:

At the moment to delete a package you have to go to 'state' and change from 'active' to 'deleted'.

For me this was not very obvious, and someone recently wrote to me about this too.

Its a small issue - but unless you know what 'state: active' refers to, its not obvious what this means, nor that this is where to go if you want to delete a package.

From UI perspective this could be replaced by either a trash can or some text that says 'delete', perhaps with some prompt that explains what happens when you delete a package - explaining that non-admins won't be able to see it, and how you purge, etc.

At the moment if you have an related item with an image associated to a dataset that appears in the side column it's resized but doesn't keep it's original aspect ratio.

On each page of package search results, all the items are neatly sorted apart from the last item of the page. SOLR gets the sorting of the results incorrect.

This is a known issue: ​https://issues.apache.org/jira/browse/SOLR-1777 affecting SOLR 1.4 only (which comes with Ubuntu 10.04)

It is highlighted in CKAN test ckan/tests/functional/test_pagination.py:TestPaginationPackage of commit 39096ed54bda86d043521b08b2e14fc5e283a0ff which fails most of the time it is run (passes intermittently).

The Package model should have a boolean field added to it describing its visibility beyond that supplied via auth.

The related permission for viewing packages where public is set to False would be package.view (show, whatever), and *never* visible for unauthenticated users.

Where does the data go?

In a postgres database configured by the ckan.datastore_write_url config option which is a sqlalchemy url.

The user should have rights to create tables.

Whats the api like?

We will just implement it as logic functions like the rest of CKAN and will part of core. After that we may add some nicer api functions that use these but that is a secondary concern.

What are the initial logic functions?

• datastore_create
• datastore_delete
• datastore_show

What is the JSON input format for datastore_create

To begin with it can have the following keys. It is fairly consistent with Max Ogdens' gut servers. Except adds resource_id.

{
resource_id: resource_id # the data is going to be stored against.
fields: a list of dictionaries of fields/columns and their extra metadata.
records: a list of dictionaries of the data eg  [{"dob": "2005", "some_stuff": ['a', b']}, ..]
}

• The first row will be used to guess types not in the fields and the guessed types will be added to the headers permanently. Consecutive rows have to conform to the field definitions.
• rows: can be empty so that you can just set the fields
• fields are optional but needed if you want to do type hinting or add extra information for certain columns or to explicitly define ordering.

eg: [{"id": "dob", "type": "timestamp" }, {"id": "some_stuff", "type": "text"}, ...]. A header items values can not be changed after it has been defined nor can the ordering of them be changed. They can be extended though.

• Any error results in total failure!! For now pass back the actual error.
• Should be transactional

What json does datastore_delete take?

{
resource_id: resource_id # the data is going to be deleted.
filters: dictionary of matching conditions to delete
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "delete from table where key1 = 'a' and key2 = 'b' ".
    No filters (either not present or not defined) then delete the table. If we want truncate then add truncate: true to truncate the table.
}

What json does datastore_search take?

{
resource_id: resource_id # the data is going to be selected.
filters : dictionary of matching conditions to select
    e.g  {'key1': 'a. 'key2': 'b'}  this will be equivalent to "select * from table where key1 = 'a' and key2 = 'b' "
q: full text query
limit: limit the amount of rows to size default 100
offset: offset the amount of rows
fields:  list of fields return in that order, defaults (empty or not present) to all fields in fields order.
sort: comma separated field names with ordering e.g "fieldname1, fieldname2 desc"
}

Some free code: ​https://gist.github.com/3163864

What json does datastore_search return?

{
fields: same type as datastore_create accepts (i.e. with metadata)
offset: The same offset that was supplied in datastore_show
limit: The original limit
filters: The filters that were applied in data_show
total: # total matching records without size or offset
records: [same as data_create] # list of matching results
}

On error will return:

{
__error__: … sql error …
}

What types are allowed?

Aim to support as many postgres/postgis types that have string representations.

​http://www.postgresql.org/docs/9.1/static/datatype.html

​http://www.postgresql.org/docs/9.1/static/sql-createdomain.html

IDs

Each row in a table will be given an _id column which has an id generated by us which you can use in queries.

Other Features

Each row will store the _full_text index of all the data in the row. At some later point there will most likely be a way to index fields add constraints etc.

This needs to be thought out ...

The current oAuth extension tightly couples repoze.who, ckan, and data.no. These components should be decoupled; they also need more tests

Currently the default authz for a package is hard-coded to:

 <PackageRole user="visitor" role="editor" context="Package">,
 <PackageRole user="logged_in" role="editor" context="Package">,

This should be configurable in the config, so that you can have a more locked down instance etc.