{22} Trac tickets (2647 matches)

Super: #1594

• qa/{username}
• qa/{groupname}
• paginate QA results
• search / filter QA results
• list organisation report by default, but can disable via config option (done)
• UX tidy up of report pages - hide border if no sidebar, etc

Super: #1594

From Richard Cyganiak on the CKAN Discuss list:

Regarding the fifth star (is the dataset linked to others?). This cannot be automatically determined just by looking at the format. It either requires inspection of the actual data, or information about links in the metadata. As you're probably aware, we've established conventions for recording information on data links in CKAN [1], as part of the work of the lodcloud group on the Data Hub. Link information is captured for hundreds of datasets. I would claim that we have the majority of four-star datasets covered there, and hence you can determine if they should get the fifth star by checking for the presence of a links:xxx field.

Superceded by #2234

Super ticket: #1508

Etherpad: ​http://ckan.okfnpad.org/wordpresser

The wordpresser extension does not work with ckan 1.5.1. The wordpress page is not loaded.

Looks like this changeset may be involved ​https://github.com/okfn/ckanext-wordpresser/commit/0efa0649896a7339865f178a302be639ddc561e1

I think that even before that changeset, the page was shown without any margins.

In any case, the extension seems to be too sensible to changes in the ckan layout so perhaps it should take into account the CKAN version and have a fixed set of xpaths and templates that work for each version

Superticket: #1343

This avoids difficulties when they forget their password.

• (?) Email validation (navl does not have one!)
• Require email confirmation to be activated (?) (Answer: no, not at present -- see discussion below)

Superticket: #1397

Write a worker daemon to download all resources from a CKAN instance to a local repository.

Questions

• Do we only want to download openly licensed information? ANS: no, we do everything (though do need to think about this re. IP issues)
• Should we have clever ways to dump APIs? ANS: no.
• Do we respect robots.txt even for openly licensed information? ANS: No (we're not crawling we're archiving)
• Use HTTP/1.1 Caching headers? ANS: if not changed since we last updated don't bother to recache.
  • Complete support for ETags
  • Expires, Max-Age etc.
• Check

Functionality

• Download files via HTTP, HTTPS (will not do FTP)

Process:

1. [Archiver.Update checks queue (automated as part of celery)]
2. Open url and get any info from resource on cache / content-length etc
   1. If FAILURE status: update task_status table (could retry if not more than 3 failures so far). Report task failure in celery
   2. Check headers for content-length and content-type ...
      • IF: content-length > max_content_length: EXIT (store outcomes on task_status, and update resource with size and content-type and any other info we get?)
      • ELSE: check content-type.
        • IF: NOT data stuff (e.g. text/html) then EXIT. (store outcomes and info on resource)
        • ELSE: archive it (compute md5 hash etc)
      • IF: get content-length and content-length unchanged GOTO step 4
3. Archive it: connect to storage system and store it. Bucket: from config, Key: /archive/{timestamp}/{resourceid}/filename.ext
   • Add cache url to resource and updated date
   • Add other relevant info to resource such as md5, content-type etc
4. Update task_status

Optional functionality

• If result object is HTML, search for references to "proper data" (CSV download pages etc.)
• Download from POST forms (accepting licenses or weird proprietary systems)
• Support running on Google Apps Engine to save traffic costs.

Existing work

• ​https://bitbucket.org/okfn/ckanext-qa/overview
• out of date: ​https://bitbucket.org/pudo/ckanextarchive - Old archiver extension, largely experimental.
• out of date: ​https://bitbucket.org/ollyc/ckan/changeset/1b16fbe9aa65 - Openness scores by ollyc

Support for listing files (manifest) contained within a package.

Traditional manifests just list the files. Suggest in addition we have support for optional metadata in form of key, value pairs.

Support uploading files to a storage system from WUI with a simple upload form.

Implementation

• Implement as an Extension
• Locate at /upload
• Add new system action/permission (FILE_UPLOAD) and restrict by default to logged in users
• Use OFS + boto -> generate form key (and create the bucket if it doesn't exist? Or do we assume bucket already exists)
• This wants to be tied in to general package workflow. See parent ticket:852.

Switch to repoze.who + openid plugin for authentication from authkit.

Already did this in microfacts so should be fairly easy.

TD look at issues and then email ckan-dev

meeting notes from 21-3-2012 general ideas non specific

Problem:

• We want to remove the Transformer usage due to performance and its close tie to genshi
• Want a standard way for core code and extensions to put code into the templates where *templates request it* (templates control insertion rather than extensions ...)
• Stacking versus overwriting on entry points
  • Relationship to h.requires('jquery', version) - ​http://codex.wordpress.org/Function_Reference/wp_enqueue_script

Options:

• placeholders or helper methods
• {{user _info}} {{sub_menu}} is this equivalent to <span class="insert-user-info"></span>
  • [xyz arguments] -> h.load_snippet('xyz', ...)
• ${h.snipit()}

TSV support doesn't seem to work very well, may be the mimetype ( text/tab-separated-values )

See ​http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96

Tags landing page tag

• Both search for tags *and* browse the list of tags right here
• Big cloud of tags (sorted by popularity/usage?) filterable with a search box.
• Tags search page tag/search and list tag/list are both redundant and should be merged straight into tag.

Take out import of paginate in setup.py. Use paginate in webhelpers instead. Make changes to take account of any i/f changes.

Take screenshots of existing ckan instances esp those mentioned <​http://wiki.ckan.net/Theming> and put on flickr in ckan or ckan-theme group so we can create a gallery ... (both to illustrate theming but also to show ckan instances that are around -- could add to ​http://wiki.ckan.net/Instances)

Taken from ​http://lists.okfn.org/pipermail/ckan-dev/2012-May/002218.html

A few months ago, the RDF provided when dereferencing a dataset's URI was very useful. It described the datasets using many of the "right" RDF vocabularies (dcterms, void, etc).

The RDF that is returned now no longer has these nice descriptions. Instead, it seems to be a mindless transcription of the CKAN record.

For example, the older descriptions gave:

<​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>

void:subset [

void:target <​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>, <​http://thedatahub.org/dataset/geonames-semantic-web> ; void:triples 50 ; a void:Linkset

], [

void:target <​http://thedatahub.org/dataset/dbpedia>, <​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states> ; void:triples 52 ; a void:Linkset

], [

void:target <​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>, <​http://thedatahub.org/dataset/govtrack> ; void:triples 52 ; a void:Linkset

] ; void:triples 130005 ;

but now it's simply:

<​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states>

dct:relation [

rdf:value "130005" ; rdfs:label "triples"

], [

rdf:value "52" ; rdfs:label "links:dbpedia"

], [

rdf:value "50" ; rdfs:label "links:geonames-semantic-web"

], [

rdf:value "52" ; rdfs:label "links:govtrack"

] ;

My old copies of the dataset descriptions mentioned a "GoLD CKAN 1.0" that was credited for the nicer modeling.

Where did that go? Can we get it back?

Thanks, Tim Lebo

<​http://semantic.ckan.net/record/91d2c0de-75a4-4bb6-b260-bc2946e1be8b>

dc:source <​http://ckan.net/> ; opmv:wasGeneratedBy [

opmv:used <​http://ckan.net/api/rest/package/farmers-markets-geographic-data-united-states>, <​http://ckan.net/api/rest/revision/9c2545b8-ebf1-42bf-ac92-633ff5ceba3f> ; opmv:wasControlledBy [

a foaf:Agent ; foaf:name "GoLD CKAN 1.0"

] ; opmv:wasPerformedAt [

a time:Instant ; time:inXSDDateTime "2012-01-31T05:34:32Z"^^xsd:dateTime

] ; a opmv:Process

] ; a dcat:CatalogRecord ; dcat:dataset <​http://thedatahub.org/dataset/farmers-markets-geographic-data-united-states> ;

…

Tasks include:

Estimate 4d

• Update requirements (git)
• Search facets error
• Error on dataset read (extras in context)
• Url generator for rdf
• Make templates work with latest theme
• Use dicts in templates (isopen())

Test DbIndex? with sqla 0.5 (and get it working if it does not work).

Cost: ??

Priority: low because DbIndex? is not currently that important to us (also wonder whether we should try using CKAN code here to stop us reinventing the wheel).

Test coverage for ckan.lib.cli is 34%. Need to improve that.

Test coverage of ckan.model.user is 53%. Notable exception - setting password. Room for improvement?

Tests marked decorated "@search_related" should only be run against postgresql, but in fact they don't get run at all.

The "Preview" button is a nice idea, but it doesn't seem to actually "preview" anything if the file MIME type would ordinarily cause the browser to download the file. If so, the browser does indeed just download the file.

This is notable in the context of most hosted file services (including Google Storage) which will deliberately serve a MIME type of application/x-some-junk-here in order to force a download.

The 'editor', 'anon_editor' and 'reader' roles are intended to have immutable actions. This was designed to prevent their names being subverted - e.g. an editor should always be able to edit! It also meant that when we add Actions (e.g. DELETE-PACKAGE) then it can be added sensibly to these roles in an upgrade just by changing the defaults table (ckan/model/authz.py).

The problem is that this immutability is only enforced on 'db upgrade'. So you can happily change the editor role using the paster command and it works, right up until you do an upgrade and realise permissions are different.

We should stop the paster commands being able to edit these roles. Or get rid of the immutability completely. Views?

The AlphaPage? implementation assumes that all packages start with a latin character. This is not true for Russian, Chinese, Arabian and Persian languages etc. We should either use a unicode solution (i.e. display a full class of characters based on the current locale) or use generic paging instead.

The CKAN instance on data.gov.uk serves invalid URIs out of its API.

For example the following can be found,

​http://uk.sitestat.com/homeoffice/rds/s?rds.hosb0509tabsxls&ns_type=pdf&ns_url=[http://www.homeoffice.gov.uk/rds/pdfs09/hosb0509tabs.xls]

In this URI, the : and / characters after the ? in the query part are invalid according to section 3.4 of RFC2396

Also URIs are not stripped of whitespace at the end.

This causes problems when other software with a more correct interpretation of what a valid URI is attempts to consume data from CKAN. In this instance the Talis triplestore complains about such URIs.

"Be liberal in what you accept and conservative in what you send" would seem apt.

Actions

• Validation of urls as part of form entry or data loading
  • Need to consider situation where this should happen out-of-band (i.e. we allow load even with invalid data and then flag bad dates in separate validation process). In general doubtful that we should do this here because url invalidity is such a big deal
• This code should support analysis of existing data so we can go through existing database and find invalid urls
  • Also useful to have this so we can do out of band validation

The CSW harvesters implemented at the moment were developed with the DGU project in mind, and they assume all remote CSW servers to implement the Gemini 2 specification. Gemini 2 is the profile defined in the UK for INSPIRE complying metadata, so obviously catalogs from other countries or non-INSPIRE complying ones won't be able to be harvested.

The changes needed to support generic CSW servers (i.e. those implementing the ISO 19139 profile) are:

• Handling the validators (right now are hardcoded in the harvester

code). This probably involves issues discussed in the CREP 3 (ticket #1134)

• Changes in the model to adapt the specification to ISO 19139

• Renaming objects and classes which are now Gemini-centric

List of CSW servers tested:

​https://spreadsheets.google.com/spreadsheet/ccc?key=0Atp3cZFjuIOAdDBVQWRINnlfN1d0b2lleHVEdjBSb2c&hl=en_US&authkey=CNu4hsEB#gid=0

The HTML and CSS for the activity streams templates needs to be tidied up. Currently the CSS is inlined in the templates themselves, and there is much unnecessary repetition of CSS.

Perhaps put this one in hold until there has been some design discussion about how the activity streams and the user, dataset and group pages with activity streams should look?

The Handle System is an initiative to provide persistent references for resources. That is, it's basically a proxy system for preventing link rot.

Its documentation is here: ​http://www.handle.net/. Servers running CKAN could host a "Local Handle Service", which redirects a hash of a resource to an actual URL.

Some suggested use cases:

• Researcher would like to cite where data came from
• Agencies would like to have a way to prevent vendor lock-in from CKAN if they decide to move to another platform

The JavaScript? just isn't going to work without it. Look into either the newer JSON3:

​http://bestiejs.github.com/json3/

or the old faithful:

​https://github.com/douglascrockford/JSON-js/

The Package model should have a boolean field added to it describing its visibility beyond that supplied via auth.

The related permission for viewing packages where public is set to False would be package.view (show, whatever), and *never* visible for unauthenticated users.

The Wordpresser extension seems to be working well for DataGM, but needs caching and tests.

The about page for the DataHub? (thedatahub.org/about) could be improved. More importantly the default about page for a generic CKAN instance should be completely different - focus more on Open Data rather than the community hub idea (as this is more relevant for most installations) and remove specific references to tdh.

The admin extension allows deleted datasets to be viewed, but there is no equivalent in the API. The package_list API call filters to just 'state=active' datasets.

'state' could be a parameter on the package_list call

• Easy to call and implement
• But getting permissions right for the different state values may not be easy - maybe a separate logic function is better?

The attributes we need to read in are being defined here: ​http://okfnpad.org/uklii

The design advice is to prepare, test, and maintain a dictionary of XPaths for each of the attributes.

Work effort is to be directed to the maintainability of the XPath statements. We need to know which are needed for which documents, and to have a way of preventing cruft.

The basic purpose of this extension is provide a way to record 'apps' (applications) that relate to datasets in CKAN. Both existing apps and ideas for apps should be permitted.

The checks to show or not the button need to be updated for the latest datastore version

The command line tool 'rights' is quite handy but it is glued to the CLI. I'm going to factor out the bit which searches for objects etc so it can be used by CreateTestData? etc and will be used by DGU.

The coverage tool (run by buildbot in the ckan build) reports that only 38% of lines of ckan.lib.authenticator are run in tests. This suggests a need for more tests.

The current behaviour of TemplateController?.view() (which is the fallback controller should all others fail) is to attempt to render (as a genshi template) the requested file.

Although this may be a feature that some instances want. In general, it leads to:

• 500s when attempting to access a normal template (eg - ​http://datahub.io/importer/preview)
• A way of inadvertantly serving things you may not want to serve. (Small risk, as it needs to be renderable as a genshi template).

Solution:

• Change the controller to 404
• Ensure there's a way for existing ckan instances to override that behaviour should they need it.

The current formalchemy setup conflates view, controller and model code in a way that makes it hard to debug and customise.

Review existing (and potentially non-existing) frameworks with a view to porting forms over to something more explicit and lightweight.

Implement the current Package forms as an example of how this would work.

Document and circulate.

Sub-ticket of #961 (form, validation, model sync meta-ticket)

The current implementation of Workers in ckanext-queue is broken. Basically the various consume / callback functions expect three arguments (routing_key, operation, payload) when they are in fact receiving only two of them (message_data, message). This is fairly easy to fix, but the question is if Workers add an extra complexity to use the messaging library directly.

The current search implementation uses the output of the the as_dict method of the domain Package object to update the index

​https://bitbucket.org/okfn/ckan/src/56c79e3fc44c/ckan/lib/search/index.py#cl-48

It also uses package_to_api1 in the SynchronousSearch? plugin:

​https://bitbucket.org/okfn/ckan/src/f9dfb0506594/ckan/lib/search/__init__.py#cl-93

This prevents extensions from being able to index custom properties (e.g. faceting by custom extras not included in the model).

The search should use the logic function to get the package properties:

get_action('package_show')(context,data_dict)

The debian package "ckan" with the two scripts "ckan-create-instance" and "ckan-instance-maintenance" depends against "postgresql". But "ckan-create-instance" is quite handy even when the DB is remote: it creates all the data dirs with the correct permissions, and the ckan and apache configs.

Please add a flag "--without-local-db" to "ckan-create-instance" and remove the postgres dependancy from the debain package.

The default config file should contains examples (maybe commented out) for all config variables. Makes it easier to edit the config. For example openid_enabled is not in there.

The default site was www.ckan.net, which had an Apache 301 redirect to ckan.net, but the problem was urllib was not rePOSTing stuff. This affected search parameters:

In [1]: import ckanclient

In [2]: c = ckanclient.CkanClient()

In [3]: c2 = ckanclient.CkanClient('http://ckan.net/api')

In [4]: c.package_search('', search_options={'groups':'openspending'})
Out[4]: 
{u'count': 2102,
 u'results': <generator object _result_generator at 0x10168b1e0>}

In [5]: c2.package_search('', search_options={'groups':'openspending'})
Out[5]: {u'count': 29, u'results': <generator object _result_generator at 0x10168b410>}

Thanks to borior for finding this and raising it.

The demo branch fails huge numbers of tests.

many are going to be template based - these need to be sorted - ideally we need to get them to work with legacy/new templates but see how we go on that one

anyhow this is not a priority whilst we are branched but needs resolving before we merge into master

The documentation for setting up Apache uses sites-enabled instead of sites-available and a2ensite demo.ckan.net

The email sending functionality (e.g for password reset) does not work on thedatahub.org and IATI (and probably some other instances) when using an address which is not a okfn.org one.

Could not send reset link: SMTPRecipientsRefused({u'amercadero@…': (550, 'relay not permitted')},)

As I said, adria.mercader@… works fine. The SMTP server used mail.okfn.org

The extension needs upgrading:

No module named webstore.database

The other option is to delay this functionality and merge with the publisher dashboards after the group refactor.

John, David Raznick, David Read, Ira, what do you think?

The feeds controller, used to construct atom feeds, duplicates code found in the package controller's search action (in order to construct the custom feed).

Refactor this to remove duplication.

The fields parameter of resource_search was deprecated when fixing #2438. It can be removed in release 1.9, and the action tidied up as a result.

The following pages are missing complete breadcrumbs.

Add related page:

​http://s031.okserver.org:2375/dataset/adur_district_spending/related/new

Needs to look like.

Dataset > {dataset_title} > Related > Create Related

Template here: ​https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/related/pages/form_page.html#L6

Same deal with Edit related.

​http://s031.okserver.org:2375/dataset/adur_district_spending/related/edit/5f9e665b-3e3c-428f-88b1-8554189cf14a

Needs to look like.

Dataset > {dataset_title} > Related > Edit Related

​https://github.com/okfn/ckan/blob/feature-2375-demo-theme/ckan/templates/related/edit.html#L5