{22} Trac tickets (2647 matches)

Results (1701 - 1800 of 2647)

Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#1751 enhancement icmurray icmurray ckan-sprint-2012-02-20 closed fixed DGU dataset form: collection of updates from feedback

A collection of improvements to the dataset creation/edition form (feedback form DGU).

  • [x] validation error formatting
  • [x] geograohic coverage: add local authority
  • [x] resource format field
  • [x] resource format auto-complete
  • [x] renamed data tab -> files tab
  • [x] integrate Adris'a INSPIRE changeset, and migration scripts as part of build
  • [x] remove "discontinued" from the update frequency
  • [x] disable the disabling of save
  • [x] Remove ability to edit url on the edit-form (and remove "url not available")
1328526612000000 1329733458000000
#1752 enhancement icmurray icmurray ckan-sprint-2012-02-20 closed fixed DGU: collection of updates from feedback
  • [x] hide black stars on the data-view page
  • [x] Edit on view form (if you have the right permission)
  • [x] RES_FORMAT ~> Resource Format
  • [x] Map Based Search -> UK Location Map Based Search
  • [x] Note at the top right expalining that these are seach filters (if they are) -> this requires "add a dataset" and "other access" to be moved to a different place (just some whitespace is fine)
  • [x] Display publisher and provider (UKLP records only).
  • [x] analysis of filter by OGL / non-OGL
  • [x] provide /dump on the build server
1328526750000000 1329733515000000
#1753 enhancement kindly rgrp ckan-sprint-2012-03-05 closed fixed Last fixes of Webstorer + Deploy on the DataHub 1328527932000000 1330990061000000
#1754 enhancement seanh seanh ckan-sprint-2012-02-06 closed fixed Add activities for extras

When an extra is added to or removed from a dataset or is updated, an activity should go in the dataset's and user's activity streams.

1328528318000000 1328528330000000
#1755 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU Alpha paged view of publishers

The publisher index page should show an alpha paged list of publishers.

1328530634000000 1328631064000000
#1756 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU Treeview of hierarchical publishers

The publisher index page should show a treeview of the publishers within the system. Probably behind a tab.

1328530689000000 1328709846000000
#1757 enhancement ross ross ckan-sprint-2012-03-05 closed fixed DGU Hide administrators from public view

On publisher view page - hide administrators from public view - only logged in admins of group should see them.

1328530775000000 1338205187000000
#1758 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU Check user management auth

Ensure publisher edit page only allows admins to manage users. #1675 is related.

1328530915000000 1328709897000000
#1759 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU Create publisher form access

Only sysadmin can access create publisher form.

1328530956000000 1329732744000000
#1760 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU Extract current publisher hierarchy

Get access to live Drupal MySQL database to get latest publisher list for migration.

Write a one-off script for import of the data.

1328531001000000 1328709560000000
#1761 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU History visibility

History on user profile should only be visible to a user who is a member of at least one group.

1328532889000000 1328716686000000
#1762 enhancement ross ross ckan-sprint-2012-02-20 closed fixed DGU Join publisher form

A new form for create publisher -> Add publisher access to your account' New wizard Autocomplete publisher name if linked from publisher view. Email address used to publish should be activated/validated on first use. Original email still used to log in.

1328532992000000 1329729305000000
#1763 enhancement seanh seanh ckan-v1.7 closed wontfix Add some simple model-level tests for activity streams 1328536285000000 1338204708000000
#1764 enhancement seanh seanh ckan-v1.7 closed wontfix Move activity streams API tests into tests/logic

Which is where the other API tests have now been moved.

1328536499000000 1335878051000000
#1765 enhancement seanh seanh ckan-sprint-2012-02-20 closed fixed Enhance Tag and Package models with vocabularies

Add a vocabulary column to the tags database table, change tags to have unique (tag_name, vocabulary_id) instrad of unique tag name, update methods in the Tag and Package classes to deal with the fact that tags may belong to vocabularies and that tag names are no longer unique.

1328537061000000 1329131124000000
#1766 enhancement johnglover johnglover ckan-sprint-2012-02-20 closed fixed Update and deploy EC Portal theme 1328537446000000 1328541227000000
#1767 enhancement johnglover johnglover ckan-sprint-2012-02-20 closed fixed Write end-to-end functional tests for tag vocabularies

Need to test:

  • API call to create new vocab
  • API call to add tags with a given vocab
  • API call to update tags
  • Dummy plugin to test that updating tags works via WUI
1328537586000000 1329326678000000
#1768 enhancement icmurray icmurray ckan-v1.7 closed fixed DGU: 7a. Public Publisher Dashboard (including QA Work and notifications) MUST PHASE 1 1328539740000000 1337159873000000
#1769 enhancement icmurray icmurray ckan-sprint-2012-02-06 closed fixed 6. Publisher Registration Improvements MUST PHASE 1 1328539789000000 1329737597000000
#1770 enhancement icmurray icmurray ckan-sprint-2012-02-06 closed wontfix 3. Themes/Taxonomy as well as tags WONT PHASE 2 1328539834000000 1329733935000000
#1771 enhancement icmurray icmurray ckan-sprint-2012-02-20 closed fixed 32. Browse by Tags PHASE 1 MUST 1328539862000000 1329733648000000
#1772 enhancement icmurray icmurray ckan-sprint-2012-02-06 closed fixed 13. Provider labelling MUST PHASE 1 [UKLP #14 MUST] 1328539902000000 1329733962000000
#1773 enhancement icmurray icmurray ckan-sprint-2012-02-20 closed fixed 17. Filter “UK Location Records” MUST PHASE 1 [UKLP #21 SHOULD] 1328540130000000 1329733534000000
#1774 enhancement icmurray icmurray ckan-sprint-2012-02-20 closed fixed 22. Mixed Licenses MUST PHASE 2 1328540153000000 1329733550000000
#1775 enhancement seanh seanh ckan-sprint-2012-03-05 closed fixed Enable adding many tags at once to controlled vocabularies via API

We already have API calls for adding one tag or removing one tag at a time to/from a vocab. Want more convenient ways to add or remove many in one call, or just pass in a list of tags to completely replace a vocab's current list.

  • Add tags_create() function that calls the existing tag_create() many times?
  • Also tags_delete()?
  • Or just let tag_create() and tag_delete() accept a list?
  • Add tags argument to vocabulary_update() to pass in a list of tags to replace the vocab's current tags? (But it should still be possible to simply rename a vocab without changing its tags and without having to pass in the tag list.)
1328541056000000 1329754459000000
#1776 enhancement seanh seanh ckan-sprint-2012-02-20 closed fixed Granular editing of vocabulary tags

Add API calls for adding one or more tags to and removing one or more tags from a vocabulary, without affecting the other tags in that vocabulary and without having to pass the full list of the vocabulary's tags.

1328541213000000 1329302364000000
#1777 enhancement kindly kindly ckan-sprint-2012-02-20 closed fixed Write about multilingual features for ec.

Basic description of multilingual features intended to be supported.

1328542835000000 1329393295000000
#1779 enhancement kindly kindly ckan-sprint-2012-02-20 closed fixed Add multilingual translation table.

A new table with 3 columns should be added. term, term_tranlastion, language_code. This table will be used for all translations, including tags. The table should have indexs on both the term and (term, language_code) combination.

1328548631000000 1329393759000000
#1780 enhancement kindly kindly ckan-backlog closed duplicate Api to add translations.

Add api to translation to the term_translation table.

  • add to logic 2d
  • test 2d
1328548908000000 1328578479000000
#1781 enhancement kindly kindly ckan-sprint-2012-02-20 closed fixed Api to add translations.

Add api to translation to the term_translation table.

  • add to logic 2d
  • test 1d
1328549031000000 1329393822000000
#1783 defect dread dread ckan-sprint-2012-02-20 closed fixed Locale change exception when unicode in URL

If the current page URL has unicode in it (e.g. u'/tag/biocombust\xedveis') then I'm not sure how why it isn't more encoded, but we are seeing links such as http://beta.dados.gov.br/dados/locale?locale=hu&return_to=/dados/tag/biocombust%C3%ADveis&hash=7fd941b5ba1a28bc84406c4191bf07b45a42a251 which cause an exception when the foreign character is passed to ckan.lib.hash.get_message_hash().

1328611485000000 1328612744000000
#1784 enhancement seanh seanh ckan-v1.7 closed wontfix Links to renamed objects break

The HTML templates for some activity stream activities link to objects (packages, resources, etc.) by their names, e.g. /dataset/foo, but if the object has since been renamed then the link will be broken.

Possible solutions are to use the object ID to find its current name at rendering time and link to that, or link to the object by ID rather than by name.

Linking by ID is both simpler to implement and better. Links by ID are ugly, but they are permanent and don't break when an object is renamed. We should be using links by ID everywhere on CKAN, so that when third-party sites copy-paste our URLs and link to us, their links don't break later.

Fixing this means fixing the dataset_link(), resource_link(), group_link() etc. functions in ckan/lib/helpers.py which will change the links everywhere in CKAN where those functions are used to link to objects (which should be everywhere where we link to an object).

If we want to have nice looking by-name URLs and what them to be permanent, then we need to change CKAN so that objects cannot be renamed.

1328613555000000 1338204797000000
#1786 enhancement dread [email protected] ckan-sprint-2012-02-20 closed fixed Relationships lost when editing package

If I have a relationship between two packages, and then I edit any property of one of them (Author email, for example), the relationship will automatically be lost.

1328635226000000 1328640597000000
#1788 defect zephod zephod ckan-sprint-2012-03-05 closed fixed IE7 compatibility problems

From Antonia Acuno, via Ira:

It appears that the current layout for the datahub does not work in IE7, which would mean a problem for us as most gov uses it and is one of those we have to support...any views?

Tested in several gov departments computers, so a common enough issue.

http://imgur.com/ip4xa

From Rufus: We haven't officially supported IE7 so far but can put engineering effort into this if absolutely needed. I note that IE 7 is now 3y out of date and is likely a security risk. Obviously this is something to ask Antonio about.

1328799250000000 1333133365000000
#1791 defect dread dread ckan-sprint-2012-02-20 closed fixed Changing locale on /dataset/new causes exception

When you are on the /dataset/new page and you try and change locale then you get a 500 error.

This is because it adds the 'cache' parameter, to ensure any proxy cache in the chain does not just send the cached page.

e.g. http://127.0.0.1:5000/dataset/new?__cache=37713707

1329134556000000 1329138315000000
#1792 enhancement toby kindly ckan-sprint-2012-03-05 closed fixed [super] api to use same validation and logic as wui forms.

The api currently uses the default validation schemas regardless of the form you specified on the front end. There should be a way to post through the api using the same validation rules.

1329146487000000 1333037710000000
#1793 enhancement ross ross closed duplicate Publisher changes

Changes to publishers for customer

1329214299000000 1329214522000000
#1794 enhancement ross ross closed fixed Publisher changes

Changes to publishers for customer that would otherwise clutter up the master branch.

1329214301000000 1332148972000000
#1795 enhancement ross ross closed wontfix Add approval_status to Package (as for Group)

The Package model should have a approval_status as the Group model does. See migration 049.

1329234060000000 1338206587000000
#1796 enhancement ross dread ckan-future closed fixed Get rid of lxml dependency

lxml is used in CKAN core in two places:

  • ckan/lib/helpers.py:239 Parsing Markdown and ensuring it is valid XML (i.e. tags close nicely so the HTML of the rest of the CKAN page is not disruptable).
  • reading the SOLR config XML to determine what version it is.

Both of these could be done with xml.dom built into Python and would save us a sizeable and slightly problematic dependency.

lxml is needed by these extensions: ckanext-inspire, ckanext-harvest, ckanext-csw & ckanext-wordpresser, so we'd have to add this to their requirements.

1329244440000000 1340635557000000
#1797 enhancement rgrp rgrp ckan-sprint-2012-03-05 closed fixed [super] Webstore v2 - Integrated and based on ElasticSearch

Main tasks:

  • Integrated seamlessly into CKAN (authorization etc)
  • Based off ElasticSearch?

Etherpad: http://ckan.okfnpad.org/webstore

Tasks

  • New API point (config + controller) [0.75d]
    • /api/data/{resource-id}
  • Authorization [0.5d]
  • Documentation [0.25d]
  • Upgrade data viewer to use ES [0.5d]
  • Convert webstorer to new setup - http://github.com/okfn/ckanext-webstorer [1d] - #1753
    • See below
  • "Create DataStore? Resource" in Resource creation (??)
    • Just creates a resource with blank url and datastore enabled (webstore_url non null)
  • Tick box to enable DataStore? in Resource edit form (for time being would just set webstore_url to any non-null/non-empty value)
    • Also need to use this info in DataStore? extension?
  • Create nice instructions on Resource page about using the DataStore?
  • Rename webstore -> datastore
  • Deploy [0.5d]

Details

Layout on ES

elasticsearch/{index}/{type-id}

index = ckan-{ckan-id}
type-id = resource-id (1)

Request path

Req => Nginx => CKAN App (authz) => req to elasticsearch and returns results

See as an example: https://github.com/okfn/elastic-proxy

Questions

  1. How do we decide what to use in the viewer e.g. Webstore data versus DataProxy?
    • Check for existence of webstore_url on Resource (should really be webstore_enabled boolean or similar, however we stick with url for the time being because it exists and we avoid migration). If exists use Webstore o/w don't.
  2. How do we decide when webstore is activated
    • Ans: see previous.
  3. Do we (and how do we) allow for resources which are just for webstores
    • Ans: we have to set webstore_url = True ... on save. Or add a big button saying: "Create Blank Resource" ...
  4. How does webstorer work ...
    • CSV import - NS: make sure we add e.g. rownum or id = rownum ...
    • Separate mini-service (webstorer.thedatahub.org ...)
  5. How do we selectively disable recline where not appropriate (e.g. query support makes no sense for DataProxy?)
    • Ans: do not bother at present. Focus on Webstore route.
  6. Where does geodata support come in (in Recline or separately)
    • This is a separate issue

User Story 1

User -> CKAN instance -> Add Dataset -> Adds a Resource via uploading a CSV file -> Save

=> Result = New Dataset + Resource + Data API to resource and nice Preview of the data

Solution:

  1. Allow normal upload process
  2. Webstorer queue task which should get triggered by new resource. Check the resource and if it can convert into the associated resource webstore database.
  3. Update the resource in some way (set webstore_url)

User Stories

User Story 2

User -> CKAN instance -> Add Dataset -> Adds a Resource via linking to some remote file (CSV or otherwise) or uploading some data (CSV or otherwise)

=> Result = New Dataset + Resource + nice Preview of the data (wherever possible)

User Story 3

User -> CKAN instance -> Add Dataset -> Adds a Resource which will be webstore database in CKAN -> User saves data to this data store.

=> Result = New Dataset + Resource (Webstore resource) which can now be written to or read from by the user

Solution: just create a resource with blank url (or better, have a new option to create webstore resource ...)

1329306185000000 1331412644000000
#1798 defect dread closed fixed API search in non-q fields has exception for unicode characters

You get an exception if you use the API to search packages and specify a non-ascii character in a field other than q.

For example:

http://catalogue.data.gov.uk/api/search/package?title=%E2%80%93

This "N-dash" (Unicode character 2013) causes this exception:

Module ckan.controllers.api:460 in search
<<                      if ver in u'12':
                               # Otherwise, put all unrecognised ones into the q parameter
                               params = convert_legacy_parameters_to_solr(params)
                           query = query_for(model.Package)
                           results = query.run(params)
>>  params = convert_legacy_parameters_to_solr(params)
Module ckan.lib.search.query:38 in convert_legacy_parameters_to_solr
<<      for search_key in non_solr_params:
               value_obj = legacy_params[search_key]
               value = str(value_obj).replace('+', ' ')
               if search_key == 'all_fields':
                   if value:
>>  value = str(value_obj).replace('+', ' ')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 30: ordinal not in range(128)

This problem affects CKAN 1.5 and 1.5.1 only.

1329395420000000 1329395560000000
#1799 defect dread dread ckan-sprint-2012-03-05 closed fixed To login as a different user or register, you need to logout first

When you are logged in, you cannot login as a different user before first logging out. (This is a restricting in repoze.who, which gives priority to an existing recognised identity over another one you supply.)

Yet, the CKAN login form appears when you try to do something you're not allowed to, even if you are logged in already. You login with a different identity and nothing happens - you remain as the first identity - this is incorrect.

(It is also a little confusing to be sent to the log-in page, although we do have a flash message explaining, and you might want to relogin as a superuser if you can.)

Whilst in this area, it would be worth fixing the problem for developers where you are logged in, do "paster db clean" and then try to log in - AuthTkt? recognises your old cookie but the User object doesn't exist, so you get an error logging in. In this instance we should tell people to log-out.

1329402820000000 1330002176000000
#1801 enhancement dread ckan-backlog closed fixed No links to password reset

You can reset your password (#1186) but you have to know the URI (/user/reset) - there is no link to this page!

1329405290000000 1343144718000000
#1802 enhancement dread dread ckan-sprint-2012-03-05 closed fixed Investigate CKAN deployment on Windows

See how possible it is to run CKAN on a Windows server.

Plan:

  • Get CKAN code and dependencies installed on Windows
    • ideally as similar to standard package install as possible
    • virtualenv/pip should work
    • try Cygwin to help
    • SOLR and Postgres could potentially run on other servers if necessary.
  • Try to run on Apache
  • See if there are any issues
1329409512000000 1329830097000000
#1803 enhancement toby toby ckan-sprint-2012-03-19 closed fixed clean up routes 1329483497000000 1332333091000000
#1804 defect toby dread ckan-sprint-2012-03-05 closed fixed CKAN mounted at URL - changing language problem

e.g. http://189.9.137.65/dados/ clicking on Deutsch link is http://189.9.137.65/dados/locale?locale=de&return_to=%2Fdados%2F&hash=1dc17c315c419df850da0dd3599eefa9da76fbeb and redirect goes to http://189.9.137.65/dados/dados/?__cache=97995106 so /dados/dados/ when it should be /dados.

Affects CKAN 1.6b only (not yet released).

1329484956000000 1330347185000000
#1805 enhancement toby closed fixed Error pages do not translate

eg 404 page

create a 404 error via a bad url - translation links go to /document/error not the actual bad url

1329489216000000 1329760795000000
#1806 task toby toby ckan-sprint-2012-03-05 closed fixed make drop down language selector

language chooser should look like the ec one

1329741393000000 1331295310000000
#1807 enhancement seanh seanh ckan-sprint-2012-03-05 closed fixed Recently added/updated datasets for EC portal

For EC Portal need a page (or HTML snippet embedded in the front page?) listing recently added or updated datasets.

  • Could be generated from the activity streams
  • Should it list datasets only, or other activities as well?
  • May be related to #1785 (Replace 'Revisions' page with site-wide activity stream)
1329742326000000 1330809641000000
#1808 enhancement seanh seanh ckan-sprint-2012-03-19 closed fixed Translation of content on dataset view pages

All data is that is translatable should be translated when viewed.

This includes tag names will be ID codes. When viewing a dataset page, look up the current language of the interface (ckan_lang or something in the environs) and display the string for the tag in that language.

Related to the dataset view extension points for translation, recently added by kindly/=.

1329742530000000 1331146804000000
#1809 enhancement johnglover johnglover ckan-sprint-2012-03-05 closed fixed Catch request exceptions in archiver link_checker task

Some request exceptions are currently not being caught (see the celery log on thedatahub for examples)

1329746267000000 1330528828000000
#1810 enhancement johnglover johnglover ckan-backlog closed wontfix Store list of Eurovoc tag names and IDs
  • Tag list not be added to CKAN core, what is the best way to do this in an extension?
  • Does the list need to be updated via API/WUI?
1329746612000000 1335874745000000
#1811 enhancement johnglover johnglover ckan-backlog closed wontfix Translate Eurovoc tags from ID to dataset language 1329747684000000 1335874765000000
#1812 enhancement johnglover johnglover ckan-backlog closed fixed Restrict editing rights/permissions based on publisher
  • see how this currently works with DGU first
1329747889000000 1335874864000000
#1813 task amercader amercader ckan-sprint-2012-03-05 closed fixed Upadate ckanext-pdeu to work with CKAN 1.6

Tasks include:

Estimate 4d

  • Update requirements (git)
  • Search facets error
  • Error on dataset read (extras in context)
  • Url generator for rdf
  • Make templates work with latest theme
  • Use dicts in templates (isopen())
1329757128000000 1330102557000000
#1814 enhancement amercader amercader ckan-sprint-2012-03-19 closed fixed Check publicadata.eu harvesters

Estimate 2d

Once ckanext-pdeu is running on CKAN 1.6, upgrade ckanext-harvest to be able to update the CKAN harvesters (default tags and extras).

Also check non-CKAN harvesters (specially scrappers) to see if they are still working.

Make a list of current harvesters with status and potential ones.

1329757408000000 1332152596000000
#1816 enhancement amercader amercader ckan-sprint-2012-03-19 closed fixed Update ckanext-apps to work with CKAN 1.5.1 - 1.6

Estimate [2-3d]

If possible, it should work with both 1.5.1 and 1.6.

Needed for PDEU, also at some point for IATI

1329757864000000 1331302835000000
#1817 enhancement johnglover johnglover ckan-sprint-2012-03-05 closed fixed Importer for EC publisher info
  • write importer for publisher info (currently in JSON format), add to ckanext-ecportal
  • add group translations to translations table
  • run on test server
1329762290000000 1329922818000000
#1819 enhancement kindly kindly ckan-sprint-2012-04-02 closed fixed change search index code to use package_dictize instead of obj.to_dict

Search index code needs to use package_dictize so that it can get the new vocablurary information and is more consistant with the rest of the system. This is in preperation for their translation being put in the search indexer.

1329770555000000 1338193886000000
#1820 enhancement kindly kindly ckan-sprint-2012-03-19 closed fixed Index multilingual data when mulilingual extension is added.

All translated fields and vocabularies need to be added to search index in the correct fields.

1329770903000000 1332163374000000
#1821 enhancement kindly kindly ckan-sprint-2012-03-05 closed fixed create internal multilingual extension that brings together all multilingual features

This extension point needs to combine.

  • Making sure the correct data get put in search index.
  • Make the search queries are modified to weight multilingual search correctly.
  • View pages, including search results are translated.
  • Bring together all search schema modifications.
1329771267000000 1330990111000000
#1822 enhancement kindly kindly ckan-sprint-2012-03-05 closed fixed Weight mulilingual searches correctly

Dismax query across languages. Titles and selected languages, weighted more highly.

1329772080000000 1330990127000000
#1825 enhancement seanh seanh ckan-sprint-2012-03-05 closed fixed Write docs for vocabularies API

Done on branch feature-1698-tag-taxonomies

1329845122000000 1330807330000000
#1826 enhancement kindly kindly ckan-sprint-2012-03-05 closed fixed weight queries so that title is more important than rest of body

Currently everything is considered equally when doing a normal search. Use dismax when there is not a fielded search to get round this. This is part of the work for #1822 as the dismax option is needed for multilingual search.

1329924260000000 1330990247000000
#1828 enhancement rgrp rgrp ckan-sprint-2012-03-05 closed fixed Upgrade to Recline v0.3

Recline v0.3 has introduced some breaking changes in API and usage. Upgrade to use it.

1329943304000000 1330525412000000
#1829 defect dread dread ckan-sprint-2012-03-05 closed fixed Changing back to English prints the flash message in the previous non-English language

On the homepage click "francais" and then "English". The flash message reads "Le langage a été fixé à: français" when it should say "The language is now: English".

1330000660000000 1330001990000000
#1830 enhancement dread dread ckan-sprint-2012-03-05 closed fixed OpenID disablable

Allow OpenID to be disabled as a login option.

Currently (CKAN 1.6) there is a config option openid_enabled which stops the login option being displayed, but:

  • In the User edit form you can still add your OpenID
  • The repoze.who.openid middleware is still required - can it be avoided?
1330002137000000 1330089912000000
#2198 enhancement zydio closed fixed API documentation is missing Storage Metadata API info

Now that ckanext-storage is back into the core (v1.6), CKAN documentation should probably contain info on Storate Metadata API.

1330421743000000 1339771453000000
#2199 enhancement seanh seanh ckan-sprint-2012-03-19 closed fixed Translation of content on dataset search pages 1330440107000000 1331146832000000
#2201 enhancement rgrp ckan-v1.7 closed duplicate Add citation info to Dataset and Resource page

Suggest:

%title. %author. Retrieved %date. %site_title.

For resource: %title = %dataset_title. %resource_name.

1330765080000000 1330765400000000
#2204 enhancement ross rgrp ckan-sprint-2012-04-30 closed fixed [super] Related (Stuff) Extension

This is a reworking of the existing apps extension.

Initial proposal at http://wiki.ckan.org/Proposals#Apps_in_CKAN and http://wiki.ckan.org/Proposals#References.2FLinks_in_CKAN

Naming

What do we call this extension?

  • related
    • RP best IMO but perhaps too close to separate relationships concept?
  • relatedstuff
  • links
  • references

Proposal

"Related Stuff": Apps as in an application (website/service/tool) that uses this dataset (as in Apps and Ideas extension)

Implementation

New table named Related with following structure

|| id (int) || type || title || description (markdown) || image_url || owner_id || url || created (timestamp)
  • type = Idea | App | API | Visualization | Post | Paper | News Article
    • Suggest we make this a ckan.ini config option (comma separated ...?)
      • Do we want the possibility of different templates for different types of Related objects?
  • image: ?? Depends where we store images. Simplest option would be to change to image_url and leave it to users to have already uploaded an image somewhere. If not we need to support image uploading and storage. See #1692 (add image attribute to datasets and groups) for more discussion, once implemented the URL here can be an internal url.
  • owner_id = user_id or creating user (see authorization below)

Related2Dataset (note that related_id, dataset_id tuple should be unique). This allows for m2m connections. If a given related item is only with one dataset this could be simplified. May contain status so dataset owner can turn this on/off.

|| id || dataset_id || related_id || status

status should be used to allow for a dataset owner (for dataset_id) to de-activate the relationship between the dataset and the related.

Url

  • /dataset/{dataset-name}/related/{related-item-id}/{related-item-title-stringified}
    • If a reference item could exist in its own right (and perhaps refer to multiple datasets then it should get its own url at e.g. /related/{id}
  • /dataset/{dataset-name}/related/add => Modal dialog on related tab so we can use API to create them.

/dataset/{id}/related <- list

  • use image_url for small icon etc, title description (shortened?)
  • Click through to full related item (optional)
  • dataset owner is shown show / hide button ... (or on /dataset/{id}/related/{id} )
  • related owner sees an edit button / icon (pops up modal)

/dataset/{id}/related/{id} (optional)

/dataset/{id} will have a Related tab (with bubble with count).

  • Drop down with Add Related -> Pop-up modal and save via API

(Not used: /related/add with dataset prefilled ... )

/related/{id}/edit

Authorization

Addition of related item be considered orthogonal to datasets (and hence with separate authorization i.e. i can add the info that site X uses dataset Y without needing permission to edit dataset Y).

Thus any logged in user could add a Related item. We set the owner of the related item to creating user and going forward only that user or a sysadmin can update or delete.

NB: we could have a system where datasets owners have to approve related items before they show up next to their dataset. This would add substantial complexity so I propose we leave out of phase 1.

Tasks and estimates {7.5d}

[x] Model + Migration for Related table. {0.75d}

[x] Controller for Related (or relevant sections in Package controller). {0.75d}

[x] Routing setup. {0.25d}

[x] Schema for related. {0.5d}

[x] Logic layer actions. {1.0d}

[x] Auth (default + publisher). {0.5d}

[x] Templates + Dataset changes (new tab etc). {1.0d}

[x] JS Application for adding Related objects in a modal. {1.0d}

[x] Testing. {0.75d}

[x] Dataset owner disabling of Related (via M2M table). {0.5d}

  • Updated to allow author of related to delete as well

[x] Documentation. {0.25d}

1330862992000000 1337594723000000
#2205 enhancement johnglover johnglover ckan-sprint-2012-03-19 closed fixed Document Tag Vocab schema/form setup 1330941067000000 1330956840000000
#2206 enhancement johnglover johnglover ckan-sprint-2012-03-19 closed fixed Change site header to match latest ODP template 1330958095000000 1331046486000000
#2207 defect seanh seanh ckan-sprint-2012-03-19 closed invalid Fix recently changed datasets activity stream on exportal site

Apparently it's working when developers run it locally but not on the demo site.

1330961739000000 1332154835000000
#2208 enhancement seanh seanh ckan-sprint-2012-06-25 closed fixed Export strings from ckan and ckanext-ecportal to combined pot file

All strings from the ckanext-ecportal extension (and from CKAN core) need to be exported to a pot file to send to tenforce.

1330961806000000 1339596371000000
#2209 enhancement ross ross ckan-sprint-2012-03-19 closed fixed Object->RDF mapping

Super is [1649]

We need to provide customisable mappings between CKAN models and RDF. This needs to be easily configurable for different clients and should take into account objects related to the model being mapped.

Current contenders for impl are:

  1. ordf.org's Object Description Mapper (which uses OWL)
  1. RDFAlchemy, although this looks like it may result in fixed mappings
  1. A custom scheme with a simple json file
  1. A template based approach using Genshi templates?

Approach is to get a mapping complete of the attributes directly in the package class, and then provide mappings for related objects (tags/resources etc).

We'll then need to be able to publish the RDF to a triplestore.

See http://ckan.okfnpad.org/rdfmapping for further analysis.

1331027277000000 1336566051000000
#2210 enhancement ross ross ckan-sprint-2012-03-19 closed fixed Add method to group for adding members

It is necessary to add a helper method to Group for adding members to that group so that

  • nobody needs to work with the Member model
  • It handles existing membership and existing membership with a different capacity
1331028470000000 1331549369000000
#2211 enhancement ross ross ckan-sprint-2012-03-19 closed fixed Logic layer access to publisher functionality

Most of the get_group/add_member style calls should be handled, or at least provided via logic layer.

Don't need to break existing code, but do need to provide access via logic layer

1331028751000000 1331549406000000
#2212 enhancement johnglover johnglover ckan-sprint-2012-03-19 closed fixed Import latest data into demo site 1331047333000000 1331720191000000
#2213 enhancement johnglover johnglover ckan-sprint-2012-04-02 closed fixed Update geographical coverage tags 1331047525000000 1332342835000000
#2214 enhancement seanh seanh ckan-sprint-2012-03-19 closed fixed Translation of content on group view pages 1331146864000000 1331147119000000
#2215 enhancement seanh seanh ckan-sprint-2012-03-19 closed fixed Translation of content on group list page 1331148579000000 1331653117000000
#2216 enhancement seanh seanh ckan-sprint-2012-04-02 closed fixed Translation of selected facets on search pages 1331148648000000 1332956791000000
#2217 enhancement seanh seanh ckan-sprint-2012-03-19 closed fixed Translation of content on tag list page 1331148704000000 1331652833000000
#2218 enhancement seanh seanh ckan-sprint-2012-03-19 closed fixed Translation of content on tag pages 1331148734000000 1331747842000000
#2219 enhancement seanh seanh ckan-sprint-2012-03-19 closed fixed Translation of datasets and tags on user pages

User names and descriptions are not translated.

1331148895000000 1331750370000000
#2220 enhancement toby toby ckan-sprint-2012-03-19 closed fixed make ckan cacheable

need to sort cookies etc

should work with nginx and varnish

1331228643000000 1338205222000000
#2221 enhancement rgrp zephod closed fixed Tests are broken for some of us: NotAPairTreeStoreException

Since my last pull I'm getting a strange new error which is presumably to do with my local config or assumptions about the storage extension.

Test output here: https://gist.github.com/2007985

1331318781000000 1331561095000000
#2222 enhancement rgrp zephod closed duplicate Tests are broken for some of us: NotAPairTreeStoreException

Since my last pull I'm getting a strange new error which is presumably to do with my local config or assumptions about the storage extension.

Test output here: https://gist.github.com/2007985

1331318783000000 1331318825000000
#2223 enhancement zephod rgrp ckan-sprint-2012-03-19 closed fixed Switch to bootstrap from blueprint

Why?

  • We already use quite a bit of boostrap
  • We already plan to use bootstrap JS
  • Bootstrap is excellent and pretty light
1331384974000000 1332842692000000
#2224 enhancement rgrp rgrp ckan-sprint-2012-04-02 closed wontfix Simplify javascript and css dependencies and add minified version
  • Get rid of jquery UI
  • Remove modernizr (what do we use it for?)
  • Convert jquery.tmpl to jquery mustache (?)
  • Remove ckanjs moving core parts of it directly into our code.

Compress everything for releases and add minified version to build (with switch to enable use of this?)

1331385623000000 1338205358000000
#2225 enhancement aron.carroll rgrp demo phase 2 closed fixed Run jshint on our javascript and clean up as needed 1331407316000000 1343220502000000
#2226 enhancement rgrp rgrp ckan-sprint-2012-03-19 closed fixed [super] Refactor and improve documentation (v1.7)

Documentation is key to the success of projects!

Move material into main docs from wiki

Create a User Guide

Basic step-by-step on getting started with CKAN (See start on set of slides here e.g. http://blog.thedatahub.org/2012/03/02/tutorial-publish-data-with-the-datahub/)

  • Publishing data
  • Authorization and workflows
    • Publisher Profile (Workflow)

Break into sections

Suggest something like:

# General
* Intro
* Conceptual Overview
* What CKAN helps you do (http://ckan.org/)
* FAQ ? (or point to the wiki)

# For Administrators

* Installation and Setup
  * Configuration options
* Customization - Theming etc
* Authorization and Workflows
* Storage

# For Users (Publishers, Data Wranglers, etc etc)
* Walkthrough of publishing a dataset
* Storage
* API (see Developers section)

# CKAN Developers

* Domain Model
* API
* Extensions
* i18n

CKAN Developers
* Buildbot
1331481965000000 1332475494000000
#2227 enhancement toby toby ckan-sprint-2012-03-19 closed wontfix Tests are slow and broken

not all tests run and they take too long :(

1331556570000000 1332843126000000
#2228 enhancement ross ross ckan-sprint-2012-03-19 closed fixed IGroupForm should allow overriding of non-edit templates

IGroupForm should allow overriding of non-edit templates so that it can specify the index +/or search +/or view.

1331637803000000 1332149030000000
#2229 enhancement kindly kindly ckan-sprint-2012-03-19 closed fixed Cleanup plugin system after some test failed to run.

The logic test did not have init. This caused lots of tests to fail because there were mock extensions that ran automatically in them. Fix plugin system so this can work.

1331721611000000 1332163408000000
#2230 refactor seanh ckan-v1.7 closed fixed Tidy up of search facets code duplication

Because of a clash between two development branches there is some duplication of code to do with code facets (note: at the time of writing the code duplication exists only on the feature-1821-multilingual-extension branch, but this will be merged into master at some point):

The package_search() function is adding the search facets to the search results twice with two different data structures, with keys "facets" and "new_facets". It should be reduced to just the new facets (with the key changed to "facets").

Also the group and package controllers are adding both facets and new_facets to the context, should be new_facets only (but renamed to facets).

The facet_items() function in helpers.py should be removed, it uses the old facets structure and shouldn't be needed anymore with the new facets structure.

In facets.html, facet_sidebar() should be removed as it uses the old facets structure and facet_div() implements the same functionality but uses the new facets.

In facets.html, facet_list_items() will have to be updated to not use the facet_items() helper and to use the new facets structure instead.

Anywhere that "new_facets" appears it will have to be changed to "facets" (e.g. in the ckanext/multilingual/plugin.py.

This is the merge commit that introduced the duplication: https://github.com/okfn/ckan/commit/1153aa876f54c22289e460aeececea22d1d4d51d

This is the earlier commit where the search facets were refactored: https://github.com/okfn/ckan/commit/3970e52008b75933fda1be1d488bed2578d98c9c

1331744085000000 1338204707000000
#2231 defect amercader amercader ckan-sprint-2012-04-02 closed fixed Base harvester broken on 1.6.1

On its current version, ckanext-harvest passed a custom schema to package_create and package_update to allow defining an id for new packages (the same as in remote ckan instances) [1]

Recent changes in the logic layer have made that the context object is no longer directly read from the context, but only get from plugins [2]. This makes the schema sent from ckanext-harvest to be ignored and the harvesting fails because the id field is not expected

ValidationError: {'Id': u'The input field id was not expected.'}

We need a way to either:

  1. Be more flexible accepting schemas on the logic functions
  2. Modify the harvester plugin to implement the necessary interface

[1] https://github.com/okfn/ckanext-harvest/blob/master/ckanext/harvest/harvesters/base.py#L115

[2] https://github.com/okfn/ckan/blob/0c07d3be846777318b35c31325215dff00e99313/ckan/logic/action/create.py#L38

1331815499000000 1333374735000000
#2232 defect amercader amercader ckan-sprint-2012-04-02 closed fixed Unicode Exception when rebuilding the search index

In some cases this exception is fired when building the search index. As explained in #1616 this makes the whole process stop.

  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 326, in wrapper
    return self._update(content, query)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 550, in _update
    rsp = self._post(selector, request, self.xmlheaders)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 639, in _post
    return check_response_status(self.conn.getresponse())
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/solr/core.py", line 1096, in check_response_status
    raise ex
SolrException: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.
Traceback (most recent call last):
  File "/var/lib/ckan/pdeu/pyenv/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/var/lib/ckan/pdeu/pyenv/lib/python2.6/site-packages/paste/script/command.py", line 218, in run
    result = self.command()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/cli.py", line 298, in command
    rebuild()
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/__init__.py", line 134, in rebuild
    {'id': pkg.id}
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 54, in insert_dict
    return self.update_dict(data)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 79, in update_dict
    self.index_package(pkg_dict)
  File "/var/lib/ckan/pdeu/pyenv/src/ckan/ckan/lib/search/index.py", line 153, in index_package
    raise SearchIndexError(e)
ckan.lib.search.common.SearchIndexError: HTTP code=400, reason=ParseError at [row,col]:[1,2354] Message: An invalid XML character (Unicode: 0x1) was found in the element content of the document.
1331816582000000 1332331660000000
Note: See TracReports for help on using and creating reports.