{22} Trac tickets (2647 matches)

Results (201 - 300 of 2647)

1 2 3 4 5 6 7 8 9 10 11 12 13
Id Type Owner Reporter Milestone Status Resolution Summary Description Posixtime Modifiedtime
#201 enhancement rgrp dread v1.0 closed fixed Stress test ckan

Check:

  • high numbers of simultaneous users
  • incomplete transactions
  • etc.
1259844836000000 1265284419000000
#202 enhancement tonfa dread v0.11 closed fixed Internationalization (i18n) try-out

Try i18n for one language (German) for ckan (de.ckan.net).

Basic approach to i18n is well-covered in Pylons book: http://pylonsbook.com/en/1.1/internationalization-and-localization.html

However, we need to integrate with our Genshi templates. Details on how to do this in comment: http://wiki.pylonshq.com/display/pylonsdocs/Internationalization+and+Localization

Steps

  1. i18n templates
    • Extract content using Babel
    • Get genshi translation working
    • Do the translation!
  2. i18n all strings in controllers (if any)
  3. i18n forms
1260179192000000 1265892368000000
#203 enhancement dread dread v0.11 closed fixed show package counts for groups in WUI

In two places in the WUI

  • list of groups
  • group page in title e.g. ukgov (5)

Cost: 1h

1260187690000000 1265891080000000
#204 enhancement dread dread v0.11 closed fixed Custom package input form

As a

admin

I want to

input government data in a constrained way

so that

you get consistency in the db which is easily searched, discovered and automatically linked.

implementation

Form has fields suitable for government data. (related to custom excel importer ticket)

In pylons config specify the form schema to use - whether the existing one or the government one.

On submission, during validation, warn if a field isn't completed (unless it's optional - agency, maintainer, maintainer_email)

Simple fields

  • all basic ckan fields
  • default license to OKD::Crown Copyright
  • co_id, precision, taxonomy url, agency

Date fields

Date format: machine readable. Can be specific or vague: year or month/year or day/month/year.

  • date_released
  • date_updated
  • temporal_coverage_from, temporal_coverage_to - two date fields combined

Strict selection

Drop down combo

  • department

Selection or free text

Editable combo

  • update_frequency - selections: never, weekly, monthly, annually, discontinued
  • geographic_granularity - selections: national, regional, local authority, ward, point
  • temporal_granularity - selections: yes, months, weeks, days, hours, point

Specialised fields

  • geographic_coverage - UI is tick boxes. store a comma separated list of all that apply from: england, scotland, wales, northern ireland, overseas, global

(Idea for future version: Select the most relevant one from a choice of OS hierarchical geographic IDs in the UK. Stored in the database as the RDF URI. Question - what to do beyond the UK?)

  • national_statisic - UI is check box, store as 'Yes' or 'No'
  • resources - use standard table, but suggestions for file type are: RDF, CSV, SPARQL, REST, XBRL, SDMX, Other XML, HTML+RDFa, XLS, DOC. Have warning label saying it is not ideal to have PDF, zip or html targets.
  • categories - multi values from strict selection of ONS categories. Stored as comma-separated.

Field data processed

  • tags - in addition to words entered here, search for known keywords in other fields: title, notes, department, agency, geographic coverage & categories.

Future - optional

Each form schemas is made available (or not) in the pylons config, along with the default schema for new packages. If there is more than just the 'basic' schema available then the 'new package' form has at the top a drop-down to select the schema with a 'switch' button. This submits the form's current data and parameter schema=xyz and the form arrives back populated with the existing options. No data should be lost switching views.

When you edit a package it detects the schema based on the extra fields that are present.

1260188965000000 1285082169000000
#205 enhancement rgrp dread closed wontfix Custom package importer

Importer function is exposed to logged in users.

Alongside specifying file to import, user chooses between 'form schemas' - 'basic' and 'government'.

  • Basic is as it is currently
  • Government form has pre-defined Extra fields, careful validation of lots of fields, code formats/processes some field data for storage. e.g. searching for tag keywords.

All fields should be in step with the government custom input form.

For fields which have suggested values but the user can input his own value, this is achieved in the spreadsheet with two columns - the first being a drop-down and the second free text.

Estimate:

1260197819000000 1297068450000000
#206 enhancement nickstenning nickstenning v1.0 closed fixed UI review

Investigate and list (on the wiki at wiki:UiReviewNotes) CKAN's UI issues, in particular:

  • Overview pages: reduction in visual noise and unnecessary elements.
    • best example: package overview page
    • consider openness/downloadability icons
    • display common metadata fields in a more structured fashion?

Est cost: 2 hrs.

1260199941000000 1260881123000000
#207 enhancement dread nickstenning v1.0 closed fixed Reimplement tag selection in jQuery

Remove the current dependency on Yahoo YUI toolkit for doing tag autocompletion on package edit forms.

The (KForge-derived) behaviour files for CKAN can easily be switched to the new jQuery-based ones: see ticket:212. Once this is done, it would make sense to implement the tag-selection code with jQuery.

Relevant pages include:

1260200340000000 1271250740000000
#208 enhancement dread rgrp v0.11 closed fixed Obfuscate author email and maintainer email field

Obfuscate author email and maintainer email field.

Use: webhelpers.html.tools.mail_to e.g. webhelpers.html.tools.mail_to('[email protected]…', encode='javascript')

Do we want to? Surely only public addresses should be on CKAN in which case why obfuscate?

1260203938000000 1265891070000000
#209 enhancement rgrp rgrp closed fixed Validate email address when creating an enquiry

p:wui 2009-09-09 validate email address when creating an enquiry

1260288671000000 1260288728000000
#210 enhancement johnbywater johnbywater closed invalid User registration

Native registration scenario: Complete registration form, require unique email address (indicate this not displayed on site), full name (indicate this is displayed on site), password. Dispatch email confirmation message. Respond to email confirmation message causes user to be logged in. Users are assigned a uuid.

1260292578000000 1266509788000000
#211 enhancement johnbywater johnbywater closed invalid User login

Native login scenario: Complete login form, require email address. Any session is resumed.

Mandated technical requirement: use Repoze (see plugin:form of repoze.who); borrow from BlastOff?.

See CKAN or Microfacts for usage of repose with openid.

1260292624000000 1266509848000000
#212 enhancement rgrp rgrp v0.11 closed fixed Switch to using m.okfn.org "kforge" media (css and behaviour)

We are already using standard kforge css though copied locally and with a few modifications.

We should switch to using common copy on m.okfn.org. Some work will be required to factor out any changes we have made.

Cost: 2h

1260703728000000 1260703969000000
#213 enhancement dread dread v0.11 closed fixed Automate CKAN deployment

fabric script to automate ckan deployment

1260811707000000 1265891061000000
#214 enhancement nickstenning dread v0.11 closed fixed UI Review - General template
  • Set hover names on navigation links.
  • Do we really need to display "About CKAN" everywhere? Just on home page.
  • Redirect in routing /groups and /tags rather than /group and /tag. At the very least make the plural versions forward to the singular versions.
1260879581000000 1263406056000000
#215 enhancement rgrp dread v1.0 closed fixed UI Review - Home page
  • 'Login via OpenID' -> 'Login with OpenID'?
  • Elements:
    • Search
    • about CKAN
    • 3 column Packages view (see ticket:219)
    • Small tag cloud (RHS)

Also, what about ticket:143 ?

1260879732000000 1266513523000000
#216 enhancement nickstenning dread v1.0 closed fixed UI Review - Search
  • allow clicking on 'only openly licensed'/'only downloadable' to select/deselect checkboxes: make them <label>s.
  • cut 'only's -- use word 'filter' instead
  • doesn't appear to match substrings!? A search for 'anna' doesn't return match the 'annakarenina' package.
  • autocomplete package names & tags
1260879802000000 1273596102000000
#217 enhancement nickstenning dread v0.11 closed fixed UI Review - Search results

Package search results page package/search

  • need clearer delineation between search entry form and results.
  • 0 results needs to be clearer, and the alternative suggestion: 'would you like to create a new package' needs to not look like a search result.
  • "N tags found" -- why do I care? this should be presented as "filter results by tag" -- very unlikely that I'm looking for a tag: I'm looking for a package, but tags might help me narrow down my search.
  • tags should be displayed more like the way they're displayed when you add them to a package (i.e. blue 'tag' background)
  • filter out particular tags
  • tag cloud?
  • copywriting: "Packages - Search" -> "Search packages"
1260879870000000 1263406092000000
#218 enhancement nickstenning dread v0.11 closed fixed UI Review - Package listing
  • openness/downloadability ticks/crosses are totally inscrutable to non-techies (those who don't know about the title attribute). need better icons, and a key in a box on RHS.
  • maybe highlight two-tick entries with a particular colour, but the open knowledge badge is horrendous, and breaks visual cadence.
  • results data should be tabulated, with the project names and descriptions made most prominent. <li> blobs are unnecessary visual noise.
  • maybe links should go on the full titles? Get rid of package names.
  • Should have filter by name (à la KForge alphanum widget) and also by tag: list of most common tags in box on RHS?
  • same comments as search listing (ticket:217)
1260879903000000 1263406033000000
#219 enhancement nickstenning dread closed wontfix UI Review - Package
  • Propose three-column package listing: two lists of highest-scorers on some metric and last column is an invitation to register a new package, with a little copy explaining what's involved and why you'd want to.
  • Add in search widget
1260879947000000 1294914901000000
#220 enhancement nickstenning dread v0.11 closed fixed UI Review - Package read

Package page package/mypackagename

  • Design
    • In general: a huge amount of visual and typographic noise: "Package:" is redundant.
    • "{edit}", "{history}" -- ugly, and it's not clear that these are primary actions on this page. Notably, why on earth is "make an enquiry with Is It Open?" bigger than these?
    • Openness/downloadability icons: need help, as discussed earlier
    • "Rating:" redundant, as is the second set of stars -- why can't both the display and rate widget
      • Rating - alternative to GET on set_rating.
    • "Metadata:" technospeak, probably redundant.
    • <li> bullets add nothing.
    • Title, Version, URL can probably go in one line.
    • Resources can be made more prominent.
    • Author, author email and Maintainer, maintainer email can be merged.
    • Groups, tags in little boxes?
    • Notes made more delineated -- Markdown has the capacity to look like other parts of the page: perhaps monospace the typeface?
    • Don't show extras unless there are any.
1260880136000000 1265286499000000
#221 enhancement dread dread v1.0 closed fixed UI Review - Package edit

Package edit page package/edit/mypackagename

  • Ability to add the package to groups on this page.
  • Fields could be grouped.
  • Inconsistent capitalisation (Url -> URL).
  • Tag editor YUI script appears to break occasionally.
  • Perhaps move to two column forms to save vertical space?
  • Notes field to monospace?
  • Extras fields need some jQuery love -- shouldn't have a fixed number available.
  • Typography is a bit of a mess.

1260880175000000 1271756757000000
#222 defect nickstenning dread closed fixed UI Review - Package history

Package history page package/history/mypackagename

  • "Revisions" is redundant
  • "Compare" button is in a ridiculous place. It should be above/below (both) the table, but not *in* it!
  • Ditto to overall "Recent Changes" page comments w.r.t. timestamps, GUIDs (ellipsis), and (if poss) author links.
1260880198000000 1282909280000000
#223 enhancement nickstenning dread v1.0 closed fixed UI Review - Tag

Tags landing page tag

  • Both search for tags *and* browse the list of tags right here
  • Big cloud of tags (sorted by popularity/usage?) filterable with a search box.
  • Tags search page tag/search and list tag/list are both redundant and should be merged straight into tag.
1260880276000000 1273596111000000
#224 enhancement nickstenning dread v0.11 closed fixed UI Review - Groups

Groups landing page group

  • Yeesh! What's up with the typography for "Just login..."?
  • Truncated description along with group name.
  • Explanation text about what groups are for (RHS)
  • Another case of <li> bullets adding nothing.
1260880360000000 1267100560000000
#225 enhancement rgrp dread closed invalid UI Review - swap URL order

Not so keen on URLS such as /package/edit/mypackagename. These seem to be task-oriented rather than resource oriented (which would be /package/mypackagename/edit), and unstable: there is a common root for /package/edit/mypackage and /package/history/mypackage which is not reflected in the URL structure.

1260880487000000 1311178276000000
#226 enhancement nickstenning dread v1.1 closed fixed UI Review - History

Repository History revision

  • What is this page called? "Recent Changes", "Repository History", "/revision": standardise. Between the link in the nav and the page <title> particularly, but the route is also important. Perhaps /changes or something similar?
    • Will change page title on /revision/ to Revision History. Will not change route for the time being.
  • needs copyediting.
  • Pagination has similar issues to elsewhere. Also, most obvious here, is the fact that we don't need to display a link to every possible page. Please can we limit it to, say, a dozen nearby pages and an ellipsis.
  • Without looking at dates, its not clear whether I'm seeing most recent or oldest changes. Change pagination to say "Older"/"More recent" rather than "Previous"/"Next". (wontfix: now have text saying we are showing most recent changes)
  • Table layout is pretty ugly (yes, I'm aware this is my fault).
    • wontfix - nothing better at the moment
  • Timestamps are horribly unreadable. At the absolute minimum get rid of the micros. Hover for more detail?
  • Can we link to an author page? Yes!
  • Atom feed should have a feed icon~~
  • Why are we adding another <link rel="alternate"> to this page? We already have one for recent changes on every page in the site and the one we're adding has a less descriptive title. Which is correct? Use that one as a feed for every page.
1260880515000000 1279130390000000
#227 enhancement nickstenning dread v1.1 closed fixed UI Review - General checks
  • There is huge inconsistency in the titles of pages across the site: to give one example "Edit Package:" vs "Package: mypkg [not linked]" vs "History - mypkg [linked to package page]"
    • Normalize and use terminology: "Data Package" - also change in navbar (but nowhere else for the time being). So hvae Data Package - mypkg, Data Package - mypkg - Edit etc
  • pagination is currently lost in amongst the results list. needs to be *much* more obvious, and should appear top and bottom, or at the very least at the bottom!
1260880552000000 1279192621000000
#228 enhancement rgrp rgrp closed duplicate Deal with duplicate packages

This needs to be thought out ...

1262085763000000 1290596875000000
#229 enhancement dread dread v0.11 closed fixed Resource hashes

New field for resources - hash of the resource file.

  • CKAN itself will not calculate the hash value - user just pastes it in.
  • Display text field in resource table.
  • Requires migration script.

Questions for the field's value:

  1. Which hash to use? Restrict to python hashlib and other major hash libraries.
  1. Should we use merkle trees?

Thanks to Julien D'Assanges for the suggestion.

1262686287000000 1265891612000000
#230 task dread dread v1.0 closed fixed Refactor 'package preview' to use 'package read'
  • Requires Package Read to not use the side-bar
  • Need to sync params to a package object that is taken out of the session so that it doesn't go into the database.
1262788084000000 1265293953000000
#231 defect pudo rgrp ckan-v1.3 closed fixed Make group versioned

Make group object revisioned/versioned.

  • Involves upgrade script both for group object and group_revision object
  • No changes required to WUI for present
1262853923000000 1288516929000000
#232 enhancement rgrp rgrp closed invalid Add strapline to CKAN logo

Strapline (?): Sharing, discovery and reuse of data and content

Need to consult on this.

Cost: 1h

1262975291000000 1297246297000000
#233 enhancement rgrp rgrp v1.1 closed fixed Allow simple site-specific customization/overriding of templates

Options:

  1. Allow for specification of genshi template paths to search in config. This way people can introduce their own templates and these templates can selectively override existing templates. (Already implemented this in shakespeare and it works)
  1. Include an extra site-specific genshi template which can then be used to customize site (e.g. by having specific calls to py:def that user can define but which are ignored if they don't exist).

Can do this using:

<xi:include href="base.html"><xi:fallback /></xi:include>

1263000396000000 1273080352000000
#234 enhancement memespring rgrp closed wontfix UI Review - Autocomplete package names & tags in search

Broken out from ticket:216

1263056678000000 1338206486000000
#235 enhancement tobes dread ckan-v1.9 assigned Resource format normalization and detection

Try to gather proper MIME information for all package resources in CKAN. This is a shared ticket with dcat-tools (https://bitbucket.org/pudo/dcat-tools), i.e. opendatasearch.org. This can then also be used by ckanrdf, the CKAN RDF conversion service.

Sub-tasks:

  • Create a Google Spreadsheet with two Worksheets: "MIME-Mappings", i.e. "CSV" -> "text/csv" and "Name mappings", i.e. "text/csv" -> "Comma-Separated Spreadsheet".
  • Collect and map surface forms from all CKANs
  • Access this via Swiss and apply, store as a PackageResource? extra field pending #826 (Resource extras).
  • Add heuristics for format auto-detections:
    • Map well-known file extensions
    • Recognize obvious magic (Zip, Tar)
    • Peek into Zipfile/Tarfiles?
  • Define a convention for generic data types (many CKAN packages have only "Spreadsheet" defined, either detect specific type or set MIME to */tabular-data or similar)
  • See also: #816 (Autocomplete for the resource format field)
1263827604000000 1340627624000000
#236 enhancement johnbywater johnbywater closed fixed The system shall present to all users a form for starting data openness enquiries 1264157100000000 1264157324000000
#237 enhancement johnbywater johnbywater closed fixed The system shall accept submissions from authenticated owners of activated accounts using the data openness enquiry form by presenting the enquiry summary and prompting for confirmation 1264157377000000 1264157420000000
#238 enhancement johnbywater johnbywater closed invalid The system shall accept submissions from unauthenticated users of the data openness enquiry form by creating an anonymous pending enquiry action, and by redirecting the user to login form, passing a reference to the pending action 1264157814000000 1266509806000000
#239 task dread dread v1.0 closed fixed Test migrate scripts

Integrate new test upgrade into testmigrate stuff.

New "migrate.ini" config file programmed with the db to be used for migrate tests and a param saying where the db dump is.

1264439926000000 1265286521000000
#240 defect johnbywater rgrp v1.0 closed fixed Do not allow creation of PackageResource object without a url

Currently it is possible (v0.11) to have a PackageResource? object without a url. This should be impossible to do (either at creation time or via modification).

1265399747000000 1272383770000000
#241 defect rgrp dread v1.0 closed fixed License doesn't preview correctly

4 failing tests, including 2 in misc/package_saver and 2 in functional/test_package.

1265625546000000 1270569769000000
#242 defect pudo rgrp v1.1 closed fixed Miscellaneous tidying up (v0.11)

Dumping ground for various minor and miscellaneous items (mainly refactorings):rgrp

  • controllers/tag.py: why does this not use lib/search.py (but controllers/packages.py search method does)?

done:

  • lib/search.py: inheritance would be nicer than switching on entity in search (i.e. have SearchPackage?, SearchTag? etc)
1265715103000000 1280823876000000
#243 defect rgrp dread closed worksforme Repetition of number of packages in a group

http://ckan.net/group/publicdomain has:

"There is 16 package in this group. There are 16 packages in this group."

1265750250000000 1266837796000000
#244 defect rgrp dread closed fixed Several links preceded by link URL

e.g. Text on ckan.net home page:

"All material available under an /licenseopen license"

Also seen on the package edit page: "/user/loginClick here to sign in"

I believe this is due to the i18n additions from Benoit. Maybe needs new Genshi. Maybe best to work round in the meantime.

1265750419000000 1271248968000000
#245 enhancement rgrp rgrp closed duplicate Support for composite primary keys

Problem here is that foreign key then becomes "complicated" (composite).

  • Could also deprecate continuity_id field in favour of the basic foreign key on ie
1265882630000000 1297066620000000
#246 enhancement rgrp rgrp closed duplicate Support for primary key not named id

At the moment setting of continuity_id depends on base table pkcol being id. Should not be hard to change this -- and may get for free as part of ticket:245 (composite primary keys)

1265882862000000 1297066757000000
#247 enhancement dread rgrp v1.0 closed fixed Alternative formats links on package page

"Alternative formats". Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f).

  • How do we know if RDF available? Ans: see details in ticket:90.

Cost: 1h

1265890326000000 1271248813000000
#248 enhancement dread rgrp v1.0 closed fixed i18n of forms

Need to do i18n of forms (not in templates but in code). Should consult with tonfa about this.

1265892313000000 1270745304000000
#249 enhancement rgrp dread closed invalid Regex search

Search of package name and title (and other fields) using regular expressions.

Current example use-case: Wanting to specify packages with title beginning with 'B'.

Issues:

  • Syntax for specifying regex over natural language search - could it be contained in the q param so be available to users of the WUI, or do we simply make it alternative fields?

Implementation:

  • Postgres reg ex searching detailed here:

http://www.postgresql.org/docs/current/static/functions-matching.html#FUNCTIONS-POSIX-TABLE

1265994509000000 1311182450000000
#250 enhancement icmurray dread ckan-v1.9 assigned RDF link in Atom feed

Add link to RDF representation of a package in our Atom feed.

1266507695000000 1340631430000000
#251 enhancement johnbywater johnbywater v1.0 closed fixed Present in the API a list of revision IDs, and present individual revisions when referenced by ID

Want to have:

  • /api/search/revision/

Support querying by "since time" and "since revision". Each individual revision should be available in the REST API (read-only):

  • /api/rest/revision/{revision ID}
    • should include revision attributes plus revision.packages e.g. like { "id": "...", ... "packages": [ list of packages names from e.g. revision.packages ] }
1266519743000000 1271423917000000
#252 enhancement dread johnbywater closed invalid Change revision object so that it has parent(s) attribute 1266519767000000 1296477560000000
#253 enhancement dread ckan-backlog assigned Package relationships

Overview

Functionality to formally associate packages. We see a need for specific parent-child, inheriting or dependency relations. Not only should this help navigation between packages in the web interface, but it also provides a mechanism to automatically pull dependencies when downloading a data package, in a similar manner as we see in software package management.

Examples

  1. There are 27 packages in data.gov.uk to do with the Data4NR's Health Poverty Index. There is currently no common link between these, unless you search for 'HPI' (which also brings up House Price Index), or look under tag 'health' (which also has 600 other results). There should be a link on each HPI package page to navigate to the other 'sibling' HPI packages, and to a 'root' package that has info about the set. This could be partially achieved using the existing tag or group concepts, but a more explicit/official/obvious marking of their relationship could be beneficial.
  1. In ckan.net is freedict, a collection of translation dictionaries. You could make each dictionary a child package and use this system. But it would probably be better to make each dictionary a different resource in the same package. (There are other ideas to denote a resource as the data making up a 'portion' of package, or a 'whole' of the package, to help people downloading datasets in the software package style.)
  1. OSM has had some Naptan data imported (bus stops), with special permission - i.e. a more liberal license. It would be useful to show this link on both OSM and Naptan packages in CKAN: OSM 'derives from' Naptan with a comment about the license change. I'm not sure this is useful to an automatic download or use of these datasets, but may aid exploration on the CKAN website and understanding the provenance of the bus stop data on it.
  1. IPCC collection of data linked / mirrored. Not sure if there are useful relationships here?
  1. Dracos gets postbox locations from crowd sourcing and OSM. We could say Dracos 'derives from' OSM.

See more examples discussed here: http://trac.ckan.org/ticket/253

Implementation

This is split into four tickets:

No need for write access to be provided API for the moment.

This ticket also encompasses ticket:169 (Package derivations) and ticket:176 (Package dependencies).

1266854721000000 1339774726000000
#254 enhancement dread dread v1.0 closed fixed Package relationships - 1. model

New domain object: PackageRelationship? (revisioned)

Attributes:

  • subject (Package reference)
  • object (Package reference)
  • type (string)
  • comment

Relationship type values: depends_on (dependency_of) derives_from (has_derivation) child_of (parent_of)

Relationship type is stored as a string, as given in the first column. The reverse relationship (bracketed) is given just for display purposes only.

Subject / Object - as in a sentence: "Dracos is derived from OSM" is <subject> <predicate> <object>. We'll use 'type' instead of 'predicate' as that is a more familiar word.

1266928449000000 1273596180000000
#255 enhancement dread dread v1.0 closed fixed Package relationships - 2. Read in WUI

WUI:

  • View: show both sides of the relationship (but think carefully -- e.g. a given package may have *many* dependents ...)
1266928542000000 1273596174000000
#256 requirement dread ckan-backlog assigned Package relationships - 3. Edit in WUI

WUI:

  • Editable as part of package or separately? (e.g. like authz)
  • Do we normalize to only one type name of the pair?
  • Do we allow create of relationship from both ends (e.g. only from dependency to dependent or either way?)
1266928561000000 1339774714000000
#257 enhancement dread dread v1.0 closed fixed Package relationships - 4. Read in API

API:

  • Appear in package listing Example: 'relationships': [{'is_dependency_of':'osm', comments:'Since version 0.2'}, {'is_parent_of':'bobs_maps'}]
1266928630000000 1273596170000000
#258 enhancement rgrp dread v1.0 closed fixed Add uuids into package in REST

Add uuids into package in REST interface.

1266954722000000 1273596163000000
#259 defect rgrp dread v1.0 closed fixed isitopen enquiry broken

Pressing the 'make an enquiry' link on the package read page gives 404. (Also links should also be changed from isitopen.ckan.net to isitopendata.org)

Example link is:

http://isitopen.ckan.net/enquiry/create/?ckan_package=zeno

which gets forwarded to:

http://www.isitopendata.org/enquiry/create/?ckan_package=zeno

which gives 404

1267012666000000 1273596153000000
#260 enhancement dread rgrp v1.0 closed fixed Display group package list in standard (richer) way

Currently use old "bullet-point" style. Should change to now standard style used on e.g. package search results page.

Cost: 0.5h

1267100660000000 1267122137000000
#261 defect dread rgrp v1.0 closed fixed License does not work through REST API

Presence of license_id field in JSON input show below breaks REST API (with weird error about int not being subscriptable as if license_id field is being subscripted).

Using normal license field does not cause breakage but has no effect.

(Also think we should remove license_id from the API since there is no guarantee this cross system compatible since locally generated on install).

    ourjson = '''{
   "maintainer" : "Applications Division",
   "version" : "1999-01-01",
   "license_id" : 9,
   "name" : "geogratiswaterfraction1km_v100",
   "author_email" : "...",
   "author" : "Dissemination Branch",
   "tags" : [
      "GeoGratis",
      "Canada",
      "GCMD",
      "Hydrosphere",
      "Ground Water",
      "Surface Water",
      "Hydropattern",
      "Earth Science",
      "Land Surface",
      "Land Cover",
      "Landscape"
   ],
   "extras" : {
      "License" : "http://geogratis.ca/geogratis/en/licence.jsp",
      "Ministry" : "Natural Resources Canada",
      "Level of Government" : "Federal"
   },
   "maintainer_email" : "[email protected]",
   "notes" : "This product is a raster coverage",
   "url" : "http://geogratis.cgdi.gc.ca/geogratis/en/collection/metadata.do?id=67",
   "title" : "Geogratis - 1-km Water Fraction From National Topographic Data Base Maps, Canada"
}
'''
    def test_15_random(self):
        offset = '/api/rest/package'
        import urllib
        tmpjson = simplejson.dumps(simplejson.loads(self.ourjson))
        postparams = urllib.urlencode({tmpjson: 1})
        res = self.app.post(offset, params=postparams, status=[200],
                extra_environ=self.extra_environ)
        model.Session.remove()
1267134778000000 1267207776000000
#262 enhancement rgrp rgrp closed fixed Mark package as duplicate of another package

Need a way to mark packages as duplicate.

Ideas/Questions?:

  • Could work off a tag (meta-duplicate or somesuch)
  • What happens if we want to delete the package (to we leave a place-holder, do we merge content?)
1267139682000000 1296468392000000
#263 enhancement johnbywater rgrp v1.1 closed fixed Improve and test openid login

At the moment the user has to figure out to login with providers -- this is not always obvious (e.g. google).

We could improve this with a small bit of javascript. As an example see: http://standalone.demo.civicrm.org/standalone/

Not sure exactly what library that is using (or if bespoke) so alternatives include (NB: we already use jquery):

In addition to implementing this we also need to test logging in with main providers: google, wordpress -- as we have had reports of it not working with e.g. wordpress (not sure if this testing can be automated really -- best hope would be selenium I think -- so OK if done by hand).

Cost: 3h (2h UI), (1h testing)

1267605039000000 1279292730000000
#264 defect dread rgrp v1.0 closed fixed Investigate why there are revisions with no associated packages

Revision.packages should not include all changes so why are we getting revisions with no associated changes? See e.g. http://ckan.net/revision/read/814e39e1-2540-4449-90d1-ac9a92ab40fb?

Investigate and fix.

NB: if this is bugfix this should go against stable or metastable branch if at all possible and then be merged into default.

Cost: 1.5h (?)

1267605428000000 1272390013000000
#265 enhancement johnbywater dread v1.0 closed fixed More detail shown in Atom feed

Use case

As a user I want to stay abreast of package changes, such as a new package being created, newer data is available for a package or a new download is available for a package.

Implementation

Add into the Atom feed:

  1. a package is a new one
  2. the resources have been changed
  3. the last_updated field has been changed
1267708364000000 1271636891000000
#266 enhancement johnbywater dread v1.0 closed fixed Package feed

As a user I want to track changes to a particular package.

Provide an Atom feed, linked from the package read page.

1267716417000000 1271636901000000
#267 defect johnbywater rgrp vdm-0.7 closed fixed Diffs for object that is created in that revision

Currently this raises an exception because there is no package/package_revision in existence at from_revision.

1268041383000000 1272960518000000
#268 defect rgrp dread closed duplicate Select groups in Package edit form 1268068896000000 1285070682000000
#269 enhancement rgrp dread closed fixed Improve gov package form

If the notes field could use a WYSIWYG editor with word cleanup this would really help users who may well be pasting in text from Word (and I guess may have been approved or written by someone else)

having auto-complete on tags would both make the system easier to use - and reduce the risk of synonymous tags being created inadvertently.

Does the department drop down options list interact with user permissions - so that users who only have rights to one department only see that option ?

Similarly for licenses - can this reduce to the set of allowed options for this user/

What level of validation is there on the fields?

1268220853000000 1291897538000000
#270 enhancement johnbywater johnbywater closed fixed Refactor ckan/lib/importer and ckanclient/loader.

There is code to create packages in-process (ckan/lib/importer) and via HTTP (ckanclient/loaders/base). There is also code to read different kinds of package source (Google Spreadsheets spreadsheet, other spreadsheet).

Propose to refactor all this to have template method in base command class (subclasses may implement command line interface, graphical user interface, or Web interface), where the template method calls on aggregated strategy objects for: reading raw spreadsheet data from spreadsheet application; for inferring CKAN package entities from the raw spreadsheet data; and for writing package entities to CKAN service.

Spreadsheet readers to be factored to allow for variations of spreadsheet application.

Raw data inferrers to allow for variations in spreadsheet cell structure.

CKAN writer strategies to allow for either CKAN client or direct linking.

Actual cases then supported by specifying existing strategies (or defining new ones) in a script that invokes the command class.

1268671804000000 1290596640000000
#271 enhancement johnbywater johnbywater v1.0 closed fixed The system shall support distributing changes.

See DistributingChanges for analysis and functional requirements.

1268672137000000 1273596144000000
#272 enhancement rgrp rgrp closed fixed Atom/RSS feeds for individual tags and groups

I want to be able to easily check when a package is added (or removed?) from a given tag or groups. To do this I would like access to an Atom/RSS feed for a given tag or group.

  • This would allow people to embed feed widgets in other places
1268988652000000 1340631775000000
#273 task pudo rgrp v1.1 closed fixed Investigate search index options and create tickets

Work already here: http://knowledgeforge.net/ckan/trac/wiki/SearchEngine

Add knowledge there or to:

http://wiki.okfn.org/SoftwareTools/Search

1268996933000000 1278578527000000
#274 defect rgrp rgrp closed fixed Allow searching by any extra field

At the moment only support a very limited set of extra fields (see docs for details). Should support arbitrary extra fields (note no need to put these in special full text index though ...)

Cost: 1h

1269034562000000 1287402800000000
#275 defect dread rgrp closed fixed Add forms renderer for JsonType fixing bugs in admin interface

This will solve e.g. issues in formalchemy admin interface (which is due to lack of renderer for Json type)

Cost: 0.5h

from ckan.model.types import JsonType

forms.FieldSet.default_renderers[JsonType] = formalchemy.fields.TextFieldRenderer
1269122740000000 1281002082000000
#276 defect dread rgrp v1.0 closed fixed Resources in Package form seen multiple times upon preview

Create a new package with a name 'blah' and resource format 'blah'. Hit preview. There is an error because of the lack of resource url, but in the resource input boxes, there are now four resources with format 'blah'.

1269255399000000 1272996237000000
#277 enhancement zephod dread ckan-backlog assigned Set some config options / settings in WUI (extension)

Use case

As a ckan administrator I want to easily change options about the CKAN install.

Implementation

Settings to be in DB

Suggested:

## Title of site (using in several places including templates and <title> tag
ckan.site_title = CKAN

## Logo image to use (replaces site_title string on front page if defined)
ckan.site_logo = http://assets.okfn.org/p/ckan/img/ckan_logo_box.png

## Site tagline / description (used on front page)
ckan.site_description = 

## Used in creating some absolute urls (such as rss feeds, css files) and 
## dump filenames
ckan.site_url =

## Favicon (default is the CKAN software favicon)
ckan.favicon = http://assets.okfn.org/p/ckan/img/ckan.ico


## An 'id' for the site (using, for example, when creating entries in a common search index) 
## If not specified derived from the site_url
# ckan.site_id = ckan.net

## API url to use (e.g. in AJAX callbacks)
## Enable if the API is at a different domain
# ckan.api_url = http://www.ckan.net

## html content to be inserted just before </body> tag (e.g. google analytics code)
## NB: can use html e.g. <strong>blah</strong>
## NB: can have multiline strings just indent following lines
# ckan.template_footer_end = 

NB: these will still need to be stored somewhere for loading on initialization. do this in db init function ...

Settings / Options / KeyValues? Table

Columns:

  • [namespace]: ? only if KeyValues? (for settings this would then always be settings)
  • key
  • label
  • value (json)
  • type (e.g. date and to specify in advance what type should be)
  • description
  • tags: ?? (for grouping ...)

Loading settings from DB

Do this in ckan/config/environment.py

WUI

  • /ckan-admin/settings
  • Show label, plus description plus text field

Depends

  • Would be part of ckan-admin section and hence build on ticket:833 (Administrative dashboard)
1269274861000000 1318247121000000
#278 defect dread dread v1.0 closed fixed Spaces in extra field keys

If the key to an extra field has a space in it then various form code doesn't work. Either need to disallow spaces or cope with them.

1269520845000000 1271173752000000
#279 enhancement johnbywater dread v1.0 closed fixed Ensure package's license is an option when it's edited

With forms configurable to show a subset of licenses and ckan instances moving packages from server to server, there is a chance that a user will edit a package and the existing package's license is not displayed in the form. If the user submits this form without touching the license dropdown, then the value is lost.

So for this case we should add the package's license to the list displayed.

1270205931000000 1272451384000000
#280 defect dread dread v1.0 closed fixed Disappearing extra fields

When I edit a package on no.ckan.net with lots of extra fields, lots of them completely disappear.

Metastable branch.

1270209684000000 1271173769000000
#281 enhancement dread dread v1.0 closed fixed Refactor forms to be plugin-able 1270542248000000 1270723675000000
#282 enhancement thejimmyg dread closed wontfix Provide diverts when package name changes

When a package's name is changed, references to the old name (in the WUI and REST) are redirected to the new name (assuming the old name is not being reused by another package).

1270659003000000 1338206417000000
#283 enhancement rgrp dread closed wontfix Manage deletions of unwanted packages

Use case

As a user I want to notify the CKAN admins of a spammed or unsuitable package for deletion.

Suggested solution

In the package view side-bar, there is a note: "To have this package completely removed, contact the [ca.ckan.net administrators [email protected]…]."

Other solutions

A more complicated solution would be to allow packages to be tagged for deletion, which would auto-alert administrators, and allow easier administration of this. But this might be overkill.

c.f. http://en.wikipedia.org/wiki/Deleting_an_article

1270660210000000 1311325526000000
#284 defect johnbywater rgrp v1.0 closed fixed Move tag/read/xxx to simple tag/xxx like package and group

Also put in redirect from old locations (see examples with plurals in routing.py)

Estimated cost: 20m

Actual cost: 40m

1270664552000000 1271852451000000
#285 enhancement rgrp assigned Paginate list of packages on tag read page

Is this worth doing? On hmg.ckan.net start to have a lot of packages with a given tag ...

1270664606000000 1340631923000000
#286 enhancement dread dread v1.0 closed fixed Refactor forms templates

Use Nick's examples as a basis to revamp the formalchemy usage, principally in the package edit form.

1270723513000000 1270723629000000
#287 enhancement dread dread v1.0 closed fixed Customise Canadian package edit form

Spec - see attached, which is a copy of the Pirate Pad from Lauren: http://piratepad.net/2C2iwiLDhd

1270801086000000 1270801210000000
#288 defect dread dread v1.0 closed fixed Foreign and escaped characters in extras fields

These don't work.

1271171899000000 1271173777000000
#289 enhancement rgrp dread v1.0 closed fixed Document config options 1271248690000000 1271249368000000
#290 defect johnbywater dread v1.0 closed fixed Revision API - docs

doc/api.rst needs to cover the new Revision REST interface.

1271268759000000 1271636910000000
#291 defect rgrp dread closed worksforme Investigate search options encoding

On ckan.net there is this unchecked-in 'temporary hack'. It seems to be to do with foreign characters in search options. See what its doing and if necessary, put check it in.

diff -r 813ad8b5de0b ckan/lib/search.py
--- a/ckan/lib/search.py	Mon Mar 01 22:23:36 2010 +0100
+++ b/ckan/lib/search.py	Thu Apr 15 19:11:42 2010 +0200
@@ -63,6 +63,9 @@
     def query(self, options):
         '''For the given search options, returns a query object.'''
         self._options = options
+        # temporary hack!
+        if self._options.q:
+            self._options.q = self._options.q.encode('utf8')
         general_terms, field_specific_terms = self._parse_query_string()
 
         if not general_terms and \

1271351670000000 1273254895000000
#292 defect dread dread v1.0 closed fixed PackageResources are created new on every package edit

When you edit a package, all of its resources are deleted and a fresh set created. This causes all history of them to be lost.

1271755964000000 1272286005000000
#293 defect johnbywater dread v1.0 closed fixed REST PackageResource update causes exception

This is caused by editing a package with resources over the REST interface.

Exception:

URL: http://de.ckan.net/api/rest/package/destatis
Module weberror.errormiddleware:162 in __call__
<<              __traceback_supplement__ = Supplement, self, environ
                   sr_checker = ResponseStartChecker(start_response)
                   app_iter = self.application(environ, sr_checker)
                   return self.make_catching_iter(app_iter, environ, sr_checker)
               except:
>>  app_iter = self.application(environ, sr_checker)
Module repoze.who.middleware:107 in __call__
<<          wrapper = StartResponseWrapper(start_response)
               app_iter = app(environ, wrapper.wrap_start_response)
       
               # The challenge decider almost(?) always needs information from the
>>  app_iter = app(environ, wrapper.wrap_start_response)
Module beaker.middleware:73 in __call__
<<                                                     self.cache_manager)
               environ[self.environ_key] = self.cache_manager
               return self.app(environ, start_response)
>>  return self.app(environ, start_response)
Module beaker.middleware:152 in __call__
<<                          headers.append(('Set-cookie', cookie))
                   return start_response(status, headers, exc_info)
               return self.wrap_app(environ, session_start_response)
           
           def _get_session(self):
>>  return self.wrap_app(environ, session_start_response)
Module routes.middleware:130 in __call__
<<                  environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1]
               
               response = self.app(environ, start_response)
               
               # Wrapped in try as in rare cases the attribute will be gone already
>>  response = self.app(environ, start_response)
Module pylons.wsgiapp:125 in __call__
<<          
               controller = self.resolve(environ, start_response)
               response = self.dispatch(controller, environ, start_response)
               
               if 'paste.testing_variables' in environ and hasattr(response,
>>  response = self.dispatch(controller, environ, start_response)
Module pylons.wsgiapp:324 in dispatch
<<          if log_debug:
                   log.debug("Calling controller class with WSGI interface")
               return controller(environ, start_response)
           
           def load_test_env(self, environ):
>>  return controller(environ, start_response)
Module ckan.lib.base:51 in __call__
<<          # available in environ['pylons.routes_dict']
               try:
                   return WSGIController.__call__(self, environ, start_response)
               finally:
                   model.Session.remove()
>>  return WSGIController.__call__(self, environ, start_response)
Module pylons.controllers.core:221 in __call__
<<                  return response(environ, self.start_response)
               
               response = self._dispatch_call()
               if not start_response_called:
                   self.start_response = start_response
>>  response = self._dispatch_call()
Module pylons.controllers.core:172 in _dispatch_call
<<              req.environ['pylons.action_method'] = func
                   
                   response = self._inspect_call(func)
               else:
                   if log_debug:
>>  response = self._inspect_call(func)
Module pylons.controllers.core:107 in _inspect_call
<<                        func.__name__, args)
               try:
                   result = self._perform_call(func, args)
               except HTTPException, httpe:
                   if log_debug:
>>  result = self._perform_call(func, args)
Module pylons.controllers.core:60 in _perform_call
<<          """Hide the traceback for everything above this method"""
               __traceback_hide__ = 'before_and_this'
               return func(**args)
           
           def _inspect_call(self, func):
>>  return func(**args)
Module ckan.controllers.rest:145 in update
<<                  fs = ckan.forms.group_fs_combined
                   fs = fs.bind(entity, data=request_fa_dict)
                   validation = fs.validate_on_edit(entity.name, entity.id)
                   if not validation:
                       response.status_int = 409
>>  validation = fs.validate_on_edit(entity.name, entity.id)
Module ckan.forms.package:385 in validate_on_edit
<<              temp_name = orig_pkg_name
                   self.data['Package-%s-name' % record_id] = u'something_unique'
               validation = self.validate()
               if temp_name:
                   # restore it
>>  validation = self.validate()
Module formalchemy.forms:175 in validate
<<          if self.readonly:
                   raise Exception('Cannot validate a read-only FieldSet')
               return AbstractFieldSet.validate(self)
       
           def sync(self):
>>  return AbstractFieldSet.validate(self)
Module formalchemy.forms:104 in validate
<<          success = True
               for field in self.render_fields.itervalues():
                   success = field._validate() and success
               # run this _after_ the field validators, since each field validator
               # resets its error list. we want to allow the global validator to add
>>  success = field._validate() and success
Module formalchemy.fields:763 in _validate
<<          try:
                   value = self._deserialize()
               except validators.ValidationError, e:
                   self.errors.append(e)
>>  value = self._deserialize()
Module formalchemy.fields:1354 in _deserialize
<<          if self.is_composite_foreign_key:
                   return self.query(self.relation_type()).get(python_pk(self.renderer.deserialize()))
               return self.renderer.deserialize()
>>  return self.renderer.deserialize()
Module formalchemy.fields:183 in deserialize
<<          if self.field.is_collection:
                   return [self._deserialize(subdata) for subdata in self._serialized_value()]
               return self._deserialize(self._serialized_value())
       
           def _deserialize(self, data):
>>  return self._deserialize(self._serialized_value())
Module formalchemy.fields:167 in _serialized_value
<<          if self.field.is_collection:
                   return self._params.getall(self.name)
               return self._params.getone(self.name)
       
           def deserialize(self):
>>  return self._params.getone(self.name)
Module formalchemy.base:63 in getone
<<          if v is None or isinstance(v, basestring) or isinstance(v, cgi.FieldStorage):
                   return v
               return v[0]
           def getall(self, key):
               v = dict.get(self, key)
>>  return v[0]
TypeError: 'int' object is unsubscriptable

Thanks to Friedrich for spotting this.

1271756102000000 1271940083000000
#294 enhancement thejimmyg dread closed duplicate Add/remove extra fields in Package edit form

Currently the package form gives you 3 fields for extras. To get more you have to hit preview. This is obscure. It would be better to have some buttons to add/remove fields, just like with the resources.

1271756591000000 1291830960000000
#295 enhancement dread dread v1.0 closed fixed Add multiple packages when editing a group

In Group edit page, you can only add one package at a time. It would be better to use some scripting to allow adding more than one at once, to make it easier and quicker when you have a long list to add.

Ticket is split off from ticket:221

See also ticket:155 (duplicate).

1271756713000000 1272384758000000
#296 enhancement johnbywater johnbywater closed duplicate Commit CKAN revisions to changeset system 1272279521000000 1294407032000000
#297 enhancement johnbywater johnbywater closed duplicate Update CKAN repository from changeset system 1272279556000000 1294407051000000
#298 enhancement johnbywater johnbywater closed duplicate Pull changesets from remote CKAN instance 1272279591000000 1294407080000000
#299 enhancement johnbywater johnbywater closed duplicate Merge diverging lines of changesets 1272279698000000 1294407099000000
#300 defect rgrp dread v1.0 closed fixed Resource ordering issue

Failing test: ckan.tests.models.test_resource.TestResourceLifecycle?.test_03_reorder_resources

Not clear how visible this is to the user.

Related to ticket:292

1272285994000000 1272384474000000
1 2 3 4 5 6 7 8 9 10 11 12 13
Note: See TracReports for help on using and creating reports.