Search of form: tags: ... behaves differently depending on whether there is a leading space:

• tags: postcode - tags found correctly but no packages found
• tags:postcode - tags incorrectly found but correct packages found

Let's fix this.

Cost: 0.5h

Sister to ticket:90 (Link to RDF version of CKAN data on Talis Connected Commons).

Talis have already kindly done an initial conversion. We should repeat this process regularly and re-upload the data to Talis CC.

In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required.

Attached is the ruby script used by Talis for conversion

Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)

Should have function/facility to get a diff of a version object between 2 revisions. Function should return a dictionary of fields with diffs.

Details:

• Do we deal with m2m relationships (and m2one on the many side)?
• What is diff?
  • For text fields output of python diff command
  • For non-text field guess just simple +/- for what was there before and now

Cost: 4h (don't think this is a huge request)

As A

sysadmin

I Want To

Have revisions to the CKAN data filtered in order to reduce the spam in the system.

Details

In the long run this is a quite a generic problem common across several OKF systems and probably can become a general component in the okfmisc repo. For time being focus on a well-factored CKAN-specific solution.

Suggest we follow path of trac: ​http://trac.edgewall.org/wiki/SpamFilter

Could have a general engine that aggregates spam scores from many different 'plugins' and then marks spam appropriately (actions should be configurable depending on spam level from 'purge' to 'delete' (mark revision as inactive) to 'flag' to 'do nothing').

Main initial plugins would be:

• regex filter (this would seem very useful here, e.g. do not allow urls in commit messages ...)
  • could augment using the badcontent list approach (can find list on e.g. moinmoin)
• spambayes and/or akismet

Use python sphinx to build documenation in ./doc and then upload it somewhere publicly accessible.

NB: improving the documentation is another matter (as is integrating e.g. existing api docs).

Upload location (these are docs for CKAN codebase/concept not the ckan service at ckan.net so good not to associate it too closely with ckan.net):

• ​http://knowledgeforge.net/ckan/ckan/doc/
• (Other options: ​http://www.ckan.net/doc/ ​http://doc.ckan.net)

Split out from ticket:43

Multiple download links, including links to mirrors and multiple formats/versions

Where an item has an association listing (e.g. package's tags or group's packages) these should be sorted in a sensible way. For present choose to sort alphabetically by name.

Cost: 2h

As a

admin / user

I want to

Admin: export all HEAD data to JSON / CSV / RDF format dump.

User: wants to get hold of this dump.

Details

• Just do JSON & CSV for now.
• Put this in CLI.
• Just do list of packages - as_dict info. CSV needs a bit of flattening.

Cost: 2h

p:wui 2009-09-09 validate email address when creating an enquiry

Investigate and list (on the wiki at wiki:UiReviewNotes) CKAN's UI issues, in particular:

• Form usability
  • best example: ​http://ckan.net/package/new
  • "Extras" fields
  • Adding to groups (from group page or from package page?)

• Overview pages: reduction in visual noise and unnecessary elements.
  • best example: package overview page
  • consider openness/downloadability icons
  • display common metadata fields in a more structured fashion?

Est cost: 2 hrs.

• openness/downloadability ticks/crosses are totally inscrutable to non-techies (those who don't know about the title attribute). need better icons, and a key in a box on RHS.
• maybe highlight two-tick entries with a particular colour, but the open knowledge badge is horrendous, and breaks visual cadence.
• results data should be tabulated, with the project names and descriptions made most prominent. <li> blobs are unnecessary visual noise.
• maybe links should go on the full titles? Get rid of package names.
• Should have filter by name (à la KForge alphanum widget) and also by tag: list of most common tags in box on RHS?
• same comments as search listing (ticket:217)

Package search results page package/search

• need clearer delineation between search entry form and results.
• 0 results needs to be clearer, and the alternative suggestion: 'would you like to create a new package' needs to not look like a search result.
• "N tags found" -- why do I care? this should be presented as "filter results by tag" -- very unlikely that I'm looking for a tag: I'm looking for a package, but tags might help me narrow down my search.
• tags should be displayed more like the way they're displayed when you add them to a package (i.e. blue 'tag' background)
• filter out particular tags
• tag cloud?
• copywriting: "Packages - Search" -> "Search packages"

rel=nofollow or robots.txt ?

Based on number of views.

Related to ticket:143.

Package page package/mypackagename

• Design
  • In general: a huge amount of visual and typographic noise: "Package:" is redundant.
  • "{edit}", "{history}" -- ugly, and it's not clear that these are primary actions on this page. Notably, why on earth is "make an enquiry with Is It Open?" bigger than these?
  • Openness/downloadability icons: need help, as discussed earlier
  • "Rating:" redundant, as is the second set of stars -- why can't both the display and rate widget
    • Rating - alternative to GET on set_rating.
  • "Metadata:" technospeak, probably redundant.
  • <li> bullets add nothing.
  • Title, Version, URL can probably go in one line.
  • Resources can be made more prominent.
  • Author, author email and Maintainer, maintainer email can be merged.
  • Groups, tags in little boxes?
  • Notes made more delineated -- Markdown has the capacity to look like other parts of the page: perhaps monospace the typeface?
  • Don't show extras unless there are any.

'created' attribute on User, Group and Rating

datetime object: default=datetime.datetime.now()

See example on revision

Requires migrate script

• Requires Package Read to not use the side-bar
• Need to sync params to a package object that is taken out of the session so that it doesn't go into the database.

When viewing a group you see a list of the users who have admin rights to it. (Akin to owners / moderators)

'paster db upgrade' should result in the same database as 'paster db create' on a clean system. In particular, all tests should pass.

Note: equivalent of paster db upgrade isn't being run from tests at the moment (we haven't been able to get this to work).

Cost: 0.5h (?)

Conjecture this went missing in cset:a35db862a841

Suggest form is: key:value (with optional space after ':'.

Keys would be for an object, i.e. 'package', 'tag', 'group'.

Becomes a link.

This would be implemented by implementing a 'Parser' or 'Processor' object which is run before markdown is done. Regex replace.

Cost: 4h

Have a news section (suggest as a sidebar item).

News section will link to latest 3/4 blog posts on CKAN from blog.okfn.org.

Details:

• Suggest pulling via rss or similar.
• Will want to cache this ...

Cost: 4h?

Cost: 1h

Get rid of modes in the RESTful API. Do json stuff directly in controllers.

• package will have to_dict and from_dict methods (called by rest controller) which provide and consume JSON friendly dictionaries representing the object. from_dict - class_method. stuff in forms for tags as_string, maybe factor out. reuse validation stuff.

Obfuscate author email and maintainer email field.

Use: webhelpers.html.tools.mail_to e.g. webhelpers.html.tools.mail_to('mike@…', encode='javascript')

Do we want to? Surely only public addresses should be on CKAN in which case why obfuscate?

the "only openly licensed packages" tick box seems to return blank page every time - as do queries for only openly licensed and only downloadable packages. The "only downloadable packages" tick box by itself seems to work fine. E.g. try searching for 'science'.

Raised by Jonathan Gray

Only first tag is updated.

Related to: ticket:90 Link to RDF version of CKAN data on Talis Connected Commons ticket:164 Creating RDF data from CKAN

Regularly re-upload the CKAN RDF data to Talis CC.

In the long run may wish to only re-convert packages changed since the last upload. However given relatively smaller size of full dataset this optimization is probably not yet required.

Cost: ? (1d+ depending on e.g. how easy integration with Talis CC is)

New field for resources - hash of the resource file.

• CKAN itself will not calculate the hash value - user just pastes it in.
• Display text field in resource table.
• Requires migration script.

Questions for the field's value:

1. Which hash to use? Restrict to python hashlib and other major hash libraries.

2. Should we use merkle trees?

Thanks to Julien D'Assanges for the suggestion.

Standard search should search notes field in addition to name, title and tags (discussed in ticket:108 but not done). For this to work we need proper text search since o/w we get poor ordering and lots of bad results.

If we do this we need:

1. To weight across fields in a sensible way
2. We can also use proper text search on title or ...

Easiest way to do this is to use existing facilities in dbs e.g. postgres has full text support since 8.3: ​http://www.postgresql.org/docs/8.3/static/textsearch.html

Using this with sqlalchemy: ​http://lowmanio.co.uk/blog/entries/postgresql-full-text-search-and-sqlalchemy/

Issues with fulltext search:

• tags not indexed, so would need to 'or' search of tags. This would cause problems with the order_by of the query, since the tags wouldn't have a ranking.
• if tags are indexed then perhaps we don't want them converted into lexemes? Exact match could well be better.
• can we split the name on dash or underscore before being indexed?
• natural language search doesn't do partial words, so search for 'gov' doesn't bring up 'government'.
• do we keep the existing search system usable with a config file switch for if we install on a db aside from postgres?
• we want to weight name and title higher than other fields - achievable with custom trigger.

Native registration scenario: Complete registration form, require unique email address (indicate this not displayed on site), full name (indicate this is displayed on site), password. Dispatch email confirmation message. Respond to email confirmation message causes user to be logged in. Users are assigned a uuid.

Native login scenario: Complete login form, require email address. Any session is resumed.

Mandated technical requirement: use Repoze (see plugin:form of repoze.who); borrow from BlastOff?.

See CKAN or Microfacts for usage of repose with openid.

• 'Login via OpenID' -> 'Login with OpenID'?
• Elements:
  • Search
  • about CKAN
  • 3 column Packages view (see ticket:219)
  • Small tag cloud (RHS)

Also, what about ticket:143 ?

See how we did it already for other things.

Note: on ckan.net older PackageRevision?.id might not be identical to Package.id but this may need sorting at this point.

A 'Derived' relationship can be applied from one package to another.

e.g. sussex-demography is derived from census-2001

'Derived' relationship is:

• directional
• many:many
• stateful

'derived' table columns:

• id (primary key)
• source_package (foreign key)
• result_package (foreign key)
• description (markdown text)

Further tickets:

• WUI - package view - shows 'derives from package x' and 'derived package y' with UML-like diagram of x -> this package -> y
• WUI - package edit form - new option to say it 'derives from' or 'has derivation' and you select the appropriate
• REST if - expose reading and writing this property

Groups landing page group

• Yeesh! What's up with the typography for "Just login..."?
• Truncated description along with group name.
• Explanation text about what groups are for (RHS)
• Another case of <li> bullets adding nothing.

Presence of license_id field in JSON input show below breaks REST API (with weird error about int not being subscriptable as if license_id field is being subscripted).

Using normal license field does not cause breakage but has no effect.

(Also think we should remove license_id from the API since there is no guarantee this cross system compatible since locally generated on install).

    ourjson = '''{
   "maintainer" : "Applications Division",
   "version" : "1999-01-01",
   "license_id" : 9,
   "name" : "geogratiswaterfraction1km_v100",
   "author_email" : "...",
   "author" : "Dissemination Branch",
   "tags" : [
      "GeoGratis",
      "Canada",
      "GCMD",
      "Hydrosphere",
      "Ground Water",
      "Surface Water",
      "Hydropattern",
      "Earth Science",
      "Land Surface",
      "Land Cover",
      "Landscape"
   ],
   "extras" : {
      "License" : "http://geogratis.ca/geogratis/en/licence.jsp",
      "Ministry" : "Natural Resources Canada",
      "Level of Government" : "Federal"
   },
   "maintainer_email" : "[email protected]",
   "notes" : "This product is a raster coverage",
   "url" : "http://geogratis.cgdi.gc.ca/geogratis/en/collection/metadata.do?id=67",
   "title" : "Geogratis - 1-km Water Fraction From National Topographic Data Base Maps, Canada"
}
'''
    def test_15_random(self):
        offset = '/api/rest/package'
        import urllib
        tmpjson = simplejson.dumps(simplejson.loads(self.ourjson))
        postparams = urllib.urlencode({tmpjson: 1})
        res = self.app.post(offset, params=postparams, status=[200],
                extra_environ=self.extra_environ)
        model.Session.remove()

Research apt-get especially wajig and python wrappers for apt-get. Post results on the wiki.

As of 80:099bdc5b07a6 datapkg has 4 test fails and 1 test error.

There is no documentation on how to run them, so I am using 'python setup.py test'.

datapkg.tests.test_cli.TestCLI.test_ckan ... FAIL
datapkg.tests.test_cli.TestCLI.test_walkthrough ... FAIL
datapkg.tests.test_index.TestDbIndex.test_get ... /Users/james/Documents/virtualenvs/okfn/src/datapkg/datapkg/index.py:97: SADeprecationWarning: Use session.add()
  self.session.update(pkg)
FAIL
datapkg.tests.test_index.TestDbIndex.test_get_when_loaded_as_new_and_init_not_called ... /Users/james/Documents/virtualenvs/okfn/src/datapkg/datapkg/tests/test_index.py:57: SADeprecationWarning: Use session.expunge_all()
  self.index.session.clear()
FAIL

4 failing tests, including 2 in misc/package_saver and 2 in functional/test_package.

Spec - see attached, which is a copy of the Pirate Pad from Lauren: ​http://piratepad.net/2C2iwiLDhd

When I edit a package on no.ckan.net with lots of extra fields, lots of them completely disappear.

Metastable branch.

"Alternative formats". Has logo for RDF (link to RDF) and JSON (link via CKAN REST i/f).

• How do we know if RDF available? Ans: see details in ticket:90.

Cost: 1h