{22} Trac tickets (2647 matches)

Presence of license_id field in JSON input show below breaks REST API (with weird error about int not being subscriptable as if license_id field is being subscripted).

Using normal license field does not cause breakage but has no effect.

(Also think we should remove license_id from the API since there is no guarantee this cross system compatible since locally generated on install).

    ourjson = '''{
   "maintainer" : "Applications Division",
   "version" : "1999-01-01",
   "license_id" : 9,
   "name" : "geogratiswaterfraction1km_v100",
   "author_email" : "...",
   "author" : "Dissemination Branch",
   "tags" : [
      "GeoGratis",
      "Canada",
      "GCMD",
      "Hydrosphere",
      "Ground Water",
      "Surface Water",
      "Hydropattern",
      "Earth Science",
      "Land Surface",
      "Land Cover",
      "Landscape"
   ],
   "extras" : {
      "License" : "http://geogratis.ca/geogratis/en/licence.jsp",
      "Ministry" : "Natural Resources Canada",
      "Level of Government" : "Federal"
   },
   "maintainer_email" : "[email protected]",
   "notes" : "This product is a raster coverage",
   "url" : "http://geogratis.cgdi.gc.ca/geogratis/en/collection/metadata.do?id=67",
   "title" : "Geogratis - 1-km Water Fraction From National Topographic Data Base Maps, Canada"
}
'''
    def test_15_random(self):
        offset = '/api/rest/package'
        import urllib
        tmpjson = simplejson.dumps(simplejson.loads(self.ourjson))
        postparams = urllib.urlencode({tmpjson: 1})
        res = self.app.post(offset, params=postparams, status=[200],
                extra_environ=self.extra_environ)
        model.Session.remove()

Plugins implementing some IPackageController or IGroupController operations (i.e. read, create, edit, delete) may need the context used in the logic action. For instance, to access the Session that created a package to perform additional queries from the extension (In that case the main logic action will still perform the final commit).

Plan to migrate from mercurial git

• Convert from mercurial to git
  • Map named branches to normal branches
  • Also, if possible map usernames for RP: rgrp <​http://rufuspollock.org/> (mercurial) -> rgrp <rufus.pollock@…>
• Place at ​http://github.com/okfn/ckan (can delete current repo there as an out of date mirror)

Process:

• Do trial run
• Announce conversion date / time
• Require everyone to have pushed all outstanding changes at that time
• Do conversion
• Test
• Announce on list and ckan.org/

Periodically we see some CKAN servers fall over because they run out of disk space. We need to find out if there is a common cause and fix it.

One problem in the past has been file handles running out when creating lots of tiny files in the data directory.

Another problem has been several enourmous backups being created every day - pdeu on eu25.

Perhaps the importer deletes them before re-importing. We shouldn't have deleted resources, so let's investigate.

Percieved benefits of switching to Jinja 2

• Block inheritance for base templates which is much simpler to understand.
• Large number of developers (and designers) are familiar with it due to its similarities with the Django templates.
• Should be much faster at parsing and rendering templates.
• Has localisation support.
• Good documentation both on it's website and on sites like Stack Overflow.

We should convert a couple of existing templates to test the switchover and then get peoples feedback.

Part of super-ticket #1506.

Follows on from #826.

We can configure some extra fields in resources and can edit them in the Web UI, but we can't create new columns in the Web UI. Update the WUI to handle tis.

Page views and Resources clicks need to be tracked.

User Stories

US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded.

US1a Next to a resource or a dataset see how often it has been downloaded / viewed

US1b I want to see datasets or resources ranked by most downloaded or viewed

US1c See a trend graph for a dataset (and resources)

Adminstrative Dashboard (?)

• I want to see the traffic breakdown by country to my site ...
• Ditto for browser type, language, etc etc
• I want to see it graphed over time ...

Implementation Details

1. How do we store this data in CKAN?
2. How do we track (and store)?
3. How do we display

• Config option ckan.status.enabled = False (by default)

Storing Data

How does ckanext-googleanalytics do this? Current table:

package_id | count_recent | count_total

Move to a new stats_summary table

id | item_id | object_type | stats_type (total, month_yyyy_mm, ...) | value

Do we store this data into the search (solr) so we can search by it?

Displaying Data

• Helper functions / dictize:
  • Helper function: h.stats_get(object_type, id, stats_type)
    • h.stats_top_ranked(object_type, number) -> returns object_dicts or just labels or ...
  • Change to dictize
• Location in the default theme (do we show for example in search results too!)
• Support for ranking by most popular in search?

Tracking Data

• Our own solution (just write to site_tracking)
• Google analytics (plus extension for retrieving data) <- would need a refactor
• Piwiki

Own Solution

(For later: not as part of this ticket probably)

site_tracking table

id | url | timestamp | action (page_view, resource_download) |

• Make javascript to make request to ckan to store clicks and page views.
• Add middleware so these requests do not go through pylons and just store data quickly.

Package search results page package/search

• need clearer delineation between search entry form and results.
• 0 results needs to be clearer, and the alternative suggestion: 'would you like to create a new package' needs to not look like a search result.
• "N tags found" -- why do I care? this should be presented as "filter results by tag" -- very unlikely that I'm looking for a tag: I'm looking for a package, but tags might help me narrow down my search.
• tags should be displayed more like the way they're displayed when you add them to a package (i.e. blue 'tag' background)
• filter out particular tags
• tag cloud?
• copywriting: "Packages - Search" -> "Search packages"

Package history page package/history/mypackagename

• "Revisions" is redundant
• "Compare" button is in a ridiculous place. It should be above/below (both) the table, but not *in* it!
• Ditto to overall "Recent Changes" page comments w.r.t. timestamps, GUIDs (ellipsis), and (if poss) author links.

Package edit page package/edit/mypackagename

• Ability to add the package to groups on this page.
• Fields could be grouped.
• Inconsistent capitalisation (Url -> URL).
• Tag editor YUI script appears to break occasionally.
• Perhaps move to two column forms to save vertical space?
• Notes field to monospace?
• Extras fields need some jQuery love -- shouldn't have a fixed number available.
• Typography is a bit of a mess.

Package CKAN as a debian package so it is easier to install.

PDEU should be in a mode where:

• Users cannot edit or create packages
• Users cannot sign up on their own
• AuthorizationGroups? and RevisionHistory? is completely gone from the UI
• Groups have been repurposed as Subjects

Over-enthusiastic removal of some model items

this breaks qa extension

Order the items so that none user fields are first from ticket #2707

Options:

1. Allow for specification of genshi template paths to search in config. This way people can introduce their own templates and these templates can selectively override existing templates. (Already implemented this in shakespeare and it works)

1. Include an extra site-specific genshi template which can then be used to customize site (e.g. by having specific calls to py:def that user can define but which are ignored if they don't exist).

Can do this using:

<xi:include href="base.html"><xi:fallback /></xi:include>

Only tests with failing --ckan-migration fail, due to authz not being initialised.

Only happens if the user does not update the password. Most probably introduced during #1229.

One user reports that after resetting the password via the reset form he was unable to log in again. He did have an email address associated and received the reset email. After updating his password, login failed

"Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"

Once we have storage, make the data available in the following ways:

• Now have a cached_url field can show in the frontend ...
• Add a [<a href="${cached_url}">cached</a>] link to right of real url on resource listing on dataset page.
• On resource page: will not add it yet.
  • At the moment no clear place to pu this given nice big download button (could put in list of items on left but that does not seem right and note that it will turn up in big list of info at bottom)
• Add test (?)
• Deploy

On thedatahub.org the favicon doesn't display. i.e. the CKAN logo should appear in the browser's tab.

Original ticket #48

On the package controller the package_type is not passed to the lookup function, so the setup_template_variables defined on the extensions is never called

On the homepage click "francais" and then "English". The flash message reads "Le langage a été fixé à: français" when it should say "The language is now: English".

On the edit dataset page... there needs to be a way for users to be able to see all the resources associated to the dataset and edit them individually.

On the add dataset form, slug generation fails with JS error:

Uncaught TypeError: Object [object Object] has no method 'incompleteFormWarning'    basic-form.js:5

On step 1 for adding a dataset can we add a ajax ping that checks if the URL that is generated by the JS is taken? Behaviour should be as follows:

1. User types title in box (as current)

2. URL key is regexed by JS (as current)

3. The JS does a ajax call to check if the generated URL key is valid. The ajax callback returns a JSON dump which contains what the URL key should be (e.g. if 'test' was taken it returns 'test-1')

4. If the URL key returned in the ajax is different than the one generated by the JS then:

4a. If the user hasn't clicked the edit button then: it automatically updates the URL key field with the URL key supplied in the ajax

4b. If the user has clicked the edit button then: a form error next to the input appears saying that the URL key is taken and within that error is suggests the URL key returned by the ajax as alternative (the user can click said URL key within the error to automatically choose that one)

The URL key isn't reserved by the ajax and upon form submission the usual validation should still apply.

On same cases this exception is thrown when logging in:

File '/home/adria/dev/envs/iati/src/ckan/ckan/controllers/user.py', line 252 in logged_in
  user_dict = get_action('user_show')(context,data_dict)
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 414 in user_show
  revision_dict = revision_show(context,{'id':revision.id})
File '/home/adria/dev/envs/iati/src/ckan/ckan/logic/action/get.py', line 332 in revision_show
  ref_package_by=ref_package_by)
File '/home/adria/dev/envs/iati/src/ckan/ckan/model/__init__.py', line 317 in revision_as_dict
  for pkg in revision.packages]
AttributeError: 'NoneType' object has no attribute 'name'

On publisher view page - hide administrators from public view - only logged in admins of group should see them.

On ​http://getthedata.org/ or stackoverflow there is a pop-up bar for new users that give them some simple instructions.

Talking with users it is clear that some people are not clear how ckan.net (or other sites) work and what they are and are not 'allowed to do. This could help make this better.

Implementation

• Pop-up bar (based on a cookie or just being not logged in?) - uses ticket:938 (message flashing)
• FAQ/instructions page (use the new wiki?)

Depends

• ticket:938

Estimate

• Cost: 1h

On each page of package search results, all the items are neatly sorted apart from the last item of the page. SOLR gets the sorting of the results incorrect.

This is a known issue: ​https://issues.apache.org/jira/browse/SOLR-1777 affecting SOLR 1.4 only (which comes with Ubuntu 10.04)

It is highlighted in CKAN test ckan/tests/functional/test_pagination.py:TestPaginationPackage of commit 39096ed54bda86d043521b08b2e14fc5e283a0ff which fails most of the time it is run (passes intermittently).

On ckan.net there is this unchecked-in 'temporary hack'. It seems to be to do with foreign characters in search options. See what its doing and if necessary, put check it in.

diff -r 813ad8b5de0b ckan/lib/search.py
--- a/ckan/lib/search.py	Mon Mar 01 22:23:36 2010 +0100
+++ b/ckan/lib/search.py	Thu Apr 15 19:11:42 2010 +0200
@@ -63,6 +63,9 @@
     def query(self, options):
         '''For the given search options, returns a query object.'''
         self._options = options
+        # temporary hack!
+        if self._options.q:
+            self._options.q = self._options.q.encode('utf8')
         general_terms, field_specific_terms = self._parse_query_string()
 
         if not general_terms and \

On ckan.net there are a few packages which cause a 403 exception when you try to rate them: ​http://www.ckan.net/package/rate/coins-data?rating=3

The exception is occuring here: ckan.rating:39 in set_rating rating_obj = rating_query.one() InvalidRequestError?: Multiple rows returned for one()

It looks like this package got rated twice in quick succession, creating two similar PackageRating? objects. This race condition needs to be solved and these particular rating objects repaired.

On a production server we produce literally millions of little files saying how long every single request took to process.

They are here:

hmg.ckan.net.2/pylonsdata/call_timing

They were added by ckan/lib/base.py in cset:da438a9085d3.

I don't believe anyone uses these stats.

On a dataset page on demo.ckan.org, the left sidebar is confusing.

• It starts with some random links. Actually they are links to groups which the dataset is in, but this isn't clear.
• The sidebar elements that are actually part of the dataset are 'Datset extent' and 'License', so these should be right at the top (if they belong in the sidebar at all), instead of which they are right at the bottom in the junk part of the page (i.e. probably lower than the bottom of the main page, and hence lower than anyone will scroll).

Occasionally we seem to get revisions that are not connected to packages. These shouldn't appear, since all revisioned objects are linked to a package aren't they?

They appear on the 'Recently changed' list on the home page with an empty 'Packages' column.

Occasionally data4nr removes a data package from its registry. When the latest full data4nr data is imported, these removed packages should be found and removed.

Obfuscate author email and maintainer email field.

Use: webhelpers.html.tools.mail_to e.g. webhelpers.html.tools.mail_to('mike@…', encode='javascript')

Do we want to? Surely only public addresses should be on CKAN in which case why obfuscate?

Now that we have release branches we should deprecate the stable branch (ie. make sure it is no longer a head and then do --close-branch and merge into default one last time).

Cost: 10m (giving high priority because of low cost)

(Assigning to dread as he has been managing the stable branch).

Now that ckanext-storage is back into the core (v1.6), CKAN documentation should probably contain info on ​Storate Metadata API.

Nothing happens if you set resources=[] or resources=null.

Not used and extra overhead. Very useful to strip this out and low cost.

Places where it seems to be:

• lib/base.py: c.time_call_started
• ....

Not so keen on URLS such as /package/edit/mypackagename. These seem to be task-oriented rather than resource oriented (which would be /package/mypackagename/edit), and unstable: there is a common root for /package/edit/mypackage and /package/history/mypackage which is not reflected in the URL structure.

Not just links that need to be displayed. Other html should be displayed too, such as tables which can't be represented in markdown. Do a search of the existing fields to see what is needed.

Not all solr facet parameters are available through the pcakage_search action. In particular, facet.sort has been asked for; but this ticket should check to see if there are other parameters that would be easy to add too.

See: ​http://wiki.apache.org/solr/SimpleFacetParameters#facet.sort

No real benefit of having State object as opposed to a simple text field with enumerated values and this second option is both simpler and more flexible.

Main hassle here is that it requires migration in all projects that use vdm.

If we're going to do this we should do it sooner rather than later.

Cost: 2h (+ 1h for a conversion method)

No links to home/license and it contains out of date references to knowledgeforge. Remove it.

Nils flagged up that Ubuntu launches another LTS in April which we should probably move to. This ticket is to get the ball rolling with what we do. We can always include in our package the exact package versions of deps that we want, but it is good to be in line with the release if possible.

Here's how to see the packages included with it: ​http://packages.ubuntu.com/search?keywords=python-pylons&searchon=names&suite=all&section=all

Pylons 0.9.7 -> 1.0-2 Requires a small amount of work SQLAlchemy 0.7.3-2 OK Babel 0.9.6-1 OK Genshi 0.6-2 Perfect

Another alternative is to move to another web framework, especially minding that Pylons is not being further developed.

New js code running produced in the work following #1294 contains various string literals, and needs proper internationalisation.

See files:

• application.js
• templates.js

Furthermore, efforts can be made to tidy up strings as described in #1329.

New field for resources - hash of the resource file.

• CKAN itself will not calculate the hash value - user just pastes it in.
• Display text field in resource table.
• Requires migration script.

Questions for the field's value:

1. Which hash to use? Restrict to python hashlib and other major hash libraries.

2. Should we use merkle trees?

Thanks to Julien D'Assanges for the suggestion.

Needs to update on:

1. language - It's broken on thedatahub.org if you change to German and then click on the CKAN icon top-left to go to thedatahub.org again.

AND

2. whether signed in or not - #1373

BUT not latest revisions (which is what it was)

Or get rid of etag caching on this page and others?