You can reset your password (#1186) but you have to know the URI (/user/reset) - there is no link to this page!

I have no idea whether this was a deliberate decision or not, but there is a total absence of any UI with which to delete resources from the currently deployed version of thedatahub.org.

Have a news section (suggest as a sidebar item).

News section will link to latest 3/4 blog posts on CKAN from blog.okfn.org.

Details:

• Suggest pulling via rss or similar.
• Will want to cache this ...

Cost: 4h?

Deleted means things will get purged at some point.

Archived means they stay around but get hidden from search results and a big warning notice gets displayed saying this is archived / deprecated.

@richard cyganiak ...

In 0.5 and 0.6 have been more oriented to getting file indexes working to support the convenience of using packages on disk directly. However we now need to go back to having a better central/default index and the best model for this is a simple db index using sqlite (which is now supported in the standard library).

NB: this is a retrospective ticket (been working on this for a couple of months)

​http://localhost:5000/dataset/new_metadata/dataset

Controller : package Action : new_metadata

template package/snippets/package_metadata_form.html

We should simplify upload and storage of files, initially only to local storage with archiver eventually being fixed to archive data externally. WIP pad is ​http://ckan.okfnpad.org/uploads

Simplifying uploads

Currently uploads are too painful/difficult/fiddly to use and/or configure. We want to simplify uploads so that they are done directly to the CKAN server, without support for remote services (S3 etc) and/or the dependencies it introduces.

We want to fix:

• File uploads themselves
• Storage of uploaded files
• Notification of the upload to other components

File uploads

Things file upload should do:

• Allow sysadmin to disable
• Allow auth'ed users to upload
• Store whatever they send on disk, and store DB entry linking the file to the person
• When creating the resource, the user should be able to choose from all of the files they have uploaded but not yet associated with a resource. This will allow for bulk upload and then a delayed association. Whenver a user creates a resource they either upload a file now, or see previously uploaded files.

Can we do the upload asynchronously and then associate the 
uploaded key with the resource before the save ? What happens 
if the user tries to submit before asymc upload finishes ? Should 
we delay them?

The upload workflow should look like...

1. File upload should be a straightforward file upload with normal auth checks and normal processing of the posted data.
2. 1. When called via ajax then the ID of the newly created file should be returned,
   2. When called via WUI then it should also be given the url to redirect to after the file upload has been handled - the id will be passed as a query param.
3. The resource save should check whether it has a file id and in that case updates the file object to point to the resource.

This should enable:

• Separate file upload into a user's temporary store, either individually or as a batch.
• Creating resources and simply choosing from previously uploaded, unassigned files
• Adding files/data to a resource after the fact.

File storage

File storage should be local to the CKAN install, and not a remote service. Any archiving to remove storage providers should be outside of the main request.

File storage should:

• allow moving data, a sysadmin should be able to move the storage root and change configuration and have the system continue running (i.e. don't store absolute paths).
• provide maintainability, it should be easy to determine which old files are not associated with resources and thus can be cleaned up.
• allow for collection of information (i.e. estimate of storate space used)
• check whether there is enough space and handling the conequences cleanly
• ensure files to be written only underneath its own root folder, checks should be made after any path generation that the file begins with the location of the file storage.
• Have a configurable maximum accepted blob size during upload.
• Should store what meta-data was provided with the upload, such as mimetype.

Somewhere in the DB we should store ...

Generating paths should try and separate the files, perhaps based on username of the owner, or some other mechanism to avoid a single folder full of files.

Notifications

We need to make sure that it is possible to notify other components within the system that an upload has taken place, or at least make it easy for them to be notified. The primary use case for this is to notify the component that will translate/upload certain formats to the data store.

We could do this based on the post-upload update to the file model (i.e. when we record the total received size of the file).

With up to date statistics such as:

• total packages
• total tags
• total users
• most active users
• last update
• etc.

Could also be nice to have a more dedicated stats page - with information such as graphs showing rate of packages being added and such like. See ticket:184

This is what we need the JSON to look like. Taken from the Jed documentation (​http://slexaxton.github.com/Jed/).

{

      // The empty string key is used as the configuration
      // block for each domain
      "" : {
        // Domain name
        "domain" : "the_domain",

        // Language code
        "lang" : "en",

        // Plural form function for language
        "plural_forms" : "nplurals=2; plural=(n != 1);"
      },

      // Other keys in a domain contain arrays as values
      // that map to the translations for that key.
      // Gettext suggests that you use english as your keys
      // in case the key isn't found, and it can just pass
      // the value directly through.
      // Note: by convention, the 0-index location of the translations
      // is never accessed. It's just a thing, I guess.
      "a key" : [ null, "the translation", "the plural translations", ... ],

      // The plural form string is converted into a function
      // and the value that's passed into the gettext call
      // is passed into the plural form function. It's result
      // (plus 1) is the index that the translation comes from.

      // We're using sprintf interpolation on our keys so we can
      // then sub in the _actual_ values into the result.
      "%d key" : [ null, "%d key", "%d keys" ],

      // Contexts are keys that are just prefixed with a context string
      // with a unicode \u0004 as the delimiter.
      // You can use it for anything. Usually it's just for being content aware
      // in some way (e.g. male vs. female, product vs. category)
      "context\u0004%d key": [ null, "context %d key", "context %d keys" ]
    }

This looks like an appropriate python library to do the formatting. ​http://pypi.python.org/pypi/pojson/ but I'll leave to your discretion.

Finally an API endpoint that returns the appropriate request would save us making requests to the i18n directory.

/api/i18n/{locale}

For example if the current url is ​http://localhost:5000/pl/dataset then the client would request:

http://localhost:5000/api/i18n/pl

This should return the Content-Type: application/json and a 200 with an appropriate JSON body (this can be the language requested or an appropriate fallback). I think that if the fallback is the default language (in most cases English) then we just return a 204 No Content to save data.

Requirements

We need to clone the demo ODS site for DataGM as soon as we have a mostly working demo so that the DataGM team can try it out and comment/feedback on functionality.

Interface

None

User Stories

None

Tasks

[ ] Deploy demo DataGM site.

Estimates

like we have on the form here

​http://s031.okserver.org:2375/harvest/new

you said it partly exists so maybe I just need some training

When navigating to ​http://s031.okserver.org:2375/group on the demo server, the Group link in the navbar is not highlighted in the same was as the other links are when visiting those pages.

There is some bizarre language flag issues. 1) Log in to ​http://s031.okserver.org:2375/ (I'm logged in as danieljohnlewis) 2) Visit the apps page: ​http://s031.okserver.org:2375/apps 3) Have a look at the languages drop down, flags look fine (although Deutsche does have an EU flag rather than the German flag.. but that's irrelevant) 4) Change filters to, say, type: Application and sort: Newest, and apply. Web address becomes: ​http://s031.okserver.org:2375/apps?page=1&type=application&sort=created_desc 5) Look at the languages drop down: Flags are largely incorrect, with more than one country having the same flag.

Please see the attached screenshot.

Note: this was originally tested with Google Chrome on Linux (Ubuntu).

Multiple download links, including links to mirrors and multiple formats/versions

Enable the multilingual plugins:

ckan.plugins = stats synchronous_search multilingual_dataset multilingual_group multilingual_tag

and set your default language to one not supported by the multilingual plugin, e.g.

ckan.locale_default = cs_CZ

now run CKAN and try to add a dataset:

File '/home/seanh/Projects/ckan171/ckan/ckanext/multilingual/plugin.py', line 141 in before_index

text_field_items+ default_lang?.extend(all_terms)

KeyError?: 'text_cs_CZ'

It doesn't matter what language you are viewing the site in in your browser, the default language setting in the ini file determines whether it crashed or not.

A number of supported languages are defined at the top of ckanext/multilingual/plugin.py. I think if the default language is not one of these it crashes.

I think this affects all versions of CKAN since the multilingual plugin was added so at least 1.7, 1.8 and 2.0

The default backing store for webstore should be Postgres and not sqlite. This was agreed as part of the #1574 storage changes and on ​http://ckan.okfnpad.org/newstorage

When vdm was created there was no SessionExtension so we use MapperExtension for doing revisioning. Now that SessionExtension? exists we should use it. We can also follow the existing SQLAlchemy recipe: <​http://www.sqlalchemy.org/docs/orm/examples.html?highlight=versioning#versioned-objects>

At the moment the assignment of user roles on package creation is done in setup_user_roles method in ckan/model/authz.py and is hard-coded.

This can be a pain to override (you can use a Plugin listening for package create events) and makes some things such as putting system in restricted mode more complex than it should be (see ticket:833).

An elegant solution would be to move this into an Extension this simplifies the code and make it easier for people override (just remove the default extension and plugin your own).

• Move into core (keeping as an extension) and default enable
• Do we need tests for controller? (No: already there)
• Documentation (?) - e.g. of basic config options
• Remove lib/stats.py which is still in core
  • dread moved it to extension in ​https://github.com/okfn/ckanext-stats/commit/293b2544737d8ab7610884bd5e999ee37f1ab813 but did not remove from core
• Deprecation notice on ckanext-stats repo and update extension registry

Est: 0.5d.

Questions:

• Why do this?
  • tiny extension with few dependencies - and really nice to have out of the box
  • trial for doing this on larger scale
• Do we keep as extension (even if in core)?
  • Ans: Yes, keep as extension because:
    • Already set up that way
    • Cleaner
    • Easier to disable / enable
• How do we integrate with main them (e.g. have stats link)
  • Ans: not sure (this is part of more general issue of how we update theme for varying changes elsewhere). Best answer is to have some widgetization in theme.
• Hide ratings section (at least until we reinstate ratings #1598)
  • Ans: no, let's not bother (and having ratings there encourages us to do #1598 and/or find out whether people are interested in ratings)

Currently autocomplete method on package controller. This method should be in API (like other autocomplete methods).

Will need to update client code (just forms atm I think).

ckan/controllers/importer.py provides a Pylons UI for importing spreadsheet data into CKAN. This would better live in an extension than core CKAN.

In the same move, ckan/tests/misc/test_spreadsheet_importer.py, which is used by importer.py as well as various dgu data importers, should move into ckanext.

From Toby: strip more functionality from controllers and add via template helpers eg activity streams

Functions like convert_to_extras and convert_from_extras will be widely used from extensions and should be on ckan core.

Archiver is included in some places just for access to download(), we should move this to lib in core.

Which is where the other API tests have now been moved.

Based on number of views.

Related to ticket:143.

When attempting to add more than one resource at once, if more than one resource invalidates, this results in a js error, leaving the form in an inconsistent state.

Repro:

1. Go to /dataset/new
2. Add a new resource. Fill in one of the fields with an invalid value. eg - last_modified, or size...
3. Add another resource, doing the same thing: make one of the fields invalid.
4. Try to save the dataset.
5. The entered resource information will be lost, and a js error "Uncaught Error: Can't add the same model to a set twice,: backbone.js:586" will be thrown.

Use case

As a user I want to stay abreast of package changes, such as a new package being created, newer data is available for a package or a new download is available for a package.

Implementation

Add into the Atom feed:

1. a package is a new one
2. the resources have been changed
3. the last_updated field has been changed

If you modify your user name, it looks like you are logged out, but you are actually logged in and therefore can't login. (Reported by Pablo)

As a work around, you need to logout and log back in.

implementation

• We should force this by auto-logging you out (DR: maybe!)

I've installed the Moderated Edits extension (ckanext-moderatededits) and am editing a package imported from IATIregistry.org, with an extra field which contains a bit of HTML.

The editor indicates the field has changed, although the content hasn't (see screenshot). All I can find so far is a minor difference: in the field content, there is a code &#8212 and in the rendered table that is an &mdash;

Where said "v0.4" would be the result of converging four lists mentioned in requirement #711.

Subticket of: #1294

Rather than having to visit a dedicated page it would be good if registration and login could be done from a modal form (separate or combined ...).

Why

• It could be used from dataset creation page in situations where user needs to be registered / logged in to create a dataset (so we could allow someone to start creating a dataset and only get them to login at the end ...)
• It allows for quicker and easier logging in

Implementation

• See Friedrich's work on the datahub

e.g. ​http://thedatahub.org/dataset/osm%402011-07-12T12%3A16%3A47.590358 gives:

This is the current revision of this dataset, as edited at 2011-07-12 12:16.

but it should say the date 2011-07-12 (which is the data being displayed).

The problem is looking at PackageRevision?, when it might be a tag or extra.

Group edit dataset add form needs some work

• Dataset name is not cleared when you add
• No way to remove item from list of datasets to be added if I make a mistake
• (2nd Apr 2012) It now seems that option to add multiple datasets at once has disappeared (perhaps during the CSS/HTML refactor ...)

Plan to migrate from mercurial git

• Convert from mercurial to git
  • Map named branches to normal branches
  • Also, if possible map usernames for RP: rgrp <​http://rufuspollock.org/> (mercurial) -> rgrp <rufus.pollock@…>
• Place at ​http://github.com/okfn/ckan (can delete current repo there as an out of date mirror)

Process:

• Do trial run
• Announce conversion date / time
• Require everyone to have pushed all outstanding changes at that time
• Do conversion
• Test
• Announce on list and ckan.org/

Move ckan trac to new ckan.org location. Will also need to re-enable user accounts etc.