{22} Trac tickets (2647 matches)

Need proper user stories but ...

Publisher admins/editors may need a more useful group read page showing things like:

• The current search

• Recent activity

• People within the group

• Followers

• Others?

Activities about private datasets should not appear in public activity streams.

I don't think you want to actually purge the activities from the db, because you might still want them to appear in private activity streams.

I do think that when a dataset goes private all its past activity should go private, because I imagine that users are going to want to hide everything about the dataset and not have any past activities 'leaking out'

I don't think you want to consider whether the dataset was private when the activity happened, rather if a dataset is private now then all its past activities are private (and the simplest thing would be to say that if a dataset is public now then all its past activities become public as well, but is that a privacy concern?)

The easiest way to implement this is going to be by modifying the *_activity_list() action functions in get.py, after they pull their activity lists out of the db they should pass them through a function that filters out stuff about private datasets.

An activity about a private dataset is one whose object_type is 'dataset' and whose object_id matches the id of a private dataset. You should also check the object_type and object_id of all of the activity object's activity detail objects, if any of those match a private dataset then mark the whole activity as private.

Currently all activity streams are public so should have all private datasets filtered out from them, except for the dashboard activity stream which is private to the individual user. In this case private datasets that the user has permission to see should not be filtered.

we now have a user dashboard that needs theming not sure if we need sam to look at it

​http://localhost:5000/user/dashboard

Not loading for all resources as far as I can tell; e.g. ​http://s031.okserver.org:2375/dataset/afghanistan-election-data/resource/f6331f99-51f6-44d9-95b9-b20f3b74f360

Fine on demo.ckan.org

Includes: missing fields, existing field checks (i.e. whether a name/dataset already exists with that name) during input (i.e. no need to submit form to check)

needs review & speccing out e.g. datasets created by a user who belongs to a certain publisher (group) get auto added to this group

We should simplify upload and storage of files, initially only to local storage with archiver eventually being fixed to archive data externally. WIP pad is ​http://ckan.okfnpad.org/uploads

Simplifying uploads

Currently uploads are too painful/difficult/fiddly to use and/or configure. We want to simplify uploads so that they are done directly to the CKAN server, without support for remote services (S3 etc) and/or the dependencies it introduces.

We want to fix:

• File uploads themselves
• Storage of uploaded files
• Notification of the upload to other components

File uploads

Things file upload should do:

• Allow sysadmin to disable
• Allow auth'ed users to upload
• Store whatever they send on disk, and store DB entry linking the file to the person
• When creating the resource, the user should be able to choose from all of the files they have uploaded but not yet associated with a resource. This will allow for bulk upload and then a delayed association. Whenver a user creates a resource they either upload a file now, or see previously uploaded files.

Can we do the upload asynchronously and then associate the 
uploaded key with the resource before the save ? What happens 
if the user tries to submit before asymc upload finishes ? Should 
we delay them?

The upload workflow should look like...

1. File upload should be a straightforward file upload with normal auth checks and normal processing of the posted data.
2. 1. When called via ajax then the ID of the newly created file should be returned,
   2. When called via WUI then it should also be given the url to redirect to after the file upload has been handled - the id will be passed as a query param.
3. The resource save should check whether it has a file id and in that case updates the file object to point to the resource.

This should enable:

• Separate file upload into a user's temporary store, either individually or as a batch.
• Creating resources and simply choosing from previously uploaded, unassigned files
• Adding files/data to a resource after the fact.

File storage

File storage should be local to the CKAN install, and not a remote service. Any archiving to remove storage providers should be outside of the main request.

File storage should:

• allow moving data, a sysadmin should be able to move the storage root and change configuration and have the system continue running (i.e. don't store absolute paths).
• provide maintainability, it should be easy to determine which old files are not associated with resources and thus can be cleaned up.
• allow for collection of information (i.e. estimate of storate space used)
• check whether there is enough space and handling the conequences cleanly
• ensure files to be written only underneath its own root folder, checks should be made after any path generation that the file begins with the location of the file storage.
• Have a configurable maximum accepted blob size during upload.
• Should store what meta-data was provided with the upload, such as mimetype.

Somewhere in the DB we should store ...

Generating paths should try and separate the files, perhaps based on username of the owner, or some other mechanism to avoid a single folder full of files.

Notifications

We need to make sure that it is possible to notify other components within the system that an upload has taken place, or at least make it easy for them to be notified. The primary use case for this is to notify the component that will translate/upload certain formats to the data store.

We could do this based on the post-upload update to the file model (i.e. when we record the total received size of the file).

The new ckan creates datasets as part of a 3 phase process. To allow for this partially created datasets can have a state that is 'draft' or 'draft-complete'. These datasets should not be seen as active by the activity stream.

If we click 'add dataset' and then complete the first phase of adding a dataset then we end up having a activity stream created. When we add a resource in the next phase (add data) again an activity stream item xxx added resource to dataset is created

We do not want these adding. Essentially if a dataset has a state.startswith('draft') then we want the activity stream to ignore all actions involving it.

finally when the state is changed form state.startswith('draft') to state=='active' we want a xxx has created dataset ... to be added to the stream.

The best way to do this would be to branch from 2375-demo-theme-stable and get it to work there.

Let me know if you need any help with this ticket or a better explanation of the problem.

3. Uploading a file behaves counter-intuitively (I would suggest wrongly).

When adding a new resource by uploading a file, I select a file called say create-group.png. I expect the following to happen:

• the pathname of that file is filled in the box;
• nothing is actually uploaded till I hit 'add' (confirming that I've got the right file etc).

Instead of this,

• the file is immediately and, seemingly, irrevocably uploaded;
• the box is filled with a mysterious URL (​https://commondatastorage.googleapis.com/ckan-demo ...)
• the mysterious URL is so long I can't even see the final element, which is my file name.

For example see: ​http://s031.okserver.org:2375/qa/dataset/five_stars

Which has: Example dataset as a score of: 4 After examination of the dataset ( ​http://s031.okserver.org:2375/dataset/example-dataset ) it is made up of a CSV and an HTML file.

In progress

1. [major] q does not seem to work reliably. e.g. using the setup from this gist ​https://gist.github.com/1930806 and doing a ?q=DE yields no results (does not work with "q=de" either)
   • q=second does work ...
2. [major] q does not work with 2 values (see below)
3. Query on search with limit 0 results in total of 0 (should either be null or correct total). Queries with other limits yield correct total AFAICT
   • Also weird fact that limit is returned but as as as string - should it not be an integer
4. Types on fields: could these not be canonical and as per recline (or is it important to allow exact sql types ...)

Multiple query values

Try a query such as: "second UK" and you will get 500 error:

​http://localhost:5000/api/3/action/datastore_search?resource_id=4f1299ab-a100-4e5f-ba81-e6d234a2f3bd&q=second%20UK

ProgrammingError: (ProgrammingError) syntax error in tsquery: "second UK" 'select "_id", "id", "date", "x", "y", "z", "country", "title", "lat", "lon", count(*) over() as "_full_count"\n from "4f1299ab-a100-4e5f-ba81-e6d234a2f3bd" where _full_text @@ to_tsquery(%s) limit 100 offset 0' (u'second UK',)

Suggestions

Filter support: should think in more detail about this (may want to follow recline style)

Simple filters in query parameters would be nice too ...

Users could write ckan extensions that offer previews for a certain datatype.

Advantages:

• We can say that you can add your own previews
• Every preview would have to define it's own dependencies
• External development (community creates new extensions?)

Questions to answer:

• Specification of the interface
• Precedence of previews for the same data types
• Plugins only define the datatype that they are responsible for or do we call a function that returns whether the extension is responsible.

As far as I can see, this should be fairly easy to implement by following these instructions: ​http://docs.ckan.org/en/latest/writing-extensions.html#writing-a-plugin-interface

Replace the /revision page with a /activity page showing a site-wide activity stream of all activities in the site. Or perhaps the site-wide activity stream can go somewhere on the front page of the site instead?

Should minify and concatenate JavaScript? and CSS as part of #2354

Currently we have far too many types that are essentially the same format. The new demo theme is using icons for common types. So we need to normalise the common formats into pretty strings. e.g. application/json, JSON, .json and json all should be output as “json”

See the following Basecamp thread for UI examples and discussion.

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61906351/comments#comment_179681056

This is not trivial, requires a database upgrade. But 0.12.3 has some nice new features (multiple vcs repository support, ticket comment editing and nicer preview) and the latest spam filter plugin requires 0.12.

​http://trac.edgewall.org/wiki/ChangeLog

there's a setting for this

Probably use ansible to do this. To create an instance, create a dir at /etc/ckan/MYSITE, and put MYSITE.wsgi, MYSITE.ini and who.ini files in it. Also put a MYSITE file in /etc/apache2/sites-available. See the example files already present in ckanbuild. Booting a new site should be a single command.

May not handle the postgres/solr/elastic-search side of things yet, could just require the user to set these up herself first and then pass them as args to the create-instance command.

When there are multiple ckan sites installed on a single server via ckanbuild, there needs to be some way of upgrading them all to a new ckan version at once.

It should run the script to create the debian package, boot a VM, install the debian package on the VM, boot a CKAN instance, then run the tests.