{22} Trac tickets (2647 matches)

I think it would be nice to have all form pages monitor their forms - excluding search ones - and if a field has changed then warn the user that they have not saved the form data but are leaving the page.

This would help with things like changing resources on the add dataset resource form etc.

First I noticed that the gold prices dataset preview was not displaying & has data API enabled Secondly I tried enabling datastore for ​http://datahub.io/dataset/adur_district_spending/resource/281dffa6-ea9b-4446-be41-05dced06591f and after I saved the preview no longer worked. Unticking the datastore & data api checkbox brought it back

Is this a known issue?

Currently the group_index page just shows the entire list of groups, forcing the ordering to be by name. It would be better if it could be sortable by name (or reversed) or by package_count (or reversed)

See attachment. Just need to amke sure that there's enough spacing around the edit button and that it lines up so it doesn't look too wierd

the "add your own data" link in "Try another search term, browse the datasets below or add your own data." here ​http://s031.okserver.org:2375/dataset?q=stuff doesn't take you to add a dataset, just back to the search/datasets page

we need to have proper fix for this - not sure if there is some caching issue

this around the group search

in create/edit dataset forms

for description / other fields with markdown support

lang box improvements: styling, images, submit upon selection

tooltip on popular datasets with number of views facets to update automatically creating a dataset without reloading page between steps hover on licences information autocomplete on search terms group filtering social share buttons in lightboxes dataset counts on homepage

when editing a dataset (e.g. here: ​http://s031.okserver.org:2375/en/dataset/edit/afghanistan-election-data) add sidebar list of resources for user to click & browse through taking you to the relevant resource edit pages

i.e. mimetype, size, last updated etc

contextpackage_limits? = { 'tags': 5, <- get first 5

'extras': 0, <- get all

}

only get what you ask for have to be explicit

contextgroup_limits? = {} only main item

start with datasets/groups expand if we like it

you know the score

we need a delete_resource logic action and enable for resources

see ​http://demo.ckan.org/dataset/vif-2012-population-totals/resource/e7b8401a-c410-4902-9e5a-6dbd78834873

what XLS previews do we support?

​http://s031.okserver.org:2375/dataset/hi-people/resource/4558c365-fd7a-4f0e-9f2a-90ad47db6b39

trying to add a dataset to another group means it's no longer part of the first group

​http://demo.ckan.org/dataset/edit/afterfibre

​http://s031.okserver.org:2375/user

needs some spacing between the image & user name

We could also display some stats here (e.g. number of dataset created, timestamp of last activity)

1. go to a group, e.g. ​http://s031.okserver.org:2375/group/data-explorer
2. Order by anything, e.g. choose 'popular' and click go
3. "sorry no datasets found" message appears instead of ordering the group datasets in wanted order

​http://s031.okserver.org:2375/group/data-explorer?q=&sort=views_recent+desc

1. One of the datasets here: ​http://s031.okserver.org:2375/en/user/shevski has [DRAFT] after it. It's not clear why this has been added -> user confusion
2. The draft dataset is not clickable in the dataset list
3. The ff dataset (same as the ff [DRAFT] dataset) in the Activity Steam section a) is clickable and b) does not have [DRAFT] after it

1. If a user enters the wrong format on a file that can be previewed - it simply won't be previewed (e.g. a CSV or XML file that can be filled in with JSON in format will just not work or check this

2. If I incorrectly edit format to one that data preview will try to preview it will work even for a format that it doesn't accept (sometimes) e.g. this PDF file I changed the metadata to HTML ​http://s031.okserver.org:2375/en/dataset/test-dataset/resource/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28

'test-group', which has been deleted,shows up on main search page under groups - and can be filtered by - see ​http://s031.okserver.org:2375/dataset?groups=test-group

Pricing <ckan.org/datasuite/solutions> needs to go to <ckan.org/datasuite/services>

http://ckan.org/datasuite/services/

We should simplify upload and storage of files, initially only to local storage with archiver eventually being fixed to archive data externally. WIP pad is ​http://ckan.okfnpad.org/uploads

Simplifying uploads

Currently uploads are too painful/difficult/fiddly to use and/or configure. We want to simplify uploads so that they are done directly to the CKAN server, without support for remote services (S3 etc) and/or the dependencies it introduces.

We want to fix:

• File uploads themselves
• Storage of uploaded files
• Notification of the upload to other components

File uploads

Things file upload should do:

• Allow sysadmin to disable
• Allow auth'ed users to upload
• Store whatever they send on disk, and store DB entry linking the file to the person
• When creating the resource, the user should be able to choose from all of the files they have uploaded but not yet associated with a resource. This will allow for bulk upload and then a delayed association. Whenver a user creates a resource they either upload a file now, or see previously uploaded files.

Can we do the upload asynchronously and then associate the 
uploaded key with the resource before the save ? What happens 
if the user tries to submit before asymc upload finishes ? Should 
we delay them?

The upload workflow should look like...

1. File upload should be a straightforward file upload with normal auth checks and normal processing of the posted data.
2. 1. When called via ajax then the ID of the newly created file should be returned,
   2. When called via WUI then it should also be given the url to redirect to after the file upload has been handled - the id will be passed as a query param.
3. The resource save should check whether it has a file id and in that case updates the file object to point to the resource.

This should enable:

• Separate file upload into a user's temporary store, either individually or as a batch.
• Creating resources and simply choosing from previously uploaded, unassigned files
• Adding files/data to a resource after the fact.

File storage

File storage should be local to the CKAN install, and not a remote service. Any archiving to remove storage providers should be outside of the main request.

File storage should:

• allow moving data, a sysadmin should be able to move the storage root and change configuration and have the system continue running (i.e. don't store absolute paths).
• provide maintainability, it should be easy to determine which old files are not associated with resources and thus can be cleaned up.
• allow for collection of information (i.e. estimate of storate space used)
• check whether there is enough space and handling the conequences cleanly
• ensure files to be written only underneath its own root folder, checks should be made after any path generation that the file begins with the location of the file storage.
• Have a configurable maximum accepted blob size during upload.
• Should store what meta-data was provided with the upload, such as mimetype.

Somewhere in the DB we should store ...

Generating paths should try and separate the files, perhaps based on username of the owner, or some other mechanism to avoid a single folder full of files.

Notifications

We need to make sure that it is possible to notify other components within the system that an upload has taken place, or at least make it easy for them to be notified. The primary use case for this is to notify the component that will translate/upload certain formats to the data store.

We could do this based on the post-upload update to the file model (i.e. when we record the total received size of the file).

Steps to reproduce:

1. go to edit a resource, e.g. ​http://s031.okserver.org:2375/dataset/ted/resource_edit/a621a6ad-6f1c-46d1-b858-03cdb7a25c07

2. Change the format field to something it doesn't get. e.g. add a comma "HTML,"

3. Get message: "The form contains invalid entries:

Last modified: Date format incorrect Size: Invalid integer"

4. Format field reverts, try to re-save/submit form - i.e. click on "update resource"

5. nothing happens (same if you even try changing the format field

We need to check the archiver to make sure it works as we would expect after the changes in #2732:

• Receive notification of a file that needs processing
• Check the file's DB entry and upload the file to the configured remote service
• Update the URL of the file (both in the file's DB entry and the resource)
• Once we're sure the file is safely archived, we should mark the file as being archived

so that we know it can safely be deleted at some point.

Maybe this could/should be implemented as a service rather than a celery task?

Why is datastore not enabled for this resource ​http://s031.okserver.org:2375/dataset/frere/resource/cdeddb39-b0cb-4762-aa5b-f80fc892d4dd ?

Is this going to be a site wide setting? How do new resources get added to datastore? Or are we just using the dataproxy for demo?

When adding a resource, the breadcrumb should say "add data" or "add resource" not "create dataset" as you see here: ​http://s031.okserver.org:2375/dataset/new_resource/frere

​http://localhost:5000/ckan-admin/authz

you need to be an admin

paster sysadmin add <username> -c dev.ini

like #2741 we need other admin links

you may want to make the inputs use your form macros - I've just done a basic port

also we will need to get all the tables done but looks like we need to get auth groups working for that

​http://s031.okserver.org:2375/dataset/new

license dropdown - when open we need some line spacing at least in FF I think it'd be nice to fix this in phase 2

Hey, this is still missing. It's the one with the add dataset text

When editing a resource you see the current and any other existing resources in sidebar

(e.g. see ​http://s031.okserver.org:2375/en/dataset/format-error-test/resource_edit/d1eac556-c16f-44af-8148-5e3467b57cf8?inner_span=True)

Would be good to have a pretty 'add new' slightly transparent resource folder/pointer undearneath - letting you add resources from the end resource page

we need to ensure that translations in .js files are ripper for translation

look at what is available as preference to creating our own

When navigating to ​http://s031.okserver.org:2375/group on the demo server, the Group link in the navbar is not highlighted in the same was as the other links are when visiting those pages.

Specifically the 'add resource' button and the 'add item' button in related section

It doesn't pass extras_as_string

We get an error which should be improved the actual problem is this but not passed to user

⇝ KeyError?: 'ofs.impl'

View as: Interactive (full) | Text (full) | XML (full) Module ckan.controllers.storage:2 in auth_form view Module ckan.lib.jsonp:26 in jsonpify view

data = func(*args, kwargs)

Module ckan.controllers.storage:407 in auth_form view

authorize(method, bucket, label, c.userobj, self.ofs)

Module ckan.controllers.storage:200 in ofs view

StorageAPIController._ofs_impl = get_ofs()

Module ckan.controllers.storage:71 in get_ofs view

storage_backend = configofs.impl?

Module paste.registry:146 in getitem view

return self._current_obj()[key]

KeyError?: 'ofs.impl'

see spacing between 'download' button and 'data api' button ​http://s031.okserver.org:2375/dataset/afterfibre/resource/66f5b980-06db-4ccd-a005-c18de4d68a4b

/home/seanh/Projects/ckan/ckan/ckan/tests/functional/test_related.py

ImportError? (cannot import name assert_regexp_matches)

Definitely do:

• Local file filestore: Remove pairtree (and OFS) and do something very simple

Options:

1. Local file only. Allow for uploaders to GS / S3 in the background
   • Advantages: simpler, if local upload can do progress bars etc etc
   • Disadvantages: strain for web-app (upload a 2GB file what happens)
     • This is probably fixable ...
2. Stick with how we are

New filestore without pairtree

Each uploaded file is located on disk at:

{uuid}/{filename}

Alternative:

yyyy/mm/dd/{uuid}/{filename}

Need the uuid to avoid collisions.

Metadata: Store no metadata.

The new ckan creates datasets as part of a 3 phase process. To allow for this partially created datasets can have a state that is 'draft' or 'draft-complete'. These datasets should not be seen as active by the activity stream.

If we click 'add dataset' and then complete the first phase of adding a dataset then we end up having a activity stream created. When we add a resource in the next phase (add data) again an activity stream item xxx added resource to dataset is created

We do not want these adding. Essentially if a dataset has a state.startswith('draft') then we want the activity stream to ignore all actions involving it.

finally when the state is changed form state.startswith('draft') to state=='active' we want a xxx has created dataset ... to be added to the stream.

The best way to do this would be to branch from 2375-demo-theme-stable and get it to work there.

Let me know if you need any help with this ticket or a better explanation of the problem.

so that data proxy works

Luca De Santis reported these strings not marked in 1.7.1:

• /user/login.html ("Remember me" & "Sign in")
• /package/search.html and template/package/group/read.html (Tags, Resource Formats, Groups)
• /package/resource_read.html (the "No description" message)
• the "Add related item" dialog (some strings are i18n savvy, others not)
• the stats page (labels on the graphs)
• the history page (as far as the log messages are concerned).

lets check that they are marked in 1.8

Upload the new pot file/resource on transifex, put out the call for translations. Some recent commits on master need to be merged into 1.8 first

We need to be able to do the following:

• Rip the localised strings from the JavaScript? source code. These are strings that look like _('String'), translate('String') and isPlural('String') .
• An endpoint that provides the translations in the correct locale for the user.
• The endpoint should provide the locale as JSON (possibly using ​http://jsgettext.berlios.de/doc/html/po2json.html or similar tool)

The JavaScript? can then read this file on page load.

to use on demos instead of the ckan logo & tag line.

Design here: ​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/63387712/comments#190457381 (First, text only one at the bottom)

apps dashboard available at ​http://localhost:5000/apps in main dev branch

I have got this to work in the demo however

a) it is a bit ugly

b) the template could do with some sorting eg the dropdowns should be being created in a nicer way

Since admins can see deleted datasets - there should be a way for them a) to know they are currently deleted & not viewable by normal users - ticket: #2779 b) way to undelete such datasets - this ticket

I suggest a button on the edit form instead of the delete button i.e. remove normal delete button with 'deleted dataset, only admins can view' with undelete button next to the message?

use ​http://leaflet.cloudmade.com/ properly to generate map backgrounds for demo instances