{22} Trac tickets (2647 matches)

so we find newly created templates

You can't run the search-index reindex on a live server because it will give us bad results for 2 to 3 hours while it runs. Can there be an option that doesn't delete the entire index at the start?

Instead it could just delete any items that don't exist any more, then delete them and regenerate them one by one. So the total number of datasets doesn't change much.

New recline version has significant improvements like the SlickGrid? based grid view, it'd be good to have it on 1.8.

Dataset extent map and spatial filter need to adapted to the new theme, as they are not showing up now

Discussion:

​https://okfn.basecamphq.com/projects/9558659-demo-ckan-front-end/posts/61696765/comments

Server setup scripts need to have two distinct steps:

1) Download all files that aren't in RHEL repositories, create RPMs where necessary (python virtualenv).

2) Install from local files.

make it as a snippet calling thing like aron would love

When a resource doesn't have a description, it is rendered as markdown from the description of the dataset.

If the resource does have a description it isn't rendered as markdown ..

See ​http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96

TSV support doesn't seem to work very well, may be the mimetype ( text/tab-separated-values )

See ​http://thedatahub.org/dataset/wikipedia-e3-timestamp-position-modification/resource/d883ab44-07f4-4992-800a-3e4bf5d53a96

Try: ​http://localhost:5000/api/storage/auth/form/photo-of-a-duck

Gets:

File '<string>', line 2 in auth_form
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/lib/jsonp.py', line 26 in jsonpify
  data = func(*args, **kwargs)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/storage.py', line 407 in auth_form
  authorize(method, bucket, label, c.userobj, self.ofs)
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/storage.py', line 200 in ofs
  StorageAPIController._ofs_impl = get_ofs()
File '/Users/Aron/.virtualenvs/ckan/src/ckan/ckan/controllers/storage.py', line 71 in get_ofs
  storage_backend = config['ofs.impl']
File '/Users/Aron/.virtualenvs/ckan/lib/python2.7/site-packages/paste/registry.py', line 146 in __getitem__
  return self._current_obj()[key]
KeyError: 'ofs.impl'

History button in top right of ​http://demo.ckan.org/group/data-explorer goes to ​http://demo.ckan.org/group/history/data-explorer and needs hiding

in 2375-demo-theme-development

templates/package/confirm_delete.html needs to be made to look nice note some writing in black

delete button needs nice home on dataset/edit

you can add js support add ?confirm=yes to url on dataset button to force delete

Please note delete is not yet implemented - but will be in the next few hours

There is a new template for editing resources outside of the add dataset logic

this is the template it just throws some crap onto the page at the moment - we need to create a form - the form will end up with the usual data/edit stuff but for now you can use the resource to populate the form and I can then change it to use data - I just want to know which fields we'll have

templates/package/resource_edit.html

when done reassign to me and I'll get it functioning

I've added [Draft] to partial completed datasets in the package list and read commit 6a8e74d0

templates/snippets/package_list.html

templates/package/read.html

do you want to style them in some way?

According to sqlalchemy docs [1], when calling Session.is_modified(), the passive argument needs to be passed in as True.

[1] ​http://docs.sqlalchemy.org/en/rel_0_7/orm/session.html?highlight=is_modified#sqlalchemy.orm.session.Session.is_modified

Need to add JS (presumably?) to the related app box on dataset page: ​http://s031.okserver.org:2375/dataset/gold-prices to allow user to scroll through when multiple related items exist

Needs to be a way for admins (owners of dataset) to delete related items. I suggest adding a button to the edit page

​http://s031.okserver.org:2375/dataset/gold-prices/related/edit/d3249863-040a-4f55-a5d8-c86697d0b7b8

Super admins & owners/creators of groups need a way to delete them

Is the auth model in place for all this stuff? May be worth sitting down and going through how this should work

Currently can change state from active to deleted - this is not ery user friendly (want a button with a 2 step process maybe?) & I'd like to know who can view this option? Everyone who can edit a group? Who can edit?

stop bots/browsers etc following links and auto deleting stuff

Clicking on 'save & add another' button here ​http://s031.okserver.org:2375/dataset/new_resource/yo takes you to step 3 instead of leaving you on step 2 when you haven't added any data.

Instead it should show an alert/ message that no data has been added & that you need to link to a file or upload something

Also, can upload be option 2 instead of 3?

404 Not Found The resource could not be found.

in dataset creation / edit form

Includes: missing fields, existing field checks (i.e. whether a name/dataset already exists with that name) during input (i.e. no need to submit form to check)

needs review & speccing out e.g. datasets created by a user who belongs to a certain publisher (group) get auto added to this group

disqus??

with add button on dataset page and in 'data & resource' section when there's no data

e.g. on ​http://s031.okserver.org:2375/en/dataset/afghanistan-election-data/resource_edit/f6331f99-51f6-44d9-95b9-b20f3b74f360

& ensure consistant with sidebar of edit dataset page

This currently looks terrible. Some simple things we should do to improve this.

• Localise and captitalise known (non custom) keys into human titles. eg. "size", "created_at", "mime_type".
• Apply units to values of known keys where possible eg. "size in appropiate units"
• Convert dates into a more human friendly format. We can avoid localisation hell by using an ISO format like YYYY/MM/DD hh:mm:ss which is still better than ISO8061
• Move the non user fields to the top of the table.

There doesn't seem to be any documentation yet for Atom feeds.

it would be nice if this didn't fall over when the dir is removed maybe a linux only issue?

to reproduce run

paster color fuchsia -c development.ini

then go to ​http://localhost:5000/dataset/new

Add button is blue gradient at the top this should be overridden in variables.less

when added send this ticket to me and i'll update the color command to change it

​http://s031.okserver.org:2375/dataset/hi-people/resource/4558c365-fd7a-4f0e-9f2a-90ad47db6b39

trying to add a dataset to another group means it's no longer part of the first group

​http://demo.ckan.org/dataset/edit/afterfibre

1. One of the datasets here: ​http://s031.okserver.org:2375/en/user/shevski has [DRAFT] after it. It's not clear why this has been added -> user confusion
2. The draft dataset is not clickable in the dataset list
3. The ff dataset (same as the ff [DRAFT] dataset) in the Activity Steam section a) is clickable and b) does not have [DRAFT] after it

1. If a user enters the wrong format on a file that can be previewed - it simply won't be previewed (e.g. a CSV or XML file that can be filled in with JSON in format will just not work or check this

2. If I incorrectly edit format to one that data preview will try to preview it will work even for a format that it doesn't accept (sometimes) e.g. this PDF file I changed the metadata to HTML ​http://s031.okserver.org:2375/en/dataset/test-dataset/resource/9d27a9d9-36ec-460e-9edb-6dff7ba4fc28

which is confusing since you can only search for one tag like this at a time. I.e. tags:economics,cvs or tags:economics, csv or tags:economics+CSV doesn't work for example; therefore tag:economics, should also work!

​http://s031.okserver.org:2375/dataset?q=tags%3Aeconomics&sort=relevance+asc

We should simplify upload and storage of files, initially only to local storage with archiver eventually being fixed to archive data externally. WIP pad is ​http://ckan.okfnpad.org/uploads

Simplifying uploads

Currently uploads are too painful/difficult/fiddly to use and/or configure. We want to simplify uploads so that they are done directly to the CKAN server, without support for remote services (S3 etc) and/or the dependencies it introduces.

We want to fix:

• File uploads themselves
• Storage of uploaded files
• Notification of the upload to other components

File uploads

Things file upload should do:

• Allow sysadmin to disable
• Allow auth'ed users to upload
• Store whatever they send on disk, and store DB entry linking the file to the person
• When creating the resource, the user should be able to choose from all of the files they have uploaded but not yet associated with a resource. This will allow for bulk upload and then a delayed association. Whenver a user creates a resource they either upload a file now, or see previously uploaded files.

Can we do the upload asynchronously and then associate the 
uploaded key with the resource before the save ? What happens 
if the user tries to submit before asymc upload finishes ? Should 
we delay them?

The upload workflow should look like...

1. File upload should be a straightforward file upload with normal auth checks and normal processing of the posted data.
2. 1. When called via ajax then the ID of the newly created file should be returned,
   2. When called via WUI then it should also be given the url to redirect to after the file upload has been handled - the id will be passed as a query param.
3. The resource save should check whether it has a file id and in that case updates the file object to point to the resource.

This should enable:

• Separate file upload into a user's temporary store, either individually or as a batch.
• Creating resources and simply choosing from previously uploaded, unassigned files
• Adding files/data to a resource after the fact.

File storage

File storage should be local to the CKAN install, and not a remote service. Any archiving to remove storage providers should be outside of the main request.

File storage should:

• allow moving data, a sysadmin should be able to move the storage root and change configuration and have the system continue running (i.e. don't store absolute paths).
• provide maintainability, it should be easy to determine which old files are not associated with resources and thus can be cleaned up.
• allow for collection of information (i.e. estimate of storate space used)
• check whether there is enough space and handling the conequences cleanly
• ensure files to be written only underneath its own root folder, checks should be made after any path generation that the file begins with the location of the file storage.
• Have a configurable maximum accepted blob size during upload.
• Should store what meta-data was provided with the upload, such as mimetype.

Somewhere in the DB we should store ...

Generating paths should try and separate the files, perhaps based on username of the owner, or some other mechanism to avoid a single folder full of files.

Notifications

We need to make sure that it is possible to notify other components within the system that an upload has taken place, or at least make it easy for them to be notified. The primary use case for this is to notify the component that will translate/upload certain formats to the data store.

We could do this based on the post-upload update to the file model (i.e. when we record the total received size of the file).

Steps to reproduce:

1. go to edit a resource, e.g. ​http://s031.okserver.org:2375/dataset/ted/resource_edit/a621a6ad-6f1c-46d1-b858-03cdb7a25c07

2. Change the format field to something it doesn't get. e.g. add a comma "HTML,"

3. Get message: "The form contains invalid entries:

Last modified: Date format incorrect Size: Invalid integer"

4. Format field reverts, try to re-save/submit form - i.e. click on "update resource"

5. nothing happens (same if you even try changing the format field

We need to check the archiver to make sure it works as we would expect after the changes in #2732:

• Receive notification of a file that needs processing
• Check the file's DB entry and upload the file to the configured remote service
• Update the URL of the file (both in the file's DB entry and the resource)
• Once we're sure the file is safely archived, we should mark the file as being archived

so that we know it can safely be deleted at some point.

Maybe this could/should be implemented as a service rather than a celery task?

Why is datastore not enabled for this resource ​http://s031.okserver.org:2375/dataset/frere/resource/cdeddb39-b0cb-4762-aa5b-f80fc892d4dd ?

Is this going to be a site wide setting? How do new resources get added to datastore? Or are we just using the dataproxy for demo?

When adding a resource, the breadcrumb should say "add data" or "add resource" not "create dataset" as you see here: ​http://s031.okserver.org:2375/dataset/new_resource/frere

​http://localhost:5000/ckan-admin/authz

you need to be an admin

paster sysadmin add <username> -c dev.ini

like #2741 we need other admin links

you may want to make the inputs use your form macros - I've just done a basic port

also we will need to get all the tables done but looks like we need to get auth groups working for that

Hey, this is still missing. It's the one with the add dataset text

When editing a resource you see the current and any other existing resources in sidebar

(e.g. see ​http://s031.okserver.org:2375/en/dataset/format-error-test/resource_edit/d1eac556-c16f-44af-8148-5e3467b57cf8?inner_span=True)

Would be good to have a pretty 'add new' slightly transparent resource folder/pointer undearneath - letting you add resources from the end resource page

we need to ensure that translations in .js files are ripper for translation

look at what is available as preference to creating our own

When navigating to ​http://s031.okserver.org:2375/group on the demo server, the Group link in the navbar is not highlighted in the same was as the other links are when visiting those pages.

Specifically the 'add resource' button and the 'add item' button in related section