priority changed; milestone set

rgrp — Wed, 02 Feb 2011 09:12:07 GMT

priority changed from awaiting triage to critical
milestone set to ckan-v1.4-sprint-1

Comments from RP - http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html

Libraries I have used: FormEncode?, FormAlchemy? (what we are currently using, before that formencode).

Neither seemed perfect but I think the form issue is a 'hard' problem (perhaps with no perfect answer) [1]. FormAlchemy?, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing.

At least 3 functions involved:

Generating (or just filling) a form template with 'form data' (and errors)
Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization'
Converting form data to model data (and validating) (inverse of

previous step)

I think one can and should separate 1 from 2+3 (and one of problems with formalchemy is it doesn't -- the attraction being you don't repeat yourself as forms get generated from model but I think this is actually a false economy in medium-term).

I'm not specifically recommending the following as I haven't used them but I've looked through docs, they are active and reasonably mature:

Flatland: http://discorporate.us/projects/flatland/docs/tip/
- Only does 2+3 which is a good thing IMO

WTForms: http://wtforms.simplecodes.com/
- Used in standard flask docs: <http://flask.pocoo.org/docs/patterns/wtforms/>

description changed

rgrp — Mon, 07 Feb 2011 09:11:24 GMT

description modified (diff)

rgrp — Wed, 23 Feb 2011 19:31:57 GMT

@Seb: I believe this is now decided following discussion last week. Please could you detail results and close :)

status changed; resolution set

anonymous — Thu, 24 Feb 2011 09:59:57 GMT

status changed from new to closed
resolution set to fixed

Goals:

We want the interface for updating an object to be loosely coupled to

the method for updating it.

We might update a Package from:

HTML forms
a REST API (using JSON)
a CLI (potentially using command line arguments, YaML, XML or ini files)

Right now, data is validated using a form framework, even if we're not using forms. Data is written to the object as part of the forms framework (using the "sync()" method), making the process hard to customise and hard to discover.

Instead, there should be a standard chain for:

deserialising untyped data (such as that received from an HTTP POST or parsed from a YaML file) into valid data
returning structured errors suitable for displaying to the user
saving the validated, deserialised data

Ideally, it would look something like:

schema = MySchemaDefinition?() raw_data = open("raw.csv", "r").read() structured_data = to_python(raw_data, schema) try:

validated = validate(python_data) myobject.update_from_dict(validated) return "Updated OK"

except ValidationError?, e:

return "Error: %s" % e.to_dict()

The inverse would be something like:

structured_data = myobject.render_to_dict() raw_data.write(to_csv(structured_data, schema) print "Wrote CSV %s" % to_logformat(serialized_data, schema)

The question of how to generate and display forms should be completely decoupled from this. It should be easy to write forms by hand, which means it should be simple to flatten the serialized data to key, value pairs, and match up any validation errors to each key.

Optionally, a form widget generation framework is a nice-to-have, but not essential, as it is expected that, given enough time, the majority of forms will require manual coding to accomodate edge conditions.

A form widget generation framework should be reasonably complete if it's worth trying at all, which means it should support things like:

nested fields (at least repeating, multi-value fieldsets)
widgets for dates and file uploads
internationalisation

...but note I'd settle for *no* widget generation

Components of a serialisation / validation framework:

a simple, obvious way to define a schema
a lightweight validation implementation
- simple interface for validators
- easy to match validation errors to data structure items

Overall, I'd like to see:

loose coupling, no framework dependencies
maximal test coverage
extensive documentation with readily available examples

## Findings

I looked at flatland, formencode, FormAlchemy?, formish, WTForms, Django, web2py, deform/colander, formconvert and web.py

web2py just helps build HTML from python, so isn't what I'm after at all
web.py has rudimentary validation which is only aimed at HTML forms and is hence tightly coupled with them.
Django's forms are again tightly coupled to HTML forms (and their generation)
FormAlchemy? similarly couples validation to forms, and is focussed on inferring a schema from a data model SQLAlchemy.
WTForms again focuses on Form generation and don't make itx easy to deserialise arbitrary data

This leaves us with Flatland, Formencode, Formish, Colander/Peppercorn/Deform?, and FormConvert?.

Having reviewed all of these, I rejected Formencode on the basis of its patchy documentation and relatively low unit test coverage. I also found it mixed concerns a bit much for my taste.

Formish felt similarly sparsely documented.

Of the remainder, I'd be happy using any of them, but opted for Colander in the end as it has the most exhaustive documentation and unit tests and has been used in production for a long time. FormConvert? has a nice design but is a bit of a moving target at the moment -- worth revisiting in the future.

CKAN: Ticket #926: Pick a simpler form framework

priority changed; milestone set

description changed

status changed; resolution set