ticket PosixTime author newvalue 4 1220900713000000 rgrp Dependent on upgrade to vdm v0.2 (sqlalchemy). Once that is done should be fairly simple (can port query stuff from microfacts?). 4 1223908381000000 rgrp r368. Search by package title and name implemented. Other options will become separate tickets. 5 1157371211000000 rgrp See changeset:37 (searching for packages) and changeset:38 (A-Z finding) 5 1185473622000000 rgrp See r64 ff. 7 1157371311000000 rgrp See changeset:39 7 1204136209000000 rgrp "No longer have concept of releases in CKAN domain model (if we did with vdm could now implement simply as a package at a particular point in time). " 8 1204135181000000 rgrp Probably part of ticket:43 (generic attributes) now so downgrading priority (also release no longer exists as a part of CKAN domain model). 8 1250619147000000 rgrp Not part of domain model conception for a long time so closing. If we need something like this can open a new ticket. 11 1157371568000000 rgrp See changeset:39 12 1157371398000000 rgrp See changeset:39 12 1204135071000000 rgrp No longer have concept of releases in CKAN domain model (if we did with vdm could now implement simply as a package at a particular point in time). 13 1157373546000000 rgrp See changeset:39 13 1218545330000000 rgrp No longer have concept of releases in CKAN domain model. See ticket:12. 14 1157371389000000 rgrp See changeset:39 14 1204136163000000 rgrp "No longer have concept of releases in CKAN domain model (if we did with vdm could now implement simply as a package at a particular point in time). " 15 1204135174000000 rgrp Probably part of ticket:43 (generic attributes) now so downgrading priority (also release no longer exists as a part of CKAN domain model). 15 1250619139000000 rgrp Not part of domain model conception for a long time so closing. If we need something like this can open a new ticket. 16 1204135164000000 rgrp Probably part of ticket:43 (generic attributes) now so downgrading priority (also release no longer exists as a part of CKAN domain model). 16 1250619162000000 rgrp Not part of domain model conception for a long time so closing. If we need something like this can open a new ticket. 17 1204135155000000 rgrp Probably part of ticket:43 (generic attributes) now so downgrading priority (also release no longer exists as a part of CKAN domain model). 17 1250619180000000 rgrp Not part of domain model conception for a long time so closing. If we need something like this can open a new ticket. 18 1199787607000000 rgrp At present we do not have a person domain object on the system as we use openid for authentication and only use username for logging etc. May change this at some point in future but at that point can reopen. 19 1199787631000000 rgrp At present we do not have a person domain object on the system as we use openid for authentication and only use username for logging etc. May change this at some point in future but at that point can reopen. 20 1185473187000000 rgrp One can do this from python via the model API. Resolved with use of vdm back in r121. 21 1254740405000000 dread Done as part of ticket:126 22 1199787243000000 rgrp At present we do not have a person domain object on the system as we use openid for authentication and only use username for logging etc. May change this at some point in future but at that point can reopen. 23 1176473102000000 cheaptix "cheap airline tickets cheap plane tickets cheap tickets london cheap airplane tickets dirt cheap tickets cheap concert tickets cheap flight tickets cheap flights paris cheap flights sydney cheap air flights cheap flights dubai cheap flights rome cheap flights prague cheap flights manila cheap air flights array cheap flights to europe cheap flights uk cheap international flights cheap airfares manila" 23 1176819354000000 imagesdische "cheap air flights array cheap airplane tickets cheap cell phones cheap checks cheap cigarettes cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap shoes cheap textbooks cheap tires cheap train tickets coach outlet dirt cheap tickets discount airline tickets discount auto parts discount carpet discount clothing discount cologne discount cosmetics discount cruises discount fabric discount fragrance discount jewelry discount kitchen cabinets discount lighting discount lodging discount luggage discount motorcycle tires discount perfume discount rugs discount shoes discount sunglasses discount tire discount wheels hair straightener sears outlet" 23 1176986042000000 cheap-n-save "cheap airfare cheap air flights array cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap cell phones cheap checks cheap cigarettes cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap shoes cheap textbooks cheap tires cheap train tickets" 23 1177064500000000 discounts "discount airfare discount air fares discount airlines discount airline tickets discount air travel discount auto parts discount carpet discount clothing discount cologne discount cosmetics discount cruises discount fabric discount flights discount fragrance discount jewelry discount kitchen cabinets discount lighting discount lodging discount luggage discount motorcycle tires discount perfume discount plane tickets discount rugs discount shoes discount sunglasses discount tire discount tire direct discount tires online discount wheels" 23 1177340490000000 partsforu "aftermarket car parts antique car parts appliance parts atv parts boat trailer parts car parts chevy truck parts classic car parts corvette parts custom car parts custom motorcycle parts diesel truck parts discount motorcycle parts dodge parts dodge truck parts ford auto parts ford mustang parts ford oem parts ford ranger parts ford tractor parts ford truck parts ge appliance parts genuine toyota parts gm parts gm performance parts gun parts harley davidson parts honda atv parts honda car parts honda motorcycle parts jeep parts mercedes parts motorcycle parts napa auto parts nissan parts polaris atv parts sears appliance parts sears parts" 23 1177416347000000 airlinetickets "cheap airline tickets cheap airline tickets london airline tickets italy cheep airline tickets dirt cheap airline tickets low cost airline tickets robots dirt cheap airline tickets robots low cost airline tickets discounted airline tickets robots cheep airline tickets robots discounted airline tickets discount airline tickets robots cheap airline tickets robots cheap airline tickets flights first class low cost airline tickets cheap air flights low cost airline tickets airline tickets sexy nude bangkok manila www com first class dirt cheap airline tickets aeroflot airline tickets airline tickets online cheap first class discounted airline tickets airline flight tickets cheap airline tickets first class cheep airline tickets airline tickets sexy islands hawaii miami florida travel las vegas hotels airline tickets cheapest airline tickets first class cheap airline tickets dirt cheap airline tickets travl new zealand first class cheap airline tickets flights discount airline tickets cheap discounte airline ticket airline tickets cheep cheep tickets cheep airline tickets cheep tickets book airline tickets cheap cruises carribean airline tickets flights cheapest plane tickets flights airline flight ticket airline tickets flights paris france madrid spain airline tickets paris london rome amsterdam cheep airline ticket cheep airline tickets airline tickets discounted flights cheep airlines cheep airline tickets first class airline tickets cheap international flights airline tickets cheap last minute dirt cheap airline tickets brazil rio de janeiro airline tickets cheep cheep airline tickets airline tickets really cheap airline ticket cheap discounted airline tickets discounted airline tickets discounted flights airline tickets cheap flights airline cheep airline tickets com cheep airline tickets airline tickets cheap last minute airline tickets southwest airline tickets airline tickets student airline fares cheap airline tickets brazil panama chile airline tickets for dirt cheap dirt cheap airline tickets cheap discounted flights airline tickets cheap discounted airline tickets europe cheep airline tickets australia cheep airline tickets dirt cheap airline tickets online airline flight ticket airline tickets search cheap airfares airline tickets economy airticket cheap argentina brazil airline tickets cheapest airline tickets online cheap air flight ticket southwest reservation desk airline tickets airlines com fly cheap airline tickets cheap flights airfares discount airline tickets argentina brazil discounted airline tickets discounted airline tickets cheep airline tickets london cheep airline tickets airline tickets cheap international flights airline tickets northwest airline last minute airline tickets online cheap air flights business economy airline tickets flights low cost airline tickets low cost airline tickets airlinetickets airline tickets delta airline airline cheap dirt ticket dirt cheap airline tickets airline tickets dirt cheap dirt cheap airline tickets cheap airlines tickets manila cebu airline tickets hotels domestic airline tickets domestic travel discount airline tickets airline discount central airline tickets discount airline ticket low cost cheap airfares low cost airline tickets dirt cheap airline ticket dirt cheap airline tickets dirt cheap airline tickets usa cheap flights from usa dirt cheap airline tickets dirt cheap airline tickets airline tickets discounts flights cheap flights airfares dirt cheap airline tickets online cheap flights airfares cheap international airline tickets airline tickets cheap airline fares discount military airline tickets airline discount central cheap low cost airline tickets low cost airline tickets low cost tickets south america low cost airline tickets low cost airline ticket low cost airline tickets airline tickets cheap first class airfares coach class airfares low cost airline tickets low cost economy ticket low cost airline tickets low cost car rentals low cost airline tickets australia sydney hotels flights airline tickets australia airline tickets travel continental airline tickets cheap airline tickets online low cost hotels flights travel low cost airline tickets southwest reservation desk airline tickets airline flights cheap airline tickets philippine japan airlines tickets cargo reservation flights tokyo phillippine airline tickets flights travel business first class airline tickets philippine cheap airline tickets cheap flights to london low cost airfares low cost airline tickets airline tickets flights flights international airline tickets inexpensive airline tickets singapore flights airline tickets travel dirt cheap low cost airfares london england low cost airline tickets really cheap airline tickets low cost airline tickets flights low cost airline tickets latin america travel airline tickets airline tickets singapore japan airlines student airline tickets airline tickets online airline tickets discounted airlines cheap air flight ticket delta airline tickets free airline tickets very cheap airline tickets airline tickets travel tokyo japan nude first class tickets bargain airline tickets favicon low cost airline tickets low cost fares lax low cost airline tickets robots airline tickets cheap international flights student discount airline tickets airline discount central one way airline tickets airline tickets go american airlines purchase airline tickets online airline tickets discount airline tickets to hawaii low fare airline tickets military discount airline tickets airline tickets low price student discount airline tickets discounted airline tickets china beijing favicon cheep airline tickets discount military airline tickets online airline tickets cheap southwest airline tickets buy airline tickets online cheap last minute airline tickets low price airline tickets cheapest airline tickets cheap flights to paris cheap airline tickets tokyo japan cheap airline tickets to india orbitz airline tickets osaka tokyo nrt osa japan fligts airline tickets cheap canadian airline tickets airline tickets usa international cheap flights from usa airline tickets to florida book airline tickets flights japan hawaii airline tickets cheap airline tickets for students cheap airline tickets to las vegas last minute cheap airline tickets alaska airline tickets buy airline tickets bid for airline tickets discount international airline tickets cheap airline tickets to europe cheap airline tickets canada cheap air flights low cost airline tickets st airline tickets to orlando cheap student airline tickets cheap airline tickets to mexico cheap airline tickets low cost airfare best time to buy airline tickets first class airline tickets united airline tickets spirit airline tickets jet blue airline tickets airline tickets low fare paris france bid on airline tickets cheap airline tickets to florida cheap airline tickets to italy low airline tickets discount airline tickets to europe las vegas airline tickets cheap airline tickets to london cheap airline tickets to hawaii travel airline tickets american airline tickets airline tickets to las vegas airline tickets airline tickets cheap international flights singapore airline tickets flights travel cheap one way airline tickets cheap airline tickets europe cheap airline tickets rio de janeiro brazil low cost flights american airlines low cost airline tickets dirt cheap airline tickets online paris france travel airline tickets cheapest best airline tickets purchase airline tickets cheap bereavement airline tickets lowest airline tickets purchase airline tickets airline tickets brazil bargain discounts for students on airline tickets lowest price airline tickets buying cheap airline tickets cheapest airline tickets available priceline airline tickets college student discount airline tickets deep discount airline tickets airline tickets europe paris france airline tickets discounted flights st discount business class airline tickets delta airlines tickets new zealand airline tickets low cost affordable airline tickets dicount airline tickets buy cheap airline tickets favicon cheap airline tickets japanes airline tickets northwest airline tickets consolidator airline tickets round the world airline tickets wholesale airline tickets airline tickets united brasilia airlines airline tickets london discount airline tickets to hawaii new zealand last minute airline tickets discounted airline tickets flights japan find cheap airline tickets cheat airline tickets ardeche france travel discount student airline tickets cheap flights airline tickets cheap domestic airline tickets airline tickets delta airlines rio airline tickets low fares paris france airline tickets to italy airline tickets to london greece cheepest airline tickets profile cheep airline tickets argentina brazil airline tickets discounted tokyo airline tickets cheap fares vivtour airline tickets best price airline tickets last minute airline tickets cheap cheap airline tickets to the philippines travel tip international airline tickets travel tips budget airline tickets new zealand airline tickets cheap fares airline tickets bangkok paris france air france discount airline tickets airline tickets military discount airline tickets northwest airline st rail travel france airline tickets cheap discount ukraine cheap airline tickets asia cheap airline tickets to orlando singapore airline tickets to asia tokyo airline tickets low fare where can i find cheap airline tickets cheap air flights low cost airline tickets bermuda cheap air flights low cost airline tickets oita tokyo airline tickets for cheap air travel france price airline tickets cheap airline tickets flights japan cheap airline tickets in canada cheap airline tickets kiev brazil airline tickets usa australia sydeny cheap airline tickets flights airfares expedia airline tickets airline tickets united airlines brazil airline tickets to greece airline tickets to chicago airline tickets south africa paris france air france america travel discount and airline tickets discount airline tickets canada new zealand really cheap airline tickets new zealand bussiness class airline tickets new zealand first class airline tickets airline poland tickets europe airline tickets brazil europe airline tickets brazil online airline tickets flights brazil airline tickets flights poland airline tickets for cheap airline tickets northwest rio airlines airline tickets thailand brazil airline tickets to south america paris france frontier airline tickets tokyo buy cheap airline tickets tokyo airline tickets low fares chep airline tickets best prices on airline tickets cheep airline tickets for greece students cheep airline tickets flights japan cheep airline tickets com cheep airline tickets st spain low cost airline tickets air france airline cheap airline tickets europe cheap first class airline tickets train travel in france airline tickets low cost compare airline tickets" 23 1178240327000000 cheapjust "cheap airfare cheap air flights array cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap cell phones cheap checks cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap shoes cheap textbooks cheap tires cheap train tickets dirt cheap tickets" 23 1178878776000000 cheapticket "airline discount tickets cheap airfare cheap air flights array cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap cell phones cheap checks cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap shoes cheap textbooks cheap tires cheap train tickets dirt cheap tickets" 23 1178929178000000 shopforu "abercrombie & fitch air compressors air force ones air jordan shoes air mattress air purifiers aluminum boats aluminum dock plates american eagle outfitters antique engagement rings athletic wear atv accessories atv tires baby phat ballet shoes barely string bikini baseball jersey bass shoes bath tubs belly button bottled water brass knuckles bridal bouquets bridal gowns bridal shoes bridesmaid dresses calorie counter candle holder ceiling fans clarks shoes coffee makers coffee tables coloring sheets comforter sets cork flooring country curtains cross stitch curly hair delta faucets detox diet dress shoes duct tape dune buggy duvet covers ear plugs earth shoes egr valve electric blankets electric fence elliptical trainers embroidery designs empire carpet exercise equipment exhaust fans fireplace mantels fishing reels fishing rods fishing tackle fish tanks flax seed floor mats fountain pens futon covers" 23 1179471344000000 shopforU "aftermarket car parts antique car parts appliance parts atv parts boat trailer parts car parts chevy truck parts classic car parts corvette parts custom car parts custom motorcycle parts diesel truck parts discount motorcycle parts dodge parts dodge truck parts ford auto parts ford mustang parts ford oem parts ford ranger parts ford tractor parts ford truck parts ge appliance parts genuine toyota parts gm parts gm performance parts gun parts harley davidson parts honda atv parts honda car parts honda motorcycle parts jeep parts mercedes parts motorcycle parts napa auto parts nissan parts polaris atv parts sears appliance parts sears parts suzuki atv parts suzuki motorcycle parts toyota auto parts toyota oem parts toyota parts toyota truck parts used motorcycle parts volvo parts vw parts whirlpool appliance parts yamaha atv parts yamaha motorcycle parts airline discount tickets cheap airfare cheap air flights array cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap cell phones cheap checks cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap shoes cheap textbooks cheap tires cheap train tickets dirt cheap tickets airline tickets airline travel tickets europe airline tickets charter airline tickets booking airline tickets airline tickets uk low rate airline tickets discount on airline tickets buy airline tickets south america airline tickets airline tickets japan airline tickets go american airlines airline tickets discounts inexspensive airline tickets cheap air flights cheap airline tickets chapest airline tickets student airline tickets" 23 1179778389000000 travelling "airfares rome airline reservations airline travel air travel finder bargain flights best air fares business class flights cheap air travel discount air travel discounted fares discounted flights flight las vegas flights johannesburg flights moscow flights panama flights riga flights tokyo flights vilnius flight tracker hotel reservations last minute airfare last minute flights low air fares lowest airfares southwest airlines fares alaska cruise bahamas cruise best cruise lines caribbean cruises carnival cruise carnival cruise lines celebrity cruise line celebrity cruises cruise deals cruise lines cruise packages cruise reviews cruise ship jobs cruise ship reviews cruise ships cruise specials cruise vacations disney cruise disney cruise line european cruises hawaii cruises holland america cruise line last minute cruise deals last minute cruises mediterranean cruises norwegian cruise lines norwegian cruises oceania cruises princess cruise lines princess cruises royal caribbean cruise lines royal caribbean cruises singles cruises airline discount tickets cheap airfare cheap air flights array cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap train tickets dirt cheap tickets discount airfare discount air fares discount airlines discount airline tickets discount air travel discount cruises discount flights discount lodging discount luggage discount plane tickets" 23 1179798284000000 prilosecwprGC " pc pitstop black clit butte black women sex cumface " 23 1179822590000000 prilosecwprGC " teen peeing black clit girl underwear sidekick 2 flash tits " 23 1179837607000000 prilosecwprGC " sleep fuck dirty blondes tranny movies sidekick 2 cyberage " 23 1179860405000000 prilosecwprGC " strip dancer tarzan porn girl underwear erotic enema cumface " 23 1179881743000000 prilosecwprGC " strip dancer adult adhd butte sex school blood gangs " 23 1180036061000000 travelling "airline discount tickets cheap airfare cheap air flights array cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap cell phones cheap checks cheap cigarettes cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap shoes cheap textbooks cheap tires cheap train tickets coach outlet dirt cheap tickets discount airfare discount air fares discount airlines discount airline tickets discount air travel discount auto parts discount carpet discount clothing discount cologne discount cosmetics discount cruises discount fabric discount flights discount fragrance discount jewelry discount kitchen cabinets discount lighting discount lodging discount luggage discount motorcycle tires discount perfume discount plane tickets discount rugs discount shoes discount sunglasses discount tire discount tire direct discount tires online discount wheels hair straightener sears outlet airfares rome airline reservations airline travel air travel finder bargain flights best air fares business class flights cheap air travel discount air travel discounted fares discounted flights flight las vegas flights johannesburg flights moscow flights panama flights riga flights tokyo flights vilnius flight tracker hotel reservations last minute airfare last minute flights low air fares lowest airfares southwest airlines fares alaska cruise bahamas cruise best cruise lines caribbean cruises carnival cruise carnival cruise lines celebrity cruise line celebrity cruises cruise deals cruise lines cruise packages cruise reviews cruise ship jobs cruise ship reviews cruise ships cruise specials cruise vacations disney cruise disney cruise line european cruises hawaii cruises holland america cruise line last minute cruise deals last minute cruises mediterranean cruises norwegian cruise lines norwegian cruises oceania cruises princess cruise lines princess cruises royal caribbean cruise lines royal caribbean cruises singles cruises" 23 1180333272000000 uploajs "1000 loan payday advance cash loan online payday advance cash loan payday quick bad credit payday loans cash advance payday loan cash till payday loan default payday loan easy payday loan fast payday loans faxless payday loans guaranteed payday loans instant payday loan loan till payday military payday loan no credit check payday loans no fax payday loans no teletrack payday loans online payday loan service payday advance payday cash advance payday loan canada payday loan illinois payday loans payday loan utah same day payday loans airline discount tickets cheap airfare cheap air flights array cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap train tickets dirt cheap tickets discount airfare discount air fares discount airlines discount airline tickets discount air travel discount cruises discount flights discount plane tickets" 23 1180424387000000 cheapfurn "cheap airline tickets airline travel tickets discount airline tickets europe charter airline tickets travel airline tickets cheap airline tickets uk low rate airline tickets discout airline tickets cheapest airline tickets to europe deals airline tickets airline tickets prices airline tickets go american airlines lowest prices for airline tickets best buy airline tickets cheap airline tickets and hotels find airline tickets online soutwest airline tickets air cheap tickets cheap tickets cheap tickets for airline cheap travel tickets dirt cheap tickets find cheap tickets plane cheap tickets aluminum patio furniture antique bedroom furniture ashley bedroom furniture ashley furniture ashley furniture store ashley home furniture bedroom furniture bedroom furniture sets bush office furniture cast aluminum patio furniture cheap bedroom furniture cheap office furniture cheap patio furniture cherry bedroom furniture child bedroom furniture childrens bedroom furniture contemporary bedroom furniture contemporary office furniture discount bedroom furniture discount office furniture discount patio furniture ergonomic office furniture executive office furniture home office furniture kids bedroom furniture modern bedroom furniture modern office furniture modular office furniture new office furniture oak bedroom furniture oak office furniture office computer furniture office furniture outdoor patio furniture patio furniture patio furniture covers patio furniture cushions patio furniture sets pine bedroom furniture resin patio furniture teak patio furniture unfinished furniture unfinished wood furniture white bedroom furniture wholesale office furniture wholesale patio furniture wicker patio furniture wood office furniture wood patio furniture wrought iron patio furniture youth bedroom furniture" 23 1180523687000000 varuploa "airline discount tickets cheap airfare cheap air flights array cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap cell phones cheap checks cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap shoes cheap tires cheap train tickets coach outlet dirt cheap tickets discount airfare discount air fares discount airlines discount airline tickets discount air travel discount auto parts discount carpet discount clothing discount cologne discount cosmetics discount cruises discount fabric discount flights discount fragrance discount jewelry discount kitchen cabinets discount lighting discount lodging discount luggage discount motorcycle tires discount perfume discount plane tickets discount rugs discount shoes discount sunglasses discount tire discount tire direct discount tires online discount wheels hair straightener sears outlet 1000 loan payday advance cash loan online payday advance cash loan payday quick bad credit payday loans cash advance payday loan cash till payday loan default payday loan easy payday loan fast payday loans faxless payday loans guaranteed payday loans instant payday loan loan till payday military payday loan no credit check payday loans no fax payday loans no teletrack payday loans online payday loan service payday advance payday cash advance payday loan canada payday loan illinois payday loans payday loan utah same day payday loans" 23 1180689166000000 vacchedis "cheap airline tickets airline travel tickets discount airline tickets europe charter airline tickets travel airline tickets cheap airline tickets uk low rate airline tickets discout airline tickets cheapest airline tickets to europe deals airline tickets airline tickets prices airline tickets go american airlines lowest prices for airline tickets best buy airline tickets cheap airline tickets and hotels find airline tickets online soutwest airline tickets air cheap tickets cheap tickets cheap tickets for airline cheap travel tickets dirt cheap tickets find cheap tickets plane cheap tickets robots air travel finder travel trailers financing ohio travel last minute travel bargains event and travel planning florida travel guide tv travel shop travel agency asia tennessee travel nursing international travel checklist travel insured international travel to lake tahoe cheap train travel kelley blue book travel trailers israel travel agents europe insurance travel us rail travel in england travel deals paris travel deals las vegas caribbean travel guide inexpensive paris air travel travel india information journal of travel and tourism marketing european bus travel paris airfare discount travel train travel france travel for single people air travel china spain cancun travel agency low paris fare air travel airline discount tickets cheap airfare cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap cell phones cheap checks cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap shoes cheap textbooks cheap tires cheap train tickets coach outlet discount airfare discount air fares discount airlines discount airline tickets discount air travel discount auto parts discount carpet discount clothing discount cologne discount cosmetics discount cruises discount fabric discount flights discount fragrance discount jewelry discount kitchen cabinets discount lighting discount lodging discount luggage discount motorcycle tires discount perfume discount plane tickets discount rugs discount shoes discount sunglasses discount tire discount tire direct discount tires online discount wheels sears outlet" 23 1181111310000000 watpayloan "breitling watches bulova watches cartier watches casio watches cheap watches citizen watches designer watches discount watches gucci watches guess watches hamilton watches invicta watches ladies watches luminox watches luxury watches mens watches military watches movado watches nike watches nixon watches omega watches pocket watch replica watches rolex watches seiko watches skagen watches swatch watches swiss army watches swiss watches tag heuer watches timex watches tissot watches vintage watches wrist watches 1000 loan payday advance cash loan online payday advance cash loan payday quick bad credit payday loans cash advance payday loan cash till payday loan default payday loan easy payday loan fast payday loans faxless payday loans guaranteed payday loans instant payday loan loan till payday military payday loan no credit check payday loans no fax payday loans no teletrack payday loans online payday loan service payday advance payday cash advance payday loan canada payday loan illinois payday loans payday loan utah same day payday loans" 23 1181186079000000 furnchedis "aluminum patio furniture amish oak furniture antique bedroom furniture antique furniture antique furniture hardware ashley bedroom furniture ashley furniture ashley furniture store ashley home furniture baby furniture bedroom furniture bedroom furniture sets bedroom furniture stores bobs discount furniture broyhill furniture bush office furniture cast aluminum patio furniture cedar outdoor furniture cheap bedroom furniture cheap contemporary furniture cheap living room furniture cheap office furniture cheap patio furniture cherry bedroom furniture cherry home office furniture child bedroom furniture childrens bedroom furniture contemporary bedroom furniture contemporary furniture contemporary home office furniture contemporary living room furniture contemporary office furniture country living room furniture custom furniture custom home office furniture custom made furniture custom office furniture custom wood furniture danish modern furniture dining room furniture discount baby furniture discount bedroom furniture discount furniture stores discount home office furniture discount leather furniture discount living room furniture discount office furniture discount outdoor furniture discount patio furniture discount wicker furniture donate furniture ergonomic office furniture executive office furniture furniture stores online home office computer furniture home office furniture kids bedroom furniture kids furniture lane furniture leather living room furniture living room furniture modern bedroom furniture modern contemporary furniture modern furniture modern home office furniture modern italian furniture modern living room furniture modern office furniture modular home office furniture modular office furniture new office furniture oak bedroom furniture oak furniture oak office furniture office computer furniture office furniture outdoor bar furniture outdoor furniture outdoor patio furniture outdoor wicker furniture patio furniture patio furniture covers patio furniture cushions patio furniture sets pine bedroom furniture quality home office furniture reproduction antique furniture resin patio furniture teak outdoor furniture teak patio furniture thomasville furniture unfinished furniture unfinished oak furniture unfinished wood furniture value city furniture white bedroom furniture wholesale office furniture wholesale patio furniture wicker patio furniture wood home office furniture wood office furniture wood patio furniture wrought iron patio furniture youth bedroom furniture airline discount tickets cheap airfare cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap cell phones cheap checks cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap shoes cheap tires cheap train tickets coach outlet dirt cheap tickets discount airfare discount air fares discount airlines discount airline tickets discount air travel discount auto parts discount carpet discount clothing discount cologne discount cosmetics discount cruises discount fabric discount flights discount fragrance discount jewelry discount kitchen cabinets discount lighting discount lodging discount luggage discount motorcycle tires discount perfume discount plane tickets discount rugs discount shoes discount sunglasses discount tire discount tires online discount wheels sears outlet" 23 1181435981000000 wataircru "breitling watches bulova watches cartier watches casio watches cheap watches citizen watches designer watches discount watches gucci watches guess watches hamilton watches invicta watches ladies watches luminox watches luxury watches mens watches military watches movado watches nike watches nixon watches omega watches pocket watch replica watches rolex watches seiko watches skagen watches swatch watches swiss army watches swiss watches tag heuer watches timex watches tissot watches vintage watches wrist watches airfares rome airline reservations airline travel air travel finder bargain flights best air fares business class flights cheap air travel discount air travel discounted fares discounted flights flight las vegas flights johannesburg flights moscow flights panama flights riga flights tokyo flights vilnius flight tracker hotel reservations last minute airfare last minute flights low air fares lowest airfares southwest airlines fares alaska cruise bahamas cruise best cruise lines caribbean cruises carnival cruise carnival cruise lines celebrity cruise line celebrity cruises cruise deals cruise lines cruise packages cruise reviews cruise ship jobs cruise ship reviews cruise ships cruise specials cruise vacations disney cruise disney cruise line european cruises hawaii cruises holland america cruise line last minute cruise deals last minute cruises mediterranean cruises norwegian cruise lines norwegian cruises oceania cruises princess cruise lines princess cruises royal caribbean cruise lines royal caribbean cruises singles cruises" 23 1181886101000000 watchpayday "breitling watches bulova watches cartier watches casio watches cheap watches citizen watches designer watches discount watches gucci watches guess watches hamilton watches invicta watches ladies watches luminox watches luxury watches mens watches military watches movado watches nike watches nixon watches omega watches pocket watch replica watches rolex watches seiko watches skagen watches swatch watches swiss army watches swiss watches tag heuer watches timex watches tissot watches vintage watches wrist watches 1000 loan payday advance cash loan online payday advance cash loan payday quick bad credit payday loans cash advance payday loans cheap payday loans consolidation debt loan payday default payday loan easy payday loan fast cash advance payday loan fast cash online payday loan faxless payday loans guaranteed payday loan instant payday loan military payday loan no credit check payday loans no fax payday loans no teletrack payday loans online payday loan service payday advance payday advance loan payday cash advance payday loan illinois payday loans payday loans online payday loan utah quick payday advance loan same day payday loans" 23 1182027008000000 pillspills "http://volny.cz/pills1/aciphex.html aciphex http://volny.cz/pills1/ambien.html ambien http://volny.cz/pills1/buy-generic-cialis.html buy generic cialis http://volny.cz/pills1/buy-levitra.html buy levitra http://volny.cz/pills1/buy-tramadol-cod-pharmacy-online.html buy tramadol cod pharmacy online http://volny.cz/pills1/cheap-tramadol-online.html cheap tramadol online http://volny.cz/pills1/cheap-valium.html cheap valium http://volny.cz/pills1/cialis.html cialis http://volny.cz/pills1/cialis-levitra.html cialis levitra http://volny.cz/pills1/diflucan.html diflucan http://volny.cz/pills1/fioricet.html fioricet http://volny.cz/pills1/free-ambien.html free ambien http://volny.cz/pills1/generic-viagra.html generic viagra http://volny.cz/pills1/levitra.html levitra http://volny.cz/pills1/nexium.html nexium http://volny.cz/pills1/propecia.html propecia http://volny.cz/pills1/reductil.html reductil http://volny.cz/pills1/tramadol.html tramadol http://volny.cz/pills1/valium.html valium http://volny.cz/pills1/valium-prescriptions.html valium prescriptions" 23 1182146472000000 cheadisc "airline discount tickets cheap airfare cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap cell phones cheap checks cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap shoes cheap tires cheap train tickets coach outlet dirt cheap tickets discount airfare discount air fares discount airlines discount airline tickets discount air travel discount auto parts discount carpet discount clothing discount cologne discount cruises discount flights discount jewelry discount kitchen cabinets discount lighting discount lodging discount luggage discount motorcycle tires discount perfume discount plane tickets discount rugs discount shoes discount sunglasses discount tire discount tires online discount wheels sears outlet" 23 1182834140000000 ChepDiso "airline discount tickets cheap airfare cheap air fares cheap air flights cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap air travel cheap checks cheap concert tickets cheap cruises cheap discount airfare cheap disney tickets cheap flights cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to australia cheap flights to europe cheap flights uk cheap flight tickets cheap hotels cheap international airfare cheap international air fares cheap international flights cheap motels cheap northwest flights cheap plane tickets cheap sunglasses cheap tickets cheap tickets london cheap tires cheap train tickets cheap travel cheap travel packages cheap vacations cheap wedding favors cheap wedding invitations dirt cheap airfare dirt cheap tickets discount airfare discount air fares discount airlines discount air travel discount auto parts discount cosmetics discount cruises discount fabric discount flights discount hotels discount international flights discount jewelry discount kitchen cabinets discount lodging discount perfume discount plane tickets discount shoes discount sunglasses discount tire discount travel discount travel packages military discount flights military discount travel airfares rome airline reservations airline travel air travel finder bargain flights best air fares business class flights cheap air travel discount air travel discounted fares discounted flights flight las vegas flights johannesburg flights moscow flights panama flights riga flights tokyo flights vilnius flight tracker hotel reservations last minute airfare last minute flights low air fares lowest airfares southwest airlines fares " 23 1182925425000000 paydayloancruises "1000 loan payday advance cash loan online payday advance cash loan payday quick bad credit payday loans cash advance payday loans cheap payday loans consolidation debt loan payday default payday loan easy payday loan fast cash advance payday loan fast cash online payday loan faxless payday loans guaranteed payday loan instant payday loan military payday loan no credit check payday loans no fax payday loans no teletrack payday loans online payday loan service payday advance payday advance loan payday cash advance payday loan illinois payday loans payday loans online payday loan utah quick payday advance loan same day payday loans alaska cruise bahamas cruise best cruise lines caribbean cruises carnival cruise carnival cruise lines celebrity cruise line celebrity cruises cruise deals cruise lines cruise packages cruise reviews cruise ship jobs cruise ship reviews cruise ships cruise specials cruise vacations disney cruise disney cruise line european cruises hawaii cruises holland america cruise line last minute cruise deals last minute cruises mediterranean cruises norwegian cruise lines norwegian cruises oceania cruises princess cruise lines princess cruises royal caribbean cruise lines royal caribbean cruises singles cruises " 23 1183096002000000 cheadisconsdebt "airline discount tickets cheap airfare cheap airline flights cheap airline tickets cheap airplane tickets cheap air tickets cheap cell phones cheap checks cheap cruises cheap discount airfare cheap flights dubai cheap flights manila cheap flights paris cheap flights prague cheap flights rome cheap flights sydney cheap flights to europe cheap flights uk cheap flight tickets cheap international flights cheap plane tickets cheap shoes cheap textbooks cheap tires cheap train tickets coach outlet dirt cheap tickets discount airfare discount air fares discount airlines discount airline tickets discount air travel discount auto parts discount carpet discount clothing discount cologne discount cosmetics discount cruises discount fabric discount flights discount fragrance discount jewelry discount kitchen cabinets discount lighting discount lodging discount luggage discount motorcycle tires discount perfume discount plane tickets discount rugs discount shoes discount sunglasses discount tire discount tire direct discount tires online discount wheels hair straightener sears outlet bad credit debt consolidation consolidation loans credit card consolidation credit card debt consolidation credit consolidation debt consolidation debt consolidation loans free debt consolidation mortgage debt consolidation credit card debt credit card debt elimination credit card debt relief credit card debt settlement debt elimination debt management debt reduction debt relief debt settlement 1000 loan payday advance cash loan online payday advance cash loan payday quick bad credit payday loans cash advance payday loans cheap payday loans consolidation debt loan payday default payday loan easy payday loan fast cash advance payday loan fast cash online payday loan faxless payday loans guaranteed payday loan instant payday loan military payday loan no credit check payday loans no fax payday loans no teletrack payday loans online payday loan service payday advance payday advance loan payday cash advance payday loan illinois payday loans payday loans online quick payday advance loan same day payday loans " 23 1183510187000000 financepaydayguaranteed "car insurance rate auto loan dental plans ephedra diet pills cash advance cincinnati home refinance mortgage cheap flights london cheap flights cheap flights cairo airline tickets cell phones cell phone reverse number cell phone radiation cruises bahama cruises fast payday loans no credit check payday loans no fax payday loans payday loans online bad credit payday loans faxless payday loans payday bad credit loans online payday loans no credit check payday loans no teletrack payday loans cash advance payday loans cheap payday loans payday loans for bad credit payday loans without direct deposit cash advances online fast payday loans same day payday loans cash advance loans payday internet online payday loans no pay day fax cash advance fast payday loans payday loans no credit check fast cash online paycheck advance bad credit payday loans online payday loans pay day fax cash advance acircordm personal payday loans online fast payday cash need cash online payday loan personal loans person payday advance loans payday loan online fast cash loans by western union advance cash fast in loans online payday quick unt instant payday loans payday loan online fast cash loans western union payday loans with bad credit personal loans payday advance bad credit most approval advance payday hold personal check cash loans payday loans with low fees personal loans bad credit cash payday advances cash advance and payday loans payday loans guaranteed no fax quick payday loans get payday cash advance fast online loans no fax payday loans online cash advance loan sonic cash payday loans cheap payday loan cash advance chicago guaranteed payday loans payday loan online fast cash loans by western unio payday loans for people with bad credit national cash advance payday loans online payday loan cash advances personal loans pa fast cash payday loans fast payday online cash loans search illinois cash advance payday loans payday loans with no teletrack personal loans borrow cash payday loan online payd alaska cash advance payday loans banks issue payday loans canada student loans payday advance cash advance service payday loans minnesota cash advance payday loans payday loans no faxing cash advance payday loans software louisiana cash advance payday loans payday loan personal loans online cash advances pa washington cash advance payday loans emergency advance payday hold personal check cash loans no fax cash advance payday loans payday loans in canada personal loans payday loan online cash advance washington utah cash advance payday loans very bad credit payday loans cards payday loans for canadians with bad credit easy payday loans free payday loans no fax bad credit payday loans payday advance preferred cash loans payday loans bad credit payday loans canada payday loans cash advance personal loans preferred payday loan paydayloanpag same day payday loans bad credit advance payday loans no faxing payday loans online payday cash advance instant loans payday advance pay day online cash loans search payday loans with no credit check vermont cash advance payday loans advance cash fast get in loans online payday apple payday loans canadian payday loans cash advance money payday loans florida cash advance payday loans instant approval payday loans new payday cash advance loans njfastcash personal loans online payday loan payda payday cash advance loans payday loan online personal loans paydayloanpages payday loans cash advances online payday loans for people with bad creditno credit check payday loans for self employed personal loans for people with bad credit not payday loans personal loans online payday loan preferredpaydayl texas cash advance payday loans cash advance payday loans tv connecticut cash advance payday loans software guaranteed payday loans guaranteed payday loan payday loans guaranteed no fax advance cash guaranteed loan payday loans bad credit guaranteed payday mortgage auto uk payday guaranteed approval payday loan guaranteed no fax payday loans guaranteed no fax payday loan borrow money online with quick guaranteed payday loan no faxing payday loan online fax guaranteed guaranteed payday loans for people with bad credit no credit check guaranteed payday loan approval guaranteed no fax payday loan money guaranteed payday loan instant approval 100 guaranteed payday loans borrow money online with quick guaranteed payday loans guaranteed payday advance loan approval guaranteed unsecured payday personal loans approval faxless guaranteed loan payday guaranteed faxless payday loan guaranteed payday loan no faxing with savings account guaranteed payday loan to savings account guaranteed unsecured loans not payday 100 guaranteed approval faxless payday loans fax guaranteed loan no payday guaranteed approval payday loans guaranteed no fax payday loan credit guaranteed unsecured payday personal loan payday loan guaranteed approval payday loan guaranteed no fax guaranteed approval no faxing no credit check payday loan payday guaranteed approval cash loans southeast kansas approval guaranteed loan payday guaranteed fax payday loan guaranteed instant payday loan guaranteed long term payday loans one hour payday loan guaranteed online no fax payday loan guaranteed account guaranteed loan payday savings guaranteed approval short employment payday loan guaranteed fax payday loans guaranteed loan payday online guaranteed personal loan andnot payday payday loan with guaranteed no faxing faxing guaranteed loan no payday guaranteed online approval payday loan no fax guaranteed payday loan without a job payday guaranteed approval cash loans in southeast kansas w guaranteed payday loan 100 guaranteed approval on payday loans cash guaranteed loan payday emergency payday loan no fax guaranteed approval guaranteed 1000 payday loan no credit check guaranteed payday loan credit guaranteed personal loans payday cash no fax no hassle payday loans guaranteed online instant guaranteed payday loans long term payback payday payday guaranteed no fax payday loan faxless guaranteed approval payday loans guaranteed bad credit payday loans guaranteed payday loan up to 3000 roll over guaranteed payday loan without a bank account guaranteed personal loans payday cash loans instant guaranteed payday loan long term payday loans 100000 guaranteed payday guaranteed no fax payday loan payday loan no fax guaranteed advance approval guaranteed loan payday borrow guaranteed loan money online payday quick guaranteed faxless payday loans no teletrak guaranteed no teletrack 2c no faxing payday loans guaranteed payday loans for people with bad credit free guaranteed payday loans vian savings account guaranteed payday loans with savings account online instant guaranteed payday loans long term p payday guaranteed approval cash loans in southeast payday guaranteed payday loan " 23 1199786799000000 rgrp Now using openid for signon so this issue is not relevant. 24 1246441366000000 rgrp "Associated to ticket:60 (front page usability improvements) Fixed in r272:e6de501c384e." 25 1199821005000000 rgrp "Amended ticket substantially. Previous title was ""A registered person creates their own tags for a package"" but this was no longer relevant since anyone may add and amend tags (see ticket:30)." 25 1204131522000000 rgrp Fixed in revisions leading up to r239 and r241. Currently just have autocomplete and no 'suggested tags' as this requires pre-computation and caching of results which needs to be thought about further. 27 1199788321000000 rgrp "In many ways everything that is needed is already in place in that the package update and create controller just require raw posts to do their work. However should implement some kind of authentication/authorization to prevent automated spamming. This would also require some way of distinguishing posts from our own WUI and those coming from outside (at present no checking is performed and the 'post' to these controllers is freely allowed by anyone out there!" 27 1202400048000000 rgrp "Methods potentially in need of access control at present (on controllers/package.py) * update (_update) * create Frankly, for time being we can probably leave these unsecured but with monitoring. Should they be abused we can add access control fairly easily. So decision: no work on access control/auth for time being here. Focus instead should be on: * documentation of post parameters (rgrp) * returning restful status values (201 Created, 204 Updated, etc) * returning machine readable output from read and list (json) " 27 1202996441000000 johnbywater "So, after reading all about REST interfaces, I've written a functional requirement for a RESTfull interface to a web application domain model: http://desire.appropriatesoftware.net/requirements/60/ Basically, the 'fit criteria' require answers to these questions: * What are the URIs? * What's the format? * What methods are supported at each URI? * What status codes could be returned? I think we can hope to make a clean rest interface at (something like) ckan.net/api/rest. Having such a distinct location is fairly common (cf. flickr.com) and prevents any clutering the web browser client interface with aspects of machine client support. To make this point clearer, as RESTfullness claims to describe the web as it is, then the existing web browser interface must already be RESTfull. Hence, the main distinction with CKAN development is between CKAN's RESTfull machine interface and CKAN's RESTfull web browser client interface. Hence, if a CKAN machine client will follow the client-server interaction of a CKAN web browser client, the main requirement is to make sure the format of the responses is client-appropriate. Whilst machine clients can make POST and GET requests in the same way as a web browsers, clearly machine clients don't need lots of decorative HTML when accessing domain objects. Therefore, CKAN needs first to make user of a data format for rendering domain objects, and it needs to detect when it is required to return data in this format. Options for data format are bascially XML or JSON. It looks like JSON is preferable. Options for detecting client type are to prefix normal resource path with /api/rest/ or for the machine clients to pass a format=json parameter with each request. " 27 1202997781000000 johnbywater "So for this ticket, a REST interface for packages would involve: See [wiki:RestfulAPI this wiki page] for more information. " 27 1207683863000000 rgrp "The remaining issue is authentication. Initially was hopeful that we could use openid (perhaps with oauth) but this does not seem possible (despite much talk of openid and oauth being complementary) ... Thus we shall adopt this old-fashioned approach. * A user who wishes to use the web api must obtain a web api key. They do this by: * Going to ckan.net and signing in using their open id. * Visit /account/apikey/: this should generate them a unique uuid key (or show their existing key if they already have one). This key should then be stored in the db along with their openid. * Once a user has an api key they must include in the request when making changes. This may necessitate an update to the restful api to start having a more generic message format (this also makes it more extensible for the future): * auth: auth key * data: data body (what we currently send)" 27 1215543933000000 rgrp Massive ticket now thoroughly addressed (early start in r242 and ff. then later in r275, r276 etc). Also now have start of demo ckan REST client at browser:ckanclient 28 1200902911000000 rgrp Mark as wontfix as not sure what benefit this provides given the the sql dump of data already provided in relation to ticket:38. If necessary it can be reopened in the future. 29 1157371518000000 rgrp See changeset:40. 30 1185472559000000 rgrp "This ticket is partially resolved and partially irrelevant: * Since move to 'ckan2' in Jan 07 (r54 onwards) have had tag support. * Since move to versioned system (r121 and ff.) have allowed tags to be added and amended by anyone." 31 1177006817000000 rgrp Milestone milestone20 deleted 31 1185472236000000 rgrp Can get this functionality via Notes section of package. Might want to reopen in future if specifically need something 'comment-like' rather than 'text-wiki-like' but for time being what we have is fine. 32 1185472746000000 rgrp Added Tag support in r82 to r87 (back in February). 33 1185472909000000 rgrp Basic listing support for tags in r82 to r87. However search (and paging of tag lists) not yet provided. 33 1199788094000000 rgrp Added search support in r190. No paging of tag lists but this is now part of new ticket:42. 34 1253781550000000 dread "dread@dread-laptop:~/hgroot/ckan$ wget http://ckan.net/fff --2009-09-24 09:38:02-- http://ckan.net/fff Resolving ckan.net... 88.198.21.211 Connecting to ckan.net|88.198.21.211|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2009-09-24 09:38:02 ERROR 404: Not Found. " 35 1185471537000000 rgrp Resolved in r157. 36 1185470035000000 rgrp Resolved in r168. 37 1199786536000000 rgrp Fixed in r193 with improvements in r201 and crucial bugfix in r203. 38 1199787336000000 rgrp Partially addressed by r194 and making available of db dump on production machine. 38 1200903004000000 rgrp Closing for time being as sql dump seems sufficient. If JSON is to be provided (per package and with Atom feed of changes) this should be made into a separate ticket. 39 1200761681000000 rgrp Using current vdm system this will be very slow so reassign to v0.6 (when we should have updated to new and better vdm setup). 39 1214244190000000 rgrp Deferring as transition to new vdm has not yet happened will do as part of 0.7. Also fairly low priority ... 39 1220900869000000 rgrp Dependent on ticket:51 (upgrade to vdm v0.2). 39 1223908298000000 rgrp See r363 r364 and r366. 40 1200993319000000 rgrp Fixed in r211. 41 1199787967000000 rgrp r198 upgraded trunk to use 0.9.6 and these changes were deployed on production as of 2008-01-06 (PM). 42 1204133257000000 rgrp See r234 and previous. Pagination now implemented for packages and tags (but not for revisions yet). 43 1214244140000000 rgrp This is currently low priority and should only happen after move to vdm 0.2 (sqlalchemy etc). 43 1241779943000000 rgrp "Started work on this using ""extra"" table. See r426." 43 1246441195000000 rgrp Making this versioned/revisioned turns out to be slightly more complex than anticipated (need a stateful associated list). So defer to 0.95 43 1249410762000000 rgrp "Done in model in changeset:7f9b19d4d54a: ""Implement a fully *versioned* PackageExtra domain object in the model and attach to the Package object via a simple dict-like attribute named 'extras'."". However still need to integrate into WUI which can only happen after conversion to formalchemy (ticket:76) is complete. " 43 1253709802000000 rgrp Split out wui work in two new tickets ticket:124 (show) and ticket:125 (edit) and therefore marking this as done. 44 1215543484000000 rgrp Minor importance and at this point we will defer into v0.7. 44 1223390660000000 rgrp Done in r339. 45 1204133342000000 rgrp Fixed in r221 by JB. 46 1218545499000000 rgrp Fixed in r303. 47 1201600216000000 rgrp "Good suggestion (this has been on my mind for a while). This should be pretty cheap to do, see instructions at: http://wiki.pylonshq.com/display/authkitcookbook/OpenID+Passurl We probably just want to create a normal genshi template and then have a function that renders which can then be set in the config file." 47 1215543616000000 rgrp Fixed in r303. 48 1296663361000000 rgrp Done in cset:2a056fce7798. 49 1214244368000000 rgrp Has become less of a problem since big efforts a few months ago so this can be downgraded for time being. 49 1257244973000000 rgrp Not a problem at the moment. 50 1239018913000000 rgrp Have done a bit of digging but noot currently a priority given direction datapkg is going. 50 1259086615000000 casbon "Probably the effort to install apt cross platform would be prohibitive. I would also like to see pure python if possible so that you could run on GAE." 50 1267648356000000 rgrp Do not see there is much more to do here. 51 1223908230000000 rgrp Done. See r344 to r354. 52 1223908425000000 rgrp Turned out this was already implemented -- see r366. 53 1239133021000000 rgrp See r435. This just provides basic list of revisions affecting this. Next step would be to show diffs but that is another ticket. 54 1223909366000000 rgrp ckan v0.6 (dump only) done in r355 and r358. 54 1230211256000000 rgrp See r362, r365, r370 + more. 55 1223909475000000 rgrp See r356 and r362 and r365. 55 1223909891000000 rgrp See r356 and r362 and r365. 56 1239018857000000 rgrp See r398 + r412. Could do a bit more here to have nicer icons and a big sign on the package indicating whether open or not). 57 1239018717000000 rgrp Done. See r415, r417, r423. (This was tidying up -- most work had already been done by start of last autumn). 58 1239974365000000 rgrp See r442 and r437. 59 1239436422000000 rgrp 1. done in r429. 59 1249411064000000 rgrp Not really clear what needs to be done and how useful it would be. Removing milestone until that is clarified. 59 1272468398000000 rgrp "Think we should resolve this by: * Removing guide page. * Putting a link to ckan user guide (http://wiki.okfn.org/ckan/doc/) in RHS sidebox on front page along with other links ... -- can this link url be made a config option in ini (defaulted to that url on load in lib/base.py or wherever it is loaded) * Move current content of guide (one item about bookmarklet) to http://wiki.okfn.org/ckan/doc/faq/" 59 1273080019000000 johnbywater All points in comment:5 have been implemented. 60 1246434921000000 rgrp 1. is ticket:24 and 2+3 have been done in r258:5b1f4efd09c3, r260:934442318774 etc. 61 1246437341000000 rgrp Very low priority and creates complications (e.g. dump for backup is different from dump for public use). 61 1253612984000000 dread "Done in cset:41151725cc12 Cost: 1h" 62 1249410921000000 rgrp "Defer until after conversion to formalchemy (ticket:76) is complete. " 62 1250181376000000 dread Agreed with rgrp that tags should be allowed to be international letters, but not symbols other than these three: ._- but not uppercase characters. This is done in the checkins for ticket 76. 63 1245656000000000 rgrp Fixed in r63c48818395f 64 1246127063000000 rgrp "Fixed in r269:4a2febc4f1eb. " 65 1246434569000000 rgrp Done in r254:fa2ef73afb40. 66 1250785405000000 dread Done in changeset:f875f0f87cb1 67 1250785122000000 dread Done in changeset:f875f0f87cb1 68 1245697554000000 rgrp Fixed in r7ffc8288cb88 and rc6ac0a80ba11. 69 1246437494000000 rgrp Not sure how useful/important moving to text-only is so will defer this for the present. Use of new license list done in r270:ea466d2721ea. 70 1247827053000000 dread Done on checkins: 46d39c62396a and 44eb4f2c4822. 71 1250181211000000 dread "Done in 1.5 hrs Change set 2b434d63d5fd" 72 1251451954000000 rgrp Done in cset:9199c49a81e6 73 1266510366000000 johnbywater "Moved ticket to okfn tracker: http://www.knowledgeforge.net/okfn/tasks/ticket/256" 74 1250182938000000 dread "Done in 0.5h Checked in to dd2f9713a6a2. Third detail done - enquiry can be done for all packages. Still need to change isitopen site to make use of the parameter ?ckan-package=" 75 1296341223000000 rgrp "Closing as duplicate (invalid also): * Not sure recording view stats for a given package page is that useful -- so invalid here * Resource download is now ticket:937" 76 1250152708000000 dread "Objectives 1 and 2 done. 3 left for another ticket if req'd. Several checkins from 4cd7935adaceup to af32f6c2c5f1" 76 1250175782000000 dread I spent 30 hours on this task. 76 1251295339000000 dread Reopening to improve look of the form. Text input fields need to be wider to be like before (size=40). Try and get remaining field information text on there. 76 1251301765000000 dread Part 3 done in cset:7ace6c5d6c68. 77 1255173809000000 dread "Basic work done in cset:80c83a5be797. Extra work done (rgrp) in cset:ac2fdd73e347" 77 1255176732000000 dread Migrate script done in cset:f232b05eb875 78 1311181280000000 thejimmyg This ticket is now more than 6 months old so closing in line with our new ticketing policy. 78 1314877151000000 zephod Fixed in cset:8638a0ac2255 (fifty year expiry time on the four cookies) 79 1250789298000000 dread "Search: geo landsat Gives: 3 tags found: geo (1), geo-somthing (12), landsat (4) 11 packages found: (usual results) " 79 1250789659000000 dread "[x] only open packages [x] only downloadable packages open licenses start with 'OKD compliant' or 'OSI compliant' (same test as isopen). See model/core.py" 79 1251110610000000 dread "All done in changeset:6155a424e775 Now gives tags results. All words in query are searched for independently unless you enclose them in ""quote marks"". Options for filtering by openness/downloadable." 79 1251111212000000 dread Also changeset:856047aa2ea6 (forgot to add a couple of files) 80 1251111180000000 dread Done in changeset:1f8132df0aa8 81 1268487327000000 rgrp Done in cset:a16c18167b6c/vdm and cset:8e235d730790/vdm (was actually reasonably simple to do). 82 1256565343000000 dread "Done basic work in vdm in cset:2a51e39be179. Previous work in ckan in cset:2cfa1c47acd2 - maybe not needed." 82 1256565441000000 dread Follow on work in ckan in ticket:173. 83 1267648301000000 rgrp Done quite a while ago in cset:2a51e39be179/vdm 84 1340626385000000 icmurray "Moved to ckan-future, and assigned to kindly. @kindly : move to a different milestone, or close/won't-fix as you see fit." 85 1263206391000000 rgrp Done in cset:6c58b1f5a542/vdm. Migration script will be in ckan. 86 1249050202000000 rgrp Done in changeset:e295241f4a53 87 1258470719000000 dread Same as ticket:189 88 1264439112000000 dread Download metadata added - format, descripiton, hash. See ticket:87. 89 1260458636000000 dread API provided for tag counts in cset:5a94d1ac60ef allowing an external tag cloud. 89 1265892698000000 dread Done in cset:66144a206c6b 90 1257535604000000 rgrp All pre-requisites done. However unsure about how we integrate into WUI so downgrading for the moment. 90 1264158298000000 wwaites "see http://semantic.ckan.net/ data can be loaded into the store easily enough (cron job for now?) http://semantic.ckan.net/data/package-name the RDF equivalent of http://ckan.net/package/package-name has sparql interface at http://semantic.ckan.net/sparql not so sure about the assertions Talis has made about the ckan namespace..." 90 1265315677000000 dread "Machine-readable link added in cset:9a7726fec351 but not 303 redirect yet. Cost so far: 1.5h" 90 1265388992000000 dread "303 redirect done in cset:ee207cfe617d. Cost: 1h. rgrp said leave ""Alternative formats"" part to Nick." 90 1265890334000000 rgrp Closing as main part done and alternative formats part is now ticket:247 91 1250864156000000 dread Done in changeset:a6c3af2024da 92 1297344859000000 rgrp wontfix at the moment as we have the separate RDF service and RDFa is probably not the best option as always a bit incomplete. 93 1253613274000000 dread "Split up into smalled tickets including: ticket:114 ticket:115 ticket:116 ticket:117" 94 1250604808000000 dread Done in changeset:631431630ef2 95 1251454716000000 rgrp Done in cset:29a5928e617b 96 1311176063000000 thejimmyg This sounds like an issue with setuptools not CKAN. It has been open for more than 6 months and doesn't appear to have been an issue so closing. 97 1318181317000000 rgrp Duplicate of: https://github.com/okfn/datapkg/issues/5 98 1297210774000000 rgrp Done in cset:b7d8786614c0 - removed pastescript and just do 'by hand'. 99 1297081088000000 rgrp Won't fix as we are not using sqlalchemy any longer. 100 1318181227000000 rgrp Resolving as wontfix since not really sure any point to this. 101 1267648607000000 rgrp "Marking as fixed as Item 1 and item 2 have now been done (at least partially), using urlgrabber. However no ""distribution"" support (we just assume all downloads are ""payloads"" and get metadata from ckan). See cset:ae89cbc898c7/datapkg cset:8448f862c588/datapkg cset:df22cd0f92a1/datapkg etc." 102 1260285104000000 rgrp Done (a couple of weeks or more ago) in cset:061e3f3d253b/vdm. Migration script as used in CKAN in browser:ckan/migration/versions/008_update_vdm_ids.py. 103 1300217647000000 thejimmyg See also #1012 Add package revision history to api 103 1300363180000000 thejimmyg Closing, we'll take this up in #1012. 103 1301943140000000 dread "Didn't take this up in #1012 after all. Closing as wont fix. " 103 1308828847000000 dread Now we have this working in #1141 we can add the links from the package history page suggested here. 103 1311175065000000 thejimmyg "This is now implemented. Date parsing is fairly rudimentary but you can now do things like this: http://test.ckan.net/package/uk-crc-rural-services-data@2011-06-10" 103 1311175601000000 dread This isn't finished - see second point in description and comment 4: Now we have this working in #1141 we can add the links from the package history page suggested here. 103 1311179429000000 thejimmyg "David, I'm in the middle of a ticket refactor. Please don't open tickets I've just closed ;) This will be taken forward as part of #1233 " 103 1311180850000000 dread Ah, you should have mentioned the new ticket! ;-) 105 1252488496000000 dread "All done apart from group owners. This functionality will be left for another ticket. cset:bd4e48a544e2 cset:3263b6612fb7 cset:c6a6f88a50c2 cset:e9111a221222 Took 1d." 106 1256140649000000 dread Ticket split into ticket:163 and ticket:164 107 1257803384000000 rgrp "On the ""long run"" item " 107 1258381854000000 dread Cost: 1h 107 1296341644000000 rgrp "Superseded by plans for upload: * #877 - File upload in WUI: 2d * #878 - Integrate file upload with workflow around package resources: 3d " 108 1252318938000000 dread This has been done in changeset:ec90f59040e0. 109 1291829457000000 thejimmyg "This is effectively implemented by the util API; http://knowledgeforge.net/ckan/hg/file/tip/doc/api/version2.rst John has a separate proposal to move the util API into the REST API but that is a different discussion. Here's how you can now search on tags: /api/2/util/tag/autocomplete?incomplete=ru " 110 1252488660000000 dread This was all done in work for ticket:105 111 1252331626000000 rgrp Done in cset:c1bad304e1a0 112 1264876463000000 rgrp No easy way to test this and now have ticket:239 (test migrate scripts) 113 1252318818000000 dread "Test database directly and via wui. Paste.Fixture to be used via proxy in paster." 113 1252328204000000 dread Done in cset:57b263f51a6b 114 1253034529000000 dread "Done in cset:895ae4371377. Remaining problem in WUI for revision purging. Have not implemented functionality to cope with blank objectid in user-roles table. Adding group access control will be in future ticket." 115 1253034599000000 dread R/W access control of packages added to WUI in cset:3e79c4398a3d. 115 1253091426000000 dread R/W access control of packages added to REST api in cset:a2e0df462427. 116 1253270921000000 dread "Done in cset:683d42238666. " 116 1253709460000000 rgrp Done in cset:df8d7efdcadb and cset:ca1a5a4ef43a 117 1254735855000000 dread Done in cset:aa1435678eb6 118 1253784319000000 dread Done in cset:36b586f95e87 118 1253784902000000 dread Additional change in cset:642d0c4837fa 119 1253791147000000 dread "Done in cset:15566874efe7 We decided that admins could read a non-active package." 120 1254406544000000 dread "Found two security holes, both only in READ access (not write). Tickets raised for holes: ticket:132 - REST listing packages & groups ticket:133 - search package/group (WUI & REST) Full details of files checked: $ find . -name ""*.py"" | xargs grep ""import ckan.model"" ./ckan/presentation.py:import ckan.model as model REMOVED - unused ./ckan/tests/functional/test_group_edit_authz.py:import ckan.model as model - not runtime ./ckan/tests/functional/test_rest.py:import ckan.model as model - not runtime ./ckan/tests/functional/test_package.py:import ckan.model as model - not runtime ./ckan/tests/functional/test_user.py:import ckan.model as model - not runtime ./ckan/tests/functional/test_tag.py:import ckan.model as model - not runtime ./ckan/tests/functional/test_group.py:import ckan.model as model - not runtime ./ckan/tests/functional/test_package_edit_authz.py:import ckan.model as model - not runtime ./ckan/tests/functional/test_revision.py:import ckan.model as model - not runtime ./ckan/tests/functional/test_authz.py:import ckan.model as model - not runtime ./ckan/tests/test_search.py:import ckan.model as model - not runtime ./ckan/tests/models/test_package.py:import ckan.model as model - not runtime ./ckan/tests/models/test_user.py:import ckan.model as model - not runtime ./ckan/tests/models/test_group.py:import ckan.model as model - not runtime ./ckan/tests/models/test_extras.py:import ckan.model as model - not runtime ./ckan/tests/models/test_misc.py:import ckan.model as model - not runtime ./ckan/tests/models/test_authz.py:import ckan.model as model - not runtime ./ckan/tests/forms/test_package.py:import ckan.model as model - not runtime ./ckan/tests/forms/test_group.py:import ckan.model as model - not runtime ./ckan/tests/forms/test_authz.py:import ckan.model as model - not runtime ./ckan/tests/test_converter.py:import ckan.model as model - not runtime ./ckan/tests/getdata/test_data4nr.py:import ckan.model as model - not runtime ./ckan/tests/__init__.py:import ckan.model as model - not runtime ./ckan/tests/test_authz.py:import ckan.model as model - not runtime ./ckan/tests/test_purge_revision.py:import ckan.model as model - not runtime ./ckan/forms/common.py:import ckan.model as model - just validator ./ckan/forms/package.py:import ckan.model as model - validator & package edits ./ckan/forms/group.py:import ckan.model as model - validate & package, wui & rest edits ./ckan/forms/authz.py:import ckan.model as model - package & group wui/rest ./ckan/controllers/rest.py:import ckan.model as model - HOLE ticket:132 - REST listing packages & groups ./ckan/controllers/base.py:import ckan.model as model - paginate filters out deleted ones ./ckan/migration/versions/005_add_authorization_tables.py:import ckan.model as model - not runtime ./ckan/getdata/data4nr.py:import ckan.model as model - not runtime ./ckan/lib/search.py:import ckan.model as model - HOLE ticket:133 - search package/group (WUI & REST) ./ckan/lib/cli.py: import ckan.model as model - not runtime ./ckan/lib/cli.py: import ckan.model as model - not runtime ./ckan/lib/cli.py: import ckan.model as model - not runtime ./ckan/lib/cli.py: import ckan.model as model - not runtime ./ckan/lib/cli.py: import ckan.model as model - not runtime ./ckan/lib/cli.py: import ckan.model as model - not runtime ./ckan/lib/cli.py: import ckan.model as model - not runtime ./ckan/lib/converter.py:import ckan.model - dumper, not runtime ./ckan/lib/base.py:import ckan.model as model - just a remove ./ckan/authz.py:import ckan.model as model - only gets roles ./bin/ckan_spam.py:import ckan.models as model - not runtime ./bin/ckan-correct.py:import ckan.models - not runtime ./test_migrate.py:import ckan.model as model - not runtime " 121 1253716782000000 dread Done in cset:f5a762dae878 122 1253716695000000 dread Done in cset:f5a762dae878 122 1253716757000000 dread "Previous comment was an ERROR! This was actually done in cset:cf62473b8495" 123 1254321447000000 dread This has been done, starting with cset:7d8bada15d88. 124 1254735558000000 dread Done in cset:687f7be58dcf 125 1257535181000000 rgrp I think this is done :) -- dread do you want to close this with linked cset? 125 1258377621000000 dread Duplicate of ticket:152 126 1254740244000000 dread Done in cset:469b0ed6d682 127 1253868048000000 dread Done in cset:6e4119b4351c 128 1260294960000000 dread "Done in 57c5b5ed4737 Cost: 30 mins" 128 1275694573000000 Floallgloli "Futures Trading Key your Futures education provider Learn how to trade futures with live trading education Futures education and trading free trial live on line room Futures" 129 1260179292000000 dread Not needed at the moment (minor security issue since only for read actions). 129 1267719162000000 dread Implementing this would make it less likely that new code would be written to be insecure. But it would also make a lot of things harder. Decided not to do it. 130 1254735501000000 dread Done in cset:77ff9d3259bd 131 1254308115000000 dread Done in cset:399e2bf1e14b. 132 1273254514000000 dread This was fixed before and now works. 133 1273253977000000 dread "WUI and REST interfaces recently updated. You can't read, list or search for packages or groups not-authorised for. The only remaining view of a non-authorised group is that the group is named when viewing a package using all_fields option in REST interface. But no details of other packages in the group are given." 134 1255430998000000 dread Authorization added in cset:3484c10753e7 135 1255180858000000 dread " * authz/is_sysadmin method now looks up sysadmins in user-role table * SystemRole and System added * migration script for new SystemRole and System * CLI for adding sysadmins Done in cset:94028478608a" 136 1255169466000000 dread Done in cset:fb571c27e04b 138 1258466054000000 dread Done in cset:bfbd0bb1b91d 138 1258466074000000 dread Overall cost: 2.25h 139 1255188974000000 dread "I didn't manage to create duplicate tags - must be the old code. I fixed this particular package in ckan." 140 1257535009000000 rgrp Not sure how useful this is atm 141 1255007583000000 dread "Decision made to put it in a section alongside REST docs at api/index. Search API docs already done in cset:5562b3e53977. Refactored in cset:a096132a6c6b" 142 1256417440000000 rgrp "Might be interested here in auto-extracting a nickname from openid (or getting service to give it to us). This question is of relevance on this point: http://stackoverflow.com/questions/572939/extracting-a-username-from-an-openid-identity (though basic answer is ""you can't"")." 142 1276121257000000 paperless@timmcnamara.co.nz "See also: http://stackoverflow.com/questions/1355292/friendly-name-from-google-using-openid Looks like Google has made this difficult intentionally." 142 1289219069000000 pudo Fixed in cset:a3f713368bba pending release of repoze.who.openid==0.5.3 143 1311181336000000 thejimmyg Baze is looking into this and also into most followed packages. 144 1257533957000000 rgrp But we don't record views ... 144 1264439281000000 dread Duplicate of ticket:215 145 1255434248000000 dread Done in cset:d664c9caeb69. 146 1281002247000000 pudo After test hasn't reproduced it, let's wait for someone to notice this in production. We can analyze weberror then. 146 1291829862000000 thejimmyg I've just tested this too and it works for me. Let's close this ticket. 147 1255515162000000 dread Same as ticket:148 148 1255515222000000 dread Done in cset:4d6bfca98d97 149 1257414916000000 dread Done in cset:4d6bfca98d97 150 1256751974000000 dread Done in cset:4dcb28d339a3 151 1257414545000000 dread Duplicate of ticket:175 152 1256056193000000 dread "Done in: cset:f57dbb45418e - main work cset:78930cfd01b7 - not requiring values now." 153 1255621895000000 dread "Comment from rgrp: Yes, and want this for tags too -- this involves working out how to order joins in sqlalchemy (shouldn't be too hard)." 153 1258381364000000 dread Cost: 3h with help from rgrp 153 1258971895000000 dread Duplicate of ticket:195 154 1255621856000000 dread Comment from rgrp: I don't think we have to prescribe one or the other. 154 1257535066000000 rgrp Don't think anything obvious to fix at present (and perhaps plan a larger ticket on form customization). 155 1255621836000000 dread "Comments from rgrp: Very reasonable - and should not be too hard to do (bit of javascript and support at the backend ...)" 155 1271760041000000 rgrp Duplicate of ticket:295 156 1256145730000000 dread Done in cset:8a29d5351650 156 1258981721000000 dread Looks like it was taking out a lot of stuff. 156 1261399351000000 dread "Sorted display of by converting them to markdown format [links] (links) in cset:9c3e64104cbf" 156 1266348238000000 dread Example problem: http://ckan.net/package/patent-nber 156 1267373651000000 dread Another example: http://www.ckan.net/package/dbtune 156 1271962871000000 johnbywater Fixed by encoding elements before and decoding elements after passing text through Markdown engine. 157 1256062680000000 dread "As far as I can tell this never worked in the past. Added in cset:f4ba0dcfb1a3 Cost: .65h" 158 1255949818000000 dread Done in cset:2aaaedff9d60 159 1256060264000000 dread "Fixed in cset:e7ea5b97365e Note: no spaces allowed between search operator and the word associated with the operator. So with ""tags: postcode"" it was searching for a blank tag. I've made the change so that would ignore 'tags' in this case, producing reasonable search results for postcodes in all fields. postcodes. Cost: 1.5h" 160 1261399380000000 dread Done in 1h in cset:9c3e64104cbf. Not allowing space - non-standard. 161 1256114485000000 dread Yes, deleted tags are still listed and associated with a package. 161 1257762932000000 dread "Rufus says: Two issues: 1. search -- this has be done in internals ... 2. is: tag.packages attribute -- this is solved by using StatefulList " 161 1258381450000000 dread Cost: 4h with help from rgrp 161 1258573607000000 rgrp Resolved in cset:6d466d8b702a 162 1256142462000000 dread Done in cset:429f2c89d4de 163 1256140829000000 dread "Basic facility to do it with CLI command done in cset:a4217353a7be Still need to make it regular." 163 1257415333000000 dread "Facility to do this, pulling packages out of ckan over REST, is here: ckan/lib/ckan-to-talis.py (cset:752f346b47c2) Currently not implementing a cron job to do this regularly." 164 1256056395000000 dread Done in cset:ab50a8ee0ba3 164 1264875709000000 rgrp Reopen as now part of semantic.ckan.net 164 1271251422000000 rgrp This is done in semantic.ckan.net. Docs at http://wiki.okfn.org/ckan/doc/rdf/ 165 1311181391000000 thejimmyg This is more than 6 months old so closing. 166 1256222093000000 dread "Done in cset:bb7566d82b9f. Couldn't get logging going." 167 1256210109000000 dread Done in cset:7d129fc35e5c 168 1257414795000000 dread Duplicate of ticket:145 169 1266928708000000 dread Covered in ticket:253 170 1256572201000000 dread Done in cset:517d25bc56f3 171 1297210925000000 rgrp A lot of work on config was done in 0.7 and these refactorings and improvements either fix or render this ticket invalid so marking as fixed. 172 1256495095000000 rgrp Largely done in cset:f3782c1071cc 172 1257532331000000 rgrp Done in cset mentioned previously. See: http://knowledgeforge.net/ckan/ckan/doc/ 173 1258543707000000 dread "Done in cset:05b1a12c5e71 Cost: 8.25h but only basic fields. e.g. extras not covered, license is not pretty." 174 1256650986000000 dread Fixed in cset:d13bfdb99282 175 1257780907000000 dread Migration requires quoting of table names 'user' and 'group' - need SA migrate v0.5.6. 175 1257847996000000 dread "Done in cset:78f04724b875 Cost: 6.5h" 176 1257533003000000 rgrp "My suggestion here is that this ticket get taken to okfn-discuss/okfn-help. This isn't a trivial issue. Do we go Debian (must exist in debian and to a name) or PyPI (very flexible, to name + version) route?" 176 1266928721000000 dread Covered in ticket:253 177 1273050236000000 rgrp Not really relevant to this trac (as outside of CKAN scope) and is duplicate of http://knowledgeforge.net/okfn/tasks/ticket/252 (which is now closed as well!). 178 1257439349000000 dread First batch of changes for this in cset:b5ef888e44b6. Works in basic way. 178 1257523544000000 dread "Second batch of changes in cset:7d524a92d602. Status: You can import new packages with all package properties using Excel or CSV format. Outstanding: * Validation is displayed poorly. * Overwriting existing packages is warned about, but doesn't work. * Can't add packages to group yet. * Functional tests are only working for the basic case. * Need to centralise package rendering for package controller. Time spent: 13hrs" 178 1257523596000000 dread Expect 1d to finish outstanding features. 178 1257523668000000 dread Time spent (correction): 20h 179 1257762996000000 dread "Changeset was to solve 'deprecated' messages. See line 46 ff. of changeset." 179 1258392697000000 dread Done in cset:76e6cd49e503 179 1258392723000000 dread Cost: 1.5 h 180 1263373715000000 rgrp Looked through jquery plugins and best I could find was this one: http://addywaddy.github.com/jquery.tagcloud.js/ (http://plugins.jquery.com/project/jquery_tagcloud) 180 1264439377000000 dread Duplicate of ticket:223 181 1296339510000000 rgrp Uncertain we want to do this and rather overtaken by other events, see e.g. http://ckan.org/wiki/UIRedesignHome 182 1270567116000000 rgrp Done as part of UI overhaul for v0.11 around xmas. Current openness icons seem good enough. 183 1290604779000000 dread You can see the top rated packages in the stats box. Actually not very useful at the moment - not much rating is going on. 184 1266837414000000 dread This is done in cset:d2026bcf43f5 cset:ac7cf572cbf9 cset:51c6de2dc390 cset:58c660932fcf cset:dd9761591ffb 185 1291830039000000 thejimmyg "This is probably no longer necessary. I've implemented JavaScript to hide the help text and allow it to be revealed by clicking ""More >"". This makes the form look simpler without needing to hide actual fields." 186 1296341182000000 rgrp "Now duplicate: * #877 - File upload in WUI: 2d * #878 - Integrate file upload with workflow around package resources: 3d " 187 1258970509000000 dread "Done in cset:af3cbf266750 and e70af291455e. Issues addressed: * tags ARE indexed * if tags are converted into lexemes but we also search on exact match. * name is split on dash when indexed by postgres. * weight name and title higher than other fields. Remaining issues: * natural language search doesn't do partial words, so search for 'gov' doesn't bring up 'government'. * previous search system not yet usable with a config file switch (for if we install on a db aside from postgres) " 187 1258970766000000 dread Cost: 16h 188 1264436473000000 dread Duplicate of ticket:219 189 1259925111000000 dread "All done in 5 days in: cset:5c7f0ebd728c cset:20374a1ee763 cset:02ce73aef595 cset:36eda4112f72 cset:dd285dd1b821 Also made relevant changes in importer, create-search-test-data and data4nr. It wasn't clear that download_url could be made a SA proxy, so instead put download_url proxy in the REST interface to maintain compatibility with existing clients." 190 1280686074000000 pudo "This will be solved using an external plugin and the disqus service. A current test version of the external code is at: http://bitbucket.org/pudo/ckandisqus " 190 1280820852000000 pudo fixed as of cset:1389 191 1305732285000000 dread "Will's suggestion is to have ""modified_since"" param, just as we have for Revision Search. Maybe we don't need a range. 'Order by modification' should be on by default for queries with 'modified_since' param." 191 1305732414000000 dread Do this after refactor #1129 191 1305814900000000 dread "Maybe allow searching by package creation date too? Suggestion from kindly: when indexing a package, have the search backend also store the metadata_modified value, to make it easy to search on it. " 191 1322762567000000 dread "So you can do this sort of search: {{{ curl http://thedatahub.org/api/action/package_search -d '{""q"": ""groups:lodcloud"", ""sort"": ""metadata_modified asc""}' }}} but it doesn't work because solr doesn't store the metadata_modified field yet." 191 1323171514000000 thejimmyg John will just check that the API version 3 does support modified after the new solr schema handled by Adria is in place. 191 1323706210000000 johnglover working in current master and 1.5.1 191 1323708844000000 dread A test for this would be great, and maybe add as an example in the search docs too? 191 1323970011000000 dread "Added a fix for the problems caused by the #1546 fix. SOLR indexing was excepting for new packages (tests such as ckan/tests/functional/api/model/test_package.py:TestPackagesUnversioned.test_entity_update_indexerror failed). The problem was (somehow) related to last_modified function using a new connection causing problems for the modified_metadata in finding the revision & package table entries created during the commit. Changeset: [release-v1.5.1c 2c595ae] and cherry picked to [origin/defect-191-modification-date f98a4b2]" 191 1323973116000000 dread "I put the sort syntax example in the apiv3.rst table and moved the full curl example to http://wiki.ckan.org/Searching_CKAN I added tests for metadata_modified and prepared this branch for closure. Changeset: [origin/defect-191-modification-date bebf7a5] John, please review and close/merge if you think it is all ok." 191 1324030875000000 johnglover "Thanks for this David. This one in particular was causing me some grief: https://github.com/okfn/ckan/commit/f98a4b2a5f5013fa4aed475bd8b3237bb7847fcc Good spot. All looks good to me." 191 1330020677000000 dread Went into CKAN 1.6 191 1330020983000000 dread "Replying to [comment:15 dread]: > Went into CKAN 1.6 Sorry, I meant 1.5.1 " 192 1261487668000000 dread Similar stuff is being done in ticket:204. Consider transferring this field into default package schema? 192 1291733895000000 dread The gov form has had this for months. 194 1260456304000000 dread "Judging by the logs, a great number of bots are hitting the star ratings. In cset:8985cd53f8e5 I've added a robots.txt as well as cli functions to reset star ratings." 195 1258585500000000 rgrp "Easy for groups because simple secondary=m2m table setup in sqlalchemy (done in cset:9e17314583ce). For packages+tags was hard because PackageTag is versioned and therefore only get the package_tags relation to work with in mappers. Tried various methods but gave up (see cset:ea4a790e9db9 for details). My conclusion was that if you want the fields ordered let's just do it by hand with a dedicated attribute or method and added packages_ordered and tags_ordered attributes to Tag and Package respectively (cset:3d3c1025a311)." 196 1264876030000000 rgrp This will be fixed via using meta http-equiv (see ticket:90). 196 1265389771000000 dread "In cset:7eadcdc94b3a package URI changed to: http://ckan.net/package/32000-naples-florida-businesses-kml This resolves to html or rdf+xml - see ticket:90 " 197 1258989700000000 dread "Done in cset:0db8ee29a8fa Cost: 1.5h" 198 1264688525000000 dread Bulk of work for this is in cset:d213e942245b 198 1266837606000000 dread This was finished off by rgrp in cset:ca995c2596c6 199 1267649255000000 rgrp These tests are no longer failing afaict. To run tests you can just use nosetests -- documented in cset:ebdfc89ccd86/datapkg 200 1311176118000000 thejimmyg I know this is for datapkg but it is over 6 months old so closing in line with our current ticketing policy. 201 1264439427000000 dread Done by rgrp 202 1264436548000000 dread Started in cset:8b6263a422d9 202 1265399429000000 dread "Step 1 previously done by Benoit Step 2 done now in cset:cf13a82ab243 Step 3 remains" 202 1265892350000000 rgrp Closing. Bulk of work has been done and we have deployments at fr.ckan.net and de.ckan.net. Remaining work on forms needs some reflection and has been turned into new ticket:248. 203 1260456150000000 dread "The list of groups was already done. The group view is now done in cset:a9506c0c9c68. Cost: .5h " 204 1260191598000000 dread "First, do simple fields, department and a date field. See how it goes. " 204 1262605982000000 dread "Basically done in changesets particularly: cset:06b1232321d7 cset:adc4ad5c5b3f cset:2b4c0f723307 cset:39dc4b106dd2" 204 1285081954000000 anonymous [http://www.grattage.pro Casinos pro] 204 1285082169000000 anonymous [http://www.casinotop10.fr Top10 des casinos sur internet] 205 1297068450000000 rgrp "wontfix as: 1. Easier to just to an import from google docs for which we have several importer examples in ckanclient 2. (More important) importing is something that is quite bespoke (changes for each cirumstance). As a result pre-written importers tend to be very fragile and it is not a good use of time to try and some customizable WUI one when you can just write a custom importer in code (code is much more flexible than whatever we can write!) " 206 1260881123000000 dread Done in wiki:UiReviewNotes and ticket:214 - ticket:227 207 1270569952000000 dread Done by nick, put into forms branch and will be merged in soon. 207 1271250740000000 dread This was done in forms branch - merge: cset:c22854a52319 208 1261413421000000 dread Email addresses now URL-encoded in the package/read/id HTML. Done in cset:af07f3bd9a04 209 1260288728000000 rgrp Done in cset:4983f1da3a27/isitopen (a while ago). 210 1266509788000000 johnbywater Isitopen tickets are actually on the OKFN tracker. 211 1266509848000000 johnbywater Isitopen tickets are actually on the OKFN tracker. 212 1260703969000000 rgrp Done in cset:3b68a54bd2b0 213 1262689477000000 dread There is a fab file for microfacts already. 213 1264436021000000 dread Done from cset:f342b4928466 214 1260889758000000 dread 3rd item (Routing redirects) done in cset:1b2edc315d1b 214 1263056226000000 nickstenning Routing now redirects /packages to /package, in addition to /packages/* to /package/*. Introduced in cset:9eda6ff974ae. 214 1263057591000000 nickstenning Hover names haven't been addressed yet. Otherwise mostly done. Closing. 215 1266513523000000 johnbywater All done, see e.g. tag cloud in #89. 216 1260891028000000 dread "First two items done in cset:f49a987492c6 Third item: If this book was correctly input the title would be 'Anna Karenina' and would match. Matching substrings isn't that useful and is difficult to achieve. Google doesn't do it, only doing similar matches. We do some of this - donkey and donkies are indexed and query the same. Fourth item: still TODO" 216 1260891603000000 dread On item 3: It does match 'geospatial' if you search for 'geo' or 'spatial' - I think it splits up words like these sensibly. Annakarenina doesn't split only because they are proper nouns. 216 1273050561000000 rgrp "Closing. Everything done except for item 4: ""autocomplete package names & tags"" which I'm not sure needs doing and certainly won't be done any time soon. Have created a new ticket for this: ticket:308" 217 1260891718000000 dread Last item done - copywriting - in cset:74ffa691c7b4 217 1263055918000000 nickstenning Mostly done. 218 1263055964000000 nickstenning Mostly done. Alphanumeric filtering à la KForge is probably still worth doing at some point. 219 1260892527000000 dread Item 2 done in cset:399c37b2ed35. 219 1294914901000000 dread """three-column package listing"" has not been prioritised for over a year now - marking won't fix." 220 1264436377000000 dread Done mainly in cset:a61604d082ed 221 1271756757000000 dread " * Ability to add the package to groups on this page. MOVED to ticket:295 * Fields could be grouped. DONE * Inconsistent capitalisation (Url -> URL). DONE * Tag editor YUI script appears to break occasionally. DONE * Perhaps move to two column forms to save vertical space? IGNORE * Notes field to monospace? IGNORE * Extras fields need some jQuery love -- shouldn't have a fixed number available. MOVED into ticket:294 * Typography is a bit of a mess. DONE " 222 1282909280000000 dread Done already. 223 1264439367000000 dread "From rgrp: Looked through jquery plugins and best I could find was this one: http://addywaddy.github.com/jquery.tagcloud.js/ ( http://plugins.jquery.com/project/jquery_tagcloud) " 223 1273247748000000 dread Tag cloud is already on the front page. The consolidation of browse and search of tags has already been done. 224 1267100542000000 rgrp As far as I can tell all of these have been done by early Jan. 225 1266837873000000 dread Note, in the REST API we are resource oriented. 225 1311178276000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. 226 1273247829000000 dread Lots of excellent suggestions here. 226 1279130390000000 dread "Remaining items done in cset:7c63c10f5681. Took: 1h" 227 1273254223000000 dread The pagination issue was corrected a while ago. The inconsistency of page titles needs looking at still. 227 1279192621000000 dread "Page titles sorted in cset:2324447eb60f (related to ticket:350) Cost: 1.5h" 228 1290596875000000 dread Duplicate of #262 229 1262799850000000 dread "Done mainly in cset:dcde72b325d8 plus migrate script in cset:3905a1c08d01 and minor corrections in cset:fe0dba15703b. Still to resolve questions about what sort of hash to store." 230 1263055318000000 nickstenning Package read view is now more-or-less self-contained. Refactoring of the template to use a c.pkg object is all that remains to be done and it then can be factored out and used for the preview. 230 1264435915000000 dread Done in cset:ad3bcfb10002 to cset:fa3bc8c93a91 231 1288516929000000 pudo fixed in cset:00bc33fbc3ff 232 1297246297000000 thejimmyg This has been here for over a year and is not a technical ticket. Suggest we close it? 233 1273078975000000 rgrp Introduction option 1. 233 1273080345000000 rgrp Done in cset:75756e565b6a 234 1294410993000000 thejimmyg "Cygri opened #815 which I've closed as a duplicate. He requests: ""The search field (on the homepage and in the top right corner of each page) should have autocomplete for package name. If a package name is selected, it should not do a search but go straight to the package page."" @memespring - Is this something you are looking at?" 234 1294657094000000 memespring @ rgrp - No, I think I suggested something simular for adding new tags to a package though. 234 1338206486000000 ross 18 months, no activity. 235 1302508788000000 pudo "The first three sub-items of this ticket are done in datautil and dcat-tools: Basic GDocs-based normalizer: * https://bitbucket.org/okfn/datautil/src/8bba83b4fa45/datautil/normalization/table_based.py Example of use: * https://bitbucket.org/okfn/dcat-tools/src/0ec5012bf12a/dcat/core/normalize.py#cl-32 Spreadsheet (as referenced in datautil source, should be a kwarg): * https://spreadsheets.google.com/ccc?key=0AplklDf0nYxWdE8tVlRrN1F3bG9PdDBFUDNZcENDNEE&hl=en#gid=0 Experience so far has been that Google rate limits the current implementation so we should perform all ops in one or two big calls rather than ""piece by piece"". " 235 1340627057000000 icmurray Moved to ckan-future and unassign so that this ticket will be picked up in triage. 235 1340627624000000 icmurray "Replying to [comment:3 icmurray]: > Moved to ckan-future and unassign so that this ticket will be picked up in triage. Sorry, this comment bears no resemblemnce to what I actually did! Assigned to tobes for 1.9." 238 1266509806000000 johnbywater Isitopen tickets are actually on the OKFN tracker. 239 1264875924000000 dread This was done in cset:d213e942245b 240 1272383770000000 johnbywater Fixed in revision 5567025e6d5e. 241 1270569769000000 dread This was fixed by John a couple of weeks ago with his new license stuff. 242 1279286215000000 pudo item 1 is done with the refactoring 242 1280823876000000 pudo "Fixed in cset:1393 In the long run, we'll want to remove those SQL-based search from the search code. The upside to using the search backend right now is that we pipe things through the query parser, which means multiple terms are looked up properly. Not a big gain." 243 1266837796000000 dread Looks like this has been fixed already. 244 1271248968000000 dread This has been fixed by Benoit (backed out changes, waiting for upgrades to babel / genshi) 245 1297066620000000 rgrp Closed in favour of https://bitbucket.org/okfn/vdm/issue/1/support-for-composite-primary-keys 246 1297066757000000 rgrp Moved to https://bitbucket.org/okfn/vdm/issue/2/support-for-primary-key-not-named-id 247 1271248813000000 dread Done in cset:36c2c01a480c 248 1270745304000000 dread "Done in cset:193280d51050. Cost: 0.8 days" 249 1311182450000000 thejimmyg This is over 6 months old so closing in line with our ticketing policy. Also, I shouldn't think we really want such a complicated search anyway. 250 1340631430000000 icmurray "Package entries already include an enclosure for the json representation of the package: {{{ }}} so it should be easy enough to include something like the rdf link that already exists on a dataset-view page: {{{ }}}" 251 1270564365000000 rgrp " 1. /api/rest/revision/ returns nothing 2. /api/rest/revision/{id} does not return revision.packages and should do." 252 1294410341000000 thejimmyg David, do you know where this requirement has come from? Is is still relevant? 252 1296477560000000 dread I've no idea what this is 253 1270632458000000 dread Closing in favour of the 4 tickets it is split into. 253 1295610188000000 rgrp Re-opening as this is a useful meta-ticket and should only be closed when all subtickets done. 253 1319709714000000 dread Closed by mistake 254 1266954625000000 dread Done with migration script in cset:663ff0c78e6e 255 1268220526000000 dread Done simply in cset:b98ce3951f0a 256 1290463475000000 dread This is a test 257 1269277815000000 dread "Done in changesets leading up to cset:24734bc10f80 Documented in API." 258 1267208356000000 dread Was already there. Added test in cset:54d61cb0bd9a 259 1267207964000000 dread rgrp fixed this in cset:45e16431add9 260 1267122137000000 dread "Done in cset:ffae8f89e577. Also made Group Read WUI format more like that for Package Read. Cost: 1h" 261 1267207776000000 dread "Fixed in cset:c024937fc6cb. license_id removed in API." 262 1267205195000000 dread "It could be also be a package relationship. However it's recorded, however it's recorded, it would be good to record the merge in the db. If the system 'knew' there was a merge then it could: * leave a redirect page * the merge would sync well" 262 1296468392000000 rgrp At the moment we get by with adding meta.duplicate tag and putting this in the notes (and possibly putting duplicate in deleted state). If this deemed as insufficient can reopen. 263 1271690219000000 johnbywater All seems to work. Reported Wordpress trouble may result from user having Wordpress account, but no blog (ie they have 'myname' login, but haven't got a 'myname.wordpress.com' blog. Having the blog makes it work (otherwise you get told that you aren't the owner of the identity). 263 1273072985000000 rgrp "Issue where if you click on google or yahoo seems to ""remember"" login so next time you click on google or yahoo item it takes you to what you clicked on previously (e.g. click on google then go back and click on yahoo and you are taken to google login). Also think this plugin may be nicer: http://code.google.com/p/openid-selector/ Also: * typo in page (indentity) * Not sure ""sign in or create new account"" is clear without some indication of how openid works e.g. better to say something like: ""Login to CKAN using Open ID""" 263 1273083179000000 johnbywater " 1. Have suspended cookie setting (which was remembering the provider). 2. We are already using openid-selector. 3. Fixed type in page ('indentity' => 'identity') 4. Have changed 'sign in or create new account' to 'Login to CKAN using Open ID'" 263 1273135032000000 rgrp "Sorry, I meant this plugin: http://jvance.com/pages/JQueryOpenIDPlugin.xhtml I also do not understand how I can login using my own openid e.g. one not provided by any of those providers (at least not directly ...): by default no box is showing and to make one appear I have to click on a specific provider (with unknown results if i then enter something). Basically, I think this plugin is poor because it does not offer me a login box by default (as the other plugin seems to ...). What we really want is a simple login box with autocomplete based on provider like the civirm example!" 263 1273136715000000 dread "Replying to [comment:4 rgrp]: > I also do not understand how I can login using my own openid e.g. one not provided by any of those providers (at least not directly ...): by default no box is showing and to make one appear I have to click on a specific provider (with unknown results if i then enter something). I found it pretty obvious to login using openid - clicking on the 'openid' box made sense. The input box may serve to confuse the majority who login via Yahoo/Gmail. I don't think it's worth changing AGAIN." 263 1273155234000000 rgrp OK, I think we're agreeing to close this now since reasonable and not worth the switch to anything different atm. Still concerned about usability but if necessary we can open a new ticket. 263 1277717180000000 rgrp I consistently get sent to google login even though I choose openid meaning I am unable to login! 263 1279292730000000 rgrp JB advised permanently remove ckan.net cookies (may be left over from much earlier). Having done this problem went away. 264 1272390013000000 dread "Mainly sorted in ticket:292. Also related changes in cset:ed4c500fcd90" 267 1272960518000000 rgrp Fixed in cset:9e2e66cced90/vdm 268 1285070682000000 dread Duplicate of ticket:652 269 1288036269000000 rgrp I think this has been long obsoleted by other work (e.g. more recent work on gov form). 269 1288037103000000 dread Several of these points haven't been considered in the recent work. 269 1291830780000000 thejimmyg "Just discussed this with Evan... notes field could use a WYSIWYG * No, Evan wants to discourage fancy features, plain text/markdown is fine * auto-complete on tags - DONE * department drop down options list interact with user permission - Evan building the API we need for this now. * licenses -> drop down is fine, let's just OGL as default So just default licence and replacing department with provider and via to be implemented on this ticket. Evan will provide: organisation.one() to look up one organsisation by ID organisation.many() to look up a list of organsisation by ID all at once organisation.match() to match a string and return a organsisation ID organisation.department() to take a organsisation ID and return the organsisation ID of the department it represents. " 269 1291897538000000 dread "Licence is defaulted in CKAN cset:5bfbcd457426 (merged into default) and DGU cset:2d798e8af3d7. ""replacing department with provider"" is covered in ticket: https://trac.dataco.coi.gov.uk/projects/datagov/ticket/742" 270 1290596640000000 dread Nearly everything mentioned here has been achieved with the SpreadsheetData, DataRecords<-SpreadsheetDataRecords, PackageImporter<-SpreadsheetPackageImporter design. New imports can take advantage of this. 271 1272280005000000 johnbywater "Initial spike solution has been written, covering four user stories: 1. Commit CKAN revisions to changeset system (#296) 2. Update CKAN repository from changeset system (#297) 3. Pull changesets from remote CKAN instance (#298) 4. Merge diverging lines of changesets (#299) Emails to ckan-discuss include: * http://lists.okfn.org/pipermail/ckan-discuss/2010-March/000109.html * http://lists.okfn.org/pipermail/ckan-discuss/2010-March/000154.html" 272 1271764003000000 dread Package feed done in ticket:255 272 1290004225000000 cygri A strong +1! Feeds for groups and tags would be extremely useful. 273 1268996987000000 pudo cf http://lists.okfn.org/pipermail/ckan-discuss/2010-February/000042.html 273 1270717895000000 pudo "SOLR Requirements * 4GB Memory * Sun Java * Tomcat * Scala (for Etherpad) * MySQL 5 (for Etherpad) * Cheap bandwidth/low latency to the CKAN servers. " 273 1278578527000000 dread rgrp: We plan to use SOLR. May investigate Xapian. Nothing more to do in this investigation. 274 1278700842000000 dread A significant chunk of the work towards this done in cset:742adebb707c (refactoring search options). 274 1279890237000000 pudo This is included in the Solr indexing engine and will become available as Solr is adopted. 274 1280262229000000 rgrp Need to work on postgres and test there. 274 1281452015000000 dread The docs are now out of date and there doesn't seem to be a test for this. 274 1287398398000000 dread I fixed the docs a couple of weeks ago. Need to ensure there is a test though. 274 1287402155000000 pudo "Replying to [comment:7 dread]: > I fixed the docs a couple of weeks ago. Need to ensure there is a test though. there is as of cset:c2e66cec3610 " 274 1287402800000000 dread "Replying to [comment:8 pudo]: > Replying to [comment:7 dread]: > > I fixed the docs a couple of weeks ago. Need to ensure there is a test though. > > there is as of cset:c2e66cec3610 Error: Invalid Changeset Number" 275 1280737701000000 rgrp "Also, now if you visit ckan.net/admin/Package get 500 error and following in logs: [Mon Aug 02 08:17:35 2010] [error] [client 86.26.8.30] Error - : 'ascii' codec can't encode character u'\\xf4' in position 988: ordinal not in range(128)" 275 1280999840000000 pudo cset:1396 fixes this 275 1280999963000000 pudo "Replying to [comment:3 pudo]: > cset:1396 fixes this Where ""this"" is the field renderer. " 275 1281001911000000 pudo "Replying to [comment:2 rgrp]: > Also, now if you visit ckan.net/admin/Package get 500 error and following in logs: > > [Mon Aug 02 08:17:35 2010] [error] [client 86.26.8.30] Error - : 'ascii' codec can't encode character u'\\xf4' in position 988: ordinal not in range(128) Cannot reproduce this at the moment. Will wait for it to occur again and then create another ticket. " 276 1271250866000000 dread I could not recreate this. I think it is only for particular packages? 276 1272276020000000 dread Estimate: 0.5day 276 1272996237000000 dread Fixed in cset:060e2df72148 277 1294416367000000 thejimmyg "My opinion is that having configuration in a database is a bad idea. We are currently considering moving to a system where CKAN is installable using apt-get. Since we're already moving functionality into CKAN extensions, choosing what you want kind of CKAN you would like would then be as simple as chosing the package to install. Configuring it would just be editing the config file. I don't think this is as relevant as it was 10 months ago. Anyone mind if I change this to wontfix?" 277 1296340723000000 rgrp I think this is useful but there may be complexities giving the non-reloading nature of python apps. Have also converted to an extension 277 1296470458000000 kindly "I think generally this is a bad idea. I think in a few controlled circumstances some configuration is worth changing at runtime, however looking through the development.ini file I do not see hardly anything in there that does not require a restart anyway. It would be good to have some clear examples of things that would be in the extension." 278 1271173752000000 dread Fixed in cset:ad64bd0f6073 - copes with spaces now. 279 1272451384000000 johnbywater Fixed in 5b6029c72f9a. 280 1271173769000000 dread Fixed on default in cset:ad64bd0f6073 281 1270723675000000 dread Done in cset:5e9f8ce150c2 282 1338206417000000 ross This was discussed with Toby who has a ticket for this same feature because of disqus requirements. 283 1270715817000000 rgrp "Can I *strongly* suggest we just use the existing perfectly-good-system for flagging stuff called ""tags"" :) I suggest we have agree in the community a standard set of ""meta"" tags for this kind of stuff. E.g. i'm already using the ""duplicate"" tag for marking duplicates (I also add in notes the link to duplicate package but that's optional). So I suggest we: 1. Create ""reserved"" tag prefix ""meta"" 2. Create following specific tags (suggestions/comments welcome): * meta.duplicate - duplicate of another package. If possible indicate in notes or an extras field (to be decided) what it is duplicate of * meta.spam Editors can then just visit http://ckan.net/tag/read/meta.spam and work through list of packages there. If ""push"" notifications are required as well as ""pull"" then I suggest this be put in an external service (e.g. rss2email) rather than integrated into CKAN core. " 285 1340631923000000 icmurray Unassign to be triaged. 286 1270723629000000 dread Done in cset:76560fa09db8 and cset:752a634a3095 and error handling in cset:aa021336d64d 287 1270801210000000 dread Done in cset:76560fa09db8 through cset:ea397fc03587 288 1271173777000000 dread Fixed in cset:ad64bd0f6073 289 1271249368000000 dread Done in cset:bf98b63331cf 291 1273254895000000 dread "This seems spurious. The options.q is unicode finds foreign characters fine. The hack has since been taken out." 292 1272286005000000 dread "Achieved in cset:56b02fda195e (rgrp), cset:95498407d15e and cset:f5af59a3365c. Remaining broken test put in ticket:300." 293 1271885457000000 johnbywater Can't reproduce this exception. Have added tests covering adding, removing and updating resources, and it all seems to work. 293 1271940083000000 johnbywater "With the metastable revision of CKAN, the package resource data structure in ckanclient scripts must have all four keys set in the Resource-Dict. Setting 'hash' in the resource data cures this issues with ""metastable"". The ""bad"" code was: resources.append((res['url'], res['format'], res['description'], res['hash'])) KeyError: 'hash' This code was adjusted in revision 40c4fe04038d, to default unspecified resource attributes to the empty string. It was changed further in subsequent revisions, to use parameters of the Package.add_resource method to default unspecified values. However, the API doc doesn't mention the resource hash, although it is required, but I just fixed that in the source (in revision 0f20bfb45d13). http://knowledgeforge.net/ckan/doc/ckan/api.html " 294 1291830960000000 thejimmyg Duplicate of #812 295 1272384758000000 dread "Done in cset:18edc4d95f2f. Took: 3h" 296 1294407032000000 thejimmyg Merging into #896. 297 1294407051000000 thejimmyg Merging into #896. 298 1294407080000000 thejimmyg Merging into #896. 299 1294407099000000 thejimmyg Merging into #896. 300 1272384474000000 dread Rufus fixed this in cset:e6e3 301 1274832112000000 tim Duplicate of #190, although that ticket has a different implementation. It prefers comments as blog comments. 301 1281342885000000 rgrp Should not have been closed. ticket:190 is about comments this is about a wiki-like discussion page which is very different. 301 1340632055000000 icmurray Unassign in order for it to be triaged. 302 1272453821000000 johnbywater Fixed in 61c8b3107f0e. 303 1272454025000000 johnbywater We could also fix up the temporal model. 303 1272912573000000 dread Package history page now shows revisions for tag, extra and resources. Needs tidying up and adding to REST. Done in cset:dc99df3ab4bd 303 1272988619000000 dread Done in cset:dc99df3ab4bd cset:beb72a0aa810 cset:96bab1eb53f5 and vdm cset:bb9f97b1c4b0 304 1272447296000000 johnbywater Fixed in c4bf92996b8a. 305 1272994804000000 rgrp Closed in cset:8136e7369c0c 306 1318181194000000 rgrp Duplicate of https://github.com/okfn/datapkg/issues/4 307 1288027815000000 rgrp Done in cset:90e318c3c7dc/datapkg cset:0036b5c505eb/datapkg etc 308 1275302577000000 rgrp Duplicate of ticket:234. 309 1280743432000000 pudo fixed in cset:1382 310 1279300525000000 dread "Fixed in cset:a0acf179355c Cost: 2h" 311 1274282065000000 rgrp "Resolved (sort of) in cset:489007a10bb9. This was a migration issue. Tracked this down to fact that on ckan.net we have: ""package_resource_revision_pkey"" PRIMARY KEY, btree (id) When it should be: ""package_resource_revision_pkey"" PRIMARY KEY, btree (id, revision_id) Looking in browser:ckan/migration/versions/012_add_resources.py find: {{{ Column('revision_id', UnicodeText, ForeignKey('revision.id')), #NB revision_id should have been primary_key too (joint with id) }}} How come this was not corrected here or at least noted for upgrade of ckan.net??? I have now fixed this so that others doing migration (at least with v1.0) will end up with correct code. I have also fixed issue on ckan.net by manual sql!" 312 1311176173000000 thejimmyg This ticket is more than 6 months old so closing in line with our current ticketing policy. 313 1275404524000000 johnbywater Fixed in changeset 06c949266644. 315 1275846764000000 dread Fixed in cset:61548ced8b7d - Quote marks correctly read in for data4nr data, which makes this problem record ok (which opened in openoffice fine incidentally). Fields in package are now dumped in correct order to make it clearer. Not changed resource serialisation - if you want tidy json, then use the json dump. No real call for half-way house dump. 316 1274366801000000 dread "This exception occurs for ckan.net with just this one character: http://ckan.net/package/search?q=%C2 (you can wget it) But I can't recreate it on my machine. Maybe it's a version issue. The client that is making all these crazy calls is googlebot." 316 1291831177000000 thejimmyg "I've just tested this on ckan.net and it gives a sensible message: There was an error while searching. Please try another search term." 317 1279005278000000 pudo this has been in for a while now but still needs to be extended to include the indexing of entities (ckan.model.search_index) 317 1279286041000000 pudo should be done after refactoring the search functions. 318 1274377385000000 wwaites "Some more datapoints from Leigh Dodds of Talis: I'm still having no joy with this I'm afraid. I'm test parsing the data locally using the TDB command-line tools, specifically tdbcheck which will parse the data and generate warnings/exceptions. This uses the same parsing code, data and URI validation code as we're using on the Platform. Currently its giving me warnings for invalid lexical values for dates, e.g: Lexical not valid for datatype: ""2008""^^http://www.w3.org/2001/XMLSchema#date While these aren't a major issue, looking at some of the data suggests that there are more underlying data problems that need checking and fixing up, e.g: Lexical not valid for datatype: ""n/a""^^http://www.w3.org/2001/XMLSchema#date Lexical not valid for datatype: ""27/04/2006 13:56""^^http://www.w3.org/2001/XMLSchema#date Lexical not valid for datatype: ""Real time calculation""^^http://www.w3.org/2001/XMLSchema#date Lexical not valid for datatype: ""varies by country""^^http://www.w3.org/2001/XMLSchema#date And there are still some invalid URIs, e.g: Code: 30/ILLEGAL_PERCENT_ENCODING in QUERY: The host component a percent occurred without two following hexadecimal digits. Can I suggest you try running the converted data through tdbcheck to iron out any problems? Then I can push it into the Platform. " 318 1275320677000000 dread "We can't change any of the metadata without permission from the various departments who supplied it. I think ""Don't shoot the messenger"" is apt here. Adding this to the form validation isn't going to change any of the existing data. I think this is better off in the data quality scoring." 318 1276271343000000 wwaites "url validation reputed to be here: http://www.livinglogic.de/Python/url/Howto.html" 318 1276438793000000 wwaites "Some good news, ll.url seems to take bad urls and make them into good urls. viz: {{{ In [1]: from ll import url In [2]: print url.URL(""https://mqi.ic.nhs.uk/IndicatorDataView.aspx?query=NRLS%3&ref=3.02.16"") ------> print(url.URL(""https://mqi.ic.nhs.uk/IndicatorDataView.aspx?query=NRLS%3&ref=3.02.16"")) /Users/ww/Work/OKF/ckanrdf/lib/python2.6/site-packages/ll/url.py:2358: UserWarning: truncated escape at position 4 value = _unescape(namevalue[1].replace(""+"", "" "")) https://mqi.ic.nhs.uk/IndicatorDataView.aspx?query=NRLS%253&ref=3%2E02%2E16 }}}" 318 1276438832000000 wwaites "Also fyi, getting ll.url is done like so {{ pip install ll-xist }}}" 318 1276438907000000 wwaites "I've updated ckanrdf to strip out datatypes and use this ll.url on external references so that should be sufficient to hold off talis. Still need to work particularly on validating dates though..." 318 1280737620000000 rgrp Important but low priority according to CO so bumping into next milestone (v1.2). NB: did not seem able to update milestone in trac interface! (Perhaps due to agilo stuff?) 318 1283179768000000 wwaites CO may not realise the implications when they said it was low priority. The implication of this lack of validation is that it is impossible to generate valid URIs in the RDF which means it cannot be imported by Talis. So until there is a solution to this, no RDF catalog. 318 1296340768000000 rgrp Still not sure what the priority is so moving to awaiting triage. 318 1296467308000000 pudo This will be implicit in #852, thus not building something specific for it now. 318 1296482049000000 wwaites We still require form validation to check URIs. They are not free-form strings. This is not the same as 852 or necessarily included in it. 318 1311176497000000 thejimmyg Assigning to John so that he can see whether the QA code correctly flags these kinds of problems. If it does, we can close this ticket because although the API will serve invalid URLs, the publishers will be notified to clean up. 318 1311770683000000 johnglover The QA code should identify invalid URLs. Resources with invalid urls will have an 'openness_score' of 0 and an 'openness_score_reason' of 'Invalid url scheme' or 'invalid URL'. 318 1349778662000000 dread "Here's a real example - one of many from MOD {{{http://www.dasa.mod.uk/applications/newWeb/www/index.php?page=48&thiscontent=180&date=2011-05-26&pubType=1&PublishTime=09:30:00&from=home&tabOption=1}}} Browsers accept colons and slashes happily, which is the main usage of our links. The URL looks better with the colons and slashes, rather than the encoded version. The average departmental user doesn't understand that the reason to encode them is for some academic RFC and RDF which is not ""liberal in what it accepts"". Since the RDF tool has a satisfactory way to encode links, this problem is essentially solved. Therefore I'm changing ckanext-archiver to accept these unencoded links, I'm afraid." 319 1274366882000000 dread Fixed in cset:a1ef783d27d2 on default and metastable. 320 1279105983000000 dread "site_title added in cset:b4c0e0a5630d site_logo is changeable in one place in the template so not essential" 320 1279130535000000 dread Took 1.5h 321 1291831399000000 anonymous This has now been superseded with this proposal: #787 322 1274773856000000 pudo This looks very reasonable. Maybe we should have a webhooks client as a simple demo for this? 322 1274807530000000 pudo "Replying to [comment:2 pudo]: > This looks very reasonable. Maybe we should have a webhooks client as a simple demo for this? c.f. #327 " 322 1277722821000000 dread Done but not pushed. Took 3.5 days. 323 1277722845000000 dread Done but not pushed yet. Took 3.5 days. 324 1274807970000000 pudo I am currently writing a Solr subclass for the search index (#317) and would propose adding standard methods to the ckan.libn.search.Search class: index_package(), index_tag(), index_group(). Those could then be called by a generic queue consumer, irrespective of the used search back-end. I will prototype such a consumer soon, so we should talk to avoid doing some duplicate work here. 324 1278599927000000 dread Done in changesets leading up to cset:ca565562129d. 325 1278599979000000 dread Both sending and received tickets closed. 326 1274789296000000 dread Done in cset:66c21d78d2f8. Took 20mins. 327 1281342690000000 rgrp Remove from v1.1 as this awaiting triage now and we are not sure when exactly we will do this. 327 1296467361000000 pudo Nerdy solution that doesn't really seem to catch on, does nothing that cannot be done through queue workers. 328 1275318745000000 dread Done in cset:170cac0b50ac and uploaded to kforge. 329 1275079189000000 dread Fixed in cset:d264f9d57477 and cset:07701ef4085e 330 1275303122000000 dread Fixed in cset:2f18d0e661fd on metastable and default branches. 331 1282833125000000 dread "CKAN timestamps should not be in a timezone, since when the clock goes back, it could cause problems for vdm. But there may be cases when CKAN is running on a machine that needs the clock set for a particular country (say for a front-end running on the same machine), and so vdm should be changed to create timestamps using UTC specifically (rather than add a timezone, since a mixture of timezones won't sort). And when we display it (or reply to a request) we convert it to the local timezone, as suggested in the description." 332 1275306933000000 dread "Switched from YUI a couple of months ago. " 332 1280743320000000 pudo was fixed in cset:1381 333 1275407987000000 dread "Use case 1: decided that when the user is redirected back to the front-end system, the URL contains a parameter with the package just edited. (In addition to the notification message.) Use case 2: decided that if the load on the front-end is not high from 100 non-web requests. Should it become a problem in future, the queue consumer could be adapted to slow down / amalgate multiple requests." 334 1280743667000000 pudo fixed in cset:1380 335 1275997752000000 dread Done in cset:5c0c0b6e1342 335 1276179605000000 dread On discussion with rgrp it's clear that it's also useful to set the redirect url in a config variable - then the client doesn't have to change. This was done in cset:b9fdd208dd45 336 1276162601000000 dread " " 336 1278700021000000 dread Done in cset:742adebb707c and cset:1748e6554e77. 336 1278700266000000 dread Took 0.75 days. 336 1279368544000000 Donny "http://ckan.net/api/search/resource?url=http://scraperwiki.com&all_fields=1&callback=ckantest yields Bad search option: Field ""callback"" not recognised in Resource search." 336 1279373842000000 dread Fixed in cset:e719f449bc74 337 1279300972000000 dread "Fixed in cset:f5cc13ade0e8 Cost: 30m" 338 1279886392000000 johnbywater Putting this into API Version 2 (similar to package references). 340 1276010234000000 dread Done in cset:f77639ddcf0d 340 1328807317000000 dread Went into CKAN 1.0.2? 341 1277483030000000 dread Done in cset:c7a9ba55db0d on metastable and merged to default. 342 1276278485000000 dread Done in cset:61f145b7d4a8 343 1277824018000000 johnbywater "Regarding the package search, if I do: $ paster db clean && paster db init && paster create-test-data and then: $ paster serve development.ini & and then: $ curl http://127.0.0.1:5000/api/1/search/package?name=warandpeace I get: {""count"": 1, ""results"": [""warandpeace""]} and with: $ curl http://127.0.0.1:5000/api/2/search/package?name=warandpeace I get: {""count"": 1, ""results"": [""c90b6c00-9496-4c8c-b7fa-7bdd3ef65c72""]} Am I missing something? " 343 1277892699000000 johnbywater Okay, so in version 2, names were still being used in the relationships part of the packages entity. But I don't know why these entities can't be retrieved independently, and references to the entities returned in representations of the entities which reference them. 344 1277477712000000 dread "Done in cset:13737a7ba4d9 " 345 1291831615000000 thejimmyg "This is a bit out of date. We have moved to a system of ""stable"" and ""default"" branches with feature branches for features, bugs and tickets. We already have default and stable tested by buildbot. " 346 1294410298000000 thejimmyg Could you take a look at this at some point please David? If it is already resolved could you please close the ticket? Thanks! 346 1296477510000000 dread "We no longer use the ""Gdu"" SoS doc." 349 1277820679000000 anonymous Mostly done, but issue regarding departments still outstanding: can the association between packages, groups, and departments be placed elsewhere? 350 1277072822000000 glen@open.org.nz "Also check out the SEO review done by Charles Coxhead - http://www.quicksitereviews.com/ckan-net/ Pagination – In the tag section of the site particularly I’d suggest replacing the traditional numbered pagination with alphabetical links, ie, display 26 links A-Z and then on each of those pages displaying the relevant tags. The point being that the tag pages represent an opportunity to index pages which intersect with very targeted search demand, so let’s give them all every opportunity to get crawled and indexed. Linking to all 26 alphabetically arranged pages from the main tag page (and maybe even the home page) will bring all the tag pages closer to the home page and give them all some more link popularity. Also suggest something similar to this for the listings in the Packages section, so arrange these on alphabetical pages. Tag page titles – Take a programmatic, but more focussed approach to these. Currently all tag pages have the title “CKAN – Comprehensive Knowledge Archive Network – Tags – [tag]” – I would make this something like “[tag] – Open Data – CKAN” or something like that which puts the emphasis on the tag keyword and adds some meaningful qualifiers to that. Tag page listings – On the tag pages there are related open data sets listed and next to each the associated tags. This leads to masses of tag links on the page and loads of duplicated links. If possible I recommend removing the tags from each data set listing, and instead displaying a cloud of “Related tags” on the right hand side. Group pages – The user curated group pages are possibly even better than the tag pages from a search perspective because they all intersect with very clear search demand, so would also benefit from having improved page titles. I suggest a similar approach to the above where the group name becomes the main keyword with some qualifiers added. It might even be a good idea to let group owners define their own page titles. Data set pages – These also could do with improved page titles, presumably the data set name, eg. “[data-set-name] – CKAN” Heading tags – not a big deal but I’m always in favor of using H tags strictly for headlines (rather than wrapping an H1 around the site logo for example)." 350 1279230686000000 dread "Done in cset:b9b82e7ae078 Cost 9 hours, including: alphabetical pager, title changes, removed links from tags in package list." 353 1280262363000000 pudo " cset:1366:e719f449bc74 fixes this" 353 1280737966000000 pudo Add a supervisord example config 353 1280756379000000 pudo This is done as of cset:8 (of ckanext!) 355 1311177552000000 thejimmyg Our policy is to recommend the use of hyphens, but not to enforce it. The new package name suggestion autocomplete JavaScript uses hyphens. 356 1278931816000000 rgrp Done in cset:a1af5f8fe59e. Have not done advanced search link normal package search currently provides no info about how to use advanced features. 357 1277461466000000 johnbywater Fixed in 79c426c0acb6. 358 1303122109000000 kindly "This ticket needs to have a more thorough spec which needs to include. * Examples of put/post requests to resources and if they are needed. * Dealing with resources that do not have a related packages in terms of authorization. Do they have a new action? how granular is the authorization? per resource? system level? etc. * The rules related to authorization for resources attached to packages. i.e you only get read permissions when the related package has read permissions? do they have their own rules? " 358 1303123611000000 dread "This ticket was designed only for reading resources, following an ongoing requirement from the Scraperwiki collaboration. Assume PUT/POST is out of scope. I suggest dealing with resources that aren't attached to packages in an entirely new ticket or CEP, as the implications are wider than this aspect of the API." 358 1310128782000000 thejimmyg Merging with #922 go there for latest updates. 359 1291135692000000 rgrp Done in cset:90e318c3c7dc/datapkg and cset:0036b5c505eb/datapkg and others. 360 1288004891000000 rgrp Done in cset:beaa842ed502/datapkg and following. 361 1291135756000000 rgrp Done in cset:7305c1d04692/datapkg 362 1296469470000000 rgrp Rating are currently disabled (invisible) so moving this down. 362 1311176564000000 thejimmyg This ticket is more than 6 months old so marking as invalid in line with out ticketing policy. 363 1291733459000000 dread Changes to user properties aren't linked to a package. 363 1298840718000000 kindly revision objects are made everytime a new revision is made even if their are no changes. 363 1310125872000000 thejimmyg This will eventually be fixed as part of braoder VDM changes. This work cannot be prioritised above other things we want to do. 364 1281451132000000 dread But this works with the new SOLR search now - close? 364 1291637291000000 rgrp Have now switched to solr search (and maybe working in postgres by now). Note correct link is http://ckan.net/package?q=statistics 365 1279300621000000 dread "Fixed in cset:c11738dcb1ba Cost: 1d" 366 1297075053000000 rgrp Now #938 is done this is straightforward. 366 1299845116000000 dread I'm very pleased that this now works when you try to edit a package that is not allowed. Are there other circumstances we should cover or can we close this? 366 1299845781000000 pudo You're right, that's done! 366 1300212171000000 dread changeset d7a8df888f44 367 1279303693000000 dread "Done in cset:79200de013e1 Cost: 1h" 368 1291831811000000 thejimmyg I don't have enough information to debug this problem. I'm assuming that since this has been a while that the problem is solved? If not please re-open the ticket and add your contact details. 371 1292257189000000 nils.toedtmann "(I know the term ""QoS"" as a very specific networking term about classifying and prioritising network traffic. I assume here it means ''uptime'', ''availability'', ''performance monitoring''?) There seem to be are at least three monitors already in place: * http://munin.okfn.org/ on eu1 monitoring eu[0-7] and us1, gathering additional health information via locally installed daemons. Munin's notification subsystem is not configured. * http://nagios.hmg.ckan.net/ on hmg.ckan.net monitoring the CKAN-HMG service group (network monitoring only). Notfications are not configured (or?) * We have a http://wasitup.com/ account which is watching some OKFN services (e.g. {ca,de,www}.ckan.org, {blog,www}.okfn.org) and sending loads of alerts to sysadmin@okfn.org. Only checking for ""HTTP 200 OK"" and whether the response contains a configurable string. My 2ct: We should consolidate. What do we want? * A webservice like https://www.pingdom.com/ ($40/month incl 30 checks and 200 SMS, $0.5/month per extra check, $0.14-20/SMS) or http://www.serverdensity.com/ ($10/server-month plus 5-10p/SMS)? * Or run our own monitor (nagios, opsview, monin)? In the latter case we want to have a separate machine which is not in on EC2 (but e.g. ByteMark), dedicated to monitoring only. We should also include root mails into the alert/notification policies. Root mails should be trimmed down to important warnings and errors only. " 371 1292257389000000 nils.toedtmann The nagios fork [http://www.opsview.com/ OPSview] might be worth a look. 371 1292704716000000 nils.toedtmann "Replying to [comment:9 nils.toedtmann]: > There seem to be are at least three monitors already in place: [[BR]] Correction: at least four, we seem to have a Montastic account, too:[[BR]] On 18/12/10 15:03, noreply@montastic.com wrote: {{{ Dear okfn, This is a monthly reminder that you have an account on Montastic, the website monitor service. ### ACCOUNT INFORMATION Signup date: 2009-10-06 Email you signup with: sysadmin@okfn.org ### 20 WEBSITES MONITORED [OK] - http://www.ckan.net/ [OK] - http://www.knowledgeforge.net/ [OK] - http://okfn.org/ [not monitored] - http://blog.okfn.org/ [...] ### EMAIL ALERT RECIPIENTS - sysadmin@okfn.org - rufus.pollock@okfn.org - martin.keenan@okfn.org [...] To make changes to your account or contact us, go to www.montastic.com. [...] }}}" 371 1294411939000000 thejimmyg It is implied in this that the performance of sites should beat the QoS criteria, therefore closing #485. Ensuring this happens is an ongoing process. 371 1294417434000000 thejimmyg "From #440 we'll also need to ""Write and pass comprehensive performance tests""" 371 1294417553000000 thejimmyg "From #395: At the moment, some pages within CKAN tend to load slowly. We should create a profiling setup in which we can measure response times for complete requests and individual methods calls. This could be used to identify bottlenecks and find an appropriate caching or tuning strategy to improve CKAN performance. NB: We should also agree on a maximum request latency. TODO: Read up on all those QoS tickets to avoid overlapping efforts. " 371 1294676093000000 anonymous "Mainly handled in http://knowledgeforge.net/okfn/tasks/ticket/564 now. Close here? " 371 1300217820000000 thejimmyg "Marking as closed since http://knowledgeforge.net/okfn/tasks/ticket/600 now takes on this ticket. I will check nils has added the new DGU Bytemark servers are added to Nagios." 372 1280514163000000 johnbywater Moved from sprint 1.1.1 373 1286376071000000 dread Done 374 1280223744000000 johnbywater Requested by DGU. 377 1280323309000000 johnbywater "Thanks a lot! I totally agree and have already started working on this. For example, try pointing your browser to http://ckan.net/api/search/revision (ie without any arguments). Also, there is probably a lot less HTML than there was, since the errors are not handled as if it was an error within the ""Web"" user interface. At any rate, perhaps we could work through the error cases, filling in the tests as we go? I think it's just unworked ground, rather than anything deliberate. :-)" 377 1338206330000000 ross We have another ticket for fixing the API 379 1280326254000000 johnbywater Referencing ticket #378 has changed sprint. 380 1280326254000000 johnbywater Referencing ticket #378 has changed sprint. 381 1294410466000000 thejimmyg There is no mention of what the test defect is. All tests pass so marking invalid. 383 1280348216000000 johnbywater Referencing ticket #382 has changed sprint. 383 1280514163000000 johnbywater Moved from sprint 1.1.1 386 1280356628000000 johnbywater Referencing ticket #372 has changed sprint. 386 1280514164000000 johnbywater Referencing ticket #372 has changed sprint. 386 1281088994000000 johnbywater Not doing for Apache, since it does not run as a user process. 390 1282214629000000 dread "Done - removed 'test' package: 88c485ec-fb70-44b2-9e18-b5dbcb7de57e 2010-07-28 14:35:30.922018 frontend2" 391 1280514163000000 johnbywater Moved from sprint 1.1.1 394 1294407189000000 thejimmyg Duplicate of #371. 395 1281110035000000 johnbywater Moved from sprint 1.1.2 395 1294417538000000 thejimmyg Merging with #371 396 1281110035000000 johnbywater Moved from sprint 1.1.2 398 1282661799000000 rgrp Referencing ticket #481 has changed sprint. 398 1288003624000000 dread Is this to be completed, or are we scrapping it? 398 1294407372000000 thejimmyg "Considering this closed because of this blog post documenting the use of the API and linking to the CKAN docs: * http://data.gov.uk/blog/announcing-datagovuk-catalogue-apis" 399 1282227985000000 dread Finished all code for this today. Cost: 4.5d (+1d to fix ticket:432) 400 1282661799000000 rgrp Referencing ticket #481 has changed sprint. 400 1288003690000000 dread This was done a couple of weeks ago. 401 1281373828000000 pudo "Coming along nicely: * http://bitbucket.org/pudo/ckanextworker " 402 1281373707000000 pudo "This is beginning to go somewhere: * http://bitbucket.org/pudo/ckanextarchive" 402 1296467635000000 pudo De-dup: #891 403 1287392451000000 pudo Available at http://bitbucket.org/pudo/repod 404 1294415587000000 thejimmyg Duplicate of #896 404 1294415965000000 rgrp This is in datapkg, not a duplicate. Partly done in fact but not complete (Indexes are now pluggable). 404 1297072955000000 rgrp Nothing more required here I think. 405 1296467471000000 pudo "Rufus, you mentioned this could be done very quickly - could you have a look? " 405 1297211204000000 rgrp By mime-type and all resources done in cset:7bd693614c80 and previous (with other improvements to download system). 405 1297214793000000 rgrp Completed in cset:a7df5071f200 405 1297214833000000 rgrp (That is did the interactive version) 406 1291139434000000 rgrp Renaming done as part of v0.7 (cset:0ea44c864153/datapkg) but pluggability for downloaders is currently disabled (half-implemented and then decided to hold back pending further thoughts). 406 1296467427000000 pudo Has existed for a while, not implementing any more except for what is in #852. 407 1281506792000000 aidan@scraperwiki Need to agree who is doing this. ATM Julian is on holiday and Richard is focussed on visualisation. Will pick up after holidays - hope this is ok. 407 1291149724000000 anonymous Related bug on ScraperWiki tracker: https://bitbucket.org/ScraperWiki/scraperwiki/issue/42/finish-ckan-integration 407 1330769956000000 rgrp Long out of date ... 408 1281609572000000 pudo "* Plugin for this is located at: http://bitbucket.org/pudo/ckanextdeliverance * Usage is documented in HOWTO.txt (http://bitbucket.org/pudo/ckanextdeliverance/src/tip/HOWTO.txt) * Changes to CKAN were in: cset: 1422" 409 1297074067000000 rgrp Done item (2) http://packages.python.org/datapkg/extending.html#commands and Command class is already pluggable. 410 1281619720000000 pudo Done in http://bitbucket.org/pudo/ckandisqus/changeset/be90fd4d386c 411 1287393033000000 pudo Now using squid in production on eu6; will only need one cache at a time. 413 1282661798000000 rgrp Referencing ticket #481 has changed sprint. 413 1294407429000000 thejimmyg There is no caching in the current DGU CKAN code. 413 1294407458000000 thejimmyg (or rather all caching is disabled). If we re-enable it we'll want to test its behaviour. 414 1282661799000000 rgrp Referencing ticket #481 has changed sprint. 414 1288003770000000 dread Catalogue API went live with caching a couple of weeks ago 415 1288003954000000 dread rdf now available at semantic.ckan.net courtesy of ww 418 1291637919000000 rgrp Not clear what ticket entails and think this has happened (should be on dgu in any case). 421 1282909772000000 dread Done 422 1283165156000000 dread Closed by mistake - task remains. 422 1292586586000000 dread Ticket remaining is #427. No need for this story ticket now. 423 1282812768000000 dread Examples for creating packages with curl are now in the API docs. 424 1281606707000000 johnbywater Referencing ticket #422 has changed sprint. 424 1282919429000000 dread "Done in: https://trac.dataco.coi.gov.uk/projects/datagov/ticket/496#comment:2" 426 1281606707000000 johnbywater Referencing ticket #422 has changed sprint. 426 1282667892000000 dread Referencing ticket #422 has changed sprint. 426 1282925305000000 dread No responses unfortunately - maybe they are fine! 427 1281606707000000 johnbywater Referencing ticket #422 has changed sprint. 427 1283165089000000 dread Posting a new package is done. Just licenses to explain. 427 1287997311000000 dread "To cover in the README: * Deployment of the service - DONE * Adding a licence * Adding a licence group CKAN docs requires: * Example of pointing CKAN to a licence service URI, local file. * Explanation of which licences are in CKAN by default (if you don't point it to a URI)." 427 1289822093000000 dread Referencing ticket #412 has changed sprint. 427 1294407533000000 thejimmyg We may need an updated form of the license service to support UKLP. 427 1297686183000000 thejimmyg Documentation of the licenses service was handled in #973. Changing this ticket to be about matching the license service in UKLII. 427 1299164063000000 thejimmyg This is done in the latest release to test. 428 1285757316000000 dread This was achieved a couple of weeks ago 429 1285757274000000 dread Will Waites fixed this a couple of weeks ago 430 1298283075000000 thejimmyg "We are doing other refactoring that is more important than this such as: * Plugin APIs to enable extensions * Form refactroing This ticket is 6 months old so closing." 432 1282065261000000 dread Fixed in cset:5adbb79d33a2 432 1282552819000000 dread "Looks like the line with the 'location' header went in in this changeset: changeset: 1207:0525981e3b78 user: johnbywater date: Wed Jun 02 12:22:11 2010 +0100 summary: Added support for using package ID as a package reference in all operations of the API. " 433 1282081297000000 wwaites "So one more modification, the datapkg_sources argument is no longer a dictionary but a string, similar to the metadata.txt but with the name as the section heading rather than [DEFAULT] so as to be able to support more than one download. As of now, this works:: {{{ This class treats an installed python package as a data index. For instructions on creating such a package, what needs to go in its setup.py and such, see :func:`datapkg.pypkgtools.datapkg_sources`. Here we are concerned with how to use such a package. An example of one such package can be installed like so:: % pip install hg+http://bitbucket.org/ww/ukgov_treasury_cra Once installed, datapkg can be used to inspect it and install parts wherever desired:: % datapkg list egg://ukgov_treasury_cra cra2009 -- Country and Regional Analysis 2009 % datapkg install egg://ukgov_treasury_cra/cra2009 file:///tmp [...] % ls -l /tmp/cra2009/ total 11112 -rw-r--r-- 1 ww wheel 5681852 May 12 15:48 cra_2009_db.csv -rw-r--r-- 1 ww wheel 292 Aug 17 22:37 metadata.txt }}} Of course the related python code and machinery hasn't been ported over to that package yet, but that's quite another matter." 433 1283183548000000 wwaites changed datapkg_sources to datapkg_index and updated to work with the new changes to how the downloader works. 434 1283189807000000 rgrp Looking at stack trac clear this was an issue with genshi/babel and i18n. Net search led to http://trac.edgewall.org/ticket/9171 which confirmed this. Fact that error was on every page when logged in suggested this was a string in template showing up on log in. Quick search confirmed suspicion ('You are logged in as' string) and fix in http://bitbucket.org/bboissin/ckan-i18n/changeset/8e0c25102cc0 435 1298284084000000 thejimmyg Haven't seen this myself and it is 6 months old now. 436 1288003983000000 dread Not seen since. 437 1288004009000000 dread This was fixed a while ago 438 1294414132000000 thejimmyg Any idea what this ticket means? Should we mark as invalid and close? Thanks. 438 1294661079000000 dread This was to ensure it didn't parse ONS data on the CKAN server, which is complete. 439 1294413903000000 thejimmyg These have been agreed and are awaiting signoff. 440 1294417436000000 thejimmyg Merging with #371. We'll do this as part of the move to the new DGU servers. 441 1282724509000000 dread Comment from pudo: Apache version is documented here: http://knowledgeforge.net/okfn/tasks/ticket/466 441 1282724585000000 anonymous Comment from pudo: CKAN should have a read-only maintenance mode with a nice little banner on all pages, appropriate REST messages etc. Bonus points if this is triggered via an environment variable and thus can be triggered by the surrounding apache. 441 1292586309000000 dread No need for requirements. Story tickets exist: #763 & #765 442 1294414214000000 thejimmyg Is this still important? Do we need to set aside some time for it? Thanks. 442 1294661028000000 dread "I'll close this now. Only thing to note is the importer code pudo wrote in ckanclient that could usefully be merged with the ckanext importer code." 443 1282662560000000 rgrp Referencing ticket #457 has changed sprint. 443 1282735925000000 dread "Now produces daily dump in ~/dumps. Was running as root, not user. Previously it was updating ONS data but not dump because of path issues of being root. Serving files using Apache file index. Added docs for database dumping. Cost: 2h" 444 1294414008000000 thejimmyg Merging with #691 445 1282662530000000 rgrp Referencing ticket #477 has changed sprint. 446 1282423615000000 johnbywater "I think ""an admin writes to the CKAN API"" is a mishearing." 446 1294414077000000 thejimmyg I don't understand this ticket. Since no-one has contributed to this ticket in 5 months I'm closing it. 447 1294410666000000 thejimmyg Hi Richard, could you take a look at this please? It is probably invalid now, in which case feel free to close it. Many thanks, James 447 1294410738000000 memespring Have you got a link or screengrab? 448 1286386256000000 dread Has this all been finished yet? 448 1294417061000000 thejimmyg We do this on an ongoing basis. I don't think we need a ticket for it too. 449 1294411301000000 thejimmyg Will, if this is something you are interested in do you want to kick it off. Otherwise maybe close the ticket? 449 1311182945000000 thejimmyg This ticket is now more than 6 months old so marking as invalid in line with our ticketing policy. 451 1288038146000000 johnbywater Moved from sprint 1.3.0 451 1292587233000000 dread Duplicate of #742 452 1287997540000000 dread John did this a while ago. 453 1294411369000000 thejimmyg This isn't really something that should be a CKAN ticket. It is more an ongoing thing. 454 1286376044000000 dread Done 455 1291637172000000 rgrp Closing as invalid as not clear what task is and now almost certainly out of date. 458 1294415537000000 thejimmyg N/a anymore. 459 1282921783000000 dread "I've now fixed this. default: 1.1 metastable: 1.1 stable: 1.1 / 1.0.2 (two heads) ultrastable: 1.0.2" 460 1282330016000000 richard@cyganiak.de "Note, I'm not a sysadmin but I ''can'' see the Status dropdown for this package, with values ""active"" and ""deleted"". Not knowing much about the permission system, I was assuming that's because I created the package. I am however certain that I did not modify (or even notice) this dropdown in the mysterious edit that led to the state change from ""active"" to ""None""." 460 1285443859000000 richard@cyganiak.de "I believe the initial report is incorrect. It states that the status was changed from ""active"" to ""deleted"". I believe that it was actually changed from ""active"" to ""None"". This might indicate a bug in the code: The value of the status field is lost." 460 1285489964000000 rgrp To add to this ticket I note that package 'owners' are entitled to see State not just system sysadmins (this allows owners to delete packages). 460 1311176868000000 thejimmyg This is most likely fixed in the new logic layer refactor but is more than 6 months old anyway so closing in line with our new ticketing policy. 461 1282662564000000 rgrp Referencing ticket #454 has changed sprint. 462 1285757238000000 dread Available at http://data.gov.uk/data/dumps 463 1282306480000000 dread "Could be related to this shortly afterwards at 22:32: {{{ Mail Contacts Tasks Compose mail Inbox (11364) Starred Sent Mail Drafts (10) alert (5090) bts (383) Notes okfn (134) okfn-lists (26) osm (462) 8 more▼ David Read Google Docs Invite a friend >< About these ads UK Ruby on Rails £10/m - www.1stEasy.com/ruby-on-rails - 1) Phusion Passenger LAMP stack. 2) Great support. 3) Try us free :) « Back to InboxArchiveReport spamDeleteMove to Labels More actions ‹ Newer 29 of 19141 Older › WebApp Error: : None has no member named ""id"" Inbox X Reply no-reply@ckan.net to ckan-sysadmin show details 22:32 (14 hours ago) URL: http://ckan.net/revision/diff/dbtune-audioscrobbler?diff=a99c98be-767a-4e49-9025-2472b2d18b9c&oldid=66a47b9e-232a-4838-8674-66fa1a5c76e1), Module weberror.errormiddleware:162 in __call__ << __traceback_supplement__ = Supplement, self, environ sr_checker = ResponseStartChecker(start_response) app_iter = self.application(environ, sr_checker) return self.make_catching_iter(app_iter, environ, sr_checker) except: >> app_iter = self.application(environ, sr_checker) Module beaker.middleware:73 in __call__ << self.cache_manager) environ[self.environ_key] = self.cache_manager return self.app(environ, start_response) >> return self.app(environ, start_response) Module beaker.middleware:152 in __call__ << headers.append(('Set-cookie', cookie)) return start_response(status, headers, exc_info) return self.wrap_app(environ, session_start_response) def _get_session(self): >> return self.wrap_app(environ, session_start_response) Module routes.middleware:130 in __call__ << environ['SCRIPT_NAME'] = environ['SCRIPT_NAME'][:-1] response = self.app(environ, start_response) # Wrapped in try as in rare cases the attribute will be gone already >> response = self.app(environ, start_response) Module pylons.wsgiapp:125 in __call__ << controller = self.resolve(environ, start_response) response = self.dispatch(controller, environ, start_response) if 'paste.testing_variables' in environ and hasattr(response, >> response = self.dispatch(controller, environ, start_response) Module pylons.wsgiapp:324 in dispatch << if log_debug: log.debug(""Calling controller class with WSGI interface"") return controller(environ, start_response) def load_test_env(self, environ): >> return controller(environ, start_response) Module ckan.lib.base:73 in __call__ << # available in environ['pylons.routes_dict'] try: return WSGIController.__call__(self, environ, start_response) finally: model.Session.remove() >> return WSGIController.__call__(self, environ, start_response) Module pylons.controllers.core:221 in __call__ << return response(environ, self.start_response) response = self._dispatch_call() if not start_response_called: self.start_response = start_response >> response = self._dispatch_call() Module pylons.controllers.core:172 in _dispatch_call << req.environ['pylons.action_method'] = func response = self._inspect_call(func) else: if log_debug: >> response = self._inspect_call(func) Module pylons.controllers.core:107 in _inspect_call << func.__name__, args) try: result = self._perform_call(func, args) except HTTPException, httpe: if log_debug: >> result = self._perform_call(func, args) Module pylons.controllers.core:60 in _perform_call << """"""Hide the traceback for everything above this method"""""" __traceback_hide__ = 'before_and_this' return func(**args) def _inspect_call(self, func): >> return func(**args) Module ckan.controllers.revision:123 in diff << c.diff.sort() c.pkg = pkg return render('revision/diff.html') def _has_purge_permissions(self): >> return render('revision/diff.html') Module ckan.lib.base:38 in render << return literal(stream.render(method=method, encoding=None)) return render_template() #return cached_template(template_name, render_template, cache_key=cache_key, # cache_type=cache_type, cache_expire=cache_expire, >> return render_template() Module ckan.lib.base:36 in render_template << stream = _filter(stream) return literal(stream.render(method=method, encoding=None)) return render_template() >> return literal(stream.render(method=method, encoding=None)) Module genshi.core:183 in render << method = self.serializer or 'xml' generator = self.serialize(method=method, **kwargs) return encode(generator, method=method, encoding=encoding, out=out) def select(self, path, namespaces=None, variables=None): >> return encode(generator, method=method, encoding=encoding, out=out) Module genshi.output:57 in encode << _encode = lambda string: string if out is None: return _encode(''.join(list(iterator))) for chunk in iterator: out.write(_encode(chunk)) >> return _encode(''.join(list(iterator))) Module genshi.output:339 in __call__ << for filter_ in self.filters: stream = filter_(stream) for kind, data, pos in stream: cached = cache_get((kind, data)) if cached is not None: >> for kind, data, pos in stream: Module genshi.output:670 in __call__ << _gen_prefix = _gen_prefix().next for kind, data, pos in stream: output = cache_get((kind, data)) if output is not None: >> for kind, data, pos in stream: Module genshi.output:771 in __call__ << push_text = textbuf.append pop_text = textbuf.pop for kind, data, pos in chain(stream, [(None, None, None)]): if kind is TEXT: >> for kind, data, pos in chain(stream, [(None, None, None)]): Module genshi.output:586 in __call__ << def __call__(self, stream): prev = (None, None, None) for ev in stream: if prev[0] is START: if ev[0] is END: >> for ev in stream: Module genshi.core:288 in _ensure << # unchanged yield event for event in stream: yield event >> for event in stream: Module genshi.template.base:618 in _include << tmpl = self.loader.load(href, relative_to=event[2][0], cls=cls or self.__class__) for event in tmpl.generate(ctxt, **vars): yield event except TemplateNotFound: >> for event in tmpl.generate(ctxt, **vars): Module genshi.template.base:618 in _include << tmpl = self.loader.load(href, relative_to=event[2][0], cls=cls or self.__class__) for event in tmpl.generate(ctxt, **vars): yield event except TemplateNotFound: >> for event in tmpl.generate(ctxt, **vars): Module genshi.template.base:618 in _include << tmpl = self.loader.load(href, relative_to=event[2][0], cls=cls or self.__class__) for event in tmpl.generate(ctxt, **vars): yield event except TemplateNotFound: >> for event in tmpl.generate(ctxt, **vars): Module genshi.template.base:605 in _include << from genshi.template.loader import TemplateNotFound for event in stream: if event[0] is INCLUDE: href, cls, fallback = event[1] >> for event in stream: Module genshi.template.markup:378 in _match << for event in self._match(self._flatten(template, ctxt, **vars), ctxt, start=idx + 1, **vars): yield event >> ctxt, start=idx + 1, **vars): Module genshi.template.markup:327 in _match << break for event in stream: # We (currently) only care about start and end events for matching >> for event in stream: Module genshi.template.base:565 in _flatten << elif kind is EXPR: result = _eval_expr(data, ctxt, vars) if result is not None: # First check for a string, otherwise the iterable test >> result = _eval_expr(data, ctxt, vars) Module genshi.template.base:277 in _eval_expr << if vars: ctxt.push(vars) retval = expr.evaluate(ctxt) if vars: ctxt.pop() >> retval = expr.evaluate(ctxt) Module genshi.template.eval:178 in evaluate << __traceback_hide__ = 'before_and_this' _globals = self._globals(data) return eval(self.code, _globals, {'__data__': data}) >> return eval(self.code, _globals, {'__data__': data}) Module ?:11 in <<

From: ${c.revision_from.id} - ${c.revision_from.timestamp}

>> From: ${c.revision_from.id} - ${c.revision_from.timestamp} Module genshi.template.eval:324 in lookup_attr << val = obj[key] except (KeyError, TypeError): val = cls.undefined(key, owner=obj) return val >> val = cls.undefined(key, owner=obj) Module genshi.template.eval:410 in undefined << """"""Raise an ``UndefinedError`` immediately."""""" __traceback_hide__ = True raise UndefinedError(key, owner=owner) >> raise UndefinedError(key, owner=owner) UndefinedError: None has no member named ""id"" CGI Variables DOCUMENT_ROOT '/htdocs' GATEWAY_INTERFACE 'CGI/1.1' HTTP_ACCEPT 'image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/x-ms-application, application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml+xml, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, application/x-silverlight, */*' HTTP_ACCEPT_ENCODING 'gzip, deflate' HTTP_ACCEPT_LANGUAGE 'en-us' HTTP_CONNECTION 'TE, close' HTTP_HOST 'ckan.net' HTTP_TE 'deflate,gzip;q=0.3' HTTP_UA_CPU 'x86' HTTP_USER_AGENT 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; MS-RTC LM 8)' PATH '/usr/local/bin:/usr/bin:/bin' PATH_INFO '/revision/diff/dbtune-audioscrobbler' PATH_TRANSLATED '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py/revision/diff/dbtune-audioscrobbler' QUERY_STRING 'diff=a99c98be-767a-4e49-9025-2472b2d18b9c&oldid=66a47b9e-232a-4838-8674-66fa1a5c76e1),' REMOTE_ADDR '89.145.108.201' REMOTE_PORT '39586' REQUEST_METHOD 'GET' REQUEST_URI '/revision/diff/dbtune-audioscrobbler?diff=a99c98be-767a-4e49-9025-2472b2d18b9c&oldid=66a47b9e-232a-4838-8674-66fa1a5c76e1),' SCRIPT_FILENAME '/home/okfn/var/srvc/ckan.net/pyenv/bin/ckan.net.py' SCRIPT_URI 'http://ckan.net/revision/diff/dbtune-audioscrobbler' SCRIPT_URL '/revision/diff/dbtune-audioscrobbler' SERVER_ADDR '10.226.226.118' SERVER_ADMIN '[no address given]' SERVER_NAME 'ckan.net' SERVER_PORT '80' SERVER_PROTOCOL 'HTTP/1.1' SERVER_SIGNATURE '
Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2 Server at ckan.net Port 80
\n' SERVER_SOFTWARE 'Apache/2.2.9 (Debian) mod_wsgi/2.5 Python/2.5.2' WSGI Variables application beaker.cache beaker.get_session > beaker.session {'_accessed_time': 1282253553.63995, '_creation_time': 1282253553.63995} mod_wsgi.application_group 'ckan.net|' mod_wsgi.callable_object 'application' mod_wsgi.listener_host '' mod_wsgi.listener_port '80' mod_wsgi.process_group '' mod_wsgi.reload_mechanism '0' mod_wsgi.script_reloading '1' mod_wsgi.version (2, 5) paste.cookies (, '') paste.parsed_querystring ([('diff', 'a99c98be-767a-4e49-9025-2472b2d18b9c'), ('oldid', '66a47b9e-232a-4838-8674-66fa1a5c76e1),')], 'diff=a99c98be-767a-4e49-9025-2472b2d18b9c&oldid=66a47b9e-232a-4838-8674-66fa1a5c76e1),') paste.registry paste.throw_errors True pylons.action_method > pylons.controller pylons.environ_config {'session': 'beaker.session', 'cache': 'beaker.cache'} pylons.pylons pylons.routes_dict {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'} repoze.who.logger repoze.who.plugins {'openid': , 'auth_tkt': } routes.route routes.url webob._parsed_query_vars (GET([('diff', 'a99c98be-767a-4e49-9025-2472b2d18b9c'), ('oldid', '66a47b9e-232a-4838-8674-66fa1a5c76e1),')]), 'diff=a99c98be-767a-4e49-9025-2472b2d18b9c&oldid=66a47b9e-232a-4838-8674-66fa1a5c76e1),') webob.adhoc_attrs {'language': 'en-us'} wsgi process 'Multi process AND threads (?)' wsgi.file_wrapper wsgiorg.routing_args (, {'action': u'diff', 'controller': u'revision', 'id': u'dbtune-audioscrobbler'}) }}}" 463 1294916148000000 dread Haven't seen this for ages. 464 1282325194000000 johnbywater Duplicates #458. 465 1282724701000000 dread This was an idea, to be able to monitor users overloading. It may not be required though. 465 1294411534000000 thejimmyg Let's reopen this ticket if it comes up again. For now not an issue. Maybe also related to caching tickets #840 #841 #856 #668 #543 #540 #537. 466 1282723866000000 dread As mentioned yesterday, I'm not convinced this is the best way to achieve the actual requirement, which is to access the API securely, and we should discuss this further. 466 1282921608000000 dread We've agreed that we need another header for this, although it's not clear that it needs to be configurable. We could just accept Authorization OR the new one. 466 1294411633000000 thejimmyg What was the original requirement? What's wrong with the current HTTP header name? Can we not mark this as wontfix for now and re-open if needed? 466 1294835610000000 dread Agreed 467 1294411656000000 thejimmyg Duplicate of #467 467 1294411681000000 thejimmyg Sorry, that should have been #466. 468 1282662774000000 rgrp Referencing ticket #467 has changed sprint. 469 1282662778000000 rgrp Referencing ticket #471 has changed sprint. 470 1282662774000000 rgrp Referencing ticket #467 has changed sprint. 470 1292587187000000 dread apikey_header_name was set to X-CKAN-API-Key some time ago I believe. 474 1294916760000000 dread Questions are out of date now 475 1282662794000000 rgrp Referencing ticket #472 has changed sprint. 476 1294411741000000 thejimmyg Where did this requirement come from? Setting to wontfix for the timebeing. Feel free to re-open if there is a clear use case for this. 477 1294411761000000 thejimmyg Duplicate of #476 478 1282662798000000 rgrp Referencing ticket #473 has changed sprint. 479 1282662526000000 rgrp Referencing ticket #462 has changed sprint. 479 1282920017000000 dread Still discussing. Current thinking that the apache file listing would easiest be opened up together with CKAN read-only web i/f. 479 1288004211000000 dread Done 480 1294411813000000 thejimmyg We are currently updating the specification and the servers this runs on. 480 1300281551000000 thejimmyg This is complete (albeit with a different architecture). 481 1294248359000000 thejimmyg This is now implemented for DGU. 482 1294417310000000 thejimmyg At some point we may want to introduce rate limiting on the CKAN API. 482 1298284158000000 thejimmyg This is now 6 months old and there still doesn't seem to be a requirement for this. Marking wontfix and we can come back to it if it comes up again. 483 1294417216000000 thejimmyg There isn't a catalogue web UI, just CKAN. I don't understand this, marking invalid. See instead #482 and #484. 484 1294417248000000 thejimmyg There isn't a catalogue web UI, just CKAN. I don't understand this, marking invalid. See instead #482. 485 1294411946000000 thejimmyg Merging with #371. 486 1291639321000000 rgrp This is a requirement ticket which we no longer support and a duplicate #847. 487 1291639404000000 rgrp Obsolete story type and looks like duplicate of #353 (and other parts of ticket:847) 488 1320930240000000 dread Not a current reqt. 489 1294414420000000 thejimmyg Which RDF service? Why does it need model events? Will, any ideas or shall I close it? Thanks. 489 1294416189000000 wwaites "We still do need something like this. Right now the rdf generation is a cron job that crawls the API. Really we want two things: * only generate RDF for those packages that have changed * enable multiple client/worker processes, potentially run by third parties, to generate different parts of the RDF description. For example, it is reasonable that we generate the DCat representation ourselves, however the voiD authors (DCat is generic, voiD is about RDF datasets in particular) want to generate the voiD-specific annotations themselves and contribute it back to augment the catalogue. Another example: group curators may also want to annotate packages in their group with group-specific metadata. Yet another example: checks on the coherence, availability, quality, openness, etc. of a package should be done from time to time or when a package changes, which can result in further annotations (see the curate tool). Because of the ""third party"" aspect we cannot do this with the internal package representation and the rabbit queue. Most likely it is feasible to do this by looking at the revisions in the API to get all changes since the last time a script was run in which case most likely the answer is, yes, there is nothing to do here and the ticket can be closed." 490 1283770756000000 wwaites Presuming this is for INSPIRE compliance, support for Opensearch should be added 496 1294407899000000 thejimmyg "This would only apply to packages that had a HarvestedDcoument record (ie had been harvested). There are two approaches for this: * Build the capability to respond to CSW ""GetCapabilities"" and ""Get Records"" requests * Export to an existing CSW server to provide the mechanism for us" 496 1297684298000000 thejimmyg "The latest plan after update with PP is as follows: * CKAN will have a CSW interface * OS GenoNetwork will use this interface to determine: * New documents * Modified documents * Removed documents * OS will then handle the serving back to the EU since GeoNetwork already implements the custom filters the EU may require (their docs are ambiguous). This means the creation of the CSW server extension is now a high priority but it only needs to know about documents in the harvested_documents table. I've already implemented a REST API to get the those documents (but depending on the implementation it may need changing)." 496 1299164106000000 thejimmyg Will has implemented this now and OS have confirmed their export to GeoNetwork works. 497 1294407705000000 thejimmyg Merged with #497 497 1294407718000000 thejimmyg Sorry that should have been #496. 498 1294412520000000 thejimmyg This is something that the Drupal team would need to do. There is no ability or requirement to embed the map widget into the CKAN search system yet. 499 1296593038000000 thejimmyg Not on the SoR. 500 1287747652000000 rgrp Looks like a duplicate of ticket:463 (also added by dread -- so hope I'm not making a mistake!) 501 1282563166000000 pudo * Apache version is documented here: http://knowledgeforge.net/okfn/tasks/ticket/466 501 1282724566000000 dread Good thinking - see previous ticket on this: ticket:441 502 1292586466000000 dread Not doing data4nr at the moment. 503 1314031851000000 dread "Relationships exist now on ckan.net. e.g. http://ckan.net/package/rkb-explorer-ibm No current DGU interest." 505 1288606561000000 dread Referencing ticket #504 has changed sprint. 505 1298368280000000 dread Now complete 507 1282909852000000 dread Done 508 1282822372000000 dread "ultrastable branch created from 1.0.1 - version on dgu. Developing policy at wiki:BranchingPolicy" 508 1282908795000000 dread Policy discussed on list. 509 1285757116000000 dread Will Waites did this a couple of weeks agoWill Waites did this a couple of weeks ago 509 1291734435000000 dread Story no longer required. Work to do is still described in #510 510 1282925375000000 dread Referencing ticket #509 has changed sprint. 510 1292955569000000 dread Now setup - just need to check the cron fires ok. 510 1294138332000000 dread Completed 21st Dec 2010. 511 1297075354000000 thejimmyg There have been quite a lot of improvements, this is now OK. 512 1294917121000000 dread Same as #513 513 1302774329000000 dread In enh-1046-dictize-the-api we remove the distinction of extras, so we can't do this. 514 1282757391000000 dread Same as ticket:515 515 1302774253000000 dread This is now fixed in enh-1046-dictize-the-api. Both groups and packages return the location header. 516 1288002933000000 dread Someone has fixed this. 517 1283897199000000 pudo See forms/iati, fixtures.py, fixtures.json as of ckanextiati: a02467cb67ba 518 1282893844000000 pudo " * Priority: 5 * Web user interface for publishing entities " 518 1283536932000000 pudo " * Still needs ""archive file"" and ""donors country"" fields, PE questions to be answered. " 518 1283896718000000 pudo functional as of cset:ae4d4263b0c0 in ckanextiati 519 1283536828000000 pudo * Mostly implemented, still have to consider further cleanup. 520 1283538080000000 pudo * http://bitbucket.org/okfn/iati/changeset/378431974c76 solr facets 521 1282893545000000 pudo "Ability to search over the IATI metadata stored about the IATI records that have been registered. * Priority: 5 (may merge into 1 to some extent)" 521 1283538342000000 pudo " * still needs to have schema adapted a bit * work so far is at eu4:~/var/solr/iati/conf/schema.xml" 521 1283897124000000 pudo Schema has been adapted to suit IATI 522 1285595317000000 pudo "Still missing some internal authorization work but on the UI side, things are mostly functional. cf. http://bitbucket.org/okfn/ckanextiati/changeset/b2588091fa56 ff" 522 1287392999000000 pudo Has been fixed for quite some time. 523 1283897688000000 pudo " * Auth will be finished in #524. * PE write access etc. via standard API, compare ckanextiati:a02467cb67ba" 524 1285594971000000 pudo fixed as of cset:470d1e1c4460 525 1285595080000000 pudo "As of now, PE creation can only be done by sysadmins. When opened to normal users, they will not be able to set the group type and thus moderation will be required of sysadmins. " 526 1320930201000000 dread Is this still relevant or can it be closed? 526 1340626152000000 icmurray Won't fix as it's 20 months old, and no feedback on whether it's still relevant or not. 527 1293097531000000 rgrp Just to note: did this relate to IATI or ...? Any way to add component and milestone? 528 1283536475000000 pudo Done at http://iati.ckan.net 529 1283536554000000 pudo at https://spreadsheets5.google.com/ccc?key=tuOtQjD0Psoqr1pWTS8EXZQ&hl=en#gid=0 530 1282899463000000 pudo "The core of the COI/data.gov.uk standard seems to look like: Identifier Title Abstract Department Contact Contact e-mail Licence Resource format Resource URL Resource ID Publisher And the IATI-specific ones we'd want are: Publishing Entity: Publishing Entity Type: (Donor, Recipient, Community Data..) Donor Country Activity period: Verification status: enumeration of statuses (checked, not checked etc) Resource links: to the actual IATI record Number of activities: ... Date record updated: Date data updated: License: Need this field even if it may be a standard license So naively mashing these together, we get something like: Identifier Title Abstract Donor Country Publisher Publisher Type Verification Status Department Contact Contact e-mail Licence Resource format Resource URL Resource ID Activity period Number of activities Date record updated Date data updated" 531 1283536676000000 pudo done: http://bitbucket.org/okfn/ckanextiati/src/tip/ckanext/iati/fixtures.py 533 1292957374000000 dread Not a current issue afaik. 534 1288002762000000 dread I fixed this a while back. 535 1283167040000000 rgrp Duplicate of #434 536 1286376029000000 dread I did this a few weeks ago. 537 1283325156000000 wwaites See #540 for a story about Varnish. Strongly favouring squid at this juncture. 537 1288023097000000 dread Can this ticket be updated? Were any tasks listed here done? Anything remaining still planned? 537 1311178929000000 thejimmyg Consolidation of caching has been moved in ticket #995. 538 1294412635000000 thejimmyg Rufus, is this something you want to take on or shall we close the ticket as wontfix? 538 1294414537000000 rgrp We don't use reports any more -- just the query module -- so closing as wontfix. 539 1303118486000000 thejimmyg Old ticket, not sure exactly what it is referring to, closing. 540 1283324947000000 wwaites "Cut-and-paste from ckan-discuss: I had a look at Varnish and I agree that the configuration language is complicated. In fact by default Varnish disregards cache control headers and in general behaves in a very standards non-compliant way. I have no doubt that it is very fast -- if you are willing to spend the efford to customise its configuration for the exact layout of pages and headers and such that each web site it is going to be used with will use. In other words, there is a large administrative burden. So I decided to change tack and see where the Squid proxy has gotten to in the decade or so since I last met it. Squid is a general purpose caching proxy that can be configured as an http accelerator. The configuration is simple. You tell it where your web servers are for which sites. The web servers make sure to set the cache control headers appropriately. Here are some results from my testing, against http://de.ckan.net/package/list?page=B which is an example of a slow page. Except for the first, which only did 100 requests, the tests were set to 8 simultaneous connections and a total of 1000 requests. {{{ No caching of any kind: Requests per second: 0.44 [#/sec] (mean) Beaker Cache (filesystem): Requests per second: 43.16 [#/sec] (mean) SQUID setting cache control headers correctly: Requests per second: 421.33 [#/sec] (mean) }}} The results are clear. Using the application cache is about 100 times faster than doing nothing. Using squid is about 1000 times faster. (Doing both wouldn't necessarily help very much). I'm sure we could squeeze a bit more performance out of it if we used Varnish, but probably not an order of magnitude and I don't think it is worth the administrative burden. If we set up a production Squid instance (or farm), with a bare minimum of work it can cache for any number of sites, not just CKAN. For the python coders, here's what you have to do to set the headers properly so that squid will cache the page: {{{ del response.headers[""Pragma""] del response.headers[""Cache-Control""] from time import gmtime, strftime response.headers[""Last-Modified""] = strftime(""%a, %d %b %Y %H:%M:%S GMT"", gmtime()) response.cache_expires(seconds=3600) }}} A further advantage is that the *browsers* will also understand these cache-control headers and do their own caching - just setting them properly without even using Squid should result in some subjective performance improvements. That's all for now, I suggest we dedicate a machine to just running squid, the more RAM the better and big discs are good, and put it between the world and the ckans. Oh, and comb through the controllers setting the headers correctly where appropriate... " 540 1302694845000000 dread Closing - all the suggestions have been implemented: squid instance and cache headers set for high traffic pages. 541 1294924872000000 wwaites this is implemented with the cookie javascript technique. not so suitable for pages where content is radically differerent depending on the logged in user but for small bits like the logged in status works perfectly well, likewise for edit links. 542 1283278896000000 wwaites "Implemented with: * cset:2bbc186459cb * cset:eba23cc027e5 * cset:e89e0b82b24e * cset:943ed812e237" 543 1311178918000000 thejimmyg Consolidation of caching has been moved in ticket #995. 544 1291638966000000 rgrp Duplicate of #672 546 1286896167000000 anonymous [http://www.pokers.li Jeux de poker en ligne] 546 1286896450000000 anonymous [http://www.bonus-pokers.eu Poker en ligne] 546 1286896814000000 anonymous [http://www.salle-de-poker-legal.com Jouer au poker en ligne] 548 1283340866000000 johnbywater Referencing ticket #546 has changed sprint. 549 1283340866000000 johnbywater Referencing ticket #546 has changed sprint. 550 1283340866000000 johnbywater Referencing ticket #546 has changed sprint. 551 1283340866000000 johnbywater Referencing ticket #546 has changed sprint. 552 1283340874000000 johnbywater Referencing ticket #547 has changed sprint. 553 1283340874000000 johnbywater Referencing ticket #547 has changed sprint. 554 1283340873000000 johnbywater Referencing ticket #547 has changed sprint. 555 1283340873000000 johnbywater Referencing ticket #547 has changed sprint. 556 1283340873000000 johnbywater Referencing ticket #547 has changed sprint. 558 1283781883000000 pudo fixed in cset:2c4c43302f21 559 1288085627000000 pudo Still missing: the possibility to remove an added entry before committing the form (i.e. you look up a package and then decide not to include it after all). 559 1310126313000000 shevski I don't understand this one. It is older than 6 months so marking invalid. 559 1310127694000000 dread Makes sense to me... Works for me too, so leaving closed. 560 1285430085000000 ehs@pobox.com I think I'm seeing a variation on this behavior: I'm retrieving the packages in the lodcloud group (http://ckan.net/api/rest/group/lodcloud) and finding bibbase, and then fetching the package details (http://ckan.net/api/rest/package/bibbase) and getting a 403 Access Denied. If access to the package isn't available I would think it wouldn't be listed in the lodcloud group response. 560 1288002971000000 dread This is still an issue. 560 1297084192000000 kindly "changeset https://bitbucket.org/okfn/ckan/changeset/b899085071a8 cset:b899085071a8" 562 1284063574000000 pudo fixed by ww by upgrading python-openid 563 1294231961000000 thejimmyg There is a choice here as to whether we provide an export to GeoNetwork or support a minimal CSW interface ourselves. 563 1296592472000000 thejimmyg This is a duplicate of #496 564 1294412714000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing. 565 1294412708000000 thejimmyg Don't understand this one? Anyone else know about it? 565 1294417752000000 thejimmyg OK, just looked this up. We are moving to a publisher/provider model as a first step, not moving to code lists. Marking invalid. 566 1294232284000000 thejimmyg To a large extent it does now so closing this ticket in favour of more specific ones. 568 1285957164000000 johnbywater Moved from sprint 1.2.5 568 1286786419000000 johnbywater Moved from sprint 1.2.6 568 1287392122000000 johnbywater Moved from sprint 1.2.8 568 1287776309000000 johnbywater Moved from sprint 1.2.9 569 1285957164000000 johnbywater Moved from sprint 1.2.5 569 1286786420000000 johnbywater Moved from sprint 1.2.6 569 1287392122000000 johnbywater Moved from sprint 1.2.8 569 1288038146000000 johnbywater Moved from sprint 1.3.0 569 1294408084000000 thejimmyg "Our metadata validation will consist of: * Testing the schema * Testing the schematron * Checking we can get out the data we need " 569 1297177220000000 wwaites so this means a dependency on amara (finally). on the plus side it is one of the better xml python libraries, xpath, xslt, etc. on the down side it how many xml libraries do we depend on and use now? this is for the schematron... 569 1297177522000000 wwaites *sigh* but it relies on 4suite in version 1.x and amara2 doesn't have the schematron anymore... 569 1297177695000000 wwaites *sigh* again. the gemini schematron schema isn't compatible with the amara scimitar... despite the instructions to use the implementation from schematron.com... 569 1297347214000000 thejimmyg This is now complete in ckanext-csw. 570 1286786420000000 johnbywater Moved from sprint 1.2.6 570 1287392123000000 johnbywater Moved from sprint 1.2.8 570 1288038146000000 johnbywater Moved from sprint 1.3.0 570 1294407974000000 thejimmyg Merged with #569. 572 1285957164000000 johnbywater Moved from sprint 1.2.5 572 1286786420000000 johnbywater Moved from sprint 1.2.6 572 1287392122000000 johnbywater Moved from sprint 1.2.8 572 1287776309000000 johnbywater Moved from sprint 1.2.9 575 1288012749000000 johnbywater This was supported with harvesting jobs. 576 1284212162000000 wwaites "* cset:9b1255e02e5e removes a reference to Package.c which is deprecated even in 0.4 * cset:d7e583948c95 wraps the session creation in if/else block on SQAlchemy version as the default behaviours and options have changed Unit Tests appear to work as before... Boy do they take a long time to run..." 576 1284215009000000 wwaites cset:7a04e78cec97 ''pool_threadlocal=True'' is no longer the default with 0.5. Setting it to ''False'' (the default) causes unit tests to hang. So explicitly set it to ''True'' to retain compatibility. It is an open question whether this is the ''correct'' setting, the defaults were changed as apparently they led to surprising behaviour. One hopes that our unit tests have already eliminated any such surprises. 576 1284222948000000 wwaites "Most of the way to 0.6 compatibility with cset:ce6b870570c1/vdm All tests still pass with 0.4 and 0.5" 576 1294753848000000 dread Seb's completed this now I believe. It's merged into head CKAN cset:68d63fda4814. 581 1284219829000000 johnbywater Referencing ticket #580 has changed sprint. 582 1284216078000000 johnbywater Referencing ticket #491 has changed sprint. 583 1284216078000000 johnbywater Referencing ticket #491 has changed sprint. 584 1284216078000000 johnbywater Referencing ticket #491 has changed sprint. 585 1284216078000000 johnbywater Referencing ticket #491 has changed sprint. 586 1284216078000000 johnbywater Referencing ticket #491 has changed sprint. 587 1284216078000000 johnbywater Referencing ticket #491 has changed sprint. 588 1284216104000000 johnbywater Referencing ticket #492 has changed sprint. 589 1284216104000000 johnbywater Referencing ticket #492 has changed sprint. 590 1284216104000000 johnbywater Referencing ticket #492 has changed sprint. 591 1284216120000000 johnbywater Referencing ticket #493 has changed sprint. 592 1284216120000000 johnbywater Referencing ticket #493 has changed sprint. 593 1284216137000000 johnbywater Referencing ticket #494 has changed sprint. 593 1284225618000000 johnbywater Referencing ticket #494 has changed sprint. 594 1284216137000000 johnbywater Referencing ticket #494 has changed sprint. 594 1284225618000000 johnbywater Referencing ticket #494 has changed sprint. 595 1284216137000000 johnbywater Referencing ticket #494 has changed sprint. 595 1284225618000000 johnbywater Referencing ticket #494 has changed sprint. 596 1284216140000000 johnbywater Referencing ticket #495 has changed sprint. 596 1284225651000000 johnbywater Referencing ticket #495 has changed sprint. 597 1284216140000000 johnbywater Referencing ticket #495 has changed sprint. 597 1284225651000000 johnbywater Referencing ticket #495 has changed sprint. 599 1284216189000000 johnbywater Referencing ticket #598 has changed sprint. 600 1284216189000000 johnbywater Referencing ticket #598 has changed sprint. 601 1284216189000000 johnbywater Referencing ticket #598 has changed sprint. 602 1284219570000000 johnbywater Referencing ticket #567 has changed sprint. 603 1284219570000000 johnbywater Referencing ticket #567 has changed sprint. 604 1284219570000000 johnbywater Referencing ticket #567 has changed sprint. 605 1284219570000000 johnbywater Referencing ticket #567 has changed sprint. 606 1284219594000000 johnbywater Referencing ticket #578 has changed sprint. 607 1284219594000000 johnbywater Referencing ticket #578 has changed sprint. 608 1284219588000000 johnbywater Referencing ticket #577 has changed sprint. 608 1284689330000000 johnbywater Referencing ticket #577 has changed sprint. 609 1284219588000000 johnbywater Referencing ticket #577 has changed sprint. 609 1284689330000000 johnbywater Referencing ticket #577 has changed sprint. 610 1284219601000000 johnbywater Referencing ticket #569 has changed sprint. 610 1284225063000000 johnbywater Referencing ticket #570 has changed sprint. 610 1285199168000000 johnbywater Referencing ticket #570 has changed sprint. 610 1285629333000000 johnbywater Referencing ticket #570 has changed sprint. 610 1286786420000000 johnbywater Referencing ticket #569 has changed sprint. 610 1287392122000000 johnbywater Referencing ticket #569 has changed sprint. 610 1287703423000000 johnbywater Referencing ticket #569 has changed sprint. 610 1288038146000000 johnbywater Referencing ticket #569 has changed sprint. 611 1284219601000000 johnbywater Referencing ticket #569 has changed sprint. 611 1284689449000000 johnbywater Referencing ticket #569 has changed sprint. 611 1285199361000000 johnbywater Referencing ticket #569 has changed sprint. 611 1285957164000000 johnbywater Referencing ticket #569 has changed sprint. 611 1286786420000000 johnbywater Referencing ticket #569 has changed sprint. 611 1287392122000000 johnbywater Referencing ticket #569 has changed sprint. 611 1287703423000000 johnbywater Referencing ticket #569 has changed sprint. 611 1288038146000000 johnbywater Referencing ticket #569 has changed sprint. 611 1294408239000000 thejimmyg I don't understand this task, but suspect it can be treated as a duplicate of #569. 612 1284219601000000 johnbywater Referencing ticket #569 has changed sprint. 612 1284689449000000 johnbywater Referencing ticket #569 has changed sprint. 612 1285199361000000 johnbywater Referencing ticket #569 has changed sprint. 612 1285957164000000 johnbywater Referencing ticket #569 has changed sprint. 612 1286786420000000 johnbywater Referencing ticket #569 has changed sprint. 612 1287392122000000 johnbywater Referencing ticket #569 has changed sprint. 612 1287703423000000 johnbywater Referencing ticket #569 has changed sprint. 612 1288038146000000 johnbywater Referencing ticket #569 has changed sprint. 612 1294408188000000 thejimmyg Duplicate of #569. 613 1284219601000000 johnbywater Referencing ticket #569 has changed sprint. 613 1284689450000000 johnbywater Referencing ticket #569 has changed sprint. 613 1285199361000000 johnbywater Referencing ticket #569 has changed sprint. 613 1285957164000000 johnbywater Referencing ticket #569 has changed sprint. 613 1286786420000000 johnbywater Referencing ticket #569 has changed sprint. 614 1284219601000000 johnbywater Referencing ticket #569 has changed sprint. 614 1284689449000000 johnbywater Referencing ticket #569 has changed sprint. 614 1285199361000000 johnbywater Referencing ticket #569 has changed sprint. 614 1285957164000000 johnbywater Referencing ticket #569 has changed sprint. 614 1286786420000000 johnbywater Referencing ticket #569 has changed sprint. 614 1287392122000000 johnbywater Referencing ticket #569 has changed sprint. 614 1287703423000000 johnbywater Referencing ticket #569 has changed sprint. 614 1288038146000000 johnbywater Referencing ticket #569 has changed sprint. 615 1284219609000000 johnbywater Referencing ticket #570 has changed sprint. 615 1284225063000000 johnbywater Referencing ticket #570 has changed sprint. 615 1285199168000000 johnbywater Referencing ticket #570 has changed sprint. 615 1285629333000000 johnbywater Referencing ticket #570 has changed sprint. 615 1286786420000000 johnbywater Referencing ticket #570 has changed sprint. 615 1287392123000000 johnbywater Referencing ticket #570 has changed sprint. 615 1287703431000000 johnbywater Referencing ticket #570 has changed sprint. 615 1288038146000000 johnbywater Referencing ticket #570 has changed sprint. 616 1284219609000000 johnbywater Referencing ticket #570 has changed sprint. 616 1284225063000000 johnbywater Referencing ticket #570 has changed sprint. 616 1285199168000000 johnbywater Referencing ticket #570 has changed sprint. 616 1285629333000000 johnbywater Referencing ticket #570 has changed sprint. 616 1286786420000000 johnbywater Referencing ticket #570 has changed sprint. 616 1287392123000000 johnbywater Referencing ticket #570 has changed sprint. 616 1287703431000000 johnbywater Referencing ticket #570 has changed sprint. 616 1288038146000000 johnbywater Referencing ticket #570 has changed sprint. 617 1284219609000000 johnbywater Referencing ticket #570 has changed sprint. 617 1284225063000000 johnbywater Referencing ticket #570 has changed sprint. 617 1285199168000000 johnbywater Referencing ticket #570 has changed sprint. 617 1285629333000000 johnbywater Referencing ticket #570 has changed sprint. 617 1286786420000000 johnbywater Referencing ticket #570 has changed sprint. 617 1287392123000000 johnbywater Referencing ticket #570 has changed sprint. 617 1287703431000000 johnbywater Referencing ticket #570 has changed sprint. 617 1288038146000000 johnbywater Referencing ticket #570 has changed sprint. 617 1294408164000000 thejimmyg Duplicate of #569. 618 1284219775000000 johnbywater Referencing ticket #571 has changed sprint. 621 1285348152000000 johnbywater Referencing ticket #568 has changed sprint. 621 1285957164000000 johnbywater Referencing ticket #568 has changed sprint. 621 1286786420000000 johnbywater Referencing ticket #568 has changed sprint. 621 1287392122000000 johnbywater Referencing ticket #568 has changed sprint. 621 1287776309000000 johnbywater Referencing ticket #568 has changed sprint. 622 1285348152000000 johnbywater Referencing ticket #568 has changed sprint. 622 1286786420000000 johnbywater Referencing ticket #568 has changed sprint. 622 1287392122000000 johnbywater Referencing ticket #568 has changed sprint. 623 1285348152000000 johnbywater Referencing ticket #568 has changed sprint. 623 1286786419000000 johnbywater Referencing ticket #568 has changed sprint. 623 1287392122000000 johnbywater Referencing ticket #568 has changed sprint. 626 1284750525000000 johnbywater Moved from sprint 1.2.3 626 1285348177000000 johnbywater Referencing ticket #572 has changed sprint. 628 1285348177000000 johnbywater Referencing ticket #572 has changed sprint. 629 1285348177000000 johnbywater Referencing ticket #572 has changed sprint. 630 1285348178000000 johnbywater Referencing ticket #572 has changed sprint. 632 1285348177000000 johnbywater Referencing ticket #572 has changed sprint. 633 1285348167000000 johnbywater Referencing ticket #572 has changed sprint. 633 1285348177000000 johnbywater Referencing ticket #572 has changed sprint. 634 1285348167000000 johnbywater Referencing ticket #572 has changed sprint. 634 1285348177000000 johnbywater Referencing ticket #572 has changed sprint. 635 1285348167000000 johnbywater Referencing ticket #572 has changed sprint. 635 1285348177000000 johnbywater Referencing ticket #572 has changed sprint. 636 1284689350000000 johnbywater Referencing ticket #579 has changed sprint. 641 1284689330000000 johnbywater Referencing ticket #577 has changed sprint. 644 1284689350000000 johnbywater Referencing ticket #579 has changed sprint. 645 1284750525000000 johnbywater Moved from sprint 1.2.3 645 1285348177000000 johnbywater Referencing ticket #572 has changed sprint. 645 1285957164000000 johnbywater Referencing ticket #572 has changed sprint. 645 1286786420000000 johnbywater Referencing ticket #572 has changed sprint. 645 1287392122000000 johnbywater Referencing ticket #572 has changed sprint. 646 1294413500000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing. 648 1284889656000000 pudo introduced in ckan-authz2 cset:934b30ec84fd 649 1284889939000000 pudo developed in cset:e65d7342ffe4 650 1286789277000000 johnbywater Moved from sprint 1.2.7 650 1287391932000000 pudo included since cset:781ef73e61cc 651 1311169144000000 thejimmyg This has been open for more than 6 months so closing it. It may even have been fixed in John's auth refactor. 651 1311171097000000 dread Yes, John fixed this. Changing to fixed. 652 1284998201000000 pudo Fixed in cset: 688cbd25b3f0 (ckan-authz2) 655 1285252726000000 johnbywater Referencing ticket #654 has changed sprint. 656 1285252726000000 johnbywater Referencing ticket #654 has changed sprint. 657 1285252726000000 johnbywater Referencing ticket #654 has changed sprint. 658 1285252726000000 johnbywater Referencing ticket #655 has changed sprint. 659 1298379892000000 dread "Smoketest scripts exist for exactly this in ckanext. It would be great to have this running on nagios. It is as simple as running: python blackbox/smoke.py -H ckan.net blackbox/ckan.net.profile.json See here for code: https://bitbucket.org/okfn/ckanext/src/default/blackbox" 659 1298424109000000 nils.toedtmann Good idea. Listed this in my nagios ticket http://knowledgeforge.net/okfn/tasks/ticket/600 659 1311183031000000 thejimmyg Since there is already an implementation and already exists as a ticket in the OKFN tasks, I'm marking this as closed. If there is a reason this is still open, please add it to a more appropriate ticket, such as the API version 3 one. 660 1311183115000000 thejimmyg This ticket is more than 6 months old so closing it in line with our ticketing policy. 661 1288271266000000 dread "Friedrich has done the main meat of this in ckanext/blackbox/smoke.py I've extended it to include http basic auth, testing wanted error statuses, checking API calls etc." 662 1286991079000000 dread "I agree we should not have the 'read-only' things like Ratings in the default returned Package Entity. What do you think of having a parameter to be able to get these if you want them though? Do you mean you *can* re-post the *entity* post response? Not sure what you mean by ""An issue for CKAN too.""? In addition to this ticket, what do you think about changing the behaviour of the Package Entity PUT/POST, so that you replace the entire Package, not just the fields you specify? So you don't keep left-over values, just because you didn't specify them as null?" 662 1286993727000000 johnbywater "It might be natural for the locator of the rating for a package to be ""/package/{id}/rating"". I've got not idea what I meant by ""An issue for CKAN too."" I may have intended to log this againt ckanclient. Anyway, it seems to be just a CKAN thing. :-) I think I would like to do this: $data = c.package_register_post({'name': 'example'}) $data['title'] = 'Example' c.package_entity_put($data['id'], $data) and this: $data = c.package_entity_get('example') $data['title'] = 'Example' c.package_entity_put($data['id'], $data) I don't think either work. We could write a test for each. I think this does work: $data = c.package_entity_get('example') $data = c.package_entity_put($data['id'], $data) $data['title'] = 'Old Example' c.package_entity_put($data['id'], $data) Which is inconsistent. The reason is that the data returned by the update operation (""entity put"") isn't given the same treatment as the read and create operations, which adds various read-only values. That's as far as I got. I inferred that one or several of the read-only values, when present in the update request data, cause the update to fail. I'm not sure how it fails. Rather than cutting down the response data, we could make the update call more robust. One fix would pick out from the request package data only the attributes that are permitted. A alternative fix would make what ever is choking on the extra values ignore them. I mean, ids are read-only, but we wouldn't want to reject an update because it has an id in the data. We do need to be careful about loading up the package entity data, but the 'id' is read-only and we aren't going to quible about that being present in the data of an update request (even if it doesn't match the referenced entity). I would rather not support ""replace the entire package"" with especial functionality and documentation. I think the model create/update/delete, where update is ""set attributes"" is sufficient, simple, and fairly optimal. To obliterate all registerd values without deleting, I would get the package entity data, loop over the keys and set the values to '', [] or {} depending on the type, and then PUT the entity. We could write a test for that." 662 1288624137000000 wwaites "Ran into this with RDF export (that then updates the CKAN package with LOD2-compatible extras from the results). Cannot use ckanclient.package_entity_put(ckanclient.package_entity_get(""ckan"")). Is this related to the modifications htat are showing up in the changelog with no apparent package (the package that should be appearing there is ""ckan"" itself)" 662 1288628902000000 dread "Yes we all agree this needs fixing it. I'm tempted by John's 'permissive' suggestion of ignoring these 'read-only' values, but to avoid confusion we should except with a 400 error if the user has changed these values. Read only fields: 'id', 'relationships', 'ratings_average', 'ratings_count', 'ckan_url' Use cases for changes between GET package, PUT package: 1. package unchanged - 200 OK 2. user changes id, ckan_url, relationships, ratings_* expecting that value to change - 400 error. 3. just license changes (but not license_id) - 400 error 4. both license and license_id change in step - 200 OK Does that sound reasonable John and Will?" 662 1297344790000000 rgrp Now part of 'model/validation/forms' meta-ticket #961 so reassigning to Seb. 662 1299755259000000 dread We want this fixed for CLG customer (DGU), so have put in a quick fix into branch 3.1.2 cset:0010a709edf0 (and merged to default) as a stopgap whilst new forms are on their way. 662 1301076443000000 dread Bug: license_id field is assigned the value of the 'license' parameter. 662 1301076463000000 dread license bug fixed in cset:00038ef33c45 663 1294414953000000 thejimmyg This is still an issue. Who is best to look into it? Assigning to me for timebeing. 663 1294660906000000 dread This is my domain, but I suggest you don't assign it to me until it becomes important. 663 1298913603000000 kindly cset:76a77439ecd0 664 1294413696000000 thejimmyg @pudo is this still valid? 664 1300371645000000 kindly "fixed cset:a5f4a49190e2" 665 1294247602000000 thejimmyg Duplicate of #884 665 1297268097000000 thejimmyg The latest version of the DMS collection interface spec says that this should be a manual process. No work to be done. 667 1285925476000000 dread This was done when caching was switched off for a few hours it seems. Still, it is worth investigating this. 667 1294414834000000 thejimmyg "This is something we'll shortly be looking into for DGU work anyway once we switch to the ""thin"" system so don't need a separate ticket for it here too. Also, the upgrade of SQLAlchmey and test performance improvements may also make this less of a problem now anyway. Marking wontfix." 667 1297069121000000 rgrp Actually marking as wontfix! 668 1294412060000000 thejimmyg Assigning this to myself but won't be looking at it any time soon, if anyone else wants to take it on I'd be very grateful. 668 1311176649000000 thejimmyg This ticket is more than 6 months old so marking as invalid in line with our ticketing policy. The API v1 and 2 were completely re-written as part of the logic layer refactor so it is likely this issue is resolved anyway. 669 1286789428000000 johnbywater Moved from sprint 1.2.7 669 1287392123000000 johnbywater Moved from sprint 1.2.8 669 1287776309000000 johnbywater Moved from sprint 1.2.9 669 1288038146000000 johnbywater Moved from sprint 1.3.0 669 1291715237000000 pudo Important sub-task: check in schema.xml from eu4 somewhere. 670 1286789428000000 johnbywater Moved from sprint 1.2.7 670 1287392123000000 johnbywater Moved from sprint 1.2.8 670 1287776309000000 johnbywater Moved from sprint 1.2.9 670 1288038146000000 johnbywater Moved from sprint 1.3.0 671 1286789428000000 johnbywater Moved from sprint 1.2.7 671 1287392123000000 johnbywater Moved from sprint 1.2.8 671 1287776309000000 johnbywater Moved from sprint 1.2.9 671 1288038146000000 johnbywater Moved from sprint 1.3.0 672 1286789428000000 johnbywater Moved from sprint 1.2.7 672 1287392123000000 johnbywater Moved from sprint 1.2.8 672 1287776309000000 johnbywater Moved from sprint 1.2.9 672 1288038146000000 johnbywater Moved from sprint 1.3.0 672 1291715338000000 pudo This is fixed for Solr now, plus a decision not to support postgres facets has been made. 673 1286786420000000 johnbywater Referencing ticket #568 has changed sprint. 674 1286786419000000 johnbywater Referencing ticket #568 has changed sprint. 675 1286786420000000 johnbywater Referencing ticket #568 has changed sprint. 675 1287392122000000 johnbywater Referencing ticket #568 has changed sprint. 675 1287776309000000 johnbywater Referencing ticket #568 has changed sprint. 676 1294413641000000 thejimmyg There was discussion of this on the list a few months ago, was there a conclusion? 676 1295869107000000 rgrp Closing as wontfix. For dev/sysadmin documentation people can follow the link in the CKAN footer to ckan.org where we are consolidating documentation. 677 1286786420000000 johnbywater Moved from sprint 1.2.6 677 1287392123000000 johnbywater Moved from sprint 1.2.8 677 1287589651000000 dread Related to https://trac.dataco.coi.gov.uk/projects/datagov/ticket/645 677 1287745278000000 dread Currently blocked, waiting for exact details of script. 677 1287746221000000 johnbywater Just to say, in cases where something needs to be delivered by somebody else, it can be useful to create a separate task ticket (with undefined remaining time) to indicate what deliverable is being waited on, and in the sprint backlog to drag it above the ticket(s) that depend on the deliverable. That means we don't feel a need to factor unknowable waiting time into our own tasks. It also means impediments are more obviously displayed, so you don't lose track when waiting for a number of things. The two different objectives also well separated so there is no question that somebody is stuck and passing time on a task. I've done this a few times recently and it does help to keep things simple. :-) 677 1287776309000000 johnbywater Moved from sprint 1.2.9 677 1288038147000000 johnbywater Moved from sprint 1.3.0 677 1292587315000000 dread Done a few weeks ago in dgu repo. 678 1286786420000000 johnbywater Moved from sprint 1.2.6 679 1286786420000000 johnbywater Moved from sprint 1.2.6 679 1287392123000000 johnbywater Moved from sprint 1.2.8 679 1287776309000000 johnbywater Moved from sprint 1.2.9 679 1288038147000000 johnbywater Moved from sprint 1.3.0 679 1292587426000000 dread James has started this, but still some things to add. 679 1294166120000000 dread I added some extra bits in cset:1ca7ba29d409. Resource formats disagree between DGU and the FAQ - have sided with DGU for now as it's simpler. I think this ticket is complete now. 681 1286786420000000 johnbywater Moved from sprint 1.2.6 681 1287160548000000 dread Looks like integration with drupal is the problem here. I'm writing tests. 681 1287392123000000 johnbywater Moved from sprint 1.2.8 681 1287776309000000 johnbywater Moved from sprint 1.2.9 681 1288036387000000 rgrp This has been fixed. 681 1288036983000000 dread The code is fixed, but not yet deployed, which requires time. 681 1290507180000000 dread This has been deployed now. 682 1286786420000000 johnbywater Moved from sprint 1.2.6 682 1287392123000000 johnbywater Moved from sprint 1.2.8 682 1287776309000000 johnbywater Moved from sprint 1.2.9 682 1288038147000000 johnbywater Moved from sprint 1.3.0 682 1297358266000000 dread Various improvements to ckanclient to enable this: cset:1bfefd7596d3 and cset:47fd07087547 and installed on buildbot now. 683 1287142289000000 dread Done apart from ultrastable - not in use at moment. 684 1286388346000000 dread Referencing ticket #683 has changed sprint. 685 1286388345000000 dread Referencing ticket #683 has changed sprint. 686 1286388346000000 dread Referencing ticket #683 has changed sprint. 686 1287997047000000 dread Don't need this yet. 689 1287392122000000 johnbywater Referencing ticket #572 has changed sprint. 689 1287776309000000 johnbywater Referencing ticket #572 has changed sprint. 690 1288013610000000 johnbywater This concern was resolved in favour of a converged set of attributes (#711). 691 1294412765000000 thejimmyg "This is more a long term idea, taking ownership for the timebeing but happy for someone else to take this on. " 691 1294412929000000 thejimmyg If we did implement this we'd need to link new sample package to previous sample package of continuous series. So closing #748. 691 1294413010000000 thejimmyg We'd also fold up continuous series in search results behind newest sample package. So closing #749. 691 1294413993000000 thejimmyg "We'd also discuss package relationships ideas with JF so closing #444. " 691 1295610145000000 rgrp Most of package relationships was implemented in ticket:253 and its subtickets. Therefore closing this as duplicate. 692 1294247841000000 thejimmyg CKAN doesn't need to implement this, Drupal does. Incidentally, the initial version is implemented on UAT anyway. 693 1286831155000000 wwaites "Fixed with cset:1e53d160abb6 If expires argument is set, Cache-Control headers get set -- so remove any values, default or otherwise from your configs. ETag gets set to last-modified." 694 1287087801000000 wwaites Installed postgres client from lenny backports on eu3. Can now access without problem, web sites still appear to work (e.g. fr.ckan.net). Note that the backup process runs directly on eu5 now as mentioned by rgrp in email 695 1287391751000000 pudo This is triggered by an issue where having local blinker notifications without asynchronous notifications will break indexing. The issues cause is still unknown, but one possible fix is running CKAN with a queue enabled, synchronous indexing on and no queue consumers attached. Not a real solution, but made possible in cset:a065dbc8041c 695 1287677169000000 ollyc "This is down to blinker not persisting the signal objects. Although the docs claim: ""Every call to signal('name') returns the same signal object, allowing unconnected parts of code (different modules, plugins, anything) to all use the same signal without requiring any code sharing or special imports"" This isn't true unless you maintain a reference to the signal object. To demonstrate this: {{{ >>> import gc >>> from blinker import signal >>> signal_id = id(signal('Package')) >>> gc.collect() 0 >>> id(signal('Package')) == signal_id False }}} The synchronous notifications code connects to signals without storing the signal objects, which are subsequently garbage collected and hence never fire. The async notifications stores references to all signals as a class attribute, so this problem is not seen when async notifications are enabled. " 695 1287677481000000 dread Superb finding Friedrich! Any chance of a quick fix for metastable then? 695 1287677524000000 dread Sorry, I meant Olly! 695 1287748240000000 ollyc "Fix and tests for this are here: * http://bitbucket.org/ollyc/ckan/changeset/48234e36ee61 * http://bitbucket.org/ollyc/ckan/changeset/f1447bbc9d65 * http://bitbucket.org/ollyc/ckan/changeset/1d3cb5378a2f * http://bitbucket.org/ollyc/ckan/changeset/e8409c84683d * http://bitbucket.org/ollyc/ckan/changeset/8e89cbce15c7 * http://bitbucket.org/ollyc/ckan/changeset/d1d63db64585 * http://bitbucket.org/ollyc/ckan/changeset/3d9eb4172f48 " 695 1287766973000000 dread "These changesets do the trick. I've: * merged Ollie's changes into default and metastable branches. * updated ckan.net to latest metastable and reindexed it. * ckan.net now searches on latest packages, including sanskrit example. * test-hmg.ckan.net also updated to latest metastable for testing with UAT. * ckanhmg is to be updated next week and DGU ticket remains for this." 696 1287392123000000 johnbywater Moved from sprint 1.2.8 698 1291742752000000 Stiivi "I see two possible options: Option A: store only mirrors of source files, have file format based plugins for querying files Option B: store mirrors of source files, have plugin based loading scripts into ""common structured format"", have single query module. I would go with option B as it is: - easier to implement - file format based transformations are simpler than file format based queries - more transparent data management process - only one simple query module (see attached ckan-srcmirror.png) The Option B will fit better to the broader data architecture context: http://democracyfarm.org/f/ckan/data_arch.png Concerning API I would suggest to try to be compatible with google spreadsheet API: http://code.google.com/apis/spreadsheets/data/3.0/reference.html " 698 1291832133000000 thejimmyg "Actually we've implemented a first version which doesn't store the data. See this post: http://blog.ckan.org/2010/12/04/open-data-day-announcing-ckan-data-proxy/ You can get data like this: http://1.latest.jsonpdataproxy.appspot.com/?sheet=1&indent=4&url=http://research.dwp.gov.uk/asd/asd4/r1_values.xls" 698 1291851275000000 Stiivi "@thejimmyg: It is neat simple solution. You have suggested a proxy API: ''There will be a new API at ``/api/spreadsheet?callback=jsonpcallback&url=`` '' There are two options: 1. Have public ckan data proxy as stand-alone service: I get package resource URL from CKAN and pass it to proxy 2. Have ckan data API (as ticket title suggests): If I am talking to CKAN, I am getting data from CKAN, I should not care about proxy or anything behind nor I should care about original data source - I care about resource data in a format that I can process (CSV/JSON). For CKAN data API I would suggest something like: {{{ /api/resource_data/RESOURCE_ID?... }}} or more human readable: {{{ /api/resource_data/PACKAGE_NAME/RESOURCE_NUMBER?... }}} This will allow others to get only CKAN resources. Moreover, allowing to get only resource data (not any URL data) would allow us to pre-process resources in the future. First version/implementation: pass each requested resource URL to your proxy service (external, not CKAN related), which determines file by file extension in URL, fail on unknown file or unprocessable file. /api/resource_data/PACKAGE/RESOURCE?output=jsonp&sheet=1... would be redirected to (for example): http://1.latest.jsonpdataproxy.appspot.com/?url=RESOURCE[""URL""]&sheet=1... Second version/implementation: Determine file type in advance and pass to appropriate conversion service when requested If you upload document on scribd or slideshare it gets processed in the background. This can be done in CKAN after any resource change. We do not need to download the file at the moment, however what can be done is: 1. try a converter by URL file extension 2. try a converter by MIME type (content-type header) 3. brute-force try all converters No need to store copies of files, just store determined file type somewhere in the resource record (as mime type). Also, it would be nice if any data conversion service would provide output in both - JSON/CSV. Therefore we would be able to have ""Download CSV"" link directly in CKAN web page for browsing users: /api/resource_data/PACKAGE/RESOURCE?output=csv... " 698 1291858346000000 Stiivi "I have created ""proof of concept"" implementation that will use external data proxy service when accessing: {{{ /api/data/PACKAGE_ID }}} like: {{{ http://127.0.0.1:5000/api/data/069c80f8-8476-452e-bfd4-0a9077666c14 }}} It just works and requires refactoring to match ckan standards. I would need help from soneone who knows ckan internals better." 698 1291859298000000 Stiivi "One more note: it would be good if packages had names/identifiers as well, as referencing internal IDs from outside world is not very good practice - they are quite volatile, mostly in regard to expected objects. PACKAGE/RESOURCE_REFERENCE Possible resource references: - 'default' - reserved keyword for 'the only one resource' if there is only one, or first resource if there are more or the one with flag 'default' - 'latest' - to be able to access 'latest' resource within package (or 'actual' or 'last'?) - alphanumeric identifier (not starting with number) - number - index of resource as human/visitor sees it on page (not the same as ""position"" attribute - as that one might contain gaps or be different (and it is in some cases)), index of resource should be something like: {{{ SELECT package_id, id, url, ROW_NUMBER() OVER (PARTITION BY package_id ORDER BY position) AS index FROM package_resource }}} " 698 1292001709000000 Stiivi "'draft"": https://github.com/Stiivi/ckanext-dataapi requires that the client handles HTTP 302 Redirect correctly." 698 1292239372000000 rgrp " 1. move repo to bitbucket 2. clone james proxy code and modify to make google spreadsheets compatible (add a test ...) 3. update the ckanext to pass on parameters .... 4. Deploy all of this to test.ckan.net 5. Rufus: check redirects with javascript" 698 1292596589000000 Stiivi "Here is the fork for (json) data proxy: https://bitbucket.org/Stiivi/dataproxy I've refactored it and moved transformations into separate modules. For each resource type there should be a module in transform/_transform.py Each module should implement ``transform(flow, url, query)`` and should return a dictionary as a result. Existing modules: * transform/csv_transform - CSV files * transform/xls_transform - Excel XLS files if there is no resource_type module, HTTP 200 Error Resource type not supported is returned. You can override URL file extension or specify type if extension is missing through type= URL option. For example if you have any URL that contains CSV data however the url is just foo.com/data then you can pass: url=http://foo.com/data&type=csv Note: Source refactored/updated in example/dataproxy, being tested by running locally localhost:8000. " 698 1292781368000000 Stiivi pushed parameter passing; change handling of unknown reply type on proxy side: do not raise exception, but reply with 200 Error - unkown reply type, use json/jsonp 698 1293472613000000 anonymous "Data proxy documentation: http://democracyfarm.org/dataproxy/api.html (included in sources) Updated ('s' as in structured) data proxy app: http://sdataproxy.appspot.com " 698 1293649815000000 rgrp "This ticket is complete: * ckanext-dataapi: working /api/data/{resource-id} with tests * https://bitbucket.org/okfn/dataproxy - the dataproxy code running at http://jsonpdataproxy.appspot.com * functioning but needs tests and improvements There a whole bunch of improvements to be done but these will be in ticket:888" 699 1294412194000000 thejimmyg I've just verified that the search pages navigation does not respect the checkboxes. Could you have a look please Friedrich? 699 1296499061000000 pudo fixed in cset:81a0f5538a5b 700 1288562082000000 rgrp "I can verify that groups are getting lost on package preview on ckan.net (steps: 1. click edit, 2. preview 3. look at edit form and group has disappeared). Given this is a bug on a production service it is urgent this gets fixed. @pudo: please can you confirm your expected fix date on this?" 700 1289823108000000 dread This is still holding back releases. @pudo please can you look at it this 703 1288355926000000 johnbywater Referencing ticket #700 has changed sprint. 703 1288356182000000 johnbywater Referencing ticket #700 has changed sprint. 703 1289909096000000 pudo fixed in cset:b23377108b6d 704 1288355926000000 johnbywater Referencing ticket #700 has changed sprint. 704 1288356182000000 johnbywater Referencing ticket #700 has changed sprint. 705 1288356302000000 johnbywater Referencing ticket #701 has changed sprint. 705 1288356319000000 johnbywater Referencing ticket #701 has changed sprint. 706 1288355926000000 johnbywater Referencing ticket #700 has changed sprint. 706 1288356181000000 johnbywater Referencing ticket #700 has changed sprint. 706 1289909028000000 pudo fixed in cset:b23377108b6d 707 1288355926000000 johnbywater Referencing ticket #700 has changed sprint. 707 1288356181000000 johnbywater Referencing ticket #700 has changed sprint. 708 1288356302000000 johnbywater Referencing ticket #701 has changed sprint. 708 1288356319000000 johnbywater Referencing ticket #701 has changed sprint. 711 1288014219000000 johnbywater See also #504. 711 1294232521000000 thejimmyg At the moment DGU has locations such as Wales, England etc. This won't change and won't try to be merged with INSPIRE datasets or bounding boxes. What we do need is a flag for INSPIRE so that the different types of package can be highlighted. 711 1296592589000000 thejimmyg "Our new approach agreed with Rod is to start with a minimum set of package extras and expand them in line with feedback from the user group. We can now show the source GEMINI record as XML (and also a very basic HTML view) for anyone who wants the full detail. No need for this ticket anymore." 712 1288038147000000 johnbywater Moved from sprint 1.3.0 712 1294408364000000 thejimmyg There is no 0.4 metadata spec yet so we can't model it. If there was, the discussion would be on #711 anyway. 713 1288026226000000 johnbywater Referencing ticket #712 has changed sprint. 713 1288038147000000 johnbywater Referencing ticket #712 has changed sprint. 714 1291733788000000 dread This was implemented with DGU ticket 614 715 1289815657000000 johnbywater Moved from sprint 1.3.3 715 1297796784000000 pudo fixed in cset:69c4210f635a 718 1288195538000000 johnbywater Referencing ticket #716 has changed sprint. 718 1288196547000000 johnbywater Referencing ticket #716 has changed sprint. 718 1288197569000000 johnbywater Referencing ticket #716 has changed sprint. 718 1288459823000000 pudo "fixed http://bitbucket.org/okfn/ckanextiati/changeset/4bf10d3f26e7" 719 1288195538000000 johnbywater Referencing ticket #716 has changed sprint. 719 1288196547000000 johnbywater Referencing ticket #716 has changed sprint. 719 1288197569000000 johnbywater Referencing ticket #716 has changed sprint. 720 1288195538000000 johnbywater Referencing ticket #716 has changed sprint. 720 1288196547000000 johnbywater Referencing ticket #716 has changed sprint. 720 1288197569000000 johnbywater Referencing ticket #716 has changed sprint. 720 1288459344000000 pudo Customer has stated they do not want this in the current iteration. 721 1288195538000000 johnbywater Referencing ticket #716 has changed sprint. 721 1288196547000000 johnbywater Referencing ticket #716 has changed sprint. 721 1288197569000000 johnbywater Referencing ticket #716 has changed sprint. 721 1288459534000000 pudo "Solved using javascript to toggle hide/show preview for each resource: http://bitbucket.org/okfn/ckanextiati/changeset/942cec028cf5" 722 1288195538000000 johnbywater Referencing ticket #716 has changed sprint. 722 1288196547000000 johnbywater Referencing ticket #716 has changed sprint. 722 1288197569000000 johnbywater Referencing ticket #716 has changed sprint. 722 1288459703000000 pudo "fixed: http://bitbucket.org/okfn/ckanextiati/changeset/fb166fde9d3e" 723 1288195538000000 johnbywater Referencing ticket #716 has changed sprint. 723 1288196547000000 johnbywater Referencing ticket #716 has changed sprint. 723 1288197569000000 johnbywater Referencing ticket #716 has changed sprint. 723 1288459610000000 pudo Fixed in cset:78b23f3f6aaf 725 1288195549000000 johnbywater Referencing ticket #724 has changed sprint. 725 1288196716000000 johnbywater Referencing ticket #724 has changed sprint. 725 1288197657000000 johnbywater Referencing ticket #724 has changed sprint. 725 1289295966000000 pudo fixed in cset:ff4264eb609c (http://bitbucket.org/okfn/iati/changeset/ff4264eb609c) 726 1288195549000000 johnbywater Referencing ticket #724 has changed sprint. 726 1288196716000000 johnbywater Referencing ticket #724 has changed sprint. 726 1288197657000000 johnbywater Referencing ticket #724 has changed sprint. 727 1288195549000000 johnbywater Referencing ticket #724 has changed sprint. 727 1288196716000000 johnbywater Referencing ticket #724 has changed sprint. 727 1288197657000000 johnbywater Referencing ticket #724 has changed sprint. 728 1294232752000000 thejimmyg WAF records may always need to be re-harvested to see if they have changed. Does CSW provide any functionality that allows us to see what has changed? 728 1304934653000000 wwaites filters not properly supported by e.g. geonetwork. so we defer. 728 1304936482000000 thejimmyg Actually, these are a different type of filter. We need to send a filter to only get modified documents, to save processing our end. 728 1310124784000000 thejimmyg Isn't this already completed? Putting in the backlog but I think it is already implemented. Adria, could you please update accordingly? 729 1288012854000000 rgrp Closed in cset:a9006fd56450 730 1291139609000000 rgrp Duplicate of ticket:402 731 1288015048000000 johnbywater Are the tasks for this user story outstading, or can we close them now? 732 1287776309000000 johnbywater Moved from sprint 1.2.9 732 1288021197000000 dread Couldn't reproduce 733 1287776309000000 johnbywater Moved from sprint 1.2.9 734 1288038147000000 johnbywater Moved from sprint 1.3.0 734 1292587603000000 dread Found problems and ticketed: #872 #873 735 1292957248000000 dread "I have been through looking for package names with a trailing underscore checking if they should indeed be separate package from those without. #872 and #873 cover creation of the duplicates in the first place." 736 1294409747000000 thejimmyg "The tests we write for this would need to: * Submit harvest source form with valid CSW location * Submit harvest source form with invalid CSW location We can now close #790 and #791. Note that discussion on end-to-end testing is now at #784." 736 1296592767000000 thejimmyg We also need to check WMS URLs added to package resources to check they are genuine WMSs. 736 1304937755000000 thejimmyg "We now check WMS links. The CSW link checking could be more robust. New proposal is: Check that a URL with a / is treated the same way as one without. Treated as the same if same port, scheme, domain and path (ie query string and fragment are ignored, port is normalised away)" 736 1304963598000000 amercader "See https://bitbucket.org/okfn/ckanext-harvest/changeset/a7b6d8c0dde7 https://bitbucket.org/okfn/ckanext-harvest/changeset/40de12fada74" 737 1319730303000000 dread "Something like this one on github: {{{ # This is an

tag ## This is an

tag ###### This is an

tag Text styles *This text will be italic* _This will also be italic_ **This text will be bold** __This will also be bold__ *You **can** combine them* Lists Unordered * Item 1 * Item 2 * Item 2a * Item 2b Ordered 1. Item 1 2. Item 2 3. Item 3 * Item 3a * Item 3b Miscellaneous Images ![GitHub Logo](/images/logo.png) Format: ![Alt Text](url) Links http://github.com - automatic! [GitHub](http://github.com) Blockquotes As Kanye West said: > We're living the future so > the present is our past. }}}" 738 1294408864000000 thejimmyg This needs more discussion. It is a separate problem from deleting a harvest source from the admin interface which is dealt with in #884. 738 1294409432000000 thejimmyg "Note from John Bywater on duplicate ticket #783 says: ""We need a condition for 'disappears'. Asked AW about this (prob. in July) but should chase for an answer.""" 738 1297685069000000 thejimmyg "It may be handy to have a history of harvested documents and also be able to delete documents but still get their revisions. The package comparison code will need to look at these revisions." 738 1299761436000000 thejimmyg This is now complete and on UAT. 739 1294247758000000 thejimmyg This needs to be implemented via an API call that Drupal can use 739 1296592842000000 thejimmyg Implemented today. May want a better implementation later. 740 1294408931000000 thejimmyg This will have to be done via the API though for DGU/UKLP. Should probably have it on the package interface in main CKAN too though. 740 1296592889000000 thejimmyg Duplicate of #739. 741 1291988087000000 rgrp Done in cset:49d5bd0a6a99 and cset:68522feabfeb among others. Documentation of progress on http://ckan.okfnpad.org/plugins 742 1288038147000000 johnbywater Moved from sprint 1.3.0 743 1294412807000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing. 744 1294412813000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing. 745 1294412820000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing. 746 1294412827000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing. 747 1294412834000000 thejimmyg This is a Drupal requirement rather than a CKAN one for the timebeing. 748 1294412916000000 thejimmyg Merging with #676. 748 1294412976000000 thejimmyg Sorry, that should have been #691. 749 1294412986000000 thejimmyg Merge with #691 750 1289483616000000 johnbywater Referencing ticket #728 has changed sprint. 750 1294408970000000 thejimmyg How would this be done? Is it part of the CSW spec? 750 1296592940000000 thejimmyg Duplicate of #728 752 1294408472000000 thejimmyg Duplicate of #712. See #711 now. 753 1289209646000000 johnbywater Moved from sprint 1.3.2 753 1289815657000000 johnbywater Moved from sprint 1.3.3 753 1294247922000000 thejimmyg Not an explicit requirement yet so closing ticket. Will re-open if needed. 754 1289209646000000 johnbywater Referencing ticket #753 has changed sprint. 754 1289815657000000 johnbywater Referencing ticket #753 has changed sprint. 754 1294408726000000 thejimmyg "We may not need to do any filtering in practice but if we did, the implementation would have these stages: * Add filter attribute to harvest source entity * Add filter field to harvest source form * Construct and send filtered CSW GetRecords request Thus these tickets can now be marked as duplicated: #755 #756 #759." 754 1294409145000000 thejimmyg "Also: * Change API documentation to indicate harvest source entity has filter attribute * Change ""CSW Get Records"" request to accept and use given CSW filter Now #758 and #760 can be closed too." 755 1288041215000000 johnbywater Referencing ticket #754 has changed sprint. 755 1288264918000000 johnbywater Referencing ticket #754 has changed sprint. 755 1289209646000000 johnbywater Referencing ticket #753 has changed sprint. 755 1289815657000000 johnbywater Referencing ticket #753 has changed sprint. 755 1294408632000000 thejimmyg Duplicate of #754 756 1288041215000000 johnbywater Referencing ticket #754 has changed sprint. 756 1288264917000000 johnbywater Referencing ticket #754 has changed sprint. 756 1289209646000000 johnbywater Referencing ticket #753 has changed sprint. 756 1289815657000000 johnbywater Referencing ticket #753 has changed sprint. 756 1294408642000000 thejimmyg Duplicate of #754 757 1288041215000000 johnbywater Referencing ticket #754 has changed sprint. 757 1288264918000000 johnbywater Referencing ticket #754 has changed sprint. 757 1289209646000000 johnbywater Referencing ticket #753 has changed sprint. 757 1289815657000000 johnbywater Referencing ticket #753 has changed sprint. 757 1294233016000000 thejimmyg Don't fully understand this ticket. Will return to it. 757 1296593448000000 thejimmyg Duplicate of #754. If we added filtering we'd need to write a migration script anyway. 758 1288041215000000 johnbywater Referencing ticket #754 has changed sprint. 758 1288264917000000 johnbywater Referencing ticket #754 has changed sprint. 758 1289209646000000 johnbywater Referencing ticket #753 has changed sprint. 758 1289815657000000 johnbywater Referencing ticket #753 has changed sprint. 758 1294409053000000 thejimmyg Also a duplicate of #754. 759 1289209646000000 johnbywater Referencing ticket #753 has changed sprint. 759 1289815657000000 johnbywater Referencing ticket #753 has changed sprint. 759 1294408652000000 thejimmyg Duplicate of #754 760 1288041238000000 johnbywater Referencing ticket #759 has changed sprint. 760 1288265044000000 johnbywater Referencing ticket #759 has changed sprint. 760 1289209646000000 johnbywater Referencing ticket #759 has changed sprint. 760 1289815657000000 johnbywater Referencing ticket #753 has changed sprint. 760 1294409111000000 thejimmyg Duplicate of #754 763 1288606918000000 dread Referencing ticket #441 has changed sprint. 763 1292775248000000 thejimmyg "Notes from my discussion with David a while ago: Following irc discussion, it looks like read-only mode is simply achieved by Apache config: * 503 for PUT/POST operations - stops writes * 503 for GETs to URIs containing: /edit, /create, /new, /authz - stops providing forms that lead to a write * setenv CKAN_READONLY=""Undergoing maintenance 12.00 UTC for one hour"" which can be picked up by CKAN to be displayed as we see fit in the future. Friedrich's current IATI sprint may link into this and grey out edit links etc." 764 1289209648000000 johnbywater Moved from sprint 1.3.2 764 1289815657000000 johnbywater Moved from sprint 1.3.3 764 1310133334000000 shevski will be done as part of #1094 765 1288606918000000 dread Referencing ticket #441 has changed sprint. 766 1289815658000000 johnbywater Moved from sprint 1.3.3 766 1294248066000000 thejimmyg I haven't heard this mentioned yet, but yes, let's try to implement it if possible. Appears to expose a CSW interface? http://webhelp.esri.com/geoportal_extension/9.3.1/index.htm#ext_csw_clnts.htm 766 1296593519000000 thejimmyg This is a duplicate of #767 767 1289209649000000 johnbywater Moved from sprint 1.3.2 767 1289815658000000 johnbywater Referencing ticket #766 has changed sprint. 767 1294409336000000 thejimmyg "Our CSW code should be able to cope with ArcGIS too though surely? Worth testing though. To do this we'd need to: * Obtain example ArcGIS CSW server * Write a test Now we can close #768 and #769." 767 1296593504000000 thejimmyg "I haven't heard this mentioned yet, but yes, let's try to implement it if possible. Appears to expose a CSW interface? http://webhelp.esri.com/geoportal_extension/9.3.1/index.htm#ext_csw_clnts.htm " 767 1304936413000000 thejimmyg We are now live and have used a variety of servers in testing. 768 1288190630000000 johnbywater Referencing ticket #767 has changed sprint. 768 1289209649000000 johnbywater Referencing ticket #767 has changed sprint. 768 1289815658000000 johnbywater Referencing ticket #766 has changed sprint. 768 1294409308000000 thejimmyg Duplicate of #767 769 1288190630000000 johnbywater Referencing ticket #767 has changed sprint. 769 1289209649000000 johnbywater Referencing ticket #767 has changed sprint. 769 1289815658000000 johnbywater Referencing ticket #766 has changed sprint. 769 1294409318000000 thejimmyg Duplicate of #767 770 1289209650000000 johnbywater Moved from sprint 1.3.2 770 1289815658000000 johnbywater Moved from sprint 1.3.3 770 1294413390000000 thejimmyg "Running CLI harvester command without arguments or with --help would return help for the harvester (currently crashes). We can therefore close #772, #773 and #774" 770 1296593925000000 thejimmyg I don't care about this. It is too trivial. If I get around to fixing it they great but I don't need it logged as a ticket. 771 1288190596000000 johnbywater Referencing ticket #770 has changed sprint. 771 1288190682000000 johnbywater Referencing ticket #770 has changed sprint. 771 1288190711000000 johnbywater Referencing ticket #770 has changed sprint. 771 1289209650000000 johnbywater Referencing ticket #770 has changed sprint. 771 1289815658000000 johnbywater Referencing ticket #770 has changed sprint. 771 1294413256000000 thejimmyg "This works for me. {{{ paster help }}} and {{{ paster --help }}} both show help text for me. " 772 1288190596000000 johnbywater Referencing ticket #770 has changed sprint. 772 1288190682000000 johnbywater Referencing ticket #770 has changed sprint. 772 1288190711000000 johnbywater Referencing ticket #770 has changed sprint. 772 1289209650000000 johnbywater Referencing ticket #770 has changed sprint. 772 1289815658000000 johnbywater Referencing ticket #770 has changed sprint. 772 1294413313000000 thejimmyg Merging with #770 773 1288190596000000 johnbywater Referencing ticket #770 has changed sprint. 773 1288190682000000 johnbywater Referencing ticket #770 has changed sprint. 773 1288190711000000 johnbywater Referencing ticket #770 has changed sprint. 773 1289209650000000 johnbywater Referencing ticket #770 has changed sprint. 773 1289815658000000 johnbywater Referencing ticket #770 has changed sprint. 773 1294413400000000 thejimmyg Merging with #770 774 1288190596000000 johnbywater Referencing ticket #770 has changed sprint. 774 1288190682000000 johnbywater Referencing ticket #772 has changed sprint. 774 1288190711000000 johnbywater Referencing ticket #772 has changed sprint. 774 1289209650000000 johnbywater Referencing ticket #772 has changed sprint. 774 1289815658000000 johnbywater Referencing ticket #772 has changed sprint. 774 1294413412000000 thejimmyg Merging with #770 775 1295260144000000 pudo fixed in cset:98c9997e80e8 776 1297066840000000 rgrp Moved to https://bitbucket.org/okfn/vdm/issue/3/avoid-generating-vdm-warnings 777 1289209649000000 johnbywater Referencing ticket #764 has changed sprint. 777 1289815658000000 johnbywater Referencing ticket #764 has changed sprint. 777 1292586843000000 dread James did this a while ago. 778 1289209652000000 johnbywater Moved from sprint 1.3.2 778 1289815658000000 johnbywater Moved from sprint 1.3.3 778 1294416567000000 thejimmyg This is now done. 779 1289209652000000 johnbywater Moved from sprint 1.3.2 779 1289815658000000 johnbywater Moved from sprint 1.3.3 779 1294416581000000 thejimmyg This is now done. 780 1289209653000000 johnbywater Moved from sprint 1.3.2 780 1289815658000000 johnbywater Moved from sprint 1.3.3 780 1294416608000000 thejimmyg This is now done. 781 1289209653000000 johnbywater Moved from sprint 1.3.2 781 1289815658000000 johnbywater Moved from sprint 1.3.3 781 1294415081000000 thejimmyg This is a duplicate of #812 so closing. 782 1289218497000000 pudo "This has existed for quite a while, but misses documentation. Has been documented in cset:db18a16a5b4d " 783 1294409399000000 thejimmyg Exact duplicate of #738. 784 1289815658000000 johnbywater Moved from sprint 1.3.3 784 1294410089000000 thejimmyg "In order to have end to end testing for UKLII we need to: * Review document from PP with the different things we need to test * Set up a GeoNetwork instance with sample documents to be tested We may also want to test ArcGIS, see #767" 784 1304936251000000 thejimmyg The UKLP test manager is looking at a new test proposal, we have had approval to go live without this, we can re-open later if necessary. 786 1289815658000000 johnbywater Moved from sprint 1.3.3 786 1294410004000000 thejimmyg Duplicate of #884. 787 1289295207000000 pudo "Alternative: * The Drupal system will expose a resource at data.gov.uk/profile which wil contain the reqesting user's nickname, fullname, email etc. in JSON (or XML) form. * The Drupal system will also run an OAuth server (and provide /request_token, /access_token, /authorize). * CKAN's login form will be rewired to initiate a client OAuth request on the /profile resource. * Recognizing CKAN's callback URL at catalogue.data.gov.uk, Drupal will automatically grant the request if a user is signed in. * Upon return, a user is created, optionally using the OAuth access token as the users API key, thus making it known both to Drupal and CKAN. * The authorizer will be extended to check access to the /profile resource for OAuth accounts (slow but safe). Advantages: * Well-known protocol, does not depend on cookies (which are strange and never behave as defined, or even the same in multiple browsers) * Python code is available: http://oauth.googlecode.com/svn/code/python/oauth/example * Drupal seems to have very complete support: http://drupal.org/node/296205 * Can be fully implemented as a plugin, using an OAuthClientController for callback and adding hooks to the Authorizer (perhaps need to reconfigure who.ini) " 787 1289815658000000 johnbywater Moved from sprint 1.3.3 787 1303118054000000 thejimmyg The AuthAPI now exists as an IMiddleware plugin, we really need the permission system moved into CKAN before it is useful though and this depends on a refactor of the Auth system. See #1094 787 1315821118000000 thejimmyg "The joint authentication was implemented a long time ago and is deployed on catalogue.data.gov.uk. We'll build the authorisation layer in ticket #1326 so marking this as fixed. " 788 1289815658000000 johnbywater Referencing ticket #786 has changed sprint. 788 1294410007000000 thejimmyg Duplicate of #884. 789 1294248289000000 thejimmyg Will, this means that we just get given a URL to harvest from and we need to determine whether it is a CSW or a WAF. Can you look at this please? The code may already do it, I'm not sure. 789 1296593257000000 thejimmyg Covered by #736 now. 790 1289815658000000 johnbywater Moved from sprint 1.3.3 790 1294409709000000 thejimmyg Duplicate of #736. 791 1294409723000000 thejimmyg Duplicate of #736 793 1289815658000000 johnbywater Moved from sprint 1.3.3 793 1294409836000000 thejimmyg "Anyone else know what this means? I can't find changeset aa9aa32e00a9 in the repository so marking as invalid." 794 1289815658000000 johnbywater Moved from sprint 1.3.3 794 1294248216000000 thejimmyg This work is now underway in the underlying DGU implementation. 794 1297686491000000 thejimmyg Actually, for the timebeing we will match but not do anything with that matched information, until there is a clear use case. Publisher is simply the publisher for which the source was registered. Closing this ticket. 794 1300877517000000 thejimmyg I think we need to look at this again in relation to scotland. 795 1296593361000000 thejimmyg This is a duplicate of #794 because if we needed to do this it would be because we need to support harvest sources registered on behalf of publishers by agents, 796 1291640040000000 rgrp We no longer have an alpha page due to refactor of search and usage of search for list view (see ticket:847 and subtickets). 797 1289402982000000 rgrp Fixed yesterday night in cset:5b522c762e71 798 1289815658000000 johnbywater Moved from sprint 1.3.3 798 1294916973000000 dread This was done a while ago. 799 1294232675000000 thejimmyg "How can we tell a WAF document has changed, we simply need to re-harvest it to see surely? Moving the issue to ticket #728 to be dealt with together." 800 1289483682000000 johnbywater Referencing ticket #799 has changed sprint. 800 1294245610000000 thejimmyg Discussion for this ticket is now at #728 801 1289483616000000 johnbywater Referencing ticket #728 has changed sprint. 801 1289483682000000 johnbywater Referencing ticket #799 has changed sprint. 801 1294233386000000 thejimmyg Perhaps this should be implemented differently so that each harvest attempt creates its own database entry with a timestamp and a status attached. We'll need to move to this when we move to a queue based system. 801 1296593735000000 thejimmyg I don't see why we'd really need to do this. Under the current system a job would be created just before the harvesting would be run so the creation date of the job is (give or take a few seconds) the date of the harvesting. If we moved to a queue system this would be done better anyway. 801 1297686706000000 thejimmyg We do have a requirement for this now. The job model has changed so that it is hidden from the user. We therefore want to know the timestamp the job started and the timestamp it finished. We'll therefore need migrations adding too. 801 1300196714000000 thejimmyg This will now be solved as part of the larger harvesting refactor. See #921. 802 1294233294000000 thejimmyg Merging with #801 803 1289489024000000 dread "I'm not sure a template is needed for the test script - surely repeated stuff can just be factored out? Creating the model dump file: * 'hg up' to ckan as it was at the last model migration * 'paster db clean', 'paster db init', 'paster create-test-data' * 'pgdump' * 'hg up default' * Check in the dumped file Do we really want a script to mess with our repo like this? Isn't it just safer to do manually with good instructions?" 803 1314031451000000 dread We now use nose to test migrations instead. 804 1294233156000000 thejimmyg This is the document I drafted after discussions with each party which was approved by JF. 805 1298379084000000 dread "Migration tests added to buildbot using kindly's new nose option #965. Also removed legacy system of migration testing in: ckan/migration/tests and updated docs. cset:643673c7db3e" 806 1290071159000000 dread This is done now I think? Please list changesets, time taken vs estimate and close ticket. 806 1291128886000000 rgrp Duplicate of ticket:827 806 1291134978000000 dread This is not a duplicate of #827 806 1294916284000000 dread This looks finished 806 1324034356000000 dread This was introduced in CKAN 1.3 807 1297075372000000 rgrp wontfix as with traffic we have this is not a massive issue and these are very minor items (we can reopen later if this comes up again). 808 1297783658000000 pudo implemented in cset:8200247e74e9 809 1297075561000000 thejimmyg This is something to think about more in the future. 810 1297679091000000 pudo At the moment, this crashes the groups field for some mysterious reason. Since this is going to be redundant with the new forms and the ticket has a low priority, I'm bumping this back 2 weeks. 810 1300093797000000 rgrp Moving back to backlog for v1.4 as should be dependent on forms overhaul and seems to be problematic (and not that urgent). 810 1310128477000000 thejimmyg This ticket is over 6 months old so closing. 811 1294415158000000 thejimmyg See also related extras field tickets #811 and #893 812 1294658293000000 dread Just added stuff from duplicate #781 813 1294411047000000 thejimmyg "This is now done. The text reads ""Add a Package"" in the main navigation. Thanks!" 814 1294411109000000 thejimmyg "This is now done. It's the ""About"" link in the main application. Thanks!" 815 1294410951000000 thejimmyg This is a duplicate of #234. Please have a look at existing tickets or re-open closed ones rather than adding new ones so that we can keep the old discussions. I've added your point to #234. Thanks! 816 1311171466000000 thejimmyg "John, could you have a look at this one please. We'd need an entry in the logic layer for it and then some autocomplete JavaScript on the format field. Can we standardise on jQueryUI going forward. http://jqueryui.com/demos/autocomplete/" 816 1312192499000000 rgrp "What about idea of 'model' refactor for resource, specifically to have: * format: human created format string with possible nesting e.g. zip:csv * kind: file | api | example | service * mimetype: standard mimetype (e.g. for zipped csv would be application/zip) * mimetype-inner: mimetype of innermost object (so for example would be text/csv) See http://lists.okfn.org/pipermail/ckan-discuss/2011-April/001139.html" 816 1312212782000000 dread "I like rgrp's idea to change the model. But since it's complicated it would be good to have a UI. e.g. You might make format field simply a drop-down with only some common options: 'CSV, Excel, Zip, Other'. If you go for Other, then a more substantial UI appears, to cover more filetypes, APIs, services and a text box for Other. And if you select 'Zip' then another set of options appears to say what is in the zip. etc.etc. Another approach would be to have a background process downloading the data file and filling this in for you." 816 1312295371000000 johnglover "I have implemented the resource autocomplete so am closing this for now. I agree that more information on the resource would be a good addition. It seems a bit of a different issue to just adding autocomplete though, with potentially far more changes to the codebase, so maybe it would be better discussed in a separate ticket/CREP. However, any new fields would presumably want to either be constrained or have an autocomplete and so can reuse the work done on this feature. Cheers, John" 816 1319804698000000 dread John did this in cset:1697dfa2552c for release 1.5 816 1319812324000000 dread And branch feature-816-format-autocomplete 817 1289997628000000 cygri "I'm not allowed to create attachments for some reason. Here's the screenshot: [[Image(http://richard.cyganiak.de/2010/ckan/ckan-resources-table-proposal.png)]] http://richard.cyganiak.de/2010/ckan/ckan-resources-table-proposal.png" 817 1297073724000000 thejimmyg This appears to be fixed now. 818 1305559442000000 dread 1.4 complete 819 1323171435000000 thejimmyg This issue is fixed. Further improvements to the UX of the autocomplete will be dealt with in other tickets. 820 1311325226000000 dread This has been fixed 821 1298486642000000 dread "Investigating several of these packages, it works for me (and David Raznick). For example ni_013_migrants_english_language_skills_and_knowledge, one resource is seen created in the diffs, is displayed in CKAN, in the API and in the dumps. Yet looking at the dump from 17/11/10 when this ticket was created, the resource didn't have a URI, which by the current model is a requirement, so it suggests the data was fine underneath, but it had problems displaying this field, and is now fixed." 822 1290506354000000 dread "Fixed in cset:045cab10aa1d Thanks for the patch!" 823 1290506116000000 anonymous Fixed in cset:3845a501ed5f 824 1290595559000000 dread Done on metastable for 1.2 release. 825 1290624449000000 dread Added plenty of info in cset:627249a1c102 826 1297414412000000 dread "It's great that the extras are added in-line with other Resource properties (as opposed to package extras which are a dict not off 'package' but off 'package.extras'). However, the resource extra field keys are defined in config option ""ckan.extra_resource_fields"". This config option should be removed - extras need to be entirely flexible for our purposes. (In the next ticket we should make it possible to add/remove both keys and values from the Web UI or API.) It would also be good to tidy things in this direction: http://wiki.okfn.org/Coding_Standards I've merged from default to the branch enhancment_826_resource_extra_fields ready for you." 826 1297415095000000 pudo "I want to strongly support david in his call for fully flexible extras: one of my use cases for them is to store the various bits of fallout from an archiving process, such as: * last-status * last-crawled * last-etag * last-expires * last-md5 * failure-count * fall-back-url These are things needed to really archive the data well, but they have nothing to do with CKAN core ops. Essentially, the archiver is a seperate concern and it need not appear in the CKAN config." 826 1297416879000000 kindly "There is nothing to stop anyone from putting any extra attributes in the extra_info field dict. So any have the flexibility you need. The config option is to add some fields that act in exactly the same way as python attributes, having the same semantics as them. i.e if you have an extra field called alturl, you can do obj.alturl = 'fdsffs'. This is the best of both worlds as far as I can tell." 826 1297417900000000 kindly "I forgot to mention that he main advantage of the fixed fields, is that we can make them properly searchable i.e the values searchable. This currently does not work for package extra values as they are jsons. I have added this searchability for the sql backend." 826 1297421022000000 dread Ah - I understand. I'd like it if you could make all the extra fields work as attributes and be searchable - is that possible? 826 1297423342000000 kindly "This would be too much of a hack. You do not want users overwriting any attributes on the object. If they called the attribute ""__init__"" it would write over the actual __init__. " 826 1297429910000000 dread "Merged into default in cset:613d7bd5fc96. Future tickets in this area include: * #978 Full edit in Web UI * #979 Edit Resource extras in the API" 826 1306766057000000 dread This went into ckan release 1.3.2. 827 1290766239000000 rgrp Fixed in cset:63a0c0223da4 828 1290696586000000 rgrp Closed in cset:66a3efe16ed4 829 1290697386000000 dread Some work towards this in cset:066b1b553595. No exceptions and test skip in place. Looks like formalchemy changed its interface. 829 1303118864000000 thejimmyg Is this still relevant? 829 1303122056000000 dread Rufus originally specified it. Reassigning to him to decide whether we still want it or not. 829 1303838115000000 rgrp This is a wontfix for the present. 830 1311180263000000 thejimmyg We can now implement themes via extensions or change them via config options. See https://bitbucket.org/okfn/ckanext-exampletheme/ for an example. 831 1290760864000000 rgrp Done over the last week (finally fixed weird trac no __dict__ bug today and enabled user accounts on wednesday). 832 1296334980000000 rgrp Done (~4w ago). See https://bitbucket.org/okfn/ckanext-stats and remove from core cset:311313e4afdb. 833 1295263447000000 rgrp Have repo https://bitbucket.org/okfn/ckanext-admin 833 1298889104000000 rgrp In progress now (sysadmin view and update nearly done). 833 1302276855000000 rgrp Authz subsystem complete. 833 1303236364000000 rgrp Two main tickets done so closing. 834 1291633657000000 rgrp "Alexander: Documentation is here: http://packages.python.org/ckan/api/version2.html#search-api Those queries should work but I do suggest you use the stable version of CKAN which is v1.2 and we suggest you use that. If you still have problems please reopen. " 835 1291644820000000 pudo done in cset:d41b77099954 836 1291851041000000 rgrp Fixed in cset:2f6d54341b47 and branch feature-286-siteurl 836 1302882808000000 dread This has been a (minor) issue since release v1.3 838 1291299716000000 memespring Package page redesign: http://ckan.org/ticket/839 838 1291636351000000 memespring Merge css files http://ckan.org/ticket/846 838 1291721955000000 memespring Show welcome message: http://ckan.org/ticket/850 838 1291729819000000 memespring Prompt users to enter missing info - http://ckan.org/ticket/863 838 1291736461000000 memespring Search results changes: http://ckan.org/ticket/864 838 1291812318000000 memespring Add download formats to search results (http://ckan.org/ticket/866) 838 1295259773000000 rgrp I'm going to close this ticket as almost everything done and remaining ticket can be done on its own. 839 1291736541000000 memespring Done with the exception of the discuss/comments page. Plugin won't install on my setup. Fredrich looking into it. 840 1291325251000000 dread "dread: ] That's the proxy_cache decorator which sets a Beaker expiry time only. ] Whether that is on or off, you still end up run the ] ""etag_cache(cache_key)"" command in the package controller won't you? ] And won't that will either insert the Etags header or abort 304 for a ] repeat? I must admit I've not tried it, asking purely with the code in ] front of me and welcome you pointing out where I've gone wrong. ww: Right. So those etag calls predate the cache decorators and should probably be either moved up into the decorator (e.g. use @ckan_cache instead of @proxy_cache) or wrapped in a check for the config parameter. (And @ckan_cache could be changed to use pylons' etag_cache function rather than just setting the header...)" 840 1302694123000000 dread Basic on/off switch added, tested & documented in cset:0da189c9630e on default. 841 1300364333000000 thejimmyg See #995 842 1296468313000000 rgrp Change to awaitingtriage as definitely not critical. 842 1303474131000000 thejimmyg "As a user I come to a package: Have a todo count at that top that takes you down to the todo list (which may say nothing todo) At the bottom is a section of the package display titled ""ToDo"" where I see a list of all toDos for the package most recent at the top If I am logged in see a form for ""Add to do"" at the top of the todo section and can add one straight away I see a ""now resolved"" button next to each which goes green when you hover. When clicked the todo fades away. Otherwise I see a button that says ""login to add todo"" expands out the form The form One of the fields is category -> autocomplete the category (not constrained) Add a description Submit, the todo gets added via AJAX to the list at the top as the most recent todo Model: todo id package_id todo_category_id (required) description (required) date=NOW() resolved=False todo_category id name Prepopulate with: broken-resource-link, no-author, bad-format " 842 1303474228000000 thejimmyg " > Otherwise I see a button that says ""login to add todo"" > > expands out the form > Actually rather than expanding the form, you will go away to the login page and come back to see the expanded form (question: how does this redirect you back to the bottom?)" 843 1291652696000000 pudo The real name is also requested via AX/SReg extensions in OpenID so for new users who are not using google, this should usually be filled in automatically. 843 1319710380000000 dread " * ""we should show the openid as well to distinguish between users with the same name."" - when ""Full name"" is not distinguishable, maybe best to display the unique 'name' field as a hover-over. * ""on account creation, the user should be redirected to their personal details page to encourage them to fill in a human readable name."" - yes you always got take to the personal details page. We should use a flash message at this point if they have not filled in the ""Full name"" field to suggest they click edit and do this. * ""List is to long"" - this has been addressed - see http://thedatahub.org/user" 843 1319721601000000 dread "Ok I misunderstood this ticket. This is referring to adding a user in e.g. http://thedatahub.org/group/authz/energy-data This UI seems to be updated. You start typing the name, full name or open id of the person and it has a dropdown that autocompletes. This seems to be sufficient for Will's points 1 and 3. Would be still good to have a flash message on account creation to encourage people to add personal their Full name. This is similar to #1413 so I'll close this ticket and add it there." 844 1296340486000000 rgrp Looking at DNS this apparently has been fixed. 845 1291723492000000 dread This was completed on the feature-845-required-fields branch and merged into default in cset:3b5635ffaa7d 846 1291715212000000 rgrp Reopening and fix used illegal css ;) -- it is illegal to have an @import statement (as we do with extras.css) after normal css statements. We therefore need to move the core css back out into a ckan.css (or similar file). 846 1291719074000000 memespring It's been done on a branch and pushed to bitbucket: http://bitbucket.org/memespring/ckan/src/9e74c40ff073/ckan/public/css/style.css 847 1291639903000000 rgrp Done my best to pull all the solr related tickets and work together in one place. 847 1292844627000000 rgrp Still have ticket:669 and ticket:874 to do (431 probably won't be done for a while). 847 1295259902000000 pudo This is now functional, minus the index testing tool. 849 1291715179000000 pudo == #846 851 1292860957000000 wwaites " * link checker above uses the queue. queue not running generally * quickest way forward is just to put the curate tool in a cron job and make a suitable rule. shall do this soonest" 851 1292892515000000 wwaites "Ready to put into cron job. cf: * http://groups.google.com/group/fuxi-discussion/browse_thread/thread/47f131fc2e3817e3 (Actions) * http://groups.google.com/group/fuxi-discussion/browse_thread/thread/bf955620a6ae77d8 (denoted/calculated functions) * http://groups.google.com/group/fuxi-discussion/browse_thread/thread/71a94191e9fef384 (FuXi 1.2) * https://github.com/wwaites/curate/commit/042a96c1589c0fa4980aca733c64c080e02f111e (curate tool update)" 851 1292894661000000 wwaites currently running against ckan.net, adding broken_link tag if a broken link is found. perhaps something more elaborate should be done? works for now anyhow... 851 1292941303000000 wwaites "urllib2 is good for http(s) urls but not, unfortunately, for other types most prominently ftp. change the httpReq action to use http://curl.haxx.se/libcurl/python/" 851 1293025112000000 wwaites "Implemented curlReq that does a curl request and returns statements similar to or analogous to httpReq. Require curate<=0.8 This in a cron job is sufficient to go through all the packages and update them with a broken link tag now: {{{ curate -r https://github.com/wwaites/curate/raw/master/examples/tagging.n3 -s -k API_KEY }}}" 852 1291723207000000 wwaites plugin documentation: http://packages.python.org/ckan/plugins.html 852 1295259620000000 rgrp Removing from sprint and moving to release as a master ticket. 852 1315821628000000 rgrp Moved dataset archiving into new ticket #1327 and this is therefore now done. 853 1291723143000000 wwaites done in http://bitbucket.org/ww/ofs need to merge back into main ofs repo 853 1294047550000000 pudo "Looking at the changeset this cannot be functional yet: where is the implementation of the policy document exchange. It seems to me like this is currently adding the actual credentials to the request (self.ofs.conn.add_aws_auth_header(headers, 'PUT', path)). c.f.: * http://doc.s3.amazonaws.com/proposals/post.html#Form_Fields * http://code.google.com/apis/storage/docs/reference-methods.html#postobject " 853 1294594581000000 wwaites "we don't need a policy document exchange. it's simpler than that. the ""server"" instance has already permissions to upload. it just calculates the headers and such that are needed (based on the ""client""'s initial headers) and gives them to the ""client"" the client then uploads without knowing the ""server""'s credentials. The ""client"" never needs any of its own goostor credentials at all. the only separate step is to make the widget readable by the world. ticket #879 is to expose this as a small set of API calls." 854 1304351843000000 johnlawrenceaspden "Coverage now up to 84% and 81%. Remaining untested code is error conditions, which we decided weren't worth the effort of locking down. fixed on feature-854-tests-for-authz-groups, now merged into default code." 855 1311176988000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. The auth code is in the process of a major refactor anyway. 856 1311177066000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. We know that test coverage needs to be improved, particularly in the logic layer. 857 1311177075000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. We know that test coverage needs to be improved, particularly in the logic layer. 858 1291734297000000 dread On branch feature-858-diff removed ckan.lib.diff as it was not being used. Merged into default in cset:f9ba1ae63ddd 859 1311177461000000 thejimmyg This ticket is more than 6 months old so closing it in line with our new ticketing policy. We know that test coverage needs to be improved, particularly in the logic layer. 860 1291726067000000 wwaites http://bitbucket.org/ww/ckanrdf/changeset/67df6dc33ec4 861 1311168845000000 thejimmyg This ticket has been open for than 6 months so I'm closing it. 862 1291726261000000 wwaites not to mention sparql endpoint... 863 1295259827000000 rgrp Removing milestone as not certain when we'll do this. 863 1338206455000000 ross UI has changed rather a lot in last 18 months, so I killing this bug. 864 1291741028000000 memespring Havent implemented all text changes because worried about impact on other themes. 865 1294414639000000 thejimmyg Isn't this in there now? Can we close? Thanks 865 1295259773000000 pudo This has been fixed on the respective branch and merged into default.. 867 1291916278000000 dread Done in ckanclient cset:d40fb101aba9 867 1292410515000000 dread Also ckan cset:c5b018bfe9bb 867 1299866685000000 rgrp This was a breaking change for loaders code. Obviously we don't have tests for that so would not have been noticed ... Fixed in cset:af81e54bd590/ckanclient 868 1292890251000000 kindly@gmail.com "Below is a patch to make the tests run at least 2.5 times faster (about 15 mins on my old laptop). Instead of dropping the tables each time, it just deletes everything in them, using a low level connection. All the tests pass this way. It's a surprisingly clean patch. Here are a few points concerning it. * I tested truncating the tables but it's slower. If there are any big tables in the tests this way is the fastest (faster than drop). * The sequences (id columns) will start from where they left off. * I also investigated making postgres template database and cloning it, but the complication was not worth it. * sqlalchemy iterates the tables in reverse dependency order, which make this possible. * I targeted rebuild_db as that what most of the tests I saw where using, however I have not checked all tests to see if they all are. * There is a slight hack on the repo object to make sure it knows that ""clean_db"" is coming from the tests. * I refactored init_db for code reuse. * I have not done a version check. sqlalchemy >= 0.5 do this in a different way as outlined in the comments. {{{ diff -r 7f2239b0f743 ckan/model/__init__.py --- a/ckan/model/__init__.py Fri Dec 17 10:34:47 2010 +0000 +++ b/ckan/model/__init__.py Mon Dec 20 23:25:04 2010 +0000 @@ -41,6 +41,9 @@ def init_db(self): super(Repository, self).init_db() + self.add_initial_data() + + def add_initial_data(self): # assume if this exists everything else does too if not User.by_name(PSEUDO_USER__VISITOR): visitor = User(name=PSEUDO_USER__VISITOR) @@ -69,6 +72,26 @@ import migrate.versioning.api as mig version = mig.version(self.migrate_repository) return version + + def clean_db(self): + # delete only added for tests + if hasattr(self, ""delete_only"") and self.delete_only: + self.delete_all() + else: + super(Repository, self).clean_db() + + def delete_all(self): + + self.session.remove() + ## use raw connection for performance + connection = self.session.connection() + ## sqla sorts in reverse dependancy order. + ## in >= 0.5 use reversed(metadata.sorted_tables()) instead of table_iterator + for table in self.metadata.table_iterator(): + connection.execute('delete from ""%s""' % table.name) + self.session.commit() + + self.add_initial_data() def setup_migration_version_control(self, version=None): import migrate.versioning.exceptions diff -r 7f2239b0f743 ckan/tests/__init__.py --- a/ckan/tests/__init__.py Fri Dec 17 10:34:47 2010 +0000 +++ b/ckan/tests/__init__.py Mon Dec 20 23:25:04 2010 +0000 @@ -55,6 +55,7 @@ import ckan.model as model model.repo.rebuild_db() +model.repo.delete_only = True class BaseCase(object): }}} " 868 1292921428000000 sebbacon "See also #867 Thanks for the patch." 868 1292921456000000 sebbacon I mean #876, of course. 868 1293401808000000 anonymous Attached are the timings I have for the tests after I upgraded to 0.57 and after a few simple test tweaks. They do not include setup and teardown time at the class level as they are not assignable to individual tests. 868 1294753596000000 dread "I've merged in David Raznick's patches: * no_autoflush_deletes.diff cset:2b9591172182 * postgres_speed.diff cset:fa1b7e3a4e0f * vdm_purge_no_autoflush.diff vdm cset 8accdd0b9b7f I've also merged in Seb's fork: cset:68d63fda4814 which closes this ticket, achieving test speeds of under 3 minutes!" 869 1292059662000000 rgrp Closed by cset:83734b5e251c which implements an IConfigurer interface. 870 1294862485000000 kindly@gmail.com "A patch is available here. https://bitbucket.org/kindly/ckan/changeset/9a1d6f55587b" 870 1294914243000000 anonymous Merged into default in cset:54ae110094be 871 1292322361000000 wwaites "Regarding rkhunter -- yes, eu1 appears to be clean Regarding the upgrade -- upgradede to 4.72 from backports which, looking more closely, appears to still have the privilege escalation bug but not the remote root exploit. Regarding exim on other hosts, there is no reason for them to be running a full mta, something like ssmtp should suffice. Also very worth the thought of moving to postfix. It's much easier to configure and I haven't known it to have any comparable bugs in the decade or so I've been running it. In fact I've never seen anyone actually use exim before..." 871 1292323656000000 nils.toedtmann "Re postfix: I second ww. I like to run some super simple local MTA (e.g. ""nullmailer"") on all but one server, using a central postfix (or a send-only GMail account) as smarthost. Am happy with postfix for >10years, it's straightforward and rock solid. " 871 1296340558000000 rgrp This is not a ckan issue. should have been on http://knowledgeforge.net/okfn/tasks 872 1292957110000000 dread Done in ckanext cset:8a7e931ef37c 873 1292939274000000 dread Fixed in dgu cset:8bfb867e247a and ckanext cset:fabd31544a73 875 1297085261000000 pudo So far opting for route 1) (not implementing facets), therefore this can be closed! 876 1292891133000000 kindly@gmail.com "I have looked into this already so I can give you a head start. I am working on a project that uses many backends so I have some experience. So here is what I have found so far. == nested transactions == VDM does not support sqlite, as it uses nested transactions. I do not think vdm needs nested transactions. It can use a flush instead. Here is the patch that works. All vdm tests pass. {{{ --- a/vdm/sqlalchemy/base.py Sat Sep 11 23:06:26 2010 +0000 +++ b/vdm/sqlalchemy/base.py Mon Dec 20 16:16:34 2010 +0000 @@ -40,9 +40,8 @@ self.setattr(session, 'HEAD', True) self.setattr(session, 'revision', revision) if revision.id is None: - session.begin_nested() session.add(revision) - session.commit() + session.flush() }}} == indexes == The index file 021_postgres_upgrade.sql in the migrate repository will not run as it uses syntax particular to postgres. Another will need to be made thats similar. sqlite does not support complex indexes like upper(text), so a work around will need to be found. == unicode == The harvesting returns utf8 encoded strings and pysqlite dbapi only supports python unicode objects (as far as I can tell). There will need to be a process in converting all strings that get into the database with string.decode(""utf8"") == dates == Have not looked into this one too much. However, as sqlite stores everything as strings the timestamps appear to be failing on conversion back into python. I have solved the above two issues before by adding attribute extensions to sqlalchemy mappers to do the conversions without effecting too much code. == in memory sqlite == Some tests need to change in order to make sure the database is created first because the database gets lost each time. In the tests that I have made pass, they run in about a seventh of the time as they do on postgres. == Other things to keep in mind. == * Need a new flag in test.ini to remove full text indexing completely, or always use it with solr. * There are enough incompatibilities between the databases that you would also want to test against postgres as well, at least before a release. * I would probably upgrade sqlalchemy first, so you will not have to the changes twice. The new versions are significantly faster too. * I have submitted a patch to #868 that makes the tests run about 2.5 times as fast and I think there are more low hanging fruit if the aim is test speed. " 876 1292924064000000 sebbacon "Thanks for the info! Re. nested transactions. I am getting repeated non-deterministic test failures against sqlite (and indeed postgres, but these failures appear more frequent against sqlite). One of them I seemed to be able to get rid of by eliminating the savepoint as per your first point. However, it appears that sqlite [http://www.sqlite.org/lang_savepoint.html does] support savepoints; to demonstrate it, the following test code appears to work in the latest sqlalchemy: {{{ #!python from sqlalchemy import * from sqlalchemy.orm import * db = create_engine('sqlite:///') metadata = MetaData(db) users = Table('users', metadata, Column('user_id', Integer, primary_key=True), Column('name', String(40)),) users.create() class User(object): pass usermapper = mapper(User, users) Session = sessionmaker() session = Session() fred = User() fred.name = ""Fred"" sue = User() sue.name = ""Sue"" amy = User() amy.name = ""Amy"" session.add(fred) session.add(amy) session.begin_nested() session.add(sue) session.rollback() session.commit() assert session.query(User).count() == 2 print ""OK"" }}} So, while I agree they're not needed, I'm not sure they're a problem. What do you think? Also, have you seen non-deterministic test errors like this?" 876 1292939711000000 kindly@gmail.com "I have read quite a lot of people having problems with savepoints with sqlite and thought they were not supported on sqlalchemy. They are at least not consistant with postgres ones. I may well be out of date on this. Here is an [http://groups.google.com/group/sqlalchemy/browse_thread/thread/dc9d1b61044bf730/65a62a33ec313842?lnk=gst&q=no+such+savepoint#65a62a33ec313842 example] even though its a bit old. I did get some non deterninistic errors, the above seemed to fix them. A failed subtransaction is not handled well by sqlalchemy and I think this causes knockon effects due to the unresolved transaction. I would stay well clear of them entirely if possible. What are the errors you are getting?? My 2 cents. ignore me at will... I would think about using a different backend for testing than production. [http://stackoverflow.com/questions/2716847/sqlalchemy-sqlite-for-testing-and-postgresql-for-development-how-to-port look here]. If you want to support both then you should test on both. There are simple ways to scrape a few more minutes off the tests. If you want real speed, then a multiprocess solution (with a database per core) would be sensible if a bit tricky. " 876 1293188088000000 anonymous "Thanks for your feedback, very useful. I don't really agree with the people in the linked discussion who say it's pointless testing against a different database from production. The goal here is to make it easy enough for people to run as many tests as possible that they actually do so. Even 15 minutes is too long in that case. With sqlite we can get it in at under 5 minutes. I would also like to identify the longest running tests (which I would characterise as ""functional"" or ""integration"" tests and make them run as a separate suite, and then encourage a culture of writing true unit tests before functional tests, so that running unit tests can happen in 1 minute and be part of the regular development cycle. That's no replacement for also running *all* tests periodically, and also running tests under postgres, which we can continue to do on the continuous integration server. Longer term I agree that it would be better to run local tests against postgres too, but that will I think involve refactoring many of the tests." 876 1293218714000000 anonymous "I agree with all your points about testing apart form using sqlite, especially splitting out the functional tests and continuous integration. > Longer term I agree that it would be better to run local tests against postgres too, but that will I think involve refactoring many of the tests. Well there are two options 1. refactor the tests 2. refactor the code to use sqlite and postress It is a value judgment as to which is more complicated. I personally think 2 is more complicated but may be wrong on that. The real danger with 2 is that you are needlessly adding complication to production code, with 1 you are only changing the tests. Upgrading to sqlalchemy 0.5+ should happen first regardless. You will need upto date documentation. There is another option too. Put the postgres data directory on tempfs/ramfs and turn off durability [http://www.postgresql.org/docs/9.0/static/non-durability.html here]. We would need a way to db init before the tests where run (or) at boot). This may be the best of both worlds. Anyway Happy xmas!! " 876 1294753889000000 dread Seb and David have completed this I believe. I've merged the changes into core CKAN in cset:68d63fda4814. 877 1297680579000000 rgrp Basic pass on an implementation (no permissions yet etc): https://bitbucket.org/okfn/ckanext-upload/changeset/9ae543f0645f 877 1298624165000000 rgrp "Various tidying in https://bitbucket.org/okfn/ckanext-upload/changeset/0fad7aa7aa97 (success messages, permissions on uploaded file - public-read) and completed permissions in https://bitbucket.org/okfn/ckanext-upload/changeset/a83ce00a1266. Still need to integrate into general workflow (e.g. create a Resource on successful upload) but that is a separate item so this ticket is now done." 878 1314404621000000 rgrp A working version of this in ckanjs as of 2 weeks ago. Need to integrate into main CKAN. 878 1315820838000000 rgrp This is now done in feature-1294-ux-improvements-dataset. see e.g. cset:c6f7f5018b4f 879 1294604066000000 wwaites first cut: https://bitbucket.org/ww/ckanext-storage/src 880 1292945137000000 dread Fixed in ckanclient cset:e7c0af586367 and ckanext cset:82d974ab6860 with test in dgu cset:c0b2c5fd95ea 881 1294410574000000 thejimmyg "Hello fccoelho, Could you please post the output you get from running pip? Thanks, James " 881 1296335072000000 rgrp Closing due to lack of clarification. 883 1300196622000000 thejimmyg Now complete. 884 1293319984000000 wwaites "Implemented in feature-884-rmsource branch at https://bitbucket.org/ww/ckan/changeset/50cbfc5a6bee " 884 1294247654000000 thejimmyg Related to ticket #665, reopening so that I can close it once I've got it working with Drupal too. 884 1294253563000000 wwaites see feature-884-rmsource in the main ckan repo deleting my own repo 884 1296592140000000 thejimmyg OK, I've implemented an API call for this too now and all is merged into default. 885 1293278564000000 wwaites Friedrich had some mixed experiences with owslib and some German CSW endpoints: http://pudo.okfnpad.org/geodaten 885 1293314219000000 wwaites "See: https://bitbucket.org/ww/ckanext-ows impossible as of yet to test against live server from the OS " 885 1294253497000000 wwaites done with feature-885-owslib 886 1294916538000000 dread Being done as part of DGU ticket: https://trac.dataco.coi.gov.uk/projects/datagov/ticket/757 887 1300196600000000 thejimmyg "This is done. See https://bitbucket.org/okfn/ckanext-harvest At the moment we are not adding migrations to remove the harvesting tables. We are designing a harvesting refactor and will write migrations once the refactor is complete so that instances that use harvesting get upgraded, and those that don't get their harvesting tables removed. Also, see this #1030 for the moving harvesting out of the REST API. " 888 1294830297000000 Stiivi "Chages to Data Proxy: * tests added with configurable list of known URLs * use brewery for transformations (included reference to brewery framework in a new vendor directory) * side effect: code to make google find external packages in vendor directory (from now on, all external packages should go there and be referenced from .hgsub if they are mercurial repositories) * changed response contents: moved from 'headers' to root, renamed 'response' to 'data', added field list as 'fields' * changed way of registering transformers (now class object is used instead of name) * added 'encoding' and 'dialect' parameters for CSV * added optional data audit (parameter 'audit') Changes: https://bitbucket.org/Stiivi/dataproxy/changeset/fccbdd275be5 Data information: http://databrewery.org/doc/data_quality.html#brewery.dq.FieldStatistics " 888 1307352537000000 thejimmyg I don't think any progress has been made on this for a bit so I'm assigning it to me. 888 1311773103000000 johnglover "Dataproxy / Dataapi now deprecated in favour of the combination of new QA archive / process commands and the webstore. Changes in relation to Dataproxy / Dataapi: * Currently only supports CSV files, but plans to add support for excel and google docs spreadsheets soon. * Uses David Raznick's CSV parser instead of Brewery for parsing, handles messy CSV data better. Changes in relation to old QA functionality: * decoupled archiving (downloading) and QA process * added a new 'process' command which parses downloaded files and adds them to a local webstore Closing for now, any improvements/feature requests should be in tickets relating to either the QA functionality or the webstore." 889 1293715109000000 rgrp Done in cset:cdbb2e6128b3 (had added item to template earlier pulling from 'g' object but had not tied that up to config as yet). Also added test and also '''removed existing google analytics code''' -- so need to explicitly set this in config from now on. 890 1318529648000000 rgrp May want to close as invalid as obsoleted by more recent queue work. 890 1318599247000000 kindly Invalid due to #1397, We will be using celery instead. 891 1318602128000000 rgrp May only do link-checker and not do full storage in this sprint. 891 1320143424000000 johnglover "Almost finished (see http://github.com/okfn/ckanext-archiver). Still to address: - check headers to see if hash / cache / max-age / expires indicates that the resource does not need to be downloaded. - add cache url to resource" 891 1320149841000000 johnglover "Added cache_url and cache_last_updated to resources after archiving. Not checking for hash value in headers. This process will generally only run when a new resource is added or someone updates a URL, so we don't expect to be regularly downloading the same resource. We will need something along these lines if this is running as a regular cron job, but in that case the logic will be added to the cron job itself (probably a paster command)." 892 1324314636000000 rgrp Moved to v1.6. 892 1324402480000000 johnglover "https://github.com/okfn/ckan/compare/bff538b%5E...d700680 Merged with master and deployed on test.ckan.org" 893 1298293527000000 thejimmyg We don't understand the use case for this requirement. Closing for now until a use case can be demonstrated. 894 1300196388000000 thejimmyg All test CSWs have been successfully harvested from. CSW harvesting is disabled on that source so we can't harvest from it. I don't think we need a ticket for this now do we? 896 1300217375000000 thejimmyg "Actually, I think this should be implemented instead by considering a different CKAN instance as a potential harvest source and then harvesting from it. By thinking of it this way in the first instance, we effectively get a read-only copy of other data in CKAN but one which will be kept up to date. Marking as duplicate. Discussion can now carry on via #987 Common harvesting framework." 897 1294914333000000 dread Same as #870 898 1294659537000000 dread From David Raznick 898 1294662408000000 dread Merged in cset:89bf917f9b46 899 1294662417000000 dread Merged in cset:89bf917f9b46 900 1294662424000000 dread Merged in cset:89bf917f9b46 901 1294662466000000 dread Merged in cset:b8a649f51cab from Seb's changes. 902 1296638257000000 rgrp "Also confirmed in FF. David (R) and I have both had a go and fixing this and it seems to be a bit tricky (very odd css behaviour!). Rather than directly fix this I'm thinking of doing a bit of rework of that set of menu tabs as they could be much nicer anyway and hence re-titling of this ticket." 902 1297078885000000 rgrp Fixed in cset:42102bf7f922 and cset:42102bf7f922 by switching to a new 'tabs' look that has none of these issues. (See branch feature-902-submenu). 903 1295265645000000 pudo This is now partially solved by filtering for deleted packages. As two further measures, we should also add a package deletion method to the Solr backend and skip deleted packages when indexing. 903 1296588070000000 pudo fixed in https://bitbucket.org/okfn/ckanext-solr/changeset/be4cffa9b987 904 1299840539000000 rgrp We're already now into improving the docs and ticket:927 is now reasonably detailed. 905 1328638536000000 dread "This works with the current SOLR search. e.g. http://thedatahub.org/dataset?q=gr%C3%B6%C3%9Ften" 906 1324299793000000 rgrp Do we need to change in core code or just configure solr? 906 1328639465000000 dread "This was done in #1701 but only in a custom SOLR schema in the ecportal extension: https://github.com/okfn/ckanext-ecportal/commit/6682926d8895f146cdf1e52ab4fbead9b065af77 Can the ASCIIFoldingFilterFactory be added to core CKAN's SOLR schema for all CKAN users to benefit from?" 906 1328782087000000 amercader Yes, I don't think there will be any problem, and we won't need to create a new version of the schema as the change is backwards compatible. 907 1296335337000000 rgrp "Basic implementation done and deployed. However plenty to improve, e.g. * Support more formats (use external systems for preview?) * json (!) * html (trivial!) * sparql * ... * Do not display preview if no preview " 907 1297072303000000 rgrp Closing this now and improvements can go in a new ticket. 908 1296403695000000 wwaites "see also from http://knowledgeforge.net/okfn/tasks/ticket/485 {{{ (pyenv)okfn@eu7:~/var/srvc/ckan.net$ sudo uwsgi -C -iH /home/okfn/var/srvc/ckan.net/pyenv --paste config:ckan.net.ini --uid www-data -s 10.48.162.201:9001 *** Starting uWSGI 0.9.6.6 (32bit) on [Sun Jan 30 16:00:13 2011] *** compiled with version: 4.4.3 Python version: 2.6.5 (r265:79063, Apr 16 2010, 13:28:26) [GCC 4.4.3] uWSGI running as root, you can use --uid/--gid/--chroot options setuid() to 33 *** WARNING: you are running uWSGI without its master process manager *** your memory page size is 4096 bytes allocated 412 bytes (0 KB) for 1 request's buffer. Setting PythonHome to /home/okfn/var/srvc/ckan.net/pyenv... binding on TCP port: 9001 your server socket listen backlog is limited to 64 connections initializing hooks...done. Loading paste environment: config:ckan.net.ini Traceback (most recent call last): File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/paste/deploy/__init__.py"", line 3, in from loadwsgi import * File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/paste/deploy/loadwsgi.py"", line 8, in import pkg_resources File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 2691, in add_activation_listener(lambda dist: dist.activate()) File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 668, in subscribe callback(dist) File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 2691, in add_activation_listener(lambda dist: dist.activate()) File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 2195, in activate map(declare_namespace, self._get_metadata('namespace_packages.txt')) File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 1790, in declare_namespace _handle_ns(packageName, path_item) File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pkg_resources.py"", line 1761, in _handle_ns loader.load_module(packageName); module.__path__ = path File ""/usr/lib/python2.6/pkgutil.py"", line 238, in load_module mod = imp.load_module(fullname, self.file, self.filename, self.etc) File ""/home/okfn/var/srvc/ckan.net/pyenv/src/ckanext-dataapi/ckanext/dataapi/__init__.py"", line 36, in from pylons import config File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/__init__.py"", line 4, in from pylons.config import config File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/config.py"", line 2, in from pylons.configuration import * File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/configuration.py"", line 25, in import pylons.legacy File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/legacy.py"", line 11, in from pylons.util import deprecated, func_move File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/pylons/util.py"", line 18, in from paste.script.appinstall import Installer File ""/home/okfn/var/srvc/ckan.net/pyenv/lib/python2.6/site-packages/paste/script/appinstall.py"", line 23, in from paste.deploy import appconfig ImportError: cannot import name appconfig }}}" 908 1296659056000000 pudo fixed in https://bitbucket.org/okfn/ckan/changeset/08c0e1a819e4 and https://bitbucket.org/okfn/ckanext-stats/changeset/6e86c071db99 908 1296660052000000 wwaites "Not fixed. Try running with: {{{ uwsgi -C -iH /path/to/your/pyenv --paste config:ckan.net.ini -s 127.0.0.1:9000 }}} Still get giant traceback relating to a circular import centered on an arbitrary plugin" 908 1296730578000000 pudo I've moved the uswgi/nginx part to #952, closing this issue since the problem it describes has been fixed. 909 1346669602000000 ross Backlog until funded. 910 1310128544000000 thejimmyg Not enough information and over 6 months old so closing. 911 1296467375000000 pudo Done. 912 1306774876000000 pudo moving to ckan 913 1299841413000000 rgrp If at all possible this should uris from the OKF licenses registry at http://licenses.opendefinition.org/ 913 1306774962000000 pudo moving to CKAN, still need a resolver 915 1296467518000000 pudo Live and running :-) 916 1297066902000000 rgrp Moved to https://bitbucket.org/okfn/vdm/issue/4/port-new-vdm-to-mongodb 917 1295280232000000 dread Fixed in cset:f2865f43d8ee 918 1315911507000000 dread Preview feature is now deprecated 919 1303202627000000 dread Simple to fix this in the process of fixing #108. Fix went in cset:304d30d85816 for release 1.3.4. 920 1295355318000000 hellmann@informatik.uni-leipzig.de my first email address was wrong... 920 1300217199000000 thejimmyg @David Should we implement this as a change to the way tags work or via a tidy up cron job? 920 1300319140000000 kindly "The only issue here is that we are listing tags that relate to 'inactive' packages. We are already not listing tags that relate to NO packages. I have fixed this. cset:cd0347eed69f The tag in the example is related to a deleted package so should not be deleted. With this patch it no longer gets displayed." 921 1300197629000000 thejimmyg Seb started this, I completed it and the code is now live. Further refactoring will be done first in #1037 and then in #987. 922 1310128789000000 thejimmyg "See also #358 which I'm closing because it gets merged into this ticket. The reason we were blockd on #358 was to do with issues around authorization." 922 1320664187000000 rgrp Duplicate of #1032 (I think) 925 1323168588000000 thejimmyg This is fixed in CKAN 1.5 926 1296637927000000 rgrp "Comments from RP - http://lists.okfn.org/pipermail/ckan-dev/2011-January/000181.html Libraries I have used: FormEncode, FormAlchemy (what we are currently using, before that formencode). Neither seemed perfect but I think the form issue is a 'hard' problem (perhaps with no perfect answer) [1]. FormAlchemy, in retrospect, was probably a mistake as it merges too much model/validation/form generation into one thing. At least 3 functions involved: 1. Generating (or just filling) a form template with 'form data' (and errors) 2. Converting model data to form data (also happens for APIs in fact) -- let's call this 'dict-ization' 3. Converting form data to model data (and validating) (inverse of previous step) I think one can and should separate 1 from 2+3 (and one of problems with formalchemy is it doesn't -- the attraction being you don't repeat yourself as forms get generated from model but I think this is actually a false economy in medium-term). I'm not specifically recommending the following as I haven't used them but I've looked through docs, they are active and reasonably mature: 1. Flatland: http://discorporate.us/projects/flatland/docs/tip/ * Only does 2+3 which is a good thing IMO 2. WTForms: http://wtforms.simplecodes.com/ * Used in standard flask docs: " 926 1298489517000000 rgrp @Seb: I believe this is now decided following discussion last week. Please could you detail results and close :) 926 1298541597000000 anonymous "Goals: We want the interface for updating an object to be loosely coupled to the method for updating it. We might update a Package from: - HTML forms - a REST API (using JSON) - a CLI (potentially using command line arguments, YaML, XML or ini files) Right now, data is validated using a form framework, even if we're not using forms. Data is written to the object as part of the forms framework (using the ""sync()"" method), making the process hard to customise and hard to discover. Instead, there should be a standard chain for: - deserialising untyped data (such as that received from an HTTP POST or parsed from a YaML file) into valid data - returning structured errors suitable for displaying to the user - saving the validated, deserialised data Ideally, it would look something like: schema = MySchemaDefinition() raw_data = open(""raw.csv"", ""r"").read() structured_data = to_python(raw_data, schema) try: validated = validate(python_data) myobject.update_from_dict(validated) return ""Updated OK"" except ValidationError, e: return ""Error: %s"" % e.to_dict() The inverse would be something like: structured_data = myobject.render_to_dict() raw_data.write(to_csv(structured_data, schema) print ""Wrote CSV %s"" % to_logformat(serialized_data, schema) The question of how to generate and display forms should be completely decoupled from this. It should be easy to write forms by hand, which means it should be simple to flatten the serialized data to key, value pairs, and match up any validation errors to each key. Optionally, a form widget generation framework is a nice-to-have, but not essential, as it is expected that, given enough time, the majority of forms will require manual coding to accomodate edge conditions. A form widget generation framework should be reasonably complete if it's worth trying at all, which means it should support things like: - nested fields (at least repeating, multi-value fieldsets) - widgets for dates and file uploads - internationalisation ...but note I'd settle for *no* widget generation Components of a serialisation / validation framework: - a simple, obvious way to define a schema - a lightweight validation implementation - simple interface for validators - easy to match validation errors to data structure items Overall, I'd like to see: - loose coupling, no framework dependencies - maximal test coverage - extensive documentation with readily available examples ## Findings I looked at flatland, formencode, FormAlchemy, formish, WTForms, Django, web2py, deform/colander, formconvert and web.py - **web2py** just helps build HTML from python, so isn't what I'm after at all - **web.py** has rudimentary validation which is only aimed at HTML forms and is hence tightly coupled with them. - **Django**'s forms are again tightly coupled to HTML forms (and their generation) - **FormAlchemy** similarly couples validation to forms, and is focussed on inferring a schema from a data model SQLAlchemy. - **WTForms** again focuses on Form generation and don't make itx easy to deserialise arbitrary data This leaves us with Flatland, Formencode, Formish, Colander/Peppercorn/Deform, and FormConvert. Having reviewed all of these, I rejected Formencode on the basis of its patchy documentation and relatively low unit test coverage. I also found it mixed concerns a bit much for my taste. Formish felt similarly sparsely documented. Of the remainder, I'd be happy using any of them, but opted for Colander in the end as it has the most exhaustive documentation and unit tests and has been used in production for a long time. FormConvert has a nice design but is a bit of a moving target at the moment -- worth revisiting in the future. " 927 1299841206000000 rgrp Major update including notes of what has been done (where not a separate ticket) and addition of a few items. 927 1300105638000000 rgrp Closing this ticket as #841 is minor. More work on docs can go in new tickets. 928 1297115097000000 rgrp Fixed in cset::540e8e548d84 928 1297115136000000 rgrp Forgot to say moved contributors material to http://ckan.org/wiki/Contributors for the time being. 929 1297686088000000 thejimmyg The licenses service was down, it is back up now. We should be able to cope with this situation better though. Renaming the ticket. 929 1299840884000000 rgrp I'm closing this ticket as a) most systems should install the licenses package (and hence have the licenses locally) and b) licenses service has now moved to s3 so should be very robust (see ticket:973 and http://knowledgeforge.net/okfn/tasks/ticket/605) 930 1296062391000000 wwaites "This is for HMG to prove to them that requests are being serviced in a timely manner * move this functionality to a plugin * make it use rrdtool instead of lots of itty bitty files " 930 1296070197000000 wwaites "actually, this is a configuration variable ckan.enable_call_timing cleaned up the code a bit, make sure this variable unset or set to false or no and there should be no millions of files as it is unclear if this is actually used or necessary, not doing the rrdtool part just yet. if there is call for it, make a new ticket for this." 930 1297066292000000 rgrp Just want to agree with wwaites assessment here. dubious this code is currently useful as a) not used b) (more importantly) probably better ways to achieve this functionality. 931 1298379187000000 dread This was completed in ckanclient in cset:1bfefd7596d3 932 1296159225000000 dread "Initial work done on branch: enh_932_sql_migrate_upgrade Not working yet - hopefully kindly will be able to help tomorrow." 932 1296496416000000 dread Completed on branch and merged into default in cset:25c94cdbd283 933 1297348975000000 dread Done on enh_933_get_rid_self_in_cls_meth and merged into default cset:5750c77e580d ready for ckan 1.4. 934 1323171047000000 thejimmyg We have other ways of solving this problem now rather than a key value store for plugins. Marking as invalid. 936 1298283172000000 thejimmyg "Hi Wayne, I'm assigning this to you but it isn't a priority yet. We'll put it in a sprint when it is time to do it. Cheers, James " 936 1303117147000000 thejimmyg How is this coming on John? 936 1303838713000000 rgrp Completed it seems :-) see https://bitbucket.org/okfn/ckanext-follower 937 1296385042000000 sebbacon "Could consider using third-party analytics tracking here, which will also give referrer etc data for free? Would probably be bes provided in the form of optional piwik or google analytics integration. Being able to say in the UI how many downloads there have been would need piwik. " 937 1297689781000000 sebbacon "I did a very quick hacky thing at the end of last week on top of the ""insert google analytics code"" extension we discussed, to work out ""most popular packages"" based off data harvested from the Google Analytics API. Needs making generic, tests etc but could be a starting point: https://bitbucket.org/sebbacon/ckanext-googleanalytics/src" 937 1297689859000000 sebbacon (and it would also need some proper caching as the GA API is very slow) 937 1298892547000000 sebbacon "The current implementation I referenced above will be a good starting point. Work that remains: * Add download click tracking to individual download links (currently we just record page views for packages, not downloads) * Somehow cache the download stats against each package (the Google API is very slow); package reddis or sqlite or similar as a local storage for the extension * Expose download information in the relevant places in the UI (all users? package owners? where?) This is about 2 days' work. Unlikely to get it done in this sprint." 937 1302513831000000 sebbacon Completed; software at https://bitbucket.org/okfn/ckanext-googleanalytics/src 938 1296753055000000 pudo fixed in https://bitbucket.org/okfn/ckan/changeset/064d74a1ead8 939 1323171158000000 thejimmyg "This is now implemented, the notification links to the about page. We might want to update that page with better information, but that would be a different ticket. " 940 1296656433000000 pudo "It seems like even with openid.realm set we could only create two ""zones"": *.ckan.net and ckan.net. We do not want *.ckan.net because it interferes with ccCKANs. My vote for the moment would be to 303 www.ckan.net to ckan.net. " 940 1296807699000000 rgrp "I agree with pudo (though it would not be the end of the world if these were treated as the same realm!). I've now created a permanent redirect for www.ckan.net to ckan.net. {{{ RewriteEngine on RewriteCond %{HTTP_HOST} ^www\.ckan\.net$ [NC] RewriteRule ^(.*)$ http://ckan.net$1 [R=301,L] }}}" 940 1297071748000000 rgrp "Listing as wontfix now since: 1. Have workaround 2. Better for most sites to converge on a single domain anyway (for SEO etc) -- via std redirect approach or otherwise 3. Seems problematic to fix this via openid realm" 941 1297246005000000 thejimmyg This will be implemented in http://bitbucket.org/okfn/ckanext-community by Wayne. 941 1297689750000000 thejimmyg "The system will need some way of plugging in the model. See ticket #989 for progress on this. Other ideas: * The apps will need an image upload * We might like a voting system for apps and ideas, potentially that could be re-used later. Let's discuss the above ideas after the basic functionality is in place." 941 1298886391000000 wwitzel3 "Continued work on the community plugin. I am still learning the layout of templates and how they work within ckan and getting figuring out Genshi templates so this is where most of the delay has been. I've been able to determine a pretty good plugin layout for extensions that create models. I am currently focusing on getting the rest of the UI in place and trying to determine the best way to get colander to do the desired validation beyond ensuring the form has all the elements. After todays work, I will push what I've done and I would like to walk through the design with someone at some point." 941 1300295545000000 wwitzel3 "https://bitbucket.org/okfn/ckanext-community/changeset/c26cc663fa00 I need to update the README to explain how to enable Edit and Delete now that I've integrated it with authorization in CKAN." 941 1301909446000000 thejimmyg See also discussion on #1069 about stub datasets. 941 1310134339000000 thejimmyg This is now implemented by pudo in publicdata.eu. For the portal case, apps and ideas will come from Drupal so we no longer need this ticket. 942 1296499160000000 pudo fixed in cset:81a0f5538a5b 943 1297076498000000 rgrp Done see http://wiki.ckan.net/ 944 1298571917000000 pudo Won't work on this for now - IATI is now running against plain CKAN but this is not deployed. We will continue work on this once IATI requests more functionality and shelf it for now. 944 1306774766000000 pudo done at iati.ckan.net 945 1301909147000000 dread Moving super ticket to 1.4 milestone 945 1325259350000000 rgrp Closing as all subcomponents now done. (parts of this duplicate the other super ticket #1032). Also move to ckan-v1.6 as that is when last parts were done. 946 1296833368000000 dread Fixed in cset:a30c31b9009d 947 1314031310000000 dread This was done in #1025 through configuration. Happily you can setup config in an extension, so covers this ticket's aims too. 948 1300271009000000 sebbacon "I would propose a ""trash can"" model: a ""deleted items"" area where only deleted items can be viewed / searched. Deleted items should not appear anywhere else, but we could include a ""also search deleted items"" option in the search for admins. (In which case, deleted items should have a ""div class='deleted'"" or similar so they can be highlighted with CSS)." 948 1300271100000000 dread This sounds like a good solution to me. This should include not just packages, but groups and all other stateful objects. 948 1300271268000000 dread "There's also the API to consider. One edge case to consider too - currently we don't allow reuse of the name of a deleted package, which is confusing. The deleted package should be moved aside (renamed), unless the deleted package is entirely the same as the new or renamed package." 948 1300385979000000 sebbacon Emptying the trashcan (which would amount to purging deleted objects) would probably be a function of the administrative dashboard #833 948 1300728602000000 dread "We have a use case now for DGU where if a user requests a deleted package, he is 302 redirected to another package (specified in the deleted package's extras). This is for the API, but could also be done in the Web UI if deleted packages are hidden away in a trashcan. But this suggests we need a trashcan in the API too. i.e we should keep the data model really synchronised between the Web UI and API, (even if they are becoming divorced from the ORM Model - or maybe we should have a trashcan in the ORM model too?) Ideas: * GET api/rest/package - lists active packages * GET api/rest/package/traffic_data - active package * GET api/rest/trash/package - lists deleted packages * GET api/rest/trash/traffic_data_old - deleted package " 948 1314112642000000 dread Note: Packages will now be excluded from the search view (see #1283). Also there is now no redirect requirement from DGU any more. 948 1323172859000000 thejimmyg Re-assigning to me temporarily to investigate now that Seb has left. 949 1296730355000000 pudo "Fixed in: * https://bitbucket.org/okfn/ckan/changeset/8017a6686838 * https://bitbucket.org/okfn/ckan/changeset/5e6634aab275" 949 1297074827000000 rgrp Hocus pocus so set owner after being fixed! 950 1296732042000000 dread "Done in: * ckan cset:e68d0704e57b * ckanext cset:0d7d223e4302 * dgu cset:c633d2608c29 Importer controller now in new repo ckanext-importer (mothballed)" 950 1297076647000000 rgrp Reopen to set an owenr. 951 1314031006000000 dread Specific problem explained by having two users with the same name - one was authorized and one wasn't. See: http://lists.okfn.org/pipermail/ckan-dev/2011-April/000550.html 953 1296809834000000 rgrp Done in cset:3275d2fc9e43 954 1300217106000000 thejimmyg I'll look at this for a v3 API with kindly as part of the dictization. It also addresses some potential XSS issues in the API we've discovered. 954 1300447466000000 thejimmyg "What would also be really nice is a `help` key in the response which was always returned unless a query parameter of `?help=False` was sent. The help text could explain what the response meant but more importantly would show examples of other calls you can make with the IDs returned such that the whole API is discoverable without anyone ever needing to read any docs! It also means the API documentation is more likely to be up to date since it will be obvious to developers when it isn't. With help=False, the JSON can have all whitespace stripped too for production use. I also think there should be no distinction between GET and POST so that people can easily link to API calls if they want to." 954 1303118513000000 thejimmyg David Raznick has implemented JSON errors for the v1 and v2 API, we'll look at this over the next few weeks. 954 1310134531000000 thejimmyg "See also this proposal about ""inlining"" extras fields #972. David Raznick and I have also agreed that for API 3, each call to the logic layer will return an object (basically a dictionary) rather than using Exceptions. This means the return values from the logic layer can exactly mirror the JSON data strucutres returned via the API. The help values can come from the docstrings of the logic layer functions." 954 1315948855000000 rgrp @kindly: can you update here. My impression is that lots was done but the current ticket description bears little relation to what was done. Suggest we move current content to a new ticket and update this with a short description of what *was* done and then close -- does this sound sensible? 954 1320142744000000 kindly "This is basically the complete now with documentation. The child tickets no longer seem to fit and are not essential for completion. " 955 1296833234000000 dread Fixed in cset:9a45a07ad95e 955 1297342534000000 dread Fixed on cset:b65b9830571c and merged into default in time for 1.3 release. 956 1299489084000000 kindly cset:1305b9192d49 957 1300216958000000 thejimmyg Let's come back to this later. There are larger UKLP refactors going on. Sorry to mess you around. 957 1314218701000000 rgrp I'm closing this as wontfix as we now have arbitrary metadata for resources so probably won't need a specific field. That said we probably will want a schema but that is not yet finalized and would be a separate and wider ticket. For current proposal of the attributes generally wanted for resources see http://wiki.ckan.net/Domain_Model/Resource (recently updated). 958 1297430414000000 dread Sorry I didn't see this and created #78. Closing this one as #978 is fuller. 958 1297430602000000 dread Apologies, #978 is subtly different - reopening. 958 1320664462000000 rgrp This was done as part of dataset resource editing in #1294 (duplicate) 960 1297080672000000 pudo Test this with a dedicated test using a unicode user name. 960 1297169056000000 pudo fixed in cset:8cb94dcdecb2 961 1297073658000000 dread "So you also need: 4. Converting form data to dict 5. Converting dict to model i.e. the dict is not the same as the serialized form data or model data." 961 1308230058000000 rgrp @kindly: can we close this ticket now. All the meat is done and the remaining related tickets are either low priority or possibly wontfix. 961 1310126100000000 shevski The main parts of this ticket are now done. The remaining parts can be dealt with separately as their own tickets. 962 1297678925000000 rgrp In progress but not yet completed so moving to next sprint. Estimate remaining time at: 2h. 962 1298889078000000 rgrp Nearly done. 962 1301364987000000 rgrp "Closing as have now reworked to: * Support plain text previews for many formats (including xml based formats) * Try to handle everything else as html with an iframe ... * Do not show preview button where not useful * Normalize formats for better recognition (e.g. text/csv, application/xls) See: https://bitbucket.org/okfn/ckanext-datapreview/changeset/c1d672a6c368 and previous. Also re-updated the dataproxy so it works again (had got out of sync when mistakenly reverted the dataproxy a couple of weeks ago). Could still do api/sparql (using sparql js wrapper) and handle json (as plain text ...?) but these can be new tickets." 963 1297850773000000 thejimmyg We will also remove all the different pip files as part of this fixing #982 at the same time. 963 1298284252000000 thejimmyg "You can now get CKAN from the repository http://apt-alpha.ckan.org/debian " 964 1297211237000000 rgrp Done in cset:fd0e2edf03b1 965 1297682632000000 kindly added with cset:553421d05ce8 971 1299245064000000 sebbacon folded into #1013 972 1300219509000000 thejimmyg I don't really agree with this. In fact I'd rather move further away from auto-magic key value pairs stored in JsonTypes. Can we discuss on Skype please? 972 1310134129000000 thejimmyg Having thought about this more David Raznick and I have decided to stick with separate extras for the timebeing. Mainly because of internationalisation issues. 973 1297678851000000 rgrp "Done. See http://licenses.opendefinition.org/ the new licenses release (0.6) http://pypi.python.org/pypi/licenses/0.6 and cset:b8e54186faee Actual cost: 4-6h (as i refactored the licenses package heavily)" 974 1297342599000000 dread Fixed in cset:9dc20731d0fd and cset:5149f503e741 for 1.3 release. 975 1297344448000000 rgrp Fixed in cset:a4caf6b8ce3d. 976 1297346954000000 dread Done in cset:1566d08d529f for release 1.3 977 1297420742000000 dread Fixed in cset:8809fbefaf8c for 1.3 and merged to default. 978 1314404858000000 rgrp Suggest we either a) get rid of columnar layout or b) (preferable) get a dedicated page for resource. That way summary on package page just shows main attributes. 978 1325259266000000 rgrp Think this should be part of a larger refactor of resource editing now that #1445 (resource page in WUI) is done 978 1328530459000000 rgrp "We already have basic editing in the Resource section of dataset edit. However, need to extend this to additional fields such as webstore_url and ""extras""." 978 1330547181000000 zephod "This was not trivial to implement. The backend supports arbitrary key/value pairings on resources, and the frontend can now handle this. Add, edit and delete resource extras according to the form state. I had to make a modification to the backend: When saving a resource, you have to submit the complete set of extras. Unsubmitted extras are assumed to be deleted. (This matches the behaviour of the package form). https://github.com/okfn/ckan/commit/a41cd0c9b04c757f5fa37acaba6be71e345a9c1f#L0R39" 979 1314404905000000 rgrp Change from Edit Resource extras in the API to Edit Resource extras in the WUI as believe we already have API editing and reference to #978 implies this is about the WUI. 979 1315221162000000 dread "Changed the title back - it was accurate. The comment was because with formalchemy, WUI and API ran off the same 'form' code. This functionality may already be there with the new stuff, but certainly needs tests." 979 1315222244000000 rgrp Is this already done? Is it under test? 980 1297501755000000 rgrp No owner :) -- also what about the issue that when trying to edit a package for which you did not have permission you got a 500 ... 980 1297503120000000 pudo "Re package editing issue, that was a ""double error"" fixed by the first two changesets mentioned. " 981 1297682773000000 kindly fixed see cset:3d1f720a2e5b 982 1297518386000000 anonymous Well we could transfer all the dependencies and version numbers to a config file for the fabfile, but we don't achieve much. 982 1297525477000000 rgrp "I don't think i understand why this has been closed (and it would surely be wontfix rather than invalid ...). Let me explain in more detail: we would move to having one and only one pip-requirements.txt in the repo at any given revision and it would simply have the correct info for whatever branch/revision it was on. At the moment we are having extra pip-requirements-....txt for labelling branch pip-requirements when we could just the branching facility of mercurial. You'd do {{{ wget http://bitbucket.org/okfn/ckan/src/metastable/pip-requirements.txt }}} rather than {{{ wget http://bitbucket.org/okfn/ckan/src/default/pip-requirements-metastable.txt }}}" 982 1297850732000000 anonymous "This is now rolled into #963. Marking as duplicate. People can get the pip from a branch over HTTP like this: https://bitbucket.org/okfn/ckan/src//path/to/file/you/want" 982 1298482394000000 dread Need to do this for older branches which isn't subject to #963. 982 1298887980000000 dread Buildbot scripts now fixed. 983 1297773407000000 dread "Error was tracked down to cset:214a8f9fc1c2 (26-9-2010): upgrade_db called validate_authorization_setup() which calls setup_default_user_roles(System()) Fixed in cset:9f51a1c8ac83 for 1.3 branch and merged to default." 984 1297628554000000 kindly "These are the errors listed in severity order. * group revision tables joins wrong in many ways. * changemask table not added. * licence_id wrong type. * package_revision.download_url and changset.status not dropped. * package.name and tag.name unique constraint not added. * update cascaded defined wrongly. Attached are the fixes that will need to be run in ckan_migration_fixes.sql" 984 1297682116000000 kindly fixed see https://bitbucket.org/okfn/ckan/changeset/d56ea86d4303 985 1298571248000000 pudo digitialiser.dk has been assigned to Stefan Marsirske to get him into this Framework, everything else is delayed. 985 1303118298000000 thejimmyg We'd like CKAN to CKAN harvesting this week if possible. 985 1306407617000000 amercader Moving this to the PDEU (publicdata.eu) project 985 1306408134000000 amercader Duplicate of #1155 and #1156 986 1297812401000000 wwitzel3 https://bitbucket.org/okfn/ckanext-qa/src/be57e20c60ef/ 987 1311177705000000 thejimmyg This has largely been implemented now for publicdata.eu. There is a CREP #1134 outstanding to take the harvesting to the next level so marking this one as duplicate for now. 989 1297700363000000 kindly "It would be nice to know some use cases. I think that plugins should control their own storage, or share a storage that is designed to be flexible (mongo, redis ...). We do not seem to be able to keep our current migrate repository in sync let alone add plugins to the mix." 989 1297700818000000 pudo "Kindly, I agree - it would be much preferable to have independent storage for plugins and this would be easy to do if we were using another type of storage already. As it stands, however, our storage mechanism is SQL. I think we should use it for what it is as much as possible and do the weird, vertical stuff (k,v tables, swapping to redis) only if we really need it. For everything else: lets use SQL as it was intended. Examples: * We want to develop an apps catalogue as a CKAN plugin. While we could certainly put this in Redis, there is no reason why we can't have the following table: application (id, name, title, description, author, project_url, site_url, code_url, image). * A watchlist plugin could essentially work on UUIDs alone. What you'd end up with is something like this: watch (id, user, scope_id). Re migrations you're right, but my first intention would be to handle that seperatly for each plugin (i.e. they need to have their own migration repositories that they keep track of, e.g. via an apps_migrate_version table)" 989 1297706620000000 kindly "I do not think we need to 'extend the model' if you intend to make the migrations separate. If the schema is decoupled, then there are no problems. So each plugin can have its own model and use sqlalchemy independently i.e have their own metadata, classes and mappers. They do not have to even use sqlalchemy. What I mean is that there is no need to do anything apart from. * Agree on a naming convention of the plugin tables (including their own migrate table each) * Agree to the rule that no plugin can add a column to an existing table. * Agree that no table can have a (database level) foreign key constraint between the core tables and itself in either direction. They *can* have implied sqlalchemy level joins. * Maybe have a hook that on db upgrade all plugins are upgraded. Each plugin will have to redefine the tables, classes and mappers they need to join onto the core tables themselves. reusing/extending the core model will not be worth the trouble. This seems to cover your use cases and this way everything is nicely decoupled. Best of all there is very little work to do... " 990 1311180850000000 thejimmyg This now works. 991 1298037717000000 dread Fixed in cset:56cccbbb9d1a in time for ckan 1.3 release. This did not affect previous releases. 992 1298060474000000 rgrp Fixed in cset:08548ef8f0e9 993 1298373114000000 dread Fixed on 1.3 cset:7708c8b521ed and merged to default. Deployed to ckan.net. 994 1298912830000000 kindly see cset:93188d42fc12 995 1300364316000000 thejimmyg "Also, update the docs: Documentation article on caching / improving performance. (To complement configuration docs.) * Different sorts of cache - beaker style, etags, package_dict in search results(?) * How each one affects performance * How to turn them on/off and configure them * Is it possible to bypass each of them in the browser or with wget/curl? Therefore closing #841." 995 1311175353000000 thejimmyg The caching still isn't brilliant but there is less urgent need to refactor it. In the future our caching methodology should be to cache at the logic layer so we don't need to refactor the existing caching at this stage. 995 1311179009000000 thejimmyg "See also previous tickets: * #537 Caching and Performance improvement * #543 Investigate partial page caching and edge-side includes" 996 1300364398000000 thejimmyg This is now fixed, the outcome will be used to inform #995 for the caching. 998 1298369862000000 dread 'paster db create' (or init) should do exactly what we ask. Surely we should simply tell people to use 'paster db upgrade' instead? 998 1298371191000000 anonymous I am happy to get rid of paster db create altogether as a compromise? Or add a depreciation warning to it? 998 1298372171000000 dread Yes I agree - either of those sounds good. I think I've always used 'db init' in preference anyway. 999 1300707328000000 rgrp Done. See https://bitbucket.org/okfn/ckan-ckan.net and updated ckan.net.ini. 1000 1298912726000000 kindly fixed cset:630513f550d5 1001 1300367555000000 thejimmyg Yes, I think the two should be consolidated, but as was noted elsewhere (I think) we don't want the API to ever be authenticated via a cookie, otherwise we open ourselves to CSRF attacks. Instead we could support the API key as a query parameter as an alternative to via the HTTP header if it is hard to set an HTTP header in JS libraries? 1001 1300625888000000 pudo "My vote on this would be to enable ""httpbasic"" auth via repoze.who and to add an API key challenger to ckan/lib/authenticator.py that accepts API keys, from within repoze.who. This way we won't have to work around the framework at all and can retain compatibility to the existing mechanisms." 1001 1300703779000000 rgrp First work in cset:6b8cbe465b61 1001 1300704249000000 pudo Wrote a WDMMG version of this yesterday: https://bitbucket.org/okfn/wdmmg/changeset/398ce0eb3901#chg-wdmmg/lib/authenticator.py 1001 1301229082000000 rgrp Completed and merged into default in cset:cb200f339dbb. At the moment have done the very simple option but pudo's approach seems better and we could refactor towards that going forward. 1001 1301307814000000 dread You mentioned writing tests? Also the CSRF question from James hasn't been addressed. 1001 1301310351000000 rgrp This ticket did not include CSRF matters - please raise in a separate ticket if you wish. I looked in some detail at the testing options and nothing seemed simple to do that didn't duplicate code we already have (since no way to access the base controller halfway through a request). This can be tested as necessary as part of the development of the new js work. 1002 1311173185000000 thejimmyg I've done this in my branch and it will get merged in. 1002 1311675754000000 dread "Can you also delete the ""paster changes"" in ckan/lib/cli with this removal?" 1002 1311676199000000 dread Reopening this, since it has not been pushed or merged in afaict. 1002 1322515684000000 dread I've redone these changes in cset:91258a6296db71d78a on master, aiming for 1.5.2. 1003 1298490126000000 rgrp Work so far in http://bitbucket.org/rgrp/ckanjs 1003 1298889293000000 rgrp Have now started refactor to use backbone and have basic inline editing working and started on Add dataset view. 1003 1300100411000000 rgrp Converted completely to backbone and now have fully-operational add dataset functionality. I'm closing this ticket now - further improvements can be in there own tickets. 1004 1311173560000000 thejimmyg "At the moment it says ""Create a new group"" and clicking it takes you to the login page. We want it to do the same thing but if you aren't logged in the label should say ""Log in to create a group""." 1004 1323171375000000 thejimmyg The yellow box on the right is now back and you don't get taken to the login page. We'll write more detailed instructions once the group refactor is done. 1004 1323172489000000 dread "There is a description of groups, but that is not the subject of this ticket. In fact if you're not logged in you do get taken to the login page (which is quite correct). I agree with your earlier comment that the text of the link needs changing. Sorry to be reopening this, but I think the previous comment misses the point. Doing this ticket would be a valuable quick fix. Putting on backlog." 1004 1323174597000000 thejimmyg "I don't understand. We just had a team meeting about this, all discussed it and agreed to close it. Yes, you get taken to the login page, but that is the correct behaviour. The problem in the past was that the link was in the yellow box and there was no explanation as to why you had to login. This time you click from the header bar and there is a clear message saying ""Unauthorized to create a group"" - exactly as a user would expect. Even if the exact text of the ticket description isn't fully implemented in the current release, the UX isn't broken anymore. Yes, it might be even nicer to have a message warning them in advanced but these improvements will be taken forward in the UX work - maybe there is an even better solution than a message? Since you are unhappy about closing it I'm marking it as ""Duplicate"" of #1521. As agreed earlier with the entire team, we'll take this forward as part of the groups refactor. See #1521 for more information. " 1004 1323177994000000 dread "My suggested fix: {{{ diff --git a/ckan/templates/group/index.html b/ckan/templates/group/index.html index 8502df9..064e066 100644 --- a/ckan/templates/group/index.html +++ b/ckan/templates/group/index.html @@ -9,7 +9,15 @@
  • What Are Groups?

    - Whilst tags are great at collecting datasets together, +

    Whilst tags are great at collecting datasets togethe +

    + + Creat + + + To create a new group, please first ${h.subnav_link(c, h.icon('group') + _('List Groups'), controller='grou

  • -
  • +
  • }}} BUT this fix doesn't completely work. When you log-in and create a group, at this point the nav bar changes from the (correct) ""Add a group"" to (incorrect) ""Log-in to add a group""." 1004 1323187352000000 dread "Here is the unclipped version: {{{

    Create a new group To create a new group, please first log-in.

    }}} {{{
  • ${h.subnav_link(c, h.icon('group_add') + _('Add a Group'), controller='group', action='new')}
  • ${h.subnav_link(c, h.icon('group_add') + _('Log-in to add a Group'), controller='group', action='new')}
  • }}}" 1004 1323194016000000 zephod "Dread, that fix wouldn't work because the class attribute was being defined in Genshi and again in HTML, with slighly undefined behaviour (in this case Genshi overwrites HTML). I have performed a very very simple fix which modifies the navbar text when you're not logged in. https://github.com/okfn/ckan/commit/a89a48731ba548170045a60ac2930e0019c299c7 I don't think the link should be restored in the sidebar, it was explicitly removed as part of a site-wide sweep to make the sidebar a passive, helpful element rather than an active element with action links. Site looks great to me, I'm closing this ticket." 1004 1323195485000000 dread "Great stuff zeph! Diff looks really good and cheers for the explanation. On master, cset:a89a48731ba548170045a60ac2930e0019c299c7 and I've cherry picked this for release 1.5.1. too." 1005 1300100085000000 dread Moved to dgu trac ticket 869 1006 1298631145000000 dread "This command is slightly different to your branch policy as of two weeks ago: {{{ stable: stable code metastable: (will soon be deprecated) for code preparing to be stable default: development HEAD }}} which I prefer. My ideal would be to get rid of the confusing name 'metastable' and unneeded 'stable' and start a new branch called 'released', which will act the same as 'master' in this diagram but with a more intuitive name: http://nvie.com/posts/a-successful-git-branching-model Then for each ckan instance we can either use the most recent release (from 'released') or choose a specific one (e.g. 'ckan-1.3' or even 'default' or 'enh-865' for getting latest features). This gives a good degree of flexibility, is more understandable to newbies and probably a more widely understood branching model." 1006 1300098217000000 dread Reassigning to rgrp for response 1006 1300105927000000 rgrp "Not sure when i described that other process but I'm definitely of the opinion that we should: * Deprecate and remove stable branch * Deprecate and remove metastable branch (going forward we can use release branches for what we did with metastable in the past) May want to update BranchingPolicy to reflect this (also should probably have some statement about closing release branches and tagging) Reassigning to David Raznick as he is our release guru now." 1006 1300372286000000 thejimmyg I'm updating the branching policy now. David has closed stable and closed metastable. Marking as fixed. 1008 1298821826000000 rgrp I've removed the eval in cset:1b8fedeb7ab0 - the more general question about caching should go in a separate ticket. 1009 1298638447000000 pudo "Some more ideas: * /user should list users, sorted by number of packages contributed/editied * /user/{name}/packages shows a list of packages to which users have contributed " 1010 1298733856000000 rgrp Complete see branch feature-1010-list-users and closing changeset cset:feature-1010-list-users. 1010 1298740889000000 rgrp Meant this cset:d2651db566ef 1011 1298820235000000 sebbacon On reflection, may as well make a Plugin interface called IAuthorizer, which allows customisation of get_authorization_groups, get_roles, and is_authorized.... 1011 1298821699000000 rgrp I agree that IAuthorizer is useful but not sure how it addresses the requirement of the ticket. AuthorizationGroups are already editable via the web interface at /authorizationgroup 1011 1298824649000000 sebbacon "The ""external source"" is an Oauth service. We need to lookup user groups from that service." 1011 1298825600000000 sebbacon Proposed implementation at https://bitbucket.org/okfn/ckan/changeset/187e65afb35f 1011 1299245206000000 sebbacon Merged to default https://bitbucket.org/okfn/ckan/changeset/e8217c317a8e 1012 1300217601000000 thejimmyg "See also #103 which says: As a user I want to view a package at a given revision: * When I visit /package/read/xyz?rev=yyy I should be shown package at revision yyy * package history page should provide links to these pages Could you please investigate how doable this would be given the current package read.html page. I expect it could be very easy once the revision API is in place because we could build a c.pkg object from the revision data instead of the model data. Perhaps another case where we could try introducing the dictization?" 1012 1300352334000000 kindly "This is entirely not trivial at the moment. Hopefully after the dictization it will become more so. Simply putting in the package revision object in place of the package does not work. It will obviously work for changes to the package object itself. However, there are no mappers on that object for getting out the related package_tags, resources and extras at that revision. You will have to construct a fake pkg object with some messy and painful queries using dates." 1012 1300362584000000 kindly "cset:5649d6e761fc The basic revision history is merged. I will keep this ticket open if it is not sufficient. All it does is give a list with the most recent first of revision_ids, authors and timestamps. i.e {{{ [{""timestamp"": ""2011-03-16T15:55:19.941961"", ""author"": ""southampton-ac-uk"", ""revision"": ""202e9eb8-afaa-4bc9-b8a1-a317561547ea""}, {""timestamp"": ""2011-03-15T17:59:16.430804"", ""author"": ""southampton-ac-uk"", ""revision"": ""8235bd0a-d39a-49e0-887a-b0f231be8a92""}] }}} " 1012 1300364620000000 thejimmyg The API part is now done. Bumping the view history part to 1.5 1012 1300368449000000 dread As I understand it, the whole point to vdm is to make it easy to get old versions of not only the package, but its related objects (like tags, extras). David take a look at this function: Package.get_as_of(revision) 1012 1301909937000000 thejimmyg No-one really seems to have requested this part. 1012 1301943096000000 dread Changing this to 'Fixed' and #103 to 'Wont fix' to ensure this feature is noted. 1013 1299245157000000 sebbacon "This is now resolved, but depends on core CKAN behaviour (specifically pluggable middleware and unicode-aware error pages) to function: https://bitbucket.org/okfn/ckan/changeset/c846794c1799 " 1014 1299245293000000 sebbacon Run out of time for decoupling, but tests and README.txt written (including pointers about how to customise for anyone who needs to decouple in the future) 1015 1298902753000000 kindly The migration fixes should sort this out, but I will keep the ticket open to check. 1015 1299788821000000 rgrp Today kindly applied the sql fixes and I can confirm this is now fixed. Well done kindly for all the great work here. 1017 1299668555000000 pudo fixed in cset:86d49a775fd3 1018 1299073340000000 dread Done in cset:e4167f8b3f80 on default 1019 1299166930000000 pudo fixed in https://bitbucket.org/okfn/ckanext-webhooks/changeset/034647931921 1020 1300196215000000 thejimmyg This is now on DGU live. 1021 1299518828000000 pudo fixed in cset:9f1b38add19f 1022 1299512991000000 pudo "We're now using fileConfig to configure the logger API from the worker config file and this enables us to use SMTPHandler to send out error messages on queue processing failures. Marking as fixed. " 1023 1299514847000000 pudo "Tried implementing this with AMQPs msg.requeue() and channel.basic_recover() but RabbitMQ yield a NOT_IMPLEMENTED error. Bit clueless on how to proceed. " 1024 1299668648000000 pudo re-created this (#1027) 1025 1299598708000000 dread Done in enh-1025-config-default-authz and goes into release 1.3.2. 1025 1299750097000000 rgrp (Link to changeset). Could you briefly clarify why this was needed in config -- we already have a process for putting things into a more restricted mode (see ticket:833) and have been working on creating a WUI to be able to do this automatically (see that ticket). 1025 1299751045000000 dread "Thanks for pointing this out. Although changing the definition of 'editor' to not allow edit (as you admit) is a bit hacky, and I think prone to confusion. James and I weren't aware there was a precedent for doing it this way, but if we had, we may well have followed it. I mentioned the branch for this cset, in preference to the csets." 1026 1299605128000000 dread Done in cset:2dde3bd563fd for branch 1.3.2 1027 1299666326000000 pudo " 1. home controller -> __before__ (check ""site-read"" on model.System) 2. user -> each individually (repoze-who pseudo action must not be blocked) * user-read (index/read/update pages for users) * user-create (register) 3. revision -> __before__ (check ""site-read"" on model.System) 4. tag -> site-read (__before__) functional/test_authz.py * denies site-read ... * checks for visitor / logged in user .. * checks you can still visit login " 1027 1299682082000000 pudo fixed in cset:532c3ad2743b 1028 1299760360000000 dread Fixed in cset 4d860a53fbad on 1.3.2 and merged to default. 1029 1301311643000000 thejimmyg This was fixed by kindly as part of the 1.3.2 release. 1030 1300103984000000 amercader "This includes: * Take out all references to harvestsource, harvestingjob and harvesteddocument in the rest API * Move the harvesting bits of ckan/lib/cli.py into ckanext-harvest. * Move ckan/controller/harvesting.py and cknan/model/harvesting.py to ckanext-harvest as well * Update ckanext-csw to be able to find the code it needs in the new place." 1030 1300209975000000 thejimmyg "Also: * Delete harvest sources * Implement a Genshi template filter so that if there is an INSPIRE package_extra set to True, you can link to the XML and HTML generation API calls in ckanext-csw * Write tests in ckanext-dgu for all the API calls that Drupal is making " 1030 1303117978000000 amercader All the points described in this ticket have been completed. Harvesting work is being done mainly under #1037 1031 1299863222000000 rgrp Done, see cset:78d96b520679 1032 1300380606000000 rgrp This is urgent for me as without upload does not really function. 1032 1303117292000000 thejimmyg Rufus, I think this is potentially quite a big change and needs to be done as part of the entity refactor rather than as a quick hack. I'd like David Raznick to work on timeseries this week so I'd like to understand a bit more about this ticket. Could you flesh it out please or let's discuss in the catch up? 1032 1303118226000000 thejimmyg See related ticket #922 1032 1304094162000000 rgrp Agreed to defer this. Resource editing, to the extent it happens can be kept very simple and part of package editing. 1032 1330348463000000 zephod Streamlining supertickets. #1506 is the new parent of #978; all tasks remaining in this ticket are complete. 1033 1308310446000000 dread Fixed in 5c7caca30737 for v1.4.1 1034 1320174353000000 dread Same problem as #1321 I believe 1035 1323172908000000 thejimmyg Re-assigning to me temporarily to investigate now that Seb has left. It may be that this should be deleted given that it is over 6 month's old. 1035 1338206251000000 ross Removing until we can come up with a case for implementing 1037 1303117000000000 thejimmyg "We spent last week integrating the new harvesting architecture and testing the code but there are still some areas that need looking at * The source type and label should be part of the plugin, not named in DGU. * Need warnings if a document changes but its date doesn't -> do we have these? * I noticed there are some tests in DGU, should these perhaps be in ckanext-harvest? * If active is False, the job should not be put on the queue * Log if the wrong type of URL is entered as an error the user can see * Deny if the source is already registered * Overwrite all extras, not just merge new ones. * During the import stage use iswms.py to add an extra during import if it is a WMS so that we can add a link to the WMS later https://gist.github.com/900878 * Can errors/warnings be logged in the import stage? Do all fetched documents get passed to import in one go? " 1037 1304937601000000 thejimmyg Closing this now, any outstanding small issues will be logged in new tickets. 1038 1300212841000000 dread Done in cset:0373737f0866 for release 1.3.2 1039 1300212856000000 dread Done in cset:97d563bbf01d for release 1.3.2 1040 1300213922000000 dread Fixed by James in 904f1f36a672 for 1.3.2 1040 1328805778000000 dread "The problem arises when you do two things together: 1. Have code like this in a CKAN extension's plugin.py: {{{ config['extra_public_paths'] = ','.join([our_public_dir, config.get('extra_public_paths', '')]) }}} which will have a trailing blank value (unless you set extra_public_paths in the config file) {{{ extra_public_paths=['/some/path/', ''] }}} 2. Run CKAN with this code: {{{ extra_public_paths = config.get('extra_public_paths') if extra_public_paths: static_parsers = [StaticURLParser(public_path) \ for public_path in \ extra_public_paths.split(',')] + static_parsers }}} Introduced in #340 https://github.com/okfn/ckan/commit/0593c3e5050882a9dbf54fb6c263f4a2f1de2097#ckan/config/middleware.py in CKAN 1.0.2 This was made safer in https://github.com/okfn/ckan/commit/e0d5f02168116a18a7aeb7b058e4160bbd402a02#ckan/config/middleware.py for CKAN 1.3.2 (it ignores blank paths) " 1040 1328806824000000 dread "Here are all the CKAN instances potentially involved in this issue: == ckan.net / DGU == These suffered this problem briefly and were fixed 15/3/11 == opengov_es http://opengov.es/ 1.2 == This does not have a theme extension. Is not serving files from root. == nosdonnees_fr http://www.nosdonnees.fr/ 1.3.1a == This does have a theme extension but does not seem to have a trailing blank public path since it is not serving files from root. " 1041 1300284809000000 thejimmyg Also, this means moving documentation out of the current CKAN dev wiki at trac.ckan.org and into the community wiki. Core developer docs should remain in the Sphinx-based .rst format in CKAN. Much-needed new tutorials go on the community wiki. 1041 1310135309000000 thejimmyg Child ticket of #1142 1041 1312372367000000 rgrp James says this is not priority and he does not plan to do docs for debs until he has got that system functioning. 1042 1300969265000000 dread Done in ckanext cset:7610512277bd and ckanext-importlib cset:7737716e0d7b. 1042 1300969865000000 dread "Remaining parts of ckanext are: * blackbox (rgrp, pudo, dread) * hammer (ww) * link_checker (ww) * solr configuration (pudo) & indexer (pudo) * amqp listener command (ww) * geo form field (rgrp) * GData Loaders: * aidprojdata (johnbywater) * norwaygovdata (johnbywater) * ordsurv (johnbywater) * Gemini2 (iso19139) (johnbywater, ww)" 1043 1300321033000000 kindly "cset:c894f92c5b9a Sesion.remove() needed to be run before configure as we want the original session made not to be used over the newly configured ones." 1044 1300359562000000 rgrp Think this would be resolved by refactoring implied in ticket:1001 (specifically checks for api key and remote user should lead to availability of same set of identification information). 1044 1300382367000000 dread "I agree with leaving #1001 solving this issue. Centralising this is good. This ticket can instead focus on: * returning the correct 403 error (rather than the 302 currently) * creating some tests * documenting * consider renaming the SITE_READ variable and rethinking purpose" 1044 1300449905000000 dread "I've added in tests for SITE_READ and corrected the 403 response - see branch bug-1044-site-read. The tests have uncovered some inconsistencies in the use of SITE_READ between reading/editing a package in the Web interface vs the REST API. Friedrich, do you want to take a look at ckan/tests/functional/test_authz.py:TestSiteRead.test_pkggroupadmin_read for details and have a think about how this should best work. How about making SITE_READ more widespread in the WUI, to protect read/editing/searching things, in addition to the READ and EDIT RoleActions. Once that is decided, it should be pretty easy to refer to the tests and document SITE_READ." 1044 1300627007000000 pudo "David, thanks for writing those tests - perhaps we should combine them with the ones below (""TestLockedDownUsage"") which attempt to basically do the same. As for the inconsistency introduced by the global check in the REST API you're right: Of course it is strange that WUI access checks are more granular than the API checks. The alternative is that we either move authz checks into all relevant REST places (a major refactoring I would be suspicious of) or that we introduce duplicate checks on the WUI actions (I actually have performance concerns about that, authz is incredibly slow - and it introduces another level of special authz that I think we really should not have). Given the choice I'd opt for the REST refactor - there is no good reason to make SITE_READ a duplicate check where authz already applies. On the other hand, this is a function we really don't want to enable or even have and that was only added to satisfy some very specific user demands. Given that these are fulfilled, I'm actually OK keeping the inconsistency for now - nobody will see it in normal operations and in a locked-down environment, users will need to have API keys anyway. Regarding the naming, I'm pretty opionion-less: SITE_READ was proposed by rgrp and I think its pretty fitting, while OTHER_READ would just confuse me. PUBLIC_READ might work, though. " 1044 1300712223000000 dread "I'm happy to leave the inconsistencies in SITE_READ for now, since this will be resolved when authz is centralised between WUI and API in the dictization refactor. I'm also happy to leave both TestSiteRead and TestLockedDownUsage as they complement each other nicely actually. So I think all that remains for this ticket is to add a paragraph to the authz documentation about SITE_READ. Pudo would you mind doing this? It's well worth noting where it applies and where it doesn't." 1044 1301304871000000 dread I think all that remains for this ticket is to add a paragraph to the authz documentation about SITE_READ. Pudo would you mind doing this? It's well worth noting where it applies and where it doesn't. 1044 1302096155000000 dread I've added in the docs for this in cset:013da53052d1 ready for release 1.3.3. 1045 1300457708000000 dread Fixed in cset:98eb8b8d063e on default, scheduled for CKAN version 1.4. 1045 1300787161000000 dread Want to identify groups by ID also in the Web interface. 1045 1300793261000000 dread Fixed in cset:71621df50983 on default ready for ckan 1.4 1046 1302777668000000 kindly cset:35ba6ad033ae 1048 1300702752000000 dread Done in changeset 9d7bfa124757 on default and expect it to go into 1.4 release. 1049 1323169424000000 thejimmyg This is over 6 months old. In accordance with wiki policy, closing. 1050 1300452200000000 johnlawrenceaspden there's already a file ckan/tests/test_authz.py, which looks like the appropriate place for new tests. 1050 1300458454000000 dread "I may misunderstand how you intend to build the WUI Admin interface, but I think most of the stuff in authztool is just marshalling command line parameters anyway. The bits which do anything are factored out. For example, to list rights you just loop over obj_classes to call {{{model.Session.query(obj_class).all()}}} and then display the values: {{{type(subj).__name__, subj.name, role, type(obj).__name__, obj.name}}}. To change a right you simply call {{{model.add_user_to_role(subj, role, obj)}}} or {{{model.add_authorization_group_to_role(subj, role, obj)}}}. But of course if there is useful stuff to factor out then be my guest!" 1050 1303118188000000 thejimmyg This should also feed into #1075 which will be being worked on this week. 1050 1315394117000000 dread There's been no objections to my comment 6 months ago saying this is invalid, so marking it invalid. Feel free to reopen if there is indeed a case here. 1051 1301305079000000 sebbacon This has been completed in https://bitbucket.org/okfn/ckan/changeset/64a949990e0b 1052 1300895410000000 dread Fixed in branch feature-DGU#889-authorization-in-lockdown and merged to default at cset:e6643cf1324c. Only remaining issue is listing in Web interface of package relationships, which shows links to packages even to ones you can't read. 1053 1310125444000000 thejimmyg This is now fixed as part of the API refactor and revisioning updates David Raznick has done. 1053 1310126546000000 dread "Looks to me like DELETE always changed the state to Deleted, so an invalid ticket. We can deal with purging in another ticket if needed." 1054 1300989764000000 dread I've added some tests in the branch: defect-1054-resource-order 1054 1300992195000000 dread I've put in a potential fix for this into the branch - do take a look and see what you think. 1054 1301002995000000 kindly "Looks great. Had a look and changed a minor thing because I was not confident with the handling of the null values. I made a fake resource and did an asdict on that to make the identity. Part of my patch I reverted as I mucked up my version of the vdm, so it was not needed. " 1054 1301305615000000 dread This branch is now merged. 1055 1300987241000000 dread This amounted to 14 tests. Fixed in cset:5bbd0005c57e ready for ckan release 1.4. 1056 1302626373000000 dread Fixed in cset:a025b5b3293c on default. 1056 1302882616000000 dread This was broken in cset:b681bbedfa62a68b71260ef48a0da6063109734 which was released in 1.3.2 1057 1329150236000000 toby fixed in commit 3d7cbf0 1058 1301303221000000 dread Fixed in cset:8e1817ab8d1c on default, ready for 1.4 release. 1059 1301312516000000 dread Fixed in ckanext-importlib cset:f6d19129ac43. 1060 1301312487000000 dread Fixed in ckan-importlib cset:bc52bba86d71 and ckan cset:2751f76fb17a in time for ckan 1.4 1061 1301397271000000 dread Fixed in cset:fc3bc103db8c ready for release 1.3.3 1061 1301922325000000 dread Actually there was a link at the bottom of the package edit form that my grep didn't pick up: h.url_for(str('/license')). So it is not orphaned at all. 1061 1301922350000000 dread Backed out original change fc3bc103db8c here: 7ae9aff8bc68 1062 1311178145000000 thejimmyg "Hi John, Can you please check that the new webstore+preview works correctly with this one please? Cheers, James" 1062 1311773731000000 johnglover The file is in Turtle format (serialisation format for RDF, which we currently don't archive and have no preview code for. The new preview code will not even create a preview button for this file type. Moving to the backlog for now, support for RDF and related formats should be added along with support for excel docs and google spreadsheets. 1063 1301522483000000 rgrp "This sounds absolutely sensible and suggest we even have a helper method such as lib.helpers.group_link(group) (as we do for user_link). (@Seb: apologies for misunderstanding your query on the list -- I was talking about the group listing table at /group)" 1063 1302514033000000 sebbacon Fixed in https://bitbucket.org/okfn/ckan/changeset/145eaae3168a 1064 1323169787000000 thejimmyg Marking as duplicate because #1464 will take it on. 1065 1301914004000000 sebbacon "To reflect further conversations: * We are parking this ticket until dictization is complete * We would prefer to see roles as asserted globally by default, with packages just storing exceptions. E.g. If I am a ""reader"" globally, then I have ""read-package"" permission on all packages new and existing, unless otherwise asserted locally on a package. * Also w.r.t. algorithm above, consider repoze.what's model; ""possible"" is analagous to ""has_permission"". See http://what.repoze.org/docs/1.0/Manual/Predicates/Builtin.html" 1065 1302033244000000 dread "Wow, lots of solutions here, and not enough evidence of the problems encountered / difficult use cases. I have tried here to extract what John and Seb might be suggesting is difficult with our current http://en.wikipedia.org/wiki/Role-based_access_control system. 1. (Seb) It seems crazy to change the User-Object-Roles for every single package to 'lockdown' a CKAN instance. Instead of using Roles 'reader' & 'editor' we start using Roles 'logged_in_user' and 'anon_user'. Then, with only changing a couple of lines of the Role-Action table for these Roles, you can give or remove permissions to edit / create packages, groups etc. 2. (John) Actions are being added all the time by migrations and extensions. I'm not sure what the problem is here - can someone explain? The separation of Users and the new Actions by adding the Role 'join' means the migration/extension can add existing Users with the new Actions. For example when we added the 'Group' protection object, logged in users could previously create groups, so it is natural to add this action to the 'editor' role. If you get rid of Roles, as John suggests, then you end up having to add as many Rights objects as users. And you might end up trying to infer new Rights on the basis of old Rights, rather than a role, which would be more natural. (Is the only reason to denormalise the UOR and RA tables the sake of ease of understanding? I would think we can explain the concept better now...!) 3. (John) This Group Hierarchy description sounds a bit like earlier incarnations of the DGU requirements. ""or any of his groups group-children (but not user-children)"" I'm not sure I understand the user-children bit, and I'm wary of any difficult-to-grasp concept. This modelling seems to take the UserGroup/User hierarchy model and use that as a hard-coded interpretation of the Authorized permission. In contrast, in DGU Evan split the Organisation Hierarchy model from the Authorization model. Each User has explicit permissions for each Organisation he is allowed to add/edit packages for. This helps for these two use cases we have to contend with: when a user is an admin for unrelated branches of the Organisation hierarchy, and when an Organisation is actually in two branches of the Organisation hierarchy. It also seems more generally flexible. " 1065 1302075994000000 sebbacon "Re the point against my name, yes -- I think the fundamental principles of the current system are fine, but the implementation means asserting things for every single object, whereas we should be able to optimise it for the general cases -- something along the lines you suggest. I would be very happy to write up a full, detailed proposal for the system before we implement anything, if only as a proper straw man to have debates around." 1065 1302081747000000 dread Great stuff Seb, cheers for that. 1065 1303118756000000 thejimmyg This is now more urgent and I think we have enough consensus to go ahead. See #1094 too. 1065 1311174208000000 thejimmyg Merging with #1094 so see the discussion there too. 1065 1318528721000000 rgrp Reassigning to zephod as he is looking at this. I think most of this is wontfix or will become obsolete with other stuff but worth looking at. Also moving to v1.6 at the same time. 1065 1324550041000000 dread "All subtickets are done now, apart from #1198 which I've moved into superticket #1477 Group refactor Change name of AuthzGroup? to UserGroup? to reflect what it is for I think this is completely covered off in #1477 Group refactor Get rid of Roles, and replace them with direct assignment of actions, even though there are many actions, and extensions can add arbitrary ones. Debatable whether we should cut the number of actions to correspond to the three roles defined by the base system. This is not clear - please raise a ticket for this if still promising. Have a method of finding roles (or, in future, actions) relevant to a given protection object (e.g. FILE-UPLOAD(ER) not relevant to Packages) These three points I've moved into a new ticket #1596 as it was getting lost rather in this one. Change UserGroups? so that they can have a hierarchical structure, I think this is completely covered off in #1477 Group refactor Conclusion: closing this ticket as all points are covered or moved into tickets elsewhere" 1066 1301904246000000 dread Need a new role 'ANON_EDITOR' which is the default role for Visitor, which can create packages, but not groups. 1066 1301930311000000 dread "Migrations: * Default (open) CKAN instances will have visitor as a reader on the system, and will have to upgrade them to anon_editor. * DGU & DataGM instances have visitor as a reader on system and must stay like that. * Pudo's specially locked instance, where editor has been changed to only read can now use the 'reader' role (assuming he's happy for them to create users) Migration script (37) is designed to cope with the default (open) setup, and the other special cases must be dealt with using CLI at the time of upgrade: DGU & DataGM must run: {{{ paster rights remove visitor anon_editor system: paster rights make visitor reader system: }}}" 1066 1301932136000000 dread Done on branch defect-1066-reader-too-permissive and merged into release-1.3.3 1067 1302186503000000 dread Done in cset:d75ab5fc3311 for release 1.3.3 1068 1302109505000000 dread "The problem was the use of mktime which is localised. Also spotted a problem with the use of localised time stored in the datestamp created in migration script 29, when initialising the repo. Fixed in a2094932e5e4 in release 1.3.3 branch." 1069 1301909423000000 thejimmyg "This is what the new ""Ideas"" section in the ckanext-community extension is for. Marking as duplicate of #941, can we have discussion there please." 1069 1302510609000000 rgrp Mistakenly assigned to this sprint. 1069 1310133794000000 thejimmyg Yes, and this will be more important in thedatahub.org 1069 1340632215000000 icmurray Assigned to @tobes as he's looking at publication workflows. 1070 1310117129000000 rgrp Added link to http://wiki.ckan.net/Domain_Model and closing as work mostly done (and nothing clear remaining to be done -- agreement perhaps!) 1071 1301943180000000 dread Done in cset:db9e2c4f65bb for release-v1.3.3 1072 1302106474000000 dread Done in cset:2a97d2d9ba4a for release 1.3.3. 1073 1302185825000000 dread Done in cset:9dfc60db90ed on default branch. 1074 1314278296000000 rgrp johnlawrenceaspden did quite a bit on this in feature-1074-refactor-authz-wui branch (including implementing for package and group). However, did not implement for authzgroup and also repeated code a lot. Reassigning to me for completion. 1074 1314283492000000 rgrp "Also work on branch feature-1074-authz-wui (that is primary one in fact) " 1074 1314303581000000 rgrp Closed in cset:18a5b48bbee6. Note the templating isn't completely reused but rather uses common methods from _util.html. Likely that a bit more refactoring could be done to share common code but not worth it at present. 1075 1303227982000000 johnlawrenceaspden "closing ticket see http://wiki.ckan.net/Authorization and https://bitbucket.org/okfn/ckanext-admin" 1076 1302515674000000 sebbacon See also http://trac.ckan.org/ticket/948 1076 1302731255000000 rgrp All main work done as of cset:556efcb91f19 in branch feature-1076-improve-purge-system (+ associated work in ckanext-admin trash bin management). All that remains is to add some tests. 1076 1303236302000000 rgrp Test of trash functionality in ckanext-admin in https://bitbucket.org/okfn/ckanext-admin/changeset/e86c102a3c60 so now closing (could do with some functional tests in the main interface but can leave for later -- were none before so situation no worse ;0 ). 1077 1319797284000000 dread I guess this is not a priority now. 1078 1305828965000000 kindly complete: cset:843b78bae016 1079 1302777496000000 kindly Complete see cset:35ba6ad033ae 1081 1303332388000000 johnlawrenceaspden Actually it seems to be worse than that. If I add a user to an authz group then I get added too, and then there's no way to remove me. 1081 1303489474000000 johnlawrenceaspden "Fixed as part of feature-854-tests-for-authz-groups branch. Original behaviour was that creating/modifying user is explicitly added to the users in the authz group. " 1082 1315317825000000 dread "The flash message 'Language set to: English' was always in the previous language, not the new one. Pudo fixed this in cset:eaf342823caf and I have transplanted this into release-1.4.3. I've also added/corrected translations for all languages for 'Language set to: English' so that it makes sense! ""Set language back to English causes server error:"" I can't reproduce this. The exception looks like a problem with the forwarding, which improvements were made in #1126." 1082 1315917217000000 dread Language switch problem has been around since it was introduced in the 1.3.1 release. 1083 1304975681000000 johnlawrenceaspden "Current behaviour: from the web interface, russianfan can be added as an admin on warandpeace twice. and you can then delete the two roles separately. desired behaviour is that you can only add him once. from the command line, you can only add him once, but if he's in there twice, you can't remove him because the command line bombs desired behaviour is that he can be removed from the shell, you can add him twice but can't remove him if he's been added twice. if you try, it bombs. The desired behaviour in all cases is that adding and removing are effective and idempotent in all three cases." 1083 1305051061000000 johnlawrenceaspden It appears to be possible to get into a state where you can't add or remove roles from the command line. Investigating. 1083 1305058621000000 johnlawrenceaspden "Fixed the problem where you can't delete a role if it's been added twice, but had to leave the add function as it was since putting a commit in it breaks many tests. It appears that the WUI doesn't call these functions anyway. Left that as it is since we're planning to replace it. The cli rights command doesn't appear to add the roles that it says it adds. Possibly because it's not doing the required commit? Reported this as a different bug." 1084 1303894555000000 dread ww you did this didn't you? Perhaps you could update us with the branch/changeset it was in and close this ticket please. 1085 1302621951000000 rgrp "I don't think this is very useful output :) Most of this is test/demo/rdf and therefore not relevant. Also have you checked your ckan.site_url in your development.ini -- which configures where what is used as based for all external urls in the templates? IMO there is nothing to fix here." 1085 1302624602000000 dread As discussed on IRC, the default ckan.site_url should be blank, rather than ckan.net. 1085 1302625314000000 dread Fixed in cset:01b3de5d4e31 on default. 1086 1303381200000000 johnlawrenceaspden There is no (obvious) way to delete an authzgroup. 1086 1323346552000000 dread Authorization groups are deprecated with the incoming Groups Refactor #1477 1087 1310134714000000 thejimmyg Child ticket of #954 1087 1320866159000000 dread "Implemented in cset:0b7e2b0bed44 as status_show logic action, heading for 1.5.1 release. Took 0.5h" 1088 1302630624000000 wwaites "you use the negotiate function in autoneg.py with a cfg dictionary like this, {{{ cfg = { ""mime_types"": [ (""application/xhtml+xml"", [""html""]), (""text/html"", [""html""], (""application/rdf+xml"", [""rdf"", ""owl""]), (""application/turtle"", [""ttl""]), (""text/plain"", [""nt""]), (""text/x-graphviz"", [""dot""]), ] for content_type, exts in negotiate(cfg, accept_header): if ""html"" in exts: render page normally else: redirect to semantic.ckan.net } }}}" 1088 1303035487000000 wwaites "fixed in 5c091fc08474 (bug-1088-autoneg-is-wonky) tip " 1089 1302631733000000 dread Done in cset:6d41ecc431e3 on default. 1090 1302635699000000 dread Fixed in cset:e0af88edf0f0 on default. 1091 1303746408000000 rgrp "This is a well known issue (see some discussoin in #142). There is nothing directly we can do about it (this is what google gives us) but we could make usernames editable (this has been discussed but not ticketed I believe). Given our general ""downer"" on openids I'm not sure this is a big priority." 1091 1323102767000000 rgrp wontfix plus we now requires usernames see #1386 1092 1305570822000000 kindly completed cset: ca1ac4112ea2 1093 1302865470000000 dread Fixed by this API call use the existing license cache. Done in cset:a95888ea39be on release-v1.3.4 1094 1303129670000000 rgrp "See also comments on the mailing list. Item 1 seems fine (what is difference from current extension mechanism?) Item 2: concerns here. What about list views? What about editing 'permissions'? I also think getting rid of System object isn't really a benefit (if anything may be a cost). Item 3: feel this may be better as part of big domain model change (also gives us time to really dig into this -- this is an important requirement/conceptual issue). Item 4: No objections but seems very minor gain for fairly significant migration work. " 1094 1304354036000000 johnlawrenceaspden "UserGroup/PackageGroup might also be confusing. A PackageGroup is *just* a group of packages. A UserGroup is both a group of users, and a thing affecting authorizations. Perhaps PackageGroup and UserAuthorizationGroup? Or PackageGroup and AuthorizedUserGroup? I was quite confused by all this at first. I think I understand how the whole thing works pretty well now, and I still can't think of good names for the two concepts, although I can already feel the normal English meanings of the words changing to what I now know they are for. We should be a little wary of this. Things that are even slightly difficult to understand end up being understood by very few people. Would any of us be prepared to sit an exam on exactly how UNIX file permissions work, even though we all use them? " 1094 1311173649000000 thejimmyg Merging with #1065 and closing. 1096 1307444626000000 nils.toedtmann See also http://ckan.okfnpad.org/multisite 1096 1310125173000000 thejimmyg "Renaming to CKAN Hosted because whether we introduce a site domain object for subdomain tenants or not is an implementation detail. As far as progress on CKAN hosted goes, we now have: * A build tool to package ckanext-dgu, ckanext-std etc into .deb files * The ability to apt-get install CKAN * A script that can set up multiple non-conflicting CKAN instances on the same machine What we need next is the ability to generate a ckanext-xxx repository for each new instance we want to host. At that point most of the underlying techniques for hosting multiple CKAN instances concurrently will be in place. Once we have that working and know what clients want, we can look at going further and speccing up a multisite solution off a single DB, rather than the multi-instance solution proposed here. " 1096 1311182262000000 thejimmyg Milestone ckan-v-future deleted 1097 1303293476000000 dread Done in cset:5dec7eb609ab on branch release-v1.3.4. 1098 1303387896000000 dread Fixed in b2b37e39de4d in release-1.3.4. Was only broken on this branch a few days before. 1098 1303406017000000 dread Also found problem with some tests using delete_all. Fixed in cset:f9764dd43a0a release_1.3.4. 1099 1324057106000000 dread Authorization groups deprecated with the Group refactor #1477 1100 1303406103000000 dread Done in release_1.3.4 cset:580e2b11b4ef and release 1.3.3 cset:cac55b8a0bd9 1101 1330082908000000 dread Stats already in site nav - adjusting title to just mention google analytics. 1102 1303744552000000 rgrp (Same ticket created 3 times by accident I think -- see #1104). 1103 1303744575000000 rgrp (Same ticket created 3 times by accident I think -- see #1104). 1104 1303744486000000 rgrp "You haven't done paster search-index rebuild I think. If you haven't the search index will not have been built (Create Test Data code creates data without indexing it). This could be considered a bug of Create Test Data but I'm not sure -- if you think so re-open this ticket and change description." 1104 1303920492000000 dread I agree this should be fixed. 1104 1303920791000000 dread Fixed on default cset:52a3fb230074 - create-test-data now indexes the package it creates. 1106 1303834069000000 rgrp fixed in branch:defect-1106-bugs-with-routing. 1107 1310134646000000 thejimmyg Child ticket of #954 1107 1320321170000000 dread "Notes to complete this: * ckan/logic/action/get.py:'def package_autocomplete' exists and is the one that should be used * need to ensure all places in the templates and public/scripts use this action directly, not dataset/autocomplete (or its alias package/autocomplete) * ckan/controllers/package.py:'def autocomplete' should be removed. * ckan/config/routes.py dataset/autocomplete should be removed." 1107 1340632612000000 icmurray autocomplete action has been deprecated in favour of /api/2/util/dataset/autocomplete : https://github.com/okfn/ckan/blob/master/ckan/controllers/package.py#L655 1108 1310128678000000 thejimmyg Now that we are using ckan.net as thedatahub.org we should go full steam ahead and create a brand new UI to reflect thedatahub.org's new role. See http://ckan.org/2011/07/08/ckan-vision-update/ 1108 1311180204000000 thejimmyg See also comment on #1200 about using the pdeu theme. 1108 1314878772000000 zephod "First, here is a repo with the new DataHub theme: https://bitbucket.org/okfn/ckanext-datahub-theme We plan to integrate this into ckan-core, thereby replacing the current theme. " 1108 1315140879000000 rgrp Completed in https://bitbucket.org/okfn/ckan/changeset/9be1ae232ec3 cset:9be1ae232ec3 1109 1303862352000000 kindly It is fixed now in 1.3.4.1. 1109 1303916487000000 dread Close ticket? 1109 1304698621000000 dread Did we decide that this facility for storing non-strings via the API is a new feature, rather than a bug? 1109 1305123852000000 dread What's the status? 1109 1305124697000000 kindly I am happy this is fixed in cset:445fc04333dd. 1111 1304935139000000 lucychambers "CKAN FAQ now complete (Sections 1,2,5,6 - needs uploading onto site): http://ckan.okfnpad.org/FAQ At CKAN Community Meetup it was decided that Sections 3 and 4 would form a separate contributor's manual. " 1112 1304358643000000 dread This is complete in the branch feature-1112-allow-search-all for release 1.3.5 1113 1304024611000000 kindly cset: 52a3fb230074 1114 1304085484000000 dread Done in cset:d7bd4b0f89de on default. 1115 1324054704000000 dread Authorization groups deprecated with the Group refactor #1477 1116 1304358664000000 dread This is complete in branch defect-1116-boolean-search-options for release-1.3.5 1117 1304277240000000 nils.toedtmann "Oh, it *does* depend against python-pastescript. Ignore. For some reason it was not (completely/correctly) installed with ckanext-datanl on us4, but is a different issue then. Might be due to missing locale ""en_GB.utf8"" and dpkg-configrue failing." 1118 1311174062000000 thejimmyg "We haven't ever come across this. Perhaps if you could provide an example we can re-open it. Cheers, James " 1119 1304104689000000 rgrp " * Merged ckanext-upload into ckanext-storage * Got existing API systemin in ckanext-storage fully working (e.g. api section was broken -- maybe due to changed boto ...) * API now supports GET requests and JSONP where appropriate * Added auth form to API (could be useful for ajax form stuff going forward) * Improved UI of upload including having ckan based file links (using redirects) * Set uploaded-by field on uploaded files * Checked on AWS And GS Also did: * Improved ofs in various ways including change to standard metadata handling and improved testing (see e.g. https://bitbucket.org/okfn/ofs/changeset/36fac29b7775) * Tracked down [https://github.com/okfn/boto/commit/0d9635c8a9785c9b20b44ee93a0679c002961592 bug with boto and metadata updating on GS] and provided patch." 1121 1307358426000000 dread We don't need to support JSON extras at the moment. 1122 1304368075000000 dread @pudo I notice you tackling solr indexing of lists/tuples in ckanext-solr cset:ecc2d69df991 so have updated to use this on ckan.net and reindexed, but #1122 still seems an issue - any chance you can take a look? 1122 1305718009000000 dread This user is now storing his list as a string so is happy. But I'll leave this ticket open, for pudo to decide whether there is an issue with SOLR indexing tuples/lists, as per the previous comment. 1122 1306662099000000 pudo I don't think there's really a nice way to do this with the current solr indexing library, but I've commented out the list handling code which should really make handling this cleaner. 1122 1306747706000000 dread "Changing this to ""won't fix"" just to be clear" 1123 1307374691000000 dread "I'll change the architecture to ""all"" as per this article: http://linuxreviews.org/man/deb-control/" 1123 1307790563000000 thejimmyg "This isn't necessarily a trivial thing. Let's get the build system for the packages stable before we start changing it all to support alternative architectures. Once the packaging is working well it would be trivial to switch those servers to amd64. I'm sorry, but this isn't worth the investment in manpower at the moment." 1123 1307795508000000 rgrp You can't get small amazon instances as 64 bit and many people's own machines are 32 bit. How costly is packaging no-arch exactly (it must be more than a simple 'switch' in the build if you are saying this is costly :-) ) 1123 1311863798000000 nils.toedtmann "Looks like the packages on apt.okfn.org are now architecture ""all"" - great. To avoid future confusion, i change this ticket's solution to ""fixed""." 1124 1304601771000000 nils.toedtmann I installed the python-ckanext-solr package rom http://apt-alpha.ckan.org/datanl-dev onto us10/DataGM too, and it seems to work just fine, see http://datagm.test.ckan.net/package?q=manchester 1124 1323168132000000 thejimmyg "Solr is now properly supported in the ckan-1.5 repository. " 1124 1323168156000000 thejimmyg Re-open this if it still affects DataGM. 1126 1304587078000000 dread Fixed in cset:61bb142a6b7c for ckan v1.3.5. 1127 1304602084000000 wwaites "The proposal is not comparable to PEP and DEP because the projects have vastly different requirements. The API stability needs of a programming language or an operating system (e.g. fundamental building blocks that you don't expect to change often or radically) are very different from a web application. The plone one is comparable. The idea itself is a double-edged sword. It will promote stability which is good but can also tend towards rigidity and stagnation which is bad. Each added bit of bureaucracy and process means fewer people will be willing to collaborate or participate in improving the software. Overall, -1. " 1127 1304606094000000 dread Fixed links 1127 1305128000000000 thejimmyg This is great idea and it has taken on well. +1 from me. We can update/create new CREP policies as needed. 1127 1305458020000000 rgrp "I'm obviously a strong +1 but feel we should stick with the CEP name (we have PEP and DEP and CREP really does sound weird compared to CEP). Also I think we may need to do a bit more to clarify when you just do a good ticket and when one does a CEP (looking at what has occurred already). I think this may be one reason to keep CEPs separate in drafting from trac as will clearly distinguish CEPs from standard tickets. " 1127 1305458489000000 rgrp one other thing: i think the plan is for CEPs to be in restructured text (for easy transfer to docs). This should be mentioned and the template in the wiki should be in ReST. 1128 1310567803000000 dread "Loader in ckanext/bin/loadscotland.py. Done upload to ckan.net: {{{ (pyenv-ckanext)dread@dread-laptop:~/hgroot/ckanext$ scotland_loader -k xyz -H http://ckan.net/api /home/dread/ckan-local/data/scotland-2010-10-18.csv /home/dread/hgroot/pyenv-ckanext/lib/python2.6/site-packages/pylons/templating.py:610: UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages) Engine = entry_point.load() No handlers could be found for logger ""vdm"" 2011-07-13 15:29:17,983 INFO [ckanext.importlib.loader] Loading scottish_neighbourhood_statistics 2011-07-13 15:29:20,109 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:29:22,089 INFO [ckanext.importlib.loader] Loading traveline_scotland 2011-07-13 15:29:24,579 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:29:27,045 INFO [ckanext.importlib.loader] Loading sqa_qualifications 2011-07-13 15:29:30,869 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:29:32,561 INFO [ckanext.importlib.loader] Loading scotxed 2011-07-13 15:29:34,017 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:29:35,286 INFO [ckanext.importlib.loader] Loading iscjis_vocabularies 2011-07-13 15:29:36,592 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:29:37,782 INFO [ckanext.importlib.loader] Loading ecare_multi-agency_store_data_model_and_vocabulary 2011-07-13 15:29:39,043 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:29:40,267 INFO [ckanext.importlib.loader] Loading digital_archive_scottish_archive_network 2011-07-13 15:29:41,690 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:29:43,727 INFO [ckanext.importlib.loader] Loading the_scottish_register_of_tartans 2011-07-13 15:29:45,159 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:29:46,816 INFO [ckanext.importlib.loader] Loading scotlands_images 2011-07-13 15:29:48,563 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:29:49,983 INFO [ckanext.importlib.loader] Loading national_library_of_scotland 2011-07-13 15:29:51,709 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:29:53,122 INFO [ckanext.importlib.loader] Loading scottish_government_statistics 2011-07-13 15:29:55,857 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:29:57,118 INFO [ckanext.importlib.loader] Loading scotbis 2011-07-13 15:29:58,883 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:00,264 INFO [ckanext.importlib.loader] Loading historic_scotland 2011-07-13 15:30:02,086 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:03,609 INFO [ckanext.importlib.loader] Loading scheduled_monuments_in_scotland 2011-07-13 15:30:05,057 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:06,987 INFO [ckanext.importlib.loader] Loading scotlands_house_prices_service_consumer_information 2011-07-13 15:30:08,687 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:09,932 INFO [ckanext.importlib.loader] Loading air_quality_in_scotland 2011-07-13 15:30:11,278 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:12,637 INFO [ckanext.importlib.loader] Loading river_and_coastal_flooding_updates_for_scotland 2011-07-13 15:30:13,940 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:15,637 INFO [ckanext.importlib.loader] Loading public_contracts_scotland 2011-07-13 15:30:17,280 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:18,666 INFO [ckanext.importlib.loader] Loading scottish_government_consultations 2011-07-13 15:30:20,673 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:22,234 INFO [ckanext.importlib.loader] Loading scottish_public_health_observatory 2011-07-13 15:30:24,025 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:26,297 INFO [ckanext.importlib.loader] Loading scotlands_information_landscape 2011-07-13 15:30:28,308 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:29,758 INFO [ckanext.importlib.loader] Loading isd_scotland 2011-07-13 15:30:31,847 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:33,074 INFO [ckanext.importlib.loader] Loading scotlandspeople 2011-07-13 15:30:34,541 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:35,754 INFO [ckanext.importlib.loader] Loading general_register_office_for_scotland_statistics 2011-07-13 15:30:37,273 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:39,330 INFO [ckanext.importlib.loader] Loading baseline_scotland_-_groundwater_chemistry_data 2011-07-13 15:30:40,716 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:41,919 INFO [ckanext.importlib.loader] Loading the_centre_for_the_study_of_retailing_in_scotland_data_sets 2011-07-13 15:30:43,275 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:44,515 INFO [ckanext.importlib.loader] Loading growing_up_in_scotland 2011-07-13 15:30:46,215 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:47,582 INFO [ckanext.importlib.loader] Loading scottish_parliament_msp_allowances 2011-07-13 15:30:49,580 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:51,272 INFO [ckanext.importlib.loader] Loading scottish_government_ministerial_hospitality 2011-07-13 15:30:53,764 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:55,642 INFO [ckanext.importlib.loader] Loading scottish_government_expenditure_over_25000 2011-07-13 15:30:58,012 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:30:59,674 INFO [ckanext.importlib.loader] Loading scottish_local_government_financial_statistics 2011-07-13 15:31:01,869 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:31:03,382 INFO [ckanext.importlib.loader] Loading scottish_government_disclosure_of_high_earners 2011-07-13 15:31:07,086 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:31:11,493 INFO [ckanext.importlib.loader] Loading scotlands_places 2011-07-13 15:31:14,452 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:31:31,743 INFO [ckanext.importlib.loader] Loading scottish_national_heritage_facts 2011-07-13 15:31:36,223 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:31:37,830 INFO [ckanext.importlib.loader] Loading scottish_natural_spaces 2011-07-13 15:31:40,245 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:31:41,764 INFO [ckanext.importlib.loader] Loading scotland_national_address_gazetteer 2011-07-13 15:31:43,810 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:31:45,069 INFO [ckanext.importlib.loader] Loading scottish_environment_protection_agency 2011-07-13 15:31:47,975 INFO [ckanext.importlib.loader] ...creating package 2011-07-13 15:31:49,525 INFO [ckanext.importlib.loader] Loading traffic_scotland 2011-07-13 15:31:52,396 INFO [ckanext.importlib.loader] ...creating package }}}" 1128 1310568028000000 dread Note, I added a title column and made one or two minor corrections that I spotted - see attached CSV for the result. 1129 1305116419000000 thejimmyg "Great proposal, here's my thinking: * The current VDM model (where we have things like a single package table which logs changes to a package_revision table) is very well tested and works well and we should only consider a radical change with good reason * Although we are making a move from *model objects* being the central components to the *logic layer functions* being the central components, there is no need yet to have the database structures we store (for revisioning or otherwise) start to look like the dictized representation. The logic layer itself is designed to do that mapping. * Since we do have this logic layer though, there is less need for ""magic"" SQLAlchemy objects to support dicts and lists (eg the stateful objects used to make a list of tags on a package behave like a list), a few simple (and easily debuggable) queries in the logic layer would work better. In the longer term we may want to look at serialising more dictized-looking objects and that may lend itself to a more dictized changeset model or even a more dictized storage system (eg no-SQL) but for the time being we are not at that point. I recommend the following: * We continue to use the current default branch of VDM (not the changeset branch) * We continue to treat the package table (and other non-revision tables) as the most recent revision (even though the *active* revision displayed in CKAN might well be in the revision tables because more recent changes haven't been moderated yet) * We slowly stop using stateful lists and dicts in CKAN because we have move control with explicit queries in the logic layer Sound good? " 1129 1305139980000000 rgrp "It seems like this ticket is getting a bit confused with ticket:1076 (changes re vdm) :-) Some general conclusions relevant to both: * General feeling is not to change to changeset model * I'm personally +0 on changeset model * I would note main benefit is simplicity and orthogonality of changesets to core system. I also think cost of change is small. * That said we can simplify current model as well. * There seems some misunderstanding: change to have logic layer has almost nothing to do with being able to remove main stateful stuff in vdm. To be able to remove most of stateful stuff in vdm requires us to make some other changes (re foreign keys from revision objects to continuity) * There are other simplifications we should make to vdm before embarking on this (e.g. move to SessionExtension from MapperExtension). This is easy as that work has been done in changeset branch and can be backported. What I don't see in this CEP as yet is any discussion of the complexities around pending stuff (e.g. do we allow multiple pending, do new edits get shown current or latest pending). I'll try and comment more on this but a lot of this was in original vdm discussion last summer." 1129 1305209601000000 rgrp "Replying to [comment:2 rgrp]: > It seems like this ticket is getting a bit confused with ticket:1076 (changes re vdm) :-) I of course ;0 meant ticket:1077" 1129 1305212467000000 kindly "> * There seems some misunderstanding: change to have logic layer has almost nothing to do with being able to remove main stateful stuff in vdm. To be able to remove most of stateful stuff in vdm requires us to make some other changes (re foreign keys from revision objects to continuity) The logic layer does not automatically help out, however it makes our life easier if we want to handle state ourselves. For example take package tags, if we remove the stateful_m2m properties and just use normal sqlalchemy relations. We will still want statefullness (i.e active, pending, deleted) on the package_tags table. We should update those on the table ourselves in the logic layer. > * There are other simplifications we should make to vdm before embarking on this (e.g. move to SessionExtension from MapperExtension). This is easy as that work has been done in changeset branch and can be backported. I agree but event thought the MapperExtension way is not great, it is very well field tested." 1129 1325268100000000 rgrp More UI work remains in #1141 1130 1340633514000000 icmurray Unassign for triage. 1131 1305537897000000 dread Fixed on branch release-v1.3.5 cset:3461ed6bab0c 1132 1307352675000000 dread "Yes, these are apparently search-related and so are skipped when testing under sqlite. See README.rst for more info. Compare with this: {{{ $ nosetests --ckan --with-pylons=test-core.ini ckan/tests/functional/test_authz.py ............................................ ---------------------------------------------------------------------- Ran 44 tests in 69.287s OK }}}" 1133 1324057072000000 dread "Works for me - change occurs successfully. {{{ (pyenv-ckan)dread@dread-laptop:~/gitroot/ckan$ paster rights add russianfan admin warandpeace /home/dread/gitroot/pyenv-ckan/lib/python2.6/site-packages/pylons/templating.py:610: UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages) Engine = entry_point.load() User russianfan -> is admin on -> Package warandpeace (pyenv-ckan)dread@dread-laptop:~/gitroot/ckan$ paster rights /home/dread/gitroot/pyenv-ckan/lib/python2.6/site-packages/pylons/templating.py:610: UserWarning: Unbuilt egg for setuptools [unknown version] (/usr/lib/python2.6/dist-packages) Engine = entry_point.load() 22 results User visitor -> is editor on -> Package annakarenina User logged_in -> is editor on -> Package annakarenina User testsysadmin -> is admin on -> System system User logged_in -> is editor on -> System system User visitor -> is reader on -> Group david User annafan -> is admin on -> Package annakarenina AuthorizationGroup anauthzgroup -> is reader on -> Package warandpeace User visitor -> is editor on -> Package warandpeace AuthorizationGroup anauthzgroup -> is editor on -> Group david User logged_in -> is editor on -> Package warandpeace User logged_in -> is reader on -> Group david User visitor -> is reader on -> Group roger User logged_in -> is reader on -> Group roger User russianfan -> is admin on -> Package warandpeace User visitor -> is admin on -> Package warandpeace AuthorizationGroup anotherauthzgroup -> is editor on -> AuthorizationGroup anauthzgroup User russianfan -> is admin on -> Group roger User russianfan -> is admin on -> Group david AuthorizationGroup anauthzgroup -> is admin on -> AuthorizationGroup anotherauthzgroup User visitor -> is anon_editor on -> System system User visitor -> is admin on -> Group roger AuthorizationGroup anauthzgroup -> is editor on -> System system }}} Tests would be nice, but it currently works, is only for admins and is therefore not as high priority as other other tests, such as front-end js." 1134 1305125228000000 thejimmyg "Great stuff. Yes, I agree with all this, but as initial thoughts I suggest that for phase 1, the info() method returns one more key called ""form_config_interface"" which takes a string representing a the type of interface for the config field on the form. If the key is missing it is treated as ""None"". The two possible values are: None - the field is not present and must always be stored as NULL Text - a single text field will be provided on the interface Whatever the interface, the value built will always be stored in the config column of the table. We then also provide a ""get_schema()"" method that returns a schema cablable of parsing the data submitted by the form and storing it as a single key named ""config"". ckanext-inspire will then add other functions to the schema for URL etc so that regardless of what the harvest plugin does, the key fields ckanext-harvest are always there. This then allows: * Customiseable config interfaces * Customisable valiation * A consistent place to store config in the database Sound good?" 1134 1305623531000000 sebbacon "Agreed with James that we should consider *from the start* how to provide for a nice UI -- my worry is if we start out with people editing raw JSON we'll end up keeping that for years :) " 1135 1305210134000000 rgrp "Work on this is largely complete in changeset branch: https://bitbucket.org/okfn/vdm/src/changeset/ Overview docs: https://bitbucket.org/okfn/vdm/src/changeset/doc/changeset.rst" 1135 1340632267000000 icmurray Re-assigned to kindly 1136 1305210925000000 rgrp "Most of this change has been implemented in the changeset branch [1] but could be backported reasonably easily to main. [1]: https://bitbucket.org/okfn/vdm/src/changeset/" 1136 1305216218000000 kindly This would be nice but it is not necessary. The current mapper implementation may be nice but not is battle tested.. 1136 1305217553000000 kindly "Replying to [comment:2 kindly]: > This would be nice but it is not necessary. The current mapper implementation may be nice but not is battle tested.. I meant the mapper extension ""IS"" battle tested. " 1136 1340632980000000 icmurray Re-assigned to @kindly. 1137 1305211672000000 rgrp "Most of this change has been implemented in the changeset branch [1] but could be backported reasonably easily to main. [1]: https://bitbucket.org/okfn/vdm/src/changeset/" 1137 1305217449000000 kindly "I am not sure this needs to be done. I think we should keep the continuity object always in the table, even if it is deleted. The querying should be done through the logic layer so the deleted state should not be an issue. The clients should be entirely state aware. The only thing that needs to be done is to remove all statefullness of relations. These are the only things that are complicated. This would make vdm just a simple copy on write mechanism, with the client controlling the state." 1138 1316965357000000 rgrp Cannot see this issue any more on default (as deployed on e.g. test). 1140 1312491320000000 kindly "fixed cset:987da68ea4f6 The package group table needed to trigger a reindex of the package. " 1140 1312537257000000 dread Broken since CKAN v1.1. I have added this fix to v1.4.2 and v1.4.3 releases. 1141 1308828928000000 dread "I've merged this into default in preparation for release 1.4.2, but there may be a couple of useful additions that could be done on this or another branch/ticket. Getting a package at a specified revision is really useful. This can be used: * links in the package history (#103) (while in there, the 'moderated' flag should be shown in the package history too if not already) * should also be added to the package API (as a replacement for the AJAX call?) The history_ajax call looks like an almost duplicate of the Package Revision API. Can these be merged and put just in the API? " 1141 1318862183000000 dread What's the status of this now? The ckanext-moderatededits README still says you need a branch of ckan. Can you say which version of CKAN is required instead? I say we close this if ajax call is now in a better place in the API. (I expect it is now, with the logic layer). 1141 1318862773000000 johnglover Well, it runs on datacatalogs.org with the current default so CKAN 1.5, but it needs quite a bit of updating to work with all of the new 1.5 UI changes, and as it has been a low priority for quite a while I haven't scheduled time to work on it. Really I would need to spend a few days on it to tidy it up for standard 1.5. 1141 1325352507000000 dread Closing, with remaining tidy up work for newer CKAN versions split off into new ticket: #1604 1142 1305880901000000 sebbacon "The guys at Wikimedia also produced this spreadsheet including notable users and features of other software. Could be useful.: https://spreadsheets2.google.com/ccc?authkey=CLWu1cgP&hl=en&key=tYiwc9kUVLLlALOQEzFqujw&hl=en&authkey=CLWu1cgP#gid=0 " 1142 1305881244000000 lucychambers "Replying to [ticket:1142 rgrp]: > Previous super ticket (from 3m ago): http://trac.ckan.org/ticket/927 > > * CKAN 1-page overview (for enterprise and for data hackers) > * Administrator's Guide (including install) > * Extensions Guide > * Separate CKAN.net info from Software Documentation (?) > > == Minor Items == > * Include guide on how to configure local filesystem storage, instead of Google storage. (http://lists.okfn.org/pipermail/ckan-discuss/2011-May/001235.html) " 1142 1308326118000000 annapowellsmith "List of new docs tasks agreed by APS and RGRP: * Tutorial on the wiki (long-term): Integration with Wordpress and Drupal and Javascript * Overview of domain model both for average users and for devs * 2-page glossy brochure * Vision for CKAN http://notebook.okfn.org/2011/04/27/data-hubs-data-management-systems-and-ckan/ * Comparison : use http://wiki.ckan.net/Related_Software as starting point - build on wiki - Socrata and OGDI * Renaming: separation between ckan.org and ckan.net * More documentation in the code ... * Config options generated from code or move this to WUI ... See also proposed overhaul at http://wiki.ckan.net/Documentation_Plans" 1142 1313775665000000 rgrp This was completed 4 weeks ago and can now be marked as complete. See http://docs.ckan.org/ and http://wiki.ckan.net/ and http://ckan.org/ 1143 1307024681000000 dread "All done in ckanext-stats cset:bc54790cacd4 (apart from ""fix x axis to start at first revision"" which was not a problem after all)" 1148 1305969925000000 kindly complete cset:96a43c9d8bd7 1149 1305971672000000 kindly cset:b1634d405066 1149 1306090663000000 kindly "Failed dgu tests due to as_dict not working on deleted objects. fixed by cset:c9b9cf513e44" 1150 1311178163000000 thejimmyg "Hi John, Can you please check that the new webstore+preview works correctly with this one too please? Cheers, James" 1150 1311774141000000 johnglover Unicode data should be fine in the new datapreview code, for example: http://test.ckan.net/package/afghanistan-election-data 1151 1330908188000000 rgrp Not sure we will be able to fix this easily as we need a dataproxy for general case. Think it better to focus on the case where data in DataStore and we use Recline. As such closing as wontfix for the present. 1152 1307352165000000 amercader Bumping to ckan-v1.5-sprint-3 and updating the CC email addresses so people actually get any updates. 1154 1314120497000000 johnglover Fixed in core (branch feature-1275-solr-search). 1154 1314177477000000 dread "Looking at the changeset, it now does a connection check at start-up of CKAN. If it fails then CKAN runs but doesn't show any packages, apart from via the Tags interface, and the search page and widget on the front-page are removed. (BTW Is this a useful thing to do? Without search, perhaps CKAN should simply not start?) I believe the problem pudo is referring to is temporary periods when the SOLR server is slow/overloaded/down. During this time pudo and I get exception emails saying connection failed for request. e.g. {{{ Module ckan.controllers.home:29 in index << query = query_for(model.Package) query.run(query='*:*', facet_by=g.facets, limit=0, offset=0, username=c.user) c.facets = query.facets c.fields = [] >> limit=0, offset=0, username=c.user) Module ckan.lib.search.common:115 in run << self.query = QueryParser(query, terms, fields) self.query.validate() self._run() self._format_results() return {'results': self.results, 'count': self.count} >> self._run() Module ckanext.solr.backend:63 in _run << # this wrapping will be caught further up in the WUI. log.exception(e) raise SearchError(e) finally: conn.close() >> raise SearchError(e) SearchError: [Errno 111] Connection refused }}} (recently from nl.ckan.net)" 1154 1314178325000000 johnglover "> Looking at the changeset, it now does a connection check at start-up of CKAN. If it fails then CKAN runs but doesn't show any > packages, apart from via the Tags interface, and the search page and widget on the front-page are removed. (BTW Is this a > useful thing to do? Without search, perhaps CKAN should simply not start?) When discussing the ticket we agreed that we should allow ckan to start without search. It is possible that people simply want to list/view information without searching for it and we decided not to prevent them from doing so. > I believe the problem pudo is referring to is temporary periods when the SOLR server is slow/overloaded/down. During this > time pudo and I get exception emails saying connection failed for request. Ok, other options then: * if an error occurs during search, return 0 results * if an error occurs during search, redirect to a generic 'search is unavailable' page * if an error occurs during search, retry the query a certain number of times before giving up and doing one of the above. What do you suggest?" 1154 1314179290000000 dread "Search requests - option one seems like a bad user experience, and option three seems too complicated. I should do option 2, explain it may well be a temporary problem, and let the user press reload, as per any normal web request. A more difficult problem though is what to do about a failed search indexing. Because of the try/except catch we've had until the patch yesterday, we get no exception emails for this, but I'm worried that this has caused packages to not be indexed, and hence are essentially lost in CKAN (unless someone realises there is a problem and does a reindex at some point). BTW I added paster command 'search-index check' for looking at this problem, but I don't think it was ever added to solr. Is it easy to add get_all_entity_ids for the purpose of checking for this problem?" 1154 1314191496000000 johnglover "Ok I'll implement option 2 for search query issues. Yes it is easy to get a list of IDs from Solr so I'll have a look at that command and make sure that it works. I think a case could be made for not allowing a package to be added if it cannot be indexed however, and making the user simply retry the request." 1154 1314192324000000 dread Yes, forcing a retry may be possible, although I guess on these occasions you'd be looking at rolling back the creation of the package object too, which is not really in the spirit of the notifications system at present. This is why we had the queue for solr for occasions like this - is it here no longer? 1154 1314192753000000 johnglover Not in core, there is a queue feature in the extension still which I could add to core. I think though it was a case of using either the queue or synchronous search, I don't think it would automatically fall back to the queue if there was an indexing problem, but we could look at that. Was there not talk of changing the queue system recently though? There are also no tests for the Solr queue so that would take a bit of work as well. 1154 1314193633000000 johnglover RE search queries failing: Just having a look through the controller code and the package search controller and the search api both perform option 2 already, so I'll just update the home controller. 1154 1314195684000000 dread "Ok, I chatted to David about this and we feel it should go like this: notification to search indexer fails and raises an exception, this gets passed through the notification system back up to the code that creates the package, and you do a roll back. Does this tally with what you mentioned in the last comment or is what's there better? Thanks for the info on the queue and ckanext-solr. It sounds like this isn't deployed successfully anywhere (or does pudo know if it is?). In this case we should be clear that it is not what it says it is: ""This extension adds support for package search using Apache Solr to CKAN"" and maybe simply mark it as deprecated. Pudo what do you think?" 1154 1314196683000000 johnglover "> Ok, I chatted to David about this and we feel it should go like this: notification to search indexer fails and raises an exception, > this gets passed through the notification system back up to the code that creates the package, and you do a roll back. Does this > tally with what you mentioned in the last comment or is what's there better? This sounds good to me, definitely better than what is currently there. > Thanks for the info on the queue and ckanext-solr. It sounds like this isn't deployed successfully anywhere (or does pudo > know if it is?). I'm not actually sure if the queue works or not, but if it does I'll have to write some tests for it before we could add it to core. A working queue system would be useful generally however, as there are other things that it would be useful for (such as QA), but yes, going forward the Solr extension should really be deprecated." 1154 1314287519000000 johnglover "All attempts to index packages should now throw a SearchIndexError if Solr is unavailable and the create/update will be rolled back. These errors are caught in the main package controller (redirecting to an error page) and the API controller. I have added tests for both controllers. Also updated the paster 'search-index check' command to work with solr." 1155 1306773174000000 pudo "Now implemented based on CSV export available on the catalogue page: * http://data.london.gov.uk/catalogue" 1156 1306855111000000 pudo Both are now implemented but may need further work to adhere to developing conventions wrt to location etc. 1159 1307615133000000 pudo Rudimentary RDFa has been added with seeAlso to API. 1161 1306408026000000 amercader Duplicate of #1157 1164 1308647224000000 amercader Done. See it in action here: http://publicdata.eu/map 1165 1306413953000000 nils.toedtmann this is similar, but not equal to #1096 1165 1307444733000000 nils.toedtmann See also http://ckan.okfnpad.org/multisite 1166 1306847326000000 thejimmyg Also, rather than INSPIRE=true we should probably have information about the harvesting mechanism and the harvest object type? This also relates to our use of the kind field on the resource. 1166 1317381932000000 amercader Fixed on d5bee3de9957 1167 1306856992000000 sebbacon "Yes to all the above. Further question: which extensions should be installed as standard? I would argue: * ckanext-stats * ckanext-googleanalytics * ckanext-disqus * ckanext-solr (implies installing a running solr, too...) * ckanext-follower * ckanext-admin Note that googleanalytics and disqus both require user accounts set up at the corresponding service. Not sure how we would handle this in the instance setup. " 1167 1306857142000000 sebbacon Another thing to consider: how do we ensure we keep a single, up-to-date version available? (I'm thinking wrt developer / sysadmin workflow) 1167 1306857750000000 thejimmyg "Creating the AMI is really just the icing on the cake. The hard bit is getting the packaging right so that's where we need to concentrate first. Once the packaging works we just make sure apt-get upgrade is run regularly. +1 to the list above. I also agree we should install solr by default. Who has actually done a CKAN solr install? I haven't but together with whoever has I can try to package it up. Wrt to usernames and passwords, I'll look into how dpkg manages those blue pop-up screens for entering configuration options." 1167 1307358318000000 dread "debconf seems the preferred way to question the user post-install and run setup scripts. It seems a bit of a faff. The person that is able to spin up an EC2 instance is no doubt technically able enough to follow a couple of instructions in README.txt saying edit /etc/ckan//.ini and add your Google Analytics credentials. So I don't think this ticket is the driver for a debconf setup." 1167 1311178516000000 thejimmyg "I've now written a 20 page guide on using CKAN on Amazon EC2. I haven't explicitly created an AMI, but I don't think we should do that anyway for a few reasons: * it is fiddly * there is no advantage to the user as CKAN installation is so easy and so well documented * we don't want to have to maintain the image going forward * we might not want to promote a non-open approach The guide will form part of Anna PS's work on the overall documentation but you can also see the original version here: https://bitbucket.org/okfn/ckan-debs-public/src/489a5ecf369f/ec2/" 1168 1310134901000000 thejimmyg This is now working on dgu-buildbot.okfn.org. The code that manages it is partly the repo.py script from http://hg.3aims.com/public/BuildKit/ and partly the fab code in https://bitbucket.org/okfn/buildbot-scripts 1168 1310396585000000 dread This might run manually on the us7, but the buildbot is not running these automatically or any sort of testing, which is what this ticket is about. 1168 1325866142000000 thejimmyg "Quick update... we now have: * An automated way to build the deb packages * An automated way to install them into a fresh virtual machine We don't currently run them via buildbot or have smoke tests. " 1169 1306754833000000 dread Done on default. Will appear on ckan.net next time it is branched and updated. 1170 1306864422000000 dread Done in cset:4d2d0857281f on default 1171 1347358705000000 ross "I have implemented this on the datahub, but this change likely needs to be made to CKAN core. Probably as part of 2.0. The format on tdh is currently: ${TITLE}. ${AUTHOR}. Retrieved ${UTC time}. ${DATASET LINK} Best discuss and agree on what format should be supported. Mark, I've assigned this to you as it is tagged academia ;) " 1172 1312991332000000 dread I've started this on branch defect-1172-exceptions 1172 1314118281000000 dread "Is in cset:47225f03e2af and merged to default for 1.4.4. It now barfs instead of failing silently: * if the ckan config file has any errors * search indexing error There are some other ""except Exception"" statements " 1172 1314118883000000 dread "I took out ""raise"" by mistake in ckan/lib/search/worker.py:26 but John has added it in in the corresponding place in ckan/lib/search/__init__.py on branch feature-1275-solr-search anyway." 1173 1307000249000000 pudo "I propose we do a bit of frameworkiness here: set up a two-part content type registry and refactor the API controller to use it. The idea would be to do the following things: 1. Create a registry mapping format extensions to mime types: RepresentationRegistry.add_format('json', 'application/json') 2. Create a registry of (mime types, entity types) -> converter functions that yield an appropriate representation of the entity. 3. Hook this into the _finish method of the API controller based on Accept: handling 4. Add support for /api/rest/ENTITY/NAME.{format} and /ENTITY/NAME.{format} to routing, use registry from (1) to rewrite accept headers. 5. Register converters in load_environment or via IConfigurable plugins 6. Document What do you think? " 1173 1307615271000000 pudo Now available in ckanext-rdf and linked to from WUI. 1174 1307615200000000 pudo "wontfix as thejimmyg wants to turn REST API into RPC API and this doesn't fit in. Not an excellent plan in my idea but this is to be discussed elsewhere. " 1175 1307350866000000 dread Flavio, can you post here the exception that you get? 1175 1307372833000000 fccoelho Thats the thing, I don't get any error message, no even in the logs. But I can't even load a page. 1175 1314030683000000 dread Hi Flavio, is this still a concern? I'd like to close this ticket unless you can provide details of the error from your logs or exception email. 1175 1325355170000000 dread This error (which will be in the apache or ckan log) must be unrelated. Stats extension runs fine elsewhere. 1176 1310557852000000 dread This is occurring several times a day now - James can you suggest what's happening? 1176 1314273012000000 dread "This occurs when you browse {{{/error/document}}} directly, rather than being directed there on error. In this case there is no 'original_response'. Fixed in cset:c67d5934fb0b for ckan 1.4.3 and merged to default." 1177 1307360643000000 rgrp What revision of the CKAN code are you using (do hg id to get revision if installing from source). Have you modified the package template in any way? Have you restarted apache after upgrading? 1177 1307361157000000 fccoelho "to upgrade I used the method describeb on bitbucket, after upgrading pip-requirements: pip -E pyenv install -r pip-requirements.txt after that I did a paster --plugin ckan db upgrade --config ckan.emap.fgv.br.ini the revision id on my pyenv/src/ckan is: 5730c79db461+ tip yes, I have restarted apached after upgrading" 1177 1307365496000000 rgrp "The issue here is that the line should never be reached because g.has_commenting should be False: https://bitbucket.org/okfn/ckan/src/5730c79db461/ckan/lib/app_globals.py @Flavio: have you modded your code in any way? is self.has_commenting False in your app_globals.py A quick fix would be to remove the template lines calling to remove the lines in the template: {{{ ${h.subnav_link(c, h.icon('comments') + _('Comments & Questions'), controller='package', action='comments', id=c.pkg.name)} }}}" 1177 1307366621000000 fccoelho "I have not modded the code in any way... I'll look for those lines in the template.which template is it? read.html does not have a line such as this." 1177 1307367354000000 fccoelho OK I did an fgrep on all package template and found the line on layout.html, I'll comment the block and test. 1177 1307367923000000 fccoelho "Ok it fixed it! thanks a lot!! Is there any side effects to the removal of this line, I should know of?" 1177 1307374897000000 dread "I think rgrp was suggesting you have mistakenly set has_commenting to ""True"" in app_globals or elsewhere. But either way, this should have no side-effects." 1178 1308045351000000 rgrp Fixed. This was due to incorrect config of bucket option. 1180 1307544223000000 dread "Both issues solved using a whitelist on anchor tags. Second issue was a link with unicode expression of the quote. e.g. somelink" 1181 1307532816000000 dread Fixed in ckan default in cset:72f7d48d7f31. Have left Package url and Resource url links though, as these are the key links we want google to see. So we need to check these carefully for spam. 1183 1311181719000000 thejimmyg "Hi John, Can you have a look at this one too please? Cheers, James " 1183 1311771069000000 johnglover Fixed in new datapreview code (currently running on the test.ckan.net), which now gets the preview data from a webstore instead of the dataproxy API. It also only now shows a preview button for resources that were successfully archived and saved in the webstore. 1186 1310556519000000 dread This was done by pudo in branch feature-1198-mailer and went into default, lined up for release 1.4.2. 1187 1308143605000000 dread "Also stopped creating revision for user edits. Done in cset:795ccd6405ba for 1.4.1 release" 1189 1323173227000000 thejimmyg Closing this ticket in line with ticketing policy since it is over 6 months old. If someone would like to develop an extension that uses spideroak as a back end we can look at it again. 1190 1320143592000000 rgrp A very substantial portion of this got done as part of 0.5 including the webstore #1208 and the majority of automated conversion #1398. #1432 is currently deferred so we're going to close this ticket on basis that item (3) is now out of scope. 1191 1308650930000000 dread Done in cset:c3822feffb38 for 1.4.1 release 1192 1308337652000000 dread "I really think forms are a key asset of CKAN, and keeping the docs about customising and integrating them is important - i.e. forms.rst and forms-integration.rst. There are ideas being suggested for replacing bits of front-end, but forms hard to do, and David's new system is a significant offering on top of the CKAN back-end, so it seems crazy to me to abandon it. forms.rst has been updated by David Raznick in the release_v1.4.1 branch. i18n.rst - include here the creation of the pot file, deploying a translation, transifex admin load-testing.rst - I think performance should be mentioned in the glossy guide (some sort of basic metrics of speed profile, to return web pages, run searches and API operations, for 100 or 1000 packages, on a given spec machine) and have it based on a report showing how we measured it. The report would look a bit like this file, and it might do as a stub, but it seems pretty lacking. distributed.rst - agreed, remove. design.rst - agreed, delete. loaders.rst / loader_scripts.rst - yes move to wiki feeds.rst - this should be split into a tutorial for the wiki and the reference page left in sphinx importer.rst - this should be moved to ckanext-importer/doc admin.html - this has already gone in 1.4.1 authorization.rst - This is a bit jumbled. We're talking about changes to the code, so maybe not worth working hard to make this good, so I suggest leaving it, but delete the boring section ""Requirements and Use Cases"". deb.rst - this is an administrator task, so should stay why move this to the wiki? buildbot.rst - ok to stay here? vm.rst - I assume there's no problem with this" 1192 1310135468000000 thejimmyg This is now a child ticket of #1142 1192 1312191462000000 rgrp Marking this as closed as Anna has completed and this is now deployed at http://docs.ckan.org/. 1193 1309768960000000 kindly fixed cset:87d6140e06ad 1194 1317043212000000 dread "Suggest changing this to ""Welcome, Rob"" rather than ""Welcome back, Rob"", avoiding the problem." 1194 1317053674000000 dread Done in cset:49e2c05e69ea 1194 1317053688000000 dread Will go into ckan 1.5. 1195 1311332049000000 dread Also this one: http://ckan.net/storage/f/file/ 1195 1311841763000000 rgrp Fixed in https://bitbucket.org/okfn/ckanext-storage/changeset/49734285f1ef 1196 1308743857000000 dread "Reason here: http://www.freewisdom.org/projects/python-markdown/Tickets/000018 Using the suggested workaround. Fixed on default for 1.4.2 in cset:715212cd220c" 1197 1340010256000000 toby "personally I'm against these for 4 reasons a) The site should be self explanatory and if it is not this is a failure of design b) a cookie would be needed to prevent these happening all the time - cookies are bad as they interfere with caching proxies. c) I find stuff like this patronising/annoying for the user d) They are a maintenance burden for ckan as we do not have a single instance Therefore I'd suggest closing this as a won't fix - I agree this is just a personal view point and therefore am not going to actually close the ticket but just record my opinion." 1197 1340018903000000 seanh "Hi Toby, yes I think I agree with you, it reminds me of when I install new Android apps on my phone, some of them start off by bothering me with some sort of guided tour or long instructional text, I hate this and always skip through without reading it, expecting to be able to figure out the interface for myself. The one thing that something like this could be useful is, rather than showing someone how to use the site, showing them what the site is capable of so they can assess whether it meets their needs/is something they're interested in. However, we already have a good guided tour of this type on ckan.org/features, those pages are really pretty good. Close this as wontfix?" 1197 1340019675000000 toby "Thinking horrible thoughts, there was a rumour of a ckan tour, for something like that this would be useful. Not sure where that plan is at? if there is a ticket for it maybe link to this one and close?" 1197 1340020357000000 seanh "AFAIK there isn't a ticket for that, it was something originally requested as part of the demo site, but since the devs think it's a horrible idea I don't think it's going to happen, easiest way to make a CKAN tour would be a series of static pages like the one we already have on ckan.org, I think that combined with a real functioning demo site probably meets the requirement. Anyway, closing this as wontfix" 1198 1308821489000000 pudo "This ticket is essential I think. I would propose we should be a bit more radical here, though: * Introduce an ""Agent"" SQLAlchemy object and derive both publishers and users via single table inheritance. * Make agent the central object of authorization, e.g. by introducing c.agencies including the user account, visitor, logged_in, etc. * Abolish AuthorizationGroups in favor of publishers. Yes, they are different but the use case for what AuthzGroups do and Publishers don't is just nerdery. * Restructure URLs to be // rather than /package/ and have / be a useful list of managed resources rather than a revision list (users current home page). * Make Agents a primary facet for all kinds of navigation. Of course this would be done incrementally but I think we should agree on some such scheme as a marching direction. " 1198 1308824870000000 dread "I think it is a good idea to allow both users and publishers to be the object of authorization. The user_object_role table and all logic surrounding it would be simpler. I deliberately didn't suggest any detail on the authz front because I know other related changes are afoot so it would be useful for David Raznick to contribute here. I really like the url ideas - more like github. / having a list of packages and a search box, plus a link to the latest revisions would be great. /package/ would usefully redirect to //. I do think this should be split off into a separate ticket though. What do you think?" 1199 1308822557000000 rgrp Not sure we need exposure this via the API (at least at this point) as things like auto mass emailing could be in done in separate component (that talks to CKAN). 1199 1308922389000000 pudo merged to default with password reset forms 1199 1310555589000000 dread "This was done in branch feature-1198-mailer. No tests - ticketed here: #1222. Additional feature in this branch 'password reset' was ticket #1186." 1200 1311180218000000 thejimmyg This is a duplicate of #1108. Let's have the discussion there please. 1202 1314030352000000 dread "The CKAN package page links to datapkg at: http://blog.okfn.org/2010/02/23/introducing-datapkg/ (it has always been this link). I think Rolf is referring to the three links to knowledgeforge.net in the blog post itself. These need to be edited by Rufus. Note datapkg has moved again to github." 1202 1315821804000000 rgrp This now irrelevant and relevant content has gone from thedatahub theme. 1204 1311179980000000 thejimmyg "Sorry, is this a CKAN issue or a datacatalogs.org one? If it is datacatalogs.org do you mean a catalog rather than a package? If so, can you give an example of the sort of change you've made? Marking as invalid until we get more information." 1205 1309768720000000 kindly "fixed in default cset:5e2070688e54 The upgrade now works locally, so the upgrade should work now." 1205 1310568631000000 dread This worked - cheers David. 1206 1309450216000000 dread "Fixed with cset:149be76faabc on default branch, aiming for ckan 1.4.2. Thanks for help from Eric Lemoine." 1207 1311325343000000 dread This has been fixed. 1208 1310124599000000 thejimmyg Rufus and Friedrich are working on this at the moment so putting into the current sprint. I'm considering this as preliminary investigation to inform the wider project that David Raznick is leading on. 1208 1311525306000000 rgrp Propose this ticket can be closed as webstore is now functional and being used. Can open new tickets for specific improvements / adaptations as the need arises. 1208 1312191646000000 pudo "This should be closed: * https://github.com/okfn/webstore - See README for details * Python client lib: https://github.com/okfn/webstore-client * http://webstore.thedatahub.org/ - Demo install * http://wiki.ckan.net/Webstore - Overview " 1209 1310571715000000 dread Was fixed in cset:d959a70a19ea. Bug was introduced and fixed before a release took place. 1210 1309974781000000 dread Fix in ckan for AssertionError in cset:8f6ba8ef63f3 lined up for CKAN release 1.4.2. 1211 1315910650000000 dread I've listed what was checked into this branch 6 weeks ago and is going into release 1.4.3. Is this ticket ready to close now or are there future plans? 1211 1315948703000000 rgrp Let's close and re-open (or open a new ticket) for future work. 1214 1309982828000000 dread "I'm working through these on branch defect-1214-api-improvements. ""Server returns text/html for errors"" - now fixed in cset:0a533a8da3df" 1214 1310041083000000 dread "Aron, I've fixed the second item (on my branch), so you can DELETE a package without Content-Length. But I'm not sure what you mean here: ""Should be 405 Method Not Allowed?"" DELETE should be allowed. Or is that a response you're getting? David" 1214 1310041605000000 dread """Tag returned as a JSON object when updating but as a string when requesting."" I'm not sure what you mean. http://test.ckan.net/api/2/rest/package/reference-example gives tags: {{{""tags"": [""country-uk"", ""example-package""],}}} Is it this - returning package names, rather than ids? Here is certainly a bug I'll fix. http://test.ckan.net/api/2/rest/tag/country-uk" 1214 1310042268000000 dread "Fixed tag entity returning package names for apiv2 - it now gives IDs. The problem mentioned about ""Extras"" content - I'm not sure what's meant here. Package extras are just strings. You might choose to store a package name in an extra, but there is no magic to convert it to a package id should you retrieve the package via apiv2. Is this just a misunderstanding or have I missed your point?" 1214 1310044531000000 dread "Deleting an extra using 'null' works for me: {{{ $ curl -d '{""name"":""dtest2"", ""extras"":{""1"":""1"", ""2"":""2"", ""3"":""3""}}' http://test.ckan.net/api/rest/package -H ""Authorization: tester"" ... $ curl -d '{""name"":""dtest2"", ""extras"":{""1"":""1"", ""2"":""2"", ""3"":null}}' http://test.ckan.net/api/rest/package -H ""Authorization: tester"" {""maintainer"": null, ""name"": ""dtest2"", ""relationships_as_subject"": [], ""author"": null, ""url"": null, ""relationships_as_object"": [], ""notes"": null, ""title"": ""dtest2"", ""maintainer_email"": null, ""revision_timestamp"": ""2011-07-07T12:57:18.454890"", ""author_email"": null, ""state"": ""active"", ""version"": null, ""groups"": [], ""license_id"": null, ""revision_id"": ""f0ff31c0-027b-49ce-9daf-94a73d96a913"", ""tags"": [], ""id"": ""fdeeb287-2783-4aac-9fc7-a6717e54e22f"", ""resources"": [], ""extras"": [{""state"": ""active"", ""value"": ""\""1\"""", ""revision_timestamp"": ""2011-07-07T12:57:18.454890"", ""package_id"": ""fdeeb287-2783-4aac-9fc7-a6717e54e22f"", ""key"": ""1"", ""revision_id"": ""f0ff31c0-027b-49ce-9daf-94a73d96a913"", ""id"": ""d1937073-7bfc-48c5-b6ff-b00d90b451ae""}, {""state"": ""active"", ""value"": ""\""2\"""", ""revision_timestamp"": ""2011-07-07T12:57:18.454890"", ""package_id"": ""fdeeb287-2783-4aac-9fc7-a6717e54e22f"", ""key"": ""2"", ""revision_id"": ""f0ff31c0-027b-49ce-9daf-94a73d96a913"", ""id"": ""8147886f-9769-440c-8c35-b7d6a2f46de7""}]} }}} " 1214 1310045026000000 dread "Ok I see the problem with extras. When you create a package you get back 'simple' extras: {{{ $ curl -d '{""name"":""dtest3"", ""extras"":{""1"":""1""}, ""tags"":[""australia"", ""barbeque""]}' http://test.ckan.net/api/rest/package -H ""Authorization: tester"" --trace log {""maintainer"": null, ""maintainer_email"": null, ""id"": ""33d75910-ae5e-45b0-a154-d00022555a43"", ""metadata_created"": ""2007-04-10T21:19:38"", ""relationships"": [], ""metadata_modified"": ""2011-07-07T13:06:06.228718"", ""author"": null, ""author_email"": null, ""state"": ""active"", ""version"": null, ""license_id"": null, ""resources"": [], ""tags"": [""australia"", ""barbeque""], ""groups"": [], ""name"": ""dtest3"", ""license"": null, ""notes_rendered"": """", ""url"": null, ""ckan_url"": ""http://test.ckan.net/package/dtest3"", ""notes"": null, ""title"": ""dtest3"", ""ratings_average"": null, ""extras"": {""1"": ""1""}, ""ratings_count"": 0, ""revision_id"": ""31cb22d4-3452-4adc-b572-cd173aea7d }}} i.e. ""extras"": {""1"": ""1""} and that's what you get if you GET the package: {{{ $ curl http://test.ckan.net/api/rest/package/dtest3 {""maintainer"": null, ""maintainer_email"": null, ""id"": ""33d75910-ae5e-45b0-a154-d00022555a43"", ""metadata_created"": ""2007-04-10T21:19:38"", ""relationships"": [], ""metadata_modified"": ""2011-07-07T13:06:06.228718"", ""author"": null, ""author_email"": null, ""state"": ""active"", ""version"": null, ""license_id"": null, ""resources"": [], ""tags"": [""australia"", ""barbeque""], ""groups"": [], ""name"": ""dtest3"", ""license"": null, ""notes_rendered"": """", ""url"": null, ""ckan_url"": ""http://test.ckan.net/package/dtest3"", ""notes"": null, ""title"": ""dtest3"", ""ratings_average"": null, ""extras"": {""1"": ""1""}, ""ratings_count"": 0, ""revision_id"": ""31cb22d4-3452-4adc-b572-cd173aea7d11""} }}} but if you then edit it you get them expressed much more verbosely: {{{ $ curl -d '{""name"":""dtest3"", ""extras"":{""1"":""2""}, ""tags"":[""australia"", ""barbeque""]}' http://test.ckan.net/api/rest/package/dtest3 -H ""Authorization: tester"" --trace log {""maintainer"": null, ""name"": ""dtest3"", ""relationships_as_subject"": [], ""author"": null, ""url"": null, ""relationships_as_object"": [], ""notes"": null, ""title"": ""dtest3"", ""maintainer_email"": null, ""revision_timestamp"": ""2011-07-07T13:06:06.228718"", ""author_email"": null, ""state"": ""active"", ""version"": null, ""groups"": [], ""license_id"": null, ""revision_id"": ""31cb22d4-3452-4adc-b572-cd173aea7d11"", ""tags"": [{""revision_timestamp"": ""2011-07-07T13:06:06.228718"", ""state"": ""active"", ""id"": ""4aa0e776-ac2a-4a8b-82ba-d80237d35596"", ""name"": ""australia""}, {""revision_timestamp"": ""2011-07-07T13:06:06.228718"", ""state"": ""active"", ""id"": ""e890354d-e170-44c1-94b1-f0d8b38a49fc"", ""name"": ""barbeque""}], ""id"": ""33d75910-ae5e-45b0-a154-d00022555a43"", ""resources"": [], ""extras"": [{""state"": ""active"", ""value"": ""\""2\"""", ""revision_timestamp"": ""2011-07-07T13:21:02.220368"", ""package_id"": ""33d75910-ae5e-45b0-a154-d00022555a43"", ""key"": ""1"", ""revision_id"": ""61703821-a2f1-41f8-8f39-71130c5e6c1b"", ""id"": ""87d7d4fc-7eb7-4760-b199-e460ce505632""}]} }}} I'm going to ask David Raznick about what's best here. It would be good to make them uniform." 1214 1310053225000000 aron "Replying to [comment:2 dread]: > Aron, > I've fixed the second item (on my branch), so you can DELETE a package without Content-Length. But I'm not sure what you mean here: ""Should be 405 Method Not Allowed?"" DELETE should be allowed. Or is that a response you're getting? > David If I send a HEAD request to the server eg. {{{ curl -i -XHEAD http://test.ckan.net:80/api/2/rest/package/ec9cb930-d15f-441f-a1e1-36f4d5df19bf }}} I get the following in the Access-Control-Allow-Methods header {{{ Access-Control-Allow-Methods: POST, PUT, GET, OPTIONS }}} I've just realised that the ""Access-Control"" headers are for cross origin browser requests so I think you can ignore my comments about the correct response being 405 Method Not Allowed but perhaps DELETE should be added to the Access-Control-Allow-Methods list? " 1214 1310054305000000 aron "Replying to [comment:3 dread]: > ""Tag returned as a JSON object when updating but as a string when requesting."" > I'm not sure what you mean. > > http://test.ckan.net/api/2/rest/package/reference-example > gives tags: {{{""tags"": [""country-uk"", ""example-package""],}}} I'm getting back objects for each tag in the response body from a PUT request. For example ""country-uk"" would be something like. {{{ { ""id"" : ""600dc72e-6127-4704-b801-bee00474ec0c"", ""name"" : ""country-uk"", ""revision_timestamp"" : ""2011-07-06T09:09:21.578034"", ""state"" : ""active"" } }}} Hopefully this gist will illustrate the issue. > Is it this - returning package names, rather than ids? Here is certainly a bug I'll fix. > http://test.ckan.net/api/2/rest/tag/country-uk That was the fifth point on the list. " 1214 1310054863000000 aron "Replying to [comment:5 dread]: > Deleting an extra using 'null' works for me: Hmm, it works for me too when there are other keys remaining in the ""extras"" object. However I can't seem to delete the last one. Take the following example. {{{ curl -i -XPUT -d'{""extras"": {""Tester"": null}}' -H""X-CKAN-Type: application/json"" http://test.ckan.net:80/api/2/rest/package/ec9cb930-d15f-441f-a1e1-36f4d5df19bf }}} Gives: {{{ { ""author"" : """", ""author_email"" : """", ""extras"" : [ { ""id"" : ""88cc50ca-9e29-4499-a542-09d364f5f64f"", ""key"" : ""Tester"", ""package_id"" : ""ec9cb930-d15f-441f-a1e1-36f4d5df19bf"", ""revision_id"" : ""e5c3ca9c-1dae-4a86-837f-b8c19ac31964"", ""revision_timestamp"" : ""2011-07-07T16:01:05.696025"", ""state"" : ""active"", ""value"" : ""\""Test Value\"""" } ], ""groups"" : [ ], ""id"" : ""ec9cb930-d15f-441f-a1e1-36f4d5df19bf"", ""license_id"" : """", ""maintainer"" : ""aron"", ""maintainer_email"" : """", ""name"" : ""my-test-package"", ""notes"" : ""Heading\r\n===\r\nThis _is_ some text"", ""relationships_as_object"" : [ ], ""relationships_as_subject"" : [ ], ""resources"" : [ ], ""revision_id"" : ""688d33fb-5629-4ab3-9f59-a649fc7caa00"", ""revision_timestamp"" : ""2011-07-06T10:37:50.182894"", ""state"" : ""active"", ""tags"" : [ { ""id"" : ""600dc72e-6127-4704-b801-bee00474ec0c"", ""name"" : ""test-tag"", ""revision_timestamp"" : ""2011-07-06T09:09:21.578034"", ""state"" : ""active"" } ], ""title"" : ""My Test Package"", ""url"" : """", ""version"" : """" } }}} " 1214 1310055169000000 aron "Another edge case that popped up while verifying the above issues. A PUT request without a body throws a 500 Internal Sever Error {{{ curl -i -XPUT -H""X-CKAN-API-KEY: tester"" -H""Content-Type: application/json"" http://test.ckan.net:80/api/2/rest/package/ec9cb930-d15f-441f-a1e1-36f4d5df19bf }}} " 1214 1310057942000000 aron "One more for today. Please do let me know if I should be filing these as separate tickets? For some reason performing a search query for packages with an underscore ""_ "" as a query string key fails to return any results. {{{ curl ""http://test.ckan.net/api/2/search/package?q=osm"" }}} Gives me 4 results. {{{ curl ""http://test.ckan.net/api/2/search/package?q=osm&_=1310056826904"" }}} Gives me none. The underscore is generally used by JavaScript libraries as a way of bypassing the browser cache when making JSONP calls. It's easily worked around but is odd none the less. " 1214 1310062570000000 dread "Fixes have gone into the branch for ""Tag returned as a JSON object when updating but as a string when requesting. Same with ""extras"" content."" and ""Doesn't return rendered_text property on package update."". I've added DELETE to the Access-Control-Allow-Methods on the server, so you can try that now. I've added a failing test for the problem of deleting the last extra. Am discussing with David Raznick how to fix that best. I think the only other ones left to address are: * Relationships use ""object"" key rather than id or package_id. * underscore in search parameters" 1214 1310062621000000 dread "I believe ""A PUT request without a body throws a 500 Internal Sever Error"" was fixed yesterday in the code. (Not deployed to test.ckan.net yet though)" 1214 1310067327000000 dread " Relationships use ""object"" key rather than id or package_id. I'm not totally clear on this one. The URLs for relationship editing can be package ids or names. When you GET a relationship packages are referred to in IDs for v2 API. When you create or update a relationship it returns package IDs for v2 API. Are you referring to the relationships listed in the response when you PUT a package? (If so that would be fixed by my previous fix.)" 1214 1310067831000000 dread " underscore in search parameters I think its trying to filter search results by values of an extra field called ""_"", even though this probably doesn't occur. If it is not too ugly, I suggest using one of your other ideas for solving the cache issue. I suggest you bring it up with Rufus or the ckan-dev list to get opinions." 1214 1310072808000000 dread I've put the fixes onto test.ckan.net - repoen the ticket if there are any issues. 1214 1310396399000000 aron "The groups property in the package resource is always empty (or is for all packages I have viewed). The following group lists one package. {{{ http://test.ckan.net/api/2/rest/group/0ac963e7-ba29-49bc-83c8-98f8c1991649 }}} But when viewing the package the groups array is empty. {{{ http://test.ckan.net/api/2/rest/package/758c26d4-5949-4347-8b4d-023374146d94 }}} " 1214 1314029628000000 dread This appears to have been fixed by version 1.4.3. 1215 1310335541000000 kindly "fixed cset: 8a317eadbb36 If you delete the last row then it just clears it instead of deleting the row." 1215 1310385485000000 dread This is now deployed on ckan.net (requires Ctrl-R to reload js). 1217 1320663277000000 dread I haven't seen this for a while - could have been due to database corruption which has been fixed. 1218 1310385419000000 dread This has been done in cset:bd0f83a2e287 for release 1.4.2. Deployed to ckan.net. 1219 1310471415000000 dread This is a known issue and of low importance. 1219 1310670324000000 erilem "The issue is due to IE7 (and IE6) not supporting {{{inline-block}}}. [attachment:patch-1219-A0.diff] works around the issue. I'm reopening this ticket in case you want to consider my patch. But note that I won't be offended if you close it as wontfix again. I agree this is pretty minor, and that working around old browsers' limitations isn't always a good idea." 1219 1310740534000000 dread Patch applied in cset:8c05d27db224 for 1.4.2 release. 1221 1310475683000000 dread "Exception details: {{{ Module ckan.controllers.user:194 in request_reset << h.flash_error(_('No such user: %s') % id) try: mailer.send_reset_link(user) h.flash_success(_('Please check your inbox for a reset code.')) redirect('/') >> mailer.send_reset_link(user) Module ckan.lib.mailer:69 in send_reset_link << def send_reset_link(user): user.reset_key = make_key() model.Session.add(user) model.Session.commit() >> user.reset_key = make_key() AttributeError: 'NoneType' object has no attribute 'reset_key' }}}" 1221 1310556607000000 dread "Other edge cases and error conditions do not work properly either. Fixed in cset:4737c86d1d81 for release 1.4.2." 1222 1310556618000000 dread Fixed in cset:4737c86d1d81 for release 1.4.2. 1223 1310573893000000 dread pudo did this in cset:1ab655b7b76a and cset:7a9c6b1060cf ready for release 1.4.2. 1225 1310642407000000 dread Done in ckanext repo cset:70fb690a0769 1226 1314029434000000 dread This appears to have gone away. No sign of it for two weeks on any OKF sites. 1228 1311086612000000 dread Done in cset:88e47b68e42d for 1.4.3 1229 1312191530000000 amercader Done, except for Revision related stuff, which probably overlaps with work being done by David Raznick. 1229 1319639472000000 dread I found another direct use of the d.b. in the home controller. 1229 1319645778000000 dread Replace code in home in cset:f19f9c5bec94 on default for release 1.5.1. 1230 1311154142000000 kindly "The standard way to add tables in a plugin has converged upon putting the tables in the iconfigurable plugin. This runs at the correct point for when the application runs normally. For testing however there are issues due to the tables potentially being dropped, especially for the sqlite case. The fix is to make sure the iconfigurables are run at the start of each set of tests, hence adding it to the ckan_nose_plugin. This is not pretty, but good enough. cset:8531b9fc1ee2 " 1231 1315948921000000 rgrp @kindly: Does this fit in with current plans? If not suggest we close as invalid/wontfix or put into the backlog (though given lack of content not sure that is very useful ...) 1231 1325474087000000 rgrp Closing as wontfix (or duplicate). We now have stats in footer as of #1576. googleanalytics (e.g. http://thedatahub.org/analytics/dataset/top) would be nice to have but should be part of stats 1231 1325474447000000 rgrp See #1101 for other duplicate ... 1232 1315948536000000 rgrp Removing from ckan-v1.5 as do not think any of this is top priority (correct me if I'm wrong!) 1233 1315948668000000 rgrp Removing from v1.5 as not a priority atm. 1234 1311183328000000 dread Fixed in cset:1ed3fde56a61 for 1.4.3. 1236 1311351476000000 dread Working on it on branch enh-1236-view-package-revision 1236 1311699843000000 dread This is merged to default (for rel 1.2.5) but isn't brilliant yet because of bug #1238. 1236 1311759816000000 dread The bug is in fact in the way I work out what revision I'm displaying - it just looks at the PackageRevision. 1236 1323090508000000 dread This feature is now released, but there is a remaining bug which I've put in this ticket: #1509 1236 1330019788000000 dread Went into CKAN 1.4.3 1238 1311583876000000 kindly "I cannot reproduce this, to me it looks like it does get the correct revision. If you for example look at the package a millisecond before i.e http://ckan.net/package/osm%402010-11-30%2000%3A21%3A49.627829 the tags are no longer there that were added in that revision." 1238 1311759784000000 dread Apologies, I have a problem instead with the way I identify the revision in #1236. 1239 1311442163000000 rgrp Fixed in cset:adf254b7c507 (see https://bitbucket.org/okfn/ckan/changeset/adf254b7c507) 1239 1312453156000000 dread Fixed in 1.4.2 release. Broken in UI in 1.4.1 and API in 1.4 & 1.4.1. 1240 1314180488000000 dread "I basically agree, but just thought I'd add a couple of comments into the mix. > GET at root returns whole package set (a *bad* idea) Sure it has a cost, but I don't see what the problem with this is if the list is cached. Forcing developers to paging through the list of packages is putting unnecessary obligations. I'd say that the top two RESTful operations are listing objects and getting a particular one, and you'd be taking away one of those. > Unify on /api/v{version num}/... structure Where is it not currently unified?" 1240 1316022145000000 rgrp add autocomplete 1240 1324319119000000 rgrp assigning to kindly for review 1240 1324320107000000 dread "Although I'm generally in favour of Action API over RESTful, Action API does need some sorting out wrt listing methods, advertising what params are allowed and checking those that are passed. Using the Action API is so hard without reading the code! The RESTful API is handy when you want to do something quickly in a browser. I'd not want to lose these two, for example: * to look at a package /api/rest/dataset/xyz * API version etc. at /api/util/status (coming with ckan 1.5.2) It's also really handy in demos and to send as links to people on the list." 1240 1325473312000000 rgrp IMO this is non-urgent and can move out of v1.6 as we have enough more important stuff to do. 1242 1323173292000000 thejimmyg This text is no longer present on the login page. 1243 1328000405000000 rgrp invalid as things have changed and this has been fixed or obsoleted. 1244 1323089995000000 dread No progress yet 1245 1311864530000000 annapowellsmith "Would be nice to list some extensions somewhere on this site (features page perhaps). Along the lines of: http://docs.ckan.org/extensions.html#finding-extensions It wasn't until I wrote this list that I realised just how many extensions there were, and what it said about the CKAN community that there were so many... So I think it would be a good sales point. " 1245 1317422640000000 rgrp Closing as now mostly done and not really a ticket item. 1246 1311863583000000 dread "We have an unresolved issue with the display of the 'open data' icon that showed up when we upgraded ckan.net to the latest beta last week, but I didn't realise the search page http://ckan.net/package was showing all packages as open (the revision page and package pages seem fine). Last week when I upgraded from 1.4.1 to 1.4.2beta, there was an exception thrown when loading the home page. The exception was in doing the isopen call. It was solved by making some code changes directly on the ckan.net server to make it an attribute query instead of function call. It was strange because this behaviour didn't occur on my machine and all the tests passed. Here are the manual patches on ckan.net (the application.js change is unrelated, by someone else): {{{ diff -r adf254b7c507 ckan/public/scripts/application.js --- a/ckan/public/scripts/application.js Sat Jul 23 18:16:15 2011 +0100 +++ b/ckan/public/scripts/application.js Thu Jul 28 14:10:23 2011 +0000 @@ -2,6 +2,7 @@ $(document).ready(function () { CKAN.Utils.setupUserAutocomplete($('input.autocomplete-user')); CKAN.Utils.setupAuthzGroupAutocomplete($('input.autocomplete-authzgroup')); + // $('.resource-url').parent( }); }(jQuery)); diff -r adf254b7c507 ckan/templates/_util.html --- a/ckan/templates/_util.html Sat Jul 23 18:16:15 2011 +0100 +++ b/ckan/templates/_util.html Thu Jul 28 14:10:23 2011 +0000 @@ -43,7 +43,7 @@ standard package listing -->
    • + class=""${'fullyopen' if (package.isopen and package.resources) else None}"">
      ${h.link_to(package.title or package.name, h.url_for(controller='package', action='read', id=package.name))} @@ -60,7 +60,7 @@
      - +
    • @@ -72,7 +72,7 @@
    • - +
    • ${h.icon('lock')} Not Openly Licensed diff -r adf254b7c507 ckan/templates/home/index.html --- a/ckan/templates/home/index.html Sat Jul 23 18:16:15 2011 +0100 +++ b/ckan/templates/home/index.html Thu Jul 28 14:10:23 2011 +0000 @@ -64,6 +64,7 @@

      Recently changed packages

      + ${package_list_from_dict(c.latest_packages)}

    • ${h.nav_link(c, _('Add a package'), controller='package', action='new', id=None)}
    • +
    • Upload Data
    • ${h.nav_link(c, _('Tags'), controller='tag', action='index', id=None)}
    • ${h.nav_link(c, _('Groups'), controller='group', action='index', id=None, highlight_actions = 'new index')}
    • ${h.nav_link(c, _('About'), controller='home', action='about', id=None)}
    • diff -r adf254b7c507 ckan/templates/package/read.html --- a/ckan/templates/package/read.html Sat Jul 23 18:16:15 2011 +0100 +++ b/ckan/templates/package/read.html Thu Jul 28 14:10:23 2011 +0000 @@ -56,7 +56,7 @@
    • - +

      This Package is Open

      @@ -80,7 +80,7 @@

      - +

      This package is Not Open

      Either because it is not openly licensed or is missing }}}" 1246 1311885360000000 pudo "I'm afraid to say I don't understand any of this: the faulty display of openness is triggered as removing the brackets turns the isopen() check into a check for the existence of the method. Reverting the changes on the server seems to work without generating exceptions. What exactly is the issue that was supposed to be fixed originally? " 1247 1311955176000000 rgrp Done. See http://ckan.readthedocs.org/ and http://docs.ckan.org/ will propagate soon. 1248 1319797763000000 dread "Fixed in cset:188d27a56c89c35f97f3a79fdd0a6460223f86cf on branch release-v1.4.3. This was only seen in CKAN 1.4.3beta." 1249 1311959356000000 pudo fixed in cset:0398243111d4 1250 1312370201000000 pudo "The issue is caused after search returns from solr and tries to get objects from the database. These are by default ordered by name and we need to restore the original order after fetching them from the database. Fixed in module." 1251 1314406332000000 rgrp Really non-urgent and can be done once #1252 is done (which is dependent on v1.5) 1251 1317315159000000 dread rgrp did this last weekend. 1252 1314406271000000 rgrp Move to v1.5 since this will coincide with release. 1252 1316876747000000 rgrp Announced. thedatahub.org now running for 2+ months. Today did wiki.ckan.net redirect and hope to do thedatahub.org redirect. 1252 1317315135000000 dread This is done now? 1252 1318160334000000 rgrp This was done ~2 weeks ago. 1256 1312409296000000 kindly This has also caused problems in authorization for deleted packages. 1256 1312813529000000 kindly fixed cset:357cf9377b25 1256 1319812556000000 dread It sounds like this became an issue with CKAN 1.4.3 and the fix goes into CKAN 1.5. 1258 1312813614000000 kindly cset:e22d4e385fc8 1258 1319812452000000 dread I believe this was broken with the introduction of moderated edits in CKAN 1.4.3 and this fix went into 1.5. 1262 1314029150000000 dread "Pudo - I'm not sure what's meant here. Are you talking about moderation for new users? Or some other sort of enforcement? Is it spam you want to target? The links you're concerned about - are they the spam ones in the User's 'about' field? If so, I think we *should* allow users to have links, although continue with applying them with 'nofollow' and attack spammers back in a concerted strategy across all editable fields - see ticket #1257." 1263 1314029279000000 dread This was fixed in cset:8d293393a9f2 for release 1.4.3. 1264 1314406294000000 rgrp Nothing urgent about this so moving to backlog. 1264 1318244619000000 zephod Created #1377 to handle bugs revealed while doing this work. 1264 1318245716000000 zephod Complete; cset:b216952644aa 1265 1312566052000000 dread Fixed in cset:a1b8dd169f4e on 1.4.3 release and merged to default. 1266 1312797477000000 dread Fixed in 987dc4fe078e on 1.4.3 branch and default. 1267 1312889130000000 dread John thinks it is a problem with WebOb 1.0.6 1267 1312890962000000 dread This error has been occurring on at.ckan.net which has webob 0.9.7.1-1 1267 1312894071000000 dread Running tests with WebOb 0.9.7.1, we see the error! Am now trying to fix. 1267 1312897724000000 dread "Fixed in cset:db5bf5b45b4f for ckan 1.4.3. Later versions of webob add the charset header for unicode errors (webob/exc.py:generate_response): {{{ extra_kw = {} if isinstance(body, unicode): extra_kw.update(charset='utf-8') resp = Response(body, status=self.status, headerlist=headerlist, content_type=content_type, **extra_kw ) }}} This was fixed in webob cset:25fc14d49623 which went into WebOb 1.0. This problem affects versions from CKAN 1.4.2 back to 1.3.1. We had been avoiding this in CKAN previously by only doing abort() with a plain string. " 1268 1317212499000000 kindly fixed cset:eebbe6071741 1270 1312912188000000 dread Fixed in cset:ca5295c9c8cf and released as ckanclient version 0.9 1271 1313753663000000 rgrp Closed in cset:749cb3a087e3 1272 1313745867000000 rgrp "Assume these are the 3 spatial fields listed on http://wiki.ckan.net/Geospatial_Capabilities. I'm a big +1 here. Starting point would be a bit more documentation on that wiki page. Questions: * Is this work going in ckanext-spatial? * What is being proposed here, I assume, is to automatically index those fields in our geospatial search. * Should we develop a simple display widget using google maps or similar (this is orthogonal to the WMS preview i think) which *just* uses the core fields. This is probably a separate ticket and possibly separate from ckanext-spatial extension." 1275 1314110016000000 johnglover "Completed in branch feature-1275-solr-search. Up to date with current default but not yet merged into default. John" 1275 1319812967000000 dread Going into release 1.5. 1276 1314886010000000 johnglover "Added to branch feature-1275-solr-search Don't seem to be able to apply weighting to fields at index time using solrpy (see open issue: http://code.google.com/p/solrpy/issues/detail?id=6). So, switched to using the newer Solr edismax query parser and have applied weights to fields at search time. Current weights are as follows: name: 4 title: 4 group: 2 tag: 2 text (everything else): 1" 1276 1315948394000000 rgrp Closing since done (though not yet in main!) 1277 1314868204000000 johnglover Done in branch feature-1275-solr-search for all package searches (which use Solr). Resource and Tag searches still use the old ckan query parser. 1277 1315297186000000 rgrp This just requires merging into main but has been deferred as once solr branch is merged solr is *required* to run CKAN at all (we are thinking of having basic support for sqlite/dbs out of the box ...) 1277 1315948417000000 rgrp Closing since done (though still need to merge into default) 1279 1314303732000000 rgrp Wow, that's a pretty big omission. authz groups are important and are used. I think we should discuss this in next planning meeting. 1279 1314304326000000 rgrp In addition it appears that authz group was left out of the form refactoring so new and edit page use old formalchemy stuff. Should fix this. 1279 1338212028000000 ross Duplicate of #2366 1280 1328786670000000 dread David tells me that this was fixed in CKAN when we moved to SQLAlchemy 0.7 #1433 which went into CKAN 1.5.1. 1281 1314021919000000 dread "Done by pudo in cset:eaf342823caf on default branch, so headed for release 1.4.4. ""Work around the fact that locale changes do not affect the running request and therefore led to an incorrect flashing message. Now manually changing language.""" 1282 1314404540000000 rgrp "Done in cset:4b6838a820bb. Have not done either of the optional items. JS testing for what we have is relatively hard and of dubious value. Re CDN stuff requires work and of doubtful of immediate value since can just configure cache or main site to serve static stuff directly. (Likely more value from compressing js and css ...)." 1283 1326720891000000 dread Since we switched completely to SOLR, this has been true. SOLR doesn't index deleted packages so you can't search them, but they can be read/edited by admins. 1283 1330083933000000 dread This went into CKAN 1.6 1284 1315212258000000 dread "Fixed by rgrp in cset:1952445d2802 on release-v1.4.3 and default. Comment: do not create revision for changes to users as they are not revisioned. * This had already been removed in cset:8d6fde0e2196 (then in user controller) but Adria seems to have reinstated this when refactoring things into the logic layer. This 'bug' explains the huge number of empty revisions we have been seeing on ckan.net." 1284 1315212603000000 dread I've added tests for this (to prevent yet another regression!) in cset:1b77ce8ce560. Closing. 1284 1315214358000000 rgrp I'm not sure this is the sort of thing we need tests for (we can have too many tests!). Something worse discussing ... 1284 1315215626000000 dread Really? How many times do you want to fix this bug again? For the sake of three lines of test... 1285 1314269414000000 dread Another instance is when the database tables are not created and you want to both display a sensible error and have an email alert. #1290 1285 1323173125000000 thejimmyg "I could write some middleware to handle this. Can we come up with a list of exceptions not to catch or should I create a base exception called ""NoEmailReportTriggeredException()"" which exceptions for this purpose have to be derived from or which code has to raise? " 1285 1323178678000000 dread Isn't the point is we don't to raise an exception at all - we want to call a method to send the email and then return from the controller a normal html page. Don't we just need a method to get the error email address from the config and send an email. 1286 1342436420000000 dread This can happen now, if it hasn't already. 1289 1317315211000000 dread Discussions have not resolved this either way. Decided to leave it like it is for now. 1290 1314270894000000 dread "Improved on option 2. For requests to Home controller, the first thing it does is check the user. I catch SQL errors that are to do with missing tables and that now produces a template-less error saying the site is off-line and then mentions database initialisation. Done in cset:484e50c4f989 on default for ckan 1.4.4." 1291 1315948475000000 rgrp @kindly: is this not done? (I know there is more work to be done here but though table creation was done ...). If not need to move to next milestone so we can close this one. 1292 1315999202000000 dread i'm def +1 for removing API stuff entirely from i18n for the present 1292 1320145677000000 dread " Review process of releases and updating strings (pot) and translations (po) files c.f. http://lists.okfn.org/pipermail/ckan-dev/2011-May/000718.html http://wiki.ckan.net/Release_process updated with basic process Get language changing thing to get home page to mostly work - push to key strings translated (or don't advertise the translation?) Welcome message is translated in all languages. Most language have upped their percentage translated significantly and Swedish and Polish added as well. Review long list of strings for improvement Consolidate any? Get rid of i18n for strings in api About 50% of the strings are templates, 30% are in the logic layer and 20% are in the controllers. Removed a handful from ckan/lib/base.py that were only displayed in the api, but the logic layer ones are generally errors that can be displayed in the Web UI as well as the API, so not ideal to remove. Try and remove javascript Tom did this. Add in extensions This could be done. Have raised ticket #1292 for this, next release. #1374 Fix switching to en from other default language Done #1417 Fix browser language detection Done" 1292 1320145732000000 dread These fixes all went into release 1.5. 1293 1314701675000000 johnglover "== Changes == * All templates * Routing: /package/ -> /dataset/ (including API) * Tests: h.url_for(controller='dataset') rather than package ... * One test to check redirect works? * Docs == Leave == * Stay with Package in code e.g. stay with 'PackageController' * also leave stuff like pkg = .... (gradual migration) * Leave extensions == Notes == * Package -> Dataset * Data Package -> Dataset * respect capitalization" 1293 1314905799000000 johnglover "Branch: feature-1293-rename-package-to-dataset Routing and test editing should be finished (all tests passings). Have not added test for redirect yet." 1293 1315214527000000 rgrp Reassign for completion (templates and docs) as john is away. 1293 1315246628000000 zephod "cset:https://bitbucket.org/okfn/ckan/changeset/4a98589af6c1 Fixed" 1294 1315948112000000 rgrp Removing #1311 as that is not directly related to dataset edit / create. 1294 1315948257000000 rgrp Done with merge into default in cset:8bb0720a2150 and deployment to http://thedatahub.org/ 1295 1315820564000000 rgrp This is mostly done but some final tweaks and name disabling to be done. See work in feature-1294-ux-improvements-dataset 1295 1315948009000000 rgrp Completed with merge into default in cset:8bb0720a2150 1296 1315403275000000 zephod No longer doing inline resource editing on dataset/view; instead use the edit form. 1296 1315821293000000 rgrp Have now done resource add pretty effectively and most of edit done (should complete today). 1296 1315948027000000 rgrp Completed with merge into default in cset:8bb0720a2150 1297 1315403878000000 zephod No longer doing inline editing on the view page. 1297 1315403974000000 zephod Largely done in cset:8901e722b158 1297 1315404322000000 zephod all done: cset:71096cd7b27d 1298 1321199769000000 seanh I've been working on this in this branch: https://github.com/seanh/ckan/commits/feature-1298-activities-table 1298 1323106307000000 seanh See the new super ticket and its branch: #1515 1298 1323710593000000 dread Sean says: This is done on the activity stream branch. Just would like David Raznick to review it. 1298 1324384191000000 seanh "David, Ignore my link above to a branch on my ckan fork. Now that I have permissions I'm pushing my branches to the okfn ckan repo on github. My super branch for the activity streams feature is '''feature-1515-activity-streams''' on the okfn ckan repo on github. This page comparing my branch to master is particularly useful: https://github.com/okfn/ckan/compare/master...feature-1515-activity-streams (click on the Files Changed tab) For reviewing this ticket, the relevant changes to review are: * ckan/lib/activity.py, all of it * ckan/model/activity.py, all of it * My changes to ckan/model/meta.py * My changes to ckan/model/package.py * My changes to ckan/model/resource.py * ckan/tests/models/test_activity.py, all of it The other changes on my branch are for other activity streams tickets that follow this one. The super ticket #1515 has an overview of it all. " 1299 1325866115000000 rgrp No estimate as yet? (Or is it in super ticket? (Link?)). Also isn't this a duplicate of #1510 (or vice-versa)? 1300 1315906283000000 rgrp page_heading and optional_footer done in #1108 1300 1315908961000000 rgrp Move of js to optional_footer done in cset:dcb21f0d976f. As part of #1294 have done a lot of css adding. Not sure how much more to do here. 1300 1319794520000000 rgrp Side bar menu is switchable (though not sure how -- zephod has done this!) CSS tidying and labelling of divs done in #1380. So closing! 1303 1314978842000000 dread Adria fixed this in cset:7637d8694388 on default. I transplanted it to release-1.4.3 and have updated thedatahub.org. 1305 1314971164000000 nils.toedtmann "Always use ""localhost"" as mail relay. On all servers, the local MTA should be configured correctly (e.g. in this case to relay via mail.fry-it.com), also to be able to send rootmail. I fixed it on IATI (s049.okserver.org = ex eu23) and thedatahub.org (s055.okserver.org). Please verify that it's working now." 1305 1315317033000000 nils.toedtmann [15:22:02] Adrià Mercader: Yes it looks like it works on both thedatahub.org and iatiregistry.org 1306 1319797204000000 dread These are expressed as functional API tests - see tests/functional/api/test_action.py. There's not a lot of difference (apart from Paste coming in there) so no real need to duplicate API and logic layer tests, or to switch them to talk directly to the logic layer. 1307 1315246706000000 zephod cset:2c31fe24e1c9 1307 1315247298000000 zephod "Replying to [comment:2 zephod]: Oops: cset:cbd6677ec41e " 1309 1315586335000000 dread Duplicate of #1318 1310 1315820419000000 dread This was fixed cset:e49781cb74fd for 1.4.3. 1312 1315327743000000 dread "The default Content-Type is ""application/x-www-form-urlencoded"" (that is what curl sends, and what CKAN defaults to). Therefore these characters need url encoding (""=&;"" -> ""%3D%26%3B""). Here is an example of it working: {{{ (pyenv-ckan)dread@dread-laptop:~/hgroot/ckan$ curl -i http://localhost:5000/api/rest/package -d '{""name"": ""test7"", ""title"": ""Test2 %3D%26%3B""}' -H ""Authorization:tester"" HTTP/1.0 201 Created Server: PasteWSGIServer/0.5 Python/2.6.5 Date: Tue, 06 Sep 2011 16:36:32 GMT Pragma: no-cache Cache-Control: no-cache Location: http://localhost:5000/api/rest/package/5866a0f5-905a-44ff-91c3-c1489374a73e Content-Type: application/json;charset=utf-8 Content-Length: 571 {""maintainer"": null, ""maintainer_email"": null, ""id"": ""5866a0f5-905a-44ff-91c3-c1489374a73e"", ""metadata_created"": ""2011-09-06T14:06:40.735488"", ""relationships"": [], ""metadata_modified"": ""2011-09-06T16:36:32.543461"", ""author"": null, ""author_email"": null, ""state"": ""active"", ""version"": null, ""license_id"": null, ""resources"": [], ""tags"": [], ""groups"": [], ""name"": ""test7"", ""license"": null, ""notes_rendered"": """", ""url"": null, ""notes"": null, ""title"": ""Test2 =&;"", ""ratings_average"": null, ""extras"": {}, ""ratings_count"": 0, ""revision_id"": ""92ee0b2e-c5e4-47b9-b00c-b8e29662b950""} }}} Alternatively you can set the content type to ""application/json"" and you don't need to URL encode. e.g.: {{{ curl -i http://localhost:5000/api/rest/package -d '{""name"": ""test5"", ""title"": ""Test &""}' -H ""Authorization:tester"" -H ""Content-Type: application/json;charset=utf-8"" }}} To help work out these problems with encoding in the future, I've improved the error message to show the decoded JSON string. cset:55273629a552 which is headed for CKAN 1.5 release." 1313 1317076432000000 rgrp "Error was due to maintenance. Need both config options for recaptcha to work so either one can be used to check config. Re docs see #1358 (auto generation of docs on config from the config source). Given that only docs remain and that is part of #1358 am closing as duplicate." 1315 1315414218000000 dread Pretty good start added in cset:7a0e7a486df3 on default, aiming for release 1.5. Just needs some loose ends tidying up. 1315 1320173768000000 dread Tidied up in cset:39daf1afd64a for release 1.5. 1316 1338193724000000 kindly now get_or_bust function handles this. 1317 1315416649000000 dread "Ok, I think this example may be when searching for ""ira"". ""irina"" works me, but ""ira"" gives: {{{""ira"" matched several users}}}" 1317 1315822670000000 rgrp This is very minor IMO. I would defer this as part of a more general user system ux refactor. 1317 1325473762000000 rgrp Moving out v1.6 as minor. 1318 1315569362000000 dread "Problem tracked down to Adria's cset:4f1ca1eb2630 on branch feature-1229-db-out-of-controllers that missed out repoze.who in the redirects after login, when refactoring the user registration. This code went into default on 2011-07-28 for release 1.4.3. Fixed in cset:93731cd3b22e for release 1.4.3 and doesn't affect previous releases. Tests for user registration added too." 1319 1315476057000000 shevski Ideally we should also verify people's email addresses. Do we think that's overkill? 1319 1315904592000000 dread "No responses received about this on ckan-discuss, so I'm taking that as no objections to requiring email address on registration. I think that verifying email address is not worth it in this case, because the value of supplying an email address is only for resetting a password. The chance of needing a password reset AND having inputted an incorrect email address is very low, so I don't think it justifies everyone having to do this extra step. If we used email address more then that may change." 1319 1318426807000000 rgrp @dread: hope it is OK to re-assign to me but this is very similar to #1386 (and is a su bticket of #1343) 1319 1318527613000000 rgrp Fixed in https://bitbucket.org/okfn/ckan/changeset/130cad62163c 1319 1324036947000000 dread Goes into release 1.5.1 1320 1317391018000000 zephod Extend misc.py to regex find http and https, surrounding it with <>. ALSO note that CKAN flavoured markdown will not currently recognise dataset:my_data links (still uses package). 1320 1317730999000000 zephod "Slightly painful when links appear inside naughty HTML tags or in markdown tags. Restriction: Links are auto-linked if they appear at the start of a line or after whitespace. cset:f9dfb0506594" 1320 1318164603000000 rgrp Completed in https://bitbucket.org/okfn/ckan/changeset/f9dfb0506594 1321 1319558004000000 dread "We should not cache if there are flash messages to show (otherwise they get queued up for the next non-cached page view). Fixed in cset:945e2a6ce74f for release 1.5" 1323 1315594333000000 dread "thedatahub has a couple of auth groups with no name. The form allows it! Fixed in cset:bebd0a46ebc6 * form requires the name field * existing auth groups without name can be viewed/edited if necessary (referring by ID) This has probably been a problem since auth groups were added to CKAN." 1324 1315948336000000 rgrp Done and merged into main at cset:8bb0720a2150 (and deployed!) 1325 1315587113000000 dread Fixed in cset:0dd4d4457c97 for release 1.4.3. Previous Python 2.5 incompatible versions include 1.4.2 - 1.3.4.1 and maybe further. 1327 1320662446000000 rgrp Duplicate of #1397 1328 1340191065000000 ross Let's see if this is still an issue in 1.8 1329 1316624026000000 zephod "I've tried to do this wherever I could as I looked through ckan.pot for broken pieces of string. Generally it triggers errors unless things are very neatly structured (no nested links or ${a.variable} accesses, it seems) but things are quite a lot tidier. cset:d5bf4c7ba293 Does that count as 'fixed', or is this a permenant issue?" 1329 1317118323000000 dread I'll take a look to see if I can do any more and then will close it. 1329 1317404566000000 dread "POT looks far better. I couldn't see any outstanding problems. This has gone into ckan 1.5." 1330 1327060201000000 dread "This is simply incorrect. test_authz.py (I assume you mean the one in ckan/tests/functional?) tests that authz works all across the Web UI and API in enough detail. The 'total site lockdown' test - perhaps you mean the TestLockedDownViaRoles class which makes up a small part of test_authz? It was a test added due to an untested use case implemented in IATI, DataGM etc. These are currently active so still need testing. Regarding refactoring the tests to be ""proper"" then please open a new ticket, providing references to why it is not ""proper"" to test by searching for text in a web response." 1331 1325355631000000 dread This appears to be fixed in 1.5, 1.5.1 and 1.5.2a. 1332 1315999104000000 dread "Fixed in cset:860b357a2aec for release 1.4.3 and updated no.ckan.net. This problem was due to {{{[1:authorization group]}}} being translated as {{{[autoriseringsgruppe]}}} and I corrected it to {{{[1:autoriseringsgruppe]}}}. This word has a bold tag, hence the brackets and the number is used to identify which bracketed word is which (when you have more than one). It is a shame that Transifex doesn't draw attention to this problem and Babel doesn't spot the error on compilation." 1334 1316086680000000 dread cset:b01de680bf86 copes with the problem. Now to find out why revision.packages has some None values - very odd! 1334 1319798961000000 dread I think this has been a problem since 1.3.3. Fixed in v1.5. 1335 1315910117000000 dread "Done to a basic extent in branch feature-1211-drupal for release 1.4.3. Further tickets: * #1322: Action API improvements * #1315: Document /api/action" 1338 1316951545000000 amercader Reopening, because another change is needed to support custom schemas 1338 1317117738000000 amercader Fixed in eca1edce3a0f 1339 1316010607000000 kindly "I have fixed the isodata and made a slightly modified int_converter for this case. In the correct place and raises invalids on not being an int. cset:a4af115116bb The thinking was that the input of these fields would be through the api so the empty string case did not arise. These should clearly be converted to None (Null). What issues in general? Having done this lots of times before you always end up needing to write your own little validators as the standard ones never do what you want. Thats the point of them. Look in ckan/lib/validators if you need examples. So what you did was correct..." 1340 1315991534000000 dread Fixed in default in cset:33856d3ce84e aimed at release 1.5 1341 1315996368000000 kindly "This was run to delete the users and their mistaken revisions that where created. {{{ BEGIN; delete from revision r where r.id in ( select r.id from ""user"" join revision r on r.author = ""user"".name left join resource_revision rr on rr.revision_id = r.id left join package_revision pr on pr.revision_id = r.id left join group_revision gr on gr.revision_id = r.id where ""user"".created between '2011-08-15' and '2011-09-06' and gr.id is null and rr.id is null and pr.id is null and (""user"".name similar to '%[0-9]' or ""user"".fullname similar to '[A-Z][a-z]*[A-Z]%') and ""user"".name not like 'http%' ); delete from ""user"" u where u.id in ( select ""user"".id from ""user"" left join revision r on r.author = ""user"".name where r.id is null and ""user"".created between '2011-08-15' and '2011-09-06' and (""user"".name similar to '%[0-9]' or ""user"".fullname similar to '[A-Z][a-z]*[A-Z]%') and ""user"".name not like 'http%' ); COMMIT; }}} " 1341 1319649573000000 dread The majority of our 6000 users are still spammers. Can we simply elimenate those who've not made any revisions? Even if we take a few genuine users with it, these are pretty dead accounts anyway. 1341 1319709997000000 dread Reopening - this is still a serious problem. We can't link to the user page because it is so embarrassing: http://thedatahub.org/user 1343 1316078440000000 dread I don't think email confirmation is necessary - see discussion at #1319 1343 1318528138000000 rgrp Remove from ckan-v1.5 since remaining non-complete items (user page improvements and modal login won't get done in v1.5. 1344 1316078757000000 kindly fixed cset:f08215845dab 1345 1337961353000000 nils.toedtmann This becomes an actual problem on the CKAN farm s057, see http://trac.okfn.org/ticket/1245 1345 1338551926000000 nils.toedtmann See #2485 for workarounds containing the leak(s). 1345 1340190737000000 ross Moved out of milestone so that it gets looked at again soon 1345 1343215489000000 amercader See https://github.com/okfn/ckan/pull/73 1346 1316086314000000 dread "Fixed in cset:df299e69d456 on default for release 1.5. Problem exists on releases 1.4.3 - 1.3.1 and probably earlier too." 1346 1316094632000000 dread Also Transposed to 1.4.3.1. 1347 1316094608000000 dread Fixed on cset:7c3cc0fd6ae4 on branch release-v1.4.3.1. 1349 1316277490000000 zephod Done. cset:e074aee19923 1350 1316624288000000 zephod "Was going to reimplement an old workaround described by Friedrich, where a language.js file is served through the home controller (with appropriate settings in routing.py) and so ${_('string literals')} will be translated. See: * https://bitbucket.org/okfn/ckan/src/2832e49d6baa/ckan/controllers/home.py#cl-48 * https://bitbucket.org/okfn/ckan/src/2832e49d6baa/ckan/templates/home/language.js However the string-extraction mechanism seems geared towards Genshi templates and hacks are required to make it read a javascript file neatly (ie. without putting the *entire file* into a string for translation). A more elegant solution seems to be using an xi:include to pull in a genshi template which contains an html }}} where the jquery.html5.preie9fix.js is simply the [gist](https://gist.github.com/887560) published by Akkuma in a [jquery-tmpl issue (#36)](https://github.com/jquery/jquery-tmpl/issues/36#issuecomment-918495). This script patches JQuery on the fly on IE < 9 to rework in memory html 5 blocks handled by JQuery, so that it doesn't break. And this fixes the ""Upload a File"" button on IE. I didn't commit/pull this specific fix for the following reasons: * the script embeds innerShiv, which is a previous incarnation of html5shiv (now included in CKAN). It doesn't sound fine to include 2 pieces of code with similar effect, but I couldn't reproduce the fix with html5shiv! * in my environment I have included this fix sitewide even if I didn't spot other problematic features..I don't know if it would be better to include it only on the dataset edit page in the official CKAN code. Summary: I leave to you guys the choice on how to deal with this problem." 1788 1330624155000000 zydio "Erh...sorry for some formatting errors in my comment. I used markdown formatting for links, and used the #NUMBER format to cite issues, which obviously is interpreted by Trac as references to his own issues...my bad." 1788 1330644056000000 rgrp Re the Upload file button not sure what's best. We do have plans to replace jquery-tmpl with mustache in the near future (but not clear exactly *how* near). We'll think about this and get back to you. Thanks for the detailed debug report. 1788 1333133365000000 zephod "I've been performing a full sweep of the site to rebuild the markup & stylesheets using Bootstrap, and in doing so have managed to close off this ticket. Bootstrap allows us to delegate lots of clever functionality and layouts to Twitter's tried-and-tested code, making it easier to be compatible with IE7. It has brought up plenty of issues: Lots of Javascript was broken; plenty of pages generate broken doctypes causing the browser to go into quirks mode (which in turn causes the page to explode all over the screen); most dynamic content would fail to generate properly in one way or another.... (see the #1788 branch for a log of issues encountered). I've been refactoring our JS and have managed to strip ckanjs, too, because the only significant element we'd used was the File Upload view (which also now works in IE7). test.ckan.net has been redeployed on master; I'm browsing the site in IE7 and everything is looking great :-)" 1791 1329138315000000 dread Fixed in [release-v1.6 beaeaed] 1792 1329483249000000 toby package create update use correct schema in get_action 1792 1332755148000000 toby closing as done all that I was asked to do as part of this ticket 1792 1332872069000000 dread "What is the changeset for this? There is evidence of a branch ""enhancement-1792-api-logic"" but perhaps this name got fast-forwarded out through a bad merge? Is this merged into 1.6.1 release?" 1792 1332928847000000 toby This is in the 1.6.1 as far as I'm aware. The branch was deleted after the merge into master 1792 1333037710000000 dread "I have tracked down the changesets to: * branched off master at dff9f7a8 * merged master in for the last time (and became the 'new master'?) at 1a9227d9ff73c " 1796 1339586499000000 ross LXML removed, update requirements on the extensions 1797 1330550304000000 rgrp New API endpoint, Authorization and documentation done and merged to master. Time so far. 1.5d. https://github.com/okfn/ckan/commit/a054071e2e29e70e7cfa69df8c117ad5d5871a24 1797 1330863639000000 rgrp Data Viewer support for new DataStore in https://github.com/okfn/ckan/commit/9ab8b0283bb086eb4cd663ff73c27066bdd3c79a 1797 1331412469000000 rgrp "All done!! Help for data api done in https://github.com/okfn/ckan/commit/1d8e464f8542d4c33286bb93f4de50060665799f Checkbox for datastore enabled in https://github.com/okfn/ckan/commit/3f1320cd92ae0e775fde1b5eada156260c55e0a6" 1797 1331412644000000 rgrp "Merge fix https://github.com/okfn/ckan/commit/476a5bd32a3fac5d2dd85614f5d86c79f4ff6547 " 1798 1329395560000000 dread "Was fixed by Ian in https://github.com/okfn/ckan/commit/5a9054459e3833443bed3e118bbbb6c442e55b0b on branch feature-1453-flexible-tag-names. This has gone into CKAN 1.6." 1799 1330000389000000 dread Fixed in [master ea2d824] for both logging in and registering whilst logged in. Cherry picked to release-v1.6 1801 1343144718000000 amercader Closing as both old and new theme now show password reset links 1802 1329409718000000 dread Current progress - I have got most of the dependencies installed - many by pip, but some I had to revert to Windows msi installers. Pad with details here: http://ckan.okfnpad.org/windows. So far done: 1 day. 1802 1329490081000000 dread "I've achieved CKAN running with paster and without SOLR (using simple_search). Running locally on an old XP machine, performance seems fine - no noticeable issues. I installed most things into a pyenv using pip. pyutilib, psycopg2 and lxml were slightly problematic, but I have workarounds. I had problems pip and svn and the Cygwin window, so reverted to DOS prompt. I still had the Cygwin tools installed though, so next step is to try it without these at all - they are a big overhead." 1802 1329734716000000 dread Spent so far 1.75d 1802 1329830097000000 dread "* Refined install without using Cygwin - this seems a little easier. * Added example Apache & modwsgi deployment. * Documented extensively at: http://wiki.ckan.org/CKAN_install_on_Windows * Have not tried SOLR on Windows - I'm not convinced this is high priority, and normal install should be fine as it is so separate from CKAN. Spent another 0.5d. Total spent 2.25d" 1804 1329491156000000 toby This should be fixed in branch feature-1653-i18n_url_rewriting as we use different logic. However it may affect other return_to calls used elsewhere like adding a new package. 1804 1330000705000000 dread I'll fix this on release-v1.6 and then reassign to Toby to fix on feature-1653-i18n_url_rewriting too if the same issue crops up there. 1804 1330037202000000 dread "Fixed this on release-v1.6 in cset 557e72330db7. Spent 0.5h. Reassigning for Toby to look at branch feature-1653-i18n_url_rewriting " 1804 1330347185000000 toby This has been fixed in master for a while and was only a 1.6 issue 1805 1329760795000000 toby this is fixed in branch feature-1653-i18n_url_rewriting 1806 1330942356000000 toby version one completed 1806 1331295310000000 toby closing as initial work has been done if we need to amend then create a new ticket 1807 1330809641000000 seanh "Implemented the recently-changed-datasets activity stream on branch feature-1807-recently-changed-datasets in CKAN and merged it into master (but the default templates don't make use of it). Added the recently-changed-datasets activity stream to the front page template on branch master in ckanext-ecportal." 1809 1330528828000000 johnglover Caught exceptions in QA instead, error message stored as openness_score_reason 1810 1329918479000000 johnglover Deferred for now, moving to backlog. 1810 1335874745000000 johnglover Not using eurovoc tags. 1811 1329918517000000 johnglover Deferred for now, moving to backlog 1811 1335874765000000 johnglover Not using eurovoc tags. 1812 1335874864000000 johnglover Ross has fixed this via group authentication. 1816 1330863079000000 rgrp "As per detailed comments in http://wiki.ckan.org/Proposals#Apps_in_CKAN I think Apps is a confusing name. What we are talking about here is the ""Related Stuff"" item mentioned in that ticket. Rather than update ckanext-apps I suggest we do a rewrite. I've created a detailed super ticket at #2204 and assigned to you Adria for review and thoughts. (While I understand #2204 may involve more work which we maybe cannot do before relaunch I'd still recommend starting on that rather than spending time upgrading this extension -- if we have to disable it for relaunch I don't think that is a big deal ...)" 1816 1331302835000000 amercader "+1 to a new, more flexible extension. But in the meantime I've just spent a couple of hours making it work with latest CKAN, which was easier than expected, so we can deploy it with the new version of PDEU." 1819 1332163324000000 kindly Currently using package_show_rest. Should be moved to just use package_show but that is another ticket. 1828 1330525412000000 rgrp Fixed in https://github.com/okfn/ckan/commit/e6da48d9337575a83bc6a30896e68d1081b43847 1829 1330001828000000 dread "This was due to https://github.com/okfn/ckan/commit/f07bfdb9#diff-1 - string ""Language has been set to: English"" shouldn't have been translated. I've put in a patch to make the code intentions clearer and added a test: [release-v1.6 07d3a02]. This only affected the 1.6 beta, and is fixed for the release." 1830 1330089912000000 dread "Now if you set openid_enabled to false, you are not given the option in user settings to associate your account with openid. Cset: [master b295bde] I looked at removing the openid middleware, but we use it for logging out users, whether they logged in or not - it looks like friendly form plugin is not as good as the openid one in this respect! We could hack the logout function into the friendly form to sort this out, but not sure the effort is worth it at the moment." 1831 1330956631000000 ross Already have a branch for this. 1831 1331549284000000 ross Email is not currently 'unique' in the user model. Find out if this is intentional or not. 1831 1331655458000000 ross Decided to delay this until later but code is in feature-1831-login-by-email 2198 1339771453000000 kindly Already in the docs. 2201 1330765400000000 rgrp Duplicate of #1171. 2202 1338204729000000 toby these are available through the pkg_dict 2202 1338204801000000 toby "re-reading this it's slightly different re-opening" 2203 1348568961000000 dominik There is work on this in the dataviewer-iframe branch. 2204 1332341044000000 toby needs discussion - planned for 23-3-2012 2204 1332341080000000 toby http://ckan.okfnpad.org/feature-2204-related-stuff 2204 1332843232000000 toby still awaiting discussion 2204 1334077936000000 rgrp "Important comment: not sure how much we have thought through using this for storing queries / views / visualizations coming from our data viewer. In particular, wonder if this necessitates some kind of support for arbitrary json data ... Also (@icmurray): does this interact with our desire to do embedding?" 2204 1334218735000000 ross "I'd be happier adding this to another ticket rather than moving the target for this ticket half way through so have created #2284 Should only be a short ticket in the next sprint." 2204 1335287636000000 aron.carroll "Making some notes of front-end changes that I think need to be implemented sooooon. {{{ Implement better truncation (the expanded text should pop out on hover rather than expand the entire row) Show the type of related item (an icon would be good here) Test in IE Make the form work without JavaScript by accepting x-www-form-urlencoded. Add classes to first item in a row to clear the row (currently uses :nth-child CSS) Touch gestures and re-ordering (David's request) }}}" 2204 1335795665000000 ross Giving this ticket to Aron to fix the front end 2204 1337582843000000 rgrp "Surely this ticket was closed 3w ago? @Ross: can you close ..." 2204 1337594723000000 ross Was waiting for documentation, can close now. 2206 1331046486000000 johnglover "Close enough for now. Still have to restyle language drop-down to match template, but will fix when updating the list of languages. Also still to decide on menu styles/locations (for CKAN login and main menu, and ODP main menu)." 2208 1332251880000000 seanh "I've attached a ckan.pot file. To make it, I edited setup.py in CKAN core like this: https://gist.github.com/2135849 and then ran python setup.py extract_messages. (And I had ckanext-ecportal installed at ~/pyenv/src/ckanext-ecportal.) I guess I may need to run this again when the ckanext-ecportal source is finalised, so leaving this ticket open for now." 2209 1331547639000000 ross Option 4 is the most appealing at this point, although you probably should look at #2209 2209 1331547674000000 ross "Previous comment should read: Option 4 is the most appealing at this point, although you probably should look at #1649 " 2209 1331551170000000 ross "Current thinking is that option 4 is a default (as per ckanext-rdf) rdf output that is generated not in code (as currently) but using a genshi xml template to read the package into an RDF format (as if it were HTML). This would then be overrideable so that for ecportal where the format of the RDF is different (change of vocabs etc) we can just point the config to a new template. Pros: Easy to implement Easy to use Not hard-coded as currently Fast execution Cons: Requires knowledge of required RDF output if default is not useful RDF and not any of the other formats yet. Only works with package/resource/tags unless more work is done " 2209 1331551393000000 ross "Can't edit comments .. so Current thinking is that option 4 is a default (as per ckanext-rdf) rdf output that is generated not in code (as currently) but using a genshi xml template to read the package into an RDF format (as if it were HTML). This would then be overrideable so that for ecportal where the format of the RDF is different (change of vocabs etc) we can just point the config to a new template. Pros: Easy to implement Easy to use Not hard-coded as currently Fast execution Cons: Requires knowledge of required RDF output if default is not useful RDF and not any of the other formats yet. Only works with package/resource/tags unless more work is done " 2210 1331549350000000 ross This is now done through the logic layer member_create/member_list/member_delete 2212 1331719868000000 johnglover Added as a new paster command in ckanext-ecportal (run 'paster ecportal' for a list of commands and arguments). 2212 1331720191000000 johnglover Deployed on test server, where it imported 4087 datasets. A small number of datasets were not created as they failed CKAN validation - most of which had strange values such as 9999 and 0 for date fields (some also didn't have unique names). 2220 1331291241000000 toby "issues: 1) want to cache static files long term - need to add a ?ver=xxx to allow changes - this will need a change to all templates etc - could be seen as a good thing 2) we can only purge specific pages not whole cache so we need to know all the urls which want to be invalidated - including language variants 3) urls should be consistent so we should use id not name for datasets etc 4) for now we can just use short expiry times as this prevents some of these problems" 2220 1332950550000000 toby "Implementation of this has been carried out at the page level. ckan.page_cache_enabled is config option to enable it requires redis need to document it somewhere " 2221 1331561095000000 ross Fixed in https://github.com/okfn/ckan/compare/1eeda11...ad2358a 2223 1331558791000000 rgrp "Have introduced boostrap js and css on merge of #1797 https://github.com/okfn/ckan/commit/476a5bd32a3fac5d2dd85614f5d86c79f4ff6547 However need to remove blueprint and make sure we keep our original theme." 2223 1332246030000000 zephod "I have been working overtime to try and get this change straightened out as it's suddenly on the critical path for something. Have a look here: https://github.com/okfn/ckan/tree/feature-2223-bootstrap It's 80% of the way there; some form CSS needs overriding because Bootstrap tends to make things very large so they spill over. But on the whole the markup is in a steady state, which is what we need." 2223 1332842692000000 zephod Merged into master here: https://github.com/okfn/ckan/commit/0bf4b33955df3806785f9b42a55acaeb45596e25 2224 1332475944000000 rgrp Autocomplete based on bootstrap looks pretty non-trivial ... 2224 1338205358000000 ross Being done by Aron 2225 1343220502000000 aron.carroll There is a jshint.json config file that all JS in the new base theme conforms to. 2226 1332159640000000 seanh These deployment docs seem worth moving into sphinx as well: http://wiki.ckan.org/Deployment 2226 1332475468000000 rgrp All done (as of Tuesday in fact!). No doubt further work to do but that's a new ticket! 2227 1332843126000000 toby being done in other tickets 2232 1332331660000000 amercader "Fixed in 6b4c6c9 This was due to some characters not being supported by XML. We now strip them before indexing." 2234 1337237687000000 rgrp This looks interesting but is this a priority :-) (also no estimate as yet :-) ) 2234 1337239588000000 ross Yes, definitely need a replacement for wordpresser and sooner rather than later. Doing some experimental noodling to work out an estimate. 2236 1332153688000000 rgrp Done. See https://github.com/okfn/ckan/commit/0d970d3a0e523a1695830e8c8c28c0ba10a8d8b9 2238 1343392681000000 seanh Ticket #2771 encompasses this and related changes. Vocabs probably won't be deployed to test.ckan.net but a working example extension using vocabs will be added to core instead. Closing this ticket as a dup 2239 1334581838000000 ross Changes to schema/rdf output 2240 1332408769000000 ross Distributed to ckan-coord 2242 1333375436000000 seanh Conclusion: jenkins should work, may need some jenkins plugin magic to get it to show what branch each build was run on. 2250 1332680690000000 rgrp Is this now how we are doing it? I thought we were integrating with RDF store (so it was more about sorting out the data viewer / recline stuff)? 2251 1332331650000000 rgrp Big +1: everyone wants to know page views. Would like detail of how this goes into interface. Downloads already being tracked. Also isn't this just an extension to ckanext-googleanalytics. 2251 1332332390000000 toby "not really part of ckanext-ga as we need to sit first in the middleware for speed reasons. I'm looking at recording the details and will do a summary page like ga extension. As far as other uses like showing popular resources etc then we need to decide where the data will live - on the resource or separate. Downloads tracked? where is this done in the code/data stored" 2251 1332340874000000 toby "notes from meeting 21-3-2012 US1 As a Site Admin / Visitor (?) I want to see how often a page has been viewed (every page) and how often resources have been downloaded. US1a Next to a resource or a dataset see how often it has been downloaded / viewed US1b I want to see datasets or resources ranked by most downloaded or viewed US1c See a trend graph for a dataset (and resources) Adminstrative Dashboard (?) * I want to see the traffic breakdown by country to my site ... * Ditto for browser type, language, etc etc * I want to see it graphed over time ... == Implementation Details == 1. How do we store this data in CKAN? 2. How do we track (and store)? 3. How do we display * Config option ckan.status.enabled = False (by default) === Storing Data === How does ckanext-googleanalytics do this? Current table: package_id | count_recent | count_total Move to a new stats_summary table id | item_id | object_type | stats_type (total, month_yyyy_mm, ...) | value Do we store this data into the search (solr) so we can search by it? === Displaying Data === * Helper functions / dictize: * Helper function: h.stats_get(object_type, id, stats_type) * h.stats_top_ranked(object_type, number) -> returns object_dicts or just labels or ... * Change to dictize * Location in the default theme (do we show for example in search results too!) * Support for ranking by most popular in search? === Tracking Data === * Our own solution (just write to site_tracking) * Google analytics (plus extension for retrieving data) <- would need a refactor * Piwiki ==== Own Solution ==== site_tracking table id | url | timestamp | action (page_view, resource_download) | * Make javascript to make request to ckan to store clicks and page views. * Add middleware so these requests do not go through pylons and just store data quickly. " 2251 1332340947000000 toby "TODO look at 1. How do we store this data in CKAN? 3. How do we display * Config option ckan.status.enabled = False (by default) " 2251 1332524123000000 rgrp Update description in great detail. 2251 1332776926000000 toby "we want resource show downloaded/viewed on resources show on the dataset total/recent" 2251 1333535572000000 toby "we have various options on tracking unique users seems the sane approach. * unique daily view - needed for nice graphing * unique total views * total unique daily views - higher numbers and feels more right - easier to calculate if original data is lost/or archived for now will collect all and then decide which to use for display as gives us flexibility if needed display ------- why is group listing of datasets inconsistent with main search (TDH at least)? should we make these consistent? group listing allows downloading of resource without tracking (this should be fixed) if we make listings uniform do we want the download links or not? I don't like the term download when it is just a link (maybe that's just me - what would be better?)" 2251 1333644538000000 toby "most work completed: outstanding issues * add a prune tracking_raw data to cli * add indexes to tracking_raw * add summary page for new data format * add api counter 1) do we record the url or action etc (action seems more sensible) 2) add in __call__ or in action? " 2251 1334074772000000 toby api counter moved to ticket #2282 as needs better specification 2251 1334144999000000 toby "For meeting today with Adria * How to add tracking to solr index. Can the data live in tracking_summary? Do we need any extra indexes on table to help indexing? * How will we add order by popularity to search? * Can we add resources too? * Will search be best way to find most popular packages/resources or should we get that pages data from tracking_summary?" 2252 1333108787000000 amercader "There is a new ticket for the same schema as forms issue: #2268 The exception was fixed on [https://github.com/okfn/ckanext-inspire/commit/0dff1f6 0dff1f6]" 2252 1333374836000000 amercader This needs more work on ckanext-dgu and testing. 2252 1334567495000000 amercader Fixed on 0dff1f6c2e 2253 1340038490000000 toby closing as nothing has ever happened with this ticket 2254 1332343198000000 toby "What do we want to be able to do? what do we use the stream stuff for at the moment? la la la" 2254 1332950678000000 toby "plan of attack a) get removal of TDH from core - this will give a better idea of the issues involved b) have a think" 2254 1338210316000000 toby "closing. we have fanstatic integration in ckan.demo work tdh is it's own extension h.snippet is in 1.7 ITemplateHelpers in 1.7 This allows most of the issues resolved. Full removal of the IGenshiStream filter still neads some form of in template hooks but this is being looked at as a possiblity in ckan demo and can be considered out of scope for this ticket closing" 2255 1332752936000000 rgrp "I would say Public is default for Organization with Private being an option (or do we make this a config option?). What about ownership by users? > New datasets can be created from the organization form, in which case the organization will be set in the dataset form dropdown (with privacy set to private - see below) Does this mean like the setup on groups on the datahub (i.e. forward to new dataset but with some things hard-coded, organization/group?)? Or does it mean actual new dataset form on organization page." 2255 1332772340000000 ross "> What about ownership by users? Good question. WIll add to description. > > New datasets can be created from the organization form, in which case the organization will be set in the dataset form dropdown (with privacy set to private - see below) > Does this mean like the setup on groups on the datahub (i.e. forward to new dataset but with some things hard-coded, organization/group?)? Or does it mean actual new dataset form on organization page. The former, although currently we will have a slightly different form in the organizations extension, until such a time as we're ready to replace the existing one. Tried merging the two forms but it ended up with such a mess of logic that it was unmanageable. " 2255 1332934041000000 ross Changed description based on docs of stuff to verify. 2255 1334582784000000 ross Review + restrict resource access (when private) 2256 1333620795000000 toby "I'd like to remove all 'custom' icons and just use bootstrap ones - is this reasonable? they are added in different ways but h.icon() could be easily adapted for this. minimal .sj .css files" 2256 1333720237000000 rgrp Not sure what ticket means (remove TDH from core?). Also agree that we should glyphicons from bootstrap where possible going forward - though that would seem to fit more naturally with #2224 (Simplify javascript and css dependencies and add minified version) and have little to do with TDH versus core :-) 2256 1334051526000000 toby "@Rufus, The idea of this ticket is to separate TDH into an extension rather than it being incorporated into core. This should help TDH by making it possible to push fixes without needing to do a new release of core. It should also help make core more stable and as slim line as possible. New features can be tested in TDH and then if they have a wider appeal they can be merged into core. There is a strong desire for this within the dev team." 2256 1334051733000000 rgrp @toby: indeed and I'm strongly in favour of the basic idea too (and largely a dev question so up to you guys though feature / functionality implications would need to be talked through). My point was that custom icon stuff was #2224 :-) 2256 1334052087000000 toby "@Rufus, Thanks, I think there is overlap and I'm happy to leave that bit of work to you (or future ticket owner) I was just adding some brief notes to this ticket on some of the outcomes I'd like to see and hence why I'd be happy to simplify the icons." 2256 1336046098000000 toby I'm closing this ticket now ckanext-datahub exists and is live - the other work is included in other tickets 2257 1332950416000000 toby pushed first round of changes into master 2257 1336731023000000 toby ckan.restrict_template_vars defaults to true for 1.8 release 2260 1333373978000000 johnglover Still some work to do on this, template needs to be finalised. 2260 1333554069000000 johnglover Finished for this iteration, waiting for design work to be completed before making further changes. 2261 1332838253000000 toby "new config option ckan.root_path = /path/from/root/with/{{LANG}}/substitution " 2261 1332840056000000 toby completed plus documentation updated 2261 1332840112000000 toby please test logging in/out as the redirect was not testable but should be correct 2264 1333635637000000 seanh "Many strings have been fixed (and committed on master). Still to go: When uploading a file: - The ""Browse..."" button. This comes from the user's browser? - Hangs when I try to upload a file, so the rest of this UI is unchecked. The licenses on the dataset add, edit and view pages, ""License Not Specified"" etc. Dataset state on dataset view page, e.g. ""active"". When adding or editing resources: - The resource types (Data File, etc.) The strings seem to come from resource.resource_type. - ""You can use Markdown formatting here"", ""Should a DataStore table and Data API be enabled for this resource?"", ""Dates are in ISO Format — eg. 2012-12-25 or 2010-05-31T14:30"". I can't get strings with hyperlinks in them to translate here. Dataset view page: - License not specified. Dataset search page: - ""Tags"", ""Res_format"" and ""Groups"" in the sidebar. Leaving because this code has been refactored on feature-1821-multilingual-extension. Fix after that branch has been merged. Dataset view page: - The resource types, e.g. ""plain text"". - The field names under additional information: cache_last_updated etc. Dataset history view: timestamps not localised. Login page: - ""Login failed. Bad username or password. (Or if using OpenID, it hasn't been associated with a user account.)"" This string is marked for translation but I can't seem to get translations to work for flash messages. Also ""Sean Hammond is now logged in"" flash message. Register page: - ""Missing value"" in error message when you try to register and don't give an email. The string seems to come from formencode. stdtrans() not working? User view page: - Activity streams strings. Activity streams genshi function needs a little refactoring to be i18n-friendly. Users page: - ""less than 1 month"". This string comes from Pylons' time_ago_in_words() function which doesn't appear to support i18n. New auth group (and edit auth group) page: - ""Unique identifier for group. 2+ chars, lowercase, using only 'a-z0-9' and '-_'"". This string is marked for translation but for some reason it isn't working. - ""Name: Please enter a value"" (in an error msg) - ""User name"". The string comes from class FormBuilder? Also, need to check that all tooltips are translated!" 2264 1334348070000000 seanh "A few strings are still left to be marked for translation: - http://trac.ckan.org/ticket/2289 - http://trac.ckan.org/ticket/2288 - http://trac.ckan.org/ticket/2287 - http://trac.ckan.org/ticket/2290" 2265 1332865837000000 dread "I started work on this and need to park it somewhere - here will do for now. I was using the 'MLT Component' http://wiki.apache.org/solr/MoreLikeThis : https://gist.github.com/2217684 But I think it would make more sense to use the 'MLT Handler' http://wiki.apache.org/solr/MoreLikeThisHandler which needs enabling by adding this to the config {{{/usr/share/solr/core1/conf/solrconfig.xml}}}: {{{ }}}" 2266 1332932634000000 dread Fixed in [release-v1.6.1 ed2772b] with test. 2267 1343135541000000 toby this is not important as far as phase 2 is concerned - but you can do if you want 2267 1343229146000000 aron.carroll Fixed in eade4e3 2271 1334567582000000 amercader This was caused by an human error. 2275 1334159668000000 johnglover Done in branch feature-2275-group-logo, still need to decide on how to style. 2278 1333441177000000 rgrp https://github.com/okfn/ckan/commit/b7b32820a9132811028e7ccc243b2d2339492d4d 2281 1333967862000000 dread Done in [release-v1.6.1 9ecbfa5][release-v1.6.1 9ecbfa5] 2283 1340623843000000 kindly No super tickets anymore. 2284 1337583675000000 rgrp This needs connecting to the user stories being generated at http://ckan.okfnpad.org/dataexplorer-user-stories 2285 1334580266000000 rgrp "Recline improvements are done. https://github.com/okfn/recline/issues/88" 2288 1338204815000000 ross Auth groups are being deprecated 2289 1340887101000000 toby needs docstrings 2292 1335021860000000 rgrp "This annoyed me so much I just fixed it: https://github.com/okfn/ckan/commit/5eca7d5e37c0ef392b768b8b3768b2c3f93448b5 Issue was that our auto-completion depended on structure of the HTML which changes with bootstrapification. BTW: have deployed this via copy and paste on datahub so i can use it. Will want to revert this when you do a deploy. @Ross: I note that the same bug probably affects organization auto-complete." 2293 1339062853000000 ross "Intercepting the group name change and forcing a reindex of all of the datasets within that group seems like the obvious approach (rather than more lookups by ID after processing search results). Should take less than a day." 2296 1334567141000000 icmurray "* ensure the read_ajax and history_ajax actions work in the routing. * ensure the moderatededits.js works with the new templating * ensure that the css works with the new templating * ensure that the stream filters work with the new templating. ~ 6 days work" 2297 1334569632000000 seanh http://ckan.okfnpad.org/27 2298 1340285049000000 toby related to http://trac.ckan.org/ticket/2579 move sort functions to helpers 2302 1335516537000000 ross Will eventually want to allow customisation of different CSS but designs are currently under review/change. 2302 1342095074000000 toby "@ross do you want to give this to aron or me?" 2302 1342097600000000 ross Have assigned it to Toby. Not sure how high priority this is, but only provides a very minimal level of configurability. 2302 1343315555000000 shevski "My ideas for this could include one day: One page where the portal admin can choose and/or change the: 1. Portal name & tag line 2. Logo (to be displayed in header) 3. About page text 4. Featured dataset & text on homepage ? 5. Featured groups for homepage ? 6. Footer links 7. Colour scheme 8. Homepage background (?)" 2302 1343596669000000 shevski I reckon getting this live, esp with a easy to fill styled text header (see Sam's design) and a UI for changing About page text, featured dataset on homepage and welcome text is key 2302 1343636153000000 toby "@shevski, please do not try to change the meaning of tickets/add features or they will just get closed as won't fix. featured datasets is nothing whatsoever to do with this ticket so adding it to the ticket just causes me to get upset / demotivated etc. please do not do this. It ruins my day. Also comments like see sams design is of no value without a url in fact it has negative value. :( I have changed the priority of this ticket to account for this but next time i'll just close it." 2302 1343820855000000 ross Just a quick note, if Rufus asks about editing about page, he's talking about this ticket. 2303 1334654727000000 ross feature-2303-rdf-export-command 2304 1335878372000000 seanh I'm setting the priority of this high just because it's a low-hanging fruit for this sprint, as it was almost done last sprint. 2304 1335885463000000 seanh Setting minor again after cleaning tickets out of this sprint 2304 1337073870000000 seanh Currently implemented in this branch https://github.com/okfn/ckan/compare/master...feature-2304-follow waiting for code review and merge into master 2304 1337158641000000 seanh Finished and merged into master 2305 1335879165000000 seanh Setting the priority to major as I think it's worth getting this done since the follow support (which is almost finished) is a bit pointless without this. 2305 1335885478000000 seanh Setting minor again after cleaning tickets out of this sprint 2305 1340879812000000 seanh Implemented by Sven 2311 1334837867000000 toby "most work has been completed comments can be added to templates via ${h.disqus_comments()} recent comments via ${h.disqus_recent()} basic i18n works for languages except norwegian which reverts to english needs ckan 1.7 awaiting clarification re urls changing from under us" 2312 1340188741000000 ross "We have a ticket for object ownership (group and package) #2548 which will make it much easier to implement this functionality of showing user owned datasets in a 'virtual group'. Will close this in preference to the other ticket which is needed for other features too." 2313 1334756921000000 seanh "Note that c.new_facets has now been renamed c.search_facets. Before this ticket is closed the old c.facets should also be removed from the template context. It's put there by ckan/logic/action/get.py:package_search(), which also puts the new c.search_facets there." 2315 1334919522000000 dread Done, with tests and bootstrapification of the form: [release-v1.6.1 b23b357] 2317 1340033433000000 kindly Getting replace as part of new theme. 2319 1340624083000000 ross Resolved with rework of UI 2322 1337339587000000 ross "+1 on these, although ../download instead of ../raw and ../view instead of ../viewer " 2322 1337864855000000 ross "Viewer almost exists with the /embed link and so have just used that with a better default size to provide a prettier url. Can still use the width=&height= that are used in the /embed url. " 2322 1339749795000000 rgrp When will this go live? Will this be in v1.7.1? 2322 1339750135000000 ross No, new new features in point releases, we only put bug features in unless there is an extremely pressing need. 2322 1339750208000000 ross "Going to try that last comment again. No, no new features in point releases, we only put bug fixes in unless there is an extremely pressing need." 2322 1344086640000000 rgrp "Can I confirm this is in v1.8 ... " 2323 1335516967000000 ross "Doesn't make sense in Organizations mode. Adding existing dataset would mean that dataset would have to already be in another organization. Leaving this open though to fix in case we decide that datasets *can* be in more than one organization at a time." 2323 1340188400000000 ross Can create a new ticket if the requirements change. 2325 1335366188000000 dread Done in [release-v1.6.1 6343a9c] 2329 1346664271000000 ross Moved to 2.0 branch for new UI. 2330 1336149571000000 icmurray "Implemented, providing two methods of passing an action's parameters. Either, each parameter is specified as a url parameter, eg {{{ curl http://127.0.0.1:5000/api/3/action/package_search?q=police }}} or, for cases where a more complex data_dict is required, the data_dict can be parsed as a json-encoded dict in a single url-parameter called ``data_dict``, ie: {{{ curl http://127.0.0.1:5000/api/3/action/package_search?data_dict={'q':'police'} }}} As part of this ticket: - I noticed that all module-public members of the action modules were being loaded as actions (including functions imported from other modules). So I've cleared up the action modules to prevent this. - I've cleaned up the action modules to use ``get_or_bust()`` when accessing a passed in data_dict outside before it's been validated by a schema. This provides better error messages for clients. - provided a decorator, ``@side_effect_free``, for users implementing their own actions through the IActions interface to be able to expose their custom/overridden actions as a GET request through the action API." 2330 1336387328000000 icmurray "Replying to [comment:3 icmurray]: > > or, for cases where a more complex data_dict is required, the data_dict can be parsed as a json-encoded dict in a single url-parameter called ``data_dict``, ie: > > {{{ > curl http://127.0.0.1:5000/api/3/action/package_search?data_dict={'q':'police'} > }}} > Another option is to have the data_dict json-encoded in the GET request's body. This is possible, but 1) Although permissable, it's not, as I understand it, conventional to have a GET request's body form the options of the request. 2) Caching proxies won't inspect the body of the request, only the URL. " 2331 1337782689000000 kindly "This is a wontfix. I think terms should be ored by default. All modern search engines work like this. If there is an issue due to relevancy (i.e if you type mulitple words and your result not coming near the top) then we should use this examples so we can tweak the results." 2331 1338390995000000 rgrp "@kindly I really don't get this but I may understand poorly. Google does not work by or-ing does it (as i add things to the search query I get *less* results not more ...). It also makes it really hard to find stuff: as I add query terms to my query I get more stuff not less which makes for terrible finding of stuff - I appreciate that there is a scoring aspect but that seems secondary." 2331 1338455981000000 kindly "Scoring is primary in my opinion. Who cares if you have 1000 results if the top few are yours. If things are hard to find we need to change our relevancy first. So if you have examples of where you think the scoring is wrong then please make a ticket for that. Google, I imagine, just has a cutoff of anything under a certain score not being shown. We could do that as well but it would take some working out what we wanted that score to be. Full ""And"" queries also limit any accidental discovery, especially of rare terms. If you do not get your search exactly correct then you get nothing, which is bad. Obviously you can still AND things or +things. The correct solution to this is adding a minimum match parameter which is a middle ground. eg you can say that you want to match over half of the terms. ""thing1 thing2 thing3 thing3"" means you have to match at least 2. There are many options described here. http://wiki.apache.org/solr/DisMaxQParserPlugin. You can change this in an extension if you want it just requries adding an mm field to the before_search in the Ipackage controller. I do not personally think we should change it as default. I am closing it as wont fix as its trivial to change and is a philosophical difference not a technical one." 2331 1343814758000000 rgrp I'm re-opening this. This is a reasonably significant UX problem -- not just a philosophical difference. It is just *weird* to get *more* results as I add search terms rather than less. Please can we have this fixed ... 2331 1343862230000000 kindly As I said I do not not agree with it being a UX problem or *wierd*. *wierd* is not acceptable in response to a thought out comment. So I am closing it again. 2331 1345191494000000 rgrp "Re-opening yet again. This is something brought up by entire client team on several occasions :-) We really do need to fix this as it is really poor UX. To give, yet another, example: http://datahub.io/dataset?q=thatcher+wages This returns 15 datasets yet only one of them has any mention of thatchers (the first one!). (The others are there because they match wages). This isn't what I expect. I expect *only* to see datasets that match *both* terms ... " 2331 1345192722000000 ross "This is the dullest game of ticket tennis ever :D Google don't OR the results, they have an explicit operator for ORing query terms and default to ensuring that all terms are present (in one form or another) in the documents. I understand the argument about cut-offs and relevancy scores, but for a large search engine they have the benefit of scale both in resource and documents. I spent 6 months of my life tweaking weightings for n-dimensional vector space search, I don't think it is likely to be a number we can, or will find by accident. Given that Marcus also got confused by the current behaviour I think we should either: a) Make it configurable b) Just change it to what has been asked for Don't care which. " 2331 1356474344000000 rgrp Now transferred to github at https://github.com/okfn/ckan/issues/258 2332 1337582891000000 rgrp @rossjones / @amercader: can you do a full review and generate sub-tickets as necessary (obviously may not all be this sprint ...) 2345 1335878420000000 seanh Setting priority high as we want this done by Friday for the 1.7 release. 2345 1336492725000000 seanh Work on this has started in branch feature-2345-action-api-autodocs 2345 1337962454000000 seanh Done but now I need to fix readthedocs again 2346 1338205553000000 johnglover Will create a separate ticket for 'organization' to 'publisher' renaming, the rest are done. 2347 1335962835000000 ross [ ] Check that dataset privacy can be taken into account. 2347 1336039458000000 ross Keeping track of UI changes required, and copied Aron 2347 1337238738000000 rgrp "This looks great Ross. Couple of things not included (which I have taken liberty of adding to the description as well - you will probably want to tidy up): * Related is a pretty terrible name in the UI - much better to have it called Apps, Ideas etc (perhaps title tag could even give more details e.g. ""Apps, Ideas, Visualization and other material related to this dataset"") -- btw i thought this was in #2332 but realize it wasn't (apologies for that!) * Documentation, documentation, documentation - AFAICT I can't see anything in v1.7 or master docs. I imagine this would be a short section like http://docs.ckan.org/en/ckan-1.7/commenting.html (and probably coming right after that in the ToC) which says * What it does * How to enable (and perhaps includes a nice-screenshot!) * How to customize (e.g. can one customize the list of options of things one can create (e.g. can I set it to just be app and idea or ...) * Also liase with Mark W re a blog post about this from the user perspective (a screenshot walkthrough ...) * Clearly mark this old extension as deprecated from v1.7 forward: https://github.com/okfn/ckanext-apps (I just met someone last week who was working on integrating this and had no idea it was replaced by something better in v1.7) " 2347 1339602763000000 shevski "Name-wise: what do we think of 'Related Media' - that suggests news articles, apps & visualisations to me I'n not sure about relating 'ideas' to specific datasets. Looking at the way ideas work in DGU & pd.eu - it's more general and portal-wide so should flesh out user stories for this before adding ideas to related extension" 2347 1340613938000000 ross Needs some UI work now, have changed status to blocker which appears to be the only way to denote something as blocked (awaiting resource). 2347 1343126819000000 icmurray "Small fix to create action, outlined in [1] to be done. Said I'd do it, so moved ticket over to myself. [1] https://github.com/okfn/ckan/pull/62" 2348 1338193014000000 ross "A fairly good start has been made on this at https://github.com/okfn/CKANClient-J Lowering priority for completion for now" 2349 1336037341000000 ross Openlink hoping to get RDF dump into Virtuoso today. 2349 1336555121000000 ross Chased Openlink today for update 2349 1338193148000000 ross Initial DNS change doesn't seem to have taken, chasing Nils to see if he can take a look. 2349 1339075361000000 ross "Created symlink from /var/www/nginx-default/dump to the rdf output folder, accessible via http://s055.okserver.org/dump/ with daily dump available at http://s055.okserver.org/dump/rdfexport.tgz " 2349 1339428913000000 ross "Notified Hugh that http://datahub.io/dump/rdfexport.tgz will contain all of the RDF files every 24 hours if they want to fetch and import them. Have asked to make sure that this is satisfactory." 2351 1340615641000000 icmurray Scripts and docs in https://github.com/okfn/ecportal-server-setup 2353 1336037264000000 ross Started the user stories at https://docs.google.com/document/d/185sWPEO9cn1SbquH83UFcLme7xKdgfhCNqOo7pdsRKw/edit 2355 1336731240000000 toby "ckan.plugins.toolkit introduced to help control access to ckan for extensions. This should reduce breakage for extensions using this. ITemplateHelpers allows extensions to add helper functions so templates can be less reliant on Genshi filter" 2355 1338212946000000 toby "Need to do documentation which is #2362 so has own ticket closing" 2358 1336641577000000 amercader Took longer than expected due to last minute changes and problems with the packaged version. 2361 1337016768000000 amercader I actually spent some more time of this because I upgraded PDEU to CKAN 1.7, which was needed to fix the search index and a good testing for the release as well. 2362 1341314244000000 toby "first part complete part2 is the auto generation merge that is for 1.9" 2362 1342085420000000 toby need to get the auto-documentation done now 2364 1335890204000000 ross Review the FRY app and investigate alternative payment gateways. 2364 1336483769000000 ross Start of doc listing payment gateways is at https://docs.google.com/document/d/1E4lX6RnRLDO0ioieKRpVs9a-My9nh6lBjvVahWDm3SY/edit 2365 1336492342000000 ross See also #1165 and #1096 2365 1340266964000000 ross After discussion with DR, it was decided that multisite wasn't something we would approach now, instead going for individual (shared codebase) installs. 2366 1336395238000000 dread For my site I need the Authorization Groups sections gone from the authz pages, for v1.7.1, so I've made these check for the presence of any Authorization Groups in commit: [master 8947302]. So it's another useful stepping stone to getting rid of them cleanly. 2366 1336555441000000 ross I think the UI aspects have been removed in 1.7 with hopefully the rest of it going in 1.8 - assuming we have a clean migration script. 2366 1336560445000000 dread "Thanks Ross. It's the dataset authz and admin authz pages which still need Authorization Groups removing. e.g. http://thedatahub.org/dataset/authz/conservative-party-uk-donors-by-sector and http://thedatahub.org/en/ckan-admin/authz" 2366 1338215334000000 ross Need to make sure there is a clear migration between authgroups and organizations if desired. 2367 1336035031000000 aron.carroll Fixed in ckanext-datahub@4aa76740e3def64a80171df4417c195210f0eb25 2371 1338481174000000 toby implementation trial in ckan demo branch #2375 2371 1340038431000000 toby awaiting release and demo theme merge 2371 1345209675000000 toby closing as part of 2375 branches 2372 1343220160000000 aron.carroll These are now in https://github.com/okfn/ckan/blob/b31109b628a21072ec0c2dec2a5d5a1c2cc0ed9c/doc/coding-standards.rst#javascript-coding-standards 2373 1336142514000000 dread Fixed in https://github.com/okfn/ckan/commit/8b7141cd2ff619a0e35aaa31bbc1d17156647498 on master (and also on release-dgu1). 2374 1340287976000000 ross "count is returned at the top level of the dict (at the same level as results) Was added by dread on 2012-05-04, must have been in a recent patch." 2375 1338292898000000 toby stealing this ticket 2375 1340710303000000 toby moving milestone 2379 1340033026000000 ross Implemented in master 2380 1337868169000000 ross "datagm is VERY broken with 1.7 Waiting for datasuite UI changes." 2380 1340627558000000 ross Dupe of #2545 2383 1337872916000000 icmurray "This helper function won't parse UTC offset if passed a UTC offset as ""+01:00""; but incorrectly parses the UTC offset ""+0100"" as 100 microseconds. Resolution: - I've decided to keep the behaviour of not parsing UTC offsets because we currently don't store timezone information in the database; and I think that either dropping the tzinfo, or adjusting for it would appear to be ""magic behaviour"" from the user's perspective. Also, as this is a helper function, I think it makes sense to keep the behaviour as is, even if it's slightly non-standard. This fix [1] maintains all the old non-ISO behaviour: - not allowing UTC offset to be specified. - arbitrary delimiters between date and time values. - microsecond precision. But won't parse ""+0100"" as microsecond precision. I tried using a couple of python modules for parsing date strings; but none maintained the same behaviour. [1] https://github.com/okfn/ckan/commit/d6c9e48f8df5e926abd3fb1a1607f4f5a30075b4" 2384 1337100810000000 dread Done in 0bc06c6495c749d6b481164d59cceeca873acefb 2386 1337967006000000 seanh Done. ckanext-spatial and ckanext-qa are not activated because they don't work with the current version of CKAN. datastorer is not installed yet. 2389 1337237502000000 rgrp Just to note these were 2 *different* options. '''Either''' you do the fallback approach '''or''' we turn off DataStore by default and only use if enabled (which happens due to explicit enabling of DataStorer turning it on ...) 2390 1337100788000000 dread Done in d1c2ed08c90c2d94e1b2ced86e8758208a61ee96 2391 1338397982000000 seanh I've told them how to do it themselves and offered to help if they get stuck 2394 1337161917000000 dread Done in [master b486244] 2398 1337963342000000 seanh I'm closing this because we've got a whole bunch of tickets from the dev meetup about fixing the dependencies and requirements file that should deal with this problem 2399 1340706702000000 amercader Fixed in 493589aaf 2400 1338205947000000 toby In testing this is not replicable due to lack of info so closing 2401 1337273042000000 dread Done in master 112e405d0116bdcf196062520e3474e231337996 2402 1337302474000000 kindly cset: 12da5e9effeeb1aca0df321c355d8438647ef426 2403 1337302700000000 kindly cset: 05144f8621ee719c345373934e70719f46e87cf6 2405 1337583749000000 rgrp Assigning to ross for review and assignment. 2405 1338212357000000 ross Suggest Sam might be able to provide you the icon 2405 1338213564000000 icmurray "Questions for client team: 1. Did you have anything in mind for improving the ""Powered by CKAN"" text and icon? I'll sort out the vertical alignment of the text and icon; and see if Sam has a better icon. 2. The branded section itself is already customizable by a snippet (https://github.com/okfn/ckan/blob/master/ckan/templates/snippets/data-viewer-embed-branded-link.html). Is that enough? Or are you asking for something else? 3. Personally I think the URI should use the resource uuid, and not name. And from recent discussions on ckan-dev, I think the consensus is the same. How important is this for you? Perhaps it's something that could be overriden in the datahub extension." 2405 1340097846000000 icmurray Bump: Questions for client team above... 2406 1337766385000000 toby "I installed CKAN yesterday afternoon from source, using the instructions at http://docs.ckan.org/en/ckan-1.7/install-from-source.html Most things worked pretty smoothly and, luckily, when they didn't I was lucky enough to have David Raznick on hand to help sort them out. As a result I've made the following notes on where the doc could be improved. For some ghastly reason I am running Linux Mint, which is some variant of Ubuntu. It's not 10.04, so no package, hence the source install. Sec 2: Not wanting a mysterious file called 'pyenv' in my home directory, I created a directory ~/ckan and typed 'virtualenv pyenv' from there. Everything worked fine. But if I had needed the 'tip' it would have been a little confusing. It would also be worth reminding the user that the dubious package is one they've supposedly just installed, so if that went OK, they won't need to worry about this. The tip also isn't clear about under what circumstances I'd need to do the 'easy_install pip' - I didn't need to (is that also only when python-virtualenv is missing?) Sec 4: This worked, but the example command for installing a particular version is confusing. It says it is for 1.5.1, but replacing '1.5.1' with '1.7' fails because there is a rogue 'c' in the command. Probably it would be better to use the latest version (1.7) as the example anyway. Or is there a way of asking for the latest released version? I understand this checks out the whole Git repository with all history - I can't imagine why I'd need to do this to build a particular version. But David assures me it's very sensible. Sec 6: This worked, but it wasn't very clear what the purpose of the 'List existing databases' step was. Sec 8: So now I had to try to get Solr working using the instructions at http://docs.ckan.org/en/ckan-1.7/solr-setup.html This didn't work at all, in ways I can't now remember, to do with Java. In the end David suggested using Tomcat rather than Jetty (don't ask me what these do, but they have something to do with getting Solr to work ...) and this turned out to be much more straightforward to install and get working, with two caveats (below). David suggested maybe Tomcat should be the default suggested in this doc. - One problem with Jetty was to do with using the wrong version of Java, which seemed to be fixed using ""sudo update-alternatives --config java"". It didn't fix Jetty - not sure if it would also have been needed with Tomcat. - Soft linking the schema file didn't work - it seems Tomcat doesn't like soft links. We made a copy. After this everything worked, and I now have my very own CKAN. Hurrah! Mark " 2406 1337766413000000 toby "The 'post-installation setup' instructions at http://docs.ckan.org/en/ckan-1.7/post-installation.html claim to be for either package or source installations, but only work as stated if you have done a package installation. E.g. after a source install, ""paster --plugin=ckan user add admin --config=/etc/ckan/std/std.ini"" doesn't work (the ini file is somewhere completely different), but just typing ""paster user add admin"" from the ckan directory is fine. The instructions should also mention that the virtual environment should be activated. Mark " 2407 1340633440000000 amercader This will require some thought. Bumping to 1.9. 2408 1341497601000000 aron.carroll This will be done as part of adding javascript in phase 2 2408 1343657014000000 aron.carroll Fixed in df502fd9269859c73458f4ad8de75939220e0315 2409 1337688965000000 ross This is in the Package controller and may be related to upcoming ticket on removal of autoneg. 2410 1338193633000000 ross Tim Lebo and Keith Alexander are both now looking at how they can implement this themselves, so for now this is wontfix. We can reopen this as part of any future work with VoID 2411 1338210872000000 toby This is a long term goal 2414 1337869578000000 ross "LXML is currently used for markdown_extract and check_solr_schema_version Changing check_solr_schema_version should be possible, Used in markdown_extract to just strip HTML tags." 2414 1338471374000000 ross Merged by kindly 2416 1338464207000000 toby commit 37622d5 does this but probably in the wrong place but works for now. Need to revisit at some point. 2416 1340710252000000 toby moving to phase 2 as not causing problems but still needs proper fix 2417 1337864155000000 toby fixed in demo-theme branch 2418 1337875172000000 toby "all facets now shown and selected have

    • @aron styling of active facets with long names breaks. we could make the names shorter in snippet/facet_list.html via truncate or make the selected taggy thing wider" 2419 1337863024000000 toby "@aron the reason for this is that controller/package.py renders the form `package/new_package_form.html` as part of it's work and then makes this available a `c.form` to the template. I'm not sure I like this as an approach but it is how it is currently done. If you split your form into a new template `package/new_package_form.html` then you get all the errors etc. I think for now we should do this and then look at if doing it all in one template pass might be more sensible. reassign to me if you want me to do this else I'll leave it with you" 2420 1337950422000000 toby merged into master 2422 1337945204000000 seanh Closing because I'm not able to reproduce and it turned out floapps had publisher auth on which was overriding his rights auth 2423 1338193199000000 ross Duplicate of #2414 2424 1338193269000000 ross Duplicate of #2413 2428 1341234933000000 seanh This awaiting merge, pull request: https://github.com/okfn/ckan/pull/46 2428 1341314475000000 seanh Merged. The only thing I didn't do was add a test case that runs pip freeze and tests that all the right versions of deps are installed. 2430 1338397815000000 seanh http://trac-hacks.org/wiki/MasterTicketsPlugin 2431 1338397906000000 seanh https://github.com/davglass/github-trac 2432 1338205526000000 aron.carroll "It would be great to use something like http://trac-hacks.org/wiki/SimpleTicketPlugin to remove most of the options when creating a ticket. Suggestions to remove theme, repository, component and type unless they're actually useful." 2432 1338325120000000 shevski "Where do new tickets go to await triaging? Is priority set by whoever creates the ticket? Does this get revised? Will we have a field for dev time estimate or where will this go?" 2433 1337960203000000 toby "the affected actions =============== package_list package_relationship_create revision_show group_list package_update_validate It would be nice to get this fixed and I'm loathed to suggest adding a version 4 hack. If we make this change it will break things for existing users. This behaviour has existed since the action api was added as far as I'm aware So should we a) leave it forever and document b) break things and fix it - when 1.7.1 or 1.8?" 2433 1338397268000000 toby after discussion it has been agreed to keep the existing behaviour and improve documentation as it is likely to break existing applications etc 2434 1341234522000000 seanh "We've moved to AWS hosting with docs generated by Jenkins instead, see: http://trac.okfn.org/ticket/1254 There are still two issues to resolve: 1. Automatically updating the symlink so that docs.ckan.org always redirects to the latest stable version 2. Finding somewhere to put links to all the other versions" 2434 1350296272000000 amercader Seems to be working again, closing. 2437 1338484676000000 seanh "Please add to the file started to doc/coding-standards.rst Also, for the Python coding standards see http://wiki.okfn.org/Coding_Standards as a starting point" 2437 1338487089000000 seanh "I've also added the commit message guidelines as they reference a trac plugin I'm going to get installed. I'll leave the rest to you. I remembered that http://wiki.ckan.org/Main_Page has a bunch of pages under Developer Resources that should probably be updated and merged into these coding standards and deleted from the wiki" 2438 1338212234000000 icmurray "Proposed solution: Add a new q field which can be specified multiple times. Each q parameter is a string: `{field}:term`. `:` characters are escaped in the `{field}` by `\:`. Mark the `fields` parameters as deprecated." 2438 1340736112000000 icmurray "Ready to be merged [1] Brief synopsis of changes: - altered return type of the action: from Resource domain objects to dictized resources. This is required if this action is to be made accessible through the action api as the domain object is not json-serializable. The 'old' return type is still available by passing in a flag in the context. In fact, this is required as the search layer uses this action, and deals with the domain objects directly. - decided that escaping the ':' (as described in above comments) wasn't necessary as resource fields shouldn't have ':' characters in them anyway. - improved action's docstring - 'fields' parameter has been deprecated ( see #2603 ). Use of it by the search layer has been removed in favour of the new 'query' parameter. But it still works, and correctly maintains backward compatibility. [1] https://github.com/okfn/ckan/pull/37" 2439 1338214158000000 icmurray "The `fields` argument doesn't provide any extra functionality that the `query` argument can't provide. 1. Figure out what the reason for this argument is, just to make sure. 2. Deprecate it, and change any internal uses of it to use `query` field instead." 2440 1339421746000000 aron.carroll "Has some styling as of b0153ba but needs content to be applied. Toby, can you pull in the top 10 tags, as well as n of the most ""popular"" datasets and n of the most recent datasets. Actually just providing the functionality as helpers would be great. Do we actually have ""featured"" datasets implemented at the moment? Not sure what we're going to use for the box on the top left when on a new base CKAN instance." 2440 1339423966000000 toby "@aron, tags what functionality do we want for this? I can just reuse the facet_list snippet but then we have all sorts of interactivity. Do we want this more a static list of tags (linked to somewhere but no add remove stuff)? seems more sensible I think featured datasets I've no idea I'll try to discover" 2440 1339430956000000 toby "you may want to look at your css as the dataset is generated by the package_list snippet I've made all packages popular for now" 2440 1339497700000000 aron.carroll Fixed this up, last thing I think we should do is ensure that the datasets passed to the homepage have descriptions. 2440 1339498655000000 toby "dataset descriptions is a data issue but I'll keep this open for now. We can probably do an exclude if needed. " 2440 1339499074000000 aron.carroll Sure, but this is more about displaying nice complete examples on the homepage. 2440 1339581622000000 toby I am changing this to assigned not accepted as this makes trac easier to use 2440 1340373461000000 toby i'm closing we can make tickets for specific issues 2441 1338287243000000 aron.carroll Order by needs hooking up to form. Currently it's just a @