id summary reporter owner description type status priority milestone component resolution keywords cc repo theme 1134 CREP0003: Description and Configuration of Harvesters amercader "'''Proposer''': Adrià Mercader == Abstract == The new harvester interface allows to create harvesters for different sources, but right now harvesters don't have many ways to describe and configure themselves. We need a way of allowing them to: * Expose their type and other details so they can be used internally and on the UI. * Define configuration settings for particular harvester instances. == The Problem == === Harvester description === The current UI for adding and editing harvest sources is the same used in ckanext-dgu, and thus the 3 harvester types used in DGU to harvest various GEMINI realted sources are hardcoded in the form. The form will be migrated to a DGU-independent one, so we need the harvesters to provide all the necessary data. There is a current {{{get_type}}} method that returns the harvester type, but for make it compatible with the DGU forms, it returns a machine-readable string (e.g. ""CSW Server""), making it error prone. === Arbitrary configuration === In the current implementation, when the harvest process is started, ckanext-harvest looks for all the available plugins that implement the {{{IHarvester}}} interface and calls the appropiate methods for the current stage ({{{gather_stage}}},{{{fetch_stage}}},{{{import_stage}}}). At these stages, harvesters have no way of applying arbitrary configuration options, so all harvesters of the same type behave on the same way. For instance, the CKAN harvester needs a way to define the API version to use when harvesting remote instances (Right now, the version 2 is hardcoded on the code). == Specification == === Harvester description === Harvesters will need to provide the following information so the UI form can be built: * name: machine-readable name (e.g. ""waf""). This will be the value stored in the database, and the one used by ckanext-harvest to call the appropiate harvester. * title: human-readable name (e.g. ""Web Accessible Folder (WAF)""). This will appear in the form's select box. * description: a description of what the harvester does (e.g. ""A Web Accessible Folder (WAF) displaying a list of GEMINI 2.1 documents""). This will appear on the form as a guidance to the user. The way to provide it will be an {{{info}}} method that all harvesters must implement, which will return a dictionary with the previous elements: {{{ { 'name': 'csw', 'title': 'CSW Server', 'description': 'A server that implements OGC's Catalog Service for the Web (CSW) standard' } }}} === Arbitrary configuration === As different harvesters will have very different needs, we need to provide a way to persist arbitrary configuration flags for each harvest source. The more flexible way given the current architecture in my opinion would be to store the configuration options as a JSON encoded object as a property of the harvest source (There already is an unused DB field called {{{config}}} in the database) (Maybe using JsonType?). This will mean adding an extra field in the harvest source form to allow entering the configuration. This could be just a simple text field where users enter the JSON encoded object or a more clever mechanism (i.e an ""Add a configuration flag"" link that adds two new text fields for the key and value for each flag, and a mechanism to later build the JSON object). In any case, this should probably be hidden in an ""Advance options"" section. == Why do it this way == === Harvester description === The {{{info}}} method would provide a single point to get all the information related to the harvester, and future properties could be added to the dictionary returned without having to modify the interface. === Arbitrary configuration === There is an already existing {{{config}}} field in the database, so we won't need to change the model. Harvesters could access the config object at any of the stages. Of course they could provide default values in their implementations so users don't need to enter them everytime. == Implementation plan == === Deliverables === === Risks and mitigations === The highest risk on the harvesters {{{info}}} method side is that harvester implementation don't offer one of the necessary properties (namely name and title). This could fire a warning when showing the UI form or using the CLI. === Participants === Adrià Mercader to do it. === Progress === None yet." CREP new major ckan-backlog ckan Harvesting david.raznick@… ckanext-harvest none