FOIwiki:ScraperSync

ScraperSync is a tool for maintaining an FOIwiki page as a mirror of a dataset from ScraperWiki. It attempts to preserve markup and changes in the page while propagating changes from the dataset.

Dataset requirements

The dataset must have a table called "swdata" containing a column called "name". The contents of this column form the texts of entries in a bulletted list in the wiki page.

List requirements

The start of the maintained list is marked by a special comment like this:

<!-- ScraperSync start { "scraper": "thing" } -->

After the word "start" is a JSON object which configures ScraperSync. It contains a single entry, "scraper", which names the scraper from which the data should be pulled.

ScraperSync understands several different list formats, and tries to remove markup and notes to work out what public authority is named by the item.

Output

ScraperSync's main output is a new version of the page source. This should be the same as the current page, but with any list entries that don't appear in the ScraperWiki dataset struck out. It also appends to this a list of those items that appeared in the source dataset but not in the wiki page. ScraperSync tries to automatically make this into links to WhatDoTheyKnow.

Creating a new page

ScraperSync has a mode for creating an entirely new page from a dataset. This can be most easily activated by running it with no options and choosing "Create new" and entering the URL name of the Scraper to use as the source.