small-waffle

This is a lightweight server for DDFCSV datasets, essentially a node.js wrapper around DDFCSV reader.
It is slower than the previous big-waffle solution in terms of selecting and filtering the datasets, but the idea is that it would have a strong cache in front of it and the speed would not matter.

limitations

can only serve years for now, see resultTransformer.js

Getting started

1. Clone this repo

to the server where you intend to run small-waffle from.
On a real server you will also need pm2 and nginx, when developing locally it doesn't matter

2. Configure the installation

See .env.example, copy it to .env and edit the variables

GITHUB_TOKEN=ghp_abcdefghijklmnopqrstuvwxyz1234567890 ALLOWED_DATASETS_GOOGLE_SPREADSHEET_ID=1u3dRq1s2zfjDXtYN1uDCa4LBiAo0igOjOWX2R185nQU

2.1 Create the control spreadsheet

spreadsheet should have a single sheet with columns id, githubRepoId (values are in form of github user/repo), branches (comma-separated), default_branch (values are optional)

<style> th, td { border: 1px solid #E8E8E8; padding: 3px 4px; } </style>

example:

id	githubRepoId	branches	default_branch
fasttrack	open-numbers/ddf--gapminder--fasttrack	autogenerated, master	master
country-flag	open-numbers/ddf--gapminder--country_flag_svg	master

2.2. Configure github token

the github token should be valid and have access to the datasets listed in the spreadhseet you can create the token in your github account under Settings -> developer settings -> Personal access tokens --> Classic. only one scope needs to be enabled: ☑️ public_repo (Access public repositories)

2.3. Get DDFCSV datasets

The datasets should follow ddfcsv format. You can start with some publicly available DDFCSV datasets on github.com/open-numbers, like in the above example

3. Start the server in DEV mode

run npm start, you should see the following output

Environment variables loaded successfully.
Starting small-waffle on PORT 4444
Updating allowed datasets from a google spreadsheet https://docs.google.com/spreadsheets/d/1u3dRq1s2zfjDXtYN1uDCa4LBiAo0igOjOWX2R185nQU
Got info about 2 datasets: fasttrack, country-flag
...
  === Loading dataset with slug fasttrack ===
  === Loading dataset with slug country-flag ===
...
  🟢 Load complete! This is not the complete sync. Run /sync to do that.

3.1. Check status

from a browser, try accessing http://localhost:4444/status you should see the output that starts with

  {
    "server": {
    "type": "small-waffle",
    "uptime_ms": 67708,

3.2. Sync datasets

from a browser access http://localhost:4444/sync in the console you should see

👉 Received a request to sync ALL datasets
Updating allowed datasets from a google spreadsheet https://docs.google.com/spreadsheets/d/1u3dR...
Got info about 2 datasets: fasttrack, country-flag
Cleaning up directories for all datasets
👉 Syncing dataset with slug fasttrack
...
Created a reader instance for fasttrack/master
🟢 Sync successful for dataset fasttrack
...
🟢 Sync complete for 3 datasets: fasttrack, country-flag

USAGE

//concept request http://localhost:4444/fasttrack?_language=sv-SE&select_key@=concept;&value@=concept/_type&=name;;&from=concepts

the link should redirect from /fasttrack to /fasttrack/master/COMMIT_HASH and return a JSON object such as

{
    "header": [
        "concept",
        "concept_type",
        "name"
    ],
    "rows": [
    [
        "gini",
        "measure",
        "Gini coefficient"
    ],[
        "journakilled",
        "measure",
        "Number of journalists killed"
    ],
    ...

//entity request http://localhost:4444/fasttrack/master?_language=en&select_key@=geo;&value@=world/_4region&=is--world/_4region;;&from=entities&where_$or@_un/_state:true http://localhost:3333/fasttrack/23e4dd45525ad1d0b7344f4935b7191972cf2ba1?_language=en&select_key@=geo;&value@=world/_4region&=is--world/_4region;;&from=entities&where_$or@_un/_state:true http://localhost:3333/fasttrack?_language=en&select_key@=geo;&value@=world/_4region&=is--world/_4region;;&from=entities&where_$or@_un/_state:true

//example of a datapoint request http://localhost:3333/fasttrack/master?_language=en&select_key@=geo&=time;&value@=pop&=lex&=gdp/_pcap;;&from=datapoints&where_geo=$geo;&join_$geo_key=geo&where_$or@_un/_state:true

//example of a concept schema request http://localhost:3333/fasttrack/master?_select_key@=key&=value;&value@;;&from=concepts.schema

//example of a datapoint schema request http://localhost:3333/fasttrack/master?_select_key@=key&=value;&value@;;&from=datapoints.schema