Add tutorial for exporting data from InfluxDB and importing into CrateDB

Author: amotl

spikes/influxdb-to-cratedb/.gitignore

@@ -0,0 +1,4 @@
+.venv*
+*.wireproto
+*.py
+influxdb-write-to-postgresql

spikes/influxdb-to-cratedb/README.md

@@ -0,0 +1,80 @@
+# About
+Tutorial about migrating data from InfluxDB to CrateDB.
+
+# Status
+This is a work in progress.
+
+# Plan
+
+- Create data
+  https://github.com/influxdata/influxdb-python/blob/master/examples/tutorial_sine_wave.py
+
+- Export data
+  https://github.com/daq-tools/influxdb-fetcher
+
+- Import data
+  https://github.com/eras/influxdb-write-to-postgresql
+
+
+# Setup
+```sh
+# Install sine wave generator
+python3 -m venv .venv
+source .venv/bin/activate
+pip install influxdb
+wget https://raw.githubusercontent.com/influxdata/influxdb-python/master/examples/tutorial_sine_wave.py
+# Comment out the line `client.drop_database(DBNAME)` at line 60/61.
+
+# Install InfluxDB Fetcher
+wget --no-clobber --output-document=/usr/local/bin/influxdb-fetcher https://raw.githubusercontent.com/daq-tools/influxdb-fetcher/develop/bin/influxdb-fetcher
+chmod +x /usr/local/bin/influxdb-fetcher
+
+# Install iw2pg: Build Docker image
+git clone https://github.com/eras/influxdb-write-to-postgresql
+cd influxdb-write-to-postgresql; docker build --tag iw2pg .; cd ..
+```
+
+
+# Run with PostgreSQL
+```sh
+# Run databases and iw2pg side by side
+docker run -it --rm --publish 8086:8086 influxdb:1.8.3
+docker run -it --rm --env "POSTGRES_HOST_AUTH_METHOD=trust" --publish 5432:5432 --name postgresql postgres:13.1
+docker run -it --rm --publish 8087:8086 --link postgresql --volume $PWD/iw2pg-config-postgresql.yaml:/app/config.yaml iw2pg --verbosity=info
+
+# Create data
+python tutorial_sine_wave.py
+
+# Export data
+influxdb-fetcher http://localhost:8086 root root tutorial "SELECT * FROM foobar" > foobar.wireproto
+
+# Import data
+psql postgres://postgres:postgres@localhost --command='CREATE DATABASE tutorial;'
+cat foobar.wireproto | http http://localhost:8087/write?db=tutorial
+
+# Verify data
+psql postgres://postgres:postgres@localhost/tutorial --command='SELECT * FROM foobar;'
+```
+
+# Run with CrateDB
+```sh
+# Run databases and iw2pg side by side
+docker run -it --rm --publish 8086:8086 influxdb:1.8.3
+docker run -it --rm --publish 5432:5432 --name cratedb crate/crate:nightly
+docker run -it --rm --publish 8087:8086 --link cratedb --volume $PWD/iw2pg-config-cratedb.yaml:/app/config.yaml iw2pg --verbosity=info
+
+# Create data
+python tutorial_sine_wave.py
+
+# Export data
+influxdb-fetcher http://localhost:8086 root root tutorial "SELECT * FROM foobar" > foobar.wireproto
+
+# Import data
+cat foobar.wireproto | http http://localhost:8087/write?db=tutorial
+```
+
+
+## Bummer
+```text
+Result status PGRES_FATAL_ERROR unexpected (expected status:PGRES_TUPLES_OK); ERROR:  Relation 'pg_indexes' unknown
+```

spikes/influxdb-to-cratedb/iw2pg-config-cratedb.yaml

@@ -0,0 +1,7 @@
+databases:
+  tutorial:
+    db_host: cratedb
+    db_port: 5432
+    db_name: tutorial
+    db_user: crate
+    db_password: crate

spikes/influxdb-to-cratedb/iw2pg-config-postgresql.yaml

@@ -0,0 +1,18 @@
+databases:
+  tutorial:
+    db_host: postgresql
+    db_port: 5432
+    db_name: tutorial
+    db_user: postgres
+    db_password: postgres
+
+    #create_table:
+    #  regexp: /.+/
+    #  method: create_table
+
+    fields_jsonb_column: null
+    tags_jsonb_column: null
+
+    #field_columns:
+    #  - value
+    #  - field2