Skip to content

Commit 296a5eb

Browse files
pauldixpauldix
committed
refactor: create tables via /api/v3/configure/table, drop __init seed rows
init.sh now POSTs each user table's tags + typed fields to /api/v3/configure/table before creating caches and triggers, replacing the old "write a __init sentinel row at t=1ns to materialize the table" pattern. LVC/DVC creation and named UI queries need the table to exist at plan/create time; explicit creation removes the race against implicit first-write creation. Drops the matching `site <> __init` filters from ui/queries.py and the andon plugin, and documents the pattern in ARCHITECTURE.md, README.md, and influxdb/schema.md.
1 parent 7308104 commit 296a5eb

7 files changed

Lines changed: 154 additions & 37 deletions

File tree

‎ARCHITECTURE.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,20 @@ across batches). Both patterns recur throughout IIoT.
113113
| Request trigger | `request_andon_board` powers the UI's andon panel via direct fetch (with latency badge) |
114114
| Custom UI | FastAPI + HTMX + Jinja2 + uPlot dashboard — no backend service for the andon view |
115115

116+
### Table creation: explicit, not implicit
117+
118+
InfluxDB 3 auto-creates tables on first write, but caches and named queries need
119+
the table to **exist at create / plan time**. `init.sh` therefore POSTs each user
120+
table to `/api/v3/configure/table` (passing tags + typed fields as JSON) before
121+
creating the LVC/DVC and before the simulator starts writing. A 409 on re-run is
122+
treated as success, so init is idempotent. Without this, the LVC/DVC `create`
123+
calls — and any UI query that fires before the simulator's first batch lands —
124+
race against implicit creation and intermittently fail with "table not found".
125+
126+
This replaces an earlier pattern that wrote a `__init` sentinel row at `t=1ns`
127+
to each table to materialize it; that approach leaked a throwaway row that every
128+
downstream query had to filter out.
129+
116130
## 8. Token bootstrap
117131

118132
The `token-bootstrap` compose service generates an offline admin token on first boot,

‎README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,11 @@ the JSON response. See `cache-last-compare` in `CLI_EXAMPLES.md`.
6868
`distinct part_id` over today (~700K events at default config) returns in a few ms via the cache (vs. hundreds of ms scanning the table); exact latency depends on the query.
6969
See `cache-distinct` in `CLI_EXAMPLES.md`.
7070

71+
> The LVC and DVC are bound to a *named* table that must already exist when
72+
> `create last_cache` / `create distinct_cache` runs. `init.sh` creates each
73+
> user table explicitly via `POST /api/v3/configure/table` before creating the
74+
> caches — see `ARCHITECTURE.md` § "Table creation: explicit, not implicit".
75+
7176
### ⬢ Custom UI calling Processing Engine directly
7277

7378
> **⚡ The andon-board panel calls the `request_andon_board` plugin via `fetch` directly

‎influxdb/init.sh‎

Lines changed: 91 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -63,31 +63,98 @@ ensure_database() {
6363
}
6464

6565
# Caches and several UI queries reference tables by name and require them
66-
# to EXIST at query/create time. Seed each user table with one sentinel row
67-
# at timestamp 1 ns (1970-01-01 — outside every recent-time window the UI
68-
# queries) using tag sets the simulator/plugins never emit.
69-
ensure_seed_tables() {
66+
# to EXIST at cache-create / query-plan time. Create each user table
67+
# explicitly via POST /api/v3/configure/table so the schema is registered
68+
# before the simulator (and plugins that read those tables) start. The
69+
# alternative — implicit table creation by first write — leaves a race
70+
# window where the LVC/DVC `create` calls and UI queries can fail with
71+
# "table not found".
72+
create_table() {
73+
local table="$1"
74+
local body="$2"
7075
local token
7176
token=$(read_token_json "${TOKEN_FILE}")
72-
local seeds=(
73-
'machine_state,site=__init,line_id=__init,station_id=__init,machine_id=__init state="__init",reason="__init" 1'
74-
'temperature,site=__init,line_id=__init,station_id=__init,machine_id=__init temp_c=0 1'
75-
'vibration,site=__init,line_id=__init,station_id=__init,machine_id=__init rms_mm_s=0 1'
76-
'part_events,site=__init,line_id=__init,station_id=__init,machine_id=__init,part_id=__init,quality=__init cycle_time_s=0 1'
77-
'alerts,source=__init,severity=__init,line_id=__init,machine_id=__init reason="__init",value=0 1'
78-
'shift_summary,line_id=__init,shift_id=__init oee=0,availability=0,performance=0,quality=0,units_total=0,units_good=0,downtime_top1_reason="__init",downtime_top2_reason="__init",downtime_top3_reason="__init",downtime_top1_seconds=0,downtime_top2_seconds=0,downtime_top3_seconds=0 1'
79-
)
80-
local body
81-
body=$(printf '%s\n' "${seeds[@]}")
82-
if curl -sf -X POST \
83-
"${INFLUX_HOST}/api/v3/write_lp?db=${INFLUX_DB}&precision=nanosecond" \
84-
-H "Authorization: Bearer ${token}" \
85-
-H "Content-Type: text/plain" \
86-
--data "${body}" >/dev/null 2>&1; then
87-
log "seeded all user tables"
88-
else
89-
log "sentinel writes may have failed (table-already-exists is fine); continuing"
90-
fi
77+
local out code
78+
out=$(mktemp)
79+
code=$(curl -s -o "${out}" -w "%{http_code}" -X POST \
80+
"${INFLUX_HOST}/api/v3/configure/table" \
81+
-H "Authorization: Bearer ${token}" \
82+
-H "Content-Type: application/json" \
83+
--data "${body}")
84+
case "${code}" in
85+
200|201|204) log "created table ${table}" ;;
86+
409) log "table ${table} already exists" ;;
87+
*)
88+
echo "[init] FATAL while creating table ${table} (HTTP ${code}): $(cat "${out}")" >&2
89+
rm -f "${out}"
90+
exit 1
91+
;;
92+
esac
93+
rm -f "${out}"
94+
}
95+
96+
ensure_user_tables() {
97+
create_table machine_state '{
98+
"db": "'"${INFLUX_DB}"'",
99+
"table": "machine_state",
100+
"tags": ["site", "line_id", "station_id", "machine_id"],
101+
"fields": [
102+
{"name": "state", "type": "utf8"},
103+
{"name": "reason", "type": "utf8"}
104+
]
105+
}'
106+
create_table temperature '{
107+
"db": "'"${INFLUX_DB}"'",
108+
"table": "temperature",
109+
"tags": ["site", "line_id", "station_id", "machine_id"],
110+
"fields": [
111+
{"name": "temp_c", "type": "float64"}
112+
]
113+
}'
114+
create_table vibration '{
115+
"db": "'"${INFLUX_DB}"'",
116+
"table": "vibration",
117+
"tags": ["site", "line_id", "station_id", "machine_id"],
118+
"fields": [
119+
{"name": "rms_mm_s", "type": "float64"}
120+
]
121+
}'
122+
create_table part_events '{
123+
"db": "'"${INFLUX_DB}"'",
124+
"table": "part_events",
125+
"tags": ["site", "line_id", "station_id", "machine_id", "part_id", "quality"],
126+
"fields": [
127+
{"name": "cycle_time_s", "type": "float64"}
128+
]
129+
}'
130+
create_table alerts '{
131+
"db": "'"${INFLUX_DB}"'",
132+
"table": "alerts",
133+
"tags": ["source", "severity", "line_id", "machine_id"],
134+
"fields": [
135+
{"name": "reason", "type": "utf8"},
136+
{"name": "value", "type": "float64"}
137+
]
138+
}'
139+
create_table shift_summary '{
140+
"db": "'"${INFLUX_DB}"'",
141+
"table": "shift_summary",
142+
"tags": ["line_id", "shift_id"],
143+
"fields": [
144+
{"name": "oee", "type": "float64"},
145+
{"name": "availability", "type": "float64"},
146+
{"name": "performance", "type": "float64"},
147+
{"name": "quality", "type": "float64"},
148+
{"name": "units_total", "type": "float64"},
149+
{"name": "units_good", "type": "float64"},
150+
{"name": "downtime_top1_reason", "type": "utf8"},
151+
{"name": "downtime_top2_reason", "type": "utf8"},
152+
{"name": "downtime_top3_reason", "type": "utf8"},
153+
{"name": "downtime_top1_seconds", "type": "float64"},
154+
{"name": "downtime_top2_seconds", "type": "float64"},
155+
{"name": "downtime_top3_seconds", "type": "float64"}
156+
]
157+
}'
91158
}
92159

93160
ensure_caches() {
@@ -142,7 +209,7 @@ main() {
142209
wait_for_api
143210
ensure_token
144211
ensure_database
145-
ensure_seed_tables
212+
ensure_user_tables
146213
ensure_caches
147214
ensure_triggers
148215
log "initialization complete"

‎influxdb/schema.md‎

Lines changed: 40 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,41 @@
11
# IIoT schema
22

3-
This file documents the IIoT demo's InfluxDB 3 schema. It is descriptive — the actual
4-
tables are created implicitly by the first write. The `init.sh` script creates the
5-
database, an operator token, the Last/Distinct caches, and registers Processing Engine
6-
triggers.
3+
This file documents the IIoT demo's InfluxDB 3 schema. The `init.sh` script creates
4+
the database, registers each user table explicitly via `POST /api/v3/configure/table`,
5+
creates the Last/Distinct caches, and registers Processing Engine triggers.
6+
7+
## Why explicit table creation
8+
9+
InfluxDB 3 will auto-create a table on the first line-protocol write — but the LVC
10+
and DVC are bound to a *named* table that must already exist when `create last_cache`
11+
or `create distinct_cache` runs, and the UI's named queries plan against tables by
12+
name at request time. If the simulator hadn't started yet (or hadn't yet written to
13+
that table) when a UI partial fired or `init.sh` tried to create a cache, the
14+
operation would fail with "table not found".
15+
16+
`init.sh` therefore registers every user table up front via:
17+
18+
```http
19+
POST /api/v3/configure/table
20+
Content-Type: application/json
21+
Authorization: Bearer <token>
22+
23+
{
24+
"db": "iiot",
25+
"table": "machine_state",
26+
"tags": ["site", "line_id", "station_id", "machine_id"],
27+
"fields": [
28+
{"name": "state", "type": "utf8"},
29+
{"name": "reason", "type": "utf8"}
30+
]
31+
}
32+
```
33+
34+
A 201 means the table was created; a 409 means it already exists (idempotent re-run).
35+
Field types: `float64`, `int64`, `uint64`, `utf8`, `bool`. This avoids the older
36+
"write a sentinel row at t=1ns with `__init` tag values to materialize the table"
37+
hack — that pattern leaves a permanent throwaway row in every table that all queries
38+
must filter out.
739

840
## Tables
941

@@ -18,6 +50,10 @@ triggers.
1850

1951
## Caches
2052

53+
Both caches are created in `init.sh` *after* the explicit table creation above —
54+
`create last_cache` and `create distinct_cache` resolve their `--table` argument
55+
against the catalog, so the table must already exist.
56+
2157
- **Last Value Cache** `machine_state_last` on `machine_state` keyed by (site, line_id, station_id, machine_id) — 24-row in-memory cache that powers the plant-state banner and is read by `request_andon_board` to assemble the andon JSON. Single-digit-ms per-machine lookup (exact latency depends on the query).
2258
- **Distinct Value Cache** `part_id_distinct` on `part_events.part_id` — accelerates the "distinct parts today" KPI. Demonstrated explicitly in `cache-distinct` CLI example.
2359

‎plugins/request_andon_board.py‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -43,13 +43,11 @@
4343

4444
def process_request(influxdb3_local, query_parameters, request_headers, request_body, args=None):
4545
# Latest state per machine via the LVC. last_cache() is a table-valued
46-
# function returning one row per cache-key combination (24 machines + 1
47-
# __init seed row that init.sh writes at t=1ns to make the table exist
48-
# before the simulator boots; filter the seed out here).
46+
# function returning one row per cache-key combination — 24 rows for
47+
# the 24 machines.
4948
state_rows = influxdb3_local.query(
5049
"SELECT site, line_id, station_id, machine_id, state, reason "
5150
"FROM last_cache('machine_state', 'machine_state_last') "
52-
"WHERE site <> '__init' "
5351
"ORDER BY line_id, station_id"
5452
)
5553

‎tests/test_queries.py‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,7 @@ def test_plant_state_query_uses_lvc():
2222
# rather than one row per tick.
2323
assert "last_cache('machine_state', 'machine_state_last')" in sql
2424
assert "machine_id" in sql
25-
# __init seed row is filtered out so it doesn't pollute the banner counts.
26-
assert "__init" in sql
25+
assert "site = 'acme-main'" in sql
2726

2827

2928
def test_kpi_units_last_window_query_filters_part_events():

‎ui/queries.py‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,13 +25,11 @@ def plant_state_sql() -> str:
2525
function, which returns one row per cache-key combination — 24 rows
2626
for the 24 machines, in single-digit ms. A plain SELECT against
2727
machine_state would scan the whole table and count every tick.
28-
The site <> '__init' filter excludes the sentinel row init.sh
29-
writes at t=1ns to make the table exist before the simulator boots.
3028
"""
3129
return """
3230
SELECT machine_id, state
3331
FROM last_cache('machine_state', 'machine_state_last')
34-
WHERE site = 'acme-main' AND site <> '__init'
32+
WHERE site = 'acme-main'
3533
ORDER BY machine_id
3634
"""
3735

0 commit comments

Comments
 (0)