Skip to content

Commit 833441f

Browse files
johankristianssjohankristianss
authored
Improved reconil (#168)
New Features Cron-based Reconciliation System Implemented automated reconciliation via server-side cron jobs (60-second interval) Crons automatically created when blueprints with handlers are added LastReconciliationProcess now updated when reconciliation process is created, not when it completes Ensures RECONCILING field accurately reflects active reconciliation state Modified pkg/server/controllers/cron_controller.go:185-224 Blueprint Reconciliation Metadata Tracking Blueprint metadata now updated at processgraph creation time with: LastReconciliationProcess - Process ID of current/latest reconciliation LastReconciliationTime - Timestamp of last reconciliation start Enables accurate reconciliation status tracking in colonies blueprint ls Modified pkg/server/controllers/cron_controller.go and pkg/database/postgresql/blueprint_table.go Enhanced RECONCILING Status Detection RECONCILING field now checks both WAITING (state 0) and RUNNING (state 1) states Previously only checked RUNNING state, causing premature "no" status Modified internal/cli/blueprint_table.go:270-282 Bug Fixes Fixed race condition in metadata updates Problem: LastReconciliationProcess was updated in TWO places: a. Cron controller (when created) → sets to NEW process ID b. Process handler (when completes) → overwrites with COMPLETING process ID c. When processes completed out of order, old IDs overwrote newer ones Solution: Removed duplicate update from process completion handler Impact: RECONCILING field now works correctly even with rapid blueprint set operations Modified pkg/server/handlers/process/handlers.go (removed lines 847-868) Fixed duplicate cron name constraint violation Problem: Cron names only used blueprint name, causing conflicts across colonies Solution: Include namespace in cron name: reconcile-{namespace}-{blueprint-name} Impact: Blueprints with same name can exist in different colonies without conflicts Modified pkg/server/handlers/blueprint/handlers.go:431 Fixed executor name inconsistency Corrected all references from local-docker-node-reconciler to local-node-docker-reconciler Aligns deployment blueprints with docker-compose.yml configuration Files: deployment/blueprints/*.json, deployment/blueprints/README.md Improvements Better error handling GetBlueprintDefinitions returns empty array instead of nil when no definitions exist Consistent with other handlers (crons, executors) Improved test coverage Coverage: 63.5% → 68.2% (+4.7%) Added 7 tests covering error paths and edge cases Fixed TestCrossColonyBlueprintIsolation failure Technical Details Cron-based vs Event-driven Reconciliation Current implementation: Cron-based (stateless, pull model) Reconciler has zero state, fetches latest blueprint from server every 60s WaitForPrevProcessGraph ensures sequential reconciliation (no concurrent updates) Event-driven reconciliation with action metadata (create/update/delete) not yet implemented
1 parent 768e724 commit 833441f

36 files changed

Lines changed: 1303 additions & 295 deletions

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ all: build
22
.PHONY: all build
33

44
BUILD_IMAGE ?= colonyos/colonies
5-
PUSH_IMAGE ?= colonyos/colonies:v1.9.1
5+
PUSH_IMAGE ?= colonyos/colonies:v1.9.3
66

77
VERSION := $(shell git rev-parse --short HEAD)
88
BUILDTIME := $(shell date -u '+%Y-%m-%dT%H:%M:%SZ')

deployment/catalog/README.md renamed to deployment/blueprints/README.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Blueprint Catalog
1+
# Blueprints
22

33
This directory contains reusable blueprint specifications for common deployments.
44

@@ -13,20 +13,20 @@ export COLONIES_PRVKEY=${COLONIES_COLONY_PRVKEY}
1313
colonies blueprint definition add --spec executor-deployment-definition.json
1414
```
1515

16-
### docker-executor-deployment.json
16+
### local-docker-executor-deployment.json
1717
Deploys a docker executor specifically on the **local/main node**.
1818

1919
**Key Settings:**
2020
- `executorType`: `docker-reconciler` - Requires a docker-reconciler
21-
- `executorName`: `local-docker-node-reconciler` - Targets the main node
21+
- `executorName`: `local-node-docker-reconciler` - Targets the main node
2222
- `replicas`: 1 - Single executor instance
2323

2424
**Deploy:**
2525
```bash
26-
colonies blueprint add --spec docker-executor-deployment.json
26+
colonies blueprint add --spec local-docker-executor-deployment.json
2727
```
2828

29-
**Result:** The deployment will run specifically on the `local-docker-node-reconciler` (main node from colonies docker-compose).
29+
**Result:** The deployment will run specifically on the `local-node-docker-reconciler` (main node from colonies docker-compose).
3030

3131
## Executor Targeting Examples
3232

@@ -39,7 +39,7 @@ colonies blueprint add --spec docker-executor-deployment.json
3939
},
4040
"spec": {
4141
"executorType": "docker-reconciler",
42-
"executorName": "local-docker-node-reconciler" // Main node
42+
"executorName": "local-node-docker-reconciler" // Main node
4343
}
4444
}
4545
```
@@ -72,15 +72,15 @@ colonies blueprint add --spec docker-executor-deployment.json
7272
},
7373
"spec": {
7474
"executorType": "docker-reconciler",
75-
"executorName": "local-docker-node-reconciler" // Specific node
75+
"executorName": "local-node-docker-reconciler" // Specific node
7676
}
7777
}
7878
```
7979
✅ Guaranteed deployment on specific node
8080
⚠️ Fails if that reconciler is down
8181

8282
**Available reconcilers in default setup:**
83-
- `local-docker-node-reconciler` - Main node (in colonies docker-compose)
83+
- `local-node-docker-reconciler` - Main node (in colonies docker-compose)
8484
- `docker-reconciler-edge` - Edge node (in docker-reconciler docker-compose)
8585

8686
### Example 3: Target Edge Node
@@ -111,7 +111,7 @@ colonies blueprint definition add --spec executor-deployment-definition.json
111111
### 2. Deploy Executor
112112
```bash
113113
# Deploy to any available node
114-
colonies blueprint add --spec docker-executor-deployment.json
114+
colonies blueprint add --spec local-docker-executor-deployment.json
115115

116116
# Check status
117117
colonies blueprint get --name docker-executor

‎deployment/blueprints/docker-deployment-definition.json‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
{
2+
"metadata": {
3+
"name": "docker-deployment"
4+
},
5+
"spec": {
6+
"group": "colonyos.io",
7+
"version": "v1",
8+
"names": {
9+
"kind": "DockerDeployment",
10+
"plural": "dockerdeployments",
11+
"singular": "dockerdeployment"
12+
},
13+
"scope": "Namespaced",
14+
"schema": {
15+
"type": "object",
16+
"required": ["instances"],
17+
"properties": {
18+
"replicas": {
19+
"type": "integer",
20+
"description": "Number of replicas to deploy (optional, defaults to number of instances specified)",
21+
"minimum": 0,
22+
"default": 1
23+
},
24+
"executorName": {
25+
"type": "string",
26+
"description": "Optional: Target specific reconciler node (e.g., 'edge-node-docker-reconciler'). If omitted, any reconciler with matching executorType can handle this deployment."
27+
},
28+
"executorNames": {
29+
"type": "array",
30+
"description": "Optional: Target multiple specific reconciler nodes. First available node gets the deployment. Alternative to 'executorName' for multi-node targeting.",
31+
"items": {
32+
"type": "string"
33+
}
34+
},
35+
"instances": {
36+
"type": "array",
37+
"description": "List of container instances to deploy",
38+
"items": {
39+
"type": "object",
40+
"required": ["name", "type", "image"],
41+
"properties": {
42+
"name": {
43+
"type": "string",
44+
"description": "Name of the container instance"
45+
},
46+
"type": {
47+
"type": "string",
48+
"enum": ["container"],
49+
"description": "Instance type (currently only 'container' is supported)"
50+
},
51+
"image": {
52+
"type": "string",
53+
"description": "Docker image to use (e.g., 'nginx:latest')"
54+
},
55+
"environment": {
56+
"type": "object",
57+
"description": "Environment variables for the container",
58+
"additionalProperties": {
59+
"type": "string"
60+
}
61+
},
62+
"ports": {
63+
"type": "array",
64+
"description": "Port mappings",
65+
"items": {
66+
"type": "object",
67+
"required": ["container", "protocol"],
68+
"properties": {
69+
"host": {
70+
"type": "integer",
71+
"description": "Host port (optional, will auto-assign if not specified)"
72+
},
73+
"container": {
74+
"type": "integer",
75+
"description": "Container port"
76+
},
77+
"protocol": {
78+
"type": "string",
79+
"enum": ["tcp", "udp"],
80+
"description": "Protocol (tcp or udp)"
81+
}
82+
}
83+
}
84+
},
85+
"volumes": {
86+
"type": "array",
87+
"description": "Volume mounts",
88+
"items": {
89+
"type": "object",
90+
"required": ["type", "mountPath"],
91+
"properties": {
92+
"type": {
93+
"type": "string",
94+
"enum": ["named", "bind"],
95+
"description": "Volume type: 'named' for Docker volumes, 'bind' for host paths"
96+
},
97+
"name": {
98+
"type": "string",
99+
"description": "Volume name (for named volumes) or host path (for bind mounts)"
100+
},
101+
"mountPath": {
102+
"type": "string",
103+
"description": "Container mount path"
104+
},
105+
"readOnly": {
106+
"type": "boolean",
107+
"description": "Mount as read-only",
108+
"default": false
109+
}
110+
}
111+
}
112+
},
113+
"command": {
114+
"type": "array",
115+
"description": "Override container command",
116+
"items": {
117+
"type": "string"
118+
}
119+
},
120+
"args": {
121+
"type": "array",
122+
"description": "Override container arguments",
123+
"items": {
124+
"type": "string"
125+
}
126+
},
127+
"privileged": {
128+
"type": "boolean",
129+
"description": "Run container in privileged mode",
130+
"default": false
131+
},
132+
"networkMode": {
133+
"type": "string",
134+
"description": "Docker network mode (e.g., 'bridge', 'host', 'none')"
135+
}
136+
}
137+
}
138+
}
139+
}
140+
},
141+
"handler": {
142+
"executorType": "docker-reconciler",
143+
"functionName": "reconcile"
144+
}
145+
}
146+
}

deployment/catalog/executor-deployment-definition.json renamed to deployment/blueprints/executor-deployment-definition.json

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"metadata": {
3-
"name": "executordeployments.compute.colonies.io"
3+
"name": "executor-deployment"
44
},
55
"spec": {
66
"group": "compute.colonies.io",
@@ -27,6 +27,17 @@
2727
"type": "string",
2828
"description": "Type of executor to deploy"
2929
},
30+
"executorName": {
31+
"type": "string",
32+
"description": "Optional: Target specific reconciler node (e.g., 'edge-node-docker-reconciler'). If omitted, any reconciler with matching executorType can handle this deployment."
33+
},
34+
"executorNames": {
35+
"type": "array",
36+
"description": "Optional: Target multiple specific reconciler nodes. First available node gets the deployment. Alternative to 'executorName' for multi-node targeting.",
37+
"items": {
38+
"type": "string"
39+
}
40+
},
3041
"cpu": {
3142
"type": "string",
3243
"description": "CPU resource request (e.g., '500m', '1')"

‎deployment/blueprints/local-docker-executor-deployment.json‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{
2+
"kind": "ExecutorDeployment",
3+
"metadata": {
4+
"name": "docker-executor"
5+
},
6+
"handler": {
7+
"executorName": "local-node-docker-reconciler"
8+
},
9+
"spec": {
10+
"image": "colonyos/dockerexecutor:v1.0.7",
11+
"replicas": 1,
12+
"executorType": "container-executor",
13+
"volumes": [
14+
{
15+
"host": "/var/run/docker.sock",
16+
"container": "/var/run/docker.sock"
17+
},
18+
{
19+
"host": "/tmp/colonies",
20+
"container": "/tmp/colonies"
21+
}
22+
],
23+
"privileged": true
24+
}
25+
}

‎deployment/catalog/docker-executor-deployment.json‎

Lines changed: 0 additions & 66 deletions
This file was deleted.

0 commit comments

Comments
 (0)