Refactor server architecture with multi-backend support and LibP2P in… (#154)

  Refactor server architecture with multi-backend support, LibP2P integration, and resource reconciliation

  Major architectural changes:
  - Backend abstraction layer supporting HTTP, gRPC, CoAP, and LibP2P transports
  - Client refactoring with automatic backend fallback and selection
  - Server handler registry pattern for better modularity
  - LibP2P P2P networking with relay, DHT, and NAT traversal
  - Kubernetes-style resource reconciliation with drift detection
  - Subscription improvements with defensive nil checking

  Resource reconciliation system:
  - Resource and ResourceDefinition types for declarative infrastructure
  - Automatic drift detection and correction mechanism
  - JSON Schema-based resource validation
  - Namespaced resource management with metadata (labels, annotations)
  - Resource status tracking with generation counters
  - Reconciliation actions: create, update, delete, noop
  - Field-level change tracking for spec, status, and metadata
  - Handler specification for executor-based reconciliation

  Backend improvements:
  - Unified backend interface for HTTP (Gin), gRPC, CoAP, and LibP2P
  - Multi-backend client with automatic failover
  - DHT-based peer discovery for LibP2P
  - Circuit relay v2 for NAT traversal
  - Server manager for coordinating multiple backend servers

  Client enhancements:
  - Modular client architecture with per-domain clients
  - Automatic backend selection and fallback
  - LibP2P client with DHT caching
  - Support for HTTP, gRPC, CoAP, and LibP2P transports

  Server refactoring:
  - Handler registry pattern for better organization
  - Moved handlers to domain-specific packages
  - Resource handler for managing ResourceDefinitions and Resources
  - Improved controller architecture
  - Better test isolation and mocking

  Database layer:
  - ResourceDatabase interface with PostgreSQL implementation
  - Resource and ResourceDefinition CRUD operations
  - Namespace and kind-based queries
  - JSON/JSONB column support for flexible schemas
  - Database migration support for resource tables

  LibP2P/P2P features:
  - P2P relay server with CLI integration (colonies p2p relay)
  - Identity generation (colonies p2p generate)
  - DHT peer discovery with caching
  - AutoNAT and hole punching support
  - Bootstrap peer management

  Resource management features:
  - Declarative resource specifications with desired state
  - Custom Resource Definition (CRD) support
  - Schema validation with required fields, types, and enums
  - Nested object and array validation
  - Helper methods for GetAPIVersion, Diff, ValidateAgainstRD
  - Resource arrays and JSON conversion utilities

  Bug fixes:
  - Fixed nil pointer panics in subscription cleanup (Gin and LibP2P)
  - Fixed DHT cache to only store active peers
  - Fixed database test isolation with unique table prefixes
  - Fixed TestSearchLogs timing boundary issue with PostgreSQL NOW()
  - Removed APIVersion field from Resource struct (moved to ResourceDefinition)
  - Updated hardcoded peer IDs to current server identity

  Testing improvements:
  - Fixed controller test database conflicts
  - Added DatabaseMock resource interface methods
  - Improved test utilities and mocking
  - Better test isolation with unique database prefixes
  - Added comprehensive backend tests
  - Added resource validation test coverage
  - Added reconciliation and diff testing

  Core domain improvements:
  - Resource type with Kind, Metadata, Spec, Status fields
  - ResourceDefinition with Group, Version, Names, Scope, Handler
  - ValidationSchema for JSON Schema validation
  - Reconciliation type for tracking old/new state with diffs
  - ResourceDiff with field-level change tracking
  - FunctionSpec integration for reconciliation workflows

  CLI enhancements:
  - New 'colonies p2p' subcommand for relay and identity management
  - Improved server startup with backend selection
  - Better configuration management

  Documentation:
  - Added Container Building Guide with multi-platform instructions
  - Docker buildx setup and QEMU configuration
  - Multi-architecture build examples (amd64, arm64)
  - CI/CD integration examples for GitHub Actions and GitLab
  - Troubleshooting guide for buildx issues

Author: johankristianss

.github/workflows/go.yml

@@ -47,7 +47,7 @@ jobs:
       run: make 
 
     - name: Test
-      run: TZ=Europe/Stockholm make github_test
+      run: TZ=Europe/Stockholm make test
       env:
         AWS_S3_ENDPOINT: localhost:9000
         AWS_S3_ACCESSKEY: minioadmin

.gitignore

@@ -8,7 +8,7 @@ coverage.txt
 *.sw*
 pkg/database/postgresql/*.sw*
 pkg/database/*.sw*
-pkg/server/*.sw*
+pkg/service/*.sw*
 pkg/core/*.sw*
 pkg/security/*.sw*
 pkg/logging/*.sw*

CLAUDE.md

@@ -19,9 +19,9 @@ make github_test       # Run tests without grc (for CI)
 
 ### Development Environment
 ```bash
-source devenv          # Set up development environment
-colonies dev           # Start development server
-make startdb          # Start PostgreSQL TimescaleDB container
+docker-compose up -d   # Start Colonies server with dependencies (TimescaleDB, MinIO)
+docker-compose down    # Stop all services
+docker-compose logs -f # View logs
 ```
 
 ### Coverage
@@ -34,13 +34,13 @@ ColonyOS is a distributed meta-orchestrator framework that creates compute conti
 
 ### Core Components
 - **Colony**: A distributed runtime environment consisting of networked Executors
-- **Process**: A computational workload defined by a FunctionSpec, with states (WAITING, RUNNING, SUCCESS, FAILED)  
-- **Executor**: Distributed microservices that execute processes, can run anywhere on the Internet
+- **Process**: A computational workload defined by a FunctionSpec, with states (WAITING, RUNNING, SUCCESS, FAILED)
+- **Executor**: Distributed workers that pull and execute processes, can run anywhere on the Internet
 - **FunctionSpec**: Specification defining what computation to run and execution conditions
 
 ### Key Packages
 - `pkg/core/`: Core domain models (Process, Executor, Colony, FunctionSpec)
-- `pkg/server/`: HTTP RPC server implementation 
+- `pkg/service/`: HTTP RPC service implementation 
 - `pkg/client/`: Go SDK for Colonies API
 - `pkg/database/postgresql/`: PostgreSQL database layer with TimescaleDB support
 - `pkg/security/`: Zero-trust security protocol implementation

Dockerfile

@@ -1,6 +1,33 @@
-FROM alpine
+# Build stage
+FROM golang:1.24-alpine AS builder
+
+WORKDIR /build
+
+# Copy go mod files
+COPY go.mod go.sum ./
+RUN go mod download
+
+# Copy source code
+COPY . .
+
+# Get build variables
+ARG VERSION
+ARG BUILDTIME
+ENV VERSION=${VERSION}
+ENV BUILDTIME=${BUILDTIME}
+
+# Build the binary for the target architecture
+RUN CGO_ENABLED=0 go build \
+    -ldflags="-s -w -X 'main.BuildVersion=${VERSION}' -X 'main.BuildTime=${BUILDTIME}'" \
+    -o /colonies \
+    ./cmd/main.go
+
+# Runtime stage
+FROM alpine:latest
 
 WORKDIR /
-COPY ./bin/colonies /bin
+
+# Copy the binary from builder
+COPY --from=builder /colonies /bin/colonies
 
 CMD ["colonies", "server", "start"]

Makefile

@@ -2,7 +2,7 @@ all: build
 .PHONY: all build
 
 BUILD_IMAGE ?= colonyos/colonies
-PUSH_IMAGE ?= colonyos/colonies:v1.8.19
+PUSH_IMAGE ?= colonyos/colonies:v1.9.0
 
 VERSION := $(shell git rev-parse --short HEAD)
 BUILDTIME := $(shell date -u '+%Y-%m-%dT%H:%M:%SZ')
@@ -17,10 +17,19 @@ build:
 	@GOOS=js GOARCH=wasm go build -o ./lib/libcryptolib.wasm internal/cryptolib.wasm/cryptolib.go
 
 container:
-	docker build -t $(BUILD_IMAGE) .
+	@echo "Building container for local architecture..."
+	docker build --build-arg VERSION=$(VERSION) --build-arg BUILDTIME=$(BUILDTIME) -t $(BUILD_IMAGE) .
+
+container-multiplatform:
+	@echo "Building multiplatform container (amd64, arm64)..."
+	docker buildx build --platform linux/amd64,linux/arm64 --build-arg VERSION=$(VERSION) --build-arg BUILDTIME=$(BUILDTIME) -t $(BUILD_IMAGE) .
+
+container-multiplatform-push:
+	@echo "Building and pushing multiplatform container (amd64, arm64)..."
+	docker buildx build --platform linux/amd64,linux/arm64 --build-arg VERSION=$(VERSION) --build-arg BUILDTIME=$(BUILDTIME) -t $(BUILD_IMAGE) -t $(PUSH_IMAGE) --push .
 
 push:
-	docker tag $(BUILD_IMAGE) $(PUSH_IMAGE) 
+	docker tag $(BUILD_IMAGE) $(PUSH_IMAGE)
 	docker push $(BUILD_IMAGE)
 	docker push $(PUSH_IMAGE)
 
@@ -32,23 +41,6 @@ build_cryptolib_ubuntu_2020:
 	cd buildtools; ./build_cryptolib_ubuntu.sh 
 
 test:
-	@cd tests/reliability; grc go test -v --race
-	@cd internal/crypto; grc go test -v --race
-	@cd pkg/core; grc go test -v --race
-	@cd pkg/database/postgresql; grc go test -v --race
-	@cd pkg/rpc; grc go test -v --race
-	@cd pkg/security; grc go test -v --race
-	@cd pkg/security/crypto; grc go test -v --race
-	@cd pkg/security/validator; grc go test -v --race
-	@cd pkg/server; grc go test -v --race
-	@cd pkg/scheduler; grc go test -v --race
-	@cd pkg/parsers; grc go test -v --race
-	@cd pkg/utils; grc go test -v --race
-	@cd pkg/cluster; grc go test -v --race
-	@cd pkg/cron; grc go test -v --race
-	@cd pkg/fs; go test -v --race
-
-github_test: 
 	@cd tests/reliability; go test -v --race
 	@cd internal/crypto; go test -v --race
 	@cd pkg/core; go test -v --race
@@ -57,7 +49,30 @@ github_test:
 	@cd pkg/security; go test -v --race
 	@cd pkg/security/crypto; go test -v --race
 	@cd pkg/security/validator; go test -v --race
+	@cd pkg/backends/gin; go test -v --race
+	@cd pkg/backends/grpc; go test -v --race
+	@cd pkg/backends/libp2p; go test -v --race
+	@cd pkg/client/gin; go test -v --race
+	@cd pkg/client/grpc; go test -v --race
+	@cd pkg/client/libp2p; go test -v --race
 	@cd pkg/server; go test -v --race
+	@cd pkg/server/controllers; go test -v --race
+	@cd pkg/server/handlers/attribute; go test -v --race
+	@cd pkg/server/handlers/colony; go test -v --race
+	@cd pkg/server/handlers/cron; go test -v --race
+	@cd pkg/server/handlers/executor; go test -v --race
+	@cd pkg/server/handlers/file; go test -v --race
+	@cd pkg/server/handlers/function; go test -v --race
+	@cd pkg/server/handlers/generator; go test -v --race
+	@cd pkg/server/handlers/log; go test -v --race
+	@cd pkg/server/handlers/process; go test -v --race
+	@cd pkg/server/handlers/processgraph; go test -v --race
+	@cd pkg/server/handlers/security; go test -v --race
+	@cd pkg/server/handlers/server; go test -v --race
+	@cd pkg/server/handlers/snapshot; go test -v --race
+	@cd pkg/server/handlers/user; go test -v --race
+	@cd pkg/server/handlers/realtime; go test -v --race
+	@cd pkg/server/utils; go test -v --race
 	@cd pkg/scheduler; go test -v --race
 	@cd pkg/parsers; go test -v --race
 	@cd pkg/utils; go test -v --race
@@ -72,3 +87,10 @@ install:
 
 startdb: 
 	docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=rFcLGNkgsNtksg6Pgtn9CumL4xXBQ7 --restart unless-stopped timescale/timescaledb:latest-pg16
+
+nukedb:
+	@echo "Nuking TimescaleDB containers and volumes..."
+	@docker stop $$(docker ps -aq --filter ancestor=timescale/timescaledb:latest-pg16) 2>/dev/null || true
+	@docker rm $$(docker ps -aq --filter ancestor=timescale/timescaledb:latest-pg16) 2>/dev/null || true
+	@docker volume rm $$(docker volume ls -q --filter dangling=true) 2>/dev/null || true
+	@echo "TimescaleDB containers and volumes destroyed"