Plugin infrastructure

[danielgerlag]

This pull request introduces significant improvements to Windows build support, plugin system documentation, and dependency management for the Drasi Server project. The main highlights are the addition of a Windows MSVC build pipeline with vendored native libraries, enhanced documentation for dynamic plugin loading and verification, and an update to core dependencies to the latest 0.4.x series. These changes improve cross-platform compatibility, clarify plugin integration, and modernize the codebase.

Windows Build and Cross-Compilation Enhancements

• Added a new GitHub Actions job (build-windows-msvc) to build the server for Windows x86_64 MSVC, using vendored static libraries for jq and OpenSSL to ensure reliable native builds. The workflow also uploads the resulting binary as an artifact.

Plugin System and Documentation Improvements

• Expanded the CLAUDE.md documentation to clearly explain runtime plugin loading, the use of cdylib shared libraries, plugin verification (including Sigstore/cosign signature verification), and integration testing for plugins. Also clarified the server’s dynamic loading architecture and plugin lifecycle.
• Added notes on plugin smoke tests, host-sdk integration tests, and the requirements for building and testing plugins.

Dependency and Workspace Updates

• Updated core dependencies (drasi-lib, drasi-core, drasi-plugin-sdk, drasi-host-sdk, etc.) to the 0.4.x series, and removed direct dependencies on individual source/reaction plugins in favor of a leaner set of core plugins. ([[1]](https://github.com/drasi-project/drasi-server/pull/76/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542L35-R38), [[2]](https://github.com/drasi-project/drasi-server/pull/76/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542L45-R68), [[3]](https://github.com/drasi-project/drasi-server/pull/76/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542L123-R117))
• Declared a Cargo workspace including the new xtask utility for managing vendored dependencies and build tasks. ([Cargo.tomlR15-R17](https://github.com/drasi-project/drasi-server/pull/76/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R15-R17))
• Added several new dependencies for build tooling, plugin verification, and improved CLI experience (e.g., console, indicatif, toml, sha2, semver). ([[1]](https://github.com/drasi-project/drasi-server/pull/76/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R83-R84), [[2]](https://github.com/drasi-project/drasi-server/pull/76/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R95-R96), [[3]](https://github.com/drasi-project/drasi-server/pull/76/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R108), [[4]](https://github.com/drasi-project/drasi-server/pull/76/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542L132-R132))

Minor Improvements and Fixes

• Updated Docker and API documentation to reflect new endpoint paths and Swagger UI location. ([[1]](https://github.com/drasi-project/drasi-server/pull/76/files#diff-972512ed94fce217ace10a2303390c321d822f967f2f008c353fb5f57152b7d0L33-R33), [[2]](https://github.com/drasi-project/drasi-server/pull/76/files#diff-972512ed94fce217ace10a2303390c321d822f967f2f008c353fb5f57152b7d0L266-R270))
• Clarified instructions and comments in .cargo/config.toml and Cargo.toml for local development and cross-compilation workflows. ([[1]](https://github.com/drasi-project/drasi-server/pull/76/files#diff-9a4f3e4537ebb7474452d131b0d969d89a51286f4269aac5ef268e712be17268R3-R12), [[2]](https://github.com/drasi-project/drasi-server/pull/76/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542L142-L166))

These changes collectively improve the project's maintainability, developer experience, and readiness for cross-platform deployments.

[github-actions]

YAML Snippet Validation Report

I validated all YAML configuration snippets found in the markdown files against the Rust models in src/config/types.rs, src/api/models/mod.rs, and src/api/models/queries/query.rs.

Overall result: 45 YAML blocks checked — 2 documentation issues found.

---

⚠️ Issues Found

1. README.md — lines 339–341: Wrong field name in trustedIdentities example

# trustedIdentities:  # optional: restrict to specific signers
#   - identity: "release@example.com"    # ❌ wrong field name
#     issuer: "(redacted)"

Problem: The field identity does not exist on TrustedIdentity. The struct is:

// src/config/types.rs:189–195
#[serde(rename_all = "camelCase")]
pub struct TrustedIdentity {
    pub issuer: String,
    pub subject_pattern: String,  // → "subjectPattern" in YAML
}

Suggested fix:

# trustedIdentities:
#   - issuer: "https://token.actions.githubusercontent.com"
#     subjectPattern: "https://github.com/my-org/*"

Note: TrustedIdentity does not have deny_unknown_fields, so identity would be silently ignored at runtime, but subjectPattern would be missing (required String field) — causing a deserialization error if this block were uncommented.

---

2. CLAUDE.md — lines 108–110: Wrong field name in trustedIdentities example

# trustedIdentities:
#   - issuer: "(redacted)"
#     subject: "builder@my-org.iam.gserviceaccount.com"    # ❌ wrong field name

Problem: Same as above — subject is not a valid field. The correct name is subjectPattern.

Suggested fix:

# trustedIdentities:
#   - issuer: "(redacted)"
#     subjectPattern: "builder@my-org.iam.gserviceaccount.com"

---

✅ Everything Else Validated

All other YAML snippets are consistent with the Rust models:

Area	Status
Top-level server config fields (DrasiServerConfig with deny_unknown_fields)	✅ All camelCase fields correct
Instance config fields (DrasiLibInstanceConfig with deny_unknown_fields)	✅ Valid
Query config fields (QueryConfigDto with deny_unknown_fields)	✅ Valid (enableBootstrap, bootstrapBufferSize, autoStart, queryLanguage, etc.)
Source subscription config (SourceSubscriptionConfigDto with deny_unknown_fields)	✅ Valid (sourceId, nodes, relations, pipeline)
Source configs (SourceConfig — flexible deserializer)	✅ Valid (kind, id, autoStart, bootstrapProvider + plugin-specific fields)
Reaction configs (ReactionConfig — flexible deserializer)	✅ Valid (kind, id, queries, autoStart + plugin-specific fields)
Multi-instance configurations	✅ Valid
Environment variable interpolation examples	✅ Valid
State store config	✅ Valid
Plugin dependency config	✅ Valid (ref field maps to PluginDependency.reference)

AI generated by YAML Snippet Validator

[Copilot]

Pull request overview

This PR overhauls drasi-server’s plugin infrastructure by introducing runtime plugin management (install/list/search/upgrade/remove), dynamic plugin loading and verification support, and improved Windows/MSVC build support via vendored native libraries and CI updates.

Changes:

• Added CLI plugin management commands plus supporting lockfile/integrity/signature UX.
• Introduced dynamic plugin loading + plugin registry wiring into API/router/factories and updated config schema to support plugin registry + auto-install + trusted identities.
• Added Windows MSVC build pipeline and vendored native library download tooling (xtask + build.rs + Cargo config), plus broad test/docs/example updates.

Reviewed changes

Copilot reviewed 128 out of 130 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
xtask/src/main.rs	Adds cargo xtask vendor CLI for pushing/pulling vendored libs.
xtask/Cargo.toml	Defines xtask dependencies.
tests/test_support/mod.rs	Adds lint suppressions around test-support modules/reexports.
tests/test_support/mock_components.rs	Updates mock reaction trait impl to satisfy new interface.
tests/test_support/config_helpers.rs	Updates test config helpers to new struct-based Source/Reaction config shape.
tests/openapi_test.rs	Wires PluginRegistry into router builder for OpenAPI tests.
tests/integration/hello-world/config.yaml	Declares plugins + auto-install for integration config.
tests/integration/getting-started/config.yaml	Declares plugins + auto-install for integration config.
tests/error_resilience_test.rs	Adds plugin error-resilience tests (unknown kinds, dynamic loading edge cases, config casing).
tests/enum_serialization_test.rs	Updates serialization tests for struct-based configs w/ JSON config payloads.
tests/dto_camelcase_test.rs	Updates (but currently weakens) camelCase DTO tests.
tests/default_query_language_test.rs	Updates default query language expectations and typed QueryLanguage.
tests/api_query_joins_test.rs	Updates DTO mapping helpers for typed query fields.
tests/api_persistence_test.rs	Updates persistence tests for new config shapes and typed query language.
tests/api_integration_test.rs	Updates router construction for plugin registry; removes dynamic creation tests note.
src/plugin/install.rs	Implements plugin install for OCI/file/http and lockfile updates.
src/plugin/list.rs	Implements plugin list with lockfile metadata + integrity/signature display.
src/plugin/remove.rs	Implements plugin remove and lockfile update attempt.
src/plugin/search.rs	Implements plugin search via registry client.
src/plugin/upgrade.rs	Implements plugin upgrade logic against registry and lockfile.
src/plugin/mod.rs	Adds plugin subcommand wiring + shared registry/auth/trust helpers.
src/dynamic_loading.rs	Adds runtime shared-library plugin loading and descriptor registration.
src/cli_styles.rs	Adds CLI formatting/spinners + signature/integrity labels.
src/main.rs	Adds plugin subcommand, plugin-dir/verification flags, and richer version output.
src/lib.rs	Re-exports plugin/dynamic loading modules and core plugin registration.
src/persistence.rs	Persists new plugin-related config fields into saved configs.
src/builder.rs	Minor builder cleanup.
src/init/mod.rs	Updates init docs to reflect plugin set changes.
src/init/builder.rs	Updates generated configs for typed query language + plugin config defaults.
src/config/types.rs	Adds plugin registry/auto-install/verification/trust config types and schema changes.
src/config/mod.rs	Re-exports new config types/helpers.
src/api/v1/routes.rs	Injects PluginRegistry into axum extensions and router signature.
src/api/v1/mod.rs	Re-exports OpenAPI plugin schema injector.
src/api/v1/handlers.rs	Threads PluginRegistry through v1 handlers into shared handlers.
src/api/shared/handlers.rs	Uses PluginRegistry when creating sources/reactions; stream logic cleanup.
src/api/models/queries/query.rs	Moves query DTO to typed QueryLanguage + plain string query/source_id.
src/api/mappings/queries/mod.rs	Updates query mapping to typed fields and removes string parsing.
src/api/mappings/mod.rs	Removes now-deleted sources/reactions mapping modules from exports.
src/api/joins_tests.rs	Updates internal join tests for typed query DTO fields.
src/api/mod.rs	Re-exports OpenAPI plugin schema injector.
src/api/models/sources/mod.rs	Removes legacy source DTO module tree.
src/api/models/sources/mock.rs	Removes legacy mock source DTOs/tests.
src/api/models/sources/grpc_source.rs	Removes legacy gRPC source DTOs.
src/api/models/sources/postgres.rs	Removes legacy Postgres source DTOs.
src/api/models/sources/platform_source.rs	Removes legacy platform source DTOs.
src/api/models/reactions/mod.rs	Removes legacy reaction DTO module tree.
src/api/models/reactions/log.rs	Removes legacy log reaction DTOs.
src/api/models/reactions/http_reaction.rs	Removes legacy HTTP reaction DTOs.
src/api/models/reactions/grpc_reaction.rs	Removes legacy gRPC reaction DTOs.
src/api/models/reactions/sse.rs	Removes legacy SSE reaction DTOs.
src/api/models/reactions/profiler.rs	Removes legacy profiler reaction DTOs.
src/api/models/reactions/platform_reaction.rs	Removes legacy platform reaction DTOs.
src/api/models/config_value.rs	Removes legacy config-value implementation (now from plugin-sdk).
src/api/mappings/sources/mod.rs	Removes legacy source mappers module tree.
src/api/mappings/sources/mock_mapper.rs	Removes legacy mock mapper.
src/api/mappings/sources/http_mapper.rs	Removes legacy HTTP mapper.
src/api/mappings/sources/grpc_mapper.rs	Removes legacy gRPC mapper.
src/api/mappings/sources/postgres_mapper.rs	Removes legacy Postgres mapper.
src/api/mappings/sources/platform_mapper.rs	Removes legacy platform mapper.
src/api/mappings/reactions/mod.rs	Removes legacy reaction mappers module tree.
src/api/mappings/reactions/log_mapper.rs	Removes legacy log mapper.
src/api/mappings/reactions/http_mapper.rs	Removes legacy HTTP mapper.
src/api/mappings/reactions/http_adaptive_mapper.rs	Removes legacy HTTP-adaptive mapper.
src/api/mappings/reactions/grpc_mapper.rs	Removes legacy gRPC mapper.
src/api/mappings/reactions/grpc_adaptive_mapper.rs	Removes legacy gRPC-adaptive mapper.
src/api/mappings/reactions/sse_mapper.rs	Removes legacy SSE mapper.
src/api/mappings/reactions/profiler_mapper.rs	Removes legacy profiler mapper.
src/api/mappings/reactions/platform_mapper.rs	Removes legacy platform mapper.
examples/trading/server/trading-sources-only.yaml	Adds plugin auto-install declarations.
examples/playground/server/playground.yaml	Adds plugin auto-install declarations.
examples/getting-started/server-config.yaml	Adds plugin auto-install declarations.
examples/getting-started/configs/getting-started-step-3.yaml	Adds plugin auto-install declarations.
examples/getting-started/configs/getting-started-step-4.yaml	Adds plugin auto-install declarations.
examples/getting-started/configs/getting-started-step-5.yaml	Adds plugin auto-install declarations.
examples/getting-started/configs/getting-started-step-6.yaml	Adds plugin auto-install declarations.
examples/configs/**.yaml	Broadly adds plugin auto-install declarations across examples.
examples/basic_setup.rs	Updates example to typed query language + new config fields.
examples/README.md	Documents plugin signature verification CLI flag.
dev-tools/vscode/drasi-server/src/schema-provider.ts	Improves kind resolution using OpenAPI discriminators.
dev-tools/vscode/drasi-server/src/component-yaml-generator.ts	Fixes schema-name-to-kind conversion for dotted schema names.
dev-tools/vscode/drasi-server/package.json	Bumps extension version.
build.rs	Adds vendored libs download + build-time version env emission.
Makefile	Adds cross-build targets + smoke tests; updates help and lint/test defaults.
DOCKER.md	Updates API docs URL paths for v1 endpoints/docs.
Cargo.toml	Adds workspace/xtask, updates drasi dependencies to 0.4.x, adds host/sdk + build deps.
CLAUDE.md	Expands docs on dynamic plugin loading, verification, and testing workflows.
.gitignore	Ignores downloaded vendor/ directory.
.github/workflows/release.yaml	Adds Windows MSVC build job + artifact/test matrix updates.
.cargo/config.toml	Adds cargo xtask alias + MSVC vendored link overrides.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ruokun-niu]

Approved. Just left some minor comments

[github-actions]

YAML Snippet Validation Report

✅ All YAML snippets validated successfully!

Scope

Validated YAML configuration snippets in the following markdown files:

• README.md (server settings, plugins, sources, queries, reactions, multi-instance examples)
• CLAUDE.md (single-instance and multi-instance config examples)
• examples/configs/README.md (schema reference examples)
• examples/getting-started/README.md (PostgreSQL source, log/SSE reaction snippets)
• DOCKER.md (environment variable interpolation fragment)

Validation Method

Cross-referenced each YAML snippet against the Rust models:

• DrasiServerConfig — #[serde(rename_all = "camelCase", deny_unknown_fields)]
• DrasiLibInstanceConfig — #[serde(rename_all = "camelCase", deny_unknown_fields)]
• QueryConfigDto / SourceSubscriptionConfigDto — #[serde(rename_all = "camelCase", deny_unknown_fields)]
• SourceConfig / ReactionConfig — custom deserializers (require kind, id, queries)
• StateStoreConfig — custom deserializer (kind: redb + path)
• TrustedIdentity — #[serde(rename_all = "camelCase")]
• PluginDependency — #[serde(rename = "ref")]

Findings

Check	Result
camelCase field naming	✅ All fields correctly use camelCase (logLevel, persistConfig, persistIndex, autoStart, sourceId, queryLanguage, enableBootstrap, bootstrapBufferSize, pluginRegistry, verifyPlugins, subjectPattern, etc.)
Required fields present	✅ All required fields (kind, id for sources; kind, id, queries for reactions) are present in complete examples
No unknown fields	✅ No fields that would be rejected by deny_unknown_fields
Type correctness	✅ Integers, booleans, strings, arrays used correctly
stateStore schema	✅ All examples use kind: redb with path field
plugins schema	✅ All entries use ref: key as expected by #[serde(rename = "ref")]
Incomplete fragments	✅ Correctly identified and intentionally excluded from runtime validation

Existing Test Coverage

The repository already has automated validation tests (tests/readme_examples_validation_test.rs and tests/example_configs_validation_test.rs) that parse YAML blocks from README.md and validate all 29 example config files on every cargo test run, providing ongoing protection against schema drift.

AI generated by YAML Snippet Validator