feat: Implement data-driven provider definition configurations with external JSON

[undead2146]

Merge after:

• feat(downloads): Add publisher card UI for content browsing #199
• refactor(content-pipeline): Add genpatcher content and implement a dependency builder #203

Summary

This PR introduces a comprehensive data-driven provider configuration system that externalizes content source settings into JSON files. This enables runtime configuration of endpoints, timeouts, catalog parsing, and provider behavior without code changes.

Key Changes

🏗️ Architecture

New Core Infrastructure

• IProviderDefinitionLoader - Service interface for loading/managing provider definitions
• ProviderDefinitionLoader - Implementation with auto-loading, caching, and hot-reload support
• ICatalogParser - Interface for pluggable catalog format parsers
• ICatalogParserFactory - Factory for obtaining parsers by format identifier
• IContentPipelineFactory - Factory for obtaining pipeline components by provider ID

Provider Definition Files (JSON)

• communityoutpost.provider.json - Community Outpost (GenPatcher) configuration
• generalsonline.provider.json - Generals Online CDN configuration
• thesuperhackers.provider.json - TheSuperHackers GitHub configuration

📁 New Files

File	Purpose
GenHub.Core/Interfaces/Providers/ICatalogParser.cs	Catalog parser interface
GenHub.Core/Interfaces/Providers/ICatalogParserFactory.cs	Parser factory interface
GenHub.Core/Services/Providers/CatalogParserFactory.cs	Parser factory implementation
GenHub/Features/Content/Services/CommunityOutpost/GenPatcherDatCatalogParser.cs	GenPatcher dl.dat catalog parser
GenHub/Features/Content/Services/GeneralsOnline/GeneralsOnlineJsonCatalogParser.cs	Generals Online JSON API parser
GenHub/Features/Content/Services/ContentPipelineFactory.cs	Pipeline component factory
GenHub/Providers/*.provider.json	Provider configuration files
docs/features/content/provider-configuration.md	Comprehensive documentation
docs/features/content/provider-infrastructure.md	Architecture documentation

🔧 Modified Files

Content Providers

All content providers now:

1. Inject IProviderDefinitionLoader
2. Override GetProviderDefinition() to provide cached configuration
3. Pass provider definition to discoverers and resolvers

Provider	Changes
CommunityOutpostProvider	Added provider definition loading and caching
CommunityOutpostDiscoverer	Uses provider endpoints instead of constants
CommunityOutpostResolver	Uses provider endpoints for manifest URLs
GeneralsOnlineProvider	Added provider definition support
GeneralsOnlineDiscoverer	Uses catalog parser factory
GeneralsOnlineManifestFactory	Uses provider endpoints for manifest URLs
SuperHackersProvider	Added provider definition support
SuperHackersUpdateService	Uses provider config for GitHub repo info

Base Infrastructure

• BaseContentProvider - Added GetProviderDefinition() virtual method
• IContentDiscoverer - Added provider-aware DiscoverAsync() overload
• IContentResolver - Added provider-aware ResolveAsync() overload

📝 Documentation Updates

• docs/architecture.md - Added section on data-driven provider configuration
• docs/features/content/index.md - Added link to provider configuration docs

---

Provider Definition Schema

{
  "providerId": "community-outpost",
  "publisherType": "communityoutpost",
  "displayName": "Community Outpost",
  "description": "Official patches, tools, and addons from GenPatcher (Community Outpost)",
  "iconColor": "#2196F3",
  "providerType": "Static",
  "catalogFormat": "genpatcher-dat",
  "endpoints": {
    "catalogUrl": "https://legi.cc/gp2/dl.dat",
    "websiteUrl": "https://legi.cc",
    "supportUrl": "https://legi.cc/patch",
    "custom": {
      "patchPageUrl": "https://legi.cc/patch",
      "gentoolWebsite": "https://gentool.net"
    }
  },
  "mirrorPreference": ["legi.cc", "gentool.net"],
  "targetGame": "ZeroHour",
  "defaultTags": ["community", "genpatcher"],
  "timeouts": {
    "catalogTimeoutSeconds": 30,
    "contentTimeoutSeconds": 300
  },
  "enabled": true
}

---

Usage Example

Provider Implementation

public class MyContentProvider : BaseContentProvider
{
    private readonly IProviderDefinitionLoader _loader;
    private ProviderDefinition? _cachedDefinition;

    protected override ProviderDefinition? GetProviderDefinition()
    {
        _cachedDefinition ??= _loader.GetProvider("my-provider-id");
        return _cachedDefinition;
    }
}

Discoverer Implementation

public async Task<OperationResult<IEnumerable<ContentSearchResult>>> DiscoverAsync(
    ProviderDefinition? provider,
    ContentSearchQuery query,
    CancellationToken cancellationToken)
{
    // Use provider-defined endpoints with fallback to constants
    var catalogUrl = provider?.Endpoints.CatalogUrl ?? MyConstants.CatalogUrl;
    var timeout = provider?.Timeouts.CatalogTimeoutSeconds ?? 30;
    
    // Use custom endpoints
    var customEndpoint = provider?.Endpoints.GetEndpoint("customKey") ?? "default";
    
    // ... discovery logic
}

Deployment Notes

Provider JSON files are automatically copied to the output directory via MSBuild:

<ItemGroup>
  <Content Include="Providers\*.json">
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
  </Content>
</ItemGroup>

---

[greptile-apps]

Greptile Overview

Greptile Summary

This PR successfully implements a data-driven provider configuration system that externalizes content source settings to JSON files, enabling runtime configuration without code changes.

Key Achievements

Architecture & Infrastructure

- **Provider Definition System**: Introduced `ProviderDefinition` model with comprehensive JSON serialization support for endpoints, timeouts, catalog formats, and mirror preferences - **Auto-Loading Mechanism**: `ProviderDefinitionLoader` loads from both bundled and user directories with thread-safe caching, synchronous fallback, and hot-reload support - **Pluggable Parsers**: `ICatalogParser` and `ICatalogParserFactory` enable format-specific catalog parsing (GenPatcher DAT, JSON API, GitHub releases) - **Pipeline Factory**: `ContentPipelineFactory` matches provider IDs to discoverers, resolvers, and deliverers with case-insensitive lookup

Provider Implementations

- **Community Outpost**: Full GenPatcher dl.dat catalog support with multi-mirror downloads, version extraction, and metadata integration via `GenPatcherContentRegistry` - **Generals Online**: Updated to use provider definition for JSON API catalog and CDN endpoints - **TheSuperHackers**: Configured for GitHub releases with repository metadata from JSON

Dependency & Version Management

- **BaseDependencyBuilder**: Abstract base for dependency creation with factory methods for common game dependencies (ZH 1.04, Generals 1.08) - **VersionConstraint**: Comprehensive semantic versioning with range support, comparison operators, caret/tilde ranges

Code Quality

• Clean separation between configuration (JSON) and implementation (C#)
• Provider-aware overloads maintain backward compatibility with fallback to constants
• Comprehensive unit tests for core infrastructure (116 test files total)
• Extensive documentation in PR description and inline comments
• MSBuild integration for automatic provider JSON deployment

Integration Points

All providers successfully integrate the new system:

• Inject IProviderDefinitionLoader via constructor
• Override GetProviderDefinition() with caching
• Pass provider definitions to discoverers and resolvers
• Use provider.Endpoints for URLs with fallback to constants

Confidence Score: 4/5

• This PR is safe to merge with minor observations - the architecture is solid, implementation is comprehensive, and testing is thorough
• Score reflects excellent architecture and implementation quality, with proper error handling, thread safety, and backward compatibility. One minor concern: synchronous file I/O in EnsureProvidersLoaded() could block on slow disks, though this is mitigated by caching and is only hit on first access
• GenHub.Core/Services/Providers/ProviderDefinitionLoader.cs - contains synchronous file I/O in LoadAllProvidersSynchronous() that could block on first access from sync methods

Important Files Changed

File Analysis

Filename	Score	Overview
GenHub/GenHub/Providers/communityoutpost.provider.json	5/5	Provider configuration for Community Outpost with catalog URL, endpoints, timeouts, and mirror preferences - clean JSON structure
GenHub/GenHub/Providers/generalsonline.provider.json	5/5	Provider configuration for Generals Online with JSON API catalog format and CDN endpoints - properly structured
GenHub/GenHub/Providers/thesuperhackers.provider.json	5/5	Provider configuration for TheSuperHackers with GitHub releases catalog format and repository metadata - well-formed
GenHub/GenHub.Core/Models/Providers/ProviderDefinition.cs	5/5	Comprehensive provider definition model with JSON serialization, endpoint management, mirror configuration, and helper methods - robust design
GenHub/GenHub.Core/Services/Providers/ProviderDefinitionLoader.cs	4/5	Loads provider definitions from bundled and user directories with caching, auto-loading, synchronous fallback, and error handling - thread-safe implementation
GenHub/GenHub/Features/Content/Services/CommunityOutpost/CommunityOutpostDiscoverer.cs	5/5	Discovers content using provider definition endpoints, catalog parser factory, and GenPatcher registry - clean separation of concerns
GenHub/GenHub/Features/Content/Services/CommunityOutpost/GenPatcherDatCatalogParser.cs	5/5	Parses GenPatcher dl.dat catalog format with mirror support, version extraction, and metadata integration - robust regex-based parsing
GenHub/GenHub/Features/Content/Services/CommunityOutpost/CommunityOutpostResolver.cs	4/5	Resolves content to manifests using provider endpoints, registry metadata, and dependency builder - comprehensive implementation with proper error handling
GenHub/GenHub.Core/Services/Dependencies/BaseDependencyBuilder.cs	5/5	Base class for dependency building with factory methods for common game dependencies - clean abstraction with well-documented patterns
GenHub/GenHub.Core/Models/Manifest/VersionConstraint.cs	5/5	Version constraint model with semantic version parsing, comparison operators, caret/tilde ranges - comprehensive version matching logic

Sequence Diagram

sequenceDiagram
    participant App as Application
    participant Loader as ProviderDefinitionLoader
    participant Provider as CommunityOutpostProvider
    participant Discoverer as CommunityOutpostDiscoverer
    participant ParserFactory as CatalogParserFactory
    participant Parser as GenPatcherDatCatalogParser
    participant Registry as GenPatcherContentRegistry
    participant Resolver as CommunityOutpostResolver
    participant Builder as ManifestBuilder
    
    Note over App,Loader: Startup: Load Provider Definitions
    App->>Loader: LoadProvidersAsync()
    Loader->>Loader: Load bundled/*.provider.json
    Loader->>Loader: Load user/*.provider.json
    Loader-->>App: ProviderDefinition[]
    
    Note over App,Provider: Content Discovery Flow
    App->>Provider: DiscoverAsync(query)
    Provider->>Provider: GetProviderDefinition()
    Provider->>Loader: GetProvider("community-outpost")
    Loader-->>Provider: ProviderDefinition (cached)
    Provider->>Discoverer: DiscoverAsync(provider, query)
    
    Note over Discoverer,Parser: Catalog Parsing
    Discoverer->>Discoverer: Get catalogUrl from provider.Endpoints
    Discoverer->>ParserFactory: GetParser(provider.CatalogFormat)
    ParserFactory-->>Discoverer: GenPatcherDatCatalogParser
    Discoverer->>Parser: ParseAsync(catalogContent, provider)
    
    loop For each catalog entry
        Parser->>Registry: GetMetadata(contentCode)
        Registry-->>Parser: ContentMetadata
        Parser->>Parser: ConvertToContentSearchResult()
        Parser->>Parser: GetPreferredDownloadUrl(provider.MirrorPreference)
    end
    
    Parser-->>Discoverer: ContentSearchResult[]
    Discoverer-->>Provider: ContentSearchResult[]
    Provider-->>App: ContentSearchResult[]
    
    Note over App,Resolver: Content Resolution Flow
    App->>Provider: ResolveAsync(searchResult)
    Provider->>Resolver: ResolveAsync(provider, searchResult)
    Resolver->>Loader: GetProvider("community-outpost")
    Loader-->>Resolver: ProviderDefinition
    Resolver->>Registry: GetMetadata(contentCode)
    Registry-->>Resolver: ContentMetadata
    
    Note over Resolver,Builder: Manifest Building
    Resolver->>Builder: WithBasicInfo(publisher, name, version)
    Resolver->>Builder: WithContentType(type, game)
    Resolver->>Builder: WithPublisher(provider.Endpoints)
    Resolver->>Builder: AddDependency(from metadata)
    Resolver->>Builder: AddRemoteFileAsync(url from provider)
    Resolver->>Builder: Build()
    Builder-->>Resolver: ContentManifest
    Resolver-->>Provider: ContentManifest
    Provider-->>App: ContentManifest

Loading

[greptile-apps]

_{86 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[ViTeXFTW]

Are the serialization intended to be a form of approved publishers baked into GenHub? Because if it is more general then it would make more sense that these are shipped with the application but that there also is a place where users can add their own publisher json files. This would then be a way for new publishers to link back to their respective husting site for downloadable content?

[undead2146]

Are the serialization intended to be a form of approved publishers baked into GenHub? Because if it is more general then it would make more sense that these are shipped with the application but that there also is a place where users can add their own publisher json files. This would then be a way for new publishers to link back to their respective husting site for downloadable content?

Regarding provider definitions, they are all stored under the following directories. With user providers with matching providerId overriding bundled providers.

Location	Path	Purpose
Bundled	{AppDir}/Providers/*.provider.json	Official providers shipped with the app
User	{AppData}/GenHub/Providers/*.provider.json	User-customized or additional providers

An addition to provider definitions is that we could implement a *.catalog.json that the provider definition Provider.Endpoints.CatalogUrl points to. This way providers will need to generate and provide users with a definition and a catalog json.

[ViTeXFTW]

And what about the custom section in the provider template? What are these used for exactly? I can't really be custom fields if we do not look / parse that specific field name.

I am not sure but it seems like we are not using it currently - but still providing some custom information in the .json files.

[undead2146]

Okay so that took a while, a couple force pushes to get the branch correct again.
I addressed all the comments.