Generate Complete Game Installation Files Registry CSV (EPIC)

[undead2146]

Description

EPIC ISSUE: As part of EPIC #108, create a comprehensive, authoritative CSV registry containing all game installation files for Command & Conquer Generals 1.08 and Zero Hour 1.04. This epic oversees the creation of the CSV registry system and is broken down into focused subtasks.

Subtasks

This epic is divided into the following focused issues:

• Create CSV Generation Utility #154: Create utility to scan installations and generate CSV
• Implement Registry Storage Structure #155: Set up repository file structure and versioning
• Create Index.json Metadata System for CSV registries. #156: Implement metadata system for dynamic discovery
• Update Documentation for CSV Registry #157: Update docs with registry procedures

Cross-Cutting Sub-Issues (EPIC #108)

• Add Language to ContentSearchQuery #150: Add Language to ContentSearchQuery
• Add CsvCatalogEntry model #151: Add CsvCatalogEntry model
• Implement LanguageDetector utility #152: Add LanguageDetector utility

High-Level Requirements

• Generate complete CSV with all game files from official installations
• Include separate columns for GameType and Language detection
• Store CSV in repository at /docs/GameInstallationFilesRegistry/
• Enable CSV resolver to reference via GitHub usercontent API
• Ensure deterministic ContentManifest generation with proper manifest IDs
• Follow constants.md patterns for configuration and paths
• Support multi-CSV approach for scalability

Generation Process

Important Clarifications:

• Developer Responsibility: CSVs are generated by developers using the GenHub.Tools.CsvGenerator utility, not by end users
• Tool Location: The generator utility is a standalone console app located at GenHub.Tools/CsvGenerator/
• Output Location: Generated CSVs are committed to the repository under docs/GameInstallationFilesRegistry/
• Path Requirements: All file paths in the CSV must be relative to the game installation root (never absolute paths)

Example CSV Row:

Data/INI/GameData.ini,12345,af34bc5678901234567890abcdef1234567890abcdef,29adef1234567890abcdef1234567890abcdef1234567890abcdef,Generals,en,true,"{""category"":""config""}"

Epic Acceptance Criteria

• FOUNDATIONAL: All dependent issues (Implement CSVDiscoverer #139-Update documentation for CSV validation feature #147) can now access CSV registry
• VERIFICATION: CSVDiscoverer can successfully load and parse registry index
• VERIFICATION: CSVResolver can download and process CSV data
• VERIFICATION: Pipeline components can function with generated registry
• Complete CSV generated with all Generals 1.08 and Zero Hour 1.04 files
• CSV includes GameType and Language columns for easy filtering
• CSV stored in repository with proper versioning
• CSV accessible via GitHub user content API
• ContentManifest generation produces consistent, deterministic IDs
• Constants.md patterns used for all configuration and paths
• Documentation updated with CSV generation and usage procedures

Dependencies

• Blocks: Issues Implement CSVDiscoverer #139-Update documentation for CSV validation feature #147 (CSV pipeline components)
• Blocked by: None (foundational work)
• Related: EPIC EPIC: Validate GameInstallation using remote CSV catalog (Generals 1.08 / ZH 1.04) with optional direct-CSV mode #108 CSV Content Provider

Implementation Strategy

Recommended Approach: Multi-CSV by Game Type

This balances performance and maintainability:

• Separate CSVs for Generals and Zero Hour
• Index.json for metadata and dynamic discovery
• GitHub raw content API for access

CSV Schema

relativePath,size,md5,sha256,gameType,language,isRequired,metadata
Data/INI/GameData.ini,12345,abcdef1234567890abcdef1234567890,sha256hash1234567890abcdef1234567890abcdef1234567890abcdef,Generals,All,true,"{""category"":""config""}"
Data/English/game.str,67890,ghijkl7890123456ghijkl7890123456,sha256hashfedcba0987654321fedcba0987654321fedcba0987654321,ZeroHour,EN,true,"{""category"":""language""}"

Important Notes:

• All paths must be relative to game installation root
• MD5 and SHA256 are both required (SHA256 for validation, MD5 for legacy compatibility)
• gameType must be exactly "Generals" or "ZeroHour"
• language values in CSV are uppercase codes: All, EN, DE, FR, ES, IT, KO, PL, PT-BR, ZH-CN, ZH-TW (input queries may be any case; system normalizes)
• isRequired is boolean (true/false)
• metadata is optional JSON string for additional file categorization

Performance Considerations

• Streaming IO: Be mindful of streaming writes for large CSVs (~80k entries). Avoid loading everything into memory at once.
• Hash Calculation: Use streaming hash computation to handle large files efficiently
• Progress Reporting: Implement progress indicators for long-running operations
• Error Handling: Gracefully handle inaccessible files/directories during scanning

File Structure

docs/
└── GameInstallationFilesRegistry/
    ├── Generals-1.08.csv
    ├── ZeroHour-1.04.csv
    └── index.json

Epic Completion

This epic is complete when all subtasks are finished and the CSV registry is fully functional with the pipeline components.