feat: Add comprehensive federated learning, LLM inference, and IPFS storage infrastructure - #16
Merged
Merged
Conversation
…reum references to Filecoin in .env.sample and related CLI files - Adjust server settings and API endpoints accordingly
…d learning session management commands in CLI - Create a new FederatedLearningClient for session operations - Add storage management commands for IPFS/Filecoin integration - Update configuration structure for Filecoin settings
…figurations - Add new environment variables for IPFS storage settings and federated learning parameters
…rated Learning - Reorganize environment variables and clarify settings for Filecoin network and federated learning parameters
…PI docs - Standardized all FL command structures with consistent patterns - Added GetTrainedModel client method for model retrieval - Enhanced configuration with proper FederatedLearning section - Fixed URL construction in federated learning client - Added comprehensive API reference documentation - Implemented consistent help text and examples across all FL commands - Added proper flag validation and error handling
…tandardization feat: add federated learning
…lm, and storage commands for consistency
- Implement complete FL session lifecycle management (create, start, get, list) - Add federated learning commands with extensive configuration options - Support for dataset upload with IPFS/Filecoin integration - Add model update submission with gradients file support - Implement trained model retrieval and export functionality - Add differential privacy configuration options - Support for custom model configuration files - Add comprehensive parameter validation and error handling - Include detailed CLI examples and usage documentation - Integrate with wallet authentication system for FL operations
…ated learning - Implement 5 partitioning strategies: random, stratified, sequential, non-IID (Dirichlet), and label skew - Add PartitionConfig struct with configurable parameters (alpha, min_samples, overlap_ratio) - Update TrainingDataInfo to include metadata for partition configuration - Enhance CLI commands with partitioning flags (--alpha, --min-samples, --overlap-ratio) - Add LoadPartitionedData method to support distributed data loading - Create comprehensive test script for all partitioning strategies - Add detailed documentation for data partitioning functionality This enables truly distributed federated learning where each participant trains on different subsets of data, simulating realistic FL scenarios with data heterogeneity and privacy-preserving data separation.
- Document the root cause of empty weights issue - Explain the technical solution across all three repositories - Provide code examples showing before/after states - Include testing instructions and benefits achieved - Add technical notes for future reference
…ity and required parameters - Revise README to include detailed federated learning features and usage instructions - Add model configuration examples and clarify required parameters for session creation - Update CLI commands to enforce required flags for aggregation method, learning rate, batch size, and local epochs - Improve error handling for missing model configuration - Enhance documentation for differential privacy and data partitioning strategies
- Revise .env.sample to include new server and blockchain identity configurations - Update CLI commands to replace PRTY with USDFC for token balance and stake operations - Add new configuration options for logging, caching, and security settings - Introduce GetReputationCommand in command structure
feat: add data partitioning for distributed federated learning
- Removed comments regarding the importance of the client and private key requirements - Streamlined the configuration options for better clarity
- Add DisableTimestamp support to gologger - Remove outdated documentation and test files - Clean up repository structure - Enhance logging configuration options
Gologger Improvements and Repository Cleanup
- Point gologger to commit 768e1ca (latest main) - Ensure consistency across all repositories
- Update gologger submodule to commit 2e729e3 - Add environment variable support for disabling timestamps - Enable runtime control of timestamp display
- Update gologger submodule to commit 3a60352 - Fix LOG_DISABLE_TIMESTAMP being ignored when using InitWithMode - Ensure environment variables are respected across all logging modes
- Update gologger submodule to commit 032e8ba - Fix stack overflow caused by recursive DefaultConfig/ConfigForMode calls - Ensure LOG_DISABLE_TIMESTAMP works properly without crashes
- Modify target URL in proxy handler to include '/api/' prefix - Update upload URL in task handler to use '/api/v1/tasks' path
…o Blockchain - Rename FilecoinNetwork to BlockchainNetwork in config and related files - Update environment variables in .env.sample and README.md - Refactor wallet adapter initialization to use BlockchainNetwork settings - Modify storage service references to reflect the change from Filecoin to Blockchain - Clean up related comments and documentation for clarity
…files - Update response body and file closure handling to log errors on close - Remove unnecessary linter configurations in .golangci.yml - Simplify lint command in Makefile - Ensure proper error handling for port listener closure in network.go - Enhance error handling for JSON body reading in http.go
- Upgrade golangci-lint-action from v3 to v4 - Specify golangci-lint version as v2.2.2 for consistency
- Remove unnecessary output format argument from golangci-lint action configuration
- Disable all linters by default and selectively enable necessary ones in .golangci.yml - Update local-prefixes for goimports to reflect new repository structure - Downgrade golangci-lint-action from v4 to v3 and update version to v1.54.2 in workflow - Adjust golangci-lint arguments for improved output formatting and timeout settings
- Add govet, ineffassign, and unused linters to .golangci.yml for improved code quality checks
…rs and adjust local-prefixes - Remove govet, ineffassign, and unused linters from .golangci.yml - Update local-prefixes for goimports to reflect new repository structure
- Add new linter for handling internal import errors in .golangci-ci.yml - Upgrade Go toolchain version to 1.24.5 in go.mod - Update Go version to 1.23 and golangci-lint version to latest in code quality workflow
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR implements a complete federated learning system, LLM inference capabilities, IPFS storage integration, and blockchain token management for the Parity Runner. The system now supports distributed machine learning, local LLM processing, and generic blockchain integration with configurable ERC20 tokens.
Major Features
Federated Learning System
LLM Inference Capabilities
IPFS Storage Integration
Blockchain Integration
Tunnel Support
Technical Implementation
Task Execution