Cleaning all over the codebase to remove orphans variables and old documentation.

Author: yohplala

AGENTS.md

@@ -13,7 +13,6 @@
 | **Tutorial & CLI usage** | [docs/TUTORIAL.md](docs/TUTORIAL.md) |
 | **CryptoCompare API details** | [docs/DATA_SOURCES.md](docs/DATA_SOURCES.md) |
 | **TOTAL2/TOTAL2b calculation** | [docs/TOTAL2_CALCULATION.md](docs/TOTAL2_CALCULATION.md) |
-| **Edge cases & solutions** | [docs/EDGE_CASES.md](docs/EDGE_CASES.md) |
 | **Deployment (GitHub Pages)** | [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) |
 
 ---

README.md

@@ -45,7 +45,7 @@ poetry run python -m main status
 
 ### 📋 References
 - **[AI Agent Context](AGENTS.md)** - Full project specification for AI agents and developers
-- **[Data Sources](docs/DATA_SOURCES.md)** - CryptoCompare API details, rate limits, caching
+- **[Data Sources](docs/DATA_SOURCES.md)** - CryptoCompare API details, rate limits, caching, data pipeline
 - **[TOTAL2 Calculation](docs/TOTAL2_CALCULATION.md)** - How the TOTAL2 market index is calculated
 - **[Deployment](docs/DEPLOYMENT.md)** - Charts generation and GitHub Pages deployment workflow
 - **[Changelog](CHANGELOG.md)** - Version history and release notes

docs/DATA_SOURCES.md

@@ -15,7 +15,7 @@ Halvix uses **CryptoCompare** as its single data source:
 | **Top coins by market cap** | `/data/top/mktcapfull` endpoint |
 | **Historical prices** | `/data/v2/histoday` endpoint |
 | **Volume data** | Included in historical data for TOTAL2 weighting |
-| **Rate limit** | 10 calls/second (free tier) |
+| **Rate limit** | Free tier: 10/sec; Halvix uses conservative 2/sec (120/min) |
 | **Historical depth** | **Unlimited** - full history available |
 
 This single-source approach provides:
@@ -41,15 +41,17 @@ This single-source approach provides:
 
 | Tier | Rate Limit | Notes |
 |------|------------|-------|
-| **Free** | 10 calls/second | No API key required |
+| **Free** | 10 calls/second (600/min) | No API key required |
+| **Halvix default** | 2 calls/second (120/min) | Conservative to avoid issues |
 | Professional | 50 calls/second | Paid |
 | Enterprise | Custom | Contact sales |
 
 ### Halvix Configuration
 
 ```python
 # src/config.py
-CRYPTOCOMPARE_API_CALLS_PER_MINUTE = 30  # Conservative (could go to 600)
+# Free tier allows 10/sec (600/min), we use conservative 2/sec (120/min)
+CRYPTOCOMPARE_API_CALLS_PER_MINUTE = 120
 CRYPTOCOMPARE_MAX_DAYS_PER_REQUEST = 2000  # Max days per request
 ```
 
@@ -272,13 +274,17 @@ All API settings are in `src/config.py`:
 # CryptoCompare
 CRYPTOCOMPARE_BASE_URL = "https://min-api.cryptocompare.com"
 CRYPTOCOMPARE_COIN_URL = "https://www.cryptocompare.com/coins"
-CRYPTOCOMPARE_API_CALLS_PER_MINUTE = 30  # Very conservative
+# Free tier allows 10/sec (600/min), we use conservative 2/sec (120/min)
+CRYPTOCOMPARE_API_CALLS_PER_MINUTE = 120
 CRYPTOCOMPARE_MAX_DAYS_PER_REQUEST = 2000  # Days per request
 
 # Retry configuration
 API_MAX_RETRIES = 5
 API_RETRY_MIN_WAIT = 1         # seconds
 API_RETRY_MAX_WAIT = 60        # seconds
+
+# Data completeness - this is a fixed constant ensuring only complete daily data is fetched
+USE_YESTERDAY_AS_END_DATE = True
 ```
 
 ---

docs/EDGE_CASES.md

@@ -19,7 +19,6 @@ python = "^3.13"
 pandas = "^2.1"
 numpy = "^2.0"
 plotly = "^5.18"
-scipy = "^1.14"
 requests = "^2.31"
 tqdm = "^4.65"
 pyarrow = "^18.0"

pyproject.toml

@@ -71,12 +71,6 @@ def __init__(self):
             re.compile(pattern, re.IGNORECASE) for pattern in EXCLUDED_PATTERNS
         ]
 
-    # Property for backwards compatibility
-    @property
-    def filtered_tokens(self) -> list[SkippedCoin]:
-        """Backwards compatibility alias for skipped_coins."""
-        return self.skipped_coins
-
     def reset(self):
         """Clear the skipped coins list."""
         self.skipped_coins = []
@@ -279,16 +273,6 @@ def should_skip_download(
 
         return (False, "")
 
-    # Backwards compatibility alias
-    def should_exclude_from_download(
-        self,
-        coin_id: str,
-        name: str = "",
-        symbol: str = "",
-    ) -> tuple[bool, str]:
-        """Backwards compatibility alias for should_skip_download."""
-        return self.should_skip_download(coin_id, name, symbol)
-
     def should_exclude_from_total2(
         self,
         coin_id: str,
@@ -377,15 +361,6 @@ def get_coins_to_download(
 
         return to_download
 
-    # Backwards compatibility alias
-    def filter_coins_for_download(
-        self,
-        coins: list[dict],
-        record_filtered: bool = True,
-    ) -> list[dict]:
-        """Backwards compatibility alias for get_coins_to_download."""
-        return self.get_coins_to_download(coins, record_skipped=record_filtered)
-
     def filter_coins_for_total2(
         self,
         coins: list[dict],
@@ -440,11 +415,6 @@ def export_skipped_coins_csv(self, filepath: Path | None = None) -> Path:
 
         return filepath
 
-    # Backwards compatibility alias
-    def export_rejected_coins_csv(self, filepath: Path | None = None) -> Path:
-        """Backwards compatibility alias for export_skipped_coins_csv."""
-        return self.export_skipped_coins_csv(filepath)
-
     def get_skipped_summary(self) -> dict:
         """
         Get a summary of skipped coins by reason.
@@ -457,8 +427,3 @@ def get_skipped_summary(self) -> dict:
             reason = coin.reason
             summary[reason] = summary.get(reason, 0) + 1
         return summary
-
-    # Backwards compatibility alias
-    def get_filtered_summary(self) -> dict:
-        """Backwards compatibility alias for get_skipped_summary."""
-        return self.get_skipped_summary()