Special mode for users from China: uses HF mirrir site if model network error

Author: meizhong986

docs/release_notes_v1.8.7b1.md

@@ -6,7 +6,28 @@ Pre-release for testing. Feedback welcome via [Issues](https://github.com/meizho
 
 ## What's New in beta.1
 
-- **#204 — SSL/VPN resilience for HuggingFace downloads** — Users behind corporate proxies or Chinese VPN services (e.g., v2rayN) no longer get blocked by SSL certificate validation errors when models are already cached locally. A single startup-time monkeypatch transparently retries failed HuggingFace Hub downloads from local cache. Covers all download paths: faster-whisper models, NeMo VAD, speech enhancement models, and ensemble subprocess workers.
+### Network Resilience for Users in China (#204)
+
+WhisperJAV now handles unstable network conditions gracefully — especially for users in China who connect through VPN/proxy services like v2rayN, Clash, or corporate proxies.
+
+**The problem:** HuggingFace Hub (where AI models are hosted) always contacts its servers to check for updates before loading a model — even when the model is already fully downloaded and cached on your machine. In China, VPN proxy tools often break SSL certificate validation, causing these checks to fail with `CERTIFICATE_VERIFY_FAILED` errors. The result: WhisperJAV refuses to start even though all required models are already on disk.
+
+**The fix:** WhisperJAV now automatically detects SSL/network failures and uses a 3-step fallback strategy. No configuration needed — it works transparently at startup.
+
+**What you'll see (3-step fallback):**
+
+1. **Step 1 — Normal download from huggingface.co:** Works as before. If this succeeds, nothing changes.
+2. **Step 2 — Local cache fallback:** If Step 1 fails with a network/SSL error, WhisperJAV checks your local model cache. If the model was downloaded before, it loads from cache and continues normally. You'll see a warning in the log, but processing is not interrupted.
+3. **Step 3 — China mirror (hf-mirror.com):** If the model is not in your local cache, WhisperJAV automatically tries downloading from `hf-mirror.com`, the official HuggingFace mirror for China. This works even when your VPN blocks `huggingface.co`. Once downloaded, the model is cached locally for future use.
+
+If all 3 steps fail, WhisperJAV shows a diagnostic summary with:
+- The exact model name and download URLs (both huggingface.co and hf-mirror.com)
+- Your local cache directory path, so you can download and place files manually
+- The `HF_ENDPOINT` environment variable you can set as a permanent workaround
+
+**Coverage:** This protection applies to all AI model downloads — Whisper models (faster-whisper), VAD models (NeMo/Silero), speech enhancement models (ZipEnhancer, BS-RoFormer, ClearVoice), and ensemble mode subprocess workers. Every entry point (CLI, GUI, ensemble workers) is protected.
+
+**For users in China:** In most cases, WhisperJAV will now "just work" — even on first run — because the China mirror fallback handles the initial download automatically. No need to disconnect your VPN.
 
 ## Carried from beta.0
 
@@ -35,7 +56,7 @@ Pre-release for testing. Feedback welcome via [Issues](https://github.com/meizho
 ## Installation
 
 ```bash
-pip install "whisperjav @ git+https://github.com/meizhong986/whisperjav.git@v1.8.7-beta.1"
+pip install "whisperjav @ git+https://github.com/meizhong986/whisperjav.git@v1.8.7b1"
 ```
 
 Or for Mac users:

tests/test_model_loader.py

@@ -235,8 +235,41 @@ def mock_download(*args, **kwargs):
             with pytest.raises(ValueError, match="Invalid repo_id"):
                 patched("bad/repo/format")
 
-    def test_ssl_fail_no_cache_raises_original_error(self):
-        """When SSL fails and cache miss, the ORIGINAL SSL error is raised."""
+    def test_ssl_fail_no_cache_tries_mirror(self):
+        """When SSL fails and cache miss, should try hf-mirror.com."""
+        import huggingface_hub
+        import whisperjav.utils.model_loader as ml
+
+        call_log = []
+
+        def mock_download(*args, **kwargs):
+            call_log.append(kwargs.copy())
+            if kwargs.get("local_files_only"):
+                raise OSError("model not found in cache")
+            if kwargs.get("endpoint") == ml._HF_MIRROR_ENDPOINT:
+                return "/mirror/model/path"
+            raise OSError(
+                "urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] cert verify failed"
+            )
+
+        ml._patched = False
+        with patch.object(huggingface_hub, "snapshot_download", mock_download):
+            self._apply_patch()
+            patched = huggingface_hub.snapshot_download
+
+            result = patched("Systran/faster-whisper-large-v2")
+            assert result == "/mirror/model/path"
+            assert len(call_log) == 3
+            # Step 1: normal attempt
+            assert not call_log[0].get("local_files_only")
+            assert not call_log[0].get("endpoint")
+            # Step 2: local cache
+            assert call_log[1]["local_files_only"] is True
+            # Step 3: mirror
+            assert call_log[2]["endpoint"] == ml._HF_MIRROR_ENDPOINT
+
+    def test_ssl_fail_no_cache_no_mirror_raises_original_error(self):
+        """When all 3 steps fail, the ORIGINAL SSL error is raised."""
         import huggingface_hub
         import whisperjav.utils.model_loader as ml
 
@@ -247,6 +280,8 @@ def test_ssl_fail_no_cache_raises_original_error(self):
         def mock_download(*args, **kwargs):
             if kwargs.get("local_files_only"):
                 raise OSError("model not found in cache")
+            if kwargs.get("endpoint"):
+                raise ConnectionError("mirror also unreachable")
             raise original_error
 
         ml._patched = False
@@ -322,6 +357,31 @@ def mock_download(*args, **kwargs):
             assert len(call_log) == 2
             assert call_log[1]["local_files_only"] is True
 
+    def test_hf_hub_download_mirror_fallback(self):
+        """SSL error + cache miss should try hf-mirror.com for hf_hub_download."""
+        import huggingface_hub
+        import whisperjav.utils.model_loader as ml
+
+        call_log = []
+
+        def mock_download(*args, **kwargs):
+            call_log.append(kwargs.copy())
+            if kwargs.get("local_files_only"):
+                raise OSError("not in cache")
+            if kwargs.get("endpoint") == ml._HF_MIRROR_ENDPOINT:
+                return "/mirror/file/path"
+            raise ssl.SSLError("record layer failure")
+
+        ml._patched = False
+        with patch.object(huggingface_hub, "hf_hub_download", mock_download):
+            self._apply_patch()
+            patched = huggingface_hub.hf_hub_download
+
+            result = patched("org/repo", filename="model.bin")
+            assert result == "/mirror/file/path"
+            assert len(call_log) == 3
+            assert call_log[2]["endpoint"] == ml._HF_MIRROR_ENDPOINT
+
     def test_hf_hub_download_non_network_error_raises(self):
         """Non-network errors should not trigger fallback for hf_hub_download."""
         import huggingface_hub

whisperjav/utils/model_loader.py

@@ -1,12 +1,16 @@
 """Resilient HuggingFace Hub downloads with network fallback.
 
-Monkeypatches huggingface_hub.snapshot_download() to gracefully handle
-network errors (SSL failures, timeouts, proxy issues) by falling back
-to locally cached models when available.
+Monkeypatches huggingface_hub.snapshot_download() and hf_hub_download()
+to gracefully handle network errors (SSL failures, timeouts, proxy issues)
+with a 3-step fallback strategy:
+
+    1. Normal download from huggingface.co
+    2. Load from local cache (local_files_only=True)
+    3. Download from hf-mirror.com (official China mirror)
 
 This is critical for users behind corporate proxies or Chinese VPN
-services (e.g., v2rayN) where SSL certificate validation often fails
-even though the model cache is already complete.
+services (e.g., v2rayN, Clash) where SSL certificate validation often
+fails even though the model cache is already complete.
 
 Architecture: A single monkeypatch applied once at startup protects ALL
 code paths that download from HuggingFace — faster-whisper, stable-ts,
@@ -26,6 +30,9 @@
 
 _patched = False
 
+# Official HuggingFace mirror for China — maintained by HuggingFace
+_HF_MIRROR_ENDPOINT = "https://hf-mirror.com"
+
 # Error strings that indicate a network/SSL problem (not a model/CUDA problem)
 _NETWORK_ERROR_INDICATORS = [
     "ssl",
@@ -75,10 +82,23 @@ def _is_network_error(error: Exception) -> bool:
     return False
 
 
+def _get_cache_dir():
+    """Get the HuggingFace Hub cache directory path for diagnostics."""
+    try:
+        from huggingface_hub import constants
+        return constants.HF_HUB_CACHE
+    except Exception:
+        return "(unknown)"
+
+
 def _make_resilient_wrapper(original_fn, fn_name):
     """Create a resilient wrapper for a HuggingFace Hub download function.
 
-    On SSL/network errors, retries with local_files_only=True.
+    3-step fallback on SSL/network errors:
+      1. Try normal download from huggingface.co
+      2. Try loading from local cache (local_files_only=True)
+      3. Try downloading from hf-mirror.com (official China mirror)
+
     Works for both snapshot_download and hf_hub_download.
     """
 
@@ -95,35 +115,97 @@ def _resilient_wrapper(*args, **kwargs):
 
             # Extract identifier for messaging (first positional arg or kwarg)
             resource_id = args[0] if args else kwargs.get("repo_id", "unknown")
+            hf_url = f"https://huggingface.co/{resource_id}"
+            cache_dir = _get_cache_dir()
 
             logger.warning(
-                "Network/SSL error while checking '%s' on HuggingFace: %s",
-                resource_id, e,
+                "[HF Download] Step 1 FAILED — network/SSL error downloading "
+                "'%s' from %s",
+                resource_id, hf_url,
             )
             logger.warning(
-                "Attempting to load from local cache..."
+                "[HF Download] Error: %s: %s",
+                type(e).__name__, e,
             )
 
+            # --- Step 2: Try local cache ---
+            logger.info(
+                "[HF Download] Step 2 — checking local cache at: %s",
+                cache_dir,
+            )
             try:
                 result = original_fn(
                     *args, **{**kwargs, "local_files_only": True}
                 )
                 logger.info(
-                    "Loaded '%s' from local cache. "
+                    "[HF Download] Step 2 OK — loaded '%s' from local cache. "
                     "Network issues did not affect model loading.",
                     resource_id,
                 )
                 return result
             except Exception:
-                logger.error(
-                    "Model '%s' not found in local cache. "
-                    "A working internet connection is required for first-time "
-                    "model download. Please check your VPN/proxy settings, "
-                    "or try disconnecting your VPN and downloading the model "
-                    "once with a direct connection.",
+                logger.warning(
+                    "[HF Download] Step 2 FAILED — '%s' not found in local "
+                    "cache.",
                     resource_id,
                 )
-                raise e  # Re-raise original network error
+
+            # --- Step 3: Try China mirror ---
+            mirror_url = f"{_HF_MIRROR_ENDPOINT}/{resource_id}"
+            logger.info(
+                "[HF Download] Step 3 — trying China mirror: %s",
+                mirror_url,
+            )
+            try:
+                result = original_fn(
+                    *args,
+                    **{**kwargs, "endpoint": _HF_MIRROR_ENDPOINT},
+                )
+                logger.info(
+                    "[HF Download] Step 3 OK — downloaded '%s' from mirror "
+                    "(%s). Model is now cached locally for future use.",
+                    resource_id, _HF_MIRROR_ENDPOINT,
+                )
+                return result
+            except Exception as mirror_err:
+                logger.error(
+                    "[HF Download] Step 3 FAILED — mirror download also "
+                    "failed: %s: %s",
+                    type(mirror_err).__name__, mirror_err,
+                )
+
+            # --- All steps failed: comprehensive diagnostics ---
+            logger.error(
+                "[HF Download] All download methods failed for '%s'. "
+                "Diagnostic summary:",
+                resource_id,
+            )
+            logger.error(
+                "  Model:      %s", resource_id,
+            )
+            logger.error(
+                "  Source URL:  %s", hf_url,
+            )
+            logger.error(
+                "  Mirror URL: %s", mirror_url,
+            )
+            logger.error(
+                "  Cache dir:  %s", cache_dir,
+            )
+            logger.error(
+                "  Error type: %s", type(e).__name__,
+            )
+            logger.error(
+                "  To download manually, visit: %s", mirror_url,
+            )
+            logger.error(
+                "  Then place the downloaded files in: %s", cache_dir,
+            )
+            logger.error(
+                "  Or set environment variable: "
+                "HF_ENDPOINT=https://hf-mirror.com"
+            )
+            raise e  # Re-raise original network error
 
     _resilient_wrapper.__name__ = f"_resilient_{fn_name}"
     _resilient_wrapper.__qualname__ = f"_resilient_{fn_name}"
@@ -137,12 +219,14 @@ def patch_hf_hub_downloads():
     and hf_hub_download (used for individual file downloads like NeMo, LLM
     GGUF files, classifiers).
 
-    On SSL/connection/timeout errors, automatically retries with
-    local_files_only=True to use cached files. Provides clear user
-    messages for each scenario:
+    On SSL/connection/timeout errors, uses a 3-step fallback with full
+    diagnostic logging at each step:
 
-    1. SSL fail + cache hit  -> warning + continues normally
-    2. SSL fail + no cache   -> error with actionable guidance
+    1. Normal download fails     -> log error type and source URL
+    2. Try local cache           -> success: continue; fail: try mirror
+    3. Try hf-mirror.com (China) -> success: model cached for future use
+    4. All failed                -> comprehensive diagnostic summary with
+                                    manual download instructions
     """
     global _patched
     if _patched: