Modify URL validation method and error handling (#159)

feat: optimize URL validation to use HEAD requests by default

Update validate_url_200 to default get_method to requests.head for improved efficiency.

Implement fallback to GET for specific status codes (403, 405, 501) to handle servers that reject HEAD requests.

It is worth noting that this change does not resolve 429 Too Many Requests errors. Because HEAD requests are still requests, they count toward your total traffic quota on the target server. By design, this function honors the server's rate-limiting policy, it does not bypass it. If you encounter a 429 error, the function will correctly raise a ValidationError as expected, signaling that the application needs to implement backoff or throttling logic rather than attempting to "fix" the request method.

Author: mushroomsandchai

courses/validators/custom_url_validators.py

@@ -25,14 +25,21 @@ def get_error_message(status_code, url):
 def validate_url_200(
     url, get_method=None, code=None, params=None
 ):
+    # Since the 
     if get_method is None:
-        get_method = requests.get
+        get_method = requests.head
 
     try:
         response = get_method(url)
+
+        if response.status_code in [403, 405, 501]:
+            response = requests.get(url)
+
         status_code = response.status_code
+            
         if status_code == 200:
             return
+            
         error_message = get_error_message(status_code, url)
         raise ValidationError(error_message, code=code, params=params)
     except requests.exceptions.RequestException as e: