Merge pull request #19 from AKSW/askretries

Askretries

Author: seebi

CHANGELOG.md

@@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](https://semver.org/)
 
+## [2.1.0] 2026-02-24
+
+### Added
+
+-ask command
+  - new options to control amount of `--retries` and the amount fo `--retry-sleep` between retries
+  - all logs from retries and eventual skips are logged into the `--retries-log`
+- tests
+  - added tests for the new retry options and logging
+
+### Changed
+
+- serve command
+  - default waiting time for answer requests is now 3 seconds instead of 0 to allow for timeout logging tests to work properly
+- fixes
+  - fixed a testing bug where after Click 8.2.0 the stdout and stderr were not properly captured in the tests with error assertion
+
 ## [2.0.0] 2026-02-11
 
 ### Added

README.md

@@ -27,20 +27,20 @@ The TEXT2SPARQL-CLIENT is the official testing CLI tool for the [International T
 
 ### CK25
 
-Usage example for the CK25 dataset from the [First International TEXT2SPARQL challenge](https://text2sparql.aksw.org/2025/). To execute tests for the CK25 dataset you will need a valid API according to the challenge specifications, [see](https://text2sparql.aksw.org/latest/participation/challenge/#process) and the directory `examples`. It contains the questions file `questions_ck25.yml` the True Result-set `ck25_true_result.json` (which can be generated with the questions file running the command `text2sparql query -o "ck25_true_result.json" questions.yml`), and the example shell file. To ask, query and evaluate according to your API open the `examples` directory and run `run_ck25.sh` passing `<API_IP> <API_NAME>`, for example:
+Usage example for the CK25 dataset from the [First International TEXT2SPARQL challenge](https://text2sparql.aksw.org/2025/). To execute tests for the CK25 dataset you will need a valid API according to the challenge specifications, [see](https://text2sparql.aksw.org/latest/participation/challenge/#process) and the directory `examples`. It contains the questions file `questions_ck25.yml` the True Result-set `ck25_true_result.json` (which can be generated with the questions file running the command `text2sparql query -o "ck25_true_result.json" questions.yml`), and the example shell file. To ask, query and evaluate according to your API open the `examples` directory and run `run_ck25.sh` passing `<API_URL> <API_NAME>`, for example:
 
 ```bash
 cd examples
-bash run_ck25.sh "http://127.0.0.1:8000" "text2sparql_qwen3"
+bash run_ck25.sh "http://localhost:8000" "text2sparql"
 ```
 
 ### DB25
 
-Usage example for the DB25 dataset from the [First International TEXT2SPARQL challenge](https://text2sparql.aksw.org/2025/). To execute tests for the CK25 dataset you will need a valid API according to the challenge specifications, [see](https://text2sparql.aksw.org/latest/participation/challenge/#process) and the directory `examples`. It contains the questions file `questions_db25.yml` the True Result-set `db25_true_result.json` (which can be generated with the questions file running the command `text2sparql query -o "db25_true_result.json" questions.yml`), and the example shell file. To ask, query and evaluate according to your API open the `examples` directory and run `run_db25.sh` passing `<API_IP> <API_NAME>`, for example:
+Usage example for the DB25 dataset from the [First International TEXT2SPARQL challenge](https://text2sparql.aksw.org/2025/). To execute tests for the CK25 dataset you will need a valid API according to the challenge specifications, [see](https://text2sparql.aksw.org/latest/participation/challenge/#process) and the directory `examples`. It contains the questions file `questions_db25.yml` the True Result-set `db25_true_result.json` (which can be generated with the questions file running the command `text2sparql query -o "db25_true_result.json" questions.yml`), and the example shell file. To ask, query and evaluate according to your API open the `examples` directory and run `run_db25.sh` passing `<API_URL> <API_NAME>`, for example:
 
 ```bash
 cd examples
-bash run_db25.sh "http://127.0.0.1:8000" "text2sparql_qwen3"
+bash run_db25.sh "http://localhost:8000" "text2sparql"
 ```
 
 ## Commands Reference
@@ -86,6 +86,9 @@ Query a TEXT2SPARQL endpoint using a questions YAML file and send each question
 |--------|------|---------|-------------|
 | `--answers-db` | Path | `responses.db` | Where to save the endpoint responses (SQLite database) |
 | `--timeout` | Integer | `600` | Timeout in seconds for each request |
+| `--retries` / `-r` | Integer | `5` | Number of retries for disconnected, http error and timed out requests |
+| `--retry-sleep` | Integer | `15` | Seconds to sleep between retries |
+| `--retries-log` | Path | `retries.log` | File to log retries to |
 | `--output` / `-o` | Path | `-` (stdout) | Save JSON output to this file |
 | `--cache` / `--no-cache` | Boolean | `True` | If possible, return a cached response from the answers database |
 

examples_ck25/run_ck25.sh

@@ -4,9 +4,9 @@
 set -euo pipefail; export FS=$'\n\t'
 
 API_NAME=${2:-}
-API_IP=${1:-}
+API_URL=${1:-}
 
-echo "Running ask, query and evaluate for all questions and responses for $API_NAME at $API_IP"
-text2sparql ask -o "${API_NAME}_ck25_answers.json" --answers-db "${API_NAME}_ck25_answers.db" questions_ck25.yml "${API_IP}"
+echo "Running ask, query and evaluate for all questions and responses for $API_NAME at $API_URL"
+text2sparql ask -o "${API_NAME}_ck25_answers.json" --answers-db "${API_NAME}_ck25_answers.db" --retries-log "${API_NAME}_ck25_retries.log" questions_ck25.yml "${API_URL}"
 text2sparql query -o "${API_NAME}_ck25_pred_result_set.json" -a "${API_NAME}_ck25_answers.json" questions_ck25.yml
 text2sparql evaluate -o "${API_NAME}_ck25_results.json" "${API_NAME}" ck25_true_result_set.json "${API_NAME}_ck25_pred_result_set.json"

examples_db25/run_db25.sh

@@ -4,9 +4,9 @@
 set -euo pipefail; export FS=$'\n\t'
 
 API_NAME=${2:-}
-API_IP=${1:-}
+API_URL=${1:-}
 
-echo "Running ask, query and evaluate for all questions and responses for $API_NAME at $API_IP"
-text2sparql ask -o "${API_NAME}_db25_answers.json" --answers-db "${API_NAME}_db25_answers.db" questions_db25.yml "${API_IP}"
+echo "Running ask, query and evaluate for all questions and responses for $API_NAME at $API_URL"
+text2sparql ask -o "${API_NAME}_db25_answers.json" --answers-db "${API_NAME}_db25_answers.db" --retries-log "${API_NAME}_db25_retries.log" questions_db25.yml "${API_URL}"
 text2sparql query -o "${API_NAME}_db25_pred_result_set.json" -a "${API_NAME}_db25_answers.json" -l "['en', 'es']" -e "http://141.57.8.18:9081/sparql" questions_db25.yml
 text2sparql evaluate -o "${API_NAME}_db25_results.json" -l "['en', 'es']" "${API_NAME}" db25_true_result_set.json "${API_NAME}_db25_pred_result_set.json"

tests/__init__.py

@@ -72,5 +72,5 @@ def run_asserting_error(command: tuple[str, ...] | list[str], match: str) -> Ann
     """Wrap the CliRunner, asserting exit 1 or more"""
     result = _run(command=command)
     assert result.exit_code >= 1, f"exit code should be 1 or more (but was {result.exit_code})"
-    assert match in result.stdout or match in str(result.result)
+    assert match in result.stdout or match in str(result.result) or match in str(result.output)
     return result

tests/test_ask.py

@@ -1,8 +1,6 @@
 """Test queries"""
 
-import pytest
-
-from tests import run, run_asserting_error
+from tests import run, run_asserting_error, run_without_assertion
 from tests.conftest import QuestionsFiles, ServerFixture, is_json_file
 
 
@@ -30,7 +28,31 @@ def test_non_successful_runs(server: ServerFixture, questions_files: QuestionsFi
     )
 
 
-@pytest.mark.skip(reason="tests that require output to be save are currently disabled")
+def test_timeout_handling_and_loggin(
+    server: ServerFixture, questions_files: QuestionsFiles
+) -> None:
+    """Test timeout handling and logging requests."""
+    command = (
+        "ask",
+        "--timeout",
+        "1",
+        "--retries",
+        "2",
+        "--retry-sleep",
+        "0",
+        "--retries-log",
+        "-",
+        "-o",
+        "output.json",
+        str(questions_files.with_ids),
+        server.get_url(),
+    )
+    result = run_without_assertion(command=command)
+    assert "Read timed out" in result.output
+    assert "Retrying" in result.output
+    assert "Maximum number of retries reached" in result.output
+
+
 def test_output(server: ServerFixture, questions_files: QuestionsFiles) -> None:
     """Test different output files."""
     output = "output.json"
@@ -42,10 +64,9 @@ def test_output(server: ServerFixture, questions_files: QuestionsFiles) -> None:
     assert is_json_file(output), "Output file should be JSON."
 
 
-@pytest.mark.skip(reason="tests that require output to be save are currently disabled")
 def test_cached_response(server: ServerFixture, questions_files: QuestionsFiles) -> None:
     """Test cached response."""
     command = ("ask", str(questions_files.with_ids), server.get_url())
-    assert "Cached response found." not in run(command=command).stdout
-    assert "Cached response found." in run(command=command).stdout
-    assert "Cached response found." not in run(command=(*command, "--no-cache")).stdout
+    assert "Cached response found." not in run(command=command).output
+    assert "Cached response found." in run(command=command).output
+    assert "Cached response found." not in run(command=(*command, "--no-cache")).output

tests/test_evaluate.py

@@ -1,8 +1,6 @@
 """Test evaluate"""
 
-import pytest
-
-from tests import run, run_asserting_error, run_without_assertion
+from tests import run, run_asserting_error
 from tests.conftest import ResultSetsFiles, is_json_file
 
 
@@ -20,17 +18,17 @@ def test_successful_evaluation(result_sets_files: ResultSetsFiles) -> None:
 
 def test_language_evaluation_error(result_sets_files: ResultSetsFiles) -> None:
     """Test language option error in evaluation."""
-    result = run_without_assertion(
+    run_asserting_error(
         command=(
             "evaluate",
             "-l",
             "en, de",
             "api_name",
             str(result_sets_files.result_set),
             str(result_sets_files.result_set),
-        )
+        ),
+        match="not a valid language list",
     )
-    assert result.exit_code >= 1, f"exit code should be 1 or more (but was {result.exit_code})"
 
 
 def test_missing_question_true_result_set(result_sets_files: ResultSetsFiles) -> None:
@@ -46,7 +44,6 @@ def test_missing_question_true_result_set(result_sets_files: ResultSetsFiles) ->
     )
 
 
-@pytest.mark.skip(reason="tests that require output to be save are currently disabled")
 def test_output_evaluation(result_sets_files: ResultSetsFiles) -> None:
     """Test evaluation with output file."""
     output = "output.json"

tests/test_query.py

@@ -1,8 +1,6 @@
 """Test query"""
 
-import pytest
-
-from tests import run, run_asserting_error, run_without_assertion
+from tests import run, run_asserting_error
 from tests.conftest import QuestionsFiles, ResponsesFiles, is_json_file
 
 
@@ -48,20 +46,19 @@ def test_language_query_error(
     questions_files: QuestionsFiles, responses_files: ResponsesFiles
 ) -> None:
     """Test query with language option error."""
-    result = run_without_assertion(
+    run_asserting_error(
         command=(
             "query",
             "-a",
             str(responses_files.responses),
             "-l",
             "en, de",
             str(questions_files.with_ids),
-        )
+        ),
+        match="not a valid language list",
     )
-    assert result.exit_code >= 1, f"exit code should be 1 or more (but was {result.exit_code})"
 
 
-@pytest.mark.skip(reason="tests that require output to be save are currently disabled")
 def test_output_query(questions_files: QuestionsFiles, responses_files: ResponsesFiles) -> None:
     """Test query with output file."""
     output = "output.json"

text2sparql_client/commands/ask.py

@@ -4,6 +4,7 @@
 import sys
 from io import TextIOWrapper
 from pathlib import Path
+from time import sleep
 
 import click
 import requests
@@ -12,7 +13,7 @@
 from pydantic import ValidationError
 
 from text2sparql_client.database import Database
-from text2sparql_client.models.questions_file import QuestionsFile
+from text2sparql_client.models.questions_file import Question, QuestionsFile
 from text2sparql_client.request import text2sparql
 
 
@@ -23,6 +24,67 @@ def check_output_file(file: str) -> None:
         sys.exit(1)
 
 
+def _retry_response(  # noqa: PLR0913
+    counter: int,
+    retries: int,
+    retry_sleep: int,
+    responses: list,
+    url: str,
+    file_model: QuestionsFile,
+    question_section: Question,
+    language: str,
+    question: str,
+    database: Database,
+    timeout: int,
+    cache: bool,
+) -> None:
+    qname = f"{file_model.dataset.prefix}:{question_section.id}-{language}"
+
+    if counter > retries:
+        logger.bind(retry=True).error(
+            f"{qname} | Maximum number of retries reached. Skipping question."
+        )
+        return
+
+    logger.bind(retry=True).info(
+        f"{qname} | Retrying ({counter}/{retries}) after {retry_sleep} seconds..."
+    )
+    sleep(retry_sleep)
+    try:
+        response = text2sparql(
+            endpoint=url,
+            dataset=file_model.dataset.id,
+            question=question,
+            database=database,
+            timeout=timeout,
+            cache=cache,
+        )
+        answer: dict[str, str] = response.model_dump()
+        if question_section.id and file_model.dataset.prefix:
+            answer["qname"] = f"{file_model.dataset.prefix}:{question_section.id}-{language}"
+            answer["uri"] = f"{file_model.dataset.id}{question_section.id}-{language}"
+        responses.append(answer)
+    except (requests.ConnectionError, requests.HTTPError, requests.ReadTimeout) as error:
+        logger.bind(retry=True).warning(f"{qname} | {error}")
+        _retry_response(
+            counter=counter + 1,
+            retries=retries,
+            retry_sleep=retry_sleep,
+            responses=responses,
+            url=url,
+            file_model=file_model,
+            question_section=question_section,
+            language=language,
+            question=question,
+            database=database,
+            timeout=timeout,
+            cache=cache,
+        )
+    except ValidationError as error:
+        logger.debug(str(error))
+        logger.error("validation error")
+
+
 @click.command(name="ask")
 @click.argument("QUESTIONS_FILE", type=click.File())
 @click.argument("URL", type=click.STRING)
@@ -40,6 +102,28 @@ def check_output_file(file: str) -> None:
     show_default=True,
     help="Timeout in seconds.",
 )
+@click.option(
+    "--retries",
+    "-r",
+    type=int,
+    default=5,
+    show_default=True,
+    help="Number of retries for disconnected, http error and timed out requests.",
+)
+@click.option(
+    "--retry-sleep",
+    type=int,
+    default=15,
+    show_default=True,
+    help="Amount of seconds to sleep before retrying a request.",
+)
+@click.option(
+    "--retries-log",
+    type=click.Path(dir_okay=False, writable=True, file_okay=True, allow_dash=True),
+    default="retries.log",
+    show_default=True,
+    help="File to log retries errors to.",
+)
 @click.option(
     "--output",
     "-o",
@@ -59,6 +143,9 @@ def ask_command(  # noqa: PLR0913
     url: str,
     answers_db: str,
     timeout: int,
+    retries: int,
+    retry_sleep: int,
+    retries_log: str,
     output: str,
     cache: bool,
 ) -> None:
@@ -71,10 +158,12 @@ def ask_command(  # noqa: PLR0913
     file_model = QuestionsFile.model_validate(yaml.safe_load(questions_file))
     logger.info(f"Asking questions about dataset {file_model.dataset.id} on endpoint {url}.")
     check_output_file(file=output)
+    logger.add(retries_log, filter=lambda record: "retry" in record["extra"])
     responses = []
     for question_section in file_model.questions:
         for language, question in question_section.question.items():
             logger.info(f"{question} ({language}) ... ")
+            qname = f"{file_model.dataset.prefix}:{question_section.id}-{language}"
             try:
                 response = text2sparql(
                     endpoint=url,
@@ -86,13 +175,25 @@ def ask_command(  # noqa: PLR0913
                 )
                 answer: dict[str, str] = response.model_dump()
                 if question_section.id and file_model.dataset.prefix:
-                    answer["qname"] = (
-                        f"{file_model.dataset.prefix}:{question_section.id}-{language}"
-                    )
+                    answer["qname"] = qname
                     answer["uri"] = f"{file_model.dataset.id}{question_section.id}-{language}"
                 responses.append(answer)
             except (requests.ConnectionError, requests.HTTPError, requests.ReadTimeout) as error:
-                logger.error(str(error))
+                logger.bind(retry=True).warning(f"{qname} | {error}")
+                _retry_response(
+                    counter=1,
+                    retries=retries,
+                    retry_sleep=retry_sleep,
+                    responses=responses,
+                    url=url,
+                    file_model=file_model,
+                    question_section=question_section,
+                    language=language,
+                    question=question,
+                    database=database,
+                    timeout=timeout,
+                    cache=cache,
+                )
             except ValidationError as error:
                 logger.debug(str(error))
                 logger.error("validation error")