Merge branch 'main' into replication_url_for_testserver

Author: vipbhardwaj

.pre-commit-config.yaml

@@ -0,0 +1,16 @@
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+    -   id: check-executables-have-shebangs
+    -   id: check-shebang-scripts-are-executable
+-   repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.11.7
+    hooks:
+    -   id: ruff
+    -   id: ruff-format
+-   repo: https://github.com/asottile/pyupgrade
+    rev: v3.19.1
+    hooks:
+    -   id: pyupgrade
+        args: [--py310-plus]

README.md

@@ -17,7 +17,7 @@ The system consists of 4 components:
 * README.md - this file
 * client - A python framework that configures the environment, runs tests and reports their results
 * dataset - A collection datasets used in tests.  Each dataset appears in two formats, as a cblite2 db, and as raw JSON
-* environment - The docker compose script that creates and runs the environment
+* environment - Scripts and such for setting up and tearing down a backend environment
 * jenkins - The jenkins pipelines and the docker compose for running the Jenkins server.
 * servers - A test server for each CBL platform.  Servers run on their platform and accept instructions from the client to run tests
 * spec - Documentation:  The specification for this system.
@@ -28,7 +28,7 @@ The system consists of 4 components:
 ### Requirements
 
 1. [Docker](https://www.docker.com/get-started)
-2. [Python 3.9+](https://www.python.org/downloads)
+2. [Python 3.10+](https://www.python.org/downloads)
 3. [OpenSSL 1.1 for CBS Python SDK](https://docs.couchbase.com/python-sdk/current/hello-world/start-using-sdk.html)
 4. [Git LFS](https://git-lfs.com)
 
@@ -40,43 +40,15 @@ The system consists of 4 components:
    ```
    This repository uses Git LFS to store binary dataset files. Ensure that you have [Git LFS](https://git-lfs.com) installed, and run `git lfs install` once to setup the extension hook before cloning the repository.
 
-2. Open a terminal window and start the environment by running the `./start_environment.py` script in the environment folder.
-   The script will start CBS and SG in the docker container in the background and wait until SG is successfully started before exiting.
-   ```
-   cd environment
-   ./start_environment.py
-   ```
-   * To shutdown the environment, run `docker compose down`.
-   
-3. Build and run the test server of the platform that you want to test.
-   * [C](https://github.com/couchbaselabs/couchbase-lite-tests/tree/main/servers/c)
-   * Android
-   * .Net
-   * iOS
-     
-4. From the tests directory, set up a python virtual environment:
-   ```
-   cd tests
-   python3 -m venv venv
-   . venv/bin/activate
-   pip install -r requirements.txt
-   ```
-   * You may need to use `python<version>` command e.g. `python3.10` if you already have `python3` for the other version.
-   * You only need to create the python venv once. To reactivate run `. venv/bin/activate`, and to deactivate run `deactivate`.
-   * When you update the repo or the Python TDK code, run `pip install ../client` to update the TDK.
+2. From the jenkins/pipelines directory of your choice, run the relevant script (such as run_test.ps1, test.sh, etc) with the various arguments regarding versions of things to use.
 
-5. Edit the file `config.example.json` with the URL of your Test server started in the Step 2.
-   ```
-   "test-servers": ["http://192.168.100.104:8080"]
-   ```
-6. Run the pytest tests as examples below.
-   ```
-   # Run one test file:
-   pytest --config config.example.json test_basic_replication.py
+### Contributing
 
-   # Run all tests:
-   pytest --config config.example.json
+Notice that this repo has a .pre-commit-config.yaml file, which means it is ready to use with the [pre-commit](https://pre-commit.com/#intro) python tool.  Essentially, after you clone this repo you should run 
 
-   # Run all with detail and without deprecation warning:
-   pytest -v --no-header -W ignore::DeprecationWarning --config config.example.json
-   ```
+```
+pip install pre-commit
+pre-commit install
+```
+
+After that git pre-commit validation will check various things for you to ensure adherence to best practices and standards.

client/smoke_tests/test_snapshot_verify.py

@@ -1,4 +1,4 @@
-from typing import Any, Dict, List, Optional, Union
+from typing import Any
 
 import pytest
 from cbltest import CBLPyTest
@@ -16,12 +16,12 @@ def setup_method(self, method):
 
     def upsert_multiple(
         self,
-        instances: List[Union[SnapshotUpdater, DatabaseUpdater]],
+        instances: list[SnapshotUpdater | DatabaseUpdater],
         collection: str,
         document: str,
-        new_properties: Optional[List[Dict[str, Any]]] = None,
-        removed_properties: Optional[List[str]] = None,
-        new_blobs: Optional[Dict[str, str]] = None,
+        new_properties: list[dict[str, Any]] | None = None,
+        removed_properties: list[str] | None = None,
+        new_blobs: dict[str, str] | None = None,
     ) -> None:
         for instance in instances:
             instance.upsert_document(
@@ -30,7 +30,7 @@ def upsert_multiple(
 
     def delete_multiple(
         self,
-        instances: List[Union[SnapshotUpdater, DatabaseUpdater]],
+        instances: list[SnapshotUpdater | DatabaseUpdater],
         collection: str,
         document: str,
     ) -> None:
@@ -39,7 +39,7 @@ def delete_multiple(
 
     def purge_multiple(
         self,
-        instances: List[Union[SnapshotUpdater, DatabaseUpdater]],
+        instances: list[SnapshotUpdater | DatabaseUpdater],
         collection: str,
         document: str,
     ) -> None:

client/smoke_tests/test_start_replicator.py

@@ -1,5 +1,3 @@
-from typing import Optional
-
 import pytest
 from cbltest import CBLPyTest
 from cbltest.api.database import Database
@@ -53,7 +51,7 @@ async def test_bad_endpoint(self, cblpytest: CBLPyTest) -> None:
         ],
     )
     async def test_known_conflict_resolvers(
-        self, cblpytest: CBLPyTest, name: str, parameters: Optional[dict]
+        self, cblpytest: CBLPyTest, name: str, parameters: dict | None
     ):
         dbs = await cblpytest.test_servers[0].create_and_reset_db(["db1"])
         db = dbs[0]

client/smoke_tests/test_update_endpoint.py

@@ -1,5 +1,3 @@
-from typing import Optional
-
 import pytest
 from cbltest import CBLPyTest
 from cbltest.api.database import Database
@@ -13,7 +11,7 @@ def setup_method(self, method) -> None:
         # will not be informed about the currently running test
         CBLPyTestGlobal.running_test_name = method.__name__
 
-        self.db: Optional[Database] = None
+        self.db: Database | None = None
 
     @pytest.mark.asyncio(loop_scope="session")
     @pytest.mark.parametrize(

client/src/cbltest/__init__.py

@@ -1,6 +1,4 @@
 from json import dumps
-from sys import version_info
-from typing import Dict, List, Optional
 
 from .api.couchbaseserver import CouchbaseServer
 from .api.syncgateway import SyncGateway
@@ -17,9 +15,6 @@
 from .logging import LogLevel, cbl_log_init, cbl_setLogLevel
 from .requests import RequestFactory
 
-if version_info < (3, 9):
-    raise RuntimeError("Python must be at least v3.9!")
-
 
 class CBLPyTest:
     """
@@ -38,7 +33,7 @@ def log_level(self) -> LogLevel:
         return self.__log_level
 
     @property
-    def extra_props(self) -> Optional[Dict[str, str]]:
+    def extra_props(self) -> dict[str, str] | None:
         """Gets the extra properties provided as parsed from the provided JSON file path"""
         return self.__extra_props
 
@@ -48,30 +43,30 @@ def request_factory(self) -> RequestFactory:
         return self.__request_factory
 
     @property
-    def test_servers(self) -> List[TestServer]:
+    def test_servers(self) -> list[TestServer]:
         """Gets the list of Test Servers available"""
         return self.__test_servers
 
     @property
-    def sync_gateways(self) -> List[SyncGateway]:
+    def sync_gateways(self) -> list[SyncGateway]:
         """Gets the list of Sync Gateways available"""
         return self.__sync_gateways
 
     @property
-    def couchbase_servers(self) -> List[CouchbaseServer]:
+    def couchbase_servers(self) -> list[CouchbaseServer]:
         """Gets the list of Couchbase Servers available"""
         return self.__couchbase_servers
 
     @property
-    def load_balancers(self) -> List[str]:
+    def load_balancers(self) -> list[str]:
         """Gets the list of Load Balancers available"""
         return self.__config.load_balancers
 
     @staticmethod
     async def create(
         config_path: str,
         log_level: LogLevel = LogLevel.VERBOSE,
-        extra_props_path: Optional[str] = None,
+        extra_props_path: str | None = None,
         test_server_only: bool = False,
     ):
         ret_val = CBLPyTest(config_path, log_level, extra_props_path, test_server_only)
@@ -93,7 +88,7 @@ def __init__(
         self,
         config_path: str,
         log_level: LogLevel = LogLevel.VERBOSE,
-        extra_props_path: Optional[str] = None,
+        extra_props_path: str | None = None,
         test_server_only: bool = False,
     ):
         _assert_not_null(config_path, "config_path")
@@ -105,7 +100,7 @@ def __init__(
             self.__extra_props = _parse_extra_props(extra_props_path)
 
         self.__request_factory = RequestFactory(self.__config)
-        self.__test_servers: List[TestServer] = []
+        self.__test_servers: list[TestServer] = []
         index = 0
         for ts in self.__config.test_servers:
             ts_info = TestServerInfo(ts)
@@ -114,7 +109,7 @@ def __init__(
             )
             index += 1
 
-        self.__sync_gateways: List[SyncGateway] = []
+        self.__sync_gateways: list[SyncGateway] = []
         index = 0
         if not test_server_only:
             for sg in self.__config.sync_gateways:
@@ -131,7 +126,7 @@ def __init__(
                 )
                 index += 1
 
-        self.__couchbase_servers: List[CouchbaseServer] = []
+        self.__couchbase_servers: list[CouchbaseServer] = []
         if not test_server_only:
             for cbs in self.__config.couchbase_servers:
                 cbs_info = CouchbaseServerInfo(cbs)

client/src/cbltest/api/cloud.py

@@ -1,6 +1,6 @@
 from json import dumps, load
 from pathlib import Path
-from typing import List, Optional, cast
+from typing import cast
 
 from opentelemetry.trace import get_tracer
 
@@ -35,13 +35,13 @@ def _check_all_indexes_removed(self, bucket: str) -> None:
             raise ValueError(f"{count} indexes remain in '{bucket}' bucket")
 
     def _wait_for_all_indexed_removed(self, bucket: str) -> None:
-        _try_n_times(10, 2, True, self._check_all_indexes_removed, type(None), bucket)
+        _try_n_times(10, 2, True, self._check_all_indexes_removed, bucket)
 
     async def configure_dataset(
         self,
         dataset_path: Path,
         dataset_name: str,
-        sg_config_options: Optional[List[str]] = None,
+        sg_config_options: list[str] | None = None,
     ) -> None:
         """
         Creates a database, ensuring that it is in an empty state when finished

client/src/cbltest/api/couchbaseserver.py

@@ -1,9 +1,7 @@
 from datetime import timedelta
 from time import sleep
-from typing import Dict, List
 
 from couchbase.auth import PasswordAuthenticator
-from couchbase.bucket import Bucket
 from couchbase.cluster import Cluster
 from couchbase.exceptions import (
     BucketAlreadyExistsException,
@@ -41,7 +39,7 @@ def __init__(self, url: str, username: str, password: str):
             self.__cluster = Cluster(url, opts)
             self.__cluster.wait_until_ready(timedelta(seconds=10))
 
-    def create_collections(self, bucket: str, scope: str, names: List[str]) -> None:
+    def create_collections(self, bucket: str, scope: str, names: list[str]) -> None:
         """
         A function that will create a specified set of collections in the specified scope
         which resides in the specified bucket
@@ -55,9 +53,7 @@ def create_collections(self, bucket: str, scope: str, names: List[str]) -> None:
             "Create Scope",
             attributes={"cbl.scope.name": scope, "cbl.bucket.name": bucket},
         ):
-            bucket_obj = _try_n_times(
-                10, 1, False, self.__cluster.bucket, Bucket, bucket
-            )
+            bucket_obj = _try_n_times(10, 1, False, self.__cluster.bucket, bucket)
             c = bucket_obj.collections()
             try:
                 if scope != "_default":
@@ -150,7 +146,7 @@ def run_query(
         bucket: str,
         scope: str = "_default",
         collection: str = "_default",
-    ) -> List[Dict]:
+    ) -> list[dict]:
         """
         Runs the specified query on the server.  The query may be formatted in a special way.