JS Server work

* JS Server: Implemented snapshot & verify

Also updated CBL dependency to 1.0.0-5

* JS Server: Implemented blobs for /updateDatabase

* JS Server: Normalize collection names everywhere

i.e. convert `_default._default` to `_default`

* JS Server: Actually save doc after updating it

* Disable more JS tests and a few more test server fixes

1. Make it an error to have a snapshot update to a document that doesn't exist in a snapshot
2. Only return result.document if the snapshot update type is an update (not delete or purge)
3. Sometimes we make a snapshot of a document that does not exist in the DB in order to verify that it didn't get pulled unintentionally, so check that (it will show up as an entry with an "undefined" value)
4. updateDatabase should be allowed to create documents
5. Bug in update remove properties handling (it was iterating the actual strings and sending them one character at a time)

* Fix API spec for conflict resolvers and run query test

* JS Server: Added auth support for replicator

* JS Server: Implemented conflict resolvers

* JS Server: Throw error when updating keypath fails

* JS Server: Implemented push/pull filters

* JS Server: Resolve relative blob URLs

* JS Server: support updatedBlobs in /verifyDocuments

* JS Server: Just prettified tdkSchema.ts

* JS Server: Fixed transaction error in /updateDatabase

Updating a blob would trigger an exception because it's illegal to
make non-database async calls within a transaction.
So use Collection.updateMultiple() instead.

NOTE: This assumes that a /updateDatabase request does not list the
same document multiple times! If so, this will probably fail.

* A few small fixes

1. Normalize collection name in replicator config
2. Correct blob base URL

* Correct snapshot tests and behavior

Only update verifications (not delete or purge) return the document.  Also, there is a distinction between null and missing for snapshot entries:  null means "I want to verify this later, but it doesn't exist right now".  undefined is "Not included in the snapshot and ineligible for verification"

* Further correct Snapshot

- Spurious `!` on line 40 messes up the test for undefined
- I accidentally used `!!` (Kotlin syntax) instead of `!` on line 59
- Using `T | null` in DocumentMap is incorrect; nulls are handled in
  the declaration of #documents, which is
  `DocumentMap<cbl.CBLDocument | null>`.

* Correct and error code and prettify message

The TDK expects an HTTP 400 when a nonexistent blob is requested.  Remove "self" from error message since it results in it being printed twice, and make the error message "returned XXX" not weirdly formatted.

* Update travel JS dataset

Needs two empty collections to be compatible with SGW's setup

* Switch SGW certificate strategy

The cert is now issued by a CA that can easily be trusted in python, as well as browser.  Existing SDK cert pinning should still be fine as well.  Also only copy prerelease RPM if it's not already on the remote machine (uploads are slow)

* Emit "transport: ws" for Javascript topology setups

* Need to separate push and pull filter

Their functions are different signatures so a premade CreateFilter is not possible

* Relax smoke test requirements for error on connection failure

Since JS has no access to that information :(

---------

Co-authored-by: Jim Borden <jim.borden@couchbase.com>

Author: snej

client/smoke_tests/test_listener.py

@@ -1,18 +1,18 @@
 import aiohttp
 import pytest
 from cbltest import CBLPyTest
+from cbltest.api.cbltestclass import CBLTestClass
 from cbltest.api.listener import Listener
-from cbltest.globals import CBLPyTestGlobal
+from cbltest.responses import ServerVariant
 
 
-class TestListener:
-    def setup_method(self, method):
-        # If writing a new test do not forget this step or the test server
-        # will not be informed about the currently running test
-        CBLPyTestGlobal.running_test_name = method.__name__
-
+class TestListener(CBLTestClass):
     @pytest.mark.asyncio(loop_scope="session")
     async def test_start_stop_listener(self, cblpytest: CBLPyTest) -> None:
+        await self.skip_if_not_platform(
+            cblpytest.test_servers[0], ServerVariant.ALL & ~ServerVariant.JS
+        )
+
         dbs = await cblpytest.test_servers[0].create_and_reset_db(["db1"])
         db = dbs[0]
         listener = Listener(db, ["_default._default"], 59840)

client/smoke_tests/test_log_endpoint.py

@@ -3,18 +3,17 @@
 import pytest
 import requests
 from cbltest import CBLPyTest
-from cbltest.globals import CBLPyTestGlobal
+from cbltest.api.cbltestclass import CBLTestClass
 from cbltest.logging import LogSlurpHandler, _cbl_log
+from cbltest.responses import ServerVariant
 
 
-class TestLogEndpoint:
-    def setup_method(self, method):
-        # If writing a new test do not forget this step or the test server
-        # will not be informed about the currently running test
-        CBLPyTestGlobal.running_test_name = method.__name__
-
+class TestLogEndpoint(CBLTestClass):
     @pytest.mark.asyncio(loop_scope="session")
     async def test_log_message(self, cblpytest: CBLPyTest) -> None:
+        await self.skip_if_not_platform(
+            cblpytest.test_servers[0], ServerVariant.ALL & ~ServerVariant.JS
+        )
         if cblpytest.config.logslurp_url is None:
             pytest.skip(
                 "No LogSlurp server configured (required to check functionality)"

client/smoke_tests/test_multipeer.py

@@ -2,19 +2,20 @@
 
 import pytest
 from cbltest import CBLPyTest
+from cbltest.api.cbltestclass import CBLTestClass
 from cbltest.api.multipeer_replicator import MultipeerReplicator
 from cbltest.api.replicator_types import ReplicatorCollectionEntry
-from cbltest.globals import CBLPyTestGlobal
+from cbltest.responses import ServerVariant
 
 
-class TestMultipeerReplicator:
-    def setup_method(self, method):
-        # If writing a new test do not forget this step or the test server
-        # will not be informed about the currently running test
-        CBLPyTestGlobal.running_test_name = method.__name__
-
+class TestMultipeerReplicator(CBLTestClass):
     @pytest.mark.asyncio(loop_scope="session")
     async def test_start_stop_multipeer(self, cblpytest: CBLPyTest) -> None:
+        await self.skip_if_not_platform(
+            cblpytest.test_servers[0], ServerVariant.ALL & ~ServerVariant.JS
+        )
+        await self.skip_if_cbl_not(cblpytest.test_servers[0], ">= 3.3.0")
+
         dbs = await cblpytest.test_servers[0].create_and_reset_db(["db1"])
         db = dbs[0]
         multipeer = MultipeerReplicator(

client/smoke_tests/test_perform_maintenance.py

@@ -1,9 +1,11 @@
 import pytest
 from cbltest import CBLPyTest
+from cbltest.api.cbltestclass import CBLTestClass
 from cbltest.api.database_types import MaintenanceType
+from cbltest.responses import ServerVariant
 
 
-class TestPerformMaintenance:
+class TestPerformMaintenance(CBLTestClass):
     @pytest.mark.asyncio(loop_scope="session")
     @pytest.mark.parametrize(
         "maintenance_type",
@@ -17,6 +19,11 @@ class TestPerformMaintenance:
     async def test_perform_maintenance_endpoint(
         self, cblpytest: CBLPyTest, maintenance_type: MaintenanceType
     ) -> None:
+        if maintenance_type != MaintenanceType.COMPACT:
+            await self.skip_if_not_platform(
+                cblpytest.test_servers[0], ServerVariant.ALL & ~ServerVariant.JS
+            )
+
         dbs = await cblpytest.test_servers[0].create_and_reset_db(
             ["db1"], dataset="names"
         )

client/smoke_tests/test_run_query.py

@@ -14,11 +14,13 @@ async def test_run_query(self, cblpytest: CBLPyTest):
         db = await cblpytest.test_servers[0].create_and_reset_db(
             ["db1"], dataset="names"
         )
-        results = await db[0].run_query("select meta().id from _ LIMIT 5")
+        results = await db[0].run_query(
+            "select meta().id from _ ORDER BY meta().id LIMIT 5"
+        )
         assert results is not None, "The query should return a result"
         assert len(results) == 5, "The query should return five results"
         i = 1
-        for r in results:
+        expected = ["name_1", "name_10", "name_100", "name_11", "name_12"]
+        for i, r in enumerate(results):
             assert "id" in r, "The result should have an ID column"
-            assert r["id"] == f"name_{i}", f"The result ID should be name_{i}"
-            i += 1
+            assert r["id"] == expected[i], f"The result ID should be {expected[i]}"

client/smoke_tests/test_start_replicator.py

@@ -2,7 +2,6 @@
 from cbltest import CBLPyTest
 from cbltest.api.database import Database
 from cbltest.api.error import CblTestServerBadResponseError
-from cbltest.api.error_types import ErrorDomain
 from cbltest.api.replicator import Replicator
 from cbltest.api.replicator_types import (
     ReplicatorActivityLevel,
@@ -35,11 +34,7 @@ async def test_bad_endpoint(self, cblpytest: CBLPyTest) -> None:
         repl.add_default_collection()
         await repl.start()
         status = await repl.wait_for(ReplicatorActivityLevel.STOPPED)
-        assert (
-            status.error is not None
-            and ErrorDomain.equal(status.error.domain, ErrorDomain.CBL)
-            and status.error.code == 5002
-        )
+        assert status.error is not None
 
     @pytest.mark.asyncio(loop_scope="session")
     @pytest.mark.parametrize(

client/src/cbltest/api/error.py

@@ -21,10 +21,14 @@ def response(self) -> TestServerResponse:
         """Gets the body of the response that had the bad status"""
         return self.__response
 
-    def __init__(self, code: int, response: TestServerResponse, *args):
+    def __init__(self, code: int, response: TestServerResponse, message: str):
         self.__code = code
         self.__response = response
-        super().__init__(args)
+        self.__message = message
+        super().__init__(message)
+
+    def __str__(self) -> str:
+        return self.__message
 
 
 class CblTimeoutError(Exception):

client/src/cbltest/api/syncgateway.py

@@ -19,6 +19,41 @@
 from cbltest.utils import assert_not_null
 from cbltest.version import VERSION
 
+# This is copied from environment/aws/sgw_setup/cert/ca_cert.pem
+# So if that file ever changes, change this too.
+_SGW_CA_CERT: str = """-----BEGIN CERTIFICATE-----
+MIIFWTCCA0GgAwIBAgIUBdrc0OhquX8RnXtZ6AiOY+57C18wDQYJKoZIhvcNAQEL
+BQAwPDEZMBcGA1UEAwwQSW50ZXJuYWwgVGVzdCBDQTESMBAGA1UECgwJQ291Y2hi
+YXNlMQswCQYDVQQGEwJVUzAeFw0yNTEwMjkwMTAzMDBaFw0yNzEwMjkwMTAzMDBa
+MDwxGTAXBgNVBAMMEEludGVybmFsIFRlc3QgQ0ExEjAQBgNVBAoMCUNvdWNoYmFz
+ZTELMAkGA1UEBhMCVVMwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCl
+vMLIQDFfEcttEUSzBKkzoRSSLJ3Z/73xmJxBenCtZ9HasLhF3iJxwyQK09nD7sLv
+RRwLeRfY8QObr/F/qJAa1cQtVA/5UxIiKsjDk+TrUibg4p6NFSgKUEg+08D0tRHG
+CF3CF/3qcM/10A+Pg2K1COaAtPrYjslOv8DoDBzwOBxibaheDZmtBdPEeHghDXZr
+DWYDe2770XGzKYqINCEDxNdyDUBdiNSzuX2h/YeZi6vGTtpAt3Iti2SIerRrCiah
+UOlykQoqiDVh4JPXts79Xhszw0oDK6YWHEBBfXmYDUdYAyF97XC6hZc+6HxiCVTB
+887mkyLCuKMGfB3dabyCqJ31fXm7gmufOs8voCfi/sKjLgcdZQUY4Gw345oYI3Yw
+O41ig/uR04KW2xASba38vXt0fEl9/50+AO3xAy9oaY36nLSnBwTV72VbvTlvevGf
+zSHbVIbtzcuovpudghYizmIqMEFguc8VsGgmwZb8mkypzB80SOoED3nJRziIK1ym
+e+NuO0DIG6xMPUhputNhwqaeYXuSmcUH5YcmLN//ewMIjzxoH33H1cwcADHFioR/
+YfIraSgVZCUhrN9aJlXdDOzDuhbVpXYJMbh5PfAiNLHPCXmo685Utf3ID+nFW1wd
+WOIyuE3aJ5KVtG8hjlgKARV7eEqtxHjIl41QtsxalwIDAQABo1MwUTAdBgNVHQ4E
+FgQUs2WdMu1wh9pJ5dPN80yN1NPAkSswHwYDVR0jBBgwFoAUs2WdMu1wh9pJ5dPN
+80yN1NPAkSswDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAgEAhiYP
+a9dvAv/33u9vBKzUo045RRrfEpv80DSZQb4ttyulrIfbaLFHxaDcf2+S+mywAgoW
+tf9SAwWO41qU7sfIBnFdCh982nP1dD707GDAZIe8ZNpl/Vu3hWY3TRQAp9ufA51w
+wxn0m2tOS18UXpv5BNX1kVaLlAiOzRzmP1ghx08v9yd8eBgnjJ89D2m1U+qFS1Xk
+egiyw66HHc3bG+eo523/l4RDqTx6KkhYnD3Bz89IxMaeK7CuynCY3VyVWPeIUfBr
+clkDBqZa4o7fD3xV6Wiu/NHZsWJx3wog1wwelvlsyOVM+mfd4IOsPGVCCdDGtpoq
+sT+f9mPDXXHKuDER8a7HiCgGK8rAQtCm/P5UFp2HUEIru/psWCXc3vh9HYVX9W79
+TwS+AVAlkeVogs1ugqAXOuGmstnevj6XzA8PszCKDSIV+t1PJSSOtypUyN4gbGGx
+sk1s9bwqy7bw2cMh3tt7HromGOoLnPxnsbQCs5HsqNdiEsPABWnI6m7epm0tFjCe
+gHDyw1LdmZlZ3R7DT+CwfyhxL6hktfs8h7goR1vkmS2q0Alxmw9faKVVpDyWnsZC
+qv6PMC0fI7jhvrr2Uf2Hhw9SQlBFwZ7LjjLqjuuJkclM4VooDElsLbPjSUbA+c5h
+WCKJ0c94mrl9GwwBmcSIKJBvd6u7uAta2fREJeE=
+-----END CERTIFICATE-----
+"""
+
 
 class _CollectionMap(JSONSerializable):
     @property
@@ -344,7 +379,7 @@ def _create_session(
         self, secure: bool, scheme: str, url: str, port: int, auth: BasicAuth | None
     ) -> ClientSession:
         if secure:
-            ssl_context = ssl.create_default_context(cadata=self.tls_cert())
+            ssl_context = ssl.create_default_context(cadata=_SGW_CA_CERT)
             # Disable hostname check so that the pre-generated SG can be used on any machines.
             ssl_context.check_hostname = False
             return ClientSession(

client/src/cbltest/requests_transport.py

@@ -85,7 +85,7 @@ async def send(
         cbl_trace(f"Received {ret_val} from {self.__url}")
         if not resp.ok:
             raise CblTestServerBadResponseError(
-                resp.status, ret_val, f"{self} returned {resp.status}"
+                resp.status, ret_val, f"returned {resp.status}"
             )
 
         return ret_val
@@ -167,7 +167,7 @@ async def send(
         cbl_trace(f"Received {ret_val} from {self.__url}")
         if ret_val.status_code != 200:
             raise CblTestServerBadResponseError(
-                ret_val.status_code, ret_val, f"{self} returned {ret_val.status_code}"
+                ret_val.status_code, ret_val, f"returned {ret_val.status_code}"
             )
 
         return ret_val

dataset/server/dbs/js/travel/index.json

@@ -2,7 +2,9 @@
     "name": "travel",
     "collections": [
         "travel.airlines",
+        "travel.airports",
         "travel.hotels",
+        "travel.landmarks",
         "travel.routes"
     ]
 }