Backport changes from development (#659)

* Backport changes from development

* Delete travis file

* Backport version checking

* Backport pbutils fixes

* Backport test write path

* Backport test suite changes

Author: gtopper

.github/workflows/pr.yaml

@@ -85,6 +85,10 @@ jobs:
       - name: Lint Python code
         run: make flake8
 
+      - uses: actions/setup-go@v3
+        with:
+          go-version: "^1.19.13"
+
       - name: Run Python tests
         run: make test-py
         env:

.github/workflows/push.yaml

@@ -87,5 +87,9 @@ jobs:
       - name: Lint Python code
         run: make flake8
 
+      - uses: actions/setup-go@v3
+        with:
+          go-version: "^1.19.13"
+
       - name: Run Python tests
         run: make test-py

.github/workflows/release.yaml

@@ -86,6 +86,10 @@ jobs:
       - name: Lint Python code
         run: make flake8
 
+      - uses: actions/setup-go@v3
+        with:
+          go-version: "^1.19.13"
+
       - name: Run Python tests
         run: make test-py
 
@@ -115,7 +119,7 @@ jobs:
 
       - uses: actions/setup-go@v3
         with:
-          go-version: "^1.19.5"
+          go-version: "^1.19.13"
 
       - name: Build binaries for ${{ matrix.go-os }}
         run: make frames-bin
@@ -187,7 +191,7 @@ jobs:
       - name: Set up python ${{ matrix.python-version }}
         uses: actions/setup-python@v4
         with:
-          python-version: 3.7
+          python-version: 3.9
 
       - name: Set version
         run: make set-version

README.md

@@ -4,9 +4,7 @@
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
 
 V3IO Frames (**"Frames"**) is a multi-model open-source data-access library that provides a unified high-performance DataFrame API for working with different types of data sources (backends).
-The library was developed by Iguazio to simplify working with data in the [Iguazio Data Science Platform](https://www.iguazio.com) (**"the platform"**), but it can be extended to support additional backend types.
-
-> **Note:** For a full API reference of the Frames platform backends, including detailed examples, see the Frames API reference in [the platform documentation](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/).
+The library was developed by Iguazio to simplify working with data in the [Iguazio MLOps Platform](https://www.iguazio.com) (**"the platform"**), but it can be extended to support additional backend types.
 
 #### In This Document
 
@@ -39,7 +37,7 @@ The library was developed by Iguazio to simplify working with data in the [Iguaz
 <a id="python-version"></a>
 #### Python Version
 
-The current version of Frames supports Python 3.6 and 3.7.
+The current version of Frames supports Python 3.7 and 3.9.
 
 <a id="initialization"></a>
 #### Initialization
@@ -59,15 +57,12 @@ All Frames client methods receive a [`backend`](#client-method-param-backend) pa
 Frames currently supports the following backend types:
 
 - `nosql` | `kv` &mdash; a platform NoSQL (key/value) table.
-  See the [platform NoSQL backend API reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/nosql/).
   <br/><br/>
   > **Note:** The documentation uses the `"nosql"` alias to the `"kv"` type, which was added in Frames v0.6.10-v0.9.13; `"kv"` is still supported for backwards compatibility with earlier releases.
 - `stream` &mdash; a platform data stream **[Tech Preview]**.
   <!-- [FRAMES-STREAMING-NO-SUPPORT]
-  See the [platform streaming backend API reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/stream/).
   -->
 - `tsdb` &mdash; a time-series database (TSDB).
-  See the [platform TSDB backend API reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/tsdb/).
 - `csv` &mdash; a comma-separated-value (CSV) file.
   This backend type is used only for testing purposes.
 
@@ -91,7 +86,6 @@ The `Client` class features the following methods for supporting operations on a
 
 When creating a Frames client, you must provide valid credentials for accessing the backend data, which Frames will use to identify the identity of the user.
 This can be done by using any of the following alternative methods (documented in order of precedence).
-For more information about the user authentication for the platform backends, see the [platform documentation](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/overview/#user-authentication):
 
 - <a id="user-auth-client-const-params"></a>Provide the authentication credentials in the call to the [`Client` constructor](#client-constructor) &mdash; either by setting the [`token`](#client-param-token) parameter to a valid authentication token (access key) or by setting the [`user`](#client-param-user) and [`password`](#client-param-password) parameters to a username and password.
   Note that you cannot set the token parameter concurrently with the username and password parameters.
@@ -124,7 +118,7 @@ Client(address=""[, data_url=""], container=""[, user="", password="", token=""]
 
 - <a id="client-param-address"></a>**address** &mdash; The address of the Frames service (`framesd`).
   Use the `grpc://` prefix for gRPC (default; recommended) or the `http://` prefix for HTTP.
-  When running locally on the platform, set this parameter to `framesd:8081` to use the gRPC (recommended) or to `framesd:8080` to use HTTP; for more information, see the [platform documentation](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/client-constructor/).
+  When running locally on the platform, set this parameter to `framesd:8081` to use the gRPC (recommended) or to `framesd:8080` to use HTTP.
 
   - **Type:** `str`
   - **Requirement:** Required 
@@ -252,7 +246,7 @@ All Frames backends that support the `create` method support the following commo
 <a id="method-create-params-tsdb"></a>
 #### `tsdb` Backend `create` Parameters
 
-The following `create` parameters are specific to the `tsdb` backend and are passed as keyword arguments via the `kw` parameter; for more information and examples, see the platform's [Frames TSDB-backend reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/tsdb/create/):
+The following `create` parameters are specific to the `tsdb` backend and are passed as keyword arguments via the `kw` parameter:
 
 - <a id="method-create-tsdb-param-rate"></a>**rate** &mdash; metric-samples ingestion rate.
 
@@ -279,8 +273,7 @@ The following `create` parameters are specific to the `tsdb` backend and are pas
 <a id="method-create-params-stream"></a>
 #### `stream` Backend `create` Parameters
 
-The following `create` parameters are specific to the `stream` backend and are passed as keyword arguments via the `kw` parameter<!--; for more information and examples, see the platform's [Frames streaming-backend reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/stream/create/)-->:
-<!-- [FRAMES-STREAMING-NO-SUPPORT] -->
+The following `create` parameters are specific to the `stream` backend and are passed as keyword arguments via the `kw` parameter:
 
 - <a id="method-create-stream-param-shards"></a>**shards** &mdash; The number of stream shards to create.
 
@@ -385,7 +378,7 @@ All Frames backends that support the `write` method support the following common
 <a id="method-write-params-nosql"></a>
 #### `nosql` Backend `write` Parameters
 
-The following `write` parameters are specific to the `nosql` backend; for more information and examples, see the platform's [Frames NoSQL-backend reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/nosql/write/):
+The following `write` parameters are specific to the `nosql` backend:
 
 <!--
 - <a id="method-write-nosql-param-expression"></a>**expression** (Optional) (default: `None`) &mdash; A platform update expression that determines how to update the table for all items in the DataFrame.
@@ -420,7 +413,7 @@ client.write(backend="nosql", table="mytable", dfs=df, expression="city='NY'", c
 <a id="method-write-params-tsdb"></a>
 #### `tsdb` Backend `write` Parameters
 
-The following `write` parameter descriptions are specific to the `tsdb` backend; for more information and examples, see the platform's [Frames TSDB-backend reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/tsdb/write/):
+The following `write` parameter descriptions are specific to the `tsdb` backend:
 
 - <a id="method-write-tsdb-param-labels"></a>**labels** &mdash; A dictionary of metric labels of the format `{<label>: <value>[, <label>: <value>, ...]}` to apply to all the DataFrame rows.
   For example, `{"os": "linux", "arch": "x86"}`.
@@ -522,7 +515,7 @@ All Frames backends that support the `read` method support the following common
 <a id="method-read-params-nosql"></a>
 #### `nosql` Backend `read` Parameters
 
-The following `read` parameters are specific to the `nosql` backend; for more information and examples, see the platform's [Frames NoSQL-backend reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/nosql/read/):
+The following `read` parameters are specific to the `nosql` backend:
 
 - <a id="method-read-nosql-param-max_rows_in_msg"></a>**max_rows_in_msg** &mdash; The maximum number of rows per message.
 
@@ -547,7 +540,7 @@ The following parameters are passed as keyword arguments via the `kw` parameter:
 <a id="method-read-params-tsdb"></a>
 #### `tsdb` Backend `read` Parameters
 
-The following `read` parameters are specific to the `tsdb` backend; for more information and examples, see the platform's [Frames TSDB-backend reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/tsdb/read/):
+The following `read` parameters are specific to the `tsdb` backend:
 
 - <a id="method-read-tsdb-param-group_by"></a>**group_by** **[Tech Preview]** &mdash; A group-by query string.
   <br/>
@@ -615,8 +608,7 @@ The following parameters are passed as keyword arguments via the `kw` parameter:
 <a id="method-read-params-stream"></a>
 #### `stream` Backend `read` Parameters
 
-The following `read` parameters are specific to the `stream` backend and are passed as keyword arguments via the `kw` parameter<!--; for more information and examples, see the platform's [Frames streaming-backend reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/stream/read/)-->:
-<!-- [FRAMES-STREAMING-NO-SUPPORT] -->
+The following `read` parameters are specific to the `stream` backend and are passed as keyword arguments via the `kw` parameter:
 
 - <a id="method-read-stream-param-seek"></a>**seek** &mdash; Seek type.
   <br/>
@@ -722,7 +714,7 @@ delete(backend, table, filter='', start='', end='', if_missing=FAIL
 <a id="method-delete-params-nosql"></a>
 #### `nosql` Backend `delete` Parameters
 
-The following `delete` parameters are specific to the `nosql` backend; for more information and examples, see the platform's [Frames NoSQL-backend reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/nosql/delete/):
+The following `delete` parameters are specific to the `nosql` backend:
 
 - <a id="method-delete-nosql-param-filter"></a>**filter** &mdash; A filter expression that identifies specific items to delete.
 
@@ -733,7 +725,7 @@ The following `delete` parameters are specific to the `nosql` backend; for more
 <a id="method-delete-params-tsdb"></a>
 #### `tsdb` Backend `delete` Parameters
 
-The following `delete` parameters are specific to the `tsdb` backend; for more information and examples, see the platform's [Frames TSDB-backend reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/tsdb/delete/):
+The following `delete` parameters are specific to the `tsdb` backend:
 
 - <a id="method-delete-tsdb-param-start"></a>**start** &mdash; Start (minimum) time for the delete operation &mdash; i.e., delete only items whose data sample time is at or after (`>=`) the specified start time.
 
@@ -822,7 +814,7 @@ All Frames backends that support the `execute` method support the following comm
 <a id="method-execute-nosql-cmds"></a>
 #### `nosql` Backend `execute` Commands
 
-The following `execute` commands are specific to the `nosql` backend; for more information and examples, see the platform's [Frames NoSQL-backend reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/nosql/execute/):
+The following `execute` commands are specific to the `nosql` backend:
 
 - <a id="method-execute-nosql-cmd-infer"></a>**infer | infer_schema** &mdash; Infers the data schema of a given NoSQL table and creates a schema file for the table.
 
@@ -845,8 +837,7 @@ The following `execute` commands are specific to the `nosql` backend; for more i
 <a id="method-execute-stream-cmds"></a>
 #### `stream` Backend `execute` Commands
 
-The following `execute` commands are specific to the `stream` backend<!--; for more information and examples, see the platform's [Frames streaming-backend reference](https://www.iguazio.com/docs/latest-release/data-layer/reference/frames/stream/execute/)-->:
-<!-- [FRAMES-STREAMING-NO-SUPPORT] -->
+The following `execute` commands are specific to the `stream` backend:
 
 - <a id="method-execute-stream-cmd-put"></a>**put** &mdash; Adds records to a stream shard.
 

clients/py/.travis.yml

@@ -245,12 +245,15 @@ def _check_version(self):
         stub = fgrpc.FramesStub(self._channel)
         request = fpb.VersionRequest()
         resp = stub.Version(request)
+        server_version = resp.version
+        if server_version.startswith("v"):
+            server_version = server_version[1:]
         if resp.version:
-            client_version = __version__[:__version__.rfind(".")]
-            server_version = resp.version[:resp.version.rfind(".")]
-            if client_version != server_version:
+            client_major_minor_version = __version__[:__version__.rfind(".")]
+            server_major_minor__version = server_version[:server_version.rfind(".")]
+            if client_major_minor_version != server_major_minor__version:
                 warnings.warn(
-                    f"Warning - Server version '{resp.version}' differs in major/minor version from client "
+                    f"Warning - Server version '{server_version}' differs in major/minor version from client "
                     f"version '{__version__}'. Some operations may not work as expected."
                 )
         else:

clients/py/v3io_frames/grpc.py

@@ -247,13 +247,20 @@ def _check_version(self):
         except json.JSONDecodeError as err:
             raise VersionError(str(err))
 
-        version = out.get('version')
-        if not version:
+        server_version = out.get('version')
+        if server_version.startswith("v"):
+            server_version = server_version[1:]
+        if not server_version:
             warnings.warn("Warning - Cannot resolve server version. Make sure client version is compatible.")
             return
 
-        if __version__ != version:
-            warnings.warn("Warning - Server version \'" + version + "\' is different from client version \'" + __version__ + "\'. Some operations may not work as expected.")
+        client_major_minor_version = __version__[:__version__.rfind(".")]
+        server_major_minor__version = server_version[:server_version.rfind(".")]
+        if client_major_minor_version != server_major_minor__version:
+            warnings.warn(
+                f"Warning - Server version '{server_version}' differs in major/minor version from client "
+                f"version '{__version__}'. Some operations may not work as expected."
+            )
 
     def _url_for(self, action):
         return self.address + '/' + action

clients/py/v3io_frames/http.py

@@ -98,7 +98,7 @@ func (csvSuite *CsvTestSuite) TestAll() {
 	err = appender.Add(frame)
 	csvSuite.Require().NoError(err)
 
-	err = appender.WaitForComplete(3 * time.Second)
+	err = appender.WaitForComplete(10 * time.Second)
 	csvSuite.Require().NoError(err)
 
 	time.Sleep(3 * time.Second) // Let DB sync