Merge pull request #347 from Sharon-iguazio/development

[IG-17510] [SITE-RESTRUCT]  Fix v3io-streams NB edits & add doc fixes and improvements + minor v3io-kv & v3io-objects NB doc fix (PR #345 fix)

Author: Sharon-iguazio

data-ingestion-and-preparation/v3io-kv.ipynb

@@ -81,7 +81,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Set path"
+    "### Set the Data Path"
    ]
   },
   {

data-ingestion-and-preparation/v3io-objects.ipynb

@@ -91,7 +91,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Set path"
+    "### Set the Data Path"
    ]
   },
   {

data-ingestion-and-preparation/v3io-streams.ipynb

@@ -11,8 +11,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The platform's Streaming API enables working with data in the platform as streams.\n"
-    "For more information, see the platform;s [streams](https://www.iguazio.com/docs/v3.0/data-layer/stream/) documentation."
+    "The platform's Streaming API enables working with data in the platform as streams.\n",
+    "    For more information, see the platform's [streams](https://www.iguazio.com/docs/v3.0/data-layer/stream/) documentation."
    ]
   },
   {
@@ -35,7 +35,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Create a dataplane client"
+    "Create a `dataplane` client:"
    ]
   },
   {
@@ -51,41 +51,46 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "> **Note**: You can pass to the client the `endpoint` and `access_key` parameters explicitly. The following code is equivalent to the default values:\n",
+    "> **Note**: You can pass to the client the `endpoint` and `access_key` parameters explicitly.\n",
+    "> The following code is equivalent to the default values:\n",
     ">\n",
-    ">``` python\n",
-    ">from os import getenv\n",
-    ">v3io_client = v3io.dataplane.Client(endpoint='http://v3io-webapi:8081',\n",
-    ">                                    access_key=getenv('V3IO_ACCESS_KEY'))\n",
-    ">```\n",
+    "> ``` python\n",
+    "> from os import getenv\n",
+    "> v3io_client = v3io.dataplane.Client(endpoint='http://v3io-webapi:8081',\n",
+    ">                                     access_key=getenv('V3IO_ACCESS_KEY'))\n",
+    "> ```\n",
     ">\n",
-    "> When calling externally, you can obtain the URL of your cluster by copying the API URL of the web-APIs service (webapi) from the **Services** dashboard page. You can select between two types of URLs:\n",
-    "- **HTTPS Direct** (recommended) &mdash; a URL of the format `https://<tenant IP>:<web-APIs port>`; for example, `https://default-tenant.app.mycluster.iguazio.com:8443`.\n",
-    "- **HTTPS** &mdash; a URL of the format `https://webapi.<tenant IP>`; for example, `https://webapi.default-tenant.app.mycluster.iguazio.com`.\n",
+    "> When running the code remotely, you can obtain the URL of your cluster by copying the API URL of the web-APIs service (`webapi`) from the **Services** dashboard page. You can select between two types of URLs:\n",
     ">\n",
-    "> For more information, see the [Data-Service Web-API General Structure](https://www.iguazio.com/docs/v3.0/data-layer/reference/web-apis/data-service-web-api-gen-struct/) documentation."
+    "> - **HTTPS Direct** (recommended) &mdash; a URL of the format `https://<tenant IP>:<web-APIs port>`; for example, `https://default-tenant.app.mycluster.iguazio.com:8443`.\n",
+    "> - **HTTPS** &mdash; a URL of the format `https://webapi.<tenant IP>`; for example, `https://webapi.default-tenant.app.mycluster.iguazio.com`.\n",
+    ">\n",
+    "> For more information see the [Data-Service Web-API General Structure](https://www.iguazio.com/docs/v3.0/data-layer/reference/web-apis/data-service-web-api-gen-struct/) documentation."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "> **Number of maximum parallel connections**: Another noteworthy parameter is `max_connections`, which defines the number of maximum parallel connections when performing batch operations. If left unspecified, the default is 8 connections. For more information see the [Put Multiple Objects](#Put-Multiple-Objects) section in this notebook."
+    "> **Number of maximum parallel connections**: Another noteworthy parameter is `max_connections`, which defines the number of maximum parallel connections when performing batch operations.\n",
+    "> If left unspecified, the default is 8 connections.\n",
+    "> For more information see the [Put Multiple Records](#Put-Multiple-Records) section in this tutorial."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Set path"
+    "### Set the Data Path"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "All data in the platform is stored in user-defined data containers. In this case we use the predefined \"users\" container.\n"
-    "For more information, see the platform's [Data Containers](https://www.iguazio.com/docs/v3.0/data-layer/containers/) documentation."
+    "All data in the platform is stored in user-defined data containers.\n",
+    "This tutorial uses the predefined \"users\" container.\n",
+    "For more information refer to the platform's [data-containers](https://www.iguazio.com/docs/v3.0/data-layer/containers/) documentation."
    ]
   },
   {
@@ -101,7 +106,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Data path where to store the kv table"
+    "Set the data path for storing the stream:"
    ]
   },
   {
@@ -165,14 +170,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Adds records to a stream."
+    "Use the `put` method to add records to a stream."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We'll define a function that will convert our text to lowercase words"
+    "The following example defines a function that converts text to lowercase words:"
    ]
   },
   {
@@ -205,7 +210,7 @@
     "    \n",
     "    return words\n",
     "\n",
-    "text1 = \"WOLF, meeting with a Lamb astray from the fold, resolved not to lay violent hands on him, but to find some plea to justify to the Lamb the Wolf's right to eat him. He thus addressed him: “Sirrah, last year you grossly insulted me.” “Indeed,” bleated the Lamb in a mournful tone of voice, “I was not then born.” Then said the Wolf, “You feed in my pasture.” “No, good sir,” replied the Lamb, “I have not yet tasted grass.” Again said the Wolf, “You drink of my well.” “No,” exclaimed the Lamb, “I never yet drank water, for as yet my mother's milk is both food and drink to me.” Upon which the Wolf seized him and ate him up, saying, “Well! I won't remain supperless, even though you refute every one of my imputations.” The tyrant will always find a pretext for his tyranny.\"\n",
+    "text1 = \"WOLF, meeting with a Lamb astray from the fold, resolved not to lay violent hands on him, but to find some plea to justify to the Lamb the Wolf’s right to eat him. He thus addressed him: “Sirrah, last year you grossly insulted me.” “Indeed,” bleated the Lamb in a mournful tone of voice, “I was not then born.” Then said the Wolf, “You feed in my pasture.” “No, good sir,” replied the Lamb, “I have not yet tasted grass.” Again said the Wolf, “You drink of my well.” “No,” exclaimed the Lamb, “I never yet drank water, for as yet my mother’s milk is both food and drink to me.” Upon which the Wolf seized him and ate him up, saying, “Well! I won’t remain supperless, even though you refute every one of my imputations.” The tyrant will always find a pretext for his tyranny.\"\n",
     "words1 = text_to_words(text1)\n",
     "len(words1)"
    ]
@@ -214,8 +219,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Convert the list of words to a record. A record is a list of dictionaries, where for each dictionary the `data` key contains the record data.\n",
-    "We display the first 5 records:"
+    "The following code converts the list of words to a record.\n",
+    "A record is a list of dictionaries, where for each dictionary the `data` key contains the record data.\n",
+    "The sample code displays the first 5 records:"
    ]
   },
   {
@@ -260,7 +266,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Write another set of records to the stream"
+    "The following code writes another set of records to the stream:"
    ]
   },
   {
@@ -286,9 +292,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Multiple consumer instances can consume data from the same stream. A consumer retrieves records from a specific shard. It is recommended that you distribute the data consumption among several consumer instances (“workers”), assigning each instance one or more shards.\n",
+    "Multiple consumer instances can consume data from the same stream.\n",
+    "A consumer retrieves records from a specific shard.\n",
+    "It's recommended that you distribute the data consumption among several consumer instances (\"workers\"), assigning each instance one or more shards.\n",
     "\n",
-    "For each shard, the consumer should determine the location within the shard from which to begin consuming records. This can be the earliest ingested record, the end of the shard, the first ingested record starting at a specific time, or a specific record identified by its sequence number (a unique record identifier that is assigned by the platform based on the record's ingestion time). The consumer first uses a seek operation to obtain the desired consumption location, and then provides this location as the starting point for its record consumption. The consumption operation returns the location of the next record to consume within the shard, and this location should be used as the location for a subsequent consumption operation, allowing for sequential record consumption."
+    "For each shard, the consumer should determine the location within the shard from which to begin consuming records.\n",
+    "This can be the earliest ingested record, the end of the shard, the first ingested record starting at a specific time, or a specific record identified by its sequence number (a unique record identifier that is assigned by the platform based on the record’s ingestion time).\n",
+    "The consumer first uses a seek operation to obtain the desired consumption location, and then provides this location as the starting point for its record consumption.\n",
+    "The consumption operation returns the location of the next record to consume within the shard, and this location should be used as the location for a subsequent consumption operation, allowing for sequential record consumption."
    ]
   },
   {
@@ -309,7 +320,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We will read from the stream, by default `get_records` limits the number of records returned to 1,000. For the sake of this demonstration we will limit the number of returned records to 10 by setting the `limit` parameter."
+    "Use the `get_records` method to read from the stream (retrieve records).\n",
+    "By default `get_records` limits the number of records returned to 1,000.\n",
+    "For the sake of this demonstration, the sample code limits the number of returned records to 10 by setting the `limit` parameter."
    ]
   },
   {
@@ -371,7 +384,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We can retrieve a stream's configuration, including the shard count and retention period."
+    "Use the `describe` method to retrieve a stream's configuration, including the shard count and retention period."
    ]
   },
   {
@@ -422,9 +435,12 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Updates a stream's configuration by increasing its shard count. The changes are applied immediately.\n",
+    "Use the `update` method to updates a stream's configuration by increasing its shard count.\n",
+    "The changes are applied immediately.\n",
     "\n",
-    "> **Note**: You can increase the shard count at any time, but you cannot reduce it. From the platform's perspective, there is virtually no cost to creating even thousands of shards. However, if you increase a stream's shard count after its creation, new records with a previously used partition key will be assigned either to the same shard that was previously used for this partition key or to a new shard.\n"
+    "> **Note**: You can increase the shard count at any time, but you cannot reduce it.\n",
+    "> From the platform's perspective, there's virtually no cost to creating even thousands of shards.\n",
+    "> However, if you increase a stream's shard count after its creation, new records with a previously used partition key will be assigned either to the same shard that was previously used for this partition key or to a new shard.\n",
     "> For more information see the platform's [stream sharding and partitioning](https://www.iguazio.com/docs/v3.0/data-layer/stream/#stream-sharding-and-partitioning) documentation.\n",
     "\n",
     "> **Spark-Streaming Note**: To use the Spark Streaming API to consume records from new shards after a shard-count increase, you must first restart the consumer application."
@@ -454,7 +470,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Describe the stream again to see the updated shard count"
+    "Describe the stream again to see the updated shard count:"
    ]
   },
   {
@@ -487,8 +503,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "In the prior section when we get the records, we didn't get all words. The reason is that, by default, the platform assigns records to shards using a Round Robin algorithm, and our consumer code reads from a single shard. If you would like to ensure a consumer gets all the words, you can optionally assign a record to specific stream shard by specifying a related shard ID by setting the `shard_id` value, or associate the record with a specific partition key to ensure that similar records are assigned to the same shard (by setting the `partition_key` value).\n"
-    "For more information, see the platform's [stream sharding and partitioning](https://www.iguazio.com/docs/v3.0/data-layer/stream/#stream-sharding-and-partitioning) documentation."
+    "In the previous section, when you retrieved the stream records you didn't get all words.\n",
+    "The reason is that by default, the platform assigns records to shards using a Round Robin algorithm, and the sample consumer code reads from a single shard.\n",
+    "If you would like to ensure that a consumer gets all the words, you can optionally assign a record to specific stream shard by specifying a related shard ID by setting the `shard_id` value, or associate the record with a specific partition key to ensure that similar records are assigned to the same shard (by setting the `partition_key` value).\n",
+    "For more information see the platform's [stream sharding and partitioning](https://www.iguazio.com/docs/v3.0/data-layer/stream/#stream-sharding-and-partitioning) documentation."
    ]
   },
   {
@@ -513,7 +531,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Write the first text to shard 10"
+    "Write the first text to shard 10:"
    ]
   },
   {
@@ -531,7 +549,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "And write the second text to shard 11"
+    "And write the second text to shard 11:"
    ]
   },
   {
@@ -549,7 +567,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now, read from shard 10"
+    "Now, read from shard 10:"
    ]
   },
   {
@@ -611,14 +629,19 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "`put_records` accepts up to a maximum of 1,000 records. If the records limit is exceeded, the request fails. Therefore, we would need to call `put_records` multiple times. In this example, we'll send large number of events. We will create a generator that returns a list of events. Each event has the following record:\n",
-    "\n",
+    "`put_records` accepts up to a maximum of 1,000 records.\n",
+    "If the records limit is exceeded, the request fails.\n",
+    "Therefore, you need to call `put_records` multiple times.\n",
+    "The following example sends a large number of events.\n",
+    "It creates a generator that returns a list of events.\n",
+    "Each event has the following record:\n",
     "``` python\n",
     "'user': <user_id>\n",
     "'time': <event_time>\n",
     "'url': <url>\n",
     "```\n",
-    "Each user will be selected at random, the URL will also be a random string and the generated events will have their event_time monotonically increasing"
+    "Each user is selected at random.\n",
+    "The URL is also be a random string, and the `event_time` of the generated events is monotonically increased."
    ]
   },
   {
@@ -663,7 +686,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Test the generator, print just a few events"
+    "Test the generator by printing a few events:"
    ]
   },
   {
@@ -695,7 +718,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Seek the latest position in one of the shards, in order to later retrieve the new data"
+    "Seek the latest position in one of the shards, in order to later retrieve the new data:"
    ]
   },
   {
@@ -742,16 +765,19 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The looped `put_records` interface above will send all `put records` requests to the data layer in parallel. When `wait` is called, it will block until either all responses arrive (in which case it will return a `Responses` object, containing the `responses` of each call) or an error occurs - in which case an exception is thrown. You can pass `raise_for_status` to `wait`, and it behaves as explained above.\n",
+    "The looped `put_records` interface in the previous code block sends all `put_records` requests to the data layer in parallel.\n",
+    "When `wait` is called, it blocks until either all responses arrive &mdash; in which case it returns a `Responses` object that contains the `responses` of each call &mdash; or an error occurs &mdash; in which case an exception is thrown.\n",
+    "You can pass `raise_for_status` to `wait`, and it behaves as previously explained.\n",
     "\n",
-    "> Note: The `batch` object is stateful, so you can only create one batch at a time. However, you can create multiple parallel batches yourself through the client's `create_batch()` interface"
+    "> **Note:** The `batch` object is stateful, therefore you can only create one batch at a time.\n",
+    "> However, you can create multiple parallel batches yourself through the client's `create_batch` interface."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Test the received record from the previously obtained location"
+    "Test the received record from the previously obtained location:"
    ]
   },
   {
@@ -815,18 +841,34 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Alternatively you can use the following commands:\n",
-    "``` python\n",
+    "Alternatively, you can use the following command:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "import shutil\n",
     "V3IO_STREAM_PATH = path.join(sep, 'v3io', CONTAINER, STREAM_PATH)\n",
-    "shutil.rmtree(V3IO_STREAM_PATH)\n",
-    "```\n",
-    "\n",
-    "or\n",
-    "\n",
-    "```\n",
-    "!rm -r $V3IO_STREAM_PATH\n",
-    "```"
+    "shutil.rmtree(V3IO_STREAM_PATH)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Or use this command:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!rm -r $V3IO_STREAM_PATH"
    ]
   }
  ],