You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"- [Getting Data from AWS S3 Using curl](data-ingest-s3-curl)\n",
39
39
"- [Running Distributed Python with Dask](#data-ingest-dask)\n",
40
40
"- [Running DataFrames on GPUs using NVIDIA cuDF](#data-ingest-gpu)\n",
41
-
"- [Creating Dashboards with Grafana](#data-grafana)"
41
+
"- [Visualizing Data with Grafana](#data-ingest-grafana)"
42
42
]
43
43
},
44
44
{
@@ -55,7 +55,8 @@
55
55
"\n",
56
56
"This tutorial provides an overview of various methods for collecting, storing, and manipulating data in the platform, and refers to sample tutorial notebooks that demonstrate how to use these methods.<br>\n",
57
57
"For an in-depth overview of the platform and how it can be used to implement a full data science workflow, see the [**platform-overview**](../platform-overview.ipynb) tutorial notebook.\n",
58
-
"For full end-to-end platform use-case application demos, see the [**welcome notebook**](../welcome.ipynb#end-to-end-use-case-applications)\n",
58
+
"<br>\n",
59
+
"For information about the available full end-to-end platform use-case application demos, see the [**welcome**](../welcome.ipynb#end-to-end-use-case-applications) notebook or the matching [**README.md**](../README.md#end-to-end-use-case-applications) file.\n",
"The platform features an extremely fast and secure data layer that supports SQL, NoSQL, time-series databases, files (simple objects), and streaming, and exposes multiple APIs for working with the different data types — including [simple-object](#data-ingest-platform-data-object-api), [NoSQL (\"key-value\")](#data-ingest-platform-nosql-api), and [streaming](#data-ingest-platform-streaming-api) APIs."
136
139
]
137
140
},
138
141
{
139
142
"cell_type": "markdown",
140
143
"metadata": {},
141
144
"source": [
142
-
"### Iguazio's Data-Object API"
145
+
"### The Data-Object Platform API"
143
146
]
144
147
},
145
148
{
146
149
"cell_type": "markdown",
147
150
"metadata": {},
148
151
"source": [
149
-
"The platform’s Simple-Object API enables performing simple data-object operations that resembles Amazon’s Simple Storage Service (S3) API. In addition to the S3-like capabilities, the Simple-Object Web API enables appending data to existing objects. The [**v3io-objects notebook**](v3io-objects.ipynb) demonstrates the API usage."
152
+
"The platform’s Simple-Object API enables performing simple data-object and container operations that resemble the Amazon Web Services (AWS) Simple Storage Service (S3) API.\n",
153
+
"In addition to the S3-like capabilities, the Simple-Object Web API enables appending data to existing objects.\n",
154
+
"For more information and API usage examples, see the [**v3io-objects**](v3io-objects.ipynb) tutorial."
150
155
]
151
156
},
152
157
{
153
158
"cell_type": "markdown",
154
159
"metadata": {},
155
160
"source": [
156
-
"<a id=\"data-ingest-iguazio-kv-api\"></a>"
161
+
"<a id=\"data-ingest-platform-nosql-api\"></a>"
157
162
]
158
163
},
159
164
{
160
165
"cell_type": "markdown",
161
166
"metadata": {},
162
167
"source": [
163
-
"### Iguazio's Key-Value API"
168
+
"### The NoSQL (Key-Value) Platform API"
164
169
]
165
170
},
166
171
{
167
172
"cell_type": "markdown",
168
173
"metadata": {},
169
174
"source": [
170
-
"The platform’s Key-Value (KV) API provides access to the NoSQL database service, which enables storing and consuming data in a tabular format. The [**v3io-kv notebook**](v3io-kv.ipynb) demonstrates the API usage."
175
+
"The platform’s NoSQL (a.k.a. Key-Value/KV) API provides access to the platform's NoSQL data store (database service), which enables storing and consuming data in a tabular format.\n",
176
+
"For more information and API usage examples, see the [**v3io-kv**](v3io-kv.ipynb) tutorial."
"The platform’s Streaming API enables working with data in the platform as streams. The [**v3io-streams notebook**](v3io-streams.ipynb) demonstrates the API usage.\n",
192
-
"\n",
193
-
"For more general information see [working with Streams](#data-ingest-streams) section."
197
+
"The platform’s Streaming API enables working with data in the platform as streams.\n",
198
+
"For more information and API usage examples, see the [**v3io-streams**](v3io-streams.ipynb) tutorial.\n",
199
+
"In addition, see the [Working with Streams](#data-ingest-streams) section in the current tutorial for general information about different methods for working with data streams in the platform."
194
200
]
195
201
},
196
202
{
@@ -346,9 +352,11 @@
346
352
"The platform's streaming engine is currently available via the platform's [Streaming Web API](https://www.iguazio.com/docs/latest-release/reference/api-reference/web-apis/streaming-web-api/).<br>\n",
347
353
"In addition, the platform's Spark-Streaming Integration API enables using the Spark Streaming API to work with platform streams, as explained in the next section ([Using Spark Streaming](#data-ingest-streams-spark)).\n",
348
354
"\n",
349
-
"The [**v3io-streams notebook**](v3io-streams.ipynb) demonstrates basic usage of the streaming API.\n",
355
+
"The [**v3io-streams**](v3io-streams.ipynb) tutorial demonstrates basic usage of the streaming API.\n",
350
356
"\n",
351
-
"The [**model meployment with streaming demo**](https://github.com/mlrun/demo-model-deployment-with-streaming) demo application includes an example of a Nuclio function that uses platform streams."
357
+
"<!-- [IntInfo] The referenced demo deson't exist.\n",
358
+
"The [**model deployment with streaming**](https://github.com/mlrun/demo-model-deployment-with-streaming) demo application includes an example of a Nuclio function that uses platform streams.\n",
359
+
"-->"
352
360
]
353
361
},
354
362
{
@@ -556,20 +564,26 @@
556
564
]
557
565
},
558
566
{
559
-
"source": [
560
-
"<a id=\"data-grafana\"></a>"
561
-
],
562
567
"cell_type": "markdown",
563
-
"metadata": {}
568
+
"metadata": {},
569
+
"source": [
570
+
"<a id=\"data-ingest-grafana\"></a>"
571
+
]
564
572
},
565
573
{
574
+
"cell_type": "markdown",
575
+
"metadata": {},
566
576
"source": [
567
-
"## Creating Dashboards with Grafana\n",
577
+
"## Visualizing Data with Grafana\n",
568
578
"\n",
569
-
"You can create a Grafana dashboard programmatically using the Iguazio API. This allows you to define a dashboard that reads from the Iguazio data layer and display tables and charts of this data. The [**grafana-grafwiz example**](grafana-grafwiz.ipynb) demonstrates how to do that."
570
-
],
571
-
"cell_type": "markdown",
572
-
"metadata": {}
579
+
"The platform has a Grafana service with predefined dashboards that leverage the monitoring service to display monitoring data, such as performance statistics, for application services.\n",
580
+
"You can also define custom Grafana dashboards for monitoring, visualizing, and understanding data stored in the platform, such as time-series metrics and NoSQL data.\n",
581
+
"You can read and analyze data from the platform's data store and visualize it on Grafana dashboards in the desired formats, such as tables and graphs.\n",
582
+
"This can be done by using the custom `iguazio` data source, or by using a Prometheus data source for running Prometheus queries on platform TSDB tables.\n",
583
+
"You can also issue data alerts and create, explore, and share dashboards.\n",
584
+
"\n",
585
+
"You can use Iguazio's [grafwiz](https://github.com/v3io/grafwiz) Python library to create an deploy Grafana dashboards programmatically, as demonstrated in the [**grafana-grafwiz**](grafana-grafwiz.ipynb) tutorial."
-[Reading from External Database](#data-ingest-external-dbs)
13
13
-[Using Spark over JDBC](#data-ingest-spark-over-jdbs)
14
14
-[Using SQLAlchemy](#data-ingest-sqlalchemy)
@@ -27,7 +27,7 @@ Learn about different methods for ingesting data into the Iguazio Data Science P
27
27
-[Getting Data from AWS S3 Using curl](data-ingest-s3-curl)
28
28
-[Running Distributed Python with Dask](#data-ingest-dask)
29
29
-[Running DataFrames on GPUs using NVIDIA cuDF](#data-ingest-gpu)
30
-
-[Creating Dashboards with Grafana](#data-grafana)
30
+
-[Visualizing Data with Grafana](#data-ingest-grafana)
31
31
32
32
<aid="data-ingest-overview"></a>
33
33
## Overview
@@ -39,7 +39,8 @@ The platform features a wide set of methods for manipulating and managing data,
39
39
40
40
This tutorial provides an overview of various methods for collecting, storing, and manipulating data in the platform, and refers to sample tutorial notebooks that demonstrate how to use these methods.<br>
41
41
For an in-depth overview of the platform and how it can be used to implement a full data science workflow, see the [**platform-overview**](../platform-overview.ipynb) tutorial notebook.
42
-
For full end-to-end platform use-case application demos, see the [**welcome notebook**](../welcome.ipynb#end-to-end-use-case-applications)
42
+
<br>
43
+
For information about the available full end-to-end platform use-case application demos, see the [**welcome**](../welcome.ipynb#end-to-end-use-case-applications) notebook or the matching [**README.md**](../README.md#end-to-end-use-case-applications) file.
@@ -77,29 +78,34 @@ For detailed information and examples on how to set the data path for each inter
77
78
The [**basic-data-ingestion-and-preparation**](basic-data-ingestion-and-preparation.ipynb) tutorial walks you through basic scenarios of ingesting data from external sources into the platform's data store and manipulating the data using different data formats.
78
79
The tutorial includes an example of ingesting a CSV file from an AWS S3 bucket; converting it into a NoSQL table using Spark DataFrames; running SQL queries on the table; and converting the table into a Parquet file.
The platform features an extremely fast and secure data layer that supports SQL, NoSQL, time-series databases, files (simple objects), and streaming, and exposes multiple APIs for working with the different data types — including [simple-object](#data-ingest-platform-data-object-api), [NoSQL ("key-value")](#data-ingest-platform-nosql-api), and [streaming](#data-ingest-platform-streaming-api) APIs.
87
88
88
-
The platform’s Simple-Object API enables performing simple data-object operations that resembles Amazon’s Simple Storage Service (S3) API. In addition to the S3-like capabilities, the Simple-Object Web API enables appending data to existing objects. The [**v3io-objects notebook**](v3io-objects.ipynb) demonstrates the API usage.
89
+
### The Data-Object Platform API
89
90
90
-
<aid="data-ingest-iguazio-kv-api"></a>
91
+
The platform’s Simple-Object API enables performing simple data-object and container operations that resemble the Amazon Web Services (AWS) Simple Storage Service (S3) API.
92
+
In addition to the S3-like capabilities, the Simple-Object Web API enables appending data to existing objects.
93
+
For more information and API usage examples, see the [**v3io-objects**](v3io-objects.ipynb) tutorial.
91
94
92
-
### Iguazio's Key-Value API
95
+
<aid="data-ingest-platform-nosql-api"></a>
93
96
94
-
The platform’s Key-Value (KV) API provides access to the NoSQL database service, which enables storing and consuming data in a tabular format. The [**v3io-kv notebook**](v3io-kv.ipynb) demonstrates the API usage.
97
+
### The NoSQL (Key-Value) Platform API
95
98
96
-
<aid="data-ingest-iguazio-streaming-api"></a>
99
+
The platform’s NoSQL (a.k.a. Key-Value/KV) API provides access to the platform's NoSQL data store (database service), which enables storing and consuming data in a tabular format.
100
+
For more information and API usage examples, see the [**v3io-kv**](v3io-kv.ipynb) tutorial.
97
101
98
-
### Iguazio's Streaming API
102
+
<aid="data-ingest-platform-streaming-api"></a>
99
103
100
-
The platform’s Streaming API enables working with data in the platform as streams. The [**v3io-streams notebook**](v3io-streams.ipynb) demonstrates the API usage.
104
+
### The Streaming Platform API
101
105
102
-
For more general information see [working with Streams](#data-ingest-streams) section.
106
+
The platform’s Streaming API enables working with data in the platform as streams.
107
+
For more information and API usage examples, see the [**v3io-streams**](v3io-streams.ipynb) tutorial.
108
+
In addition, see the [Working with Streams](#data-ingest-streams) section in the current tutorial for general information about different methods for working with data streams in the platform.
103
109
104
110
<aid="data-ingest-external-dbs"></a>
105
111
## Reading Data from External Databases
@@ -180,9 +186,11 @@ You can use the platform's streaming engine to write data into a queue in a real
180
186
The platform's streaming engine is currently available via the platform's [Streaming Web API](https://www.iguazio.com/docs/latest-release/reference/api-reference/web-apis/streaming-web-api/).<br>
181
187
In addition, the platform's Spark-Streaming Integration API enables using the Spark Streaming API to work with platform streams, as explained in the next section ([Using Spark Streaming](#data-ingest-streams-spark)).
182
188
183
-
The [**v3io-streams notebook**](v3io-streams.ipynb) demonstrates basic usage of the streaming API.
189
+
The [**v3io-streams**](v3io-streams.ipynb) tutorial demonstrates basic usage of the streaming API.
184
190
185
-
The [**model meployment with streaming demo**](https://github.com/mlrun/demo-model-deployment-with-streaming) demo application includes an example of a Nuclio function that uses platform streams.
191
+
<!-- [IntInfo] The referenced demo deson't exist.
192
+
The [**model deployment with streaming**](https://github.com/mlrun/demo-model-deployment-with-streaming) demo application includes an example of a Nuclio function that uses platform streams.
193
+
-->
186
194
187
195
<aid="data-ingest-streams-spark"></a>
188
196
@@ -284,8 +292,14 @@ The [**gpu-cudf-vs-pd**](gpu-cudf-vs-pd.ipynb) tutorial demonstrates how to use
284
292
> **Note:** To use the cuDF library, you need to create a RAPIDS Conda environment.
285
293
> For more information, see the [**virtual-env**](../virtual-env.ipynb) tutorial.
286
294
287
-
<aid="data-grafana"></a>
295
+
<aid="data-ingest-grafana"></a>
288
296
289
-
## Creating Dashboards with Grafana
297
+
## Visualizing Data with Grafana
290
298
291
-
You can create a Grafana dashboard programmatically using the Iguazio API. This allows you to define a dashboard that reads from the Iguazio data layer and display tables and charts of this data. The [**grafana-grafwiz example**](grafana-grafwiz.ipynb) demonstrates how to do that.
299
+
The platform has a Grafana service with predefined dashboards that leverage the monitoring service to display monitoring data, such as performance statistics, for application services.
300
+
You can also define custom Grafana dashboards for monitoring, visualizing, and understanding data stored in the platform, such as time-series metrics and NoSQL data.
301
+
You can read and analyze data from the platform's data store and visualize it on Grafana dashboards in the desired formats, such as tables and graphs.
302
+
This can be done by using the custom `iguazio` data source, or by using a Prometheus data source for running Prometheus queries on platform TSDB tables.
303
+
You can also issue data alerts and create, explore, and share dashboards.
304
+
305
+
You can use Iguazio's [grafwiz](https://github.com/v3io/grafwiz) Python library to create an deploy Grafana dashboards programmatically, as demonstrated in the [**grafana-grafwiz**](grafana-grafwiz.ipynb) tutorial.
0 commit comments