diff --git a/docs/docusaurus/docs/reference/learn/core_to_cloud.md b/docs/docusaurus/docs/reference/learn/core_to_cloud.md new file mode 100644 index 000000000000..7934865ffcac --- /dev/null +++ b/docs/docusaurus/docs/reference/learn/core_to_cloud.md @@ -0,0 +1,99 @@ +--- +id: core_to_cloud +sidebar_label: 'GX Core to GX Cloud Migration Guide' +title: "GX Core to GX Cloud Migration Guide" +--- + +## Overview + +This guide will enable you to migrate your GX Core configuration to a GX Cloud organization. Since GX Cloud is built on top of GX Core, the code you used to originally set up your GX Core configuration can be reused for setting up your GX Cloud organization. + +The key difference between using GX Core and GX Cloud is the Data Context. By setting the mode of your Data Context to `cloud` and then providing the appropriate credentials, you will be able to connect to your GX Cloud organization. Once you have created a Cloud Data Context, the rest of the code you have already written to configure your GX entities, such as Data Sources, Data Assets, and Expectations, can be re-run to migrate your existing configuration into GX Cloud. Similarly, any code that you have written to run validations, including Custom Actions, can also be reused. + +## Examples + +### Configuration Setup +In the example below, a File Data Context has been created, along with a Postgres Data Source and Data Asset. + +```python +import great_expectations as gx + +context = gx.get_context(mode="file") +ds = context.data_sources.add_sql(name="Postgres DB", connection_string="postgresql+psycopg2://username:passowrd@myhost.domain>:443>/sample_db") +asset = ds.add_table_asset(table_name="sample_table", name="sample_table") + +bd = asset.add_batch_definition_whole_table( + name="FULL_TABLE" +) + +suite = context.suites.get("my_suite") + +validation_definition = gx.ValidationDefinition( + data=bd, suite=suite, name="Validation Definition" +) +context.validation_definitions.add(validation_definition) +``` + +In order to recreate these same entities in GX Cloud, create a Cloud Data Context by setting the mode to `cloud` and providing your [GX Cloud Credentials](/cloud/connect/connect_python.md#get-your-credentials) for the `GX_CLOUD_ORGANIZATION_ID`, `GX_CLOUD_WORKSPACE_ID` and `GX_CLOUD_ACCESS_TOKEN` environment variables. The rest of the code remains unchanged. + +```python +import great_expectations as gx +import os + +os.environ["GX_CLOUD_ORGANIZATION_ID"] = "" +os.environ["GX_CLOUD_WORKSPACE_ID"] = "" +os.environ["GX_CLOUD_ACCESS_TOKEN"] = "" + +context = gx.get_context(mode="cloud") +data_source = context.data_sources.add_sql(name="Postgres DB", connection_string="postgresql+psycopg2://username:passowrd@myhost.domain>:443>/sample_db") +asset = data_source.add_table_asset(table_name="sample_table", name="sample_table") + +batch_definition = asset.add_batch_definition_whole_table( + name="FULL_TABLE" +) + +suite = context.suites.get("my_suite") + +validation_definition = gx.ValidationDefinition( + data=batch_definition, suite=suite, name="Validation Definition" +) +context.validation_definitions.add(validation_definition) +``` + +Running this script will now create the same Data Source, Data Asset, Batch Definition, and Validation Definition in your GX Cloud organization. + +### Running Validations + +The code snippet below runs a Checkpoint in an existing GX Core configuration. + +```python +import great_expectations as gx + +context = gx.get_context("file") +checkpoint = context.checkpoints.get("My Checkpoint") +checkpoint.run() +``` + +In order to execute a checkpoint within your GX Cloud organization, the same code snippet can be used. In the same way as the previous example, set the mode of your data context to `cloud` and provide your [GX Cloud Credentials](/cloud/connect/connect_python.md#get-your-credentials) for the `GX_CLOUD_ORGANIZATION_ID`, `GX_CLOUD_WORKSPACE_ID` and `GX_CLOUD_ACCESS_TOKEN` environment variables. + +```python +import great_expectations as gx +import os + +os.environ["GX_CLOUD_ORGANIZATION_ID"] = "" +os.environ["GX_CLOUD_WORKSPACE_ID"] = "" +os.environ["GX_CLOUD_ACCESS_TOKEN"] = "" + +context = gx.get_context(mode="cloud") +checkpoint = context.checkpoints.get("My Checkpoint") +checkpoint.run() +``` + +Your GX Cloud checkpoint can now be used to run validations wherever needed, such as within your data pipelines. + +## Limitations +Some common limitations of migrating from GX Core to GX Cloud are listed below. Refer to the [compatibility reference](/help/compatibility_reference.md) for a comprehensive list of limitations. + +- Some Data Sources that are supported in GX Core may not be supported in GX Cloud. +- Any Custom Expectations that you have created are not compatible with GX Cloud. +- Credentials stored in the `config_variables.yml` file are not supported in GX Cloud. \ No newline at end of file diff --git a/docs/docusaurus/docs/reference/learn/reference_overview.md b/docs/docusaurus/docs/reference/learn/reference_overview.md index 7e84b83dd083..0a63b65605bd 100644 --- a/docs/docusaurus/docs/reference/learn/reference_overview.md +++ b/docs/docusaurus/docs/reference/learn/reference_overview.md @@ -21,6 +21,8 @@ import OverviewCard from '@site/src/components/OverviewCard'; ## Supplemental Information + + diff --git a/docs/docusaurus/sidebars.js b/docs/docusaurus/sidebars.js index 23202f9d9f71..944682965751 100644 --- a/docs/docusaurus/sidebars.js +++ b/docs/docusaurus/sidebars.js @@ -268,6 +268,7 @@ module.exports = { } ], learn: [ + 'reference/learn/core_to_cloud', { type: 'category', label: 'GX in your data pipeline',