New Feature #13: Implemented Export / Import mechanism over Datapump Network_link

Author: thedatabaseme

Changelog.txt

@@ -2,6 +2,7 @@ Changelog of Pythia:
 
 Version 2.1.0
   - New Feature #1: Added documentation / instructions under docs/*.md for the most common tasks you can do with Pythia
+  - New Feature #13: Implemented Export / Import mechanism over Datapump Network_link
   - Fixed Issue #7: Updated README(s) and added Features to the list
   - Fixed Issue #11: All not needed Variables are moved from vars/main.yml to defaults/main.yml. Making them better overwriteable
 

README.md

@@ -9,6 +9,7 @@ Pythia is a Role for ansible that helps you to automate the following tasks when
   - Runs SQL Scripts against a specified Oracle Database
   - Upgrade a Oracle DB to a new Version
   - Duplicate a Database with RMAN "duplicate from active database"
+  - Datapump Export / Import over NETWORK_LINK
   - Install Oracle Client
 
 Thereby several Prerequisites are fullfilled and or checked. E.G.
@@ -68,6 +69,7 @@ Pythia works with tags to combine the actions of installation, configuration, pa
   - client: Specifies, that you want to install an Oracle Client. When tagging `client`, you also may to want to specify the Variable `client_version` when starting the playbook with `client` TAG.
   - duplicate: Specifies, that you want to duplicate a Source Database to a Target Database. Uses RMAN duplicate from active Database. You need to specify the `duplicate_source_host`, `duplicate_target_host`, `duplicate_source_sid` and `duplicate_target_sid` Variable. As HOSTS you need to specify the Target Host.  
   - upgrade: Specifies that you want to upgrade a Database to a new Version. You need to specify the Variables `oracle_source_version`, `oracle_target_version`, `oracle_sid` and `upgrade_mode` when starting the Playbook with the `upgrade` TAG.
+  - datapump: Specifies, that you want to do an Datapump Export / Import over NETWORK_LINK. You need to specify the `datapump_source_host`, `datapump_target_host`, `datapump_source_sid` and `datapump_target_sid` Variable. As HOSTS you need to specify the Target Host.
 
 Example: Install Oracle RDBMS with Version 19c and Enterprise Edition on Host "ansibletest", install the 19.5 RU (Patch ID: 30125133). Build a Database named "ORA19" on top:
 
@@ -111,3 +113,10 @@ Also you're able to specify, that the SPFILE of your maybe existing Target Datab
 Example: Upgrade an Oracle Database in Version 18 to Oracle Database in Version 19 on Host "ansibletest":
 
     - ansible-playbook /etc/ansible/pythia/pythia.yml -e "oracle_source_version=18EE oracle_target_version=19EE oracle_sid=ORAUPGRD upgrade_mode=deploy HOSTS=ansibletest" --tags "upgrade" -k -K -u <username>
+
+Example: Do a Datapump Export / Import over NETWORK_LINK. Source Database named EXPSRC to a Database named EXPDST on the host "vmdbsoradatapumptarget". As HOST, the Target Host must be specified:
+Attention!: The Passwords for the Source and Target Database is needed for the datapump process. The Password has to be provided within the vault.yml (using ansible-vault) as vault_datapump_<SOURCESID> and vault_datapump_<TARGETSID>
+
+    - ansible-playbook /etc/ansible/pythia/pythia.yml -e "HOSTS=vmdbsoradatapumptarget datapump_source_host=vmdbsdatapumpsource datapump_target_host=vmdbsoradatapumptarget datapump_source_sid=EXPSRC datapump_target_sid=EXPDST datapump_full=False datapump_schema_list='USER1,USER2'" --tags "datapump" -k -K -u <username> --vault-password-file /etc/ansible/vault_password.pwd
+
+    - ansible-playbook /etc/ansible/pythia/pythia.yml -e "HOSTS=vmdbsoradatapumptarget datapump_source_host=vmdbsdatapumpsource datapump_target_host=vmdbsoradatapumptarget datapump_source_sid=EXPSRC datapump_target_sid=EXPDST" --tags "datapump" -k -K -u <username> --vault-password-file /etc/ansible/vault_password.pwd

docs/07_DATAPUMP.md

@@ -0,0 +1,66 @@
+# Datapump Export / Import a Database with Pythia
+
+# UPDATE STARTS HERE!
+
+Pythia is able to do a Datapump Export / Import over a NETWORK_LINK (Database Link). You may either do a FULL Export / Import or an Import of a List of Schemas. Pythia also supports you by remapping the schemas.
+
+Be aware, that all restrictions for doing an Export / Import over a Database Link are applying here. More informations can be found [here](https://docs.oracle.com/database/121/SUTIL/GUID-0871E56B-07EB-43B3-91DA-D1F457CF6182.htm#SUTIL919)
+
+Be aware, that using the `datapump` TAG assumes, that there is a valid RDBMS in the correct Version already installed on both Systems. If this isn't the case, you need to specify the `rdbms` TAG like described [here](https://github.com/thedatabaseme/pythia/blob/master/docs/02_INSTALL_RDBMS.md)
+
+## Working with an Ansible Vault
+
+To process a Datapump Export / Import of a database, Pythia will need the Password for a User to execute the actual Export and Import for both Databases. It's recommended, that this user is the System user, but any user with the appropriate permissions will be feasable. The password needs to be provided in a `vault.yml` file, that resides under the `roles/pythia/vars` directory.
+To create a new Vault File, you can do so by:
+
+    ansible-vault create ./pythia/roles/pythia/vars/vault.yml
+
+You will get asked for a password that will protect the Vault and is needed to edit or view the Vault File in the future.
+In the upcoming editor you can enter informations, that will be stored encrypted in the Vault File. For Pythia Duplicate, you need to specify the SYS Password of the Source Database following the Syntax `vault_datapump_<SID>`. Here is an example:
+
+    vault_datapump_EXPSRC: supersecretpassword
+
+When calling a Playbook that uses Pythia, you can specify the Vault Password either by a Vault Passwordfile using `--vault-password-file` or by entering it during the Playbook call using `--ask-vault-password`.
+
+You can find more general informations about the usage of an Ansible Vault [here](https://docs.ansible.com/ansible/latest/user_guide/vault.html).
+## Variables
+
+### Mandatory Variables
+
+
+| Variable Name | Description              | Default Value |
+|---------------|--------------------------|---------------|
+|HOSTS |In this special usecase, you need to specify the Target Host as HOSTS.|none|
+|datapump_source_host|The source host on which the datapump_source_sid is running. <br>Only needed when starting the playbook with the `datapump` TAG, for doing an Datapump Export / Import a Database.|none|
+|datapump_target_host|The target host on which the datapump_target_sid is running. <br>Only needed when starting the playbook with the `datapump` TAG, for doing an Datapump Export / Import a Database.|none|
+|datapump_source_sid|The source SID of the Database to duplicate. Only needed when <br>starting the playbook with the `datapump` TAG, for duplicating a Database.|none|
+|datapump_target_sid|The target SID of the Database to duplicate to. Only needed when <br>starting the playbook with the `datapump` TAG, for duplicating a Database.|none|
+
+### Optional Variables
+
+
+| Variable Name | Description              | Default Value |
+|---------------|--------------------------|---------------|
+|datapump_full|Should a Full Export / Import be made or schema Import only? If `FALSE`, <br>you need to specify the `datapump_schema_list` Variable|`TRUE`|
+|datapump_schema_list|List of Schemas (divided by , for example `USER1,USER2`). <br>Only relevant when `datapump_full` is `FALSE`. You need to to set `datapump_full` to `FALSE` when you want to export only a certain list of Schemas.|none|
+|datapump_schema_remap|List of Schemas to Remap. For Example `USER1:USERA,USER2:USERB` <br>will remap USER1 to USERA and USER2 to USERB|none|
+|datapump_max_runtime|Number of Seconds of the maximum Datapump runtime.|`14400`|
+|datapump_source_user|User on Source Database which the Database Link uses to connect over Network_link|`System`|
+|datapump_target_user|User on Target Database which impdp uses to connect|`System`|
+|datapump_table_exists_action|Controls the impdp TABLE_EXIST_ACTION. Can either be `APPEND`, `REPLACE`, `SKIP` or `TRUNCATE`|`SKIP`|
+
+## Examples
+
+Example: Do a Datapump Export / Import over NETWORK_LINK. Source Database named EXPSRC to a Database named EXPDST on the host "vmdbsoradatapumptarget". As HOST, the Target Host must be specified:
+Attention!: The Passwords for the Source and Target Database is needed for the datapump process. The Password has to be provided within the vault.yml (using ansible-vault) as `vault_datapump_<SOURCESID>` and `vault_datapump_<TARGETSID>`
+
+    - ansible-playbook /etc/ansible/pythia/pythia.yml -e "HOSTS=vmdbsoradatapumptarget datapump_source_host=vmdbsdatapumpsource datapump_target_host=vmdbsoradatapumptarget datapump_source_sid=EXPSRC datapump_target_sid=EXPDST datapump_full=False datapump_schema_list='USER1,USER2'" --tags "datapump" -k -K -u <username> --vault-password-file /etc/ansible/vault_password.pwd
+
+    - ansible-playbook /etc/ansible/pythia/pythia.yml -e "HOSTS=vmdbsoradatapumptarget datapump_source_host=vmdbsdatapumpsource datapump_target_host=vmdbsoradatapumptarget datapump_source_sid=EXPSRC datapump_target_sid=EXPDST" --tags "datapump" -k -K -u <username> --vault-password-file /etc/ansible/vault_password.pwd
+
+## Further Documentation
+
+This is only were the fun begins, next let's have a look on how to create a Database using Pythia and how to install Patches on top of your RDBMS and Database.
+
+  - [Install a RDBMS](https://github.com/thedatabaseme/pythia/blob/master/docs/02_INSTALL_RDBMS.md)
+  - [Install Patches and Updates](https://github.com/thedatabaseme/pythia/blob/master/docs/04_INSTALL_PATCH.md)

roles/pythia/README.md

@@ -10,7 +10,7 @@ Pythia is a Role for ansible that helps you to automate the following tasks when
 - Install Patches on top of a new and existing RDBMS and or Database
 - Upgrade a Oracle DB to a new Version
 - Duplicate a Database with RMAN "duplicate from active database"
-- Duplicate a Database with RMAN "duplicate from active database"
+- Datapump Export / Import over NETWORK_LINK
 - Install Oracle Client
 
 Thereby several Prerequisites are fullfilled and or checked. E.G.
@@ -72,6 +72,17 @@ main.yml Variables (can be set when calling the playbook, see Examples):
   - current_hugepages (Default 0): Just for initializing the Variable. Must not be modified
   - hugepages (Default 0): Will be calculated during Playbook Runtime. Will be included to the sysctl Configuration when deploying a Database
   - client_install_type (Default Administrator): Choose Which Client Install Type you want to install. Can be Administrator, Runtime, InstantClient or Custom
+  - datapump_source_host (Default NULL): The source host on which the datapump_source_sid is running. Only needed when starting the playbook with the "datapump" tag, for doing an Datapump Export / Import a Database.
+  - datapump_target_host (Default NULL): The target host on which the datapump_target_sid is running. Only needed when starting the playbook with the "datapump" tag, for doing an Datapump Export / Import a Database.
+  - datapump_source_sid (Default NULL): The source SID of the Database to Export. Only needed when starting the playbook with the "datapump" tag, for doing an Datapump Export / Import a Database.
+  - datapump_target_sid (Default NULL): The target SID of the Database to Import to. Only needed when starting the playbook with the "datapump" tag, for doing an Datapump Export / Import a Database.
+  - datapump_full (Default TRUE): Should a Full Export / Import be made or schema Import only? If FALSE, you need to specify the datapump_schema_list Variable
+  - datapump_schema_list (Default NULL): List of Schemas (divided by , for example USER1,USER2). Only relevant when datapump_full is FALSE. You need to to set datapump_full to FALSE when you want to export only a certain list of Schemas.
+  - datapump_schema_remap (Default NULL): List of Schemas to Remap. For Example "USER1:USERA,USER2:USERB" will remap USER1 to USERA and USER2 to USERB
+  - datapump_max_runtime (Default 14400): Number of Seconds of the maximum Datapump runtime.
+  - datapump_source_user (Default system): User on Source Database which the Database Link uses to connect over Network_link
+  - datapump_target_user (Default system): User on Target Database which impdp uses to  connect
+  - datapump_table_exists_action (Default SKIP): Controls the impdp TABLE_EXIST_ACTION. Can either be APPEND, REPLACE, [SKIP] or TRUNCATE
   - duplicate_source_host (Default NULL): The source host on which the duplicate_source_sid is running. Only needed when starting the playbook with the "duplicate" tag, for duplicating a Database.
   - duplicate_target_host (Default NULL): The target host on which the duplicate_target_sid is running. Only needed when starting the playbook with the "duplicate" tag, for duplicating a Database.
   - duplicate_source_sid (Default NULL): The source SID of the Database to duplicate. Only needed when starting the playbook with the "duplicate" tag, for duplicating a Database.
@@ -117,21 +128,22 @@ Dependencies
 ------------
 
 Tags that can be specified:
-  - rdbms: Specifies, that you want a RDBMS installed. When tagging rdbms, you also may want to specify the Variable oracle_version when starting the playbook with "rdbms" tag.
-  - db: Specifies, that you want to install a Database on top a new RDBMS (when specifying "rdbms" as a tag alongside with the "db" tag) or an already installed RDBMS on the target system. You may also want to specify the Variable oracle_version when starting the playbook with the "db" tag. You need to specify the oracle_sid Variable when calling the playbook for naming the new Database.
-  - patch: Specifies, that you want to install a Patch on top of a new RDBMS (when specifying "rdbms" as a tag alongside with the "patch" tag) or an already installed RDBMS. When specifying the "patch" tag, you need to also specify the "install_patch" Variable when running the playbook. While running under the "patch" tag, the Patch Archive will be uncompressed to the target system. The Patch Files will be deleted after successfully installing the Patch.
-  - patchonly: Like "patch" tag but implies, that the Patch Archive already exists on the target system. No cleanup will be done after the Patch Installation.
+  - rdbms: Specifies, that you want a RDBMS installed. When tagging `rdbms`, you also may want to specify the Variable `oracle_version` when starting the playbook with `rdbms` tag.
+  - db: Specifies, that you want to install a Database on top a new RDBMS (when specifying `rdbms` as a TAG alongside with the `db` TAG) or an already installed RDBMS on the target system. You may also want to specify the Variable `oracle_version` when starting the playbook with the `db` TAG. You need to specify the `oracle_sid` Variable when calling the playbook for naming the new Database.
+  - patch: Specifies, that you want to install a Patch on top of a new RDBMS (when specifying `rdbms` as a TAG alongside with the `patch` TAG) or an already installed RDBMS. When specifying the `patch` TAG, you need to also specify the `install_patch` Variable when running the playbook. While running under the `patch` TAG, the Patch Archive will be uncompressed to the target system. The Patch Files will be deleted after successfully installing the Patch.
+  - patchonly: Like `patch` tag but implies, that the Patch Archive already exists on the target system. No cleanup will be done after the Patch Installation.
   - cleanup: Forces a cleanup after sucessfull Patch Installation. Also forces that the created backup Files under the RDBMS that are created by OPatch when patching are deleted. Handle with caution if you may want to rollback the Patch.
-  - listener: Only creates a listener for the Specified Database with the oracle_sid Variable.
-  - sqlscript: Triggers, that SQL Scripts (.sql) under files/default (or specified by local_sql_stage_directory) directory will be executed against the specified oracle_sid. Be aware, the Scripts will be executed without any precheck. Bad SQL can cause immense harm.
-  - rpm: Triggers that additional RPMs have to be installed. When specifying "rpm" tag, you need also to specify the "install_rpm" Variable when running the playbook. The RPM will be transfered to the target system. After installing the RPM, the file will be removed.
-  - autostart: Controls that the automatic startup of the Database you specified will be implemented. When specifying the autostart tag, you also must specify the oracle_sid Variable when calling the Playbook.
-  - hugepage: Triggers, that Hugepages have to be configured. Can only be triggered when also tagging db. Hugepages will be calculated by given sga_max_size (Default 2G)
-  - converttohugepage: Converts a non Hugepage configured System to use Hugepages. Disables transparent_hugepages. Needs sga_max_size Variable set to the absolute Size (in GB) of all SGA's on the System
-  - prepare: Prepares the Target System for an upcomming Oracle Installation. Can be combined with the hugepage Tag and needs sga_max_size (Default 2G) specified to calculate Shared Memory and Hugepages
-  - client: Specifies, that you want to install an Oracle Client. When tagging client, you also may to want to specify the Variable client_version when starting the playbook with "client" tag.
-  - duplicate: Specifies, that you want to duplicate a Source Database to a Target Database. Uses RMAN duplicate from active Database. You need to specify the duplicate_source_host, duplicate_target_host, duplicate_source_sid and duplicate_target_sid Variable. As HOSTS you need to specify the Target Host.
-  - upgrade: Specifies that you want to upgrade a Database to a new Version. You need to specify the Variables oracle_source_version, oracle_target_version, oracle_sid and upgrade_mode when starting the Playbook with the "upgrade" tag.
+  - listener: Only creates a listener for the Specified Database with the `oracle_sid` Variable.
+  - sqlscript: Triggers, that SQL Scripts (.sql) under files/default (or specified by `local_sql_stage_directory`) directory will be executed against the specified `oracle_sid`. Be aware, the Scripts will be executed without any precheck. Bad SQL can cause immense harm.
+  - rpm: Triggers that additional RPMs have to be installed. When specifying `rpm` TAG, you need also to specify the `install_rpm` Variable when running the playbook. The RPM will be transfered to the target system. After installing the RPM, the file will be removed.
+  - autostart: Controls that the automatic startup of the Database you specified will be implemented. When specifying the `autostart` TAG, you also must specify the `oracle_sid` Variable when calling the Playbook.
+  - hugepage: Triggers, that Hugepages have to be configured. Can only be triggered when also tagging `db`. Hugepages will be calculated by given `sga_max_size` (Default 2G)
+  - converttohugepage: Converts a non Hugepage configured System to use Hugepages. Disables `transparent_hugepages`. Needs `sga_max_size` Variable set to the absolute Size (in GB) of all SGA's on the System
+  - prepare: Prepares the Target System for an upcomming Oracle Installation. Can be combined with the `hugepage` TAG and needs `sga_max_size` (Default 2G) specified to calculate Shared Memory and Hugepages
+  - client: Specifies, that you want to install an Oracle Client. When tagging `client`, you also may to want to specify the Variable `client_version` when starting the playbook with `client` TAG.
+  - duplicate: Specifies, that you want to duplicate a Source Database to a Target Database. Uses RMAN duplicate from active Database. You need to specify the `duplicate_source_host`, `duplicate_target_host`, `duplicate_source_sid` and `duplicate_target_sid` Variable. As HOSTS you need to specify the Target Host.
+  - upgrade: Specifies that you want to upgrade a Database to a new Version. You need to specify the Variables `oracle_source_version`, `oracle_target_version`, `oracle_sid` and `upgrade_mode` when starting the Playbook with the `upgrade` TAG.
+  - datapump: Specifies, that you want to do an Datapump Export / Import over NETWORK_LINK. You need to specify the `datapump_source_host`, `datapump_target_host`, `datapump_source_sid` and `datapump_target_sid` Variable. As HOSTS you need to specify the Target Host.
 
 
 Example Playbook