Merge pull request #83 from cbrianpace/main

Updates to improve floating data type compare

Author: cbrianpace

.gitignore

@@ -3,4 +3,5 @@ target/*
 database/local_test.sql
 .DS_Store
 ._*
-test_local.txt
+test_local.txt
+TODO.md

README.md

@@ -46,6 +46,7 @@ Before initiating the build and installation process, ensure the following prere
 - Unsupported data types: blob, long, longraw, bytea.
 - Cross-platform comparison limitations with boolean type.
 - Low precission types (float, real) cannot be compared to high precission types (double).
+- All low precission types are cast using a scale of 3.  If a higher scale is required consider using the map-expression override option.
 - Different databases cast float to different values.  Use float-cast option to switch between char and notation (scientific notation) if there are compare problems with float data types.
 
 # Getting Started
@@ -72,37 +73,62 @@ At a minimal the `repo-xxxxx` parameters are required in the properties file (or
 Run the script or use the command below to set up the PostgreSQL repository:
 
 ```shell
-java -jar pgcompare.jar --init
+java -jar pgcompare.jar init
 ```
 
 ## 5. Discover Tables
 
 Discover and map tables in specified schemas:
 
 ```shell
-java -jar pgcompare.jar --discovery
+java -jar pgcompare.jar discover
 ```
 
 # Usage
 
+## Command Line
+
+```shell
+java -jar pgcompare.jar <action> <options>
+```
+
+Actions:
+- **check**:  Recompare the out of sync rows from previous compare
+- **compare**:  Perform database compare
+- **copy-table**: Copy pgCompare metadata for table.  Must specify table alias to copy using --table option
+- **discover**:  Disocver tables and columns
+- **init**: Initialize the repository database
+
+Options:
+
+   -b|--batch {batch nbr}
+
+   -p|--project Project ID
+
+   -r|--report {file} Create html report of compare
+
+   -t|--table {target table}
+
+   --help
+
 ## Define Table Mapping
 
 1. Automatic Discovery
 
     Discover and map tables in specified schemas:
 
     ```shell
-    java -jar pgcompare.jar --discover
+    java -jar pgcompare.jar discover
     ```
 
 2. Manual Registration 
 
-    Insert mappings into `dc_table` and `dc_table_map` tables in the repository.
+    Insert mappings into `dc_table`, `dc_table_map`, `dc_table_column`, and `dc_table_column_map` tables in the repository.
 
 ## Run Data Comparison
 
 ```shell
-java -jar pgcompare.jar --batch 0
+java -jar pgcompare.jar compare --batch 0
 ```
 
 Batch 0 processes all data. Use `PGCOMPARE-BATCH` or specify the batch number using the `--batch` argument to specify a batch number.
@@ -112,11 +138,20 @@ Batch 0 processes all data. Use `PGCOMPARE-BATCH` or specify the batch number us
 Revalidate flagged rows:
 
 ```shell
-java -jar pgcompare.jar --batch 0 --check
+java -jar pgcompare.jar check --batch 0
 ```
 
 # Upgrading
 
+## Version 0.4.0 Enhancements
+
+- Improved casting of low precision data types.
+- Added html report generation.
+- Refactored code for efficiency.
+- Modified arguments and added 'verb' clause to command line.
+  
+**Note:** Drop and recreate the repository to upgrade to 0.4.0.
+
 ## Version 0.3.0 Enhancements
 
 - DB2 support.
@@ -188,11 +223,11 @@ FROM dc_source s
 Properties are categorized into four sections: system, repository, source, and target. Each section has specific properties, as described in detail in the documentation.  The properties can be specified via a configuration file, environment variables or a combination of both.  To use environment variables, the environment variable will be the name of the property in upper case with dashes '-' converted to underscore '_' and prefixed with PGCOMPARE_.  For example, batch-fetch-size can be set by using the environment variable PGCOMPARE_BATCH_FETCH_SIZE.
 
 ### System
+
 - batch-fetch-size: Sets the fetch size for retrieving rows from the source or target database.
 - batch-commit-size:  The commit size controls the array size and number of rows concurrently inserted into the dc_source/dc_target staging tables.
 - batch-progress-report-size:  Defines the number of rows used in mod to report progress.
-- database-source:  Determines if the sorting of the rows based on primary key occurs on the source/target database.  If set to true, the default, the rows will be sorted before being compared.  If set to false, the sorting will take place in the repository database.
-- float-cast: Defines how float and double data types are cast for hash function (notation|char).  Default is char.
+- database-sort:  Determines if the sorting of the rows based on primary key occurs on the source/target database.  If set to true, the default, the rows will be sorted before being compared.  If set to false, the sorting will take place in the repository database.
 - loader-threads: Sets the number of threads to load data into the temporary tables. Default is 4.  Set to 0 to disable loader threads.
 - log-level:   Level to determine the amount of log messages written to the log destination.
 - log-destination:  Location where log messages will be written.  Default is stdout.
@@ -201,6 +236,8 @@ Properties are categorized into four sections: system, repository, source, and t
 - observer-throttle:  Set to true or false, instructs the loader threads to pause and wait for the observer thread to catch up before continuing to load more data into the staging tables.
 - observer-throttle-size:  Number of rows loaded before the loader thread will sleep and wait for clearance from the observer thread.
 - observer-vacuum:  Set to true or false, instructs the observer whether to perform a vacuum on the staging tables during checkpoints.
+- stage-table-parallel: Default parallel degree to set on staging table (default: 0)
+- standard-number-format: Format used to cast numbers (default:0000000000000000000000.0000000000000000000000)
 
 ### Repository
 
@@ -214,7 +251,6 @@ Properties are categorized into four sections: system, repository, source, and t
 
 ### Source
 
-- source-database-hash: True or false, instructs the application where the hash should be computed (on the database or by the application).
 - source-dbname:  Database or service name.
 - source-host:  Database server name.
 - source-password:  Database password.
@@ -226,7 +262,6 @@ Properties are categorized into four sections: system, repository, source, and t
 
 ### Target
 
-- target-database-hash: True or false, instructs the application where the hash should be computed (on the database or by the application).
 - target-dbname:  Database or service name.
 - target-host:  Database server name.
 - target-password:  Database password.

database/pgCompare.sql

@@ -49,7 +49,8 @@ CREATE TABLE dc_table (
 	pid int8 DEFAULT 1 NOT NULL,
 	tid int8 GENERATED ALWAYS AS IDENTITY( INCREMENT BY 1 MINVALUE 1 MAXVALUE 9223372036854775807 START 1 CACHE 1 NO CYCLE) NOT NULL,
 	table_alias text NULL,
-	status varchar(10) DEFAULT 'disabled'::character varying NULL,
+    enabled boolean default true,
+	--status varchar(10) DEFAULT 'disabled'::character varying NULL,
 	batch_nbr int4 DEFAULT 1 NULL,
 	parallel_degree int4 DEFAULT 1 NULL,
 	CONSTRAINT dc_table_pk PRIMARY KEY (tid)
@@ -61,7 +62,7 @@ CREATE TABLE dc_table_column (
 	tid int8 NOT NULL,
 	column_id int8 GENERATED ALWAYS AS IDENTITY( INCREMENT BY 1 MINVALUE 1 MAXVALUE 9223372036854775807 START 1 CACHE 1 NO CYCLE) NOT NULL,
 	column_alias text NOT NULL,
-	status varchar(15) DEFAULT 'compare'::character varying NULL,
+    enabled boolean default true,
 	CONSTRAINT dc_table_column_pk PRIMARY KEY (column_id)
 );
 
@@ -91,12 +92,10 @@ CREATE TABLE dc_table_column_map (
 
 CREATE TABLE dc_table_history (
 	tid int8 NOT NULL,
-	load_id varchar(100) NULL,
 	batch_nbr int4 NOT NULL,
 	start_dt timestamptz NOT NULL,
 	end_dt timestamptz NULL,
 	action_result jsonb NULL,
-	action_type varchar(20) NOT NULL,
 	row_count int8 NULL
 );
 
@@ -107,7 +106,6 @@ CREATE TABLE dc_table_map (
 	dest_type varchar(20) DEFAULT 'target'::character varying NOT NULL,
 	schema_name text NOT NULL,
 	table_name text NOT NULL,
-	parallel_degree int4 DEFAULT 1 NULL,
 	mod_column varchar(200) NULL,
 	table_filter varchar(200) NULL,
 	schema_preserve_case bool DEFAULT false NULL,
@@ -150,3 +148,59 @@ ALTER TABLE dc_table_map ADD CONSTRAINT dc_table_map_fk FOREIGN KEY (tid) REFERE
 --
 INSERT INTO pgcompare.dc_project (project_name,project_config) VALUES
 	 ('default',NULL);
+
+--
+-- Functions
+--
+-- DROP FUNCTION pgcompare.dc_copy_table(int4, int4);
+
+CREATE OR REPLACE FUNCTION pgcompare.dc_copy_table(p_pid integer, p_tid integer)
+ RETURNS bigint
+ LANGUAGE plpgsql
+AS $function$
+DECLARE
+v_new_tid bigint;
+    v_new_cid bigint;
+    r_column record;
+BEGIN
+    -- Duplicate dc_table
+INSERT INTO dc_table (pid, tid, table_alias, enabled, batch_nbr, parallel_degree)
+SELECT pid, tid, table_alias, enabled, batch_nbr, parallel_degree
+FROM dc_table
+WHERE pid = p_pid AND tid = p_tid
+    RETURNING tid INTO v_new_tid;
+
+-- Duplicate dc_table_map
+INSERT INTO dc_table_map (tid, dest_type, schema_name, table_name, mod_column, table_filter, schema_preserve_case, table_preserve_case)
+SELECT v_new_tid, dest_type, schema_name, table_name, mod_column, table_filter, schema_preserve_case, table_preserve_case
+FROM dc_table_map
+WHERE tid = p_tid;
+
+-- Duplicate dc_table_column and dc_table_column_map
+FOR r_column IN
+SELECT tid, column_id, column_alias, enabled
+FROM dc_table_column
+WHERE tid = p_tid
+    LOOP
+-- Insert into dc_table_column with new tid and potentially new cid
+INSERT INTO dc_table_column (tid, column_id, column_alias, enabled)
+SELECT v_new_tid, cid, tid, column_id, column_alias, enabled
+FROM dc_table_column
+WHERE tid = p_tid
+  AND column_id = r_column.column_id
+    RETURNING cid INTO v_new_cid;
+
+-- Duplicate dc_table_column_map
+INSERT INTO dc_table_column_map (tid, column_id, column_origin, column_name, data_type, data_class, data_length, number_precision,
+                                 number_scale, column_nullable, column_primarykey, map_expression, supported, preserve_case, map_type)
+SELECT v_new_tid, v_new_cid, column_origin, column_name, data_type, data_class, data_length, number_precision,
+       number_scale, column_nullable, column_primarykey, map_expression, supported, preserve_case, map_type
+FROM dc_table_column_map
+WHERE tid = p_tid
+  AND column_id = r_column.column_id;
+END LOOP;
+
+RETURN v_new_tid;
+END;
+$function$
+;

pgcompare.properties.sample

@@ -103,9 +103,6 @@ repo-sslmode=disable
 #       Database user to use for authentication.
 #  source|target-password:
 #       Database password.
-#  source|target-database-hash:
-#       Determines whether to generate the hash on the database (true) or by the application (false).
-#       Default: true
 #  source|target-sslmode:
 #       SSL mode to use when connecting to the database (disable|prefer|require).
 #       Default: disable
@@ -121,7 +118,6 @@ source-port=1521
 source-dbname=hr
 source-user=appuser
 source-password=welcome1
-source-database-hash=false
 source-sslmode=disable
 source-schema=appuser
 source-schema=appuser
@@ -132,7 +128,6 @@ source-schema=appuser
 #source-dbname=hr
 #source-user=postgres
 #source-password=welcome1
-#source-database-hash=true
 #source-sslmode=disable
 #source-schema=appuser
 
@@ -146,7 +141,6 @@ target-port=5432
 target-dbname=hr
 target-user=postgres
 target-password=welcome1
-target-database-hash=true
 target-sslmode=disable
 target-schema=public
 # Oracle Example
@@ -156,7 +150,6 @@ target-schema=public
 #target-dbname=hr
 #target-user=appuser
 #target-password=welcome1
-#target-database-hash=true
 #target-sslmode=disable
 #target-schema=appuser
 

pom.xml

@@ -21,13 +21,13 @@
         <dependency>
             <groupId>com.ibm.db2</groupId>
             <artifactId>jcc</artifactId>
-            <version>11.5.8.0</version>
+            <version>12.1.2.0</version>
         </dependency>
         <!-- https://mvnrepository.com/artifact/org.postgresql/postgresql -->
         <dependency>
             <groupId>org.postgresql</groupId>
             <artifactId>postgresql</artifactId>
-            <version>42.7.4</version>
+            <version>42.7.7</version>
         </dependency>
         <!-- https://mvnrepository.com/artifact/commons-cli/commons-cli -->
         <dependency>
@@ -45,25 +45,25 @@
         <dependency>
             <groupId>com.oracle.database.jdbc</groupId>
             <artifactId>ojdbc11</artifactId>
-            <version>23.4.0.24.05</version>
+            <version>23.8.0.25.04</version>
         </dependency>
         <!-- https://mvnrepository.com/artifact/org.json/json -->
         <dependency>
             <groupId>org.json</groupId>
             <artifactId>json</artifactId>
-            <version>20241224</version>
+            <version>20250517</version>
         </dependency>
         <!-- https://mvnrepository.com/artifact/com.mysql/mysql-connector-j -->
         <dependency>
             <groupId>com.mysql</groupId>
             <artifactId>mysql-connector-j</artifactId>
-            <version>8.4.0</version>
+            <version>9.3.0</version>
         </dependency>
         <!-- https://mvnrepository.com/artifact/com.microsoft.sqlserver/mssql-jdbc -->
         <dependency>
             <groupId>com.microsoft.sqlserver</groupId>
             <artifactId>mssql-jdbc</artifactId>
-            <version>12.6.2.jre11</version>
+            <version>12.10.0.jre11</version>
         </dependency>
         <!-- https://mvnrepository.com/artifact/org.apache.commons/commons-lang3 -->
         <dependency>
@@ -75,7 +75,7 @@
         <dependency>
             <groupId>org.mariadb.jdbc</groupId>
             <artifactId>mariadb-java-client</artifactId>
-            <version>3.5.2</version>
+            <version>3.5.3</version>
         </dependency>
     </dependencies>
     </dependencyManagement>
@@ -127,6 +127,12 @@
             <groupId>org.mariadb.jdbc</groupId>
             <artifactId>mariadb-java-client</artifactId>
         </dependency>
+        <!-- https://mvnrepository.com/artifact/net.snowflake/snowflake-jdbc -->
+        <dependency>
+            <groupId>net.snowflake</groupId>
+            <artifactId>snowflake-jdbc</artifactId>
+            <version>3.24.2</version>
+        </dependency>
     </dependencies>
 
     <build>