fix: remove deprecated templates (#1066)

* fix: remove deprecated templates

Remove deprecated templates and clean up files.

* fix: remove python test for deprecated templates

* fix: python init file remove reference

* fix: clean up python files

Author: rajc242

java/README.md

@@ -15,7 +15,7 @@ Please refer to the [Dataproc Templates (Java - Spark) README](java/README.md) f
 * [GCSToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#5-gcs-to-gcs) (blogpost [link](https://medium.com/@ankuljain/migrate-gcs-to-gcs-using-dataproc-serverless-3b7b0f6ad6b9))
 * [GCSToJDBC](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#4-gcs-to-jdbc) (blogpost [link](https://medium.com/google-cloud/importing-data-from-gcs-to-databases-via-jdbc-using-dataproc-serverless-7ed75eab93ba))
 * [GCSToSpanner](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#3-gcs-to-spanner) (blogpost [link](https://medium.com/google-cloud/fast-export-large-database-tables-using-gcp-serverless-dataproc-spark-bb32b1260268))
-* [GCSToMongo](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#6-gcs-to-mongo) (blogpost [link] (https://medium.com/google-cloud/importing-data-from-gcs-to-mongodb-using-java-dataproc-serverless-6ff5c8d6f6d5))
+* [GCSToMongo](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#6-gcs-to-mongo) (blogpost [link](https://medium.com/google-cloud/importing-data-from-gcs-to-mongodb-using-java-dataproc-serverless-6ff5c8d6f6d5))
 * [GeneralTemplate](/java/src/main/java/com/google/cloud/dataproc/templates/general)
 * [HBaseToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/hbase#1-hbase-to-gcs) (blogpost [link](https://medium.com/google-cloud/using-dataproc-serverless-to-migrate-your-hbase-data-to-gcs-bf1ccf4ab945))
 * [HiveToBigQuery](/java/src/main/java/com/google/cloud/dataproc/templates/hive#1-hive-to-bigquery) (blogpost [link](https://medium.com/google-cloud/using-dataproc-serverless-to-migrate-your-hive-data-to-bigquery-8e2d4fcd1c24))
@@ -33,20 +33,17 @@ Please refer to the [Dataproc Templates (Java - Spark) README](java/README.md) f
 * [MongoToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/databases#executing-mongo-to-gcs-template) (blogpost [link](https://medium.com/google-cloud/migrating-data-from-mongo-to-gcs-using-java-and-dataproc-serverless-template-390500481804))
 * [PubSubToBigQuery](/java/src/main/java/com/google/cloud/dataproc/templates/pubsub#1-pubsub-to-bigquery) (blogpost [link](https://medium.com/google-cloud/from-pub-sub-to-bigquery-streaming-data-in-near-real-time-b550aeff595d))
 * [PubSubToBigTable](/java/src/main/java/com/google/cloud/dataproc/templates/pubsub#1-pubsub-to-bigtable) (blogpost [link](https://medium.com/google-cloud/stream-data-from-pub-sub-to-bigtable-using-dataproc-serverless-3142c1bcc22a))
-* [PubSubLiteToBigTable](/java/src/main/java/com/google/cloud/dataproc/templates/pubsublite#1-pubsublite-to-bigtable) (blogpost [link](https://medium.com/google-cloud/stream-data-from-pub-sub-lite-to-bigtable-using-dataproc-serverless-2c8816f40581)) **Deprecated and will be removed in Q1 2025**
 * [PubSubToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/pubsub/README.md#2-pubsub-to-gcs) (blogpost [link](https://medium.com/google-cloud/stream-data-from-pub-sub-to-cloud-storage-using-dataproc-serverless-7a1e4823926e))
-* [RedshiftToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/databases#executing-redshift-to-gcs-template) **Deprecated and will be removed in Q1 2025**
 * [S3ToBigQuery](/java/src/main/java/com/google/cloud/dataproc/templates/s3#1-s3-to-bigquery) (blogpost [link](https://medium.com/google-cloud/export-data-from-aws-s3-to-bigquery-using-dataproc-serverless-6dc7a9952fc4))
 * [SnowflakeToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/snowflake#1-snowflake-to-gcs) (blogpost [link](https://medium.com/google-cloud/export-snowflake-query-results-to-gcs-using-dataproc-serverless-3d68f5a01ca9))
 * [SpannerToGCS](/java/src/main/java/com/google/cloud/dataproc/templates/databases#executing-spanner-to-gcs-template) (blogpost [link](https://medium.com/google-cloud/cloud-spanner-export-query-results-using-dataproc-serverless-6f2f65b583a4))
-* [TextToBigquery](/java/src/main/java/com/google/cloud/dataproc/templates/gcs#7-text-to-bigquery) **Deprecated and will be removed in Q1 2025**
 * [WordCount](/java/src/main/java/com/google/cloud/dataproc/templates/word/WordCount.java)
 
 ...
 
 ## Requirements
 
-* Java 8
+* Java 11
 * Maven 3
 
 ## Running Templates

java/pom.xml

@@ -77,12 +77,10 @@
     <shade.skip>false</shade.skip>
     <spanner.jdbc.version>2.14.0</spanner.jdbc.version>
     <google.cloud.pubsub.version>1.132.0</google.cloud.pubsub.version>
-    <google.cloud.pubsublite.version>1.14.0</google.cloud.pubsublite.version>
     <google.cloud.bigtable.version>2.54.0</google.cloud.bigtable.version>
     <google.oauth.client.version>1.36.0</google.oauth.client.version>
     <google.api.client.version>2.7.0</google.api.client.version>
     <spark.bigquery.connector.version>0.36.4</spark.bigquery.connector.version>
-    <pubsublite.spark.sql.streaming.version>1.0.0</pubsublite.spark.sql.streaming.version>
     <spark.bigtable.connector.version>0.6.0</spark.bigtable.connector.version>
     <spark.streaming.pubsub.version>2.4.0</spark.streaming.pubsub.version>
 
@@ -333,11 +331,6 @@
       <artifactId>google-cloud-spanner-jdbc</artifactId>
       <version>${spanner.jdbc.version}</version>
     </dependency>
-    <dependency>
-      <groupId>com.google.cloud</groupId>
-      <artifactId>google-cloud-pubsublite</artifactId>
-      <version>${google.cloud.pubsublite.version}</version>
-    </dependency>
     <dependency>
       <groupId>com.google.cloud</groupId>
       <artifactId>google-cloud-storage</artifactId>
@@ -358,11 +351,6 @@
       <version>${spark.bigquery.connector.version}</version>
       <scope>provided</scope>
     </dependency>
-    <dependency>
-      <groupId>com.google.cloud</groupId>
-      <artifactId>pubsublite-spark-sql-streaming</artifactId>
-      <version>${pubsublite.spark.sql.streaming.version}</version>
-    </dependency>
     <dependency>
       <groupId>com.google.cloud</groupId>
       <artifactId>google-cloud-bigtable</artifactId>

java/src/main/java/com/google/cloud/dataproc/templates/databases/README.md

@@ -173,41 +173,6 @@ You can replace the ```casscon``` with your catalog name if it is passed. This i
 
 Make sure that either ```cassandratobq.input.query``` or both ```cassandratobq.input.keyspace``` and ```cassandratobq.input.table``` is provided. Setting or not setting all three properties at the same time will throw an error.
 
-
-## Executing Redshift to Cloud Storage template
-
-General Execution:
-
-```
-export GCP_PROJECT=<gcp-project-id> 
-export REGION=<region>  
-export SUBNET=<subnet>   
-export GCS_STAGING_LOCATION=<gcs-staging-bucket-folder> 
-export JARS=gs://<cloud-storage-bucket-name>/spark-redshift_<version>.jar,gs://<cloud-storage-bucket-name>/redshift_jdbc_<version>.jar,gs://<cloud-storage-bucket-name>/minimal_json<version>.jar
-
-bin/start.sh \
--- --template REDSHIFTTOGCS \
---templateProperty project.id=<gcp-project-id> \
---templateProperty redshift.aws.input.url=<jdbc:redshift://host-name:port-number/> \
---templateProperty redshift.aws.input.table=<Redshift-table-name> \
---templateProperty redshift.aws.input.temp.dir=<AWS-temp-directory> \
---templateProperty redshift.aws.input.iam.role=<Redshift-S3-IAM-role> \
---templateProperty redshift.aws.input.access.key=<Access-key> \
---templateProperty redshift.aws.input.secret.key=<Secret-key> \
---templateProperty redshift.gcs.output.file.format=<Output-File-Format> \
---templateProperty redshift.gcs.output.file.location=<Output-GCS-location> \
---templateProperty redshift.gcs.output.mode=<Output-GCS-Save-mode>
-```
-
-There are two optional properties as well with "Redshift to Cloud Storage" Template. Please find below the details :-
-
-```
---templateProperty redshift.gcs.temp.table='temporary_view_name' 
---templateProperty redshift.gcs.temp.query='select * from global_temp.temporary_view_name'
-```
-These properties are responsible for applying some spark sql transformations while loading data into Cloud Storage.
-The only thing needs to keep in mind is that, the name of the Spark temporary view and the name of table in the query should match exactly. Otherwise, there would be an error as:- "Table or view not found:"
-
 ## Executing Mongo to Cloud Storage template
 
 Template for exporting a MongoDB Collection to files in Google Cloud Storage. It supports writing JSON, CSV, Parquet and Avro formats.

java/src/main/java/com/google/cloud/dataproc/templates/databases/RedshiftToGCS.java

@@ -237,38 +237,7 @@ bin/start.sh \
 --templateProperty gcs.mongo.output.mode="overwrite"
 ```
 
-## 7. Text To BigQuery
-
-General Execution:
-
-```
-GCP_PROJECT=<gcp-project-id> \
-REGION=<region>  \
-SUBNET=<subnet>   \
-GCS_STAGING_LOCATION=<gcs-staging-bucket-folder> \
-HISTORY_SERVER_CLUSTER=<history-server> \
-bin/start.sh \
--- --template TEXTTOBIGQUERY \
---templateProperty project.id=<gcp-project-id> \
---templateProperty text.bigquery.input.location=<gcs path for input file> \
---templateProperty text.bigquery.input.compression=<compression file format like gzip or deflate> \
---templateProperty text.bigquery.input.delimiter=<, for csv> \
---templateProperty text.bigquery.output.dataset=<Big query dataset name> \
---templateProperty text.bigquery.output.table=<Big query table name> \
---templateProperty text.bigquery.output.mode=<Append|Overwrite|ErrorIfExists|Ignore> \
---templateProperty text.bigquery.temp.bucket=<bigquery temp bucket name>
-```
-
-There are two optional properties as well with "Text to BigQuery" Template. Please find below the details :-
-
-```
---templateProperty text.bigquery.temp.table='temporary_view_name'
---templateProperty text.bigquery.temp.query='select * from global_temp.temporary_view_name'
-```
-These properties are responsible for applying some spark sql transformations while loading data into BigQuery.
-The only thing needs to keep in mind is that, the name of the Spark temporary view and the name of table in the query should match exactly. Otherwise, there would be an error as:- "Table or view not found:"
-
-## 8. Deltalake To Iceberg
+## 7. Deltalake To Iceberg
 
 `deltalake.version.as_of` is an optional parameter which is default set to `0` means we will pick up the latest change only. We are providing below example to show how you can pass the value if you require time travel based on version number.