[backlog] Different research tasks and ideas

[amotl]

About

While GH-19 has specific actionable tasks, this ticket bundles some secondary topics, mostly personal notes.

Topics

• Do CrateDB's datatime data types work correctly?
• Is it possible to already use CrateDB's advanced data types (ARRAY, OBJECT, GEO_X) in any way?
• It would be so sweet if a "DDL auto.create" feature could be supported in any way, right?
  https://docs.confluent.io/kafka-connectors/jdbc/current/sink-connector/sink_config_options.html#ddl-support
• In most cases, the CREATE DATABASE AS statement is suitable for fully automated data integration scenarios.
  https://www.alibabacloud.com/help/en/realtime-compute-for-apache-flink/latest/create-database-as-statement
  https://www.alibabacloud.com/help/en/realtime-compute-for-apache-flink/latest/create-table-as-statement
• Upsert vs. append mode.
  https://www.ververica.com/blog/streaming-modes-of-flink-kafka-connectors

Documentation

• Would it be possible to submit connectivity documentation/instructions to upstream cloud providers, to be published at places like https://www.alibabacloud.com/help/en/realtime-compute-for-apache-flink?
• Don't stop at Docker, also demonstrate usage with Podman.

---

Integrations

Kafka Connect

• Evaluate if/how "Kafka Connect" plays a sensible role in the Flink ecosystem.

JetBrains Big Data Tools for IntelliJ

• Have a look at IntelliJ's Big Data Tools, it has support for Apache Flink and more.

Resources

• https://blog.jetbrains.com/big-data-tools/2022/05/20/big-data-tools-1-6-eap/
• https://blog.jetbrains.com/big-data-tools/2022/08/01/big-data-tools-2022-2/
• https://www.heise.de/news/JetBrains-Big-Data-Tools-1-6-behalten-Flink-Jobs-im-Auge-7199626.html
• https://twitter.com/BigDataTools

What’s New in Big Data Tools 1.6

The highlights of this release include integration with Hive Metastore and the ability to monitor Flink jobs right inside your IDE, as well as SSO authentication on Amazon S3.

Apache Flink Monitoring in the IDE

You can now monitor Apache Flink applications right in your IDE with the ability to submit jobs.

Improved AWS EMR Integration

The Amazon EMR integration allows you to monitor clusters and nodes. As of v.1.6, you can view EMR clusters as a tree view and filter them by the following states: terminated last day/hour/week. We also added the ability to open logs as .txt files.

Debezium

• Some tutorial around Debezium and/or Apache Iceberg?

Resources

• https://github.com/memiiso/debezium-server-iceberg
• https://github.com/getindata/kafka-connect-iceberg-sink
• https://community.crate.io/t/replicating-data-to-cratedb-with-debezium-and-kafka/1388