Merge pull request #987 from rtdip/feature/cleanup

Removal of Deprecated Modules, Dependency Clean‑up, and Turbodbc Deprecation (RTDIP SDK v0.14.4)

Author: Amber-Rigg

docs/api/overview.md

@@ -1,4 +1,4 @@
-<center> ![rest](images/rest-api-logo.png){width=50%} </center>
+<center> <img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/api/images/rest-api-logo.png" width="50%" alt="rest" /> </center>
 
 <!-- --8<-- [start:restapi] -->
 

docs/api/rest_apis.md

@@ -3,23 +3,23 @@
 
 RTDIP REST API documentation is available in a number of formats, as described below. 
 
-<center> ![rest](images/open-api.png){width=50%} </center>
+<center> <img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/api/images/open-api.png" width="50%" alt="rest" /> </center>
 
 <!-- --8<-- [start:openapi] -->
 
 RTDIP REST APIs are built to OpenAPI standard 3.0.2. You can obtain the OpenAPI JSON schema at the following endpoint of your deployed APIs `https://{domain name}/api/openapi.json`
 
 <!-- --8<-- [end:openapi] -->
 
-<center> ![rest](images/swagger.png){width=50%} </center>
+<center> <img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/api/images/swagger.png" width="50%" alt="rest" /> </center>
 
 <!-- --8<-- [start:swagger] -->
 
 It is recommended to review the **Swagger** documentation that can be found at the following endpoint of your deployed APIs `https://{domain name}/docs` for more information about the parameters and options for each API. It is also possible to try out each API from this link.
 
 <!-- --8<-- [end:swagger] -->
 
-<center> ![rest](images/redoc-logo.png){width=50%} </center>
+<center> <img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/api/images/redoc-logo.png" width="50%" alt="rest" /> </center>
 
 <!-- --8<-- [start:redoc] -->
 

docs/blog/posts/delta_and_rtdip.md

@@ -9,7 +9,7 @@ tags:
 
 # Delta Lakehouse and Real Time Data Ingestion Platform
 
-<center> ![Delta Lakehouse](../images/delta-lakehouse.svg){width=50%} </center>
+<center> <img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/delta-lakehouse.svg" width="50%" alt="Delta Lakehouse" /> </center>
 
 Real Time Data Ingestion Platform leverages Delta and the concept of a Lakehouse to ingest, store and manage it's data. There are many benefits to Delta for performing data engineering tasks on files stored in a data lake including ACID transactions, maintenance, SQL query capability and performance at scale. To find out more about Delta Lakehouse please see [here.](https://databricks.com/product/data-lakehouse)
 

docs/blog/posts/deprecations_and_module_removals.md

@@ -0,0 +1,178 @@
+---
+date: 2026-04-01
+authors:
+  - GBARAS
+---
+
+# Important: Module Removals and Deprecations in RTDIP SDK v0.14.5
+
+<center>
+<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/framework.png" width="60%" alt="breaking-changes" />
+</center>
+
+We are pleased to announce significant updates to the RTDIP SDK that improve performance and focus the platform on its core strengths. These changes involve the removal of several modules and the deprecation of turbodbc support. This post outlines these changes and provides guidance on migration paths.
+
+<!-- more -->
+
+## Summary of Changes
+
+### Removed Modules
+
+The following modules have been completely removed from RTDIP SDK v0.14.5:
+
+#### 1. Data Quality Module (`rtdip_sdk.pipelines.data_quality`)
+The entire data quality module, including monitoring and data manipulation components, has been removed. This includes:
+
+- **Monitoring Components:** CheckValueRanges, FlatlineDetection, IdentifyMissingDataInterval, IdentifyMissingDataPattern, MovingAverage, GreatExpectationsDataQuality
+- **Data Manipulation Components:** DimensionalityReduction, DuplicateDetection, FlatlineFilter, GaussianSmoothing, IntervalFiltering, KSigmaAnomalyDetection, MissingValueImputation, OutOfRangeValueFilter
+- **Normalization Components:** NormalizationZScore, NormalizationMean, NormalizationMinMax, Denormalization
+- **Base Interfaces:** InputValidator, DataManipulationBaseInterface, MonitoringBaseInterface
+
+**Impact:** Applications using these components will need to be refactored to use alternative solutions or upgraded to implement data quality checks independently.
+
+#### 2. Forecasting Module (`rtdip_sdk.pipelines.forecasting`)
+The entire forecasting module has been removed. This includes:
+
+- **Time Series Forecasting:** ArimaPrediction, ArimaAutoPrediction
+- **Machine Learning Models:** LinearRegression, KNearestNeighbors, DataBinning
+- **Base Interface:** MachineLearningInterface
+
+**Impact:** Applications using ARIMA, AutoARIMA, or other forecasting models will need to migrate to external forecasting libraries or implement their own models.
+
+#### 3. Associated Tests
+All test files for the removed modules have been removed from the test suite, including:
+- 23 data quality tests
+- 6 forecasting tests
+- 1 logging integration test (`test_log_collection.py`)
+
+### Removed Package Dependencies
+The following packages, which were only used by the removed modules, have been removed from dependencies:
+
+- `statsmodels>=0.14.1` - Used by ARIMA models
+- `pmdarima>=2.0.4` - Used by AutoARIMA
+- `great-expectations>=0.18.8` - Used by data quality monitoring
+- `scikit-learn>=1.3.0` - Used by machine learning components
+
+### Deprecated: Turbodbc Support
+
+**Turbodbc is no longer inherently supported in RTDIP SDK (as of v0.14.5).** The connector is maintained for backward compatibility, but is deprecated.
+
+#### Migration Options
+If you require turbodbc connectivity, you have two choices:
+
+**Option 1: Use an older RTDIP version**
+```bash
+pip install "rtdip-sdk<=0.14.4"
+```
+
+**Option 2: Manual Installation**
+Install turbodbc manually in your environment:
+```bash
+pip install "turbodbc"
+```
+
+#### Alternative Connectors
+For new projects, we recommend using:
+- **DatabricksSQLConnection** (Default and recommended)
+- **PYODBCSQLConnection** (Lightweight ODBC alternative)
+- **SparkConnection**
+
+## Affected Components & Migration Guidance
+
+### If You Were Using Data Quality Monitoring
+**Previous Approach:**
+```python
+from rtdip_sdk.pipelines.data_quality.monitoring.spark import IdentifyMissingDataInterval
+```
+
+**Migration:**
+- Implement custom monitoring logic
+- Use external data quality frameworks (e.g., Great Expectations directly)
+- Implement monitoring in your data pipeline transforms
+
+### If You Were Using Data Quality Data Manipulation
+**Previous Approach:**
+```python
+from rtdip_sdk.pipelines.data_quality.data_manipulation.spark import MissingValueImputation
+```
+
+**Migration:**
+- Use PySpark SQL and DataFrame operations directly
+- Implement transform logic in pipeline steps
+- Consider external libraries for specialized operations
+
+### If You Were Using Forecasting
+**Previous Approach:**
+```python
+from rtdip_sdk.pipelines.forecasting.spark import ArimaAutoPrediction
+```
+
+**Migration:**
+- Use external forecasting libraries:
+  - [statsmodels](https://www.statsmodels.org/) for ARIMA/AutoARIMA
+  - [prophet](https://facebook.github.io/prophet/) for time series
+  - [scikit-learn](https://scikit-learn.org/) for ML models
+- Implement forecasting in separate pipeline steps or services
+
+### If You Were Using Turbodbc
+**Previous Approach:**
+```python
+from rtdip_sdk.connectors import TURBODBCSQLConnection
+```
+
+**Migration:**
+```python
+# Option 1: Use PYODBC instead (recommended)
+from rtdip_sdk.connectors import PYODBCSQLConnection
+
+# Option 2: Continue with turbodbc (manual installation)
+# Install turbodbc
+# No code changes needed, deprecation warnings will appear
+```
+
+## When to Update
+
+We recommend updating to v0.14.5 if:
+- You are not using data quality, forecasting, or turbodbc features
+- You are ready to migrate to alternative solutions for removed functionality
+- You want the latest RTDIP updates and improvements
+
+**Delay updating if:**
+- You heavily rely on data quality monitoring/manipulation
+- You use ARIMA/forecasting extensively
+- You require turbodbc connectivity
+
+For delayed updates, continue using your current RTDIP version that includes these features, or implement them as external components.
+
+## Documentation & Resources
+
+For more information on the affected modules and alternatives:
+
+- **Available Connectors:** [Connector Documentation](../../sdk/queries/connectors.md)
+- **Authentication:** [Azure Authentication Guide](../../sdk/authentication/azure.md)
+- **Installation:** [Getting Started](../../getting-started/installation.md)
+- **Removed Components Documentation:** [DATA_QUALITY_REMOVAL.md](https://github.com/rtdip/core/blob/develop/DATA_QUALITY_REMOVAL.md)
+
+## Support & Questions
+
+If you have questions or concerns about these changes:
+
+1. **Update your installation guide:** See [Installation Documentation](../../getting-started/installation.md) for the latest requirements
+2. **Open an Issue:** Visit [GitHub Issues](https://github.com/rtdip/core/issues) to discuss migration paths
+3. **Check Migration Guides:** Look for updated documentation on connector alternatives
+
+## Looking Forward
+
+These changes allow RTDIP to:
+- Focus on core time-series data ingestion and querying capabilities
+- Maintain lean, focused dependencies
+- Provide better performance and maintainability
+- Allow users to choose specialized tools for data quality and forecasting
+
+We appreciate your understanding and are committed to supporting your migration journey.
+
+---
+
+**Questions?** 
+- Open an issue on [GitHub](https://github.com/rtdip/core/issues)
+- Check the [documentation](../../sdk/overview.md)

docs/blog/posts/enhancing_data_quality_amos.md

@@ -8,7 +8,7 @@ authors:
 
 <center>
 
-![blog](../images/agile.svg){width=60%}
+<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/agile.svg" width="60%" alt="blog" />
 <small>1</small>
 </center>
 
@@ -61,9 +61,9 @@ To illustrate this visually, plotting the before-and-after DataFrames reveals th
 
 <center>
 
-![blog](../images/amos_mvi_raw.png){width=70%}
+<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/amos_mvi_raw.png" width="70%" alt="blog" />
 
-![blog](../images/amos_mvi.png){width=70%}
+<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/amos_mvi.png" width="70%" alt="blog" />
 
 </center>
 

docs/blog/posts/opcua_to_delta.md

@@ -19,7 +19,7 @@ This Historian system was developed as an ecosystem that provided a comprehensiv
 
 Hyperscalers have become a central part of global IT infrastructure and essential to operations and they also started providing industrial connectivity options out of many in this article I am going to focus on Azure , OPC UA & Open source stack which provides you all the options of data ingestion from OPC UA source to your cloud database.
 
-![blog](../images/opcua-to-deltalake.png){width=100%}
+<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/opcua-to-deltalake.png" width="100%" alt="blog" />
 </center>
 
 ## Components

docs/blog/posts/rtdip_data_quality.md

@@ -7,7 +7,9 @@ authors:
 # Ensuring Data Quality at Speed with Real Time Data
 
 <center>
-![DataQualityImage](../images/data-quality.png){width=75%} 
+
+<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/data-quality.png" width="75%" alt="Data Quality" />
+
 </center>
 
 High quality data plays a pivotal role in business success across various dimensions. Accurate and reliable data empowers business leaders to make well informed decisions and achieve operational efficiency, promoting growth and profitability. Data quality encompasses more than just accuracy it also includes completeness, consistency, and relevance. 

docs/blog/posts/rtdip_energy_forecasting.md

@@ -7,7 +7,7 @@ authors:
 # Energy Forecasting: Utilising the Power of Tomorrow’s Data
 
 <center>
-![EnergyForecastingImage](../images/energy-forecasting.png){width=75%} 
+<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/energy-forecasting.png" width="75%" alt="Energy Forecasting" /> 
 </center>
 
 Energy forecasting plays a pivotal role in our modern world, where energy consumption, production and pricing are critical factors. 

docs/blog/posts/rtdip_ingestion_pipelines.md

@@ -7,7 +7,7 @@ authors:
 <center>
 # RTDIP Ingestion Pipeline Framework
 
-![blog](../images/framework.png){width=60%}
+<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/framework.png" width="60%" alt="framework" />
 </center>
 
 RTDIP has been built to simplify ingesting and querying time series data. One of the most anticipated features of the Real Time Data Ingestion Platform for 2023 is the ability to create streaming and batch ingestion pipelines according to requirements of the source of the data and needs of the data consumer. Of equal importance is the need to query this data and an article that focuses on egress will follow in due course. 
@@ -62,7 +62,7 @@ As per the above, a pipeline job consists of a list of tasks. Each task consists
 
 |Python|Apache Spark|Databricks|Delta Live Tables|
 |---------------------------|----------------------|--------------------------------------------------|-------|
-|![python](../images/python.png){: .image-center}|![pyspark](../images/apachespark.png){: .image-center}|![databricks](../images/databricks_horizontal.png){: .image-center}|![delta](../images/dlt.svg){: .image-center}
+|<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/python.png" alt="python" class="image-center" />|<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/apachespark.png" alt="pyspark" class="image-center" />|<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/databricks_horizontal.png" alt="databricks" class="image-center" />|<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/dlt.svg" alt="delta" class="image-center" />|
 
 Pipelines will be able to run in multiple environment types. These will include:
 
@@ -79,7 +79,7 @@ Runtimes will take precedence depending on the list of components in a pipeline
 
 ## Pipeline Clouds
 
-![azure-aws-gcp](../images/aws-azure-gcp.png){: .image-center}
+<img src="https://raw.githubusercontent.com/rtdip/core/develop/docs/blog/images/aws-azure-gcp.png" alt="aws-azure-gcp" class="image-center" />
 
 Certain components are related to cloud providers and in the tables below, it is indicated which cloud provider is related to its specific component. It does not mean that the component can only run in that cloud, instead its highlighting that the component is related to that cloud provider.