SAP HANA Provider Package

[jtakish]

Would there be any interest in a provider package to support SAP HANA? I use airflow to interact with a SAP HANA database and I have hooks/operators written to use for work. DbApiHook claims to offers support for SAP HANA as evidenced by the docstring in the generate_insert_sql method but none of the methods will work with HANA as is, most of them need to be overridden.

It would not be too much work for me to make the hooks/operators that I have written public

[dabla]

We actually use SAP Hana, but will ditch it very soon, with the SQL hooks and operators without any issues. For the replace/upsert statements you only need to pass the 'replace_statement_format' hook parameter, and it should work.

transfer_table_task = GenericTransfer(
        task_id="transfer_table",
        source_conn_id="jdbc_hana_conn",
        destination_conn_id="jdbc_hana_conn",
        destination_hook_params={"replace_statement_format": "UPSERT {} {} VALUES ({}) WITH PRIMARY KEY"},
        insert_args={
            "commit_every": "5000",
            "executemany": True,
            "fast_executemany": True,
            "replace": True,
        },
        dag=dag,
        sql="SELECT * FROM STG_USERS",
        destination_table="USERS",
)

[dabla]

There is also another PR in progress, which introduces the notion of dialects which is independant of the connection type you want to use (native/odbc/jdbc/...)

[dabla]

But we managed to use SAP Hana in Airflow without any modifications, except the ones we already merged into Airflow regarding DbApiHook and GenericTransfer operator.

[jtakish]

It appears that you are interacting with HANA using JDBC and not hdbcli, correct? hdbcli is the official SAP cursor package and only dialect for sqlalchemy. Without using hdbcli, you can't create a sqlalchemy engine, or an inspector, and many operations will be executed sub-optimally

[dabla]

Yes you can and we actually do it ar our company, I’ve done a PR in the past so you can use sqlalchemy with SAP Hana through jdbc

[Joffreybvn]

Side note: If you use sqlalchemy-hana, beware that this not an officially supported SAP product (as stated in their README). Which means SAP can decide to yank all the releases on a friday afternoon (like they did for pyrfc).

[jtakish]

I have mostly moved away from sqlalchemy for HANA as hdbcli works much better but there are a few things (reflection, inspection) from sqlalchemy that are incredibly convenient.

Interestingly, the SAP documentation does direct you over to sqlalchemy, I do not recall them ever acknowledging SA in the past. I feel like they used to push hana-ml, which was their attempt at making their own proprietary pandas, which is very on-brand for SAP.

https://help.sap.com/docs/HANA_SERVICE_CF/1efad1691c1f496b8b580064a6536c2d/01e93e584e524747b570cd9083b08d2b.html

[potiuk]

And this is one of the reason we in Airflow community are reluctant to accept providers that are 3rd-party and the 3rd-party is not committed to supporting it. We do not want to deal with things like that and promise our users things we cannot promise.

[potiuk]

@j6takish -> generally approach we have in the community that we are reluctant to maintain something that is used by a handful of our users (or maybe even few handfulls - but not really massive numbers) - and where we - the apache community built around Airlfow - do not have a lot of experience with.

We have very high expectations on when we accept new providers for new technologies - simply because we do not want to increase maintenance burden on us - Airflow is developed by most people in their free time, and maintainers are also volunteers and the maintenanc effort to review, merge, fix etc. is on them (while some of them are paid to do other stuff, they are not necessarily paid to maintain technoloigy X or Y integration with Airflow - so they do it effectively in their own free time).

That's why we have not only guidelines but also very strict expectations on accepting new providers to be maintained here in the community - I recommend you to read them https://github.com/apache/airflow/blob/main/PROVIDERS.rst#community-managed-providers - there are also some examples of discussions and previous attempts (both successful and unsuccessful) of submitting providers to the community here. It also explains what are the limitations of having comunity provider.

One of the important limitations is that the community has to have an easy way to test such provider when things change - if that requires external service like SAP/HANA, that someone must also provide "system test dashboard" where system tests are run against such external service (and pay for it) - and provide public dashboard to the community so that release manager can see that the system tests passed.

You can look at the past discussions and decide yourself if you want to go that road - eventually any provider will have to be accepted by formal VOTE on the devlist, after initiall discussion there.

And you are also absolutely free to go another route - providers do not have to live here - in airlfow repo, in fact many of them don't https://airflow.apache.org/ecosystem/#third-party-airflow-plugins-and-providers - you can easily package and release your own provider in the way and manner you think it is best - you can also add link to it at our "ecosystem" page - we will merge such change gladly.

So it's really up to you to see what road you think is best for such SAP/HANA provider.

[jtakish]

Third party is definitely the way to go as I can't imagine there being a ton of HANA/airflow users, I just wanted to gauge the demand for a HANA provider package.

[potiuk]

Posted it in a wrong thread :). Moved the comment to where it belongs

[potiuk]

Yeah. I think 3rd-party is the way to go too

[potiuk]

Ideally SAP should maintain their own provider :D

[TsubasaBE]

I'm interested in an official HanaDb provider. I'm at the verge of having to implement one myself for our Airflow instance.

[potiuk]

I'm interested in an official HanaDb provider. I'm at the verge of having to implement one myself for our Airflow instance.

Feel free. As explained above we encourage people who are interested in new providers to manage and maintain them outside of Airflow official repo, because we have no expertise and we do not want to carry the burden of maintaining such providers unless they prove to be highly popular. So "at the verge of having to implement one myself" is the way to go.

[jtakish]

I published mine. It's linked on the Airflow ecosystem page. Very interested in feedback about missing features etc. Works for everything I need to do internally but I'm sure there are use cases I didn't consider.

[TsubasaBE]

@j6takish
Oh. Great! I'll check it out. Thanks.

[TsubasaBE]

@j6takish
Your provider works perfectly! Thanks for the package. You are a life saviour.
I'll star your repo.
Thanks again.

[venkatamandavilli-code]

Hi @j6takish — really glad you published this provider. I wanted to add
enterprise context to this discussion that might be helpful for anyone
evaluating the SAP HANA provider for large-scale deployments.

Background
I'm an SAP S/4HANA Project Manager with 21+ years of ERP implementation
experience. I've managed data migration pipelines and batch job orchestration
across five major S/4HANA implementations including:

• PepsiCo (worldwide plants, 100,000+ users, $91.85B revenue)
• Levi Strauss (multi-country SAMEA + Asia Pacific rollout, 10,000+ users)
• Compass Group (3-plant migration, Europe's largest foodservice company)
• King's Hawaiian ($50M implementation budget)

In each of these projects, nightly batch workflows, data validation
pipelines, and multi-phase migration sequences were central to the
implementation — exactly the scenarios where an Airflow + SAP HANA
provider would be most valuable.

Enterprise Use Cases Not Yet Covered in Documentation
Based on my production experience, here are the SAP HANA pipeline
scenarios that enterprise teams will need support for:

1. Multi-Plant Batch Orchestration — Running parallel DAGs per
   SAP plant/company code, each connecting to a different SAP HANA
   schema, with dependency gates between regional go-lives
   (e.g., Americas runs before EMEA cutover begins)

2. Legacy ECC → S/4HANA Migration Pipelines — Extracting from
   SAP ECC 6.0 source tables, transforming (cleansing, deduplication,
   enrichment), and loading into SAP S/4HANA target structures —
   specifically for OTC (VBAK/VBAP), FICO (BKPF/BSEG), and
   PTP (EKKO/EKPO) data objects

3. Delta/Incremental Load Patterns — Using SAP HANA Change Data
   Capture (CDC) or timestamp-based delta extraction to sync data
   incrementally between SAP and downstream analytics platforms
   (Airbyte, Superset, Metabase)

4. Go-Live Cutover Sequencing — Using Airflow SLA sensors to
   gate cutover steps — e.g., only proceed to SAP production
   activation after data migration validation DAG returns a
   success signal with zero error records

5. Post Go-Live Monitoring Pipelines — Scheduling automated
   data quality checks (completeness, duplicate detection,
   reconciliation) against SAP HANA in the weeks following go-live
   across multiple organizational units

My Offer
I'd like to contribute to the existing provider by:

• Writing enterprise-focused documentation covering the above use cases
• Adding a SAP HANA connection setup guide specifically for
  SAP HANA Cloud (SCP) vs. on-premise deployments
• Contributing DAG examples for the migration and batch
  orchestration patterns above

@j6takish — is there a preferred way to contribute to your provider
repo? Happy to submit

[jtakish]

I am the sole contributor, the project has 3 stars on GitHub, and I average ~100 downloads per week which are probably mostly bots. I think there is a small to medium sized demand for a HANA Airflow Hook, given how many spam emails I get about HANA/Airflow contract roles, but I don't have the time or wherewithal to promote my hobby project on social media. You can clone the repo and submit PR's and we can discuss any other features over there. Any feedback on existing features from real actual humans would be welcome, a lot of the features are "things I do in my personal workflow" and not much "things other people do in their workflows".

I will say, I think most of your suggestions are outside the scope of an Airflow hook class. I give you the ability to use Airflow operators with HANA, the rest is sort of up to the person writing the DAGs. With the exception of maybe number 2, you could do everything on your list using the hook class as is.

People migrate onto and off of MSSQL and Postgres all day every day. Those provider packages don't have any niche use cases built into them, they enable you to fetch records, upsert, etc.

I do think I need some help with the connection setup guide, particularly for cloud. My workplace used on-prem HANA so that is what I focused on.

[dabla]

We’ve had been using SAP HANA for years with just a JdbcHook and standard Airflow SQL operators without any issues. Using the SAP ODBC driver wasn’t feasible due to persistent bugs and random character encoding problems—it worked intermittently at best.

Keep in mind that, in many environments, direct access to SAP HANA via ODBC or JDBC may be restricted due to licensing limitations.

[jtakish]

Yeah you can do mostly everything with JDBC or ODBC and it all somewhat works but not as well as using the native connectors. HANA is always the black sheep, they have their own Tableau connector for example. Using a hdbcli based hook removes a lot of friction and frustration. I used HANA primarily at work for a long time and connecting with JDBC, and ODBC especially, added a low level simmering layer of rage to my workday. You can eat soup with a fork but it's much less painful to use a spoon.

Unrelated, SAP does not recommend ODBC but you can fix the character issues by setting the CHAR_AS_UTF8=TRUE keyword arg on your ODBC driver.

[dabla]

Then indeed a dedicated HANAHook would make sense.

[jtakish]

This might not be the place for it, let me know if you want me to move it to a different thread.

I was looking at the code for the GenericTransfer operator today. It uses fetchall and exponentially growing LIMIT + OFFSET to batch data. The newest version in Airflow 3.2.2, uses deferrals and async to run all of the statements concurrently. Each statement triggers a full table scan, the rows have to get ordered, and then all rows not in the LIMIT get discarded.

I have a get_rows_by_chunk method in the HANA hook. It yields batches from a generator.

    def get_records_by_chunks(
        self, sql: str, parameters: Iterable[Any] | Mapping[str, Any] | None = None, chunksize: int = 10000
    ) -> Iterator[tuple[Any, ...] | list[tuple[Any, ...]]]:
        """
        Streams records from SAP HANA, yielding chunks of rows.

        This method allows for fetching large datasets without loading them all
        into memory. Each record is passed through ``_make_common_data_structure``
        to ensure JSON serialization. The ``descriptions`` and ``last_description``
        attributes are available immediately after execution.

        :param sql: The SQL statement.
        :param parameters: Parameters to bind to the SQL statement.
        :param chunksize: The number of records per chunk.
        :return: A generator yielding lists of tuples (or a single tuple if chunksize is 1).
        """
        self.descriptions = []
        conn = None
        cur = None
        try:
            conn = self.get_conn()
            cur = conn.cursor()
            self._run_command(cur, sql, parameters)
            self.descriptions.append(cur.description)
        except Exception as e:
            if cur:
                cur.close()
            if conn:
                conn.close()
            raise e
        return chunk_handler(self, conn, cur, chunksize)

def chunk_handler(
    hook: T, conn: Any, cursor: Any, chunksize: int
) -> Iterator[tuple[Any, ...] | list[tuple[Any, ...]]]:
    """
    Yield rows in batches.

    This allows for processing large datasets without loading all data into memory.
    The ``descriptions`` and ``last_description`` attributes of the hook are
    available immediately after calling this method.

    :param hook: The ``DbApiHook`` instance.
    :param conn: The database connection object.
    :param cursor: The database cursor.
    :param chunksize: The number of records to return per chunk.
    :return: A generator yielding lists of tuples.
    """
    nb_rows = 0
    make_common_data_structure = getattr(hook, "_make_common_data_structure")
    log = getattr(hook, "log")
    try:
        while results := make_common_data_structure(fetch_many_handler(cursor, chunksize)):
            nb_rows += len(results)
            log.info("Fetched %s rows so far", nb_rows)
            yield results
        else:
            log.info("Done fetching. Fetched %s total rows", nb_rows)
    finally:
        if cursor:
            cursor.close()
        if conn:
            conn.close()

I think there is a way to maybe improve the generic transfer using a similar logic? I realize the generic transfer is intended to be used only for "results that fit into memory" but that is a little ambiguous. Also using a generator would essentially remove this restriction.