ClickHouse loader fails on NFS storage - renameat2() not supported

[vangravs]

The Problem:

1. Hard Dependency on renameat2() System Call

The seqr v2.6.7 code uses REFRESH EVERY X materialized views which require atomic renames via renameat2(). This system call is NOT supported on NFS, which is a extremely common
enterprise storage solution. This is a fundamental compatibility issue.

2. No Graceful Degradation

The code has no fallback mechanism. It should:

• Detect if the filesystem supports renameat2()
• Fall back to regular materialized views (which work fine and auto-update)
• Provide a clear error message instead of cryptic stack traces

3. Silent Failures

The clickhouse-loader daemon fails silently every 5 minutes, creating _CLICKHOUSE_LOAD_FAIL markers without any user-visible notification in the UI. Users don't know data loading
failed unless they dig through:

• Command-line marker files
• ClickHouse pod logs
• Pipeline runner logs

Potential fix:

 def refresh_materialized_views(...):
      for mv in materialized_views:
          try:
              # Try REFRESH-based approach
              logged_query(f"SYSTEM REFRESH VIEW {mv}")
          except Exception as e:
              if "UNSUPPORTED_METHOD" in str(e) or "renameat2" in str(e):
                  # Filesystem doesn't support atomic renames
                  # Fall back to regular MVs (no-op, they auto-update)
                  logger.warning(f"REFRESH views not supported on this filesystem, using auto-updating MVs")
                  continue
              raise

[bpblanken]

Hi @vangravs,

Thank you for reporting this; we have not thoroughly tested ClickHouse (or seqr in general) on NFS, so support is unfortunately minimal. It is a hole in our documentation that we do not link to filesystem support documentation here and more thoroughly here.

The hard dependency is likely internal within ClickHouse and will be challenging to work around. We are using the refreshable materialized views rather than the default increment materialized views for an explicit application logic reason- they can be fully rebuilt from scratch. From my understanding, the two kinds of materialized views are not particularly compatible; you must choose one approach or the other at table creation time. I will assign this ticket to me and investigate some potential solutions.

Regarding the debugging process, an issue like this will require digging through logs unfortunately. As the loading process becomes more polished we will try to distribute errors better within the user interface.

[bpblanken]

There are some details within Clickhouse tickets:

• Replace table statement is not working as expected ClickHouse/ClickHouse#49339
• When run in Docker REPLACE is failing with DB::Exception: System call renameat2() is not supported. ClickHouse/ClickHouse#37583
• Anybody tried Amazon EFS as backend storage for ClickHouse? ClickHouse/ClickHouse#31113

My gut feeling is it will be very challenging for us to support ClickHouse on NFS; but might be able to support NFS within other parts of seqr. The loading pipeline (for VCFs and intermediate storage) is likely where it will be most useful.

[bpblanken]

As an additional follow up, I've slotted some work to better integrate the clickhouse loader with the rest of the Luigi Pipeline, so errors will at least propagate within that user interface as a failed task. The Luigi UI leaves much to be desired, but it is at least something.