Moving from docker compose to helm

[atb1r21]

Hi,

I have attempted to move from the Docker compose style local installation to helm today.

I've had a series of issues and have not yet been able to get seqr to start.

The main pod in Kubernetes is experiencing a stack trace, related to clickhouse migrations:

Running migrations:
Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/clickhouse_driver/dbapi/cursor.py", line 111, in execute
response = execute(
^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/clickhouse_backend/driver/client.py", line 53, in execute
rv = self.process_ordinary_query(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/clickhouse_driver/client.py", line 580, in process_ordinary_query
return self.receive_result(with_column_types=with_column_types,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/clickhouse_driver/client.py", line 212, in receive_result
return result.get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/clickhouse_driver/result.py", line 50, in get_result
for packet in self.packet_generator:
File "/opt/venv/lib/python3.11/site-packages/clickhouse_driver/client.py", line 228, in packet_generator
packet = self.receive_packet()
^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/clickhouse_driver/client.py", line 245, in receive_packet
raise packet.exception
clickhouse_driver.errors.ServerException: Code: 57.
DB::Exception: Table seqr.GRCh38/SNV_INDEL/reference_data/pext/all_variants already exists. Stack trace:

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f2bff3b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009c74bec
2. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009c745ab
3. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_deleteDB::DDLGuard>&, DB::LoadingStrictnessLevel) @ 0x00000000131b853b
4. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x00000000131aa882
5. DB::InterpreterCreateQuery::execute() @ 0x00000000131be038
6. DB::executeQueryImpl(char const*, char const*, std::shared_ptrDB::Context, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptrDB::IAST&) @ 0x0000000013597788
7. DB::executeQuery(String const&, std::shared_ptrDB::Context, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000013590e86
8. DB::TCPHandler::runImpl() @ 0x00000000148f12be
9. DB::TCPHandler::run() @ 0x000000001490ffb9
10. Poco::Net::TCPServerConnection::start() @ 0x0000000018017407
11. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018017859
12. Poco::PooledThread::run() @ 0x0000000017fe2b7b
13. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000017fe105d
14. ? @ 0x00007fa1dd7791f5
15. ? @ 0x00007fa1dd7f8b00

Furthermore, I am unable to revert to the hold methodology after doing this attempted upgrade today, as the older docker-compose containers will start but nothing is serving on the Seqr frontend.

Would it be possible to get some advice?

Thanks

[bpblanken]

Hi @atb1r21

It looks like you might have a partially completed migration; that table is created in 0024_pext_reference_data. My suggestion is to check what's contained in the django_migrations table of the seqr database (USE DATABASE seqr; then select * FROM django_migrations where app='clickhouse_search';) after logging into ClickHouse with your username and password.

Unfortunately this migration is not idempotent, so you will need to DELETE all pext tables and rollback your migrations to the previous (0023) migration.

[atb1r21]

Hi,

Rather than try a migration, I attempted this from scratch with an empty /var/seqr directory. The issue recurred, could this be a bug?

Thanks,
Alister

[atb1r21]

Hi,

I have instantiaed a whole new VM for this, to ensure no lurking config or other issues.

From a fresh installation once I deploy the various Kubernetes pods:

NAME READY STATUS RESTARTS AGE
pipeline-runner-api-59f7c8776f-qn2fs 0/2 Pending 0 6m53s
pipeline-runner-ui-6b6d68f86b-bqkrt 1/1 Running 0 6m53s
seqr-b9565f958-psnfq 0/1 CrashLoopBackOff 4 (12s ago) 6m53s
seqr-check-new-samples-job-29526440-wd6v6 1/1 Running 0 5s
seqr-clickhouse-shard0-0 1/1 Running 0 6m53s
seqr-postgresql-0 1/1 Running 0 6m53s
seqr-redis-master-0 1/1 Running 0 6m53s
seqr-update-all-reference-data-29526435-x4s4g 1/1 Running 0 5m5s

The seqr container gets stuck at this stage once again:

django.db.utils.OperationalError: Code: 57.
DB::Exception: Table seqr.GRCh38/SNV_INDEL/reference_data/pext/all_variants already exists. Stack trace:

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f2bff3b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009c74bec
2. DB::Exception::Exception<String, String>(int, FormatStringHelperImpl<std::type_identity::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009c745ab
3. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_deleteDB::DDLGuard>&, DB::LoadingStrictnessLevel) @ 0x00000000131b853b
4. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x00000000131aa882
5. DB::InterpreterCreateQuery::execute() @ 0x00000000131be038
6. DB::executeQueryImpl(char const*, char const*, std::shared_ptrDB::Context, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptrDB::IAST&) @ 0x0000000013597788
7. DB::executeQuery(String const&, std::shared_ptrDB::Context, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000013590e86
8. DB::TCPHandler::runImpl() @ 0x00000000148f12be
9. DB::TCPHandler::run() @ 0x000000001490ffb9
10. Poco::Net::TCPServerConnection::start() @ 0x0000000018017407
11. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018017859
12. Poco::PooledThread::run() @ 0x0000000017fe2b7b
13. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000017fe105d
14. ? @ 0x00007fecdb00a1f5
15. ? @ 0x00007fecdb089b00

This is from an empty directory, additionally I have followed your advice and access the clichouse pod, dropped the pext dictionary and those tables.

When the seqr pod launches again it tries the same process and the issue recurs.

I think I have found a bug.

It occurs when this runs, specifically it locks up at migration 24.

python -u manage.py migrate --database=clickhouse_write

This happens from a clean directory, from trying to tidy up pext, and additionally trying to debug the seqr container is difficult due to the fact that it constantly crashes.

I'm on RHEL 9 and have followed the instructions to the letter, and I can see connections between the containers and updates happening to the clickhouse database, it looks like there's something wrong with this migration step, and the container gets restarted and it tries again infinitely, due to the fact that a pext is created, but never completes.

If you have any advice I'd be grateful.

[bpblanken]

Thank you for the detailed report! I will take a look and see if I can reproduce.

[bpblanken]

Hi @atb1r21,

I've reproduced and debugged the issue. It's a race condition between the seqr pod startup and the pipeline-runner startup; if you are creating a seqr-platform for the first time the pipeline-runner will block on downloading reference data and the seqr migrations (which queue jobs on the pipeline runner) will fail.

I'm in the process of updating the pipeline-runner initialization to not block on syncing reference data; this should be released within the next day or so.

Thank you for the patience!

[atb1r21]

Hi,

Thanks for your response and quick action.

I'll try again later this week, once you've released the fix.

All the best.

[atb1r21]

Hi,

I tried this on Friday but it still hit the same error, has this been resolved yet or are you still working on the issue?

Thanks.

[bpblanken]

Hi @atb1r21,

I'd make sure you've upgraded your helm installation to the latest seqr-platform version (3.1.0), or at the very least the last supported 2.x version (2.19.3-annotations-final).

The commands for this are:

helm repo update
helm upgrade YOUR_INSTITUTION_NAME-seqr seqr-helm/seqr-platform --version 3.1.0

If you are able to verify that the pipeline-runner pod comes up quickly (under 3 min), the reference-data-sync inits are CronJobs instead of initContainers, and you still encounter this issue, feel free to ping again on this ticket.