E2E flakiness due to syncing to a FHIR server in the FHIR-Search mode

[bashir2]

Another source of e2e flakiness is this step: "Run E2E Test for FHIR-search mode with HAPI source". Below is a sample log which shows that not all observations are seen in the sink server. This is probably an issue with HAPI indices being updated late. The log is from this run:

Already have image: us-docker.pkg.dev/cloud-build-fhir/fhir-analytics/e2e-tests:c4cc2fa
Checking if the Parquet-tools JAR exists...
Parquet-tools JAR exists in /workspace/e2e-tests/controller-spark
E2E TEST: ---- STARTING FHIR_SEARCH_HAPI TEST ----
E2E TEST: Total FHIR source test patients ---> 79
E2E TEST: Total FHIR source test encounters ---> 4006
E2E TEST: Total FHIR source test obs ---> 17279
E2E TEST: Counting number of patients, encounters and obs sinked to parquet files
E2E TEST: Total patients synced to parquet ---> 79
E2E TEST: Total encounters synced to parquet ---> 4006
E2E TEST: Total obs synced to parquet ---> 17279
E2E TEST: Parquet Sink Test Non-Streaming mode
E2E TEST: Total patient flat rows synced to parquet ---> 106
E2E TEST: Total encounter flat rows synced to parquet ---> 4006
E2E TEST: Total observation flat rows synced to parquet ---> 17279
E2E TEST: PARQUET SINK EXECUTED SUCCESSFULLY USING FHIR_SEARCH_HAPI MODE
E2E TEST: Finding number of patients, encounters and obs in FHIR server
E2E TEST: Counting number of patients, encounters and obs sinked to fhir files
E2E TEST: Total patients sinked to fhir ---> 79
E2E TEST: Total encounters sinked to fhir ---> 4006
E2E TEST: Total observations sinked to fhir ---> 17273
E2E TEST: FHIR SERVER SINK TEST FAILED USING FHIR_SEARCH_HAPI MODE

[shanjare2002]

can i take this

[bashir2]

Thanks @shanjare2002 for your interest; sure you can take this if you like. Just a heads-up that if you have not dealt with Google Cloud Build before, it might be a little bit challenging to get started.

For now I assigned this to you. Please take a look and feel free to unassign it if you decid not to work on it.

[CodeGovindz]

hey @bashir2 can I work on it ?