[Bug] Failure to fetch all resources from HAPI FHIR server

[Rkareko]

The FHIR Data Pipes is only fetching 78 patients records out of a total of 3370 from a staging server. The FHIR Pipelines Control Panel does not show any errors when the Full or Incremental runs are triggered.

The following error log was found in the fhir-data-pipes/docker/dwh logs

java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Pipeline execution failed
	at com.google.fhir.analytics.EtlUtils.runMultiplePipelines(EtlUtils.java:171)
	at com.google.fhir.analytics.EtlUtils.runMultipleMergerPipelinesWithTimestamp(EtlUtils.java:111)
	at com.google.fhir.analytics.PipelineManager$PipelineThread.run(PipelineManager.java:754)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Pipeline execution failed
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
	at com.google.fhir.analytics.EtlUtils.runMultiplePipelines(EtlUtils.java:158)
	... 2 more
Caused by: java.lang.RuntimeException: Pipeline execution failed
	at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:107)
	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:310)
	at com.google.fhir.analytics.EtlUtils$PipelineSupplier.get(EtlUtils.java:192)
	at com.google.fhir.analytics.EtlUtils$PipelineSupplier.get(EtlUtils.java:182)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
	at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
	at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:258)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
	at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
	at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
	at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
	at akka.dispatch.OnComplete.internal(Future.scala:300)
	at akka.dispatch.OnComplete.internal(Future.scala:297)
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
	at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
	at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
	at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
	at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
	at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:252)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:242)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:233)
	at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:684)
	at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
	at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:444)
	at jdk.internal.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
	at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
	at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
	at akka.actor.Actor.aroundReceive(Actor.scala:537)
	at akka.actor.Actor.aroundReceive$(Actor.scala:535)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
	at akka.actor.ActorCell.invoke(ActorCell.scala:548)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
	at akka.dispatch.Mailbox.run(Mailbox.scala:231)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
	... 5 more
Caused by: org.apache.beam.sdk.util.UserCodeException: org.apache.flink.runtime.operators.chaining.ExceptionInChainedStubException: Exception in chained task 'FlatMap (FlatMap at ParquetIO.ReadFiles/ParDo(SplitRead)/ParMultiDo(SplitRead)/ProcessKeyedElements/Drop key/Values/Map/ParMultiDo(Anonymous).output)'
	at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
	at org.apache.beam.sdk.transforms.MapElements$2$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
	at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.apache.beam.runners.flink.FlinkBatchTransformTranslators$ReshuffleTranslatorBatch.lambda$translateNode$baeabb50$1(FlinkBatchTransformTranslators.java:420)
	at org.apache.flink.api.java.operators.translation.PlanUnwrappingReduceGroupOperator$TupleUnwrappingNonCombinableGroupReducer.reduce(PlanUnwrappingReduceGroupOperator.java:120)
	at org.apache.flink.runtime.operators.GroupReduceDriver.run(GroupReduceDriver.java:145)
	at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:519)
	at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:360)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.flink.runtime.operators.chaining.ExceptionInChainedStubException: Exception in chained task 'FlatMap (FlatMap at ParquetIO.ReadFiles/ParDo(SplitRead)/ParMultiDo(SplitRead)/ProcessKeyedElements/Drop key/Values/Map/ParMultiDo(Anonymous).output)'
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:81)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction$DoFnOutputManager.output(FlinkDoFnFunction.java:221)
	at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:276)
	at org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:86)
	at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:432)
	at org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:89)
	at org.apache.beam.sdk.transforms.MapElements$2.processElement(MapElements.java:151)
Caused by: org.apache.beam.sdk.util.UserCodeException: org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in block 1 in file ReadableFile{metadata=Metadata{resourceId=/dwh/controller_DWH_TIMESTAMP_2025_06_12T11_56_57_763659295Z/VIEWS_TIMESTAMP_2025_06_13T05_25_52_780357915Z/patient_flat/ProcessGenericRecords_Patient_patient_flat_output-parquet-th-1027-ts-1749792381325-r-552323.parquet, sizeBytes=2593, isReadSeekEfficient=true, checksum=null, lastModifiedMillis=0}, compression=UNCOMPRESSED}
	at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
	at org.apache.beam.sdk.io.parquet.ParquetIO$ReadFiles$SplitReadFn$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.sdk.util.construction.SplittableParDoNaiveBounded$NaiveProcessFn.process(SplittableParDoNaiveBounded.java:360)
	at org.apache.beam.sdk.util.construction.SplittableParDoNaiveBounded$NaiveProcessFn$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:186)
	at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruningFunction.flatMap(FlinkMultiOutputPruningFunction.java:60)
	at org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruningFunction.flatMap(FlinkMultiOutputPruningFunction.java:35)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction$DoFnOutputManager.output(FlinkDoFnFunction.java:221)
	at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:276)
	at org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:86)
	at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:432)
	at org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:89)
	at org.apache.beam.sdk.transforms.MapElements$2.processElement(MapElements.java:151)
	at org.apache.beam.sdk.transforms.MapElements$2$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
	at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.apache.beam.runners.flink.FlinkBatchTransformTranslators$ReshuffleTranslatorBatch.lambda$translateNode$baeabb50$1(FlinkBatchTransformTranslators.java:420)
	at org.apache.flink.api.java.operators.translation.PlanUnwrappingReduceGroupOperator$TupleUnwrappingNonCombinableGroupReducer.reduce(PlanUnwrappingReduceGroupOperator.java:120)
	at org.apache.flink.runtime.operators.GroupReduceDriver.run(GroupReduceDriver.java:145)
	at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:519)
	at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:360)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in block 1 in file ReadableFile{metadata=Metadata{resourceId=/dwh/controller_DWH_TIMESTAMP_2025_06_12T11_56_57_763659295Z/VIEWS_TIMESTAMP_2025_06_13T05_25_52_780357915Z/patient_flat/ProcessGenericRecords_Patient_patient_flat_output-parquet-th-1027-ts-1749792381325-r-552323.parquet, sizeBytes=2593, isReadSeekEfficient=true, checksum=null, lastModifiedMillis=0}, compression=UNCOMPRESSED}
	at org.apache.beam.sdk.io.parquet.ParquetIO$ReadFiles$SplitReadFn.processElement(ParquetIO.java:827)
Caused by: org.apache.flink.runtime.operators.chaining.ExceptionInChainedStubException: Exception in chained task 'FlatMap (FlatMap at ParquetIO.ReadFiles/ParDo(SplitRead)/ParMultiDo(SplitRead).output)'
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:81)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction$DoFnOutputManager.output(FlinkDoFnFunction.java:221)
	at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:276)
	at org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:86)
	at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:432)
	at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:412)
	at org.apache.beam.sdk.util.construction.SplittableParDoNaiveBounded$NaiveProcessFn$NestedProcessContext$1.output(SplittableParDoNaiveBounded.java:528)
	at org.apache.beam.sdk.io.parquet.ParquetIO$ReadFiles$SplitReadFn.processElement(ParquetIO.java:823)
	at org.apache.beam.sdk.io.parquet.ParquetIO$ReadFiles$SplitReadFn$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.sdk.util.construction.SplittableParDoNaiveBounded$NaiveProcessFn.process(SplittableParDoNaiveBounded.java:360)
	at org.apache.beam.sdk.util.construction.SplittableParDoNaiveBounded$NaiveProcessFn$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:186)
	at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruningFunction.flatMap(FlinkMultiOutputPruningFunction.java:60)
	at org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruningFunction.flatMap(FlinkMultiOutputPruningFunction.java:35)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction$DoFnOutputManager.output(FlinkDoFnFunction.java:221)
	at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:276)
	at org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:86)
	at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:432)
	at org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:89)
	at org.apache.beam.sdk.transforms.MapElements$2.processElement(MapElements.java:151)
	at org.apache.beam.sdk.transforms.MapElements$2$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
	at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.apache.beam.runners.flink.FlinkBatchTransformTranslators$ReshuffleTranslatorBatch.lambda$translateNode$baeabb50$1(FlinkBatchTransformTranslators.java:420)
	at org.apache.flink.api.java.operators.translation.PlanUnwrappingReduceGroupOperator$TupleUnwrappingNonCombinableGroupReducer.reduce(PlanUnwrappingReduceGroupOperator.java:120)
	at org.apache.flink.runtime.operators.GroupReduceDriver.run(GroupReduceDriver.java:145)
	at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:519)
	at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:360)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.beam.sdk.util.UserCodeException: org.apache.flink.runtime.operators.chaining.ExceptionInChainedStubException: Exception in chained task 'FlatMap (FlatMap at ParDo(GroupViewIds)/ParMultiDo(GroupViewIds).output)'
	at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
	at com.google.fhir.analytics.ParquetMerger$GroupViewIds$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
	at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruningFunction.flatMap(FlinkMultiOutputPruningFunction.java:60)
	at org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruningFunction.flatMap(FlinkMultiOutputPruningFunction.java:35)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	... 47 more
Caused by: org.apache.flink.runtime.operators.chaining.ExceptionInChainedStubException: Exception in chained task 'FlatMap (FlatMap at ParDo(GroupViewIds)/ParMultiDo(GroupViewIds).output)'
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:81)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction$DoFnOutputManager.output(FlinkDoFnFunction.java:221)
	at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:276)
	at org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:86)
	at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:432)
	at org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:89)
	at com.google.fhir.analytics.ParquetMerger$GroupViewIds.processElement(ParquetMerger.java:461)
Caused by: org.apache.beam.sdk.util.UserCodeException: org.apache.flink.runtime.operators.chaining.ExceptionInChainedStubException: Exception in chained task 'FlatMap (FlatMap at CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable).output)'
	at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
	at org.apache.beam.sdk.transforms.join.CoGroupByKey$ConstructUnionTableFn$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
	at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruningFunction.flatMap(FlinkMultiOutputPruningFunction.java:60)
	at org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruningFunction.flatMap(FlinkMultiOutputPruningFunction.java:35)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction$DoFnOutputManager.output(FlinkDoFnFunction.java:221)
	at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:276)
	at org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:86)
	at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:432)
	at org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:89)
	at com.google.fhir.analytics.ParquetMerger$GroupViewIds.processElement(ParquetMerger.java:461)
	at com.google.fhir.analytics.ParquetMerger$GroupViewIds$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
	at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruningFunction.flatMap(FlinkMultiOutputPruningFunction.java:60)
	at org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruningFunction.flatMap(FlinkMultiOutputPruningFunction.java:35)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction$DoFnOutputManager.output(FlinkDoFnFunction.java:221)
	at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:276)
	at org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:86)
	at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:432)
	at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:412)
	at org.apache.beam.sdk.util.construction.SplittableParDoNaiveBounded$NaiveProcessFn$NestedProcessContext$1.output(SplittableParDoNaiveBounded.java:528)
	at org.apache.beam.sdk.io.parquet.ParquetIO$ReadFiles$SplitReadFn.processElement(ParquetIO.java:823)
	at org.apache.beam.sdk.io.parquet.ParquetIO$ReadFiles$SplitReadFn$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.sdk.util.construction.SplittableParDoNaiveBounded$NaiveProcessFn.process(SplittableParDoNaiveBounded.java:360)
	at org.apache.beam.sdk.util.construction.SplittableParDoNaiveBounded$NaiveProcessFn$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:186)
	at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruningFunction.flatMap(FlinkMultiOutputPruningFunction.java:60)
	at org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruningFunction.flatMap(FlinkMultiOutputPruningFunction.java:35)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction$DoFnOutputManager.output(FlinkDoFnFunction.java:221)
	at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:276)
	at org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:86)
	at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:432)
	at org.apache.beam.sdk.transforms.DoFnOutputReceivers$WindowedContextOutputReceiver.output(DoFnOutputReceivers.java:89)
	at org.apache.beam.sdk.transforms.MapElements$2.processElement(MapElements.java:151)
	at org.apache.beam.sdk.transforms.MapElements$2$DoFnInvoker.invokeProcessElement(Unknown Source)
	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
	at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at org.apache.beam.runners.flink.FlinkBatchTransformTranslators$ReshuffleTranslatorBatch.lambda$translateNode$baeabb50$1(FlinkBatchTransformTranslators.java:420)
	at org.apache.flink.api.java.operators.translation.PlanUnwrappingReduceGroupOperator$TupleUnwrappingNonCombinableGroupReducer.reduce(PlanUnwrappingReduceGroupOperator.java:120)
	at org.apache.flink.runtime.operators.GroupReduceDriver.run(GroupReduceDriver.java:145)
	at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:519)
	at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:360)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.flink.runtime.operators.chaining.ExceptionInChainedStubException: Exception in chained task 'FlatMap (FlatMap at CoGroupByKey/MakeUnionTable1/ParMultiDo(ConstructUnionTable).output)'
	at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:81)
	at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
	at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction$DoFnOutputManager.output(FlinkDoFnFunction.java:221)
	at org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:276)
	at org.apache.beam.runners.core.SimpleDoFnRunner.access$900(SimpleDoFnRunner.java:86)
	at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:432)
	at org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:412)
	at org.apache.beam.sdk.transforms.join.CoGroupByKey$ConstructUnionTableFn.processElement(CoGroupByKey.java:175)
Caused by: java.lang.ArrayIndexOutOfBoundsException

[bashir2]

Thanks @Rkareko for reporting this. I had a look and although it is not very clear what is going on, here are some findings and suggestions:

• It seems the error is coming from the ParquetMerger; this is triggered only during the incremental run, i.e., when the old and new Parquet files are being merged. Can you please delete/move the content of your DWH directory and try a Full Run? If the Full Run also fails, it should give you a different error; can you please share the log-trace for that too?

• I am not sure what is the root cause of the ArrayIndexOutOfBoundsException but given that this seems to be happening when the old and new views are being merged, is it possible that you edited the ViewDefinition files between runs? I mean you had some old Parquet views and then you modified the corresponding ViewDefinition before running an Incremental Run? That may explain the ArrayIndexOutOfBoundsException if the schema was changed.

• We need to investigate why you did not get an error message in the controller UI but first let's try to reproduce the problem in the Full Run scenario and work from there.

[Rkareko]

@bashir2 I cleared the DWH directory and did a Full Run. This was fecthing data from the main FHIR staging server.

1. The Pipeline Controller UI does not show any errors
2. Checking the patient_flat table in the views db shows 2000 out of a total of 3371 Patient records have been ingested.
3. Querying the patient_flat_view table connected via Spark returns no results
4. While the pipelines controller was running the following error was displayed on the docker container logs```
   spark-thriftserver | 25/08/21 07:33:09 INFO DAGScheduler: Asked to cancel job group 31bc66f9-d175-4757-bf9a-2b24b703c535
   spark-thriftserver | 25/08/21 07:33:09 ERROR SparkExecuteStatementOperation: Error executing query with 31bc66f9-d175-4757-bf9a-2b24b703c535, currentState RUNNING,
   spark-thriftserver | org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Add request failed : INSERT INTO COLUMNS_V2 (CD_ID,COMMENT,"COLUMN_NAME",TYPE_NAME,INTEGER_IDX) VALUES (?,?,?,?,?) )
   spark-thriftserver | at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:110)
   spark-thriftserver | at org.apache.spark.sql.hive.HiveExternalCatalog.createTable(HiveExternalCatalog.scala:244)
   spark-thriftserver | at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.createTable(ExternalCatalogWithListener.scala:94)
   spark-thriftserver | at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createTable(SessionCatalog.scala:373)
   spark-thriftserver | at org.apache.spark.sql.execution.command.CreateViewCommand.run(views.scala:166)
   spark-thriftserver | at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
   spark-thriftserver | at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
   spark-thriftserver | at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
   spark-thriftserver | at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
   spark-thriftserver | at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
   spark-thriftserver | at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
   spark-thriftserver | at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
   spark-thriftserver | at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
   spark-thriftserver | at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
   spark-thriftserver | at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
   spark-thriftserver | at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
   spark-thriftserver | at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
   spark-thriftserver | at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
   spark-thriftserver | at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
   spark-thriftserver | at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
   spark-thriftserver | at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
   spark-thriftserver | at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
   spark-thriftserver | at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
   spark-thriftserver | at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
   spark-thriftserver | at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
   spark-thriftserver | at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
   spark-thriftserver | at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
   spark-thriftserver | at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
   spark-thriftserver | at org.apache.spark.sql.Dataset.(Dataset.scala:219)
   spark-thriftserver | at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
   spark-thriftserver | at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
   spark-thriftserver | at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
   spark-thriftserver | at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
   spark-thriftserver | at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
   spark-thriftserver | at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
   spark-thriftserver | at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
   spark-thriftserver | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:291)
   spark-thriftserver | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:230)
   spark-thriftserver | at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
   spark-thriftserver | at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:79)
   spark-thriftserver | at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:63)
   spark-thriftserver | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:43)
   spark-thriftserver | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:230)
   spark-thriftserver | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:225)
   spark-thriftserver | at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
   spark-thriftserver | at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
   spark-thriftserver | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
   spark-thriftserver | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:239)
   spark-thriftserver | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
   spark-thriftserver | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   spark-thriftserver | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
   spark-thriftserver | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   spark-thriftserver | at java.base/java.lang.Thread.run(Thread.java:840)
   spark-thriftserver | Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Add request failed : INSERT INTO COLUMNS_V2 (CD_ID,COMMENT,"COLUMN_NAME",TYPE_NAME,INTEGER_IDX) VALUES (?,?,?,?,?) )
   spark-thriftserver | at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869)
   spark-thriftserver | at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:874)
   spark-thriftserver | at org.apache.spark.sql.hive.client.Shim_v0_12.createTable(HiveShim.scala:614)
   spark-thriftserver | at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$createTable$1(HiveClientImpl.scala:565)
   spark-thriftserver | at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
   spark-thriftserver | at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:298)
   spark-thriftserver | at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:229)
   spark-thriftserver | at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:228)
   spark-thriftserver | at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:278)
   spark-thriftserver | at org.apache.spark.sql.hive.client.HiveClientImpl.createTable(HiveClientImpl.scala:563)
   spark-thriftserver | at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$createTable$1(HiveExternalCatalog.scala:286)
   spark-thriftserver | at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
   spark-thriftserver | at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
   spark-thriftserver | ... 52 more
   spark-thriftserver | Caused by: MetaException(message:Add request failed : INSERT INTO COLUMNS_V2 (CD_ID,COMMENT,"COLUMN_NAME",TYPE_NAME,INTEGER_IDX) VALUES (?,?,?,?,?) )
   spark-thriftserver | at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:211)
   spark-thriftserver | at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
   spark-thriftserver | at jdk.proxy2/jdk.proxy2.$Proxy31.create_table_with_environment_context(Unknown Source)
   spark-thriftserver | at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2396)
   pipeline-controller | 07:33:09.322 [Thread-8] ERROR c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.createHiveTablesIfNeeded:642 - Exception while creating resource table on thriftserver for path: /dwh/controller_DWH_TIMESTAMP_2025_08_21T07_23_27_159258430Z
   spark-thriftserver | at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:93)
   pipeline-controller | java.sql.SQLException: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Add request failed : INSERT INTO COLUMNS_V2 (CD_ID,COMMENT,"COLUMN_NAME",TYPE_NAME,INTEGER_IDX) VALUES (?,?,?,?,?) )
   spark-thriftserver | at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:750)
   pipeline-controller | at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:43)
   spark-thriftserver | at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738)
   pipeline-controller | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:325)
   spark-thriftserver | at jdk.internal.reflect.GeneratedMethodAccessor111.invoke(Unknown Source)
   pipeline-controller | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:230)
   spark-thriftserver | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   pipeline-controller | at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
   spark-thriftserver | at java.base/java.lang.reflect.Method.invoke(Method.java:568)
   pipeline-controller | at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:79)
   spark-thriftserver | at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
   pipeline-controller | at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:63)
   spark-thriftserver | at jdk.proxy2/jdk.proxy2.$Proxy32.createTable(Unknown Source)
   pipeline-controller | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:43)
   spark-thriftserver | at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:859)
   pipeline-controller | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:230)
   spark-thriftserver | ... 64 more
   pipeline-controller | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:225)
   spark-thriftserver | Caused by: javax.jdo.JDODataStoreException: Add request failed : INSERT INTO COLUMNS_V2 (CD_ID,COMMENT,"COLUMN_NAME",TYPE_NAME,INTEGER_IDX) VALUES (?,?,?,?,?)
   pipeline-controller | at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
   spark-thriftserver | NestedThrowables:
   pipeline-controller | at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
   spark-thriftserver | java.sql.BatchUpdateException: A truncation error was encountered trying to shrink VARCHAR 'array<struct<id:string,linkId:string,definition:string,text:&' to length 32672.
   pipeline-controller | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
   spark-thriftserver | at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
   pipeline-controller | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:239)
   spark-thriftserver | at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:729)
   pipeline-controller | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
   spark-thriftserver | at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:749)
   pipeline-controller | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   spark-thriftserver | at org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:995)
   pipeline-controller | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
   spark-thriftserver | at jdk.internal.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)
   pipeline-controller | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   spark-thriftserver | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   pipeline-controller | at java.base/java.lang.Thread.run(Thread.java:840)
   spark-thriftserver | at java.base/java.lang.reflect.Method.invoke(Method.java:568)
   pipeline-controller | Caused by: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Add request failed : INSERT INTO COLUMNS_V2 (CD_ID,COMMENT,"COLUMN_NAME",TYPE_NAME,INTEGER_IDX) VALUES (?,?,?,?,?) )
   spark-thriftserver | at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
   pipeline-controller | at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:110)
   spark-thriftserver | at jdk.proxy2/jdk.proxy2.$Proxy30.createTable(Unknown Source)
   pipeline-controller | at org.apache.spark.sql.hive.HiveExternalCatalog.createTable(HiveExternalCatalog.scala:244)
   spark-thriftserver | at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1457)
   pipeline-controller | at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.createTable(ExternalCatalogWithListener.scala:94)
   spark-thriftserver | at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1503)
   pipeline-controller | at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createTable(SessionCatalog.scala:373)
   spark-thriftserver | at jdk.internal.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)
   pipeline-controller | at org.apache.spark.sql.execution.command.CreateViewCommand.run(views.scala:166)
   spark-thriftserver | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   pipeline-controller | at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
   spark-thriftserver | at java.base/java.lang.reflect.Method.invoke(Method.java:568)
   pipeline-controller | at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
   spark-thriftserver | at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
   pipeline-controller | at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
   spark-thriftserver | ... 76 more
   pipeline-controller | at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
   spark-thriftserver | Caused by: java.sql.BatchUpdateException: A truncation error was encountered trying to shrink VARCHAR 'array<struct<id:string,linkId:string,definition:string,text:&' to length 32672.
   pipeline-controller | at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
   pipeline-controller | at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
   pipeline-controller | at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
   pipeline-controller | at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
   pipeline-controller | at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
   pipeline-controller | at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
   pipeline-controller | at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
   pipeline-controller | at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
   pipeline-controller | at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
   spark-thriftserver | at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeBatch(Unknown Source)
   pipeline-controller | at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
   spark-thriftserver | at org.apache.derby.impl.jdbc.EmbedStatement.executeBatch(Unknown Source)
   pipeline-controller | at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
   spark-thriftserver | at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:424)
   pipeline-controller | at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
   spark-thriftserver | at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:366)
   pipeline-controller | at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
   spark-thriftserver | at org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:676)
   pipeline-controller | at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
   spark-thriftserver | at org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:405)
   pipeline-controller | at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
   spark-thriftserver | at org.datanucleus.store.rdbms.scostore.JoinListStore.internalAdd(JoinListStore.java:298)
   pipeline-controller | at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
   spark-thriftserver | at org.datanucleus.store.rdbms.scostore.AbstractListStore.addAll(AbstractListStore.java:137)
   pipeline-controller | at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
   spark-thriftserver | at org.datanucleus.store.rdbms.mapping.java.CollectionMapping.postInsert(CollectionMapping.java:157)
   pipeline-controller | at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
   spark-thriftserver | at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:522)
   pipeline-controller | at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
   pipeline-controller | at org.apache.spark.sql.Dataset.(Dataset.scala:219)
   pipeline-controller | at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
   spark-thriftserver | at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObjectInTable(RDBMSPersistenceHandler.java:162)
   pipeline-controller | at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
   spark-thriftserver | at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:138)
   pipeline-controller | at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
   spark-thriftserver | at org.datanucleus.state.StateManagerImpl.internalMakePersistent(StateManagerImpl.java:3363)
   pipeline-controller | at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
   spark-thriftserver | at org.datanucleus.state.StateManagerImpl.makePersistent(StateManagerImpl.java:3339)
   pipeline-controller | at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
   spark-thriftserver | at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2080)
   pipeline-controller | at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
   spark-thriftserver | at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2172)
   pipeline-controller | at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
   pipeline-controller | at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:291)
   pipeline-controller | ... 16 more
   pipeline-controller | Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Add request failed : INSERT INTO COLUMNS_V2 (CD_ID,COMMENT,"COLUMN_NAME",TYPE_NAME,INTEGER_IDX) VALUES (?,?,?,?,?) )
   pipeline-controller | at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869)
   pipeline-controller | at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:874)
   pipeline-controller | at org.apache.spark.sql.hive.client.Shim_v0_12.createTable(HiveShim.scala:614)
   spark-thriftserver | at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObjectAsValue(PersistableMapping.java:603)
   pipeline-controller | at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$createTable$1(HiveClientImpl.scala:565)
   spark-thriftserver | at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObject(PersistableMapping.java:357)
   pipeline-controller | at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
   spark-thriftserver | at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:191)
   pipeline-controller | at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:298)
   spark-thriftserver | at org.datanucleus.state.AbstractStateManager.providedObjectField(AbstractStateManager.java:1460)
   pipeline-controller | at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:229)
   spark-thriftserver | at org.datanucleus.state.StateManagerImpl.providedObjectField(StateManagerImpl.java:120)
   pipeline-controller | at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:228)
   spark-thriftserver | at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.dnProvideField(MStorageDescriptor.java)
   pipeline-controller | at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:278)
   spark-thriftserver | at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.dnProvideFields(MStorageDescriptor.java)
   pipeline-controller | at org.apache.spark.sql.hive.client.HiveClientImpl.createTable(HiveClientImpl.scala:563)
   spark-thriftserver | at org.datanucleus.state.StateManagerImpl.provideFields(StateManagerImpl.java:1170)
   pipeline-controller | at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$createTable$1(HiveExternalCatalog.scala:286)
   spark-thriftserver | at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:292)
   pipeline-controller | at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
   spark-thriftserver | at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObjectInTable(RDBMSPersistenceHandler.java:162)
   pipeline-controller | at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:101)
   spark-thriftserver | at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:138)
   pipeline-controller | ... 52 more
   spark-thriftserver | at org.datanucleus.state.StateManagerImpl.internalMakePersistent(StateManagerImpl.java:3363)
   pipeline-controller | Caused by: MetaException(message:Add request failed : INSERT INTO COLUMNS_V2 (CD_ID,COMMENT,"COLUMN_NAME",TYPE_NAME,INTEGER_IDX) VALUES (?,?,?,?,?) )
   spark-thriftserver | at org.datanucleus.state.StateManagerImpl.makePersistent(StateManagerImpl.java:3339)
   pipeline-controller | at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:211)
   spark-thriftserver | at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2080)
   pipeline-controller | at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
   spark-thriftserver | at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2172)
   pipeline-controller | at jdk.proxy2/jdk.proxy2.$Proxy31.create_table_with_environment_context(Unknown Source)
   spark-thriftserver | at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObjectAsValue(PersistableMapping.java:603)
   pipeline-controller | at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2396)
   spark-thriftserver | at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObject(PersistableMapping.java:357)
   pipeline-controller | at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:93)
   spark-thriftserver | at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:191)
   pipeline-controller | at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:750)
   spark-thriftserver | at org.datanucleus.state.AbstractStateManager.providedObjectField(AbstractStateManager.java:1460)
   pipeline-controller | at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738)
   spark-thriftserver | at org.datanucleus.state.StateManagerImpl.providedObjectField(StateManagerImpl.java:120)
   pipeline-controller | at jdk.internal.reflect.GeneratedMethodAccessor111.invoke(Unknown Source)
   spark-thriftserver | at org.apache.hadoop.hive.metastore.model.MTable.dnProvideField(MTable.java)
   pipeline-controller | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   spark-thriftserver | at org.apache.hadoop.hive.metastore.model.MTable.dnProvideFields(MTable.java)
   pipeline-controller | at java.base/java.lang.reflect.Method.invoke(Method.java:568)
   spark-thriftserver | at org.datanucleus.state.StateManagerImpl.provideFields(StateManagerImpl.java:1170)
   pipeline-controller | at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173)
   spark-thriftserver | at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:292)
   pipeline-controller | at jdk.proxy2/jdk.proxy2.$Proxy32.createTable(Unknown Source)
   spark-thriftserver | at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObjectInTable(RDBMSPersistenceHandler.java:162)
   pipeline-controller | at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:859)
   spark-thriftserver | at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:138)
   pipeline-controller | ... 64 more
   spark-thriftserver | at org.datanucleus.state.StateManagerImpl.internalMakePersistent(StateManagerImpl.java:3363)
   pipeline-controller | Caused by: javax.jdo.JDODataStoreException: Add request failed : INSERT INTO COLUMNS_V2 (CD_ID,COMMENT,"COLUMN_NAME",TYPE_NAME,INTEGER_IDX) VALUES (?,?,?,?,?)
   spark-thriftserver | at org.datanucleus.state.StateManagerImpl.makePersistent(StateManagerImpl.java:3339)
   pipeline-controller | NestedThrowables:
   spark-thriftserver | at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2080)
   pipeline-controller | java.sql.BatchUpdateException: A truncation error was encountered trying to shrink VARCHAR 'array<struct<id:string,linkId:string,definition:string,text:&' to length 32672.
   spark-thriftserver | at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:1923)
   pipeline-controller | at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
   spark-thriftserver | at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1778)
   pipeline-controller | at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:729)
   spark-thriftserver | at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
   pipeline-controller | at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:749)
   spark-thriftserver | at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:724)
   pipeline-controller | at org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.java:995)
   spark-thriftserver | ... 89 more
   pipeline-controller | at jdk.internal.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)
   spark-thriftserver | Caused by: java.sql.SQLDataException: A truncation error was encountered trying to shrink VARCHAR 'array<struct<id:string,linkId:string,definition:string,text:&' to length 32672.
   pipeline-controller | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   spark-thriftserver | at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
   pipeline-controller | at java.base/java.lang.reflect.Method.invoke(Method.java:568)
   spark-thriftserver | at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
   pipeline-controller | at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
   spark-thriftserver | at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
   pipeline-controller | at jdk.proxy2/jdk.proxy2.$Proxy30.createTable(Unknown Source)
   spark-thriftserver | at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
   pipeline-controller | at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1457)
   spark-thriftserver | at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
   pipeline-controller | at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1503)
   spark-thriftserver | at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
   pipeline-controller | at jdk.internal.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)
   spark-thriftserver | at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
   pipeline-controller | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   spark-thriftserver | at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeBatchElement(Unknown Source)
   pipeline-controller | at java.base/java.lang.reflect.Method.invoke(Method.java:568)
   spark-thriftserver | ... 138 more
   pipeline-controller | at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
   spark-thriftserver | Caused by: ERROR 22001: A truncation error was encountered trying to shrink VARCHAR 'array<struct<id:string,linkId:string,definition:string,text:&' to length 32672.
   pipeline-controller | ... 76 more
   spark-thriftserver | at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
   pipeline-controller | Caused by: java.sql.BatchUpdateException: A truncation error was encountered trying to shrink VARCHAR 'array<struct<id:string,linkId:string,definition:string,text:&' to length 32672.
   spark-thriftserver | at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeBatch(Unknown Source)
   spark-thriftserver | at org.apache.derby.iapi.types.SQLChar.hasNonBlankChars(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.jdbc.EmbedStatement.executeBatch(Unknown Source)
   spark-thriftserver | at org.apache.derby.iapi.types.SQLVarchar.normalize(Unknown Source)
   pipeline-controller | at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:424)
   spark-thriftserver | at org.apache.derby.iapi.types.SQLVarchar.normalize(Unknown Source)
   pipeline-controller | at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:366)
   spark-thriftserver | at org.apache.derby.iapi.types.DataTypeDescriptor.normalize(Unknown Source)
   pipeline-controller | at org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:676)
   spark-thriftserver | at org.apache.derby.impl.sql.execute.NormalizeResultSet.normalizeColumn(Unknown Source)
   pipeline-controller | at org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:405)
   spark-thriftserver | at org.apache.derby.impl.sql.execute.NormalizeResultSet.normalizeRow(Unknown Source)
   pipeline-controller | at org.datanucleus.store.rdbms.scostore.JoinListStore.internalAdd(JoinListStore.java:298)
   spark-thriftserver | at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown Source)
   pipeline-controller | at org.datanucleus.store.rdbms.scostore.AbstractListStore.addAll(AbstractListStore.java:137)
   spark-thriftserver | at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown Source)
   pipeline-controller | at org.datanucleus.store.rdbms.mapping.java.CollectionMapping.postInsert(CollectionMapping.java:157)
   spark-thriftserver | at org.apache.derby.impl.sql.execute.InsertResultSet.getNextRowCore(Unknown Source)
   pipeline-controller | at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:522)
   spark-thriftserver | at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
   pipeline-controller | at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObjectInTable(RDBMSPersistenceHandler.java:162)
   spark-thriftserver | at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
   pipeline-controller | at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:138)
   spark-thriftserver | at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
   pipeline-controller | at org.datanucleus.state.StateManagerImpl.internalMakePersistent(StateManagerImpl.java:3363)
   spark-thriftserver | ... 140 more
   pipeline-controller | at org.datanucleus.state.StateManagerImpl.makePersistent(StateManagerImpl.java:3339)
   spark-thriftserver | 25/08/21 07:33:09 INFO DAGScheduler: Asked to cancel job group 31bc66f9-d175-4757-bf9a-2b24b703c535
   pipeline-controller | at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2080)
   spark-thriftserver | 25/08/21 07:33:09 INFO SparkExecuteStatementOperation: Close statement with 31bc66f9-d175-4757-bf9a-2b24b703c535
   pipeline-controller | at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2172)
   pipeline-controller | at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObjectAsValue(PersistableMapping.java:603)
   pipeline-controller | at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObject(PersistableMapping.java:357)
   pipeline-controller | at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:191)
   pipeline-controller | at org.datanucleus.state.AbstractStateManager.providedObjectField(AbstractStateManager.java:1460)
   pipeline-controller | at org.datanucleus.state.StateManagerImpl.providedObjectField(StateManagerImpl.java:120)
   pipeline-controller | at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.dnProvideField(MStorageDescriptor.java)
   pipeline-controller | at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.dnProvideFields(MStorageDescriptor.java)
   pipeline-controller | at org.datanucleus.state.StateManagerImpl.provideFields(StateManagerImpl.java:1170)
   pipeline-controller | at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:292)
   pipeline-controller | at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObjectInTable(RDBMSPersistenceHandler.java:162)
   pipeline-controller | at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:138)
   pipeline-controller | at org.datanucleus.state.StateManagerImpl.internalMakePersistent(StateManagerImpl.java:3363)
   pipeline-controller | at org.datanucleus.state.StateManagerImpl.makePersistent(StateManagerImpl.java:3339)
   pipeline-controller | at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2080)
   pipeline-controller | at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2172)
   pipeline-controller | at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObjectAsValue(PersistableMapping.java:603)
   pipeline-controller | at org.datanucleus.store.rdbms.mapping.java.PersistableMapping.setObject(PersistableMapping.java:357)
   pipeline-controller | at org.datanucleus.store.rdbms.fieldmanager.ParameterSetter.storeObjectField(ParameterSetter.java:191)
   pipeline-controller | at org.datanucleus.state.AbstractStateManager.providedObjectField(AbstractStateManager.java:1460)
   pipeline-controller | at org.datanucleus.state.StateManagerImpl.providedObjectField(StateManagerImpl.java:120)
   pipeline-controller | at org.apache.hadoop.hive.metastore.model.MTable.dnProvideField(MTable.java)
   pipeline-controller | at org.apache.hadoop.hive.metastore.model.MTable.dnProvideFields(MTable.java)
   pipeline-controller | at org.datanucleus.state.StateManagerImpl.provideFields(StateManagerImpl.java:1170)
   pipeline-controller | at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:292)
   pipeline-controller | at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObjectInTable(RDBMSPersistenceHandler.java:162)
   pipeline-controller | at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:138)
   pipeline-controller | at org.datanucleus.state.StateManagerImpl.internalMakePersistent(StateManagerImpl.java:3363)
   pipeline-controller | at org.datanucleus.state.StateManagerImpl.makePersistent(StateManagerImpl.java:3339)
   pipeline-controller | at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2080)
   pipeline-controller | at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:1923)
   pipeline-controller | at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1778)
   pipeline-controller | at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
   pipeline-controller | at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:724)
   pipeline-controller | ... 89 more
   pipeline-controller | Caused by: java.sql.SQLDataException: A truncation error was encountered trying to shrink VARCHAR 'array<struct<id:string,linkId:string,definition:string,text:&' to length 32672.
   pipeline-controller | at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeBatchElement(Unknown Source)
   pipeline-controller | ... 138 more
   pipeline-controller | Caused by: ERROR 22001: A truncation error was encountered trying to shrink VARCHAR 'array<struct<id:string,linkId:string,definition:string,text:&' to length 32672.
   pipeline-controller | at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
   pipeline-controller | at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
   pipeline-controller | at org.apache.derby.iapi.types.SQLChar.hasNonBlankChars(Unknown Source)
   pipeline-controller | at org.apache.derby.iapi.types.SQLVarchar.normalize(Unknown Source)
   pipeline-controller | at org.apache.derby.iapi.types.SQLVarchar.normalize(Unknown Source)
   pipeline-controller | at org.apache.derby.iapi.types.DataTypeDescriptor.normalize(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.sql.execute.NormalizeResultSet.normalizeColumn(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.sql.execute.NormalizeResultSet.normalizeRow(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.sql.execute.InsertResultSet.getNextRowCore(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
   pipeline-controller | at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
   pipeline-controller | ... 140 more
   pipeline-controller |
   pipeline-controller | at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:401)
   pipeline-controller | at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:266)
   pipeline-controller | at com.mchange.v2.c3p0.impl.NewProxyStatement.execute(NewProxyStatement.java:1419)
   pipeline-controller | at com.google.fhir.analytics.HiveTableManager.executeSql(HiveTableManager.java:175)
   pipeline-controller | at com.google.fhir.analytics.HiveTableManager.createTablesForResourceOrView(HiveTableManager.java:119)
   pipeline-controller | at com.google.fhir.analytics.HiveTableManager.createResourceAndCanonicalTables(HiveTableManager.java:84)
   pipeline-controller | at com.google.fhir.analytics.PipelineManager.createHiveTablesIfNeeded(PipelineManager.java:619)
   pipeline-controller | at com.google.fhir.analytics.PipelineManager$PipelineThread.run(PipelineManager.java:759)
   pipeline-controller | 07:33:09.411 [Thread-8] INFO c.g.fhir.analytics.PipelineManager c.g.fhir.analytics.PipelineManager$PipelineThread.run:773 - Total time taken for the pipelines = 582 secs
   pipeline-controller | 07:33:13.025 [http-nio-8080-exec-6] INFO c.g.fhir.analytics.DataProperties com.google.fhir.analytics.DataProperties.createBatchOptions:190 - Converting options for fhirServerUrl https://fhir.labs.smartregister.org/fhir and dbConfig null
   pipeline-controller | 07:33:13.028 [http-nio-8080-exec-6] INFO c.g.fhir.analytics.MainController com.google.fhir.analytics.MainController.index:76 - Done serving /
   pipeline-controller | 07:33:28.673 [scheduling-1] INFO c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.checkPurgeScheduleAndTrigger:123 - Last purge run was at 2025-08-21T07:18:28.662479911 next run is at 2025-08-21T07:30
   pipeline-controller | 07:33:28.674 [scheduling-1] INFO c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.checkPurgeScheduleAndTrigger:125 - Purge run triggered at 2025-08-21T07:33:28.674070140
   pipeline-controller | 07:33:28.687 [scheduling-1] INFO com.google.fhir.analytics.DwhFiles com.google.fhir.analytics.DwhFiles.getAllChildDirectories:274 - Child directories of /dwh are [/dwh/controller_DWH_TIMESTAMP_2025_08_21T07_23_27_159258430Z/]
   pipeline-controller | 07:33:28.687 [scheduling-1] INFO c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.checkPurgeScheduleAndTrigger:127 - Purge run completed at 2025-08-21T07:33:28.687776306
   pipeline-controller | 07:33:28.694 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:33:58.695 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   spark-thriftserver | 25/08/21 07:34:23 INFO ObjectStore: ObjectStore, initialize called
   spark-thriftserver | 25/08/21 07:34:23 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
   spark-thriftserver | 25/08/21 07:34:23 INFO ObjectStore: Initialized ObjectStore
   spark-thriftserver | 25/08/21 07:34:23 INFO ObjectStore: ObjectStore, initialize called
   spark-thriftserver | 25/08/21 07:34:23 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
   spark-thriftserver | 25/08/21 07:34:23 INFO ObjectStore: Initialized ObjectStore
   spark-thriftserver | 25/08/21 07:34:23 INFO HiveMetaStore: 5: Cleaning up thread local RawStore...
   spark-thriftserver | 25/08/21 07:34:23 INFO audit: ugi=username ip=unknown-ip-addr cmd=Cleaning up thread local RawStore...
   spark-thriftserver | 25/08/21 07:34:23 INFO HiveMetaStore: 5: Done cleaning up thread local RawStore
   spark-thriftserver | 25/08/21 07:34:23 INFO audit: ugi=username ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
   spark-thriftserver | 25/08/21 07:34:23 INFO HiveMetaStore: 7: Cleaning up thread local RawStore...
   spark-thriftserver | 25/08/21 07:34:23 INFO audit: ugi=username ip=unknown-ip-addr cmd=Cleaning up thread local RawStore...
   spark-thriftserver | 25/08/21 07:34:23 INFO HiveMetaStore: 7: Done cleaning up thread local RawStore
   spark-thriftserver | 25/08/21 07:34:23 INFO audit: ugi=username ip=unknown-ip-addr cmd=Done cleaning up thread local RawStore
   spark-thriftserver | 25/08/21 07:34:23 ERROR TThreadPoolServer: Thrift error occurred during processing of message.
   spark-thriftserver | org.apache.thrift.transport.TTransportException
   spark-thriftserver | at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
   spark-thriftserver | at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
   spark-thriftserver | at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374)
   spark-thriftserver | at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451)
   spark-thriftserver | at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433)
   spark-thriftserver | at org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:43)
   spark-thriftserver | at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
   spark-thriftserver | at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425)
   spark-thriftserver | at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
   spark-thriftserver | at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
   spark-thriftserver | at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
   spark-thriftserver | at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:52)
   spark-thriftserver | at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
   spark-thriftserver | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
   spark-thriftserver | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   spark-thriftserver | at java.base/java.lang.Thread.run(Thread.java:840)
   spark-thriftserver | 25/08/21 07:34:23 ERROR TThreadPoolServer: Thrift error occurred during processing of message.
   spark-thriftserver | org.apache.thrift.transport.TTransportException
   spark-thriftserver | at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
   spark-thriftserver | at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
   spark-thriftserver | at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374)
   spark-thriftserver | at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451)
   spark-thriftserver | at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433)
   spark-thriftserver | at org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:43)
   spark-thriftserver | at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
   spark-thriftserver | at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:425)
   spark-thriftserver | at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:321)
   spark-thriftserver | at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:225)
   spark-thriftserver | at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
   spark-thriftserver | at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:52)
   spark-thriftserver | at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310)
   spark-thriftserver | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
   spark-thriftserver | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   spark-thriftserver | at java.base/java.lang.Thread.run(Thread.java:840)
   pipeline-controller | 07:34:28.696 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:34:58.697 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:35:28.698 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:35:58.700 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:36:28.702 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:36:58.704 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:37:28.705 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:37:58.706 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:38:28.688 [scheduling-1] INFO c.g.fhir.analytics.DwhFilesManager com.google.fhir.analytics.DwhFilesManager.checkPurgeScheduleAndTrigger:123 - Last purge run was at 2025-08-21T07:33:28.687771071 next run is at 2025-08-21T08:30
   pipeline-controller | 07:38:28.707 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:38:58.708 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:39:28.709 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:39:58.710 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:40:28.711 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:40:58.712 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:41:28.713 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00
   pipeline-controller | 07:41:58.714 [scheduling-1] INFO c.g.fhir.analytics.PipelineManager com.google.fhir.analytics.PipelineManager.checkSchedule:385 - Last run was at 2025-08-21T07:33:09.327319694 next run is at 2025-08-21T08:00

w Enable Watch

[Rkareko]

@bashir2 I cleared the DWH directory and did a Full Run on a different machine (from the one that did this run). This was fetching data from a preview server.

1. 255 out of a total of 355 Patients records were ingested into the patient_flat table in the views db.
2. Querying the patient_flat_view table connected via Spark returns no results
3. The Pipelines Controller showed that the run failed with the following error logged```
   java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Pipeline execution failed
   at com.google.fhir.analytics.EtlUtils.runMultiplePipelines(EtlUtils.java:171)
   at com.google.fhir.analytics.EtlUtils.runMultiplePipelinesWithTimestamp(EtlUtils.java:83)
   at com.google.fhir.analytics.PipelineManager$PipelineThread.run(PipelineManager.java:736)
   Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Pipeline execution failed
   at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
   at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
   at com.google.fhir.analytics.EtlUtils.runMultiplePipelines(EtlUtils.java:158)
   ... 2 more
   Caused by: java.lang.RuntimeException: Pipeline execution failed
   at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:107)
   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:310)
   at com.google.fhir.analytics.EtlUtils$PipelineSupplier.get(EtlUtils.java:192)
   at com.google.fhir.analytics.EtlUtils$PipelineSupplier.get(EtlUtils.java:182)
   at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   at java.base/java.lang.Thread.run(Thread.java:840)
   Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
   at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
   at org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
   at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
   at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
   at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
   at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:258)
   at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
   at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
   at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
   at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
   at org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389)
   at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
   at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
   at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
   at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
   at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
   at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
   at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
   at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
   at akka.dispatch.OnComplete.internal(Future.scala:300)
   at akka.dispatch.OnComplete.internal(Future.scala:297)
   at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224)
   at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221)
   at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
   at org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
   at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
   at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
   at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
   at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
   at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621)
   at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
   at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
   at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
   at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
   at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
   at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
   at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
   at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
   at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
   at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
   at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
   at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
   at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
   at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
   at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
   at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
   at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
   Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
   at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
   at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
   at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:252)
   at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:242)
   at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:233)
   at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:684)
   at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
   at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:444)
   at jdk.internal.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)
   at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.base/java.lang.reflect.Method.invoke(Method.java:569)
   at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
   at org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
   at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
   at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
   at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
   at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
   at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
   at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
   at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
   at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
   at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
   at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
   at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
   at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
   at akka.actor.Actor.aroundReceive(Actor.scala:537)
   at akka.actor.Actor.aroundReceive$(Actor.scala:535)
   at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
   at akka.actor.ActorCell.invoke(ActorCell.scala:548)
   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
   at akka.dispatch.Mailbox.run(Mailbox.scala:231)
   at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
   ... 5 more
   Caused by: org.apache.beam.sdk.util.UserCodeException: java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because the return value of "org.hl7.fhir.instance.model.api.IBaseExtension.getUrl()" is null
   at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
   at com.google.fhir.analytics.FetchResources$SearchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
   at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:212)
   at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:189)
   at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:62)
   at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:122)
   at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.flatMap(FlinkDoFnFunction.java:59)
   at org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:79)
   at org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
   at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSourceTask.java:208)
   at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
   at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
   at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
   at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
   at java.base/java.lang.Thread.run(Thread.java:840)
   Caused by: java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because the return value of "org.hl7.fhir.instance.model.api.IBaseExtension.getUrl()" is null
   at com.cerner.bunsen.definitions.HapiCompositeConverter.fromHapi(HapiCompositeConverter.java:218)
   at com.cerner.bunsen.avro.AvroConverter.resourceToAvro(AvroConverter.java:215)
   at com.google.fhir.analytics.AvroConversionUtil.convertToAvro(AvroConversionUtil.java:158)
   at com.google.fhir.analytics.ParquetUtil.write(ParquetUtil.java:269)
   at com.google.fhir.analytics.ParquetUtil.writeRecords(ParquetUtil.java:347)
   at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:301)
   at com.google.fhir.analytics.FetchSearchPageFn.processBundle(FetchSearchPageFn.java:292)
   at com.google.fhir.analytics.FetchResources$SearchFn.processElement(FetchResources.java:145)

[bashir2]

Thanks @Rkareko for the update. I looked at your second report (with a clean "Full Run") and it seems the error is coming from this line, which IIUC means that you have some FHIR resources with extensions but the url part of the extension is not set. Can you please check some of your Patient resources and confirm that is really the case? If that's true here are my suggestions:

• The resources really need to be fixed as url is a mandatory field of extension.
• If you want to disable extension processing, you can comment out this line (BTW, if you have your own custom extensions, you should update this line and provide the correct StructureDefinition files for your extensions).
• To prevent this exception in future, I have created PR Preventing a potential NPE #1440 (but as I said the url is a required field so it is better to fix the resources as that assumption might have been made elsewhere in the code too).

Regarding the Spark table creation issues, did you manage to fix that in the second run? From a quick look at logs it seems to me that the size of the Parquet schema is too large which I am guessing might be related to extensions again or the recursive depth; do those ring any bells?

Side note: Number of rows in a flat view can be different from the number of FHIR resources that are used to create that view (can be more or less depending on your ViewDefinition).

[bashir2]

BTW, are you still not getting any errors in the controller UI for the failed runs? If not, can you check your DWH snapshot dirs and see if an error.log file is created there?

[Rkareko]

@bashir2 Thanks for the tips.
A quick note, the issues reported here were found when fetching data from 2 different FHIR servers

1. Staging FHIR server had the issues reported on the issue description and on this comment
2. Preview FHIR server had the issue reported on this comment to which you responded here

Concerning the Preview server issues, after following your tips the following was noted

• There was indeed a Patient record that had a null Extension.url field value. After this was corrected and a new Full Run executed the FHIR Data Pipes appears to have ingested all the required data.
• After this run the patient_flat_2_view Spark table has the data but patient_flat does not have data. Not sure whether this is a cause for concern or not.
• The error was displayed on the controller UI

[Rkareko]

Concerning the Staging server issue mentioned here and here

• We are now at 2000 out of a total of 3371 Patient resources ingested.
• The controller UI does not show any errors
• The dwh folder does nto have an error log as seen below

ls controller_DWH_TIMESTAMP_2025_08_21T07_23_27_159258430Z/
Condition         Encounter  Immunization  Location     Organization  Practitioner      Procedure      QuestionnaireResponse  timestamp_end.txt    VIEWS_TIMESTAMP_2025_08_21T07_23_27_159338141Z
DiagnosticReport  Group      List          Observation  Patient       PractitionerRole  Questionnaire  Task                   timestamp_start.txt

and

ls controller_DWH_TIMESTAMP_2025_08_21T07_23_27_159258430Z/VIEWS_TIMESTAMP_2025_08_21T07_23_27_159338141Z/
condition_flat          encounter_flat     location_flat     organization_flat  practitioner_flat       procedure_flat     timestamp_start.txt
diagnostic_report_flat  immunization_flat  observation_flat  patient_flat       practitioner_role_flat  timestamp_end.txt

[Rkareko]

@ndegwamartin After cleaning the dwh folder and configuring the data pipes to fetch only on Patient resources, it appears that all the Patient resources are saved in the data warehouse and the patient_flat_view. The API shows that there are 3409 Patient records.

API response for a summary count of patient records

{
    "resourceType": "Bundle",
    "id": "41968084-98b9-4df7-9e05-5e9d6213e269",
    "meta": {
        "lastUpdated": "2025-09-24T12:40:21.563+00:00",
        "tag": [
            {
                "system": "http://terminology.hl7.org/CodeSystem/v3-ObservationValue",
                "code": "SUBSETTED",
                "display": "Resource encoded in summary mode"
            }
        ]
    },
    "type": "searchset",
    "total": 3409
}

Count of distinct patient records in the patient_flat_view

views=# select count(distinct id) from patient_flat ;
 count 
-------
  3409

[Rkareko]

@ndegwamartin After configuring the data pipes to fetch only Patient and Observation, both types of resources are fetched fully.

After updating the configurations to fetch Patient, Encounter and Observation, both Patient and Observation resources are fetched fully. However Encounter resources are partially fetched

API response for a summary count of Encounter records

{
    "resourceType": "Bundle",
    "id": "6b53890d-9a33-409a-bc9b-ddac9e734872",
    "meta": {
        "lastUpdated": "2025-09-25T08:03:06.857+00:00",
        "tag": [
            {
                "system": "http://terminology.hl7.org/CodeSystem/v3-ObservationValue",
                "code": "SUBSETTED",
                "display": "Resource encoded in summary mode"
            }
        ]
    },
    "type": "searchset",
    "total": 4611
}

Count of distinct encounter records in the encounter_flat_view

views=# select count(distinct id) from encounter_flat ;
 count 
-------
  4111

After updating the configurations to fetch only Encounters, all of the encounters are fetched.

Count of distinct encounter records in the encounter_flat_view

views=# select count(distinct id) from encounter_flat ;
 count 
-------
  4611
(1 row)

[Rkareko]

Configured Resources
Patient,Encounter,Observation,Questionnaire,Condition

Partially fetched resources
Patient - 2809 out of 3409
Encounter - 4311 out of 4611
Observation - 19045 out of 19745
Condition - 422 out of 1222
Questionnaire - 265 out of 365

[Rkareko]

Configured Resources
Patient,Encounter,Observation,Condition

Partially fetched resources
Patient - 3109 out of 3409
Encounter - 4100 out of 4611
Observation - 19245 out of 19745
Condition - 622 out of 1222