Performance impact when trying to generate profiling report for more than 200 columns

[eapframework]

Encountering performance issues when generating a profiling report for more than 200 columns across 5 million records. I am applying almost all the metrics to generate profiling report. Applied metrics such as datatype, entropy, minimum, maximum, sum, standard deviation, mean, maxlength, minlength, histogram, completeness, distinctness, uniquevalueratio, uniqueness, countdistinct, and correlation. I am trying to generate report similar to ydata-profiling(https://github.com/ydataai/ydata-profiling)

The job has been running for over 3 hours despite attempts to optimize Spark configuration. When checking the logs each metrics is calculated sequentially. Sequential computation of each metric is causing the prolonged runtime. Is it possible to parallelize this operation for improved efficiency?

[rdsharma26]

Thanks for the feedback @eapframework
We will investigate this issue and get back to you with an update.

[eapframework]

Hi @rdsharma26, I was doing more testing. By analyzing the spark execution tasks, I believe the performance issue is because for metrics such as CountDistinct, Histogram, each metrics calculation is done on each column in sequential manner. So more columns in dataframe is causing the job to run for more time. Parallelizing these calculations would enhance efficiency.

[github-actions]

Looking at the AnalysisRunner.doAnalysisRun code, deequ already optimizes scan-shareable analyzers (Completeness, Mean, Sum, StandardDeviation, etc.) by computing them in a single pass via runScanningAnalyzers. However, you're correct that grouping-based analyzers (CountDistinct, Histogram, Uniqueness, Distinctness, Entropy, UniqueValueRatio) run sequentially — they are grouped by their groupingColumns and each group triggers a separate aggregation on the grouped DataFrame.

With 200+ columns, that means 200+ separate groupBy/count passes for frequency-based analyzers, which explains the 3+ hour runtime.

Workarounds:

1. Run frequency-based analyzers in parallel yourself — split your analyzers into batches and run them concurrently using Scala futures:

import scala.concurrent.{Future, Await}
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global

val columns = df.columns.toSeq
val frequencyAnalyzers = columns.map(c => Entropy(c)) // or Histogram, CountDistinct, etc.

val batchSize = 20
val futures = frequencyAnalyzers.grouped(batchSize).map { batch =>
  Future {
    AnalysisRunner.onData(df).addAnalyzers(batch).run()
  }
}

val results = futures.map(f => Await.result(f, 2.hours)).reduce(_ ++ _)

This leverages Spark's ability to run multiple jobs concurrently (controlled by spark.scheduler.mode=FAIR).

2. Separate scan-shareable from grouping analyzers — run all scan-shareable analyzers (Completeness, Min, Max, Mean, StdDev, Sum, MinLength, MaxLength) in one call, and batch the frequency-based ones (Histogram, Entropy, CountDistinct, Distinctness, Uniqueness, UniqueValueRatio) separately with parallelism as above.

A proper fix would require changes to the AnalysisRunner to parallelize grouping analyzer execution internally. This is a known architectural limitation — the sequential loop over groupingAnalyzers.groupBy(...) in doAnalysisRun is the bottleneck. We'd welcome a PR that adds an option to execute independent grouping analyzer groups in parallel.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 5609e400) — may not be fully accurate. Reply if this doesn't help.