[addition to existing page]: Managing Partitions

[onsbardons]

Name of existing page

Managing Partitions https://best-practice-and-impact.github.io/ons-spark/spark-concepts/partitions.html?highlight=skewed#managing-partitions

Summary of Content

Now that we can use Spark>3.0 it would be good to add another automatic way of handling data skews using spark.sql.adaptive.enabled and spark.sql.adaptive.skewJoin.enabled configurations

Language Version

Python

Spark 3

Yes

Code snippets

Article:
https://krishna-yogik.medium.com/spark-aqe-a-detailed-guide-with-examples-d8b52a0a2f20

Code of Conduct

• I agree to follow this project's Code of Conduct