Use ROW-level operations instead of writing the python/scala code. This is the relevant Apache Spark [issue](https://issues.apache.org/jira/browse/SPARK-35801?jql=project%20%3D%20SPARK%20AND%20status%20in%20(Open%2C%20Reopened%2C%20%22In%20Progress%22)%20AND%20(labels%20%3D%20SPIP%20OR%20summary%20~%20%22SPIP%22)%20ORDER%20BY%20createdDate%20DESC). [Design Doc](https://docs.google.com/document/d/12Ywmc47j3l2WF4anG5vL4qlrhT2OKigb7_EbIKhxg60/edit#heading=h.82w8qxfl2uwl) for the feature.