I have been working on apache spark lately and wanted to consolidated my learnings at one place so that I can refer later and make it available to other developers as well. While working with spark you will often run into situations where the way you write code normally results in most inefficient results. You can visualise spark plan executions but it is very tedious to navigate and make sense of in terms of what exactly is cause of the problem. In such case having basic knowledge of how spark manages tasks can help arrive at solutions to improve compute efficiency. Below are a set of techniques I have used to improve compute efficiency of spark applications. SparkContext with a Checkpoint Directory In Apache Spark , the SparkContext is the entry point to interact with the Spark framework. It manages the execution of jobs, resource allocation, and coordination across clusters. A checkpoint directory in Spark is used to store checkpointed RDDs or stre...