Optimizing Apache Spark & Tuning Best Practices

This live-virtual course is perfect for

Data and Machine Learning Engineers who deal with transformation of large volumes of data and need production-quality code. Expert Data Scientists can also participate: they will learn how to get the most performance out of Spark and how simple tweaks can increase the performance dramatically.

What will you learn during Optimizing Apache Spark & Tuning Best Practices?

After this training, you will have learned how Apache Spark works internally, the best practices to write performant code, and have acquired essential skills necessary to debug and tweak your Spark applications.

Program

Fundamentals

• Spark execution model: Driver/Executors
• Spark resource managers (YARN, MESOS, K8s)
• Understanding RDDs/DataFrames APIs and bindings
• Difference between Actions and Transformations
• How to read the Query plan (Physical/Logical)

Spark internals

• Spark Memory model
• Understanding persistence (caching)
• Catalyst optimizer and Tungsten project
• Shuffle service and how is shuffle operation executed
• Concept of fair scheduling and pools
• Java and Kryo serializer
• Step into JVM world: what you need to know about GC when running Spark applications

Spark optimization: main problems and issues

• The most common memory problems
• Benefit of using early filtering
• Understanding partition and predicate filtering
• Join optimization
• Combating Data skew (preprocessing, broadcasting, salting)
• Understanding shuffle partitions: how to tackle memory/disk spill
• Downside of using UDF’s
• Executor idle timeout
• Data formats examples

Moving to production

• Debugging / troubleshooting
• Productionizing your Spark application
• Dynamic allocation and dynamic partitioning
• Profiling your Spark application (Sparklint)
• JVM profiler

Data Engineering Trainers

This Data Engineering training is brought to you by our training partner, GoDataDriven. GoDataDriven works with experts in their field who are always on the lookout for the most innovative ways to get the most out of data. Your trainer is a data guru who enjoys sharing his or her experiences to help you work with the latest tools. 

Yes, I want to know more about Apache Spark!

After registering for this training, you will receive a confirmation email with practical information. A week before the training we share literature if there's anything you need to prepare. See you soon!

Virtual or in-person training: This training can be delivered both in-person or online. When hosting the in-person training, we provide lunch, snacks and drinks to the participants. Accordingly there is a discount for virtual trainings.