US flag signifying that this is a United States Federal Government website  Official website of the Department of Homeland Security

Spark Monitoring and Tuning

Spark Monitoring and Tuning

  • Online, Self-Paced

Course Description

In this course, you will learn about various ways to monitor Spark applications such as web UIs, metrics, and other monitoring tools. You will also learn about memory tuning.

Learning Objectives

Monitoring Spark Applications

  • start the course
  • access the web user interface
  • use the Spark environment configuration parameters
  • use JSON to query monitoring tools for Spark
  • set JVM fractional memory amounts for Spark

Tuning Spark Applications

  • modify speculation controls for Spark tasks
  • describe data serialization and the role it plays in the performance of Spark applications
  • describe memory management and consumption
  • determine executor memory allocation
  • describe garbage collection tuning
  • set the level of parallelism
  • use the broadcast functionality
  • use query execution plan explainer
  • implement data compression on parquet storage

Practice: Monitoring Spark Applications

  • monitor Spark applications

Framework Connections

  • Icon that says Operate and maintain with a hammer & wrench in x shape depicted.

The materials within this course focus on the Knowledge Skills and Abilities (KSAs) identified within the Specialty Areas listed below. Click to view Specialty Area details within the interactive National Cybersecurity Workforce Framework.

Feedback

If you would like to provide feedback for this course, please e-mail the NICCS SO at NICCS@hq.dhs.gov.