Apache Spark on Kubernetes


This site is for user documentation for running Apache Spark with a native Kubernetes scheduling backend. This repository apache-spark-on-k8s/spark, contains a fork of Apache Spark that enables running Spark jobs natively on a Kubernetes cluster.

What is this?

This is a collaboratively maintained project working on SPARK-18278. The goal is to bring native support for Spark to use Kubernetes as a cluster manager, in a fully supported way on par with the Spark Standalone, Mesos, and Apache YARN cluster managers.

Relation with apache/spark

This project was put up for voting in an SPIP in August 2017 and passed. It is in the process of being upstreamed into the apache/spark repository.