Spark Integration checker

Description

Users nowadays use Alluxio to interact with many computation frameworks including but not limited to Spark, MapReduce, Hive, and HBase. Different frameworks require different configurations. Additional configuration steps are required in order to allow these frameworks to work well with Alluxio. Just missing one step will cause the failure of a framework job.

Many Alluxio users may not be familiar with configuring Alluxio with frameworks and it may be hard for them to figure out what makes jobs fail. We hope to provide a tool for users to test whether they have successfully configured Alluxio with selected frameworks. If a framework job fails, integration checker can tell them what causes the failure and provide them with useful information for future configuration and debugging.

This tool will target checking Spark with Alluxio integration at the beginning.

Environment

None
Fixed

Assignee

Lu Qiu

Reporter

Lu Qiu

Labels

None

Fix versions

Affects versions

Priority

Major