让Spark,Java和MongoDB协同工作

类似于我的问题,但这一次是Java,而不是Python,给我带来了问题.

我已经遵循建议的步骤(据我所知),但由于我使用的是hadoop-2.6.1,我认为我应该使用旧的API,而不是示例中提到的新API.

我正在研究Ubuntu和我拥有的各种组件版本

Spark spark-1.5.1-bin-hadoop2.6

Hadoop hadoop-2.6.1

Mongo 3.0.8

通过Maven包含Mongo-Hadoop连接器罐

Java 1.8.0_66

Maven 3.0.5

我的Java程序是基本的

import org.apache.spark.api.java.*;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function;
import com.mongodb.hadoop.MongoInputFormat;
import org.apache.hadoop.conf.Configuration;
import org.bson.BSONObject;

public class SimpleApp {
  public static void main(String[] args) {
    Configuration mongodbConfig = new Configuration();
    mongodbConfig.set("mongo.job.input.format", "com.mongodb.hadoop.MongoInputFormat");
    mongodbConfig.set("mongo.input.uri", "mongodb://localhost:27017/db.collection");
    SparkConf conf = new SparkConf().setAppName("Simple Application");
    JavaSparkContext sc = new JavaSparkContext(conf);
    JavaPairRDD documents = sc.newAPIHadoopRDD(
        mongodbConfig,            // Configuration
        MongoInputFormat.class,   // InputFormat: read from a live cluster.
        Object.class,             // Key class
        BSONObject.class          // Value class
    );
  }
}

edu.berkeley
  simple-project
  4.0.0
  Simple Project
  jar
  1.0
  
     
      org.apache.spark
      spark-core_2.10
      1.5.1
    
    
        org.mongodb
        mongo-java-driver
        3.2.0
    
    
      org.mongodb.mongo-hadoop
      mongo-hadoop-core
      1.4.2
    
  
  
    
        
            org.apache.maven.plugins
            maven-compiler-plugin
            
                1.8
                1.8
            
        
    

/usr/local/share/spark-1.5.1-bin-hadoop2.6/bin/spark-submit --class "SimpleApp" --master local[4] target/simple-project-1.0.jar

Exception in thread "main" java.lang.NoClassDefFoundError: com/mongodb/hadoop/MongoInputFormat
    at SimpleApp.main(SimpleApp.java:18)