» » How to create rdd in apache spark using java

  • RDD is the spark's core abstraction.
  • Full form of RDD is resilient distributed dataset.
  • That means it is immutable collection of objects.
  • Each RDD will split into multiple partitions which may be computed in different machines of cluster
  • We can create Apache spark RDD in two ways
    1. Parallelizing a collection
    2. Loading an external dataset.
  • Now we sill see an example program on creating RDD by parallelizing a collection.
  • In Apache spark JavaSparkContext  class providing parallelize() method.
  • Let us see the simple example program to create Apache spark RDD in java



Program #1: Write a Apache spark java example program to create simple RDD using parallelize method of JavaSparkContext.

  1. package com.instanceofjava.sparkInterview;
  2. import java.util.Arrays;
  3.  
  4. import org.apache.spark.SparkConf;
  5. import org.apache.spark.api.java.JavaRDD;
  6. import org.apache.spark.api.java.JavaSparkContext;
  7.  
  8. /**
  9.  *  Apache spark examples:RDD in spark example program
  10.  * @author www.instanceofjava.com
  11.  */
  12. public class SparkTest {
  13.     
  14.  public static void main(String[] args) {
  15.         
  16.  SparkConf conf = new SparkConf().setMaster("local[2]").setAppName("InstanceofjavaAPP");
  17.  JavaSparkContext sc = new JavaSparkContext(conf);
  18.         
  19.  JavaRDD<String> strRdd=sc.parallelize(Arrays.asList("apache spark element1","apache
  20. spark element2"));
  21.  System.out.println("apache spark rdd created: "+strRdd);
  22.         
  23. /**
  24.  * Return the first element in this RDD.
  25.  */
  26. System.out.println(strRdd.first());
  27.  }
  28.  
  29. }
  30.  
  31. }

Output:

  1. apache spark rdd created: ParallelCollectionRDD[0] at parallelize at SparkTest.java:21
  2. spark element1


create apache spark rdd java

Instance Of Java

We will help you in learning.Please leave your comments and suggestions in comment section. if you any doubts please use search box provided right side. Search there for answers thank you.
«
Next
Newer Post
»
Previous
Older Post

No comments

Leave a Reply

Select Menu