3. Install Spark and creating RDD in Spark

Install spark on mac

brew update
brew install scala
brew install apache-spark

Note: need to install latest JRE and JDK.

start Spark python
type pyspark in your terminal

Exit Spark
Ctrl + D

Things to know:

1. All Spark commands operate on RDD. 
2. You can create RDD from lists using command sc.parallelize (sc represents SparkContext)
3. Many commands are "lazy", (results are not computed until they are used)

Simple example
>> sc.parallelize(range(1,10)).first()
1

list1=sc.parallelize(range(1,1000))
list2=list1.map(lambda x: x*10)  # results are not computed yet
list2.reduce(lambda x,y : x+y) # not lazy, since only report one value
list2.filter(lambda x: x%100==0).collect() # results are computed when needed.

RDD methods are divided into two kinds:

• Transformation  (e.g.,  map, filter, flatmap )
• return another RDD
• are not performed until an action is called
• Action  (e.g.  reduce, take, collect, sum)
• Return a value than an RDD
• Are performed immediately

Exercise:
1. Get an RDD with number from 1 to 10
2. Get all numbers in that RDD which are dividable by 3
3. Get the products of the elements in 2.