4. Reading files and ReduceByKey

sc.textFile (urlOrPath, minPartitions, useUnicode=True)

• returns an rdd of strings (one per line)
• can read from many files, using wildcards (*)
• can read from hdfs, ...
• we normally use map right after and split/parse the lines.

Example

people=sc.textFile("../data/people.txt")

#get first line

people.first()

people=sc.textFile("../data/people.txt").map(lambda x:x.split("\t")).first()

Tuples and ReduceByKey

Many times we want to group elements first, and then calculate values for each group.

People example

we have a people.txt file with following schema:

Name | Gender | Age | Favourite Language

we can load with

people=sc.textFile("../data/people.txt").map(lambda x:x.split("\t"))

• Find number of people by gender
• first get tuples like: ("M",1), ("F", 1) ...  then reduce by key
• people.map(lambda t: (t[1],1))
• then use reduce function to sum the numbers
• people.map(lambda t: (t[1],1)).reduceByKey(lambda x,y: x+y).collect()
• Find number of people by favourite programming language
• people.map(lambda t: (t[3],1)).reduceByKey(lambda x,y: x+y).collect()
• Find youngest person per gender
• people.map(lambda t: (t[1],int(t[2]))).reduceByKey(lambda x,y: min(x,y)).collect()

TIP: always  use .first() after map() operation to check if you get the tuples you expected.

3. Install Spark and creating RDD in Spark

Install spark on mac

brew update
brew install scala
brew install apache-spark

Note: need to install latest JRE and JDK.

start Spark python
type pyspark in your terminal

Exit Spark
Ctrl + D

Things to know:

1. All Spark commands operate on RDD. 
2. You can create RDD from lists using command sc.parallelize (sc represents SparkContext)
3. Many commands are "lazy", (results are not computed until they are used)

Simple example
>> sc.parallelize(range(1,10)).first()
1

list1=sc.parallelize(range(1,1000))
list2=list1.map(lambda x: x*10)  # results are not computed yet
list2.reduce(lambda x,y : x+y) # not lazy, since only report one value
list2.filter(lambda x: x%100==0).collect() # results are computed when needed.

RDD methods are divided into two kinds:

• Transformation  (e.g.,  map, filter, flatmap )
• return another RDD
• are not performed until an action is called
• Action  (e.g.  reduce, take, collect, sum)
• Return a value than an RDD
• Are performed immediately

Exercise:
1. Get an RDD with number from 1 to 10
2. Get all numbers in that RDD which are dividable by 3
3. Get the products of the elements in 2.

2. Functional programming in Python

1. Functional tools in python

• map
• apply an operation to each element in a list, return a list of results. (original list is never changed, map just creates a new list)
• filter
• select only certain elements from a list (based on a boolean list)
• reduce
• apply a function to all pairs (non-repeated) of elements in a list, return ONE value, not a list.
• better for functions that are communicative, such as multiply/add, but not for divide/minus
• lambda
• allow us define a function with a value, without giving it a name
• itertools
• chain, chain.from_iterable
• compress
• accumulate
• takewhile,  dropwhile
• flatmap

additional reading about itertools:
https://docs.python.org/dev/library/itertools.html

2.  code examples
>>> map(lambda x: x+1, [1,2,3])
[2,3,4]
>>> filter(lambda x: x%2==1, range(1,10))
[1,3,5,7,9]
>>> reduce(lambda x, y: x+y, [1,2,3,4])
10

3. Exercises

(lambda x: x*2)(3) => ?
map(lambda x: x*2, [1,2,3]) =>?
map(lambda t: t[0], [(1,2), (3,4), (5,6)]) =>?
reduce(lambda x, y: x+y, [1,2,3]) =>?
reduce(lambda x, y: x+y,  map(lambda t: t[0], [(1,2), (3,4), (5,6)) =>?

Given a list of tuples: a=[(1,2), (3,4), (5,6)]
write an expression to get only the second element of each tuple.
write an expression to get the sum of second elements.

1. Introduction to Spark

Here are study notes taken from Spark with python lesson at PyCon 2015.
Curtesy to Speaker, Orlando Karam
Online tutorial
https://www.youtube.com/watch?v=9xYfNznjClE

1. what is Spark?

• Distributed data processing framework
• Based on Resilient Distributed datasets (RDD)
• RDD is basically a huge list of objects
• we can divide RDD in slices and keep them in different nodes
• values are only computed when needed
• fault tolerance, keep info how each slide is calculated so we can re-generate if one node fails
• Immutable! 
• Used for big data processing

Background reading:  Hadoop, RDD, JVM

2. Shared Spark variables

• Broadcast variables (e.g. if you need a constant in your node)
• copy is kept at each node
• Accumulators
• You can only add/update, main node can read end results