4. Reading files and ReduceByKey

sc.textFile (urlOrPath, minPartitions, useUnicode=True)

• returns an rdd of strings (one per line)
• can read from many files, using wildcards (*)
• can read from hdfs, ...
• we normally use map right after and split/parse the lines.

Example

people=sc.textFile("../data/people.txt")

#get first line

people.first()

people=sc.textFile("../data/people.txt").map(lambda x:x.split("\t")).first()

Tuples and ReduceByKey

Many times we want to group elements first, and then calculate values for each group.

People example

we have a people.txt file with following schema:

Name | Gender | Age | Favourite Language

we can load with

people=sc.textFile("../data/people.txt").map(lambda x:x.split("\t"))

• Find number of people by gender
• first get tuples like: ("M",1), ("F", 1) ...  then reduce by key
• people.map(lambda t: (t[1],1))
• then use reduce function to sum the numbers
• people.map(lambda t: (t[1],1)).reduceByKey(lambda x,y: x+y).collect()
• Find number of people by favourite programming language
• people.map(lambda t: (t[3],1)).reduceByKey(lambda x,y: x+y).collect()
• Find youngest person per gender
• people.map(lambda t: (t[1],int(t[2]))).reduceByKey(lambda x,y: min(x,y)).collect()

TIP: always  use .first() after map() operation to check if you get the tuples you expected.