1. Introduction to Spark

Here are study notes taken from Spark with python lesson at PyCon 2015.
Curtesy to Speaker, Orlando Karam
Online tutorial
https://www.youtube.com/watch?v=9xYfNznjClE

1. what is Spark?

• Distributed data processing framework
• Based on Resilient Distributed datasets (RDD)
• RDD is basically a huge list of objects
• we can divide RDD in slices and keep them in different nodes
• values are only computed when needed
• fault tolerance, keep info how each slide is calculated so we can re-generate if one node fails
• Immutable! 
• Used for big data processing

Background reading:  Hadoop, RDD, JVM

2. Shared Spark variables

• Broadcast variables (e.g. if you need a constant in your node)
• copy is kept at each node
• Accumulators
• You can only add/update, main node can read end results