Example from RDD and other Technologies

Moderator: Concepts and Technologies for DS and BDP

oek
Erstie
Erstie
Beiträge: 21
Registriert: 11. Sep 2013 08:42

Example from RDD and other Technologies

Beitrag von oek »

Hi,

at the exam preparation was a question about the underlying data structure from Apache Spark and we shall show a minimal example.
The technology is RDD and in the lecture about Spark/RDD is the following example:

lines = spark.textFile("hdfs://...")
errors = lines.filter(lambda s: s.startswith("ERROR"))
messages = errors.map(lambda s: s.split('\t'))
messages = cache()
messages.filter(lambda s: "foo" in s).count()

Is this example sufficient for the exam?
Is the scope of this example also sufficient for other technologies like MapReduce and Scala?


Best regards,
Oliver

salvaneschi
Moderator
Moderator
Beiträge: 49
Registriert: 29. Mär 2013 23:51

Re: Example from RDD and other Technologies

Beitrag von salvaneschi »

> Is this example sufficient for the exam?
> Is the scope of this example also sufficient for other technologies like MapReduce and Scala?

Sorry, I don't understand the question. Can you please explain what yu mean?

oek
Erstie
Erstie
Beiträge: 21
Registriert: 11. Sep 2013 08:42

Re: Example from RDD and other Technologies

Beitrag von oek »

Ist the example enough to get the full points at the exam?

salvaneschi
Moderator
Moderator
Beiträge: 49
Registriert: 29. Mär 2013 23:51

Re: Example from RDD and other Technologies

Beitrag von salvaneschi »

If the question asks for an example of RDDs yes. Depending on how the question is formulated you may want to add an explanation of why this example is relevant and where the RDDs are in the example.

Antworten

Zurück zu „Archiv“