Basic Transformation - RDD

Moderator: Concepts and Technologies for DS and BDP

saxenaankur789
Neuling
Neuling
Beiträge: 5
Registriert: 5. Mai 2016 09:23

Basic Transformation - RDD

Beitrag von saxenaankur789 »

Hi ,
Can anyone let me know how the flatMap transformation is used ? There is an example on slides which states that -

nums = sc.parallelize([1, 2, 3])

# Map each element to zero or more others
nums.flatMap(lambda x: range(0, x)) # => {0, 0, 1, 0,
1, 2}

I am not sure how we achieve this output post applying FlatMap on nums ? If anyone has any idea , kindly help me out.

Regards
AS

bauser
Neuling
Neuling
Beiträge: 2
Registriert: 12. Okt 2011 19:02

Re: Basic Transformation - RDD

Beitrag von bauser »

range(0,x) creates a sequence from zero to (not including) x

range(0,1) #=> [0]
range(0,2) #=> [0,1]
range(0,3) #=> [0,1,2]

if using map you would get:
[[0],[0,1],[0,1,2]]

flatMap creates a single sequence from all the results, so instead of a list of lists you get just one list:
[0,0,1,0,1,2]

saxenaankur789
Neuling
Neuling
Beiträge: 5
Registriert: 5. Mai 2016 09:23

Re: Basic Transformation - RDD

Beitrag von saxenaankur789 »

Thanks for the explanation !

Antworten

Zurück zu „Archiv“