MR - Reduce starting time

Moderator: Concepts and Technologies for DS and BDP

robhe
Neuling
Neuling
Beiträge: 5
Registriert: 11. Jul 2017 16:59

MR - Reduce starting time

Beitrag von robhe »

Hello,

I'm a little bit confused when the reduce task is starting relatively to the map task.
Is it correct that the reduce calculation starts only if every map task has finished, but the reduce workers are already reading the results of the map tasks while the map tasks are still in process?

Thank you
Robin

yc81reja
Erstie
Erstie
Beiträge: 12
Registriert: 18. Apr 2015 16:25

Re: MR - Reduce starting time

Beitrag von yc81reja »

I wouldn't think that the reducers start reading while mappers are still working because the scheduler shuffles the intermediate (key, values) and assigns them to reducers. If reducers start reading before all mappers finish, the partitions they read are incomplete.
Please take a look at page 15 of slide 4 where a barrier aggregating intermediate values exists between map tasks and reduce tasks

robhe
Neuling
Neuling
Beiträge: 5
Registriert: 11. Jul 2017 16:59

Re: MR - Reduce starting time

Beitrag von robhe »

Thank you for the answer but I think this is contrarily to some statements in the MapReduce paper:
"When a map task is executed first by worker A and then later executed by worker B (because A failed), all workers executing reduce tasks are notified of the reexecution. Any reduce task that has not already read the data from worker A will read the data from worker B."

yc81reja
Erstie
Erstie
Beiträge: 12
Registriert: 18. Apr 2015 16:25

Re: MR - Reduce starting time

Beitrag von yc81reja »

Well, I would assume that since each worker stores their intermediate values on local storage and reducers are reading them from a mapper, if the mapper is dead after completion, the reducer cannot read the intermediate values from the dead worker because the values are written to its local storage. And I think this should be the reason why even completed mappers must be re-executed when a worker on that node fails.

robhe
Neuling
Neuling
Beiträge: 5
Registriert: 11. Jul 2017 16:59

Re: MR - Reduce starting time

Beitrag von robhe »

Yes, I agree that this should be the reason why even completed mappers must be re-executed when a worker on that node fails.

However, "Any reduce task that has not already read the data from worker A will read the data from worker B" means that there could be reduce tasks that have already read the data from worker A and this would imply that there are reading reduce tasks while mapping is not finished.

yc81reja
Erstie
Erstie
Beiträge: 12
Registriert: 18. Apr 2015 16:25

Re: MR - Reduce starting time

Beitrag von yc81reja »

Oh, sorry I think my previous replies are wrong. Now I got what you mean. You are right on your first post.
In the paper section 3.1 step 4 it says the reducer has started reading when the output of writers are written on the local disk but until step 5 when the reducers have finished reading and sorting all intermediate (key, value) pairs, reducers do not compute. The barrier stands between reducers reading/sorting operations and reduce computation. And shuffles are not handled by reducers so they do not need to wait for all shuffles done.
Thanks a lot for pointing this subtle detail out

robhe
Neuling
Neuling
Beiträge: 5
Registriert: 11. Jul 2017 16:59

Re: MR - Reduce starting time

Beitrag von robhe »

You are right, this paper section should be the answer.
Thanks a lot for the help!

Antworten

Zurück zu „Archiv“