Need Help <Hadoop ProcessBuilder NullPointer Exception >

Moderator: Concepts and Technologies for DS and BDP

Benutzeravatar
AizazZaidee
BASIC-Programmierer
BASIC-Programmierer
Beiträge: 106
Registriert: 20. Apr 2016 22:49

Need Help <Hadoop ProcessBuilder NullPointer Exception >

Beitrag von AizazZaidee »

Hello guys,
Anyone of you successfully able to run Hadoop examples on Windows Machine? It fails to get process on Windows Machine but on MAC/Linux it's running fine. Here is the stacktrace


Exception in thread "main" java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:483)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:815)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:798)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:731)
at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:489)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:530)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:507)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:305)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:144)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
at de.tud.stg.WordCount.main(WordCount.java:59)


Benutzeravatar
AizazZaidee
BASIC-Programmierer
BASIC-Programmierer
Beiträge: 106
Registriert: 20. Apr 2016 22:49

Re: Need Help <Hadoop ProcessBuilder NullPointer Exception >

Beitrag von AizazZaidee »

Thank you professor I have tried this one as well .. here is a more detailed link http://www.codeproject.com/Articles/757 ... s-Platform but seems like it's not working on Windows some how.

I borrowed MAC from a friend and I was able to reach this following output

Jim 6
Joe 2
Kai 2
Lin 4
Sue 2

I am thinking of putting a hashSet so that I can get the desired output but I am not sure as whether this would be a good solution as right now it is running on single Node but when it would run on multiple nodes how would it relate that (1,4) and (4,1) are same? Please guide me how to solve or point out please where I am doing it wrong. I can share code as well.

meichholz
Moderator
Moderator
Beiträge: 167
Registriert: 30. Mär 2016 08:28

Re: Need Help <Hadoop ProcessBuilder NullPointer Exception >

Beitrag von meichholz »

Hi,

you can also use a virtual machine (e.g. Virtualbox - https://www.virtualbox.org) to run a Linux environment on your Windows machine.

As you can see from the input data, we assume that every friendship is bidirectional. But just because there are two input entries per pair does not mean that you have to count them twice. Maybe you should think about emitting less key-value pairs. It is not necessary to use a Set.

Best,
Matthias

Antworten

Zurück zu „Archiv“