NPE when MR tries to use Tachyon

Description

This logic is fixed in https://github.com/amplab/tachyon/pull/427, but as of master right now, MR doesn't seem to work with Tachyon

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Error: java.lang.NullPointerException at tachyon.client.FileOutStream.write(FileOutStream.java:213) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:50) at java.io.DataOutputStream.writeByte(DataOutputStream.java:153) at org.apache.hadoop.io.WritableUtils.writeVLong(WritableUtils.java:273) at org.apache.hadoop.io.WritableUtils.writeVInt(WritableUtils.java:253) at org.apache.hadoop.io.Text.writeString(Text.java:475) at org.apache.hadoop.io.SequenceFile$Writer.writeFileHeader(SequenceFile.java:1146) at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1214) at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1091) at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:270) at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:527) at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64) at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:624) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:744) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Setup

1 2 3 4 # on all nodes cp /opt/tachyon/tachyon-0.6.0-SNAPSHOT/client/target/tachyon-client-0.6.0-SNAPSHOT-jar-with-dependencies.jar /usr/lib/hadoop/lib/ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar randomtextwriter tachyon://rhel65-1.localdomain:19998/randomInput

Environment

None

Status

Assignee

miasantreble

Reporter

David Capwell

Labels

Fix versions

Affects versions

0.5.0

Priority

Blocker
Configure