summary

CLI hang in HDFS shutdown hook

Description

I can reproduce this consistently by running

hdfs dfs -mkdir alluxio://zk@@master1:2181;master2:2181/test

after launching that command, the shell becomes frozen for over a minute, not even responding to Ctrl-C. Looking at the jstack, it seems like the metrics heartbeat is to blame. We need to set a short time limit on the final heartbeat (e.g. 500ms), and ideally also make it interruptible.

"Thread-9" #24 prio=5 os_prio=0 tid=0x00007fe5001e4000 nid=0xf1e waiting for monitor entry [0x00007fe4f5938000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at alluxio.AbstractClient.close(AbstractClient.java:268)
        - waiting to lock <0x00000000ee25a3c0> (a alluxio.client.metrics.MetricsMasterClient)
        at alluxio.client.file.FileSystemContext.closeInternal(FileSystemContext.java:285)
        - locked <0x00000000ee257c40> (a alluxio.client.file.FileSystemContext)
        at alluxio.client.file.FileSystemContext.close(FileSystemContext.java:266)
        at alluxio.hadoop.AbstractFileSystem.close(AbstractFileSystem.java:151)
        at alluxio.hadoop.FileSystem.close(FileSystem.java:27)
        at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2913)
        - locked <0x00000000f3001718> (a org.apache.hadoop.fs.FileSystem$Cache)
        at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2930)
        - locked <0x00000000f30cad50> (a org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer)
        at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

Environment

None

Status

Assignee

Unassigned

Reporter

Andrew Audibert

Labels

None

Components

Core - Client

Affects versions

1.8.0

Priority

Blocker