CLI hang in HDFS shutdown hook

Description

I can reproduce this consistently by running

1 hdfs dfs -mkdir alluxio://zk@@master1:2181;master2:2181/test

after launching that command, the shell becomes frozen for over a minute, not even responding to Ctrl-C. Looking at the jstack, it seems like the metrics heartbeat is to blame. We need to set a short time limit on the final heartbeat (e.g. 500ms), and ideally also make it interruptible.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 "Thread-9" #24 prio=5 os_prio=0 tid=0x00007fe5001e4000 nid=0xf1e waiting for monitor entry [0x00007fe4f5938000] java.lang.Thread.State: BLOCKED (on object monitor) at alluxio.AbstractClient.close(AbstractClient.java:268) - waiting to lock <0x00000000ee25a3c0> (a alluxio.client.metrics.MetricsMasterClient) at alluxio.client.file.FileSystemContext.closeInternal(FileSystemContext.java:285) - locked <0x00000000ee257c40> (a alluxio.client.file.FileSystemContext) at alluxio.client.file.FileSystemContext.close(FileSystemContext.java:266) at alluxio.hadoop.AbstractFileSystem.close(AbstractFileSystem.java:151) at alluxio.hadoop.FileSystem.close(FileSystem.java:27) at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2913) - locked <0x00000000f3001718> (a org.apache.hadoop.fs.FileSystem$Cache) at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2930) - locked <0x00000000f30cad50> (a org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

Environment

None

Status

Assignee

Unassigned

Reporter

Andrew Audibert

Labels

None

Components

Affects versions

1.8.0

Priority

Blocker