ThreadPool is full and new task has been rejected

Description

Thread Pool Full

problem description

A huge warn log happened to the `master.log`.

1 2017-02-03 17:28:55,438 WARN server.TThreadPoolServer (TThreadPoolServer.java:serve) - Task has been rejected by ExecutorService 9 times till timedout, reason: java.util.concurrent.RejectedExecutionException: Task org.apache.thrift.server.TThreadPoolServer$WorkerProcess@4a20b9cb rejected from java.util.concurrent.ThreadPoolExecutor@2a85e472[Running, pool size = 14000, active threads = 14000, queued tasks = 0, completed tasks = 3]

I think thread pool must be full and reject new coming task.

My `alluxio-site.properties` is:

1 2 alluxio.master.worker.threads.max=14000 alluxio.master.worker.threads.min=2048

A sim test program

So I write a test program to simulate the problem. I let both the Alluxio master and the test program running on the same computer.

The test description is in the github readme.md

[github](https://github.com/maobaolong/testAlluxio).

Test conclusion

The handle function of the file open and close between hdfs and Alluxio are different, hdfs namenode can
close a unused connection automatically, but alluxio master hold the connection until the client
process terminate. So if there are a lot of client connect to alluxio master and keep alive,
alluxio's thread pool will full although every client call close in the right way.

Environment

CentOS Linux release 7.2.1511 64bit
jdk1.8.0_65
a hdfs client and a alluxio master node.

Status

Assignee

wangzhehan

Reporter

maobaolong

Labels

Time tracking

40h

Components

Affects versions

1.4.0
1.5.0

Priority

Critical
Configure