beeline reconnect Spark Thriftserver on alluxio cause NPE

Description

when I use beeline to connect Spark Thriftserver for the first time, its ok to run SQL.

after I close the beeline session and reconnect using beeline again, then execute a new SQL, which brought the following log:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 2018-01-15 16:39:18,167 ERROR thriftserver.SparkExecuteStatementOperation (SparkExecuteStatementOperation.scala:run(178)) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree: (execute plan tree) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:268) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:174) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:171) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:184) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-01-15 16:51:50,689 INFO thrift.ThriftCLIService (ThriftCLIService.java:deleteContext(107)) - Session disconnected without closing properly, close it now Exception in thread "HiveServer2-Handler-Pool: Thread-304" java.lang.RuntimeException: java.lang.NullPointerException at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:83) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) at com.sun.proxy.$Proxy32.close(Unknown Source) at org.apache.hive.service.cli.session.SessionManager.closeSession(SessionManager.java:280) at org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.closeSession(SparkSQLSessionManager.scala:90) at org.apache.hive.service.cli.CLIService.closeSession(CLIService.java:237) at org.apache.hive.service.cli.thrift.ThriftCLIService$1.deleteContext(ThriftCLIService.java:109) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:300) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at alluxio.client.file.FileSystemContext.acquireMasterClient(FileSystemContext.java:187) at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:232) at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:226) at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:316) at alluxio.hadoop.FileSystem.getFileStatus(FileSystem.java:27) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) at org.apache.hadoop.fs.FileSystem.processDeleteOnExit(FileSystem.java:1383) at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2006) at alluxio.hadoop.AbstractFileSystem.close(AbstractFileSystem.java:135) at alluxio.hadoop.FileSystem.close(FileSystem.java:27) at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2726) at org.apache.hadoop.fs.FileSystem.closeAllForUGI(FileSystem.java:457) at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.close(HiveSessionImplwithUGI.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) ... 15 more

Environment

spark 2.2.1, alluxio 1.6.1(two tier), hadoop2.6.0, hive 0.14.0

Status

Assignee

Yupeng Fu

Reporter

xu Bruce

Labels

Time tracking

800h

Components

Fix versions

Affects versions

1.6.1

Priority

Minor
Configure