Spark process report "Path is no longer valid, possibly due to a concurrent delete."

Description

the error log:

```
java.io.IOException: alluxio.exception.AlluxioException: Path is no longer valid, possibly due to a concurrent delete.
at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:349)
at alluxio.hadoop.FileSystem.getFileStatus(FileSystem.java:27)
at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1487)
at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1500)
at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1500)
at org.apache.hadoop.fs.FileSystem.getContentSummary(FileSystem.java:1500)
at org.apache.spark.sql.FileUtils$.getDirOrFileTotalSize(FileUtils.scala:232)
at org.apache.spark.sql.ExecuteSparkPlanOperation.updateDataSizeAndLastUpdateTime(ExecuteSparkPlanOperation.scala:696)
at org.apache.spark.sql.ExecuteSparkPlanOperation.org$apache$spark$sql$ExecuteSparkPlanOperation$$execute(ExecuteSparkPlanOperation.scala:573)
at org.apache.spark.sql.ExecuteSparkPlanOperation$$anon$1.run(ExecuteSparkPlanOperation.scala:91)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: alluxio.exception.AlluxioException: Path is no longer valid, possibly due to a concurrent delete.
at alluxio.exception.status.AlluxioStatusException.toAlluxioException(AlluxioStatusException.java:119)
at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:248)
at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:234)
at alluxio.hadoop.AbstractFileSystem.getFileStatus(AbstractFileSystem.java:345)
... 14 more
Caused by: alluxio.exception.status.InvalidArgumentException: Path is no longer valid, possibly due to a concurrent delete.
at alluxio.exception.status.AlluxioStatusException.from(AlluxioStatusException.java:148)
at alluxio.exception.status.AlluxioStatusException.fromThrift(AlluxioStatusException.java:130)
at alluxio.AbstractClient.retryRPCInternal(AbstractClient.java:429)
at alluxio.AbstractClient.retryRPC(AbstractClient.java:407)
at alluxio.client.file.RetryHandlingFileSystemMasterClient.getStatus(RetryHandlingFileSystemMasterClient.java:137)
at alluxio.client.file.BaseFileSystem.getStatus(BaseFileSystem.java:242)
... 16 more
```

Environment

None

Status

Assignee

Gene Pang

Reporter

ligq

Labels

None

Components

Affects versions

1.8.0

Priority

Major