In some cases, file deletion errors are thrown when creating a file
Description
The code:
if (!fs.exists(ignoreFile)) {
val stream = fs.create(ignoreFile)
stream.close()
}
The stack:
2018-12-05 14:13:30,727 | ERROR | Error executing query, currentState RUNNING
java.io.IOException: alluxio.exception.DirectoryNotEmptyException: Failed to delete XXX (UFS delete file failed) from the under file system
at alluxio.hadoop.AbstractFileSystem.create(AbstractFileSystem.java:204)
at alluxio.hadoop.FileSystem.create(FileSystem.java:27)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:929)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:910)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:807)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:796)
...
Caused by: alluxio.exception.DirectoryNotEmptyException: Failed to delete XXX (UFS delete file failed) from the under file system
at alluxio.client.file.BaseFileSystem.delete(BaseFileSystem.java:166)
at alluxio.client.file.BaseFileSystem.delete(BaseFileSystem.java:154)
at alluxio.hadoop.AbstractFileSystem.create(AbstractFileSystem.java:200)
... 25 more
Environment
Activity
This is because the overwrite parameter is set to true. We will attempt to delete the file if it already exists. Are there multiple concurrent creators of the file (since you did an exists check before)?
If you would rather have it fail, you can set the overwrite parameter to false and catch the exception later.
Yes, there multiple concurrent creators of the file. Thank you for your commit.
Thanks for confirming. I would suggest to do some application level locking to prevent this, or if you have a semantic of first writer wins, you can change the overwrite flag to false. If you want last writer wins, you could have them create a temporary file and then rename to the final destination file.
Is there any Alluxio change that you think would help?
A deletion error occurred while creating the file, and the user was confused.
I hope alluxio can give reasonable information.
in high level to repro: a create / delete race
