File content inconsistency between alluxio from hdfs

Description

We use presto on alluxio, mount hdfs://ns2 to alluxio /ns2.
Every alluxio worker and every presto are one-to-one correspondence.

After several hours presto-jdbc execute sql. We find that a sql query result was different from the correct result.

After deep search, we found that a alluxio in-mem file named '000274_0' was different from hdfs, but the size of the two file are same.

So, it maybe a bug that during alluxio read and cache file to the alluxio worker.

The attachment is the difference showed by `bcompare`, we find that file content was mismatch from 0x00800000 in hex-diff.
______________

Something I must explain:

The file '000274_0' on hdfs never change.


Reproduction:

  • Presto execute a sql which will query a partition contains somefiles large than 100MB.

  • The Query will fail

  • We can see this partition's files state most are in mem.

  • If you execute free command, the free command will success, but the state is still in mem, the blocks are still in worker's ram disk.

  • If you execute cat command or open file of this partition in browse page, the operation will block.

  • Diff the same block at different worker's ram disk and ufs, those are all different.(use md5sum or beyond compare to diff)


Avoidance:

  • Modified the `alluxio.worker.session.timeout.ms` from 60000 to 600000, the problem has gone.

Environment

Worker count: 15
UFS : hdfs
_______________________
Cluster Usage Summary

Workers Capacity: 900.00GB
Workers Free / Used: 1651.88MB / 898.39GB
UnderFS Capacity: 7.27TB
UnderFS Free / Used: 7.25TB / 25.55GB

——————————————————————————

alluxio.check.inconsistent.client.user hadp
alluxio.check.inconsistent.interval.minute 60
alluxio.conf.dir /software/servers/alluxio-1.4.0/conf
alluxio.debug false
alluxio.extend.worker.bind.hosts.enabled true
alluxio.fuse.cached.paths.max 500
alluxio.fuse.debug.enabled false
alluxio.fuse.fs.name alluxio-fuse
alluxio.fuse.fs.root /
alluxio.fuse.maxwrite.bytes 131072
alluxio.fuse.mount.default /mnt/alluxio
alluxio.home /software/servers/alluxio-1.4.0
alluxio.integration.master.resource.cpu 10
alluxio.integration.master.resource.mem 9216MB
alluxio.integration.mesos.alluxio.jar.url http://downloads.alluxio.org/downloads/files/1.4.0-RC1/alluxio-1.4.0-RC1-bin.tar.gz
alluxio.integration.mesos.jdk.path jdk1.7.0_79
alluxio.integration.mesos.jdk.url https://alluxio-mesos.s3.amazonaws.com/jdk-7u79-linux-x64.tar.gz
alluxio.integration.mesos.master.name AlluxioMaster
alluxio.integration.mesos.master.node.count 1
alluxio.integration.mesos.principal alluxio
alluxio.integration.mesos.role *
alluxio.integration.mesos.user
alluxio.integration.mesos.worker.name AlluxioWorker
alluxio.integration.worker.resource.cpu 10
alluxio.integration.worker.resource.mem 9216MB
alluxio.integration.yarn.workers.per.host.max 1
alluxio.keyvalue.enabled false
alluxio.keyvalue.partition.size.bytes.max 512MB
alluxio.logger.type MASTER_LOGGER
alluxio.logs.dir /data0/alluxio/log
alluxio.master.address alluxio://172.11.11.11:19998
alluxio.master.bind.host 0.0.0.0
alluxio.master.connection.timeout.ms 60000
alluxio.master.file.async.persist.handler alluxio.master.file.async.DefaultAsyncPersistHandler
alluxio.master.format.file_prefix format
alluxio.master.heartbeat.interval.ms 1000
alluxio.master.hostname 172.11.11.11
alluxio.master.journal.flush.batch.time.ms 5
alluxio.master.journal.folder /data0/alluxio/journal
alluxio.master.journal.formatter.class alluxio.master.journal.ProtoBufJournalFormatter
alluxio.master.journal.log.size.bytes.max 10MB
alluxio.master.journal.tailer.shutdown.quiet.wait.time.ms 5000
alluxio.master.journal.tailer.sleep.time.ms 1000
alluxio.master.lineage.checkpoint.class alluxio.master.lineage.checkpoint.CheckpointLatestPlanner
alluxio.master.lineage.checkpoint.interval.ms 300000
alluxio.master.lineage.recompute.interval.ms 300000
alluxio.master.lineage.recompute.log.path /data0/alluxio/log/recompute.log
alluxio.master.port 19998
alluxio.master.retry 29
alluxio.master.startup.consistency.check.enabled false
alluxio.master.tieredstore.global.level0.alias MEM
alluxio.master.tieredstore.global.level1.alias SSD
alluxio.master.tieredstore.global.level2.alias HDD
alluxio.master.tieredstore.global.levels 3
alluxio.master.ttl.checker.interval.ms 3600000
alluxio.master.web.bind.host 0.0.0.0
alluxio.master.web.port 19999
alluxio.master.worker.threads.max 14000
alluxio.master.worker.threads.min 2048
alluxio.master.worker.timeout.ms 300000
alluxio.metrics.conf.file /software/servers/alluxio-1.4.0/conf/metrics.properties
alluxio.network.host.resolution.timeout.ms 5000
alluxio.network.thrift.frame.size.bytes.max 16MB
alluxio.proxy.stream.cache.timeout.ms 3600000
alluxio.proxy.web.bind.host 0.0.0.0
alluxio.proxy.web.port 39999
alluxio.security.authentication.socket.timeout.ms 600000
alluxio.security.authentication.type SIMPLE
alluxio.security.authorization.permission.enabled true
alluxio.security.authorization.permission.supergroup supergroup
alluxio.security.authorization.permission.umask 022
alluxio.security.group.mapping.cache.timeout.ms 60000
alluxio.security.group.mapping.class alluxio.security.group.provider.ShellBasedUnixGroupsMapping
alluxio.site.conf.dir ${user.home}/.alluxio/,/etc/alluxio/
alluxio.test.mode false
alluxio.underfs.address /data0/alluxio/alluxio-data
alluxio.underfs.allow.set.owner.failure false
alluxio.underfs.gcs.owner.id.to.username.mapping
alluxio.underfs.glusterfs.impl org.apache.hadoop.fs.glusterfs.GlusterFileSystem
alluxio.underfs.glusterfs.mapred.system.dir glusterfs:///mapred/system
alluxio.underfs.hdfs.configuration /software/servers/hadoop-2.7.1/etc/hadoop/hdfs-site.xml
alluxio.underfs.hdfs.impl org.apache.hadoop.hdfs.DistributedFileSystem
alluxio.underfs.hdfs.prefixes hdfs://,glusterfs:///
alluxio.underfs.hdfs.remote false
alluxio.underfs.listing.length 1000
alluxio.underfs.object.store.mount.shared.publicly false
alluxio.underfs.oss.connection.max 1024
alluxio.underfs.oss.connection.timeout.ms 50000
alluxio.underfs.oss.connection.ttl -1
alluxio.underfs.oss.socket.timeout.ms 50000
alluxio.underfs.s3.admin.threads.max 20
alluxio.underfs.s3.disable.dns.buckets false
alluxio.underfs.s3.owner.id.to.username.mapping
alluxio.underfs.s3.proxy.https.only true
alluxio.underfs.s3.threads.max 22
alluxio.underfs.s3.upload.threads.max 2
alluxio.underfs.s3a.consistency.timeout.ms 60000
alluxio.underfs.s3a.inherit_acl true
alluxio.underfs.s3a.secure.http.enabled false
alluxio.underfs.s3a.server.side.encryption.enabled false
alluxio.underfs.s3a.socket.timeout.ms 50000
alluxio.user.block.master.client.threads 512
alluxio.user.block.remote.read.buffer.size.bytes 8MB
alluxio.user.block.remote.reader.class alluxio.client.netty.NettyRemoteBlockReader
alluxio.user.block.remote.writer.class alluxio.client.netty.NettyRemoteBlockWriter
alluxio.user.block.size.bytes.default 128MB
alluxio.user.block.worker.client.pool.gc.threshold.ms 300000
alluxio.user.block.worker.client.pool.size.max 128
alluxio.user.block.worker.client.threads 2048
alluxio.user.failed.space.request.limits 3
alluxio.user.file.buffer.bytes 3MB
alluxio.user.file.cache.partially.read.block true
alluxio.user.file.master.client.threads 64
alluxio.user.file.readtype.default CACHE_PROMOTE
alluxio.user.file.seek.buffer.size.bytes 1MB
alluxio.user.file.waitcompleted.poll.ms 1000
alluxio.user.file.worker.client.pool.gc.threshold.ms 300000
alluxio.user.file.worker.client.pool.size.max 128
alluxio.user.file.worker.client.threads 10
alluxio.user.file.write.location.policy.class alluxio.client.file.policy.LocalFirstPolicy
alluxio.user.file.write.tier.default HIGHEST
alluxio.user.file.writetype.default MUST_CACHE
alluxio.user.heartbeat.interval.ms 1000
alluxio.user.lineage.enabled false
alluxio.user.lineage.master.client.threads 10
alluxio.user.local.reader.packet.size.bytes 8MB
alluxio.user.local.writer.packet.size.bytes 64KB
alluxio.user.network.netty.channel EPOLL
alluxio.user.network.netty.channel.pool.disabled false
alluxio.user.network.netty.channel.pool.gc.threshold.ms 300000
alluxio.user.network.netty.channel.pool.size.max 1024
alluxio.user.network.netty.reader.buffer.size.packets 16
alluxio.user.network.netty.reader.cancel.enabled true
alluxio.user.network.netty.timeout.ms 60000
alluxio.user.network.netty.worker.threads 200
alluxio.user.network.netty.writer.buffer.size.packets 16
alluxio.user.network.netty.writer.packet.size.bytes 64KB
alluxio.user.packet.streaming.enabled false
alluxio.user.ufs.delegation.enabled true
alluxio.user.ufs.delegation.read.buffer.size.bytes 8MB
alluxio.user.ufs.delegation.write.buffer.size.bytes 2MB
alluxio.user.ufs.file.reader.class alluxio.client.netty.NettyUnderFileSystemFileReader
alluxio.user.ufs.file.writer.class alluxio.client.netty.NettyUnderFileSystemFileWriter
alluxio.version 1.4.0-RC1
alluxio.web.resources /software/servers/alluxio-1.4.0/core/server/src/main/webapp
alluxio.web.threads 1
alluxio.work.dir /software/servers/alluxio-1.4.0
alluxio.worker.allocator.class alluxio.worker.block.allocator.MaxFreeAllocator
alluxio.worker.bind.host 0.0.0.0
alluxio.worker.block.heartbeat.interval.ms 1000
alluxio.worker.block.heartbeat.timeout.ms 60000
alluxio.worker.block.threads.max 2048
alluxio.worker.block.threads.min 256
alluxio.worker.data.bind.host 0.0.0.0
alluxio.worker.data.folder /alluxioworker/
alluxio.worker.data.folder.tmp .tmp_blocks
alluxio.worker.data.port 29999
alluxio.worker.data.server.class alluxio.worker.netty.NettyDataServer
alluxio.worker.data.tmp.subdir.max 1024
alluxio.worker.evictor.class alluxio.worker.block.evictor.LRUEvictor
alluxio.worker.evictor.lrfu.attenuation.factor 2.0
alluxio.worker.evictor.lrfu.step.factor 0.25
alluxio.worker.file.buffer.size 1MB
alluxio.worker.file.persist.pool.size 64
alluxio.worker.file.persist.rate.limit 2GB
alluxio.worker.file.persist.rate.limit.enabled false
alluxio.worker.filesystem.heartbeat.interval.ms 1000
alluxio.worker.memory.size 60GB
alluxio.worker.network.netty.block.reader.threads.max 128
alluxio.worker.network.netty.block.writer.threads.max 128
alluxio.worker.network.netty.boss.threads 1
alluxio.worker.network.netty.channel EPOLL
alluxio.worker.network.netty.file.reader.threads.max 128
alluxio.worker.network.netty.file.transfer MAPPED
alluxio.worker.network.netty.file.writer.threads.max 128
alluxio.worker.network.netty.reader.buffer.size.packets 16
alluxio.worker.network.netty.reader.packet.size.bytes 64KB
alluxio.worker.network.netty.shutdown.quiet.period 2
alluxio.worker.network.netty.shutdown.timeout 15
alluxio.worker.network.netty.watermark.high 32KB
alluxio.worker.network.netty.watermark.low 8KB
alluxio.worker.network.netty.worker.threads 0
alluxio.worker.network.netty.writer.buffer.size.packets 16
alluxio.worker.port 29998
alluxio.worker.session.timeout.ms 600000
alluxio.worker.tieredstore.block.lock.readers 1000
alluxio.worker.tieredstore.block.locks 1000
alluxio.worker.tieredstore.level0.alias MEM
alluxio.worker.tieredstore.level0.dirs.path /mnt/ramdisk
alluxio.worker.tieredstore.level0.dirs.quota 60GB
alluxio.worker.tieredstore.level0.reserved.ratio 0.1
alluxio.worker.tieredstore.levels 1
alluxio.worker.tieredstore.reserver.enabled false
alluxio.worker.tieredstore.reserver.interval.ms 1000
alluxio.worker.tieredstore.retry 3
alluxio.worker.web.bind.host 0.0.0.0
alluxio.worker.web.port 30000
alluxio.zookeeper.election.path /election
alluxio.zookeeper.enabled false
alluxio.zookeeper.leader.inquiry.retry 10
alluxio.zookeeper.leader.path /leader

Status

Assignee

Yupeng Fu

Reporter

maobaolong

Labels

Components

Affects versions

1.4.0
1.6.1

Priority

Critical