File content inconsistency between alluxio from hdfs

Description

We use presto on alluxio, mount hdfs://ns2 to alluxio /ns2.
Every alluxio worker and every presto are one-to-one correspondence.

After several hours presto-jdbc execute sql. We find that a sql query result was different from the correct result.

After deep search, we found that a alluxio in-mem file named '000274_0' was different from hdfs, but the size of the two file are same.

So, it maybe a bug that during alluxio read and cache file to the alluxio worker.

The attachment is the difference showed by `bcompare`, we find that file content was mismatch from 0x00800000 in hex-diff.
______________

Something I must explain:

The file '000274_0' on hdfs never change.


Reproduction:

  • Presto execute a sql which will query a partition contains somefiles large than 100MB.

  • The Query will fail

  • We can see this partition's files state most are in mem.

  • If you execute free command, the free command will success, but the state is still in mem, the blocks are still in worker's ram disk.

  • If you execute cat command or open file of this partition in browse page, the operation will block.

  • Diff the same block at different worker's ram disk and ufs, those are all different.(use md5sum or beyond compare to diff)


Avoidance:

  • Modified the `alluxio.worker.session.timeout.ms` from 60000 to 600000, the problem has gone.

Environment

Worker count: 15
UFS : hdfs
_______________________
Cluster Usage Summary

Workers Capacity: 900.00GB
Workers Free / Used: 1651.88MB / 898.39GB
UnderFS Capacity: 7.27TB
UnderFS Free / Used: 7.25TB / 25.55GB

——————————————————————————

alluxio.check.inconsistent.client.user hadp
alluxio.check.inconsistent.interval.minute 60
alluxio.conf.dir /software/servers/alluxio-1.4.0/conf
alluxio.debug false
alluxio.extend.worker.bind.hosts.enabled true
alluxio.fuse.cached.paths.max 500
alluxio.fuse.debug.enabled false
alluxio.fuse.fs.name alluxio-fuse
alluxio.fuse.fs.root /
alluxio.fuse.maxwrite.bytes 131072
alluxio.fuse.mount.default /mnt/alluxio
alluxio.home /software/servers/alluxio-1.4.0
alluxio.integration.master.resource.cpu 10
alluxio.integration.master.resource.mem 9216MB
alluxio.integration.mesos.alluxio.jar.url http://downloads.alluxio.org/downloads/files/1.4.0-RC1/alluxio-1.4.0-RC1-bin.tar.gz
alluxio.integration.mesos.jdk.path jdk1.7.0_79
alluxio.integration.mesos.jdk.url https://alluxio-mesos.s3.amazonaws.com/jdk-7u79-linux-x64.tar.gz
alluxio.integration.mesos.master.name AlluxioMaster
alluxio.integration.mesos.master.node.count 1
alluxio.integration.mesos.principal alluxio
alluxio.integration.mesos.role *
alluxio.integration.mesos.user
alluxio.integration.mesos.worker.name AlluxioWorker
alluxio.integration.worker.resource.cpu 10
alluxio.integration.worker.resource.mem 9216MB
alluxio.integration.yarn.workers.per.host.max 1
alluxio.keyvalue.enabled false
alluxio.keyvalue.partition.size.bytes.max 512MB
alluxio.logger.type MASTER_LOGGER
alluxio.logs.dir /data0/alluxio/log
alluxio.master.address alluxio://172.11.11.11:19998
alluxio.master.bind.host 0.0.0.0
alluxio.master.connection.timeout.ms 60000
alluxio.master.file.async.persist.handler alluxio.master.file.async.DefaultAsyncPersistHandler
alluxio.master.format.file_prefix format
alluxio.master.heartbeat.interval.ms 1000
alluxio.master.hostname 172.11.11.11
alluxio.master.journal.flush.batch.time.ms 5
alluxio.master.journal.folder /data0/alluxio/journal
alluxio.master.journal.formatter.class alluxio.master.journal.ProtoBufJournalFormatter
alluxio.master.journal.log.size.bytes.max 10MB
alluxio.master.journal.tailer.shutdown.quiet.wait.time.ms 5000
alluxio.master.journal.tailer.sleep.time.ms 1000
alluxio.master.lineage.checkpoint.class alluxio.master.lineage.checkpoint.CheckpointLatestPlanner
alluxio.master.lineage.checkpoint.interval.ms 300000
alluxio.master.lineage.recompute.interval.ms 300000
alluxio.master.lineage.recompute.log.path /data0/alluxio/log/recompute.log
alluxio.master.port 19998
alluxio.master.retry 29
alluxio.master.startup.consistency.check.enabled false
alluxio.master.tieredstore.global.level0.alias MEM
alluxio.master.tieredstore.global.level1.alias SSD
alluxio.master.tieredstore.global.level2.alias HDD
alluxio.master.tieredstore.global.levels 3
alluxio.master.ttl.checker.interval.ms 3600000
alluxio.master.web.bind.host 0.0.0.0
alluxio.master.web.port 19999
alluxio.master.worker.threads.max 14000
alluxio.master.worker.threads.min 2048
alluxio.master.worker.timeout.ms 300000
alluxio.metrics.conf.file /software/servers/alluxio-1.4.0/conf/metrics.properties
alluxio.network.host.resolution.timeout.ms 5000
alluxio.network.thrift.frame.size.bytes.max 16MB
alluxio.proxy.stream.cache.timeout.ms 3600000
alluxio.proxy.web.bind.host 0.0.0.0
alluxio.proxy.web.port 39999
alluxio.security.authentication.socket.timeout.ms 600000
alluxio.security.authentication.type SIMPLE
alluxio.security.authorization.permission.enabled true
alluxio.security.authorization.permission.supergroup supergroup
alluxio.security.authorization.permission.umask 022
alluxio.security.group.mapping.cache.timeout.ms 60000
alluxio.security.group.mapping.class alluxio.security.group.provider.ShellBasedUnixGroupsMapping
alluxio.site.conf.dir ${user.home}/.alluxio/,/etc/alluxio/
alluxio.test.mode false
alluxio.underfs.address /data0/alluxio/alluxio-data
alluxio.underfs.allow.set.owner.failure false
alluxio.underfs.gcs.owner.id.to.username.mapping
alluxio.underfs.glusterfs.impl org.apache.hadoop.fs.glusterfs.GlusterFileSystem
alluxio.underfs.glusterfs.mapred.system.dir glusterfs:///mapred/system
alluxio.underfs.hdfs.configuration /software/servers/hadoop-2.7.1/etc/hadoop/hdfs-site.xml
alluxio.underfs.hdfs.impl org.apache.hadoop.hdfs.DistributedFileSystem
alluxio.underfs.hdfs.prefixes hdfs://,glusterfs:///
alluxio.underfs.hdfs.remote false
alluxio.underfs.listing.length 1000
alluxio.underfs.object.store.mount.shared.publicly false
alluxio.underfs.oss.connection.max 1024
alluxio.underfs.oss.connection.timeout.ms 50000
alluxio.underfs.oss.connection.ttl -1
alluxio.underfs.oss.socket.timeout.ms 50000
alluxio.underfs.s3.admin.threads.max 20
alluxio.underfs.s3.disable.dns.buckets false
alluxio.underfs.s3.owner.id.to.username.mapping
alluxio.underfs.s3.proxy.https.only true
alluxio.underfs.s3.threads.max 22
alluxio.underfs.s3.upload.threads.max 2
alluxio.underfs.s3a.consistency.timeout.ms 60000
alluxio.underfs.s3a.inherit_acl true
alluxio.underfs.s3a.secure.http.enabled false
alluxio.underfs.s3a.server.side.encryption.enabled false
alluxio.underfs.s3a.socket.timeout.ms 50000
alluxio.user.block.master.client.threads 512
alluxio.user.block.remote.read.buffer.size.bytes 8MB
alluxio.user.block.remote.reader.class alluxio.client.netty.NettyRemoteBlockReader
alluxio.user.block.remote.writer.class alluxio.client.netty.NettyRemoteBlockWriter
alluxio.user.block.size.bytes.default 128MB
alluxio.user.block.worker.client.pool.gc.threshold.ms 300000
alluxio.user.block.worker.client.pool.size.max 128
alluxio.user.block.worker.client.threads 2048
alluxio.user.failed.space.request.limits 3
alluxio.user.file.buffer.bytes 3MB
alluxio.user.file.cache.partially.read.block true
alluxio.user.file.master.client.threads 64
alluxio.user.file.readtype.default CACHE_PROMOTE
alluxio.user.file.seek.buffer.size.bytes 1MB
alluxio.user.file.waitcompleted.poll.ms 1000
alluxio.user.file.worker.client.pool.gc.threshold.ms 300000
alluxio.user.file.worker.client.pool.size.max 128
alluxio.user.file.worker.client.threads 10
alluxio.user.file.write.location.policy.class alluxio.client.file.policy.LocalFirstPolicy
alluxio.user.file.write.tier.default HIGHEST
alluxio.user.file.writetype.default MUST_CACHE
alluxio.user.heartbeat.interval.ms 1000
alluxio.user.lineage.enabled false
alluxio.user.lineage.master.client.threads 10
alluxio.user.local.reader.packet.size.bytes 8MB
alluxio.user.local.writer.packet.size.bytes 64KB
alluxio.user.network.netty.channel EPOLL
alluxio.user.network.netty.channel.pool.disabled false
alluxio.user.network.netty.channel.pool.gc.threshold.ms 300000
alluxio.user.network.netty.channel.pool.size.max 1024
alluxio.user.network.netty.reader.buffer.size.packets 16
alluxio.user.network.netty.reader.cancel.enabled true
alluxio.user.network.netty.timeout.ms 60000
alluxio.user.network.netty.worker.threads 200
alluxio.user.network.netty.writer.buffer.size.packets 16
alluxio.user.network.netty.writer.packet.size.bytes 64KB
alluxio.user.packet.streaming.enabled false
alluxio.user.ufs.delegation.enabled true
alluxio.user.ufs.delegation.read.buffer.size.bytes 8MB
alluxio.user.ufs.delegation.write.buffer.size.bytes 2MB
alluxio.user.ufs.file.reader.class alluxio.client.netty.NettyUnderFileSystemFileReader
alluxio.user.ufs.file.writer.class alluxio.client.netty.NettyUnderFileSystemFileWriter
alluxio.version 1.4.0-RC1
alluxio.web.resources /software/servers/alluxio-1.4.0/core/server/src/main/webapp
alluxio.web.threads 1
alluxio.work.dir /software/servers/alluxio-1.4.0
alluxio.worker.allocator.class alluxio.worker.block.allocator.MaxFreeAllocator
alluxio.worker.bind.host 0.0.0.0
alluxio.worker.block.heartbeat.interval.ms 1000
alluxio.worker.block.heartbeat.timeout.ms 60000
alluxio.worker.block.threads.max 2048
alluxio.worker.block.threads.min 256
alluxio.worker.data.bind.host 0.0.0.0
alluxio.worker.data.folder /alluxioworker/
alluxio.worker.data.folder.tmp .tmp_blocks
alluxio.worker.data.port 29999
alluxio.worker.data.server.class alluxio.worker.netty.NettyDataServer
alluxio.worker.data.tmp.subdir.max 1024
alluxio.worker.evictor.class alluxio.worker.block.evictor.LRUEvictor
alluxio.worker.evictor.lrfu.attenuation.factor 2.0
alluxio.worker.evictor.lrfu.step.factor 0.25
alluxio.worker.file.buffer.size 1MB
alluxio.worker.file.persist.pool.size 64
alluxio.worker.file.persist.rate.limit 2GB
alluxio.worker.file.persist.rate.limit.enabled false
alluxio.worker.filesystem.heartbeat.interval.ms 1000
alluxio.worker.memory.size 60GB
alluxio.worker.network.netty.block.reader.threads.max 128
alluxio.worker.network.netty.block.writer.threads.max 128
alluxio.worker.network.netty.boss.threads 1
alluxio.worker.network.netty.channel EPOLL
alluxio.worker.network.netty.file.reader.threads.max 128
alluxio.worker.network.netty.file.transfer MAPPED
alluxio.worker.network.netty.file.writer.threads.max 128
alluxio.worker.network.netty.reader.buffer.size.packets 16
alluxio.worker.network.netty.reader.packet.size.bytes 64KB
alluxio.worker.network.netty.shutdown.quiet.period 2
alluxio.worker.network.netty.shutdown.timeout 15
alluxio.worker.network.netty.watermark.high 32KB
alluxio.worker.network.netty.watermark.low 8KB
alluxio.worker.network.netty.worker.threads 0
alluxio.worker.network.netty.writer.buffer.size.packets 16
alluxio.worker.port 29998
alluxio.worker.session.timeout.ms 600000
alluxio.worker.tieredstore.block.lock.readers 1000
alluxio.worker.tieredstore.block.locks 1000
alluxio.worker.tieredstore.level0.alias MEM
alluxio.worker.tieredstore.level0.dirs.path /mnt/ramdisk
alluxio.worker.tieredstore.level0.dirs.quota 60GB
alluxio.worker.tieredstore.level0.reserved.ratio 0.1
alluxio.worker.tieredstore.levels 1
alluxio.worker.tieredstore.reserver.enabled false
alluxio.worker.tieredstore.reserver.interval.ms 1000
alluxio.worker.tieredstore.retry 3
alluxio.worker.web.bind.host 0.0.0.0
alluxio.worker.web.port 30000
alluxio.zookeeper.election.path /election
alluxio.zookeeper.enabled false
alluxio.zookeeper.leader.inquiry.retry 10
alluxio.zookeeper.leader.path /leader

Activity

Show:
maobaolong
April 4, 2017, 9:02 AM

Thank you for continuously watching this issue.

  • The file '000274_0' on hdfs never change.

  • We use presto as Alluxio client to read ufs use CACHE readType and never use any Alluxio cli command before problem taken place.

So, we can think about others.

Yupeng Fu
April 4, 2017, 3:38 PM

thanks for the update. Can you attach the error logs from presto for step 2?

jira.alluxio
April 6, 2017, 11:34 AM

User 'maobaolong' has created a pull request for this issue:
https://github.com/Alluxio/alluxio/pull/5021

李阳
May 16, 2018, 12:55 PM

It's also in version 1.6.1.

李阳
May 28, 2018, 2:50 AM

It works after free the MEM space.

Assignee

Yupeng Fu

Reporter

maobaolong

Labels

Components

Affects versions

Priority

Critical
Configure