flume 配置实例.md

flume 配置实例.md

从文件到kafka

# flume-directory-kafka.conf: A single-node Flume configuration# Name the components on this agenta3.sources = r3a3.sinks = k3a3.channels = c3# Describe/configure the sourcea3.sources.r3.type = execa3.sources.r3.channels = c3a3.sources.r3.command = tail -F /usr/local/nginx/logs/access.log# Describe the sinka3.sinks.k3.type = org.apache.flume.sink.kafka.KafkaSinka3.sinks.k3.topic = countly_small_sysa3.sinks.k3.brokerList = node6.hadoop.fzzqxf.com:9092,node5.hadoop.fzzqxf.com:9092,node4.hadoop.fzzqxf.com:9092,node3.hadoop.fzzqxf.com:9092,node2.hadoop.fzzqxf.com:9092a3.sinks.k3.requiredAcks = 1a3.sinks.k3.batchSize = 20a3.sinks.k3.channel = c3# Use a channel which buffers events in memorya3.channels.c3.type = memorya3.channels.c3.capacity = 100000a3.channels.c3.transactionCapacity = 1000# Bind the source and sink to the channela3.sources.r3.channels = c3a3.sinks.k3.channel = c3

从kafka 到hdfs

# list the sources, sinks and channels for the agentagent_collector.sources = kafka-src-1agent_collector.sinks = hdfs-sink-1agent_collector.channels = mem-channel-1# set channel for sourceagent_collector.sources.kafka-src-1.channels = mem-channel-1# set channel for sinkagent_collector.sinks.hdfs-sink-1.channel = mem-channel-1# properties for sourcesagent_collector.sources.kafka-src-1.type = org.apache.flume.source.kafka.KafkaSourceagent_collector.sources.kafka-src-1.channels = mem-channel-1agent_collector.sources.kafka-src-1.zookeeperConnect = master.hadoop.fzzqxf.com:2181,node1.hadoop.fzzqxf.com:2181,node4.hadoop.fzzqxf.com:2181,node5.hadoop.fzzqxf.com:2181,node6.hadoop.fzzqxf.com:2181agent_collector.sources.kafka-src-1.topic = siege-countly_log-2agent_collector.sources.kafka-src-1.groupId = flumeCountlyHdfsagent_collector.sources.kafka-src-1.batchSize=200agent_collector.sources.kafka-src-1.kafka.consumer.timeout.ms = 100# properties for channelsagent_collector.channels.mem-channel-1.type = memoryagent_collector.channels.mem-channel-1.capacity = 10000agent_collector.channels.mem-channel-1.transactionCapacity = 10000agent_collector.channels.mem-channel-1.byteCapacityBufferPercentage = 20agent_collector.channels.mem-channel-1.byteCapacity = 10485760# properties for sinksagent_collector.channels = mem-channel-1agent_collector.sinks = hdfs-sink-1agent_collector.sinks.hdfs-sink-1.type = hdfsagent_collector.sinks.hdfs-sink-1.channel = mem-channel-1agent_collector.sinks.hdfs-sink-1.hdfs.path = /data/logs/flume/realtime/countly/%Y%m%d agent_collector.sinks.hdfs-sink-1.hdfs.filePrefix = collector_1agent_collector.sinks.hdfs-sink-1.hdfs.fileType=DataStreamagent_collector.sinks.hdfs-sink-1.hdfs.minBlockReplicas=1agent_collector.sinks.hdfs-sink-1.hdfs.rollInterval=21600# unit is bytes. value is 1G*100=13421772800.agent_collector.sinks.hdfs-sink-1.hdfs.rollSize=0agent_collector.sinks.hdfs-sink-1.hdfs.rollCount=0agent_collector.sinks.hdfs-sink-1.hdfs.batchSize=1000agent_collector.sinks.hdfs-sink-1.hdfs.callTimeout=60000

作者：少帅

您的支持是对博主最大的鼓励，感谢您的认真阅读。

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。