BLOG main image
분류 전체보기 (111)
주절주절 (16)
Tabex (4)
클라우드 관련 (12)
Linux (23)
CruiseControl (3)
XP (1)
C (4)
C++ (1)
C# (2)
Boost (3)
Java (0)
DataBase (4)
Algorithm (17)
Network (1)
OS (1)
UML (1)
상식 (3)
좋은 글 (3)
회사에서... (0)
Node.js (1)
EKS (1)
Mac (2)
Visitors up to today!
Today hit, Yesterday hit
daisy rss
tistory 티스토리 가입하기!
'Flume'에 해당되는 글 1건
2013. 4. 11. 21:11

FlumeUserGuide

http://archive.cloudera.com/cdh4/cdh/4/flume-ng/FlumeUserGuide.html


1. HDFS로 로그를 write할 Agent

vim /etc/flume-ng/conf/flume.conf

agentM.sources = avro-collection-source

agentM.channels = mem-channel

agentM.sinks = hdfs-sink


# For each one of the sources, the type is defined

agentM.sources.avro-collection-source.channels = mem-channel

agentM.sinks.hdfs-sink.channel = mem-channel

agentM.sinks.hdfs-sink.type = hdfs

agentM.sinks.hdfs-sink.hdfs.path = hdfs://sm06/flumetest/events/

agentM.sinks.hdfs-sink.hdfs.filePrefix = events-

agentM.sinks.hdfs-sink.hdfs.round = true

agentM.sinks.hdfs-sink.hdfs.roundValue = 10

agentM.sinks.hdfs-sink.hdfs.roundUnit = 1


# Each sink's type must be defined

agentM.sources.avro-collection-source.type = avro

agentM.sources.avro-collection-source.bind = 0.0.0.0

agentM.sources.avro-collection-source.port = 41414


#Specify the channel the sink should use


# Each channel's type is defined.

agentM.channels.mem-channel.type = memory


# Other config values specific to each type of channel(sink or source)

# can be defined as well

# In this case, it specifies the capacity of the memory channel

agentM.channels.mem-channel.capacity = 100

sudo -u hdfs flume-ng agent --conf /etc/flume-ng/conf --conf-file /etc/flume-ng/conf/flume.conf --name agentM -Dflume.root.logger=INFO,console


2. HDFS Agent로 로그를 보내줄 Agent

agent.sources = avro-AppSrv-source

agent.channels = file-channel

agent.sinks = avro-forward-sink


agent.sources.avro-AppSrv-source.type = exec

agent.sources.avro-AppSrv-source.command = tail -F /var/log/httpd/access_log

agent.sources.avro-AppSrv-source.channels = file-channel

agent.sinks.avro-forward-sink.channel = file-channel


agent.sinks.avro-forward-sink.type = avro
agent.sinks.avro-forward-sink.hostname = 172.21.81.106
agent.sinks.avro-forward-sink.port = 41414

agent.channels.file-channel.type = memory
agent.channels.file-channel.capacity = 100

sudo -u hdfs flume-ng agent --conf /etc/flume-ng/conf --conf-file /etc/flume-ng/conf/flume.conf --name agent -Dflume.root.logger=INFO,console


prev"" #1 next