Flume

Flume Architecture

- Flume Download

[hadoop@hadoop01 ~]$ tar -zxf apache-flume-1.5.0-bin.tar.gz 

[hadoop@hadoop01 ~]$ ln -s apache-flume-1.5.0-bin apache-flume

- Flume Configuration 

$ vi apache-flume/conf/flume-env.sh

JAVA_HOME=/usr/java/java FLUME_CLASSPATH=/home/hadoop/apache-flume/lib export PATH=$PATH:/home/hadoop/hadoop/bin

$ cd apache-flume/conf/

$ vi flume_hdfs.txt

# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 ### source a1.sources.r1.type = exec a1.sources.r1.command = tail -F /home/hadoop/syslog/a.txt a1.sources.r1.channels = c1 ### sink -> hdfs a1.sinks.k1.type = hdfs a1.sinks.k1.channel = c1 a1.sinks.k1.hdfs.path = hdfs://hatest/user/hadoop/syslog/ a1.sinks.k1.hdfs.round = true a1.sinks.k1.hdfs.roundValue = 10 a1.sinks.k1.hdfs.roundUnit = minute ### Channel  a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 ### Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1

$ vi flume_avro.txt

# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 ### source a1.sources.r1.type = exec a1.sources.r1.command = tail -F /home/hadoop/syslog/a.txt a1.sources.r1.channels = c1 ### sink configuration a1.channels = c1 a1.sinks = k1 a1.sinks.k1.type = avro a1.sinks.k1.channel = c1 a1.sinks.k1.hostname = 192.168.56.101 a1.sinks.k1.port = 4545 ### channel configuration a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 ### Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1

- Test용 파일 생성

[hadoop@hadoop01 apache-flume]$ mkdir /home/hadoop/syslog

[hadoop@hadoop01 apache-flume]$ touch /home/hadoop/syslog/a.txt

[hadoop@hadoop01 apache-flume]$ hadoop fs -mkdir -p /user/hadoop/syslog

[hadoop@hadoop01 apache-flume]$ hadoop fs -ls

Found 1 items

drwxr-xr-x   - hadoop supergroup          0 2017-08-22 11:11 syslog

- Flume 데몬 기동 

$ bin/flume-ng agent --conf ./conf/ -f conf/flume_hdfs.txt -Dflume.root.logger=DEBUG,console  -n 

- 다른 창에서 로그 생성 

[hadoop@hadoop01 ~]$ echo "TEST1" >> /home/hadoop/syslog/a.txt

[hadoop@hadoop01 ~]$ echo "TEST2" >> /home/hadoop/syslog/a.txt

[hadoop@hadoop01 ~]$ echo "TEST3" >> /home/hadoop/syslog/a.txt

[hadoop@hadoop01 ~]$ echo "TEST4" >> /home/hadoop/syslog/a.txt

[hadoop@hadoop01 ~]$ echo "TEST5" >> /home/hadoop/syslog/a.txt

- hadoop data 확인 

[hadoop@hadoop01 name]$ hadoop fs -cat syslog/events-.1503541724091

?TEST3
^Vache.TEST4
^_ TEST5[hadoop@hadoop01 name]$ .io.BytesWritable�G=a[��&`#[��
^ �TEST1
^ �TEST2
^