FLUME安装过程

环境：

1、操作系统：CentOS-6.5-x86_64

2、JDK：jdk-8u40-linux-x64

3、HADOOP：hadoop-2.6.0

4、HBASE：hbase-0.98.11-hadoop2

5、MySQL-server-5.5.28-1.linux2.6.x86_64

6、HIVE1.1.0

前提：安装好HADOOP2.6.0

***********************************************************************************************************

1、下载apache-flume-1.5.2-bin.tar.gz

2、在nn1主机hadoop用户下解压，既完成安装。

***********************************************************************************************************

测试：

1、创建agent的配置文件

[hadoop2@nn1 shstest]$ cat agent_syslogudp.conf

agent1.sources = syslogudp1

agent1.sinks = hdfs1

agent1.channels = filechannel1

agent1.sources.syslogudp1.type = syslogudp

agent1.sources.syslogudp1.host = nn1

agent1.sources.syslogudp1.port = 5140

agent1.sources.syslogudp1.keepFields = true

agent1.sources.syslogudp1.channels = filechannel1

agent1.sinks.hdfs1.type = hdfs

agent1.sinks.hdfs1.hdfs.path = hdfs://nn1:19600/flume/shs

agent1.sinks.hdfs1.hdfs.fileType = DataStream

agent1.sinks.hdfs1.hdfs.writeFormat = TEXT

agent1.sinks.hdfs1.hdfs.rollInterval = 4

agent1.sinks.hdfs1.channel = filechannel1

agent1.channels.filechannel1.type = file

agent1.channels.filechannel1.checkpointDir = /data/hadoop2/flume152/shstest/checkpointdir

agent1.channels.filechannel1.dataDirs = /data/hadoop2/flume152/shstest/datadir

2、执行flume-ng，启动agent

flume-ng agent -f agent_syslogudp.conf -n agent1 -Dflume.root.logger=INFO,console

3、在另一台主机上（dn1），配置syslog(/etc/syslog.conf，centos6以后是：/etc/rsyslog.conf），添加如下内容

*.*   @nn1:5410               #后面5410与上面conf文件中配置的端口一致

4、重启dn1上syslog服务

/etc/init.d/syslog restart

（centos6以后为 /etc/init.d/rsyslog restart）

5、在dn1上手工添加syslog记录，测试

logger -p user.notice "syslog test from shs"

6、查看hdfs上，对syslog记录的结果

[hadoop2@nn1 ~]$ hdfs dfs -cat /flume/shs/*

15/05/18 10:14:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

<86>May 18 10:14:56 dn1 sshd[46777]: Accepted password for hadoop2 from 172.18.0.11 port 43446 ssh2

<86>May 18 10:14:56 dn1 sshd[46777]: pam_unix(sshd:session): session opened for user hadoop2 by (uid=0)

<13>May 18 10:15:07 dn1 hadoop2: syslogtest from shs