2016-10-19-kafka-0.10.0

下载

1	wget https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz

quitstart

https://kafka.apache.org/documentation#quickstart

依赖

java

zookeeper

配置

1 2	cp server.properties server-0.properties cp server.properties server-1.properties

vi server-0.properties

##################### Server Basics ######################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

################### Socket Server Settings #####################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = security_protocol://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092



############################# Log Basics #############################

# A comma seperated list of directories under which to store log files
log.dirs=/usr/local/kafka/logs-0

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=2



############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=192.168.100.31:2181

vi server-1.properties

##################### Server Basics ######################
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

################### Socket Server Settings #####################
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = security_protocol://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9093



############################# Log Basics #############################

# A comma seperated list of directories under which to store log files
log.dirs=/usr/local/kafka/logs-1

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=2



############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=192.168.100.31:2181

启动

1 2	bin/kafka-server-start.sh config/server-0.properties >/dev/null 2>&1 & bin/kafka-server-start.sh config/server-1.properties >/dev/null 2>&1 &

日志信息

1	tail -f logs/server.log

kafka manager

https://www.cnblogs.com/yinchengzhe/p/5126360.html

1	nohup ./kafka-manager -java-home /path/to/jdk8 -Dconfig.file=../conf/application.conf >/dev/null 2>&1 &

1	bin/kafka-manager -java-home /root/jdk1.8.0_102/ -Dconfig.file=conf/application.conf >/dev/null 2>&1 &

producer config

https://kafka.apache.org/documentation#producerconfigs

demo

https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example

consumer config

https://kafka.apache.org/documentation#consumerconfigs

demo

https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example

https://zqhxuyuan.github.io/2016/02/20/Kafka-Consumer-New/

consumer offset storage

https://kafka.apache.org/documentation#impl_offsettracking

zookeeper structures

https://kafka.apache.org/documentation#impl_zookeeper

hostname

1
2
3

vi /etc/hosts
192.168.100.34 kafka34
192.168.100.35 kafka35

加入 host 不然连接不上 kafka