Kafka 集群配置过程记录
in Note with 0 comment
Kafka 集群 记录
Kafka 集群配置过程记录
in Note with 0 comment

Kafka 集群 记录

硬件资源

  1. 使用三台机器布置3个结点.
  2. 每台机器/mnt/datas目录挂载了一个专用的SSD磁盘，用来存放数据与log.

添加用户和用户组

sudo groupadd kafka
sudo useradd -g kafka -M kafka

下载kafka

wget http://mirror.bit.edu.cn/apache/kafka/1.1.0/kafka_2.11-1.1.0.tgz
tar -xvzf kafka_2.11-1.1.0.tgz
sudo mv kafka_2.11-1.1.0 /usr/local/

配置zookeeper

编辑/usr/local/kafka_2.11-1.1.0/config/zookeeper.properties, 配置以下信息

initLimit=10
tickTime=2000
syncLimit=15
clientPort=2181
maxClientCnxns=0

dataDir=/mnt/datas/zookeeper_data/data
dataLogDir=/mnt/datas/zookeeper_data/data_log

server.1=kafka-zookeeper-1.prodenv.com:2888:3888
server.2=kafka-zookeeper-2.prodenv.com:2888:3888
server.3=kafka-zookeeper-3.prodenv.com:2888:3888

编辑/etc/hosts文件，加入以下配置

192.168.32.59    kafka-zookeeper-1.prodenv.com
192.168.32.88    kafka-zookeeper-2.prodenv.com
192.168.32.97    kafka-zookeeper-3.prodenv.com

创建zookeeper的dataDir

sudo mkdir /mnt/datas/zookeeper_data
sudo chown kafka:kafka -R /mnt/datas/zookeeper_data
sudo -u kafka -H mkdir /mnt/datas/zookeeper_data/data
sudo -u kafka -H mkdir /mnt/datas/zookeeper_data/data_log
sudo -u kafka -H mkdir /mnt/datas/zookeeper_data/logs

创建myid文件

# 每台机器需要配置不相同的id
sudo -u kafka -H bash -c 'echo 1 > /mnt/datas/zookeeper_data/data/myid'

新建zookeeper systemd service启动项 /etc/systemd/system/kafka-zookeeper.service

[Unit]
Description=Apache Zookeeper server (Kafka)
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target
ConditionPathExists=/usr/local/kafka_2.11-1.1.0/config/zookeeper.properties

[Service]
Type=simple
User=kafka
Group=kafka
Environment=JAVA_HOME=/usr/local/jdk1.8
Environment=LOG_DIR=/mnt/datas/zookeeper_data/logs
ExecStart=/usr/local/kafka_2.11-1.1.0/bin/zookeeper-server-start.sh /usr/local/kafka_2.11-1.1.0/config/zookeeper.properties
ExecStop=/usr/local/kafka_2.11-1.1.0/bin/zookeeper-server-stop.sh

[Install]
WantedBy=multi-user.target

配置kafka

编辑/usr/local/kafka_2.11-1.1.0/config/server.properties，配置以下信息

broker.id=1    # 每个结点id不能重复
listeners=PLAINTEXT://192.168.32.59:9092    # 每台机器配置相应的IP
log.dirs=/mnt/datas/kafka
zookeeper.connect=kafka-zookeeper-1.prodenv.com:2181,kafka-zookeeper-2.prodenv.com:2181,kafka-zookeeper-3.prodenv.com:2181

创建log.dirs

sudo mkdir /mnt/datas/kafka
sudo chown kafka:kafka -R /mnt/datas/kafka

新建kafka systemd service启动项 /etc/systemd/system/kafka.service

[Unit]
Description=Apache Kafka server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target remote-fs.target
After=network.target remote-fs.target kafka-zookeeper.service
ConditionPathExists=/usr/local/kafka_2.11-1.1.0/config/server.properties

[Service]
Type=simple
User=kafka
Group=kafka
Environment=JAVA_HOME=/usr/local/jdk1.8
Environment=LOG_DIR=/mnt/datas/kafka
Environment=JMX_PORT=9998
Environment="KAFKA_HEAP_OPTS=-Xmx5G -Xms5G"
ExecStart=/usr/local/kafka_2.11-1.1.0/bin/kafka-server-start.sh /usr/local/kafka_2.11-1.1.0/config/server.properties
ExecStop=/usr/local/kafka_2.11-1.1.0/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

启动服务

# 刷新systemd
sudo systemctl daemon-reload

# 配置开机自启动
sudo systemctl enable kafka-zookeeper.service
sudo systemctl enable kafka.service

# 启动zookeeper
sudo systemctl start kafka-zookeeper.service

# 启动kafka
sudo systemctl start kafka.service

每个结点都完成以上操作之后，可使用以下命令进入zookeeper shell:

/usr/local/kafka_2.11-1.1.0/bin/zookeeper-shell.sh kafka-zookeeper-1.prodenv.com:2181,kafka-zookeeper-2.prodenv.com:2181,kafka-zookeeper-3.prodenv.com:2181

结束~ 👊

本文由 Chakhsu Lau 创作，采用 知识共享署名4.0 国际许可协议进行许可。
本站文章除注明转载/出处外，均为本站原创或翻译，转载前请务必署名。

← → ↓ ↑
Responses