记新 Kafka 集群压力测试过程

机器条件

一共有 5 台机器，每台机器的物理条件如下：

CPU信息如下，两颗

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                48
On-line CPU(s) list:   0-47
Thread(s) per core:    2
Core(s) per socket:    12
座：                 2
NUMA 节点：         2
厂商 ID：           GenuineIntel
CPU 系列：          6
型号：              63
型号名称：        Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz
步进：              2
CPU MHz：             1244.628
BogoMIPS：            4993.38
虚拟化：           VT-x
L1d 缓存：          32K
L1i 缓存：          32K
L2 缓存：           256K
L3 缓存：           30720K
NUMA 节点0 CPU：    0-11,24-35
NUMA 节点1 CPU：    12-23,36-47

内存条信息如下，两条

Memory Device
    Array Handle: 0x0020
    Error Information Handle: Not Provided
    Total Width: 72 bits
    Data Width: 64 bits
    Size: 32 GB
    Form Factor: DIMM
    Set: None
    Locator: DIMM_A1
    Bank Locator: NODE 1
    Type: DDR4
    Type Detail: Synchronous
    Speed: 2400 MHz
    Manufacturer: Samsung
    Serial Number: 388A5EDE
    Asset Tag:
    Part Number: M393A4K40CB1-CRC
    Rank: 2
    Configured Clock Speed: 2134 MHz

磁盘信息如下，由两块组成 RAID 1

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 850 EVO 1TB
Serial Number:    S2RGNX0K100551F
LU WWN Device Id: 5 002538 d4294cab3
Firmware Version: EMT02B6Q
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu May 24 18:03:34 2018 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

kafka 集群

kafka 集群用 5 台上面的物理条件一样的机器搭建而成。

创建 api-test topic、3个副本、15个分区，下面对这个 topic 进行压测

压力测试

1万条消息每条消息 2kb 大小

生产者压测

执行：./kafka-producer-perf-test.sh --topic api-test --num-records 10000 --record-size 2000 --throughput 10000 --producer-props bootstrap.servers=172.25.52.170:33092,172.25.52.171:33092,172.25.52.169:33092,172.25.52.168:33092,172.25.52.167:33092 batch.size=8196

10000 records sent, 9680.542110 records/sec (18.46 MB/sec), 27.31 ms avg latency, 247.00 ms max latency, 2 ms 50th, 92 ms 95th, 108 ms 99th, 112 ms 99.9th.

消费者压测

执行：./kafka-consumer-perf-test.sh --topic api-test --zookeeper 172.25.52.170:33181,172.25.52.171:33181,172.25.52.169:33181,172.25.52.168:33181,172.25.52.167:33181 --threads 1 --messages 10000

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2018-05-24 18:39:30:930, 2018-05-24 18:39:31:061, 0.9537, 7.2800, 10000, 76335.8779

1千万条消息每条消息 2kb 大小

生产者压测

执行：./kafka-producer-perf-test.sh --topic api-test --num-records 1000000 --record-size 100 --throughput 1000000 --producer-props bootstrap.servers=172.25.52.170:33092,172.25.52.171:33092,172.25.52.169:33092,172.25.52.168:33092,172.25.52.167:33092 batch.size=8196

561241 records sent, 112248.2 records/sec (214.10 MB/sec), 130.1 ms avg latency, 487.0 max latency.
556093 records sent, 111218.6 records/sec (212.13 MB/sec), 142.5 ms avg latency, 785.0 max latency.
567376 records sent, 113475.2 records/sec (216.44 MB/sec), 145.1 ms avg latency, 784.0 max latency.
571313 records sent, 114262.6 records/sec (217.94 MB/sec), 142.5 ms avg latency, 804.0 max latency.
570905 records sent, 114181.0 records/sec (217.78 MB/sec), 142.0 ms avg latency, 794.0 max latency.
556664 records sent, 111332.8 records/sec (212.35 MB/sec), 147.7 ms avg latency, 791.0 max latency.
568587 records sent, 113717.4 records/sec (216.90 MB/sec), 144.2 ms avg latency, 740.0 max latency.
591267 records sent, 118253.4 records/sec (225.55 MB/sec), 139.1 ms avg latency, 797.0 max latency.
592039 records sent, 118407.8 records/sec (225.84 MB/sec), 136.9 ms avg latency, 820.0 max latency.
615839 records sent, 123167.8 records/sec (234.92 MB/sec), 137.8 ms avg latency, 716.0 max latency.
649346 records sent, 129869.2 records/sec (247.71 MB/sec), 124.0 ms avg latency, 575.0 max latency.
638521 records sent, 127704.2 records/sec (243.58 MB/sec), 125.9 ms avg latency, 861.0 max latency.
599791 records sent, 119958.2 records/sec (228.80 MB/sec), 132.7 ms avg latency, 818.0 max latency.
405575 records sent, 75483.9 records/sec (143.97 MB/sec), 149.8 ms avg latency, 2766.0 max latency.
595724 records sent, 119144.8 records/sec (227.25 MB/sec), 189.6 ms avg latency, 2789.0 max latency.
751120 records sent, 150224.0 records/sec (286.53 MB/sec), 109.1 ms avg latency, 375.0 max latency.
10000000 records sent, 117752.343272 records/sec (224.59 MB/sec), 137.72 ms avg latency, 2789.00 ms max latency, 92 ms 50th, 743 ms 95th, 878 ms 99th, 2773 ms 99.9th.

消费者压测

执行： ./kafka-consumer-perf-test.sh --topic api-test --zookeeper 172.25.52.170:33181,172.25.52.171:33181,172.25.52.169:33181,172.25.52.168:33181,172.25.52.167:33181 --threads 1 --messages 10000000

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2018-05-24 17:40:51:586, 2018-05-24 17:41:27:446, 17261.5051, 481.3582, 10000000, 278862.2421

1亿条消息每条消息 2kb 大小

执行：./kafka-producer-perf-test.sh --topic api-test --num-records 100000000 --record-size 2000 --throughput 100000000 --producer-props bootstrap.servers=172.25.52.170:33092,172.25.52.171:33092,172.25.52.169:33092,172.25.52.168:33092,172.25.52.167:33092 batch.size=8196

642163 records sent, 128432.6 records/sec (244.97 MB/sec), 113.7 ms avg latency, 414.0 max latency.
729273 records sent, 145854.6 records/sec (278.20 MB/sec), 111.2 ms avg latency, 730.0 max latency.
717807 records sent, 143561.4 records/sec (273.82 MB/sec), 114.5 ms avg latency, 669.0 max latency.
667852 records sent, 133570.4 records/sec (254.77 MB/sec), 121.5 ms avg latency, 465.0 max latency.
561615 records sent, 112323.0 records/sec (214.24 MB/sec), 147.2 ms avg latency, 474.0 max latency.
783997 records sent, 156799.4 records/sec (299.07 MB/sec), 102.1 ms avg latency, 478.0 max latency.
730643 records sent, 146128.6 records/sec (278.72 MB/sec), 108.2 ms avg latency, 850.0 max latency.
624260 records sent, 124852.0 records/sec (238.14 MB/sec), 134.0 ms avg latency, 869.0 max latency.
518143 records sent, 103628.6 records/sec (197.66 MB/sec), 152.0 ms avg latency, 1248.0 max latency.
262529 records sent, 52505.8 records/sec (100.15 MB/sec), 326.7 ms avg latency, 3127.0 max latency.
715721 records sent, 143144.2 records/sec (273.03 MB/sec), 116.8 ms avg latency, 585.0 max latency.
709401 records sent, 141880.2 records/sec (270.62 MB/sec), 113.5 ms avg latency, 473.0 max latency.
677999 records sent, 135599.8 records/sec (258.64 MB/sec), 120.1 ms avg latency, 564.0 max latency.
737457 records sent, 147491.4 records/sec (281.32 MB/sec), 108.1 ms avg latency, 680.0 max latency.
704827 records sent, 140965.4 records/sec (268.87 MB/sec), 115.4 ms avg latency, 661.0 max latency.
676627 records sent, 135325.4 records/sec (258.11 MB/sec), 120.0 ms avg latency, 1023.0 max latency.
590074 records sent, 118014.8 records/sec (225.10 MB/sec), 137.6 ms avg latency, 1054.0 max latency.
659174 records sent, 131834.8 records/sec (251.45 MB/sec), 126.4 ms avg latency, 962.0 max latency.
523764 records sent, 104752.8 records/sec (199.80 MB/sec), 157.3 ms avg latency, 1769.0 max latency.
739164 records sent, 147832.8 records/sec (281.97 MB/sec), 112.6 ms avg latency, 612.0 max latency.
738231 records sent, 147646.2 records/sec (281.61 MB/sec), 107.2 ms avg latency, 721.0 max latency.
369107 records sent, 73821.4 records/sec (140.80 MB/sec), 222.5 ms avg latency, 2160.0 max latency.
···
···
···
850695 records sent, 170139.0 records/sec (324.51 MB/sec), 94.8 ms avg latency, 497.0 max latency.
862939 records sent, 172587.8 records/sec (329.19 MB/sec), 94.8 ms avg latency, 664.0 max latency.
684993 records sent, 136998.6 records/sec (261.30 MB/sec), 118.5 ms avg latency, 846.0 max latency.
759886 records sent, 151977.2 records/sec (289.87 MB/sec), 108.9 ms avg latency, 507.0 max latency.
759156 records sent, 151831.2 records/sec (289.60 MB/sec), 108.2 ms avg latency, 528.0 max latency.
726247 records sent, 145249.4 records/sec (277.04 MB/sec), 112.8 ms avg latency, 509.0 max latency.
646276 records sent, 106488.1 records/sec (203.11 MB/sec), 113.3 ms avg latency, 1989.0 max latency.
555445 records sent, 111089.0 records/sec (211.89 MB/sec), 192.3 ms avg latency, 2665.0 max latency.
931803 records sent, 186360.6 records/sec (355.45 MB/sec), 85.8 ms avg latency, 497.0 max latency.
750725 records sent, 150145.0 records/sec (286.38 MB/sec), 107.9 ms avg latency, 821.0 max latency.
808483 records sent, 161696.6 records/sec (308.41 MB/sec), 104.4 ms avg latency, 646.0 max latency.
712690 records sent, 142538.0 records/sec (271.87 MB/sec), 109.7 ms avg latency, 746.0 max latency.
629610 records sent, 125922.0 records/sec (240.18 MB/sec), 131.2 ms avg latency, 784.0 max latency.
674141 records sent, 134828.2 records/sec (257.16 MB/sec), 124.9 ms avg latency, 708.0 max latency.
751786 records sent, 150357.2 records/sec (286.78 MB/sec), 109.1 ms avg latency, 429.0 max latency.
768948 records sent, 153789.6 records/sec (293.33 MB/sec), 106.1 ms avg latency, 363.0 max latency.
100000000 records sent, 140619.880557 records/sec (268.21 MB/sec), 116.07 ms avg latency, 3480.00 ms max latency, 2 ms 50th, 210 ms 95th, 451 ms 99th, 1661 ms 99.9th.

消费者压测

执行：./kafka-consumer-perf-test.sh --topic api-test --zookeeper 172.25.52.170:33181,172.25.52.171:33181,172.25.52.169:33181,172.25.52.168:33181,172.25.52.167:33181 --threads 1 --messages 100000000

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2018-05-24 17:44:44:025, 2018-05-24 17:48:14:125, 75859.4952, 361.0638, 40722223, 193823.0509

压测总结

生产者

消息量	生产速率	平均延时	最大时延	50%	95%	99%	99.9%
10000	18.46 MB/sec	27.31 ms	247.00 ms	2ms	92 ms	108 ms	112 ms
10000000	224.59 MB/sec	137.72 ms	2789.00 ms	92 ms	743 ms	878 ms	2773 ms
100000000	268.21 MB/sec	116.07 ms	3480.00 ms	2 ms	210 ms	451 ms	1661 ms

消费者

开始时间	结束时间	总容量(MB)	容量速率(MB/s)	消息总数(nMsg)	消息速率(nMsg/s)
2018-05-24 18:39:30:930	2018-05-24 18:39:31:061	0.9537	7.2800	10000	76335.8779
2018-05-24 17:40:51:586	2018-05-24 17:41:27:446	17261.5051	481.3582	10000000	278862.2421
2018-05-24 18:27:52:075	2018-05-24 18:35:19:864	188922.8821	421.9016	100000000	223319.4652

得出下面结论：

我们在 3 个副本和 15 个分区的情况下：每个分区的吞量平均在 15MB/s ，吐量平均在 30MB/s；
生产量越大，时延越大，时延高的另外一个原因是副本同步导致的；
消费不是瓶颈，生产存在瓶颈，生产瓶颈来自磁盘写入速度；

这个测试是不够充分的，但也够用，目的是大概知道这个 kafka 集群的天花板在哪里，知道集群的承受力，从而方便日后的扩容。

本文由 Chakhsu Lau 创作，采用知识共享署名4.0 国际许可协议进行许可。
本站文章除注明转载/出处外，均为本站原创或翻译，转载前请务必署名。