领英公司参考连接：https://www.slideshare.net/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600?qid=84026ff8-243f-49a7-a4d0-69976cf317b7&v=&b=&from_search=9(温馨提示，需要VPN哟～)。
一、kafka-producer-perf-test.sh使用案例
1、测试kafka写入的吞吐量
我用的是kafka版本是2.3.0，领英可能用的是0.10.0,我使用我的版本按照它的方式操作，发现不行，于是它标红线的参数我没有使用。
[root@Hexindai-C11-71 ~]# nohup kafka-producer-perf-test.sh --num-records 100000000 --record-size 1000 --topic kafka01 --throughput 1000000 --producer-props bootstrap.servers=172.20.11.71:9092,172.20.11.72:9092,172.20.11.73:9092 > /export/servers/kafka/logs/kafka01.log & 相关参数说明： —topic 指定Kafka集群的topic名称，本例为yinzhengjie-kafka --num-records 总共需要发送的消息数，本例为100000000 --record-size 每个记录的字节数，本例为1000 --throughput 每秒钟发送的记录数，本例为20000 --producer-props bootstrap.servers=172.20.11.71:9092,172.20.11.72:9092,172.20.11.73:9092 很显然是kafka集群的broker地址
2、计算生产者的吞吐量
[root@Hexindai-C11-71 ~]# tail -50f /export/servers/kafka/logs/kafka01.log 46368 records sent, 9273.6 records/sec (8.84 MB/sec), 3704.6 ms avg latency, 4446.0 ms max latency. 62016 records sent, 12403.2 records/sec (11.83 MB/sec), 3171.3 ms avg latency, 4187.0 ms max latency. 82704 records sent, 16159.4 records/sec (15.41 MB/sec), 2014.9 ms avg latency, 3001.0 ms max latency. 73904 records sent, 14780.8 records/sec (14.10 MB/sec), 2329.7 ms avg latency, 3223.0 ms max latency. 69424 records sent, 13884.8 records/sec (13.24 MB/sec), 2857.4 ms avg latency, 3674.0 ms max latency. 96688 records sent, 18565.3 records/sec (17.71 MB/sec), 1883.7 ms avg latency, 3101.0 ms max latency. 145184 records sent, 29036.8 records/sec (27.69 MB/sec), 1248.2 ms avg latency, 1856.0 ms max latency. 204400 records sent, 40880.0 records/sec (38.99 MB/sec), 825.9 ms avg latency, 1118.0 ms max latency. 214736 records sent, 42912.9 records/sec (40.92 MB/sec), 755.2 ms avg latency, 989.0 ms max latency. 218320 records sent, 43664.0 records/sec (41.64 MB/sec), 747.7 ms avg latency, 850.0 ms max latency. 154080 records sent, 30816.0 records/sec (29.39 MB/sec), 1003.2 ms avg latency, 1424.0 ms max latency. 142928 records sent, 28585.6 records/sec (27.26 MB/sec), 1161.5 ms avg latency, 1427.0 ms max latency. 141360 records sent, 28272.0 records/sec (26.96 MB/sec), 1213.5 ms avg latency, 1740.0 ms max latency. 194160 records sent, 38832.0 records/sec (37.03 MB/sec), 855.0 ms avg latency, 1178.0 ms max latency. 184848 records sent, 36969.6 records/sec (35.26 MB/sec), 884.6 ms avg latency, 1045.0 ms max latency. 188256 records sent, 37651.2 records/sec (35.91 MB/sec), 872.1 ms avg latency, 1019.0 ms max latency. 196848 records sent, 39369.6 records/sec (37.55 MB/sec), 832.6 ms avg latency, 953.0 ms max latency. 183824 records sent, 36757.4 records/sec (35.05 MB/sec), 902.8 ms avg latency, 1023.0 ms max latency. 139456 records sent, 27891.2 records/sec (26.60 MB/sec), 1105.3 ms avg latency, 1682.0 ms max latency. 73920 records sent, 14769.2 records/sec (14.09 MB/sec), 2203.0 ms avg latency, 3348.0 ms max latency. 154576 records sent, 30915.2 records/sec (29.48 MB/sec), 1200.9 ms avg latency, 3296.0 ms max latency. 189632 records sent, 37926.4 records/sec (36.17 MB/sec), 881.1 ms avg latency, 1192.0 ms max latency. 205680 records sent, 41136.0 records/sec (39.23 MB/sec), 846.0 ms avg latency, 1112.0 ms max latency. 213856 records sent, 42771.2 records/sec (40.79 MB/sec), 816.4 ms avg latency, 1162.0 ms max latency. 185888 records sent, 37170.2 records/sec (35.45 MB/sec), 894.1 ms avg latency, 1337.0 ms max latency. 211072 records sent, 42214.4 records/sec (40.26 MB/sec), 772.0 ms avg latency, 943.0 ms max latency. 205264 records sent, 41052.8 records/sec (39.15 MB/sec), 795.8 ms avg latency, 900.0 ms max latency. 219488 records sent, 43897.6 records/sec (41.86 MB/sec), 755.4 ms avg latency, 888.0 ms max latency. 218960 records sent, 43792.0 records/sec (41.76 MB/sec), 743.0 ms avg latency, 958.0 ms max latency. 197680 records sent, 39536.0 records/sec (37.70 MB/sec), 830.8 ms avg latency, 920.0 ms max latency. 210512 records sent, 42102.4 records/sec (40.15 MB/sec), 774.8 ms avg latency, 896.0 ms max latency. 211360 records sent, 42272.0 records/sec (40.31 MB/sec), 784.8 ms avg latency, 891.0 ms max latency. 209472 records sent, 41894.4 records/sec (39.95 MB/sec), 786.4 ms avg latency, 1010.0 ms max latency. 226192 records sent, 45238.4 records/sec (43.14 MB/sec), 722.6 ms avg latency, 861.0 ms max latency. 222896 records sent, 44579.2 records/sec (42.51 MB/sec), 738.3 ms avg latency, 830.0 ms max latency. 209600 records sent, 41920.0 records/sec (39.98 MB/sec), 781.4 ms avg latency, 870.0 ms max latency. 210800 records sent, 42160.0 records/sec (40.21 MB/sec), 770.1 ms avg latency, 861.0 ms max latency. 214832 records sent, 42966.4 records/sec (40.98 MB/sec), 760.5 ms avg latency, 872.0 ms max latency. 224544 records sent, 44774.5 records/sec (42.70 MB/sec), 744.2 ms avg latency, 933.0 ms max latency. 194496 records sent, 38899.2 records/sec (37.10 MB/sec), 838.0 ms avg latency, 1130.0 ms max latency. 218704 records sent, 43740.8 records/sec (41.71 MB/sec), 754.5 ms avg latency, 883.0 ms max latency. 195056 records sent, 39003.4 records/sec (37.20 MB/sec), 841.2 ms avg latency, 1160.0 ms max latency. 212320 records sent, 42430.1 records/sec (40.46 MB/sec), 762.6 ms avg latency, 871.0 ms max latency. 207264 records sent, 41452.8 records/sec (39.53 MB/sec), 802.2 ms avg latency, 938.0 ms max latency. 204480 records sent, 40887.8 records/sec (38.99 MB/sec), 801.7 ms avg latency, 947.0 ms max latency. 202432 records sent, 40486.4 records/sec (38.61 MB/sec), 810.4 ms avg latency, 934.0 ms max latency. . 207824 records sent, 41564.8 records/sec (39.64 MB/sec), 802.2 ms avg latency, 954.0 ms max latency. 202912 records sent, 40582.4 records/sec (38.70 MB/sec), 804.7 ms avg latency, 967.0 ms max latency. 202576 records sent, 40507.1 records/sec (38.63 MB/sec), 810.7 ms avg latency, 952.0 ms max latency. 219248 records sent, 39682.9 records/sec (37.84 MB/sec), 736.8 ms avg latency, 1400.0 ms max latency. ^C [root@Hexindai-C11-71 ~]#
207824 records sent, 41564.8 records/sec (39.64 MB/sec), 802.2 ms avg latency, 954.0 ms max latency. 当你执行完kafka-producer-perf-test.sh 脚本时，会输出类似上面的一行，从上面一行我们可以得出一些消息 1>.每秒平均向kafka写入了74.80 MB的数据(大概是：78433条消息，计算方式：74.8*1024*1024/1000); 2>.每次写入的平均延迟为390.67毫秒； 3>.最大延迟为6731.00毫秒； 4>.87毫秒内占95%； 5>.158毫秒内占99%； 6>.275毫秒内占99.9%；
二、kafka-consumer-perf-test.sh 使用案例
1、测试kafka消费的吞吐量
[root@Hexindai-C11-71 ~]# nohup kafka-consumer-perf-test.sh --broker-list 172.20.11.71:9092 --topic kafka01 --messages 1000000 --fetch-size 1048576 --threads 10 > /export/servers/kafka/logs/kafka01-consumer.log & [4] 29406 [root@Hexindai-C11-71 ~]# 相关参数说明： --topic 指定topic的名称，本例为 yinzhengjie-kafka --fetch-size 指定每次fetch的数据的大小，本例为1048576，也就是1M --messages 总共要消费的消息个数，本例为1000000，100w --topic 指定消费的topic --threads 指定消费的线程数为10
2、计算消费者的吞吐量
[root@Hexindai-C11-71 ~]# tail -f /export/servers/kafka/logs/kafka01-consumer.log start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec 2019-08-16 10:52:46:494, 2019-08-16 10:52:58:544, 954.0787, 79.1767, 1000424, 83022.7386, 456, 11594, 82.2907, 86288.0800 当你执行完 kafka-consumer-perf-test.sh 脚本时，会输出类似于上面的一行，从上面一行我们可以得出一些消息 1>.共消费了954.0787M的数据； 2>.消费速度为79.1767M/s； 3>.总共消费了1000424条消息； 4>.每秒消费83022.7386条消息；
三、编写脚本案例
1、执行脚本
[root@Hexindai-C11-71 ~]# cat kafka01.sh nohup kafka-consumer-perf-test.sh --broker-list 172.20.11.71:9092 --topic kafka01 --messages 1000000 --fetch-size 1048576 --threads 10 > /export/servers/kafka/logs/kafka01-consumer.log & nohup kafka-producer-perf-test.sh --num-records 100000000 --record-size 1000 --topic kafka01 --throughput 1000000 --producer-props bootstrap.servers=172.20.11.71:9092,172.20.11.72:9092,172.20.11.73:9092 > /export/servers/kafka/logs/kafka01.log & [root@Hexindai-C11-71 ~]#
2、运行脚本时查看kafka manager的监控
3、查看我的kafka性能测试结果
生产端： 总共需要发送的消息数：1亿条，每个记录的字节数是：1000字节，每秒钟发送的记录数是：1000000 消费端： 模拟了10个消费者：进行消费 指定每次fetch的数据的大小：1M 总共要消费的消息个数：1亿条 生产端结论： 1>.每秒平均向kafka写入了225.07 MB的数据； 2>.每次写入的平均延迟为 129.81毫秒； 3>.最大延迟为2422.00毫秒； 消费端结论： 1>. 共消费了95367.4316M的数据； 2>.消费速度为225.2047M/s； 3>.总共消费了1亿条消息； 4>.每秒消费236144.2369条消息；
https://www.cnblogs.com/yinzhengjie/p/9953212.html