Docker搭建帶SASL用戶密碼驗證的Kafka

最近需要用到帶有鑒權的kafka，網上基本都是使用confluentinc的kafka，試了下有各種問題，因此使用 wurstmeister/zookeeper 和 wurstmeister/kafka 搭建了一個帶有密碼驗證的kafka，簡單記錄下搭建的過程

1 配置ZOOKEEPER

1.1 新建放置配置文件的目錄

/home/tool/kafka-sasl/conf

1.2 在文件夾內創建一個新的zookeeper配置文件zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/opt/zookeeper-3.4.13/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider

requireClientAuthScheme=sasl

jaasLoginRenew=3600000

zookeeper.sasl.client=true

1.3 新建密碼驗證的配置文件 server_jaas.conf

Client {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    username="admin"
    password="12345678";
};


Server {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    username="admin"
    password="12345678"
    user_super="12345678"
    user_admin="12345678";
};

KafkaServer {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="12345678"
    user_admin="12345678";
};

KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="12345678";
};

1.4 新建配置文件 log4j.properties和configuration.xsl（也可以先不加參數直接啟動zookeeper后，從容器/opt/zookeeper-3.4.13/conf/中復制過來）

log4j.properties

# Define some default values that can be overridden by system properties
zookeeper.root.logger=INFO, CONSOLE
zookeeper.console.threshold=INFO
zookeeper.log.dir=.
zookeeper.log.file=zookeeper.log
zookeeper.log.threshold=DEBUG
zookeeper.tracelog.dir=.
zookeeper.tracelog.file=zookeeper_trace.log

#
# ZooKeeper Logging Configuration
#

# Format is "<default threshold> (, <appender>)+

# DEFAULT: console appender only
log4j.rootLogger=${zookeeper.root.logger}

# Example with rolling log file
#log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE

# Example with rolling log file and tracing
#log4j.rootLogger=TRACE, CONSOLE, ROLLINGFILE, TRACEFILE

#
# Log INFO level and above messages to the console
#
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Threshold=${zookeeper.console.threshold}
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n

#
# Add ROLLINGFILE to rootLogger to get log file output
#    Log DEBUG level and above messages to a log file
log4j.appender.ROLLINGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.ROLLINGFILE.Threshold=${zookeeper.log.threshold}
log4j.appender.ROLLINGFILE.File=${zookeeper.log.dir}/${zookeeper.log.file}

# Max log file size of 10MB
log4j.appender.ROLLINGFILE.MaxFileSize=10MB
# uncomment the next line to limit number of backup files
log4j.appender.ROLLINGFILE.MaxBackupIndex=10

log4j.appender.ROLLINGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.ROLLINGFILE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n


#
# Add TRACEFILE to rootLogger to get log file output
#    Log DEBUG level and above messages to a log file
log4j.appender.TRACEFILE=org.apache.log4j.FileAppender
log4j.appender.TRACEFILE.Threshold=TRACE
log4j.appender.TRACEFILE.File=${zookeeper.tracelog.dir}/${zookeeper.tracelog.file}

log4j.appender.TRACEFILE.layout=org.apache.log4j.PatternLayout
### Notice we are including log4j's NDC here (%x)
log4j.appender.TRACEFILE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L][%x] - %m%n

configuration.xsl

<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="html"/>
<xsl:template match="configuration">
<html>
<body>
<table border="1">
<tr>
 <td>name</td>
 <td>value</td>
 <td>description</td>
</tr>
<xsl:for-each select="property">
<tr>
  <td><a name="{name}"><xsl:value-of select="name"/></a></td>
  <td><xsl:value-of select="value"/></td>
  <td><xsl:value-of select="description"/></td>
</tr>
</xsl:for-each>
</table>
</body>
</html>
</xsl:template>
</xsl:stylesheet>

1.4 啟動命令

docker run --name zookeeper_sasl -p 2181:2181 -e SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper-3.4.13/secrets/server_jaas.conf" -v /home/tool/kafka-sasl/conf:/opt/zookeeper-3.4.13/conf -v /home/tool/kafka-sasl/conf:/opt/zookeeper-3.4.13/secrets/ --rm -it wurstmeister/zookeeper

2 配置 KAFKA

使用上面的server_jass.conf 配置文件作為密碼驗證文件

2.1 啟動命令

docker run  --name kafka_sasl -p 59092:9092 --link zookeeper_sasl:zookeeper_sasl -e KAFKA_BROKER_ID=0 -e KAFKA_ADVERTISED_LISTENERS=SASL_PLAINTEXT://10.18.104.202:59092 -e KAFKA_ADVERTISED_PORT=59092 -e KAFKA_LISTENERS=SASL_PLAINTEXT://0.0.0.0:9092 -e KAFKA_SECURITY_INTER_BROKER_PROTOCOL=SASL_PLAINTEXT -e KAFKA_PORT=59092 -e KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL=PLAIN -e KAFKA_SASL_ENABLED_MECHANISMS=PLAIN -e KAFKA_AUTHORIZER_CLASS_NAME=kafka.security.auth.SimpleAclAuthorizer -e KAFKA_SUPER_USERS=User:admin -e KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND=false -e KAFKA_ZOOKEEPER_CONNECT='zookeeper_sasl:2181' -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 -e KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0 -e KAFKA_OPTS="-Djava.security.auth.login.config=/opt/kafka/secrets/server_jaas.conf" -v /home/tool/kafka-sasl/conf/:/opt/kafka/secrets/ --rm -it  wurstmeister/kafka:2.11-0.11.0.3 

換端口再啟動兩個，可以建立一個集群

3 驗證

3.1 創建topic

kafka-topics.sh --zookeeper 10.18.104.202:2181 --create --partitions 2 --replication-factor 1 --topic  testTopic

kafka-topics.sh --zookeeper  10.18.104.202:2181  --list 

kafka-topics.sh --zookeeper 10.18.104.202:2181  --describe --topic  testTopic

3.2 產出一條消息

執行腳本前，修改/opt/kafka/bin/kafka-console-producer.sh：

export KAFKA_HEAP_OPTS="-Xmx512M -Djava.security.auth.login.config=/opt/kafka/secrets/server_jaas.conf"

修改/opt/kafka/config/producer.properties

metadata.broker.list=kafka外網IP:外網端口

添加：

security.protocol=SASL_PLAINTEXT

sasl.mechanism=PLAIN

 

執行腳本：

./bin/kafka-console-producer.sh --broker-list 10.18.104.202:59092  --topic  testTopic  --producer.config config/producer.properties

3.3 消費一條消息

執行腳本前，修改/opt/kafka/bin/kafka-console-consumer.sh：

export KAFKA_HEAP_OPTS="-Xmx512M -Djava.security.auth.login.config=/opt/kafka/secrets/server_jaas.conf"

修改/opt/kafka/config/consumer.properties

zookeeper.connect=zookeeper外網ip:外面端口

添加：

security.protocol=SASL_PLAINTEXT

sasl.mechanism=PLAIN

 

執行腳本：

./bin/kafka-console-consumer.sh --bootstrap-server 10.18.104.202:59092 --topic  testTopic  --from-beginning --consumer.config config/consumer.properties