Linux 服務器
CentOS 為例。
Kafka 安裝
獲取下載地址，下載后解壓。
$ wget https://mirror.bit.edu.cn/apache/kafka/2.5.0/kafka_2.12-2.5.0.tgz
$ tar -zxvf kafka_2.12-2.5.0.tgz && cd kafka_2.12-2.5.0
具體參考官網 Kafka - Quickstart。
啟動 Kafka
Kafka 依賴 ZooKeeper，所以需要先運行后者。Kafak 安裝目錄下自帶了 ZooKeeper，可直接運行無須單獨安裝。
$ bin/zookeeper-server-start.sh config/zookeeper.properties
運行 Zookeeper 時發現機器上沒有 Java，報錯信息如下：
$ bin/zookeeper-server-start.sh config/zookeeper.properties
/root/dev/kafka/bin/kafka-run-class.sh: line 315: exec: java: not found
所以需要先安裝 Java。
安裝 Java 過程中如果 yum 報 https 錯誤，提示 404 repo 地址無效等，
yum install java 時的報錯信息$ yum install java-1.6.0-openjdk
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror
base | 3.6 kB 00:00:00
epel | 4.7 kB 00:00:00
extras | 3.4 kB 00:00:00
https://repo.mongodb.org/yum/redhat/2.2/mongodb-org/4.2/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article
https://access.redhat.com/articles/1320623
If above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/
One of the configured repositories failed (MongoDB Repository),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Disable the repository, so yum won't use it by default. Yum will then
just ignore the repository until you permanently enable it again or use
--enablerepo for temporary usage:
yum-config-manager --disable mongodb-org-4.2
4. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=mongodb-org-4.2.skip_if_unavailable=true
failure: repodata/repomd.xml from mongodb-org-4.2: [Errno 256] No more mirrors to try.
https://repo.mongodb.org/yum/redhat/2.2/mongodb-org/4.2/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
根據提示，執行 yum-config-manager --disable mongodb-org-4.2 后再次安裝就成功了。
檢查 Java 的安裝：
$ java -version
java version "1.6.0_38"
OpenJDK Runtime Environment (IcedTea6 1.13.10) (rhel-1.13.10.0.el7_2-x86_64)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)
另，如果需要卸載，執行 yum remove java-1.6.0-openjdk
再次運行 bin/zookeeper-server-start.sh config/zookeeper.properties 發現前面安裝的 Java 版本不對，低了…
$ bin/zookeeper-server-start.sh config/zookeeper.properties
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/zookeeper/server/quorum/QuorumPeerMain : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.zookeeper.server.quorum.QuorumPeerMain. Program will exit.
根據提示最小應該為 52 的版本，根據 Unsupported major.minor version 52.0 [duplicate]可知 52 對應的語義化版本。
同時，可通過 yum list available java\* 命令查詢到 yum 上可安裝的版本，找一個滿足要求的安裝即可，yum install java-1.8.0-openjdk。
再次檢查安裝：
$ java -version
openjdk version "1.8.0_71"
OpenJDK Runtime Environment (build 1.8.0_71-b15)
OpenJDK 64-Bit Server VM (build 25.71-b15, mixed mode)
啟動 Kafka 服務
$ bin/kafka-server-start.sh config/server.properties
創建 Topic
$ bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
創建生產者
創建生產者發送消息
$ bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test
創建消費者
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
此時發消息的地方有新增數據時，消費者處會實時獲取到。
mac 本機
修改 Kafka 配置
$ vi /usr/local/etc/kafka/server.properties
將 listeners=PLAINTEXT://:9092 取消其注釋並修改成如下形式后保存：
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
- listeners=PLAINTEXT://:9092
+ listeners=PLAINTEXT://localhost:9092
# Hostname and port the broker will advertise to producers and consumers. If not set,
# it uses the value for "listeners" if configured. Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092
啟動 zookeeper
$ brew services start zookeeper
==> Successfully started `zookeeper` (label: homebrew.mxcl.zookeeper)
注： 通過 brew 和 zookeeper 自己的命令啟動時，停止服務也需要對應的命令，即，brew services stop zookeeper 只會停止 brew 啟動的服務。
啟動 kafka
$ brew services start kafka
==> Successfully started `kafka` (label: homebrew.mxcl.kafka)
上面是以服務形式常駐啟動，如果臨時啟動，可使用下面的命令：
$ zkServer start
$ kafka-server-start /usr/local/etc/kafka/server.properties
創建 Topic
$ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
如果出現如下錯誤：
Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:262)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:258)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1863)
at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:341)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:55)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
- 檢查 zookeeper 服務是否正常：
$ zkServer status 22:22:32
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Client port found: 2181. Client address: localhost.
Error contacting service. It is probably not running.
如果是像上面這樣，zookeeper 根本沒啟動成功。雖然 brew services start zookeeper 及 zkServer start 都會提示啟動成功，這是比較坑爹的地方。
然后不要相信網上列的那一系列可能原因：
- host 沒綁
- 端口占用
- dataDir 目錄不存在或權限不足
- myid 文件缺失或內容不對
- ...
此時需要做的是先停掉服務，
$ brew services start zookeeper
# 或
$ zkServer stop
然后打開 /usr/local/etc/zookeeper/log4j.properties 配置文件看日志在哪，
$ cat /usr/local/etc/zookeeper/log4j.properties
log4j.rootCategory=WARN, zklog
log4j.appender.zklog = org.apache.log4j.RollingFileAppender
log4j.appender.zklog.File = /usr/local/var/log/zookeeper/zookeeper.log
log4j.appender.zklog.Append = true
log4j.appender.zklog.layout = org.apache.log4j.PatternLayout
log4j.appender.zklog.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %c{1} [%p] %m%n
可以看到，日志文件在 /usr/local/var/log/zookeeper/zookeeper.log 里面，可以看看里面有沒有報錯以幫助排查啟動的問題。
新開窗口使用 tail -f 打開日志，然后重新嘗試啟動 zookeeper 可以看到如下輸出：
$ tail -n 50 -f /usr/local/var/log/zookeeper/zookeeper.log
2020-05-13 09:58:43 QuorumPeerMain [WARN] Either no config or no quorum defined in config, running in standalone mode
2020-05-13 09:58:43 ContextHandler [WARN] o.e.j.s.ServletContextHandler@e45f292{/,null,UNAVAILABLE} contextPath ends with /*
2020-05-13 09:58:43 ContextHandler [WARN] Empty contextPath
2020-05-13 09:58:43 NIOServerCnxnFactory [ERROR] Thread Thread[main,5,main] died
java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
at org.apache.jute.BinaryOutputArchive.stringToByteBuffer(BinaryOutputArchive.java:77)
at org.apache.jute.BinaryOutputArchive.writeString(BinaryOutputArchive.java:107)
at org.apache.zookeeper.data.Id.serialize(Id.java:50)
at org.apache.jute.BinaryOutputArchive.writeRecord(BinaryOutputArchive.java:123)
at org.apache.zookeeper.data.ACL.serialize(ACL.java:51)
at org.apache.zookeeper.server.ReferenceCountedACLCache.serialize(ReferenceCountedACLCache.java:136)
at org.apache.zookeeper.server.DataTree.serialize(DataTree.java:1218)
at org.apache.zookeeper.server.util.SerializeUtils.serializeSnapshot(SerializeUtils.java:152)
at org.apache.zookeeper.server.persistence.FileSnap.serialize(FileSnap.java:210)
at org.apache.zookeeper.server.persistence.FileSnap.serialize(FileSnap.java:227)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.save(FileTxnSnapLog.java:406)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:248)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
原因就很明朗了， java 中報錯找不到對應方法。根據 Kafka with Zookeeper 3.5.7 Crash NoSuchMethodError: java.nio.ByteBuffer.flip() 這個類似問題下的回答，解決辦法有兩個：
- 升級 java 到 1.9
- 降級 zookeeper 到 3.14.x
看了下使用 brew 安裝后的版本：
$ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.252-b09, mixed mode)
$ brew info zookeeper
zookeeper: stable 3.5.7 (bottled), HEAD
Centralized server for distributed coordination of services
...
因為是本機，升級 java 會比較容易操作些，直接：
$ brew cask install java
...
完成后查看 java 版本，已經來到了最新的 14.0：
$ java -version
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment (build 14.0.1+7)
OpenJDK 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)
再次啟動 zookeeper，
$ zkServer start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Starting zookeeper ... STARTED
同時查看日志輸出：
2020-05-13 10:35:47 QuorumPeerMain [WARN] Either no config or no quorum defined in config, running in standalone mode
2020-05-13 10:35:47 ContextHandler [WARN] o.e.j.s.ServletContextHandler@69b2283a{/,null,UNAVAILABLE} contextPath ends with /*
2020-05-13 10:35:47 ContextHandler [WARN] Empty contextPath
2020-05-13 10:35:47 ZooKeeperServerMain [ERROR] Unable to start AdminServer, exiting abnormally
org.apache.zookeeper.server.admin.AdminServer$AdminServerException: Problem starting AdminServer on address 0.0.0.0, port 8080 and command URL /commands
at org.apache.zookeeper.server.admin.JettyAdminServer.start(JettyAdminServer.java:107)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:138)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Caused by: java.io.IOException: Failed to bind to /0.0.0.0:8080
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:308)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:396)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.zookeeper.server.admin.JettyAdminServer.start(JettyAdminServer.java:103)
... 5 more
Caused by: java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Net.java:479)
at java.base/sun.nio.ch.Net.bind(Net.java:468)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:88)
at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
... 12 more
可以看到，仍然報錯，但錯誤已經變了。看來前面的步驟起作用了。再來看新的報錯，
錯誤信息里關鍵信息是 Problem starting AdminServer on address 0.0.0.0, port 8080 and command URL /commands。
看來是端口占用的問題，查看 zookeeper 配置並添加 admin.serverPort 配置，修改默認的 8080 到其他端口：
$ vi /usr/local/etc/zookeeper/zoo.cfg
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
+ admin.serverPort=8089
再次嘗試啟動並查看日志：
2020-05-13 10:56:13 QuorumPeerMain [WARN] Either no config or no quorum defined in config, running in standalone mode
2020-05-13 10:56:13 ContextHandler [WARN] o.e.j.s.ServletContextHandler@69b2283a{/,null,UNAVAILABLE} contextPath ends with /*
2020-05-13 10:56:13 ContextHandler [WARN] Empty contextPath
2020-05-13 10:56:13 ZooKeeperServerMain [ERROR] Unexpected exception, exiting abnormally
java.net.BindException: Address already in use
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Net.java:479)
at java.base/sun.nio.ch.Net.bind(Net.java:468)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:88)
at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:687)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:143)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
錯誤已經更新成了 Address already in use。這個就是之前有起過 zookeeper 但忘了關，不管怎樣，通過下面對應的命令關閉一下即可：
$ brew services start zookeeper
# 或
$ zkServer stop
再次啟動：
$ zkServer start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Starting zookeeper ... STARTED
查看日志已經沒有異樣了：
2020-05-13 11:57:01 QuorumPeerMain [WARN] Either no config or no quorum defined in config, running in standalone mode
2020-05-13 11:57:02 ContextHandler [WARN] o.e.j.s.ServletContextHandler@69b2283a{/,null,UNAVAILABLE} contextPath ends with /*
2020-05-13 11:57:02 ContextHandler [WARN] Empty contextPath
再通過 zkServer status 查看 zookeeper 狀態，不出意外可以看到 Mode: standalone 字樣表示單例啟動成功，因為沒有配集群所以是本機單例。
$ zkServer status 11:57:01
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: standalone
最后，可以成功創建 Topic 了:
$ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic test.
創建生產者
創建生產者並生成消息：
$ kafka-console-producer --broker-list localhost:9092 --topic test
> hello
以上命令執行后會進入交互命令行，每輸入一行代表一條消息。
創建消費者
消費消息：
$ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning
> hello
啟動后會從隊列起始處開始消費消息並輸出。
相關資源
- Kafka - Quickstart
- 在 MacOS 上安裝 Kafka
- Apache Kafka Installation on Mac using Homebrew
- Find (and kill) process locking port 3000 on Mac
- ZooKepper Unable to start AdminServer, exiting abnormally
The text was updated successfully, but these errors were encountered: