golang confluent-kafka-go

針對golang的 kafka client 有很多開源package，例如sarama, confluent等等。在使用sarama 包時，高並發中偶爾遇到crash。於是改用confluent-kafka-go，其簡單易用，並且表現穩定。

本文主要介紹confluent-kafka-go的使用方法。
confluent-kafka-go，是kafka官網推薦的golang package。

confluent-kafka-go is Confluent's Golang client for Apache Kafka and the Confluent Platform.

編譯環境搭建

安裝librdkafka

下載

$ git clone https://github.com/edenhill/librdkafka.git $ cd librdkafka 

配置、編譯、安裝

$ ./configure --prefix /usr
$ make
$ sudo make install

配置PKG_CONFIG_PATH

在文件~/.bashrc 末尾添加

export PKG_CONFIG_PATH=/usr/lib/pkgconfig

下載go client

$ go get -u github.com/confluentinc/confluent-kafka-go/kafka 

自動下載到GOPATH目錄下，也可到github上自行下載，然后放到GOPATH中。

Example

// Example function-based Apache Kafka producer package main /** * Copyright 2016 Confluent Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ import ( "fmt" "github.com/confluentinc/confluent-kafka-go/kafka" "os" ) func main() { if len(os.Args) != 3 { fmt.Fprintf(os.Stderr, "Usage: %s <broker> <topic>\n", os.Args[0]) os.Exit(1) } broker := os.Args[1] topic := os.Args[2] p, err := kafka.NewProducer(&kafka.ConfigMap{"bootstrap.servers": broker}) if err != nil { fmt.Printf("Failed to create producer: %s\n", err) os.Exit(1) } fmt.Printf("Created Producer %v\n", p) // Optional delivery channel, if not specified the Producer object's // .Events channel is used. deliveryChan := make(chan kafka.Event) value := "Hello Go!" err = p.Produce(&kafka.Message{TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny}, Value: []byte(value)}, deliveryChan) e := <-deliveryChan m := e.(*kafka.Message) if m.TopicPartition.Error != nil { fmt.Printf("Delivery failed: %v\n", m.TopicPartition.Error) } else { fmt.Printf("Delivered message to topic %s [%d] at offset %v\n", *m.TopicPartition.Topic, m.TopicPartition.Partition, m.TopicPartition.Offset) } close(deliveryChan) } 

注意：
如果需要鏈接靜態庫，可刪除/usr/lib/下面關於rdkafka的動態庫文件(.so文件)。然后，go build編譯時加上選項 –tags static
例如：

go build -tags static produer.go

更多example，可參考
https://github.com/confluentinc/confluent-kafka-go/tree/master/examples

參考

https://github.com/confluentinc/confluent-kafka-go