kafka命令及啟動

默認內網訪問，要在外網訪問的話，需要在修改config/server.properties中的配置

將listeners和advertised.listeners的值用主機名進行替換，在外用使用java進行生產者或消費者連接的時候，不填寫具體的IP，填寫安裝kafka的主機名，然后，在hosts目錄中，配置該主機名對應的真是IP地址即可；

以下命令都是摘抄與官網http://kafka.apache.org/quickstart

先啟動zookeeper,默認自帶的

bin/zookeeper-server-start.sh config/zookeeper.properties

然后啟動kafka服務

bin/kafka-server-start.sh config/server.properties

列舉擁有哪些topics

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

在服務器上打開一個生產者，然后把輸入的每行數據發送到kafka中的命令

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
#后面光標提示數據數據，然后回車就會發送到kafka中了

打開一個消費者

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

當有數據往kafka的test主題發送消息，這邊就會進行消費。

java調用作為生產者和消費者代碼：

項目需要引入的依賴pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.theorydance</groupId>
    <artifactId>kafkademo</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>kafkademo</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.1.1</version>
        </dependency>
    </dependencies>
</project>

生產者代碼ProducerDemo.java

package com.theorydance.kafkademo;

import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;

public class ProducerDemo {
    public static void main(String[] args){
        Properties properties = new Properties();
        properties.put("bootstrap.servers", "node125:9092");
        properties.put("acks", "all");
        properties.put("retries", 0);
        properties.put("batch.size", 16384);
        properties.put("linger.ms", 1);
        properties.put("buffer.memory", 33554432);
        properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        Producer<String, String> producer = null;
        try {
            producer = new KafkaProducer<String, String>(properties);
            for (int i = 0; i < 100; i++) {
                String msg = "This is Message " + i;
                producer.send(new ProducerRecord<String, String>("HelloWorld", msg));
                System.out.println("Sent:" + msg);
            }
        } catch (Exception e) {
            e.printStackTrace();

        } finally {
            producer.close();
        }

    }
}

消費者代碼ConsumerDemo.java

package com.theorydance.kafkademo;

import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.PartitionInfo;

public class ConsumerDemo {
    public static void main(String[] args) throws InterruptedException {
        Properties properties = new Properties();
        properties.put("bootstrap.servers", "node125:9092");
        properties.put("group.id", "group-1");
        properties.put("enable.auto.commit", "true");
        properties.put("auto.commit.interval.ms", "1000");
        properties    .put("auto.offset.reset", "earliest");
        properties.put("session.timeout.ms", "30000");
        properties.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        properties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        KafkaConsumer<String, String> kafkaConsumer = new KafkaConsumer<>(properties);
        
        while(true){
            Map<String, List<PartitionInfo>> maps = kafkaConsumer.listTopics();
            System.out.println("監聽topics="+maps.keySet());
            Set<String> sets = new HashSet<>();
            for (String topic : maps.keySet()) {
                if(topic.startsWith("Hello")){ // 制定規則，監聽哪一些的topic
                    sets.add(topic);
                }
            }
            kafkaConsumer.subscribe(sets);
            long startTime = System.currentTimeMillis();
            while (true) {
                ConsumerRecords<String, String> records = kafkaConsumer.poll(100);
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("offset = %d, value = %s, topic = %s", record.offset(), record.value(), record.topic());
                    System.out.println("=====================>");
                }
                long endTime = System.currentTimeMillis();
                if(endTime - startTime > 30000){
                    System.out.println("------------------------------------------------------------------");
                    break;
                }
            }
        }
        

    }
}

說明：在實際需求中，我需要收集在不同服務器上的日志（微服務相同模塊和不同模塊，或其他程序的日志），采用的是flume進行收集，希望能夠對收集的日志進行分類（區別是哪個程序產生的），去網上找了一下，在flume進行收集的時候，能不能在日志前面加上應用的標識進行區別，我沒有找到，如果有，看到該博客的同行，請不吝賜教。我這邊就換了種思路，就像前面我寫的消費者示例一樣，不同的程序日志，我往不同的topic中進行發送消息，在消費者監聽一定規則的topic，然后進行消費，這樣就可以區分不同的應用程序的日志了。