Windows 上連接虛擬機上docker的kafka報錯unknown host

Kafka Listeners - Explained

https://rmoff.net/2018/08/02/kafka-listeners-explained/

 The key thing is that when you run a client, the broker you pass to it is just where it’s going to go and get the metadata about brokers in the cluster from. The actual host & IP that it will connect to for reading/writing data is based on the data that the broker passes back in that initial connection—even if it’s just a single node and the broker returned is the same as the one connected to.

You need to set advertised.listeners (or KAFKA_ADVERTISED_LISTENERS if you’re using Docker images) to the external address (host/IP) so that clients can correctly connect to it. Otherwise they’ll try to connect to the internal host address–and if that’s not reachable then problems ensue.

Put another way, courtesy of Spencer Ruport:

LISTENERS are what interfaces Kafka binds to. ADVERTISED_LISTENERS are how clients can connect.

HOWTO: Connecting to Kafka on Docker

 

 

Here’s an example configuration:

listeners=INTERNAL://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=INTERNAL://ip-172-31-18-160.us-west-2.compute.internal:19092,EXTERNAL://ec2-54-191-84-122.us-west-2.compute.amazonaws.com:9092
inter.broker.listener.name=INTERNAL

　　

錯誤

[thrd:kafka1:9092/1002]: kafka1:9092/1002: Failed to resolve 'kafka1:9092': 不知道這樣的主機。 (after 2253ms in state CONNECT, 1 identical error(s) suppressed)

 

The brokers will advertise themselve using advertised.listeners (which seems to be abstracted with KAFKA_ADVERTISED_HOST_NAME in that docker image) and the clients will consequently try to connect to these advertised hosts and ports.

You will thus need to make sure the client can resolve and reach this advertised hostnames, e.g. by adding "kafka" to /etc/hosts on the client host.

192.168.0.11 是虛擬機上的IP，Windows Client 機上連不上虛擬機里面的docker網絡172.20.0.14~16，而連接里必須跟KAFKA_ADVERTISED_LISTENERS的設置地址一樣，在window上hosts文件設置

192.168.0.11 kafka1

192.168.0.11 kafka2

192.168.0.11 kafka3

 

public override void ConfigureServices(ServiceConfigurationContext context)
        {
            Configure<AbpKafkaOptions>(options =>
            {
                //options.Connections.Default.BootstrapServers = "127.0.0.1:9092";
                options.Connections.Default.BootstrapServers = "kafka1:9092,kafka2:9093,kafka3:9094";
            });

            Configure<AbpKafkaEventBusOptions>(options =>
            {
                options.GroupId = "MyGroupId2";
                options.TopicName = "MyTopicName";　

 

docker-compose.yml

version: '3.7'

networks:
  br17219:
    external:
      name: br17219

services:
  zk1:
    image: zookeeper
    hostname: zk1
    container_name: zk1
    ports:
      - 2181:2181
      - 8181:8080
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=zk3:2888:3888;2181
    volumes:
      - /usr/local/zookeeper/zk1/data:/data
      - /usr/local/zookeeper/zk1/datalog:/datalog
    networks:
      br17219:
        ipv4_address: 172.20.0.11

  zk2:
    image: zookeeper
    hostname: zk2
    container_name: zk2
    ports:
      - 2182:2181
      - 8182:8080
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zk3:2888:3888;2181
    volumes:
      - /usr/local/zookeeper/zk2/data:/data
      - /usr/local/zookeeper/zk2/datalog:/datalog
    networks:
      br17219:
        ipv4_address: 172.20.0.12

  zk3:
    image: zookeeper
    hostname: zk3
    container_name: zk3
    ports:
      - 2183:2181
      - 8183:8080
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zk1:2888:3888;2181 server.2=zk2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181
    volumes:
      - /usr/local/zookeeper/zk3/data:/data
      - /usr/local/zookeeper/zk3/datalog:/datalog
    networks:
      br17219:
        ipv4_address: 172.20.0.13
      
  kafka1:
    image: wurstmeister/kafka
    depends_on:
      - zk1
      - zk2
      - zk3
    restart: always
    hostname: kafka1
    container_name: kafka1
    ports:
      - 9092:9092
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zk1:2181,zk2:2181,zk3:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_LISTENERS: PLAINTEXT://kafka1:9092
    volumes:
      - /usr/local/kafka/kafka1/logs:/kafka
    external_links:   
      - zk1
      - zk2
      - zk3
    networks:
      br17219:
        ipv4_address: 172.20.0.14
    extra_hosts:
      - kafka1:172.20.0.14

  kafka2:
    image: wurstmeister/kafka
    depends_on:
      - zk1
      - zk2
      - zk3
    restart: always
    hostname: kafka2
    container_name: kafka2
    ports:
      - 9093:9093
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka2
      KAFKA_ADVERTISED_PORT: 9093
      KAFKA_ZOOKEEPER_CONNECT: zk1:2181,zk2:2181,zk3:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9093
      KAFKA_LISTENERS: PLAINTEXT://kafka2:9093
    volumes:
      - /usr/local/kafka/kafka2/logs:/kafka
    external_links:   
      - zk1
      - zk2
      - zk3
    networks:
      br17219:
        ipv4_address: 172.20.0.15
    extra_hosts:
      - kafka2:172.20.0.15

  kafka3:
    image: wurstmeister/kafka
    depends_on:
      - zk1
      - zk2
      - zk3
    restart: always
    hostname: kafka3
    container_name: kafka3
    ports:
      - 9094:9094
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka3
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_ZOOKEEPER_CONNECT: zk1:2181,zk2:2181,zk3:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9094
      KAFKA_LISTENERS: PLAINTEXT://kafka3:9094
    volumes:
      - /usr/local/kafka/kafka3/logs:/kafka
    external_links:   
      - zk1
      - zk2
      - zk3
    networks:
      br17219:
        ipv4_address: 172.20.0.16
    extra_hosts:
      - kafka3:172.20.0.16

  kafka-manager:
    image: sheepkiller/kafka-manager:latest
    restart: always
    container_name: kafka-manager
    hostname: kafka-manager
    ports:
      - "9010:9000"
    links:            
      - kafka1
      - kafka2
      - kafka3
    external_links:   
      - zk1
      - zk2
      - zk3
    environment:
      ZK_HOSTS: zk1:2181,zk2:2181,zk3:2181                 
      TZ: CST-8
    networks:
      br17219:
        ipv4_address: 172.20.0.20
        
  efak:
    image: ydockerp/efak:2.0.8
    restart: always
    container_name: efak-manager
    hostname: efak
    ports:
      - "8048:8048"
    depends_on:
      - kafka1
      - kafka2
      - kafka3
    environment:
      ZK_HOSTS: zk1:2181,zk2:2181,zk3:2181
    networks:
      br17219:
        ipv4_address: 172.20.0.21

　　

結果