Skip to content

Configure Kafka authentication to build Kafka client application

Preamble

In a previous blog post we have seen on how to use the Confluent Kafka REST API. In this follow-up blog post I wanted to use the two available Python packages and Java to directly address Kafka server through a Python or Java Kafka client application:

This sounds quite easy to setup but to do so you must fully configure your Kafka server to handle those request coming from different clients and not from the Kafka server itself.

Apache Confluent Kafka platform is ready for multiple authentication mechanisms clearly explained. I have chosen to use SASL/PLAIN. SASL stands for SASL (Simple Authentication Security Layer). SASL/PLAIN, is a simple username/password authentication mechanism, as passwords are sent in clear over the wire, in a production environment you should club SASL/PLAIN with TLS as transport layer.

Even if I plan to make a rapid test with SASL/PLAIN and TLS in a real enterprise environment I believe people would more do LDAP or OAuth/OIDC… Configuring only SASL/PLAIN was already not a piece of cake and Confluent documentation is not very well made for complete example configuration and I had to fight for a while…

I have used Confluent Community Platform 7.9.1. My test server is running Red Hat Enterprise Linux release 8.10 (Ootpa) with 64GB of memory and 12 cores. On my Windows client I have used Insomnia 11.1.0, Python 3.13.3 and Java JDK 21.0.7.

I have used confluent-kafka 2.10.0 and kafka-python 2.2.7 for Python and Apache Kafka client 4.0.0 for Java. I have not created example with oracledb Python package as it would be exactly the same as previous blog post…

Confluent Community Platform SASL/PLAIN setup for Kafka client application authentication

Kafka server

First stop every single component of you Kafka server, I guess you are on a test server first so should not be an issue.

In my previous blog I did not enter too much in the small tuning of the Kafka configuration file, but for SASL/PLAIN configuration you have to tune more precisely. So having this $CONFLUENT_HOME/etc/kafka/kraft/server.properties file:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@server01.domain.com:9093
listeners=BROKER://:9092,CONTROLLER://:9093
inter.broker.listener.name=BROKER
advertised.listeners=BROKER://server01.domain.com:9092,CONTROLLER://server01.domain.com:9093
controller.listener.names=CONTROLLER
listener.security.protocol.map=BROKER:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.mechanism.controller.protocol=PLAIN
security.protocol=SASL_PLAINTEXT

I have created this JAAS (Java Authentication and Authorization Service) configuration file (kafka_server_jaas.conf):

KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  serviceName="KafkaServer"
  username="admin"
  password="admin-secret"
  user_admin="admin-secret"
  user_schemaregistry="schemaregistry-secret"
  user_yjaquier="secure_password";
};

To take it into account you have to export KAFKA_OPTS variable:

export KAFKA_OPTS=”-Djava.security.auth.login.config=$CONFLUENT_HOME/etc/kafka/kraft/kafka_server_jaas.conf”

But the Confluent recommended method is to use listener.name.{listenerName}.{saslMechanism}.sasl.jaas.config server property to specify your configuration, so I have added to my server.properties file (listener names must be in lowercase or you get a stupid error). The protocol is PLAIN and NOT SASL_PLAINTEXT:

listener.name.broker.plain.sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="admin" \
  password="admin-secret" \
  user_admin="admin-secret" \
  user_schemaregistry="schemaregistry-secret" \
  user_yjaquier="secure_password";
listener.name.controller.plain.sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="admin" \
  password="admin-secret" \
  user_admin="admin-secret" \
  user_schemaregistry="schemaregistry-secret" \
  user_yjaquier="secure_password";

Then simply starting it with:

[kafka@server01 ~]$ nohup kafka-server-start $CONFLUENT_HOME/etc/kafka/kraft/server.properties > ~/kafka-server-start.log 2>&1 &

Schema Registry

I modified $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties file to match the new SASL_PLAINTEXT configuration:

kafkastore.bootstrap.servers=SASL_PLAINTEXT://localhost:9092

kafkastore.security.protocol=SASL_PLAINTEXT
kafkastore.sasl.mechanism=PLAIN
kafkastore.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="schemaregistry" \
  password="schemaregistry-secret";

Then simply starting it with:

[kafka@server01 ~]$ nohup schema-registry-start $CONFLUENT_HOME/etc/schema-registry/schema-registry.properties > ~/schema-registry-start.log 2>&1 &

Then after to issue all commands we have seen you need to be authenticated. I have created two file to authenticate with my yjaquier account and another one to authenticate with admin account:

[kafka@server01 ~]$ cat client.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="yjaquier" \
    password="secure_password";
[kafka@server01 ~]$ cat adminclient.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="admin" \
    password="admin-secret";

Then specify one of those client configuration files with different option versus the used tool. Examples:

[kafka@server01 ~]$ kafka-topics --bootstrap-server localhost:9092 --command-config client.conf --list
__consumer_offsets
_dek_registry_keys
_schema_encoders
_schemas
jsontest
oracle-kafka

[kafka@server01 ~]$ kafka-console-producer --topic oracle-kafka --bootstrap-server localhost:9092 --producer.config adminclient.conf

[kafka@server01 ~]$ kafka-console-consumer --topic oracle-kafka --bootstrap-server localhost:9092 --consumer.config client.conf --from-beginning

Python consumer and producer Kafka client application

Kafka Python

My script is:

from kafka import KafkaProducer, KafkaConsumer
from kafka.errors import KafkaTimeoutError, KafkaError
import json
import logging
import random
import argparse

def producer(topic_name: str) -> None:
  try:
    producer = KafkaProducer(bootstrap_servers="server01.domain.com:9092",
                            security_protocol='SASL_PLAINTEXT', 
                            sasl_mechanism='PLAIN',
                            sasl_plain_username="yjaquier",
                            sasl_plain_password="secure_password",
                            value_serializer=lambda m: json.dumps(m).encode('utf-8'), max_block_ms=5000)
  except Exception as e:
    print(f"Error creating Kafka producer: {e}")
    exit(1)
  else:
    print("Successfully created Kafka producer")

  try:
    producer.send(topic_name,  { "key": "Random value " + str(random.randint(0, 9999)) })
    # Wait for all messages to be sent
    producer.flush()
  except Exception as e:
    print(f"Error sending message to producer: {e}")
    exit(1)
  finally:
    # Close the producer
    producer.close()

  print("Message sent successfully.")

def consumer(topic_name: str) -> None:
  try:
    consumer = KafkaConsumer(bootstrap_servers="server01.domain.com:9092",
                            security_protocol='SASL_PLAINTEXT', 
                            sasl_mechanism='PLAIN',
                            sasl_plain_username="yjaquier",
                            sasl_plain_password="secure_password",
                            group_id="mygroup" + str(random.randint(1000, 9999)),
                            auto_offset_reset="latest")
  except Exception as e:
    print(f"Error creating Kafka consumer: {e}")
    exit(1)
  else:
    print("Successfully created Kafka consumer")

  # Topics list
  metadata = consumer.topics()
  print(f'Topics: {list(metadata)}')

  try:
    consumer.subscribe(topics=[topic_name])
  except Exception as e:
    print(f"Error subscribing to topic: {e}")
    exit(1)
  else:
    print(f"Successfully subscribed to topic {topic_name}")

  print(f"Polling messages from topic {topic_name} using kafka-python...")
  # print(consumer.partitions_for_topic(TOPIC_NAME))

  # Poll for messages
  try:
    while True:
      msg = consumer.poll(timeout_ms=1000)
      if not msg:
        # If no messages are returned, print "waiting"
        print("Waiting...")
      else:
        for topic_partition, messages in msg.items():
          for message in messages:
            print(f"Received message: {message.value.decode('utf-8')} from {message.topic} [{message.partition}]")
  except KeyboardInterrupt:
    pass
  finally:
    # Leave group and commit final offsets
    consumer.close()

def main():
  # Enable logging
  # logging.basicConfig(level=logging.DEBUG)
  logging.basicConfig(level=logging.ERROR)
  logger = logging.getLogger(__name__)

  TOPIC_NAME="oracle-kafka"
  # Create the parser
  parser = argparse.ArgumentParser(description="A simple example of using argparse.")

  # Add arguments
  parser.add_argument('-t', '--type', type=str, help='Producer or consumer ?', required=True)

  # Parse the arguments
  args = parser.parse_args()

  # Use the arguments
  if args.type == "producer":
    producer(TOPIC_NAME)
  elif args.type == "consumer":
    consumer(TOPIC_NAME)
  else:
    print("Invalid type. Use 'producer' or 'consumer'.")
    exit(1)

if __name__ == "__main__":
  main()

Execute first the consumer in one terminal:

$ python kafka_apache.py --type consumer
Successfully created Kafka consumer
Topics: ['_schemas', 'jsontest', 'my-topic', '_schema_encoders', 'oracle-kafka', '_dek_registry_keys']
Successfully subscribed to topic oracle-kafka
Polling messages from topic oracle-kafka using kafka-python...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Received message: {"key": "Random value 6900"} from oracle-kafka [0]
Waiting...
Waiting...
Waiting...
Waiting...

An the producer in another terminal:

$ python kafka_apache.py --type producer
Successfully created Kafka producer
Message sent successfully.

Confluent Kafka

Execution is exactly the same and script is:

from confluent_kafka import Producer, Consumer, KafkaException, TopicPartition
import json
import logging
import socket
import random
import argparse


def producer(topic_name: str) -> None:
  config_producer = {
    'bootstrap.servers': 'server01.domain.com:9092',
    'security.protocol': 'SASL_PLAINTEXT',
    'sasl.mechanism':   'PLAIN',
    'sasl.username':     'yjaquier',
    'sasl.password':     'secure_password'
  }
  try:
    producer = Producer(**config_producer)
  except Exception as e:
    print(f"Error creating Kafka producer: {e}")
    exit(1)
  else:
    print("Successfully created Kafka producer")

  # Delivery report callback
  def delivery_report(err, msg):
    if err is not None:
      print(f"Delivery failed for message {msg.key()}: {err}")
    else:
      print(f"Message {msg.key()} successfully delivered to {msg.topic()} [{msg.partition()}]")

  # Produce a message
  producer.produce(topic_name, key='key', value='{ "key": "Random value ' + str(random.randint(0, 9999)) + '" }', callback=delivery_report)

  # Wait for any outstanding messages to be delivered
  producer.flush()

def consumer(topic_name: str) -> None:
  config_consumer = {
    'bootstrap.servers': 'server01.domain.com:9092',
    'sasl.username':     'yjaquier',
    'sasl.password':     'secure_password',
    'security.protocol': 'SASL_PLAINTEXT',
    'sasl.mechanism':   'PLAIN',
    'group.id': 'mygroup' + str(random.randint(1000, 9999)),
    'auto.offset.reset': 'latest',
    'enable.auto.offset.store': False
  }

  try:
    consumer = Consumer(**config_consumer)
  except Exception as e:
    print(f"Error creating Kafka consumer: {e}")
    exit(1)
  else:
    print("Successfully created Kafka consumer")

  # Topics list
  metadata = consumer.list_topics().topics
  print(f'Topics: {list(metadata)}')

  def on_assign_callback(consumer, topic_partitions):
    for topic_partition in topic_partitions:
      print("Topic={}. partition={}. offset={}. error={}".format(topic_partition.topic, topic_partition.partition, topic_partition.offset, topic_partition.error))

  try:
    consumer.subscribe(topics=[topic_name], on_assign=on_assign_callback)
  except Exception as e:
    print(f"Error subscribing to topic: {e}")
    exit(1)
  else:
    print(f"Successfully subscribed to topic {topic_name}")

  # Poll for messages
  try:
    while True:
      msg = consumer.poll(1.0)
      if msg is None:
        print("Waiting...")
        # continue
      elif msg.error():
        print(f"Consumer error: {msg.error()}")
        raise KafkaException(msg.error())
        # continue
      else:
        print(f"Received message: {msg.value().decode('utf-8')} from {msg.topic()} [{msg.partition()}]")
        # print("Consumed event from topic {topic}: key = {key:12} value = {value:12}".format(topic=msg.topic(), key=msg.key().decode('utf-8'), value=msg.value().decode('utf-8')))
  except KeyboardInterrupt:
    pass
  finally:
    # Leave group and commit final offsets
    consumer.close()

def main():
  # Enable logging
  # logging.basicConfig(level=logging.DEBUG)
  logging.basicConfig(level=logging.ERROR)
  logger = logging.getLogger(__name__)

  TOPIC_NAME="oracle-kafka"
  # Create the parser
  parser = argparse.ArgumentParser(description="A simple example of using argparse.")

  # Add arguments
  parser.add_argument('-t', '--type', type=str, help='Producer or consumer ?', required=True)

  # Parse the arguments
  args = parser.parse_args()

  # Use the arguments
  if args.type == "producer":
    producer(TOPIC_NAME)
  elif args.type == "consumer":
    consumer(TOPIC_NAME)
  else:
    print("Invalid type. Use 'producer' or 'consumer'.")
    exit(1)

if __name__ == "__main__":
  main()

Java consumer and producer Kafka client application

Same as for Python I have created an unique package that will handle, through a parameter, the producer and consumer role. I have this project structure in IntelliJ:

kafka_client01
kafka_client01

My logger configuration is (simplelogger.properties):

org.slf4j.simpleLogger.defaultLogLevel=warn

The Java code:

package kafka01;

import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;
import java.util.Random;
import java.util.Collections;
import java.time.Duration;
import java.util.concurrent.ExecutionException;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class kafka01 {
  private static final String TOPIC_NAME = "oracle-kafka";
  private static final Logger logger = LoggerFactory.getLogger(kafka01.class);

  private static void producer() {
    Properties props = new Properties();
    Random random01 = new Random();
    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "server01.domain.com:9092");
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    props.put("security.protocol", "SASL_PLAINTEXT");
    props.put("sasl.mechanism", "PLAIN");
    props.put("sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"yjaquier\" password=\"secure_password\";");
    // Create the producer
    KafkaProducer<String, String> producer = new KafkaProducer<>(props);

    // Create a JSON object
    String jsonobject01 = "{ \"key\": \"Random value " + random01.nextInt(10000) + "\" }";
    // Create a producer record
    ProducerRecord<String, String> record = new ProducerRecord<>(TOPIC_NAME, "key"+random01.nextInt(100), jsonobject01);

    try {
      // Send the record and get metadata
      RecordMetadata metadata = producer.send(record).get();
      System.out.printf("Sent message to topic %s partition %d with offset %d%n",
              metadata.topic(), metadata.partition(), metadata.offset());
    } catch (InterruptedException | ExecutionException e) {
      logger.error("", e);
    } finally {
      // Close the producer
      producer.close();
    }

  }
  private static void consumer() {
    Random random01 = new Random();
    Properties props = new Properties();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "server01.domain.com:9092");
    props.put(ConsumerConfig.GROUP_ID_CONFIG, "mygroup" + random01.nextInt(1000, 9999));
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,   StringDeserializer.class.getCanonicalName());
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getCanonicalName());
    props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
    props.put("security.protocol", "SASL_PLAINTEXT");
    props.put("sasl.mechanism", "PLAIN");
    props.put("sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"yjaquier\" password=\"secure_password\";");
    // Create the consumer
    KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

    // Subscribe to the topic
    consumer.subscribe(Collections.singletonList(TOPIC_NAME));

    // Poll for new messages
    try {
      while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
        if (records.isEmpty()) {
          // If no messages are returned, print "waiting..."
          System.out.println("waiting...");
        } else {
          // Process each message in the batch
          for (ConsumerRecord<String, String> record : records) {
            System.out.printf("Received message: %s from topic %s partition %d offset %d%n", record.value(), record.topic(), record.partition(), record.offset());
          }
        }
      }
    } finally {
      // Close the consumer
      consumer.close();
    }
  }

  public static void main(String[] args) {
    if (args.length < 1) {
      System.out.println("Please specify 'producer' or 'consumer' as the first argument.");
      return;
    }

    String mode = args[0];

    if ("producer".equalsIgnoreCase(mode)) {
      System.out.println("Producer chosen");
      producer();
    } else if ("consumer".equalsIgnoreCase(mode)) {
      System.out.println("Consumer chosen");
      consumer();
    } else {
      System.out.println("Invalid argument " + mode + ". Please specify 'producer' or 'consumer'.");
    }
  }
}

Create a JAR file for this project (using Build Artifacts) and execute, first, the consumer with:

PS D:\> java -cp d:\temp\kafka.jar kafka01.kafka01 consumer
Consumer chosen
waiting...
waiting...
waiting...
waiting...
Received message: { "key": "Random value 5063" } from topic oracle-kafka partition 0 offset 94
waiting...
waiting...
waiting...

Genearate a message with (that appear as displayed just above):

PS D:\> java -cp d:\temp\kafka.jar kafka01.kafka01 producer
Producer chosen
Sent message to topic oracle-kafka partition 0 with offset 94

Connecting to Oracle

To connect to an Oracle database I have added Oracle JDBC JAR file to my project (ojdbc17.jar) and with a code like:

String jdbcUrl = "jdbc:oracle:thin:@//sever01.domain.com:1531/pdb1"; // Update with your Oracle DB details
String username = "yjaquier";
String password = "secure_password";
String result;

try (Connection connection = DriverManager.getConnection(jdbcUrl, username, password);
      Statement statement = connection.createStatement()) {

  // Execute a query
  String query = "select json_object('customer_name' is customer_name, 'total_sales' is total_sales) as json " +
                  "from (select c.name as customer_name, " +
                  "     sum(o.quantity * p.price) as total_sales " +
                  "from customer c join orders o on c.id = o.customer__id " +
                  "join product p on o.product__id = p.id " +
                  "group by c.name)";
  ResultSet resultSet = statement.executeQuery(query);
  Properties props = new Properties();
  Random random01 = new Random();
  props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "sever01.domain.com:9092");
  props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
  props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
  props.put("security.protocol", "SASL_PLAINTEXT");
  props.put("sasl.mechanism", "PLAIN");
  props.put("sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"yjaquier\" password=\"secure_password\";");
  // Create the producer
  KafkaProducer<String, String> producer = new KafkaProducer<>(props);

  while (resultSet.next()) {
    //result="{ \"customer_name\": \""+ resultSet.getString("customer_name") + "\", \"total_sales\": " + resultSet.getFloat("total_sales") + " }";
    result=resultSet.getString("json");
    //System.out.println(result);
    ProducerRecord<String, String> record = new ProducerRecord<>(TOPIC_NAME, "key", result);
    try {
      // Send the record and get metadata
      RecordMetadata metadata = producer.send(record).get();
      System.out.printf("Sent message to topic %s partition %d with offset %d%n",
              metadata.topic(), metadata.partition(), metadata.offset());
    } catch (InterruptedException | ExecutionException e) {
      logger.error("", e);
    }
  }
  producer.close();
} catch (Exception e) {
    logger.error("", e);
}

Of course it is also possible not to use the JSON feature of the database and build your own JSON object with one of the multiple Java JSON libraries (org.json, JSON.simple, Jackson, …).

Issues encountered

java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

KafkaTimeoutError: Failed to update metadata after 4999000.0 secs

Working from my Windows laptop and accessing to my Confluent Kafka Community server with this simple code:

import oracledb
from kafka import KafkaProducer
from kafka.errors import KafkaTimeoutError
import json
 
producer = KafkaProducer(bootstrap_servers=['server01.domain.com:9092'], value_serializer=lambda m: json.dumps(m).encode('utf-8'), max_block_ms=5000)

try:
  future = producer.send("oracle-kafka", b'some_message')
  result = future.get(timeout=60)  # setting the timeout to 60 seconds
except KafkaTimeoutError as e:
  print(e)
  print("Failed to send message due to timeout error")

I got ths error:

KafkaTimeoutError: Failed to update metadata after %s secs.
Failed to send message due to timeout error

But the code is working well when I execute it directly from the Kafka server… So there is a story of access rights… So the SASL_PLAINTEXT confguration of this blog post…

No serviceName defined in either JAAS or Kafka config

Required to set this parameter, undocumented on Confluent documentation web site, in JASS confiuration file if you use it (not directly if using only server.properties file). The name of the service is apparently not important but it must be specified:

serviceName="KafkaServer"

org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator

[2025-05-23 12:17:31,548] WARN [NodeToControllerChannelManager id=1 name=heartbeat] Error connecting to node localhost:9093 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
java.io.IOException: Channel could not be created for socket java.nio.channels.SocketChannel[closed]
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:368)
        at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:345)
        at org.apache.kafka.common.network.Selector.connect(Selector.java:272)
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1077)
        at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:321)
        at org.apache.kafka.server.util.InterBrokerSendThread.sendRequests(InterBrokerSendThread.java:146)
        at org.apache.kafka.server.util.InterBrokerSendThread.pollOnce(InterBrokerSendThread.java:109)
        at kafka.server.NodeToControllerRequestThread.doWork(NodeToControllerChannelManager.scala:377)
        at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:251)
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:356)
        ... 8 more
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.KafkaException: Principal could not be determined from Subject, this may be a transient failure due to Kerberos re-login
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.firstPrincipal(SaslClientAuthenticator.java:634)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.(SaslClientAuthenticator.java:205)
        at org.apache.kafka.common.network.SaslChannelBuilder.buildClientAuthenticator(SaslChannelBuilder.java:297)
        at org.apache.kafka.common.network.SaslChannelBuilder.lambda$buildChannel$1(SaslChannelBuilder.java:237)
        at org.apache.kafka.common.network.KafkaChannel.(KafkaChannel.java:143)
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:245)
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:356)
        at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:345)
        at org.apache.kafka.common.network.Selector.connect(Selector.java:272)
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1077)
        at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:321)
        at org.apache.kafka.server.util.InterBrokerSendThread.sendRequests(InterBrokerSendThread.java:146)
        at org.apache.kafka.server.util.InterBrokerSendThread.pollOnce(InterBrokerSendThread.java:109)
        at kafka.server.NodeToControllerRequestThread.doWork(NodeToControllerChannelManager.scala:377)
        at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136)

To solve this issue you have to add the following line in your server.properties file:

sasl.mechanism.controller.protocol=PLAIN

Unexpected Kafka request of type METADATA during SASL handshake

[2025-05-23 12:18:04,479] INFO [SocketServer listenerType=BROKER, nodeId=1] Failed authentication with /127.0.0.1 (channelId=127.0.0.1:9092-127.0.0.1:45820-0) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)

The general recommandation is to check if you have a running process, like a producer, a consumer, trying to connect in an unsecure way to your Kafka server. For me it was simply the Schema Registry process that was runing in the background that was trying to connect non-securely to my Kafka server.

bootstrap.servers ignored and keeps trying to connect to localhost:9092

Even if I specify correctly in my consumer configuration:

config_consumer = {
  'bootstrap.servers': 'server01.domain.com:9092',
  'sasl.username':     'yjaquier',
  'sasl.password':     'secure_password',
  'security.protocol': 'SASL_PLAINTEXT',
  'sasl.mechanism':   'PLAIN',
  'group.id': 'mygroup',
  'auto.offset.reset': 'earliest'
}

I was able to correctly connect to get the topics list but then when trying to consume event my Python script kept trying to connect to localhost and not my specified servername:

Successfully created Kafka consumer
Topics: ['_dek_registry_keys', '_schema_encoders', '_schemas', 'jsontest', '__consumer_offsets', 'oracle-kafka']
Successfully subscribed to topic 'oracle-kafka'
%3|1748249010.649|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Unknown error (after 2032ms in state CONNECT)
%3|1748249010.665|FAIL|rdkafka#consumer-1| [thrd:sasl_plaintext://localhost:9092/1]: sasl_plaintext://localhost:9092/1: Connect to ipv4#127.0.0.1:9092 failed: Unknown error (after 2043ms in state CONNECT)
%3|1748249012.847|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Unknown error (after 2041ms in state CONNECT, 1 identical error(s) suppressed)
%3|1748249014.652|FAIL|rdkafka#consumer-1| [thrd:sasl_plaintext://localhost:9092/1]: sasl_plaintext://localhost:9092/1: Connect to ipv4#127.0.0.1:9092 failed: Unknown error (after 2039ms in state CONNECT, 1 identical error(s) suppressed)
%3|1748249019.701|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Unknown error (after 2018ms in state CONNECT)
%3|1748249031.821|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Unknown error (after 2046ms in state CONNECT)
%3|1748249041.816|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Unknown error (after 2023ms in state CONNECT)
%3|1748249045.664|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: localhost:9092: Connect to ipv4#127.0.0.1:9092 failed: Unknown error (after 2050ms in state CONNECT)

Remark:
Thanks to this Python code to put my script in debug mode:

import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

I found this stackoverlow question and changing:

advertised.listeners=BROKER://server01.domain.com:9092,CONTROLLER://server01.domain.com:9093

Even with ‘auto.offset.reset’ set to ‘earliest’ you get no message

The reason is:

  • Kafka tracks the offset of messages for each consumer group. When a consumer reads a message, Kafka records the offset of the last message read.
  • If you run the consumer again with the same group ID, it will start reading from the last committed offset, which means it won’t re-read old messages.

So change the name of the consumer group id to get again all messages from the beginning…

References

About Post Author

Share the knowledge!

Leave a Reply

Your email address will not be published. Required fields are marked *