Skip to content

Kafka TLS/SSL encryption and Access Control Lists (ACLs) setup

Preamble

In my previous article (http) where I implemented SASL/PLAIN for my Kafka server I noted an important security warning in Confluent documentation:

SASL/PLAIN should only be used with TLS as transport layer to ensure that clear passwords are not transmitted on the wire without encryption.

Before going further in my testing in trying to give authorization to users this Data in Motion protection with TLS Encryption is a required step.

I also wanted to implement LDAP authentication but when I have tried to use the Confluent LDAP authentication:

listener.name.broker.plain.sasl.server.callback.handler.class = io.confluent.security.auth.provider.ldap.LdapAuthenticateCallbackHandler

I got:

Class io.confluent.security.auth.provider.ldap.LdapAuthenticateCallbackHandler could not be found.

The LDAP authentication is not in Confluent Community edition, you need the Enterprise one to implement it. There are few open source Kafka LDAP class or you can even write your own but this is a full project in itself…

Again I have struggled a lot in what was supposed to be a simple configuration to encrypt traffic between client and server and Confluent documentation is lacking a lot practical examples…

I have used Confluent Community Platform 7.9.1. My test server is running Red Hat Enterprise Linux release 8.10 (Ootpa) with 64GB of memory and 12 cores. On my Windows client I have used Python 3.13.3.

I have also used confluent-kafka 2.10.0 and kafka-python 2.2.7 Python libraries.

The high level big pictures is the following (diagrams generated with https://mermaid.js.org/ that is a really nice discover)

kafka tls-ssl and acls01
kafka tls-ssl and acls01

Kafka TLS/SSL encryption

It is strongly suggested to keep hostname verification for client connections as Confluent has written in documentation:

Host name verification of servers is enabled by default for client connections as well as interbroker connections to prevent man-in-the-middle attacks.

For testing purpose it is oaky to deactivate it but from security standpoint it is a must to keep it (particularly in production), set ssl.endpoint.identification.algorithm to an empty string in server.properties file:

ssl.endpoint.identification.algorithm=

Hostname verification can be done with two methods:

  • Common Name (CN)
  • Subject Alternative Name (SAN) (-ext SAN=DNS:hostname)

CN is relatively straightforward while the SAN one has kept me busy for multiple hours… It is strongly suggested to inject all parameters to avoid dying prompt of keytool and openssl (location, name, …). This is particularly true if, like me, you perform multiple tens of trials… Even if unsecure I am also using -storepass option of keytool to avoid typing my keystore password multiple tens times per day…

Kafka TLS/SSL with common Name (CN)

Generate the key and the certificate for each Confluent Server broker in the cluster (notice the CN=server01.domain.com that is your FQDN server name):

keytool -keystore kafka.server.keystore.jks -alias server01.domain.com -keyalg RSA -validity 730 -genkey -storepass secure_password -dname "CN=server01.domain.com, OU=IT, O=IT, L=Geneva, S=Geneva, C=CH"

Generate a Certificate Authority (CA) that is simply a public(ca.cert)-private(ca.key) key pair and certificate. To avoid answering question I have created this openssl.conf.cn file first (notice the commonName parameter):

[kafka@server01 ssl]$ cat openssl.conf.cn
[ req ]
default_bits            = 2048
encrypt_key             = no
default_md              = sha256
utf8                    = yes
string_mask             = utf8only
prompt                  = no
distinguished_name = req_distinguished_name

[ req_distinguished_name ]
countryName         = CH
stateOrProvinceName = Geneva
localityName        = Geneva
organizationName    = MyCompany
organizationalUnitName = IT
commonName          = server01.domain.com

[kafka@server01 ssl]$ openssl req -new -x509 -keyout ca.key -out ca.cert -days 730 -config openssl.conf.cn

Store the CA in client and server truststore:

keytool -keystore kafka.client.truststore.jks -alias CARoot -importcert -file ca.cert -storepass secure_password -noprompt
keytool -keystore kafka.server.truststore.jks -alias CARoot -importcert -file ca.cert -storepass secure_password -noprompt

Export the certificate from the keystore:

keytool -keystore kafka.server.keystore.jks -alias server01.domain.com -certreq -file cert-file -storepass secure_password -noprompt

Sign it with the CA:

openssl x509 -req -CA ca.cert -CAkey ca.key -in cert-file -out cert-signed -days 730 -CAcreateserial

Import both the certificate of the CA and the signed certificate into the broker keystore:

keytool -keystore kafka.server.keystore.jks -alias CARoot -importcert -file ca.cert -storepass secure_password -noprompt
keytool -keystore kafka.server.keystore.jks -alias server01.domain.com -importcert -file cert-signed -storepass secure_password -noprompt

All in one in a generate.sh.cn script:

rm *.jks ca* cert* *.pem
keytool -keystore kafka.server.keystore.jks -alias server01.domain.com -keyalg RSA -validity 730 -genkey -storepass secure_password -dname "CN=server01.domain.com, OU=IT, O=IT, L=Geneva, S=Geneva, C=CH"
openssl req -new -x509 -keyout ca.key -out ca.cert -days 730 -config openssl.conf.cn
keytool -keystore kafka.client.truststore.jks -alias CARoot -importcert -file ca.cert -storepass secure_password -noprompt
keytool -keystore kafka.server.truststore.jks -alias CARoot -importcert -file ca.cert -storepass secure_password -noprompt
keytool -keystore kafka.server.keystore.jks -alias server01.domain.com -certreq -file cert-file -storepass secure_password -noprompt
openssl x509 -req -CA ca.cert -CAkey ca.key -in cert-file -out cert-signed -days 730 -CAcreateserial
keytool -keystore kafka.server.keystore.jks -alias CARoot -importcert -file ca.cert -storepass secure_password -noprompt
keytool -keystore kafka.server.keystore.jks -alias server01.domain.com -importcert -file cert-signed -storepass secure_password -noprompt

Then the $CONFLUENT_HOME/etc/kafka/kraft/server.properties:

listeners=BROKER://:9092,CONTROLLER://:9093

inter.broker.listener.name=BROKER

advertised.listeners=BROKER://server01.domain.com:9092,CONTROLLER://server01.domain.com:9093

controller.listener.names=CONTROLLER

listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL

sasl.enabled.mechanisms=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.mechanism.controller.protocol=PLAIN
#security.protocol=SASL_PLAINTEXT
security.protocol=SASL_SSL

listener.name.broker.plain.sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="admin" \
  password="admin-secret" \
  user_admin="admin-secret" \
  user_schemaregistry="schemaregistry-secret" \
  user_yjaquier="secure_password";
listener.name.controller.plain.sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="admin" \
  password="admin-secret" \
  user_admin="admin-secret" \
  user_schemaregistry="schemaregistry-secret" \
  user_yjaquier="secure_password";

ssl.client.auth=required
#ssl.endpoint.identification.algorithm= # Avoid to set: unsecure

ssl.truststore.location=/home/kafka/ssl/kafka.server.truststore.jks
ssl.truststore.password=secure_password
ssl.keystore.location=/home/kafka/ssl/kafka.server.keystore.jks
ssl.keystore.password=secure_password

Use five below commands to ensure the CN has been correctly filled for the keystore of the server and the truststore of the server/client:

keytool -list -v -keystore kafka.server.keystore.jks -storepass secure_password
keytool -list -v -keystore kafka.server.truststore.jks -storepass secure_password
keytool -list -v -keystore kafka.client.truststore.jks -storepass secure_password
openssl x509 -in ca.cert -text -noout
openssl x509 -in cert-signed -text -noout

Start the Kafka server (broker and controller) with something like:

nohup kafka-server-start $CONFLUENT_HOME/etc/kafka/kraft/server.properties > ~/kafka-server-start.log 2>&1 &
clear;kafka-server-start $CONFLUENT_HOME/etc/kafka/kraft/server.properties

Once the Kafka broker is up you can check the CN in its certificate with:

openssl s_client -connect server01.domain.com:9092 -showcerts

My Kafka client configuration file (client.conf) is now:

security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="yjaquier" \
    password="secure_password";
ssl.truststore.location=/home/kafka/ssl/kafka.client.truststore.jks
ssl.truststore.password=secure_password
#ssl.endpoint.identification.algorithm= # unsecure

Start a consumer with:

kafka-console-consumer --topic oracle-kafka --bootstrap-server server01.domain.com:9092 --consumer.config ~/client.conf --from-beginning

In my producer Python code I now have:

producer = KafkaProducer(bootstrap_servers="server01.domain.com:9092",
                            security_protocol='SASL_SSL', 
                            sasl_mechanism='PLAIN',
                            sasl_plain_username="yjaquier",
                            sasl_plain_password="secure_password",
                            value_serializer=lambda m: json.dumps(m).encode('utf-8'), max_block_ms=5000)

You should get:

Error creating Kafka producer: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1028)

To solve this, extract the CA from the client truststore (or server or server keystore as the CA is present in all of them) with:

keytool -exportcert -alias CARoot -keystore kafka.client.truststore.jks -rfc -file CARoot.pem -storepass secure_password

Transfer this CARoot.pem file in the directory of your Python script and change the producer with:

producer = KafkaProducer(bootstrap_servers="server01.domain.com:9092",
                            security_protocol='SASL_SSL', 
                            sasl_mechanism='PLAIN',
                            sasl_plain_username="yjaquier",
                            sasl_plain_password="secure_password",
                            ssl_cafile='CARoot.pem',
                            # ssl_check_hostname=False,
                            value_serializer=lambda m: json.dumps(m).encode('utf-8'), max_block_ms=5000)

And you should get:

$ python kafka_apache.py --type producer
Successfully created Kafka producer
Message sent successfully.

Kafka TLS/SSL with Subject Alternative Name (SAN)

Even if using CN is valid SAN is recommended as stated in Confluent documentation:

Both fields are valid, however RFC-2818 recommends the use of SAN. SAN is also more flexible, allowing for multiple DNS entries to be declared. Another advantage is that the CN can be set to a more meaningful value for authorization purposes. To add a SAN field, append the argument -ext SAN=DNS:{FQDN} to the keytool command:

The Confluent documentation is not clear at all for SAN configuration and I had to fight a lot. Particularly signing the certificate with the CA and the -extfile and -extensions option of openssl. I had this issue because the controller process was not able to authenticate with broker because the signed certificate did not contain the SAN name of the server. My openssl.conf.san file is also much more complete:

[kafka@server01 ssl]$ cat openssl.conf.san
[ req ]
default_bits            = 2048
encrypt_key             = no
default_md              = sha256
utf8                    = yes
string_mask             = utf8only
prompt                  = no
distinguished_name      = req_distinguished_name
x509_extensions         = req_ext

[ req_distinguished_name ]
countryName         = CH
stateOrProvinceName = Geneva
localityName        = Geneva
organizationName    = MyCompany
organizationalUnitName = IT
commonName          = Yannick Jaquier

[ req_ext ]
basicConstraints        = CA:TRUE
keyUsage                = digitalSignature, keyCertSign
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = server01.domain.com
DNS.2 = server01
DNS.3 = localhost
IP.1  = 10.75.43.64

Remark:
I have deliberately chosen a different CN to avoid mis-configuration

Not entering again in details and my generate.sh.san script is (I have also used a CN different from server name):

rm *.jks ca* cert* *.pem
keytool -keystore kafka.server.keystore.jks -alias server01.domain.com -keyalg RSA -validity 730 -genkeypair -ext SAN=dns:server01.domain.com,dns:server01,ip:10.75.43.64 -storepass secure_password -dname "CN=Yannick Jaquier, OU=IT, O=IT, L=Geneva, S=Geneva, C=CH"
openssl req -new -x509 -keyout ca.key -out ca.cert -days 730 -config openssl.conf.san
keytool -keystore kafka.client.truststore.jks -alias CARoot -importcert -file ca.cert -ext SAN=dns:server01.domain.com,dns:server01,ip:10.75.43.64 -storepass secure_password -noprompt
keytool -keystore kafka.server.truststore.jks -alias CARoot -importcert -file ca.cert -ext SAN=dns:server01.domain.com,dns:server01,ip:10.75.43.64 -storepass secure_password -noprompt
keytool -keystore kafka.server.keystore.jks -alias server01.domain.com -certreq -file cert-file -ext SAN=dns:server01.domain.com,dns:server01,ip:10.75.43.64 -storepass secure_password -noprompt
openssl x509 -req -CA ca.cert -CAkey ca.key -in cert-file -out cert-signed -days 730 -CAcreateserial -extfile openssl.conf.san -extensions req_ext
keytool -keystore kafka.server.keystore.jks -alias CARoot -importcert -file ca.cert -ext SAN=dns:server01.domain.com,dns:server01,ip:10.75.43.64 -storepass secure_password -noprompt
keytool -keystore kafka.server.keystore.jks -alias server01.domain.com -importcert -file cert-signed -ext SAN=dns:server01.domain.com,dns:server01,ip:10.75.43.64 -storepass secure_password -noprompt

In the middle of re-starting your Kafka server ensure all certificate contains a SAN entry with those commands:

keytool -list -v -keystore kafka.server.keystore.jks -storepass secure_password
keytool -list -v -keystore kafka.server.truststore.jks -storepass secure_password
keytool -list -v -keystore kafka.client.truststore.jks -storepass secure_password
openssl x509 -in ca.cert -text -noout
openssl x509 -in cert-signed -text -noout

Example:

[kafka@server01 ssl]$ openssl x509 -in cert-signed -text -noout
.
.
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:TRUE
            X509v3 Key Usage:
                Digital Signature, Certificate Sign
            X509v3 Subject Alternative Name:
                DNS:server01.domain.com, DNS:server01, DNS:localhost, IP Address:10.75.43.64
.
.
.

The Python testing remains the same…

Access Control Lists (ACLs)

Once your users need to authenticate, one step forward in security is to give and forbid some users to perform some operations. For example how would you restrict a Kafka topics to be open for everyone for producing ?

For Kraft Kafka server add the following lie in $CONFLUENT_HOME/etc/kafka/kraft/server.properties file:

authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin

And restart your Kafka Server…

Now if I try to launch a consumer with my yjaquier account I get:

[kafka@server01 ~]$ kafka-console-consumer --topic oracle-kafka --bootstrap-server server01.domain.com:9092 --consumer.config ~/client.conf --from-beginning
[2025-06-10 15:15:07,746] WARN [Consumer clientId=console-consumer, groupId=console-consumer-18100] The metadata response from the cluster reported a recoverable issue with correlation id 2 : {oracle-kafka=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2025-06-10 15:15:07,752] ERROR [Consumer clientId=console-consumer, groupId=console-consumer-18100] Topic authorization failed for topics [oracle-kafka] (org.apache.kafka.clients.Metadata)
[2025-06-10 15:15:07,756] ERROR Error processing message, terminating consumer process:  (org.apache.kafka.tools.consumer.ConsoleConsumer)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [oracle-kafka]
Processed a total of 0 messages

Same with Python producer client:

$ python kafka_apache.py --type producer
Successfully created Kafka producer
ERROR:kafka.cluster:Topic oracle-kafka is not authorized for this client
ERROR:kafka.producer.kafka:kafkaproducer client_id="kafka-python-producer-1" transactional_id="None": Exception occurred during message send: [Error 29] TopicAuthorizationFailedError: oracle-kafka
Message sent successfully.

Remark:
The misisng authorization is strangely not generating an exception…

I have built this adminclient.conf configuration file:

[kafka@server01 ~]$ cat adminclient.conf
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="admin" \
    password="admin-secret";
ssl.truststore.location=/home/kafka/ssl/kafka.client.truststore.jks
ssl.truststore.password=secure_password

Initially I have no grants on my Kafka topic oracle-kafka:

[kafka@server01 ~]$ kafka-acls --bootstrap-server server01.domain.com:9092 --command-config ~/adminclient.conf --list --topic oracle-kafka

I add my yjaquier account to be able to read from the topic from any host (I do not specify the group option):

[kafka@server01 ~]$ kafka-acls --bootstrap-server server01.domain.com:9092 --command-config ~/adminclient.conf \
>   --add \
>   --allow-principal User:yjaquier \
>   --operation read \
>   --topic oracle-kafka
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=oracle-kafka, patternType=LITERAL)`:
        (principal=User:yjaquier, host=*, operation=READ, permissionType=ALLOW)

[kafka@server01~]$ kafka-acls --bootstrap-server server01.domain.com:9092 --command-config ~/adminclient.conf --list --topic oracle-kafka
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=oracle-kafka, patternType=LITERAL)`:
        (principal=User:yjaquier, host=*, operation=READ, permissionType=ALLOW)

I got this error. The group name is dynamically generated to impossible to guess it in advance. I could have specified it but we have already seen that it memorized the status of what we have already read but clearly this is an option to consider for extra security:

[kafka@server01 ~]$ kafka-console-consumer --topic oracle-kafka --bootstrap-server server01.domain.com:9092 --consumer.config ~/client.conf --from-beginning
[2025-06-10 15:43:13,260] ERROR Error processing message, terminating consumer process:  (org.apache.kafka.tools.consumer.ConsoleConsumer)
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: console-consumer-6192
Processed a total of 0 messages

So granting for any group name:

[kafka@server01 ~]$ kafka-acls --bootstrap-server server01.domain.com:9092 --command-config ~/adminclient.conf \
>   --add \
>   --allow-principal User:yjaquier \
>   --operation read \
>   --topic oracle-kafka \
>   --group '*'
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=oracle-kafka, patternType=LITERAL)`:
        (principal=User:yjaquier, host=*, operation=READ, permissionType=ALLOW)

Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
        (principal=User:yjaquier, host=*, operation=READ, permissionType=ALLOW)

[kafka@server01 ~]$ kafka-acls --bootstrap-server server01.domain.com:9092 --command-config ~/adminclient.conf --list --topic oracle-kafka
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=oracle-kafka, patternType=LITERAL)`:
        (principal=User:yjaquier, host=*, operation=READ, permissionType=ALLOW)

After this I’m able to launch a consumer process directly from the server but still it is not possible to produce a message:

[kafka@server01 ~]$ kafka-console-producer --topic oracle-kafka --bootstrap-server server01.domain.com:9092 --producer.config ~/client.conf
>{"key": "Random value 1234"}
[2025-06-10 16:26:20,996] ERROR [Producer clientId=console-producer] Aborting producer batches due to fatal error (org.apache.kafka.clients.producer.internals.Sender)
org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
[2025-06-10 16:26:21,006] ERROR Error when sending message to topic oracle-kafka with key: null, value: 28 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
org.apache.kafka.common.KafkaException: Cannot execute transactional method because we are in an error state
        at org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:1042)
        at org.apache.kafka.clients.producer.internals.TransactionManager.maybeAddPartition(TransactionManager.java:386)
        at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1092)
        at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:993)
        at kafka.tools.ConsoleProducer$.send(ConsoleProducer.scala:118)
        at kafka.tools.ConsoleProducer$.loopReader(ConsoleProducer.scala:91)
        at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:100)
        at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Caused by: org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.

From Python no error is produced but inside $CONFLUENT_HOME/logs/kafka-authorizer.log I found:

[2025-06-10 16:36:17,030] INFO Principal = User:yjaquier is Denied operation = WRITE from host = 10.71.17.6 on resource = Topic:LITERAL:oracle-kafka for request = Produce with resourceRefCount = 1 based on rule DefaultDeny (kafka.authorizer.logger)

I authorize my account to write on my oracle-kafka topic:

[kafka@server01 ~]$ kafka-acls --bootstrap-server server01.domain.com:9092 --command-config ~/adminclient.conf \
>   --add \
>   --allow-principal User:yjaquier \
>   --operation write \
>   --topic oracle-kafka
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=oracle-kafka, patternType=LITERAL)`:
        (principal=User:yjaquier, host=*, operation=WRITE, permissionType=ALLOW)

[kafka@server01 ~]$ kafka-acls --bootstrap-server server01.domain.com:9092 --command-config ~/adminclient.conf --list --topic oracle-kafka
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=oracle-kafka, patternType=LITERAL)`:
        (principal=User:yjaquier, host=*, operation=WRITE, permissionType=ALLOW)
        (principal=User:yjaquier, host=*, operation=READ, permissionType=ALLOW)

And now producing and consuming messages is possible using my account…

Issues encountered

org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed

When trying to start Kafka server in SAN mode I got:

[2025-05-28 17:20:10,929] ERROR [NodeToControllerChannelManager id=1 name=heartbeat] Connection to node 1 (server01.domain.com/10.75.43.64:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2025-05-28 17:20:10,930] ERROR [broker-1-to-controller-heartbeat-channel-manager]: Failed to send the following request due to authentication error: ClientRequest(expectResponse=true, callback=kafka.server.NodeToControllerRequestThread$$Lambda$807/0x00007f194c4110d8@d20b551, destination=1, correlationId=3, clientId=1, createdTimeMs=1748445610899, requestBuilder=BrokerRegistrationRequestData(brokerId=1, clusterId='PYNUg5Q3TnqyqmT8FzEsIg', incarnationId=ISQb59yUQWiuTM7jSx25LA, listeners=[Listener(name='BROKER', host='server01.domain.com', port=9092, securityProtocol=3)], features=[Feature(name='kraft.version', minSupportedVersion=0, maxSupportedVersion=1), Feature(name='metadata.version', minSupportedVersion=1, maxSupportedVersion=21)], rack=null, isMigratingZkBroker=false, logDirs=[j3LJIuLzech7S5qEFbckyA], previousBrokerEpoch=-1)) (kafka.server.NodeToControllerRequestThread)
[2025-05-28 17:20:10,930] INFO [SocketServer listenerType=CONTROLLER, nodeId=1] Failed authentication with /10.75.43.64 (channelId=10.75.43.64:9093-10.75.43.64:46476-2) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2025-05-28 17:20:10,930] ERROR [broker-1-to-controller-heartbeat-channel-manager]: Request BrokerRegistrationRequestData(brokerId=1, clusterId='PYNUg5Q3TnqyqmT8FzEsIg', incarnationId=ISQb59yUQWiuTM7jSx25LA, listeners=[Listener(name='BROKER', host='server01.domain.com', port=9092, securityProtocol=3)], features=[Feature(name='kraft.version', minSupportedVersion=0, maxSupportedVersion=1), Feature(name='metadata.version', minSupportedVersion=1, maxSupportedVersion=21)], rack=null, isMigratingZkBroker=false, logDirs=[j3LJIuLzech7S5qEFbckyA], previousBrokerEpoch=-1) failed due to authentication error with controller. Disconnecting the connection to the stale controller 1 (kafka.server.NodeToControllerRequestThread)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: No name matching server01.domain.com found
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:383)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:444)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:533)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:382)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:302)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:563)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:501)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:596)
        at org.apache.kafka.server.util.InterBrokerSendThread.pollOnce(InterBrokerSendThread.java:110)
        at kafka.server.NodeToControllerRequestThread.doWork(NodeToControllerChannelManager.scala:377)
        at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:136)
Caused by: java.security.cert.CertificateException: No name matching server01.domain.com found
        at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:234)
        at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:467)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:418)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329)
        ... 19 more

This was because the singed certificate did not contains the SAN name of the server. The Confluent documentation is incomplete so refer to my SAN creation script above…

Error creating Kafka producer: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for ‘server01.domain.com’. (_ssl.c:1028)

For testing purpose you can set this parameter in your producer/consumer Python script:

ssl_check_hostname=False

But the source error is most probably that the server name has not correctly been added in your certificates (you have also surely use the server counter part parameter called ssl.endpoint.identification.algorithm).

Error creating Kafka producer: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1028)

You must export the root certificate with something like:

[kafka@server01 ssl]$ keytool -exportcert -alias CARoot -keystore kafka.client.truststore.jks -rfc -file CARoot.pem -storepass secure_password
Certificate stored in file caroot.pem

Then transfer this file to your client and use it in your script (Python or else):

ssl_cafile='CARoot.pem'

listenerName=ListenerName(CONTROLLER), securityProtocol=SASL_SSL, buffer=null, envelope=None) is not authorized

The complete error message is:

[2025-06-10 12:40:37,973] ERROR [ControllerApis nodeId=1] Unexpected error handling request RequestHeader(apiKey=CONTROLLER_REGISTRATION, apiVersion=0, clientId=1, correlationId=0, headerVersion=2) -- ControllerRegistrationRequestData(controllerId=1, incarnationId=hJ14_M8MRrWQp2g-yMFuiA, zkMigrationReady=false, listeners=[Listener(name='CONTROLLER', host='server01.domain.com', port=9093, securityProtocol=3)], features=[Feature(name='kraft.version', minSupportedVersion=0, maxSupportedVersion=1), Feature(name='metadata.version', minSupportedVersion=1, maxSupportedVersion=21)]) with context RequestContext(header=RequestHeader(apiKey=CONTROLLER_REGISTRATION, apiVersion=0, clientId=1, correlationId=0, headerVersion=2), connectionId='10.75.43.64:9093-10.75.43.64:38898-0', clientAddress=/10.75.43.64, principal=User:admin, listenerName=ListenerName(CONTROLLER), securityProtocol=SASL_SSL, clientInformation=ClientInformation(softwareName=apache-kafka-java, softwareVersion=7.9.1-ccs), fromPrivilegedListener=false, principalSerde=Optional[org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder@1206bdbd]) (kafka.server.ControllerApis)
org.apache.kafka.common.errors.ClusterAuthorizationException: Request Request(processor=0, connectionId=10.75.43.64:9093-10.75.43.64:38898-0, session=org.apache.kafka.network.Session@5c96d4bf, listenerName=ListenerName(CONTROLLER), securityProtocol=SASL_SSL, buffer=null, envelope=None) is not authorized.
[2025-06-10 12:40:37,979] INFO [ControllerRegistrationManager id=1 incarnation=hJ14_M8MRrWQp2g-yMFuiA] RegistrationResponseHandler: controller returned error CLUSTER_AUTHORIZATION_FAILED (Cluster authorization failed.) (kafka.server.ControllerRegistrationManager)

And found in authorizer log:

[kafka@server01 ~]$ tail $CONFLUENT_HOME/logs/kafka-authorizer.log
.
[2025-06-10 12:40:37,970] INFO Principal = User:admin is Denied operation = CLUSTER_ACTION from host = 10.75.43.64 on resource = Cluster:LITERAL:kafka-cluster for request = ControllerRegistration with resourceRefCount = 1 based on rule DefaultDeny (kafka.authorizer.logger)
.

You need at least one super account to perform the BROKER/CONTROLLER stuff so add something like in your configuration:

super.users=User:admin

References

About Post Author

Share the knowledge!

Leave a Reply

Your email address will not be published. Required fields are marked *