IT World https://blog.yannickjaquier.com RDBMS, Unix and many more... Fri, 17 Jun 2022 07:36:44 +0000 en-US hourly 1 https://wordpress.org/?v=5.9.3 Kubernetes on virtual machines hands-on – part 3 https://blog.yannickjaquier.com/linux/kubernetes-on-virtual-machines-hands-on-part-3.html https://blog.yannickjaquier.com/linux/kubernetes-on-virtual-machines-hands-on-part-3.html#respond Fri, 17 Jun 2022 07:36:44 +0000 https://blog.yannickjaquier.com/?p=5326 Preamble After the creation of our first Kubernetes Nginx stateless pod let see why stateless pod is an issue (not only) for database pod. In this third article I will focus on why you need stateful container for certain types of workload. As it has kept me busy for some time we will also see […]

The post Kubernetes on virtual machines hands-on – part 3 appeared first on IT World.

]]>

Table of contents

Preamble

After the creation of our first Kubernetes Nginx stateless pod let see why stateless pod is an issue (not only) for database pod. In this third article I will focus on why you need stateful container for certain types of workload.

As it has kept me busy for some time we will also see how to create the share storage.

In next blog post we will see which Kubernetes storage plugin to use to access it.

PostgreSQL stateless deployment creation

I will obviously use the official PostgreSQL image that can be find on Docker Hub:

kubernetes07
kubernetes07

For this new deployment I have decided to create my own YAML file that I would be able to use over and over to add new functionalities. I have started by the one of my Nginx deployment as a skeleton:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - image: postgres:latest
        name: postgres
        ports:
        - containerPort: 5433
      restartPolicy: Always
      schedulerName: default-scheduler

Let’s load this new deployment:

[root@server1 ~]# kubectl apply -f postgres.yaml
deployment.apps/postgres created
[root@server1 ~]# kubectl get deployment
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
httpd      1/1     1            1           22h
nginx      1/1     1            1           6d23h
postgres   0/1     1            0           16s

After a while my deployement failed:

[root@server1 ~]# kubectl get pod
NAME                        READY   STATUS              RESTARTS   AGE
httpd-757fb56c8d-7cdj5      1/1     Running             0          22h
nginx-6799fc88d8-xg5kd      1/1     Running             0          23h
postgres-74b5d46bcb-tvv8v   0/1     ContainerCreating   0          59s
[root@server1 ~]# kubectl get pod
NAME                        READY   STATUS    RESTARTS      AGE
httpd-757fb56c8d-7cdj5      1/1     Running   0             22h
nginx-6799fc88d8-xg5kd      1/1     Running   0             23h
postgres-74b5d46bcb-tvv8v   0/1     Error     2 (18s ago)   82s

Get the log with:

[root@server1 ~]# kubectl logs postgres-74b5d46bcb-tvv8v
Error: Database is uninitialized and superuser password is not specified.
       You must specify POSTGRES_PASSWORD to a non-empty value for the
       superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".

       You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
       connections without a password. This is *not* recommended.

       See PostgreSQL documentation about "trust":
       https://www.postgresql.org/docs/current/auth-trust.html

So added to my YAML file:

env:
- name: POSTGRES_PASSWORD
  value: secure_password
- name: POSTGRES_DB
  value: testdb

The options I have changed are the postgres superuser password with POSTGRES_PASSWORD and I have asked the creation of a default database called testdb with POSTGRES_DB. The Docker Hub PostgresSQL default page gives a clear explanation and a list of all possible environment variables.

Delete and re-deploy:

[root@server1 ~]# kubectl delete deployment postgres
deployment.apps "postgres" deleted
[root@server1 ~]# kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
httpd   1/1     1            1           22h
nginx   1/1     1            1           6d23h
[root@server1 ~]# kubectl apply -f postgres.yaml
deployment.apps/postgres created
[root@server1 ~]# kubectl get pod
NAME                        READY   STATUS    RESTARTS   AGE
httpd-757fb56c8d-7cdj5      1/1     Running   0          22h
nginx-6799fc88d8-xg5kd      1/1     Running   0          23h
postgres-6d7fcf96b5-gfpxf   1/1     Running   0          5s

You can now connect to the PostgreSQL database inside the pod with (we see that the port is not 5433 as expected but still the default 5432):

[root@server1 ~]# kubectl exec -it postgres-6d7fcf96b5-gfpxf -- /bin/bash
root@postgres-6d7fcf96b5-gfpxf:/# su - postgres
postgres@postgres-6d7fcf96b5-gfpxf:~$ psql
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.

postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 testdb    | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
(4 rows)

postgres=#

Or directly from the server where is running the pod after I got the IP address of the pod:

[root@server2 ~]# psql --host=192.168.55.19 --port=5432 --username=postgres
Password for user postgres:
psql (13.4, server 14.0 (Debian 14.0-1.pgdg110+1))
WARNING: psql major version 13, server major version 14.
         Some psql features might not work.
Type "help" for help.

postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 testdb    | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
(4 rows)

postgres=#

It finally appears that the – containerPort: 5433 is not providing the expected result. The solution to change default port (5432) is:

args: ["-c", "port=5433"]

The stateless issue with database containers

Let’s create a new table in my tesdb database and insert a new row into it:

[root@server1 ~]# kubectl exec -it postgres-5594494b8f-2wsvh -- /bin/bash
root@postgres-5594494b8f-2wsvh:/# su - postgres
postgres@postgres-5594494b8f-2wsvh:~$ psql --port=5433
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.

postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 testdb    | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
(4 rows)

postgres=# \c testdb
You are now connected to database "testdb" as user "postgres".
testdb=# create table test01(id integer, descr varchar(20));
CREATE TABLE
testdb=# insert into test01 values(1,'One');
INSERT 0 1
testdb=# select * from test01;
 id | descr
----+-------
  1 | One
(1 row)

testdb=#

Now I delete the pod, like it would happen in a real life k8s cluster. The pod is automatically recreated by the deployment and I then try to get my test table figures:

[root@server1 ~]# kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP              NODE                 NOMINATED NODE   READINESS GATES
httpd-757fb56c8d-7cdj5      1/1     Running   0          23h   192.168.55.17   server2.domain.com              
nginx-6799fc88d8-xg5kd      1/1     Running   0          24h   192.168.55.16   server2.domain.com              
postgres-5594494b8f-2wsvh   1/1     Running   0          17m   192.168.55.21   server2.domain.com              
[root@server1 ~]# kubectl delete pod postgres-5594494b8f-2wsvh
pod "postgres-5594494b8f-2wsvh" deleted
[root@server1 ~]# kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP              NODE                 NOMINATED NODE   READINESS GATES
httpd-757fb56c8d-7cdj5      1/1     Running   0          23h   192.168.55.17   server2.domain.com              
nginx-6799fc88d8-xg5kd      1/1     Running   0          24h   192.168.55.16   server2.domain.com              
postgres-5594494b8f-p88h9   1/1     Running   0          5s    192.168.55.22   server2.domain.com              
[root@server1 ~]# kubectl exec -it postgres-5594494b8f-p88h9 -- /bin/bash
root@postgres-5594494b8f-p88h9:/# su - postgres
postgres@postgres-5594494b8f-p88h9:~$ psql --port=5433 --dbname=testdb
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.

testdb=# select * from test01;
ERROR:  relation "test01" does not exist
LINE 1: select * from test01;
                      ^
testdb=#

Oups, well as expected I would say, the information has gone and again in the case of a database this is clearly not acceptable. To make persistent the content we need to work a little more with persistent volume and persistent volume claim.

Creation of the cluster filesystem between Kubernetes nodes

Kubernetes has a lot of available persistent volumes plugins to allow you to mount an incredible number of different storage types on your k8s nodes. On my trial k8s cluster made of two virtual machines I have decided to use a shared disk same as we have already seen in my Oracle Real Application Cluster (RAC) configuration trial. Once this shared storage and cluster filesystem will be created the idea is to use the local Kubernetes storage plugin.

Once I have attached the shared disk let’s create a new partition:

[root@server1 ~]# fdisk /dev/sdb

Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xde5a83e7.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p):

Using default response p.
Partition number (1-4, default 1):
First sector (2048-2097151, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-2097151, default 2097151):

Created a new partition 1 of type 'Linux' and of size 1023 MiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

[root@server1 ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xde5a83e7

Device     Boot Start     End Sectors  Size Id Type
/dev/sdb1        2048 2097151 2095104 1023M 83 Linux
[root@server1 ~]# mkfs -t xfs /dev/sdb1
meta-data=/dev/sdb1              isize=512    agcount=4, agsize=65472 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=261888, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=1566, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

On second node to make the shared partition available I had to trick a bit fdisk:

[root@server2 ~]# blkid /dev/sdb1
[root@server2 ~]# fdisk /dev/sdb

Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Then as I will not have Oracle ASM you have to decide to use a shared filesystem…

GFS2

This cluster FS is sponsored by RedHat and installation is simple as:

[root@server1 ~]# dnf install gfs2-utils.x86_64

Then create the FS with:

[root@server1 ~]# mkfs -t gfs2 -p lock_dlm -t cluster01:postgres -j 8 /dev/sdb1
It appears to contain an existing filesystem (xfs)
This will destroy any data on /dev/sdb1
Are you sure you want to proceed? [y/n] y
Discarding device contents (may take a while on large devices): Done
Adding journals: Done
Building resource groups: Done
Creating quota file: Done
Writing superblock and syncing: Done
Device:                    /dev/sdb1
Block size:                4096
Device size:               1.00 GB (261888 blocks)
Filesystem size:           1.00 GB (261886 blocks)
Journals:                  8
Journal size:              8MB
Resource groups:           12
Locking protocol:          "lock_dlm"
Lock table:                "cluster01:postgres"
UUID:                      90f456e8-cf74-43af-a838-53b129682f7d

But when I tried to mount it I got:

[root@server1 ~]# mount -a
mount: /mnt/shared: mount(2) system call failed: Transport endpoint is not connected.

From the RedHat official solution I have discovered that lock_dlm modules was not loaded:

[root@server1 ~]# lsmod |grep lock
[root@server1 ~]# modprobe lock_dlm
modprobe: FATAL: Module lock_dlm not found in directory /lib/modules/4.18.0-305.19.1.el8_4.x86_64
[root@server1 ~]# dmesg | grep gfs
[ 8255.885092] gfs2: GFS2 installed
[ 8255.896751] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[ 8255.899671] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[ 8291.542025] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[ 8291.542186] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[ 8376.146357] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[ 8376.156197] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[ 8442.132982] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[ 8442.137871] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[12479.923651] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[12479.924713] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[12861.644565] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[12861.644663] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[13016.278584] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[13016.279004] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[13042.852965] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[13042.866282] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[13362.619425] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[13362.631850] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107

I have tried to install kernel extra modules:

[root@server1 ~]# dnf install kernel-modules-extra.x86_64

To finally realized that GFS2 is delivered as extra cost add-ons in Red Hat Enterprise Linux, such as the High Availability Add-On for clustering and the Resilient Storage Add-On for GFS2. I thought the FS was free but apparently GFS was free but GFS2 is not…

OCFS2

As I’m using the Oracle Linux OCFS2 sounds like a good idea and for sure OCFS2 is released under the GNU General Public License. Install it with:

[root@server1 ~]# dnf install ocfs2-tools.x86_64

On one of your node create the OCFS2 cluster with:

[root@server1 ~]# o2cb add-cluster k8socfs2
[root@server1 ~]# o2cb add-node --ip 192.168.56.101 --port 7777 --number 1 k8socfs2 server1.domain.com
[root@server1 ~]# o2cb add-node --ip 192.168.56.102 --port 7777 --number 2 k8socfs2 server2.domain.com
[root@server1 ~]# o2cb register-cluster k8socfs2
[root@server1 ~]# o2cb start-heartbeat k8socfs2
[root@server1 ~]# cat /etc/ocfs2/cluster.conf
cluster:
        heartbeat_mode = local
        node_count = 2
        name = k8socfs2

node:
        number = 1
        cluster = k8socfs2
        ip_port = 7777
        ip_address = 192.168.56.101
        name = server1.domain.com

node:
        number = 2
        cluster = k8socfs2
        ip_port = 7777
        ip_address = 192.168.56.102
        name = server2.domain.com

Copy this configuration file (/etc/ocfs2/cluster.conf) on all nodes of your OCFS2 cluster.

Initialize the OCFS2 cluster stack (O2CB) with:

[root@server1 ~]# o2cb.init --help
Usage: /usr/sbin/o2cb.init {start|stop|restart|force-reload|enable|disable|configure|load|unload|online|offline|force-offline|status|online-status}
[root@server1 /]# o2cb.init configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot.  The current values will be shown in brackets ('[]').  Hitting
 without typing an answer will keep that current value.  Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [n]: y
Cluster stack backing O2CB [o2cb]:
Cluster to start on boot (Enter "none" to clear) [ocfs2]: k8socfs2
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
checking debugfs...
Loading filesystem "ocfs2_dlmfs": Unable to load filesystem "ocfs2_dlmfs"
Failed
[root@server1 /]#
[root@server1 /]# lsmod |egrep -i "ocfs|o2"
[root@server1 /]# modprobe ocfs2_dlmfs
modprobe: FATAL: Module ocfs2_dlmfs not found in directory /lib/modules/4.18.0-305.19.1.el8_4.x86_64
[root@server1 /]# o2cb.init status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Driver for "ocfs2_dlmfs": Not loaded
Checking O2CB cluster "ociocfs2": Offline
stat: cannot read file system information for '/dlm': No such file or directory
Debug file system at /sys/kernel/debug: mounted

I realized that all issues I had (including mount.ocfs2: Unable to access cluster service while trying initialize cluster) were linked to the kernel I was using. Not the UEK Oracle kernel. All issues have been resolved at the moment I switch to Oracle UEK kernel !

Do not forget to start and enable o2cb service with:

[root@server2 ~]# systemctl status o2cb
● o2cb.service - Load o2cb Modules
   Loaded: loaded (/usr/lib/systemd/system/o2cb.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
[root@server2 ~]# systemctl start o2cb
[root@server2 ~]# systemctl status o2cb
● o2cb.service - Load o2cb Modules
   Loaded: loaded (/usr/lib/systemd/system/o2cb.service; disabled; vendor preset: disabled)
   Active: active (exited) since Mon 2021-10-18 14:04:15 CEST; 1s ago
  Process: 73099 ExecStart=/sbin/o2cb.init enable (code=exited, status=0/SUCCESS)
 Main PID: 73099 (code=exited, status=0/SUCCESS)

Oct 18 14:04:14 server2.domain.com systemd[1]: Starting Load o2cb Modules...
Oct 18 14:04:15 server2.domain.com o2cb.init[73099]: checking debugfs...
Oct 18 14:04:15 server2.domain.com o2cb.init[73099]: Setting cluster stack "o2cb": OK
Oct 18 14:04:15 server2.domain.com o2cb.init[73099]: Cluster ociocfs2 already online
Oct 18 14:04:15 server2.domain.com systemd[1]: Started Load o2cb Modules.
[root@server2 ~]# systemctl enable o2cb
Created symlink /etc/systemd/system/multi-user.target.wants/o2cb.service → /usr/lib/systemd/system/o2cb.service.

Create the FS with:

[root@server1 ~]# man mkfs.ocfs2
[root@server1 ~]# mkfs -t ocfs2 --cluster-name=k8socfs2 --fs-feature-level=max-features --cluster-stack=o2cb -N 4 /dev/sdb1
mkfs.ocfs2 1.8.6
Cluster stack: o2cb
Cluster name: k8socfs2
Stack Flags: 0x0
NOTE: Feature extended slot map may be enabled
Overwriting existing ocfs2 partition.
Proceed (y/N): y
Label:
Features: sparse extended-slotmap backup-super unwritten inline-data strict-journal-super metaecc xattr indexed-dirs usrquota grpquota refcount discontig-bg
Block size: 2048 (11 bits)
Cluster size: 4096 (12 bits)
Volume size: 1072693248 (261888 clusters) (523776 blocks)
Cluster groups: 17 (tail covers 7936 clusters, rest cover 15872 clusters)
Extent allocator size: 4194304 (1 groups)
Journal size: 33554432
Node slots: 4
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 0 block(s)
Formatting Journals: done
Growing extent allocator: done
Formatting slot map: done
Formatting quota files: done
Writing lost+found: done
mkfs.ocfs2 successful

Get the id of the device with:

[root@server1 ~]# blkid /dev/sdb1
/dev/sdb1: UUID="ea6e9804-105d-4d4c-96e8-bd54ab5e93d2" BLOCK_SIZE="2048" TYPE="ocfs2" PARTUUID="de5a83e7-01"
[root@server1 ~]# echo "ea6e9804-105d-4d4c-96e8-bd54ab5e93d2" >> /etc/fstab
[root@server1 ~]# vi /etc/fstab
[root@server1 ~]# tail -n 2 /etc/fstab
# Shared storage
UUID="ea6e9804-105d-4d4c-96e8-bd54ab5e93d2"     /mnt/shared      ocfs2    defaults     0 0
[root@server1 ~]# mount -a

With Oracle UEK kernel I now have:

[root@server3 postgres]# o2cb.init status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster "ociocfs2": Online
  Heartbeat dead threshold: 31
  Network idle timeout: 30000
  Network keepalive delay: 2000
  Network reconnect delay: 2000
  Heartbeat mode: Local
Checking O2CB heartbeat: Active
Debug file system at /sys/kernel/debug: mounted
[root@server3 ~]# df /mnt/shared
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/sdb1        1047552 78956    968596   8% /mnt/shared

And I can share file from all nodes of my k8s cluster…

I had an issue where the FS was automatically unmounted by the system:

Oct 18 15:10:50 server1 kernel: o2dlm: Joining domain E80332E942C649EB942623C43D2B35DC
Oct 18 15:10:50 server1 kernel: (
Oct 18 15:10:50 server1 kernel: 1
Oct 18 15:10:50 server1 kernel: ) 1 nodes
Oct 18 15:10:50 server1 kernel: ocfs2: Mounting device (8,17) on (node 1, slot 0) with ordered data mode.
Oct 18 15:10:50 server1 systemd[1]: mnt-postgres.mount: Unit is bound to inactive unit dev-disk-by\x2duuid-b7f61498\x2da4f0\x2d4570\x2da0ed\x2dcb50caa98165.device. Stopping, too.
Oct 18 15:10:50 server1 systemd[1]: Unmounting /mnt/shared...
Oct 18 15:10:50 server1 systemd[10949]: mnt-postgres.mount: Succeeded.

Solved with:

[root@server1 /]# systemctl daemon-reload

I also had an issue where o2cb was not able to register my cluster name:

[root@server2 ~]# o2cb register-cluster k8socfs2
o2cb: Internal logic failure while registering cluster 'k8socfs2'

Old trials not completely removed and solved it with:

[root@server2 /]# o2cb cluster-status
Cluster 'ociocfs2' is online
[root@server2 /]# o2cb unregister-cluster ociocfs2

References

The post Kubernetes on virtual machines hands-on – part 3 appeared first on IT World.

]]>
https://blog.yannickjaquier.com/linux/kubernetes-on-virtual-machines-hands-on-part-3.html/feed 0
Kubernetes on virtual machines hands-on – part 2 https://blog.yannickjaquier.com/linux/kubernetes-on-virtual-machines-hands-on-part-2.html https://blog.yannickjaquier.com/linux/kubernetes-on-virtual-machines-hands-on-part-2.html#comments Sat, 28 May 2022 07:45:05 +0000 https://blog.yannickjaquier.com/?p=5278 Preamble After a first part where we have setup the Kubernetes cluster and added a second node, this second part will be about pods creation with direct download of the image and creation of YAML file to use them later to deploy and, more importantly, modify existing pod. This second part will mainly focus on […]

The post Kubernetes on virtual machines hands-on – part 2 appeared first on IT World.

]]>

Table of contents

Preamble

After a first part where we have setup the Kubernetes cluster and added a second node, this second part will be about pods creation with direct download of the image and creation of YAML file to use them later to deploy and, more importantly, modify existing pod.

This second part will mainly focus on setup of useful tools to handle YAML files and creation of stateless pods.

Stateless Nginx web server with a deployment

The suggested stateless application deployment to test your newly installed Kubernetes cluster is a simple Nginx web server though a Kubernetes deployment. Create it simply with below command and wait a few minutes to have it ready (number of replicas is 1 by default):

[root@server1 ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@server1 ~]# kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   0/1     1            0           9s
[root@server1 ~]# kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/1     1            1           104s

Confirm the pod is running with one replicaset. By default the running node of my Kubernetes cluster is OT the control node (so server2.domain.com in my case):

[root@server1 ~]# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP             NODE                 NOMINATED NODE   READINESS GATES
nginx-6799fc88d8-zn2dm   1/1     Running   0          40s   192.168.55.2   server2.domain.com              
[root@server1 ~]# kubectl get replicaset
NAME               DESIRED   CURRENT   READY   AGE
nginx-6799fc88d8   1         1         1       6d22h

Expose the deployment as a service on port 80 and get the port on the k8s cluster of your deployment with:

[root@server1 ~]# kubectl expose deployment nginx --type=NodePort --port=80
service/nginx exposed
[root@server1 ~]# kubectl get service nginx
NAME    TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
nginx   NodePort   10.105.114.178           80:32609/TCP   177m

Then you can control it works with your web browser (accessing my worker IP address i.e. server2.domain.com on port 32609):

kubernetes04
kubernetes04

To get more information on your newly created service:

[root@server1 ~]# kubectl describe service nginx
Name:                     nginx
Namespace:                default
Labels:                   app=nginx
Annotations:              
Selector:                 app=nginx
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.105.114.178
IPs:                      10.105.114.178
Port:                       80/TCP
TargetPort:               80/TCP
NodePort:                   32609/TCP
Endpoints:                192.168.55.2:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   
[root@server1 ~]# kubectl get ep nginx
NAME    ENDPOINTS         AGE
nginx   192.168.55.2:80   46m

You can also curl inside the container with exec command using the cluster IP address with something like:

[root@server1 ~]# kubectl exec nginx-6799fc88d8-zn2dm -- curl -s http://10.105.114.178
.
.
.

Accessing a container from outside the Kubernetes cluster

Even if my network expertise is ridiculous I really wanted to access the pod with its own IP address i.e. 192.168.55.2. For this I have started to modify the routing table on my virtual machines host that is my Windows 10 desktop:

PS C:\WINDOWS\system32> route print
===========================================================================
Interface List
  6...0a 00 27 00 00 06 ......VirtualBox Host-Only Ethernet Adapter
  5...48 0f cf 33 0a 07 ......Intel(R) Ethernet Connection (2) I218-LM
  1...........................Software Loopback Interface 1
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0    10.70.101.254    10.70.101.129     35
      10.70.101.0    255.255.255.0         On-link     10.70.101.129    291
    10.70.101.129  255.255.255.255         On-link     10.70.101.129    291
    10.70.101.255  255.255.255.255         On-link     10.70.101.129    291
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    331
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    331
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    331
     192.168.56.0    255.255.255.0         On-link      192.168.56.1    281
     192.168.56.1  255.255.255.255         On-link      192.168.56.1    281
   192.168.56.255  255.255.255.255         On-link      192.168.56.1    281
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    331
        224.0.0.0        240.0.0.0         On-link     10.70.101.129    291
        224.0.0.0        240.0.0.0         On-link      192.168.56.1    281
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    331
  255.255.255.255  255.255.255.255         On-link     10.70.101.129    291
  255.255.255.255  255.255.255.255         On-link      192.168.56.1    281
===========================================================================
Persistent Routes:
  None

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    331 ::1/128                  On-link
  1    331 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

With initial configuration any TRACERT.EXE 192.168.55.18 or TRACERT.EXE 192.168.55.1 would end up with something not answering…

So (as administrator), inspired from 192.168.56.0/24 subnet configured by VirtualBox, I have issued the two below commands (that will give a route to 192.168.55.0/24 subnet). I also had to specify the interface to use the VirtualBox one (if option):

PS C:\WINDOWS\system32> route add 192.168.55.0 mask 255.255.255.0 192.168.55.1 if 6
 OK!
PS C:\WINDOWS\system32> route add 192.168.55.1 mask 255.255.255.255 192.168.56.1 if 6
 OK!
PS C:\WINDOWS\system32> route add 192.168.55.255 mask 255.255.255.255 192.168.56.1 if 6
 OK!

To remove what you added you can use:

PS C:\WINDOWS\system32> route delete 192.168.55.0
 OK!
PS C:\WINDOWS\system32> route delete 192.168.55.1
 OK!
PS C:\WINDOWS\system32> route delete 192.168.55.255
 OK!

To end up with this routing table:

PS C:\Users\yjaquier> route print
===========================================================================
Interface List
  6...0a 00 27 00 00 06 ......VirtualBox Host-Only Ethernet Adapter
  5...48 0f cf 33 0a 07 ......Intel(R) Ethernet Connection (2) I218-LM
  1...........................Software Loopback Interface 1
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0    10.70.101.254    10.70.101.129     35
      10.70.101.0    255.255.255.0         On-link     10.70.101.129    291
    10.70.101.129  255.255.255.255         On-link     10.70.101.129    291
    10.70.101.255  255.255.255.255         On-link     10.70.101.129    291
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    331
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    331
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    331
     192.168.55.0    255.255.255.0     192.168.55.1     192.168.56.1     26
     192.168.55.1  255.255.255.255         On-link      192.168.56.1     26
   192.168.55.255  255.255.255.255         On-link      192.168.56.1     26
     192.168.56.0    255.255.255.0         On-link      192.168.56.1    281
     192.168.56.1  255.255.255.255         On-link      192.168.56.1    281
   192.168.56.255  255.255.255.255         On-link      192.168.56.1    281
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    331
        224.0.0.0        240.0.0.0         On-link      192.168.56.1    281
        224.0.0.0        240.0.0.0         On-link     10.70.101.129    291
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    331
  255.255.255.255  255.255.255.255         On-link      192.168.56.1    281
  255.255.255.255  255.255.255.255         On-link     10.70.101.129    291
===========================================================================
Persistent Routes:
  None

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    331 ::1/128                  On-link
  1    331 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

This has allowed access to Nginx web server fetching 192.168.55.2 from my desktop. I can also ping the IP address directly from my desktop (so outside of the cluster). I can access the nginx server fron the server (server2.domain.com) running the pod but not from my controller node (server1.domain.com):

[root@server1 ~]# ping -c 1 192.168.55.2
PING 192.168.55.2 (192.168.55.2) 56(84) bytes of data.
From 192.168.55.1 icmp_seq=1 Destination Host Unreachable

--- 192.168.55.2 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
[root@server1 ~]# curl http://192.168.55.2
curl: (7) Failed to connect to 192.168.55.2 port 80: No route to host
[root@server2 ~]# ping -c 1 192.168.55.2
PING 192.168.55.2 (192.168.55.2) 56(84) bytes of data.
64 bytes from 192.168.55.2: icmp_seq=1 ttl=64 time=0.079 ms

--- 192.168.55.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.079/0.079/0.079/0.000 ms
[root@server2 ~]# curl 192.168.55.2
.
.
.
kubernetes05
kubernetes05

One option to access from outside the cluster is to enable port forwarding with kubectl port-forward command, so using something like:

[root@server1 ~]# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP             NODE                 NOMINATED NODE   READINESS GATES
nginx-6799fc88d8-zn2dm   1/1     Running   0          3h36m   192.168.55.2   server2.domain.com              
[root@server1 ~]# kubectl port-forward pod/nginx-6799fc88d8-zn2dm 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
[root@server1 ~]# kubectl port-forward pod/nginx-6799fc88d8-zn2dm :80
Forwarding from 127.0.0.1:40859 -> 80
Forwarding from [::1]:40859 -> 80
[root@server1 ~]# kubectl port-forward --address 0.0.0.0 pod/nginx-6799fc88d8-zn2dm 8080:80
Forwarding from 0.0.0.0:8080 -> 80

With last command on any IP address on controller node (server1.domain.com) and port 8080 I can access to my Nginx server:

kubernetes06
kubernetes06

Overall this part is not fully clear and I really need to progress on this Kubernetes area…

How to scale a pod with ReplicaSet

For example if I scale my Nginx application to 3 pods:

[root@server1 ~]# kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/1     1            1           3d5h
[root@server1 ~]# kubectl scale --replicas=3 deployment/nginx
deployment.apps/nginx scaled
[root@server1 ~]# kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/3     3            1           3d5h
[root@server1 ~]# kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   3/3     3            3           3d5h
[root@server1 ~]# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS        AGE    IP             NODE                 NOMINATED NODE   READINESS GATES
nginx-6799fc88d8-g8v2v   1/1     Running   0               20s    192.168.55.4   server2.domain.com              
nginx-6799fc88d8-zhhsn   1/1     Running   0               20s    192.168.55.5   server2.domain.com              
nginx-6799fc88d8-zn2dm   1/1     Running   1 (2d23h ago)   3d5h   192.168.55.3   server2.domain.com              

If I delete (or kill) one pod then automatically a new one is created:

[root@server1 ~]# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS        AGE     IP             NODE                 NOMINATED NODE   READINESS GATES
nginx-6799fc88d8-g8v2v   1/1     Running   0               6m16s   192.168.55.4   server2.domain.com              
nginx-6799fc88d8-zhhsn   1/1     Running   0               6m16s   192.168.55.5   server2.domain.com              
nginx-6799fc88d8-zn2dm   1/1     Running   1 (2d23h ago)   3d5h    192.168.55.3   server2.domain.com              
[root@server1 ~]# kubectl delete pod nginx-6799fc88d8-zhhsn
pod "nginx-6799fc88d8-zhhsn" deleted
[root@server1 ~]# kubectl get pod -o wide
NAME                     READY   STATUS    RESTARTS        AGE     IP             NODE                 NOMINATED NODE   READINESS GATES
nginx-6799fc88d8-9pbcf   1/1     Running   0               7s      192.168.55.6   server2.domain.com              
nginx-6799fc88d8-g8v2v   1/1     Running   0               6m40s   192.168.55.4   server2.domain.com              
nginx-6799fc88d8-zn2dm   1/1     Running   1 (2d23h ago)   3d5h    192.168.55.3   server2.domain.com              

To see if my pod would also go on my controller node I have authorized pod creation on it because by default it’s forbidden:

[root@server1 ~]# kubectl taint nodes --all node-role.kubernetes.io/master-
node/server1.domain.com untainted
error: taint "node-role.kubernetes.io/master" not found

To be honest none of my pods went to my master node and if you dig a bit on Internet you will see that the pod allocation on nodes is a recurreing issue for poeple. To come back to original situation simply do:

[root@server1 ~]# kubectl taint nodes server1.domain.com node-role.kubernetes.io/master=:NoSchedule
node/server1.domain.com tainted

Move pods from one node to another

Once I have added this additional node I wanted to move out from my control node the Nginx pod we have created above. I thought it would be a simple command but I was highly wrong… Currently it is not possible to move on-the-fly pods from one node to another. The only available option is to re-schedule the pod and use node affinity, with labels, to force a pod running on a node.

I’m not yet at this level but I have seen plenty of blog posts of people complaining that in their cluster some nodes are highly used while some others are almost idle and they have no option to solve the situation…

How to sanitize your deployment YAML files

To create a deployment or a pod or to modify existing resources you often create a YAML file of an existing resource. You can also create this YAML file directly from scratch and the official k8s documentation is full of example. One issue I have immediately see is the verbosity of the YAML file generated from an existing resource with a command like:

[root@server1 ~]# kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
httpd   1/1     1            1           21h
nginx   1/1     1            1           6d22h
[root@server1 ~]# kubectl get deployment httpd -o yaml

When you have done it you realize that you need a tool to sanitize those generated YAML file because they are really far from lean file we see in official documentation. One tool that often came in discussions is kubectl-neat. To implement it first start by installing krew with (git is a prerequisite):

[root@server1 ~]# (
>   set -x; cd "$(mktemp -d)" &&
>   OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
>   ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
>   KREW="krew-${OS}_${ARCH}" &&
>   curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
>   tar zxvf "${KREW}.tar.gz" &&
>   ./"${KREW}" install krew
> )
++ mktemp -d
+ cd /tmp/tmp.mjm5SmGWMR
++ uname
++ tr '[:upper:]' '[:lower:]'
+ OS=linux
++ uname -m
++ sed -e s/x86_64/amd64/ -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/'
+ ARCH=amd64
+ KREW=krew-linux_amd64
+ curl -fsSLO https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-linux_amd64.tar.gz
+ tar zxvf krew-linux_amd64.tar.gz
./LICENSE
./krew-linux_amd64
+ ./krew-linux_amd64 install krew
Adding "default" plugin index from https://github.com/kubernetes-sigs/krew-index.git.
Updated the local copy of plugin index.
Installing plugin: krew
Installed plugin: krew
\
 | Use this plugin:
 |      kubectl krew
 | Documentation:
 |      https://krew.sigs.k8s.io/
 | Caveats:
 | \
 |  | krew is now installed! To start using kubectl plugins, you need to add
 |  | krew's installation directory to your PATH:
 |  |
 |  |   * macOS/Linux:
 |  |     - Add the following to your ~/.bashrc or ~/.zshrc:
 |  |         export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
 |  |     - Restart your shell.
 |  |
 |  |   * Windows: Add %USERPROFILE%\.krew\bin to your PATH environment variable
 |  |
 |  | To list krew commands and to get help, run:
 |  |   $ kubectl krew
 |  | For a full list of available plugins, run:
 |  |   $ kubectl krew search
 |  |
 |  | You can find documentation at
 |  |   https://krew.sigs.k8s.io/docs/user-guide/quickstart/.
 | /
/

Control it works with:

[root@server1 ~]# export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"
[root@server1 ~]# kubectl krew
krew is the kubectl plugin manager.
You can invoke krew through kubectl: "kubectl krew [command]..."

Usage:
  kubectl krew [command]

Available Commands:
  completion  generate the autocompletion script for the specified shell
  help        Help about any command
  index       Manage custom plugin indexes
  info        Show information about an available plugin
  install     Install kubectl plugins
  list        List installed kubectl plugins
  search      Discover kubectl plugins
  uninstall   Uninstall plugins
  update      Update the local copy of the plugin index
  upgrade     Upgrade installed plugins to newer versions
  version     Show krew version and diagnostics

Flags:
  -h, --help      help for krew
  -v, --v Level   number for the log level verbosity

Use "kubectl krew [command] --help" for more information about a command.

Install kubectl-neat with:

[root@server1 ~]# kubectl krew install neat
Updated the local copy of plugin index.
Installing plugin: neat
Installed plugin: neat
\
 | Use this plugin:
 |      kubectl neat
 | Documentation:
 |      https://github.com/itaysk/kubectl-neat
/
WARNING: You installed plugin "neat" from the krew-index plugin repository.
   These plugins are not audited for security by the Krew maintainers.
   Run them at your own risk.

Then after if you would like to get a skeleton of a pod to create a similar one or simply extract a clean yaml pod file from a running pod to modify it you would do something like:

[root@server1 ~]# kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
httpd-757fb56c8d-7cdj5   1/1     Running   0          20h
nginx-6799fc88d8-xg5kd   1/1     Running   0          21h
httpd-757fb56c8d-7cdj5   1/1     Running   0          20h
[root@server1 ~]# kubectl get pod httpd-757fb56c8d-7cdj5 -o yaml

The problem is that this extracted yaml file contains a lot extra information, to remove all this redundant extra information use kubectl-neat with something like:

[root@server1 ~]# kubectl get pod httpd-757fb56c8d-7cdj5 -o yaml | kubectl neat
[root@server1 ~]# kubectl neat get pod httpd-757fb56c8d-7cdj5 -o yaml

Useful commands

Access a pod:

[root@server1 ~]# kubectl exec -it nginx-6799fc88d8-tdh4p -- /bin/bash
root@nginx-6799fc88d8-tdh4p:/# ls -l /usr/share/nginx/html
total 8
-rw-r--r-- 1 root root 494 Jul  6 14:59 50x.html
-rw-r--r-- 1 root root 612 Jul  6 14:59 index.html

Delete a deployment:

[root@server1 ~]# kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/1     1            1           3d5h
[root@server1 ~]# kubectl delete deployment nginx
deployment.apps "nginx" deleted

Print the supported API resources on the server:

[root@server1 ~]# kubectl api-resources

References

The post Kubernetes on virtual machines hands-on – part 2 appeared first on IT World.

]]>
https://blog.yannickjaquier.com/linux/kubernetes-on-virtual-machines-hands-on-part-2.html/feed 1
Kubernetes on virtual machines hands-on – part 1 https://blog.yannickjaquier.com/linux/kubernetes-on-virtual-machines-hands-on-part-1.html https://blog.yannickjaquier.com/linux/kubernetes-on-virtual-machines-hands-on-part-1.html#comments Thu, 28 Apr 2022 06:55:50 +0000 https://blog.yannickjaquier.com/?p=5267 Preamble Before describing Kubernetes (K8s for short) let’s briefly see what is container technology. Container is similar to virtualization technology, such a VMWare, but they are lightweight as the underlining operating system is shared amongst containers. Same as VM resource allocation (filesystem, CPU, memory, ..) is controlled to avoid having one container eating all resources […]

The post Kubernetes on virtual machines hands-on – part 1 appeared first on IT World.

]]>

Table of contents

Preamble

Before describing Kubernetes (K8s for short) let’s briefly see what is container technology. Container is similar to virtualization technology, such a VMWare, but they are lightweight as the underlining operating system is shared amongst containers. Same as VM resource allocation (filesystem, CPU, memory, ..) is controlled to avoid having one container eating all resources and impacting the others. Containers are also portable across OS and different Cloud providers.

Container technology is not something new and I have personally started to hear a lot about container with the rise of Docker. Even if nowadays you have plenty of products that compete with Docker: Podman, containerd and CRI-O to name a few. Even if you have not created containers on your own you might have tested an application that was containerized and if you had the underlining infrastructure you have experienced how easy it is to deploy a container and to use the application almost immediately without the burden of configuration or so.

Once you have all those applicative containers running how do you manage them ? Kubernetes of course ! Kubernetes is a open source platform to manage containerized workloads and services. Example of tasks handle by Kubernetes are scaling, managing downtime and much more…

My ultimate goal, as you might guess, is to create container running a database (I have already tested SQL Server in container) and to create what we call stateful container. It means that the container has persistent storage, yes of course you do not want to lose the content of your database in case of a container crash. This fist article will focus only on stateless container that is typically a web server where you do not mind of loosing the content…

For my testing I have used two virtual machines (with VirtualBox) running Oracle Linux Server release 8.4. Kubernetes version is 1.22.2 and Docker is 20.10.9.

This first part will be “just” about creating the cluster with the main first master node as well as adding a second worker node to handle workload…

Kubernetes installation

I have used a virtual machine under VirtualBox running Oracle Linux 8.4. One important configuration is to activate virtualization feature and nested virtualization feature (not available since long on VirtualBox) with:

kubernetes01
kubernetes01

You can confirm nested virtualization feature is active with:

[root@server1 ~]# grep -E --color 'vmx|svm' /proc/cpuinfo
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm invpcid_single pti tpr_shadow vnmi flexpriority vpid fsgsbase avx2 invpcid md_clear flush_l1d
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm invpcid_single pti tpr_shadow vnmi flexpriority vpid fsgsbase avx2 invpcid md_clear flush_l1d

Add the Kubernetes repository with:

[root@server1 ~]# cat << EOF > /etc/yum.repos.d/kubernetes.repo
  > [kubernetes]
  > name=Kubernetes
  > baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
  > enabled=1
  > gpgcheck=1
  > repo_gpgcheck=1
  > gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
  > EOF

To download packages and if, like me, you are behind a corporate proxy configura it with a .wgetrc file in the home folder of your account (root in my case). I have also added check_certificate=off to free me from certificate that are not verified by my proxy server (the literature is either mentioning check-certificate or check_certificate so I guess both are working):

[root@server1 ~]# cat .wgetrc
use_proxy=on
https_proxy=http://proxy_server:proxy_port/
http_proxy=http://proxy_server:proxy_por/
proxy_user=proxy_account
proxy_password=proxy_password
check_certificate=off
no_proxy=192.168.0.0/16,10.0.0.0/8

You also need to configure dnf package manager to go though your proxy by adding in /etc/dnf/dnf.conf file:

# The proxy server - proxy server:port number
proxy=http://proxy_server:proxy_port
# The account details for yum connections
proxy_username=proxy_account
proxy_password=proxy_password
sslverify=False

Starting from the Kubernetes getting started web page I have asked myself where to start. I have initially tried with minikube but if like me you really start with your own virtual machine and plan to add one or more worker to your Kubernetes instance then it’s a cluster managed by yourself and you should use kubeadm ! Install kubctl, kubeadm and kubelet with:

[root@server1 ~]# dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

Enable kubelet service with:

[root@server1 ~]# systemctl enable kubelet.service
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /usr/lib/systemd/system/kubelet.service.

Deactivate swap as requested:

[root@server1 ~]# cat /etc/fstab | grep swap
#/dev/mapper/vg00-swap   swap                    swap    defaults        0 0
[root@server1 ~]# swapoff -a
[root@server1 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:          7.6Gi       311Mi       5.8Gi       8.0Mi       1.5Gi       7.0Gi
Swap:            0B          0B          0B

Network prerequisites for container runtime:

[root@server1 ~]# cat /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
[root@server1 ~]# sysctl --system
[root@server1 ~]# cat /etc/modules-load.d/k8s.conf
overlay
br_netfilter
[root@server1 ~]# modprobe overlay
[root@server1 ~]# modprobe br_netfilter

SELinux already deactivated on all my nodes…

I have also tried to choose a different container runtime than Docker. I have tried using Podamn (that I already widely use) and its podam-docker package to have Docker-like commands. Never been able to configure my Kubernetes cluster. I have also trie only with containerd, same story all trials went wrong.

Few errors encountered and their solution:

  1. Podman even with Docker compatible package is not really compatible with Kubernetes, particularly for the service that is not there. So use command line option to pass false errors: –ignore-preflight-errors=all or to fine tune –ignore-preflight-errors IsDockerSystemdCheck,SystemVerification,Service-Docker
  2. [server.go:629] “Failed to get the kubelet’s cgroup. Kubelet system container metrics may be missing.” err=”cpu and memory cgroup hierarchy not unified.
    I solved this one by installing libcgroup with dnf install libcgroup
  3. [WARNING HTTPProxy]: Connection to “https://192.168.56.101” uses proxy “http://proxy_user:proxy_password@proxy_server:proxy_port”. If that is not intended, adjust your proxy settings
    I solved it by exporting this variable with export NO_PROXY=192.168.0.0/16,10.0.0.0/8
  4. [WARNING FileExisting-tc]: tc not found in system path
    Solved this one by installation iproute-tc with dnf install iproute-tc

At each run to purge previous configuration file use:

[root@server1 ~]# kubeadm reset

Sometime it worked but when looking in log files I had many error, particularly with kubelet service (if you have not yet played with kubeadm command the kubelet service is simply in activating (auto-restart)):

[root@server1 ~]# vi /var/log/messages
[root@server1 ~]# systemctl status kubelet
[root@server1 ~]# journalctl -xeu kubelet

Finally installed docker-ce, to be honest all had work like a charm at this point:

[root@server1 ~]# dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
Adding repo from: https://download.docker.com/linux/centos/docker-ce.repo
[root@server1 ~]# dnf remove -y buildah runc
[root@server1 ~]# dnf install -y containerd.io docker-ce
[root@server1 ~]# systemctl enable docker
[root@server1 ~]# systemctl daemon-reload
[root@server1 ~]# systemctl restart docker

And configured docker for proxy access and to use systemd for the management of the container’s cgroups:

[root@server1 ~]# cat << EOF > /etc/systemd/system/docker.service.d/https-proxy.conf
> [Service]
> Environment="HTTPS_PROXY=http://proxy_account:proxy_password@proxy_server:proxy_port"
EOF

[root@server1 ~]# mkdir /etc/docker
[root@server1 ~]# cat << EOF > /etc/docker/daemon.json
> {
>   "exec-opts": ["native.cgroupdriver=systemd"],
>   "log-driver": "json-file",
>   "log-opts": {
>     "max-size": "100m"
>   },
>   "storage-driver": "overlay2"
> }
EOF
[root@server1 ~]# systemctl daemon-reload
[root@server1 ~]# systemctl restart docker

I have anyway benefited from my previous trials and end-up with this kubeadm init command (I have decided to use a different subnet for the pod network 192.168.55.0/24):

[root@server1 ~]# export HTTPS_PROXY='http://proxy_account:proxy_password@proxy_server:proxy_port'
[root@server1 ~]# export NO_PROXY=192.168.0.0/16,10.0.0.0/8
[root@server1 ~]# kubeadm init --apiserver-advertise-address 192.168.56.101 --pod-network-cidr 192.168.55.0/24
[init] Using Kubernetes version: v1.22.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local server1.domain.com] and IPs [10.96.0.1 192.168.56.101]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost server1.domain.com] and IPs [192.168.56.101 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost server1.domain.com] and IPs [192.168.56.101 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 24.005381 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node server1.domain.com as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node server1.domain.com as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: wa7cwx.pml1pqv2i9tnhqkf
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.56.101:6443 --token wa7cwx.pml1pqv2i9tnhqkf \
        --discovery-token-ca-cert-hash sha256:83352a4e1e18e59b4e5a453c1d8573b1fcd718982e0e398741d9182a966472fa

Remark:
If you have lost the join command you can regenerate it with:

[root@server1 ~]# kubeadm token create --print-join-command --v=5

When required to install a Pod network addon I have chosen Flannel. To be able to download from Internet you might be needed to export your proxy configuration if like me you have a corporate proxy (KUBECONFIG cat be put directly in the profile of your root account):

[root@server1 ~]# export KUBECONFIG=/etc/kubernetes/admin.conf
[root@server1 ~]# export HTTPS_PROXY='http://proxy_account:proxy_password@proxy_server:proxy_port'
[root@server1 ~]# export NO_PROXY=192.168.0.0/16,10.0.0.0/8
[root@server1 ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
[root@server1 ~]# kubectl cluster-info
Kubernetes control plane is running at https://192.168.56.101:6443
CoreDNS is running at https://192.168.56.101:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@server1 ~]# kubectl get nodes
NAME                 STATUS   ROLES                  AGE   VERSION
server1.domain.com   Ready    control-plane,master   82m   v1.21.3
[root@server1 ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                         READY   STATUS    RESTARTS   AGE
kube-system   coredns-78fcd69978-8mtv9                     1/1     Running   0          87s
kube-system   coredns-78fcd69978-d5vpz                     1/1     Running   0          87s
kube-system   etcd-server1.domain.com                      1/1     Running   2          98s
kube-system   kube-apiserver-server1.domain.com            1/1     Running   0          103s
kube-system   kube-controller-manager-server1.domain.com   1/1     Running   0          98s
kube-system   kube-flannel-ds-4lpk4                        1/1     Running   0          33s
kube-system   kube-proxy-9c2pr                             1/1     Running   0          87s
kube-system   kube-scheduler-server1.domain.com            1/1     Running   0          98s

Kubernetes Web UI dashboard

For the Kubernetes list of addons I have chosen to install the web UI dashboard as it always help to have a graphical interface to manage things even if at the end of day you mainly work command line:

[root@server1 ~]# kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
Warning: spec.template.metadata.annotations[seccomp.security.alpha.kubernetes.io/pod]: deprecated since v1.19; use the "seccompProfile" field instead
deployment.apps/dashboard-metrics-scraper created

I had an issue and the pods did not come up, so investigated with:

[root@server1 ~]# kubectl describe pods --namespace=kubernetes-dashboard
Name:         dashboard-metrics-scraper-856586f554-lj429
Namespace:    kubernetes-dashboard
Priority:     0
Node:         server1.domain.com/192.168.56.101
Start Time:   Mon, 26 Jul 2021 15:53:35 +0200
Labels:       k8s-app=dashboard-metrics-scraper
              pod-template-hash=856586f554
.
.
.
Events:
  Type     Reason   Age                    From     Message
  ----     ------   ----                   ----     -------
  Warning  Failed   10m (x19 over 104m)    kubelet  Failed to pull image "kubernetesui/dashboard:v2.3.1": rpc error: code = Unknown desc = context canceled
  Normal   Pulling  5m19s (x20 over 105m)  kubelet  Pulling image "kubernetesui/dashboard:v2.3.1"
  Normal   BackOff  26s (x355 over 104m)   kubelet  Back-off pulling image "kubernetesui/dashboard:v2.3.1"

And pulled the Docker image manually with:

[root@server1 ~]# docker pull kubernetesui/dashboard:v2.3.1
v2.3.1: Pulling from kubernetesui/dashboard
b82bd84ec244: Pull complete
21c9e94e8195: Pull complete
Digest: sha256:ec27f462cf1946220f5a9ace416a84a57c18f98c777876a8054405d1428cc92e
Status: Downloaded newer image for kubernetesui/dashboard:v2.3.1
docker.io/kubernetesui/dashboard:v2.3.1

Finally the dashboard pods were up and running:

[root@server1 ~]# kubectl get pods --namespace=kubernetes-dashboard
NAME                                         READY   STATUS    RESTARTS   AGE
dashboard-metrics-scraper-856586f554-lj429   1/1     Running   1          2d21h
kubernetes-dashboard-67484c44f6-h6zwl        1/1     Running   1          2d21h

Then accessing this dashboard from remote (my desktop running VirtualBox) has not been that simple. Firstly, as explained in official documentation, I have used kubectl -n kubernetes-dashboard edit service kubernetes-dashboard command to expose an external port (valid only in a development spirit) and found it with:

[root@server1 ~]# kubectl -n kubernetes-dashboard get pod
NAME                                         READY   STATUS    RESTARTS   AGE
dashboard-metrics-scraper-856586f554-lj429   1/1     Running   1          2d20h
kubernetes-dashboard-67484c44f6-h6zwl        1/1     Running   1          2d20h
[root@server1 ~]# kubectl -n kubernetes-dashboard get service kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.110.133.71           443:30736/TCP   2d20h

Accessing this url I get the login url, not in the form of user/account unfortunately so a bit more of work is required:

kubernetes02
kubernetes02

So let’s create this account to get its token and login:

[root@server1 ~]# cat << EOF > dashboard-adminuser.yaml
> apiVersion: v1
> kind: ServiceAccount
> metadata:
>   name: admin-user
>   namespace: kubernetes-dashboard
> EOF
[root@server1 ~]# kubectl apply -f dashboard-adminuser.yaml
serviceaccount/admin-user created
[root@server1 ~]# cat << EOF > dashboard-authorization-adminuser.yaml
> apiVersion: rbac.authorization.k8s.io/v1
> kind: ClusterRoleBinding
> metadata:
>   name: admin-user
> roleRef:
>   apiGroup: rbac.authorization.k8s.io
>   kind: ClusterRole
>   name: cluster-admin
> subjects:
> - kind: ServiceAccount
>   name: admin-user
>   namespace: kubernetes-dashboard
> EOF
[root@server1 ~]# kubectl apply -f dashboard-authorization-adminuser.yaml
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
[root@server1 ~]# kubectl get serviceaccounts --namespace=kubernetes-dashboard
NAME                   SECRETS   AGE
admin-user             1         6m3s
default                1         2d22h
kubernetes-dashboard   1         2d22h
[root@server1 ~]# kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"
eyJhbGciOiJSUzI1NiIsImtpZCI6IkhmUkV4c1BvSmZwSGZrdk5RdEw2LXBZYklEUWdTREpHZENXclBpRnktSEEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWhkZ242Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI3ZTA3ZTg3NS1lYzU1LTQ1YTEtODUwNy1hNzRlOWJjMjQ4M2MiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.s79nHkqPi8wI3R131vKbvxzLW-N5Th6dsdEvQ8oCh31xIyjh5eOCWTFuG4Jyqra02Uu8CeHThh2SyjyRvJcMy948Oah1SIzyTmGwTxzOO0_hyYDNKCRlSYFbKqMqqKoGlaFoqTObi0-wYzgjrMmrIRMt6JYkm05fQgMVYaXBlUIZMbCx3uhBQKyZ270YQe5os1E_6yhNjI30w2SvpG6aVcrr1pDC-wT7aizJ42_oHx0ZB2REOcJhdUII1nCwF6Cd-kbfMN_kqkxLhi5AIHWGWINDoSAR89jR8-DVmd_ttG9Ou5dhiQ4anXYwcF3BhzQsZdZsY8aoEwxni-aLK9DqXQ

By suppling the token I am finally able to connect:

kubernetes03
kubernetes03

Add a node to a Kubernetes cluster

So far I have only the control-plane node in my cluster:

[root@server1 ~]# kubectl get nodes
NAME                 STATUS   ROLES                  AGE    VERSION
server1.domain.com   Ready    control-plane,master   3d1h   v1.21.3

on my second node (server2.domain.com) I configure the Kubernetes, Docker repository and I install the same packages as on server1.domain.com. I have also obviously configured all the operating system requirements same as server1.domain.com. Finally issue the suggested kubeadm command:

[root@server2 ~]# export HTTPS_PROXY='http://proxy_account:proxy_password@proxy_server:proxy_port'
[root@server2 ~]# export NO_PROXY=192.168.0.0/16,10.0.0.0/8
[root@server2 ~]# kubeadm join 192.168.56.101:6443 --token bbvzqr.z1201gns44iewbo8 --discovery-token-ca-cert-hash sha256:f8dbf9a512fe242b8b818b6528a43285ad8fc41612502a968a09907b8e5e78e7
[preflight] Running pre-flight checks
error execution phase preflight: couldn't validate the identity of the API Server: could not find a JWS signature in the cluster-info ConfigMap for token ID "bbvzqr"
To see the stack trace of this error execute with --v=5 or higher

Unfortunately my token expired (by default, tokens expire after 24 hours) so had to recreate a new one on control-plane node:

[root@server1 ~]# kubeadm token list
[root@server1 ~]# kubeadm token create
w84q7r.6v9kttkhvj34mco5
[root@server1 ~]# kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
w84q7r.6v9kttkhvj34mco5   23h         2021-07-30T17:58:10+02:00   authentication,signing                                                        system:bootstrappers:kubeadm:default-node-token
[root@server1 ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
f8dbf9a512fe242b8b818b6528a43285ad8fc41612502a968a09907b8e5e78e7

And this time it went well:

[root@server2 ~]# kubeadm join 192.168.56.101:6443 --token w84q7r.6v9kttkhvj34mco5 --discovery-token-ca-cert-hash sha256:f8dbf9a512fe242b8b818b6528a43285ad8fc41612502a968a09907b8e5e78e7
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

Node added and role has to be set manually from what I have read:

[root@server1 ~]# kubectl get nodes
NAME                 STATUS     ROLES                  AGE    VERSION
server1.domain.com   Ready      control-plane,master   3d2h   v1.21.3
server2.domain.com   NotReady                    35s    v1.21.3

Or more verbose for nodes labels:

[root@server1 ~]# kubectl get nodes --show-labels
NAME                 STATUS   ROLES                  AGE     VERSION   LABELS
server1.domain.com   Ready    control-plane,master   3d22h   v1.21.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=server1.domain.com,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=
server2.domain.com   Ready                     20h     v1.21.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=server2.domain.com,kubernetes.io/os=linux

To change the label of freshly added node to worker (from what I have read there is no naming convention for nodes labels):

[root@server1 ~]# kubectl label node server2.domain.com node-role.kubernetes.io/worker=
node/server2.domain.com labeled
[root@server1 ~]# kubectl get nodes
NAME                 STATUS   ROLES                  AGE     VERSION
server1.domain.com   Ready    control-plane,master   3d22h   v1.21.3
server2.domain.com   Ready    worker                 20h     v1.21.3
[root@server1 ~]# kubectl describe node server2.domain.com
Name:               server2.domain.com
Roles:              worker
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=server2.domain.com
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/worker=
.
.
.

Lots of system pods have been added:

[root@server1 ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE              NAME                                         READY   STATUS              RESTARTS         AGE    IP               NODE                 NOMINATED NODE   READINESS GATES
default                nginx-6799fc88d8-jk727                       0/1     ContainerCreating   0                19m               server2.domain.com              
kube-system            coredns-78fcd69978-8mtv9                     1/1     Running             0                111m   192.168.55.3     server1.domain.com              
kube-system            coredns-78fcd69978-d5vpz                     1/1     Running             0                111m   192.168.55.2     server1.domain.com              
kube-system            etcd-server1.domain.com                      1/1     Running             2                111m   192.168.56.101   server1.domain.com              
kube-system            kube-apiserver-server1.domain.com            1/1     Running             0                111m   192.168.56.101   server1.domain.com              
kube-system            kube-controller-manager-server1.domain.com   1/1     Running             0                111m   192.168.56.101   server1.domain.com              
kube-system            kube-flannel-ds-2922x                        0/1     CrashLoopBackOff    21 (2m51s ago)   87m    192.168.56.102   server2.domain.com              
kube-system            kube-flannel-ds-4lpk4                        1/1     Running             0                110m   192.168.56.101   server1.domain.com              
kube-system            kube-proxy-9c2pr                             1/1     Running             0                111m   192.168.56.101   server1.domain.com              
kube-system            kube-proxy-9p268                             1/1     Running             0                87m    192.168.56.102   server2.domain.com              
kube-system            kube-scheduler-server1.domain.com            1/1     Running             0                111m   192.168.56.101   server1.domain.com              
kubernetes-dashboard   dashboard-metrics-scraper-856586f554-mwwzw   1/1     Running             0                109m   192.168.55.5     server1.domain.com              
kubernetes-dashboard   kubernetes-dashboard-67484c44f6-ggvj8        1/1     Running             0                109m   192.168.55.4     server1.domain.com              

The Flannel container is not able to start on newly added node, this is apparently a bug:

[root@server1 ~]# kubectl -n kube-system  logs -p kube-flannel-ds-7hr6x
I1008 09:59:16.295218       1 main.go:520] Determining IP address of default interface
I1008 09:59:16.296819       1 main.go:533] Using interface with name enp0s8 and address 10.70.101.44
I1008 09:59:16.296883       1 main.go:550] Defaulting external address to interface address (10.70.101.44)
W1008 09:59:16.296945       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1008 09:59:17.400125       1 kube.go:116] Waiting 10m0s for node controller to sync
I1008 09:59:17.400262       1 kube.go:299] Starting kube subnet manager
I1008 09:59:18.400588       1 kube.go:123] Node controller sync successful
I1008 09:59:18.400644       1 main.go:254] Created subnet manager: Kubernetes Subnet Manager - server2.domain.com
I1008 09:59:18.400670       1 main.go:257] Installing signal handlers
I1008 09:59:18.401529       1 main.go:392] Found network config - Backend type: vxlan
I1008 09:59:18.401704       1 vxlan.go:123] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
E1008 09:59:18.402563       1 main.go:293] Error registering network: failed to acquire lease: node "server2.domain.com" pod cidr not assigned
I1008 09:59:18.403201       1 main.go:372] Stopping shutdownHandler...

Solved it with:

[root@server1 ~]# kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
192.168.55.0/24
[root@server1 ~]# kubectl patch node server2.domain.com -p '{"spec":{"podCIDR":"192.168.55.0/24"}}'
node/server2.domain.com patched
[root@server1 ~]# kubectl get pods --all-namespaces -o wide
NAMESPACE              NAME                                         READY   STATUS    RESTARTS      AGE   IP               NODE                 NOMINATED NODE   READINESS GATES
kube-system            coredns-78fcd69978-8mtv9                     1/1     Running   0             20h   192.168.55.3     server1.domain.com              
kube-system            coredns-78fcd69978-d5vpz                     1/1     Running   0             20h   192.168.55.2     server1.domain.com              
kube-system            etcd-server1.domain.com                      1/1     Running   2             20h   192.168.56.101   server1.domain.com              
kube-system            kube-apiserver-server1.domain.com            1/1     Running   0             20h   192.168.56.101   server1.domain.com              
kube-system            kube-controller-manager-server1.domain.com   1/1     Running   0             20h   192.168.56.101   server1.domain.com              
kube-system            kube-flannel-ds-4lpk4                        1/1     Running   0             20h   192.168.56.101   server1.domain.com              
kube-system            kube-flannel-ds-7hr6x                        1/1     Running   8 (11m ago)   22m   192.168.56.102   server2.domain.com              
kube-system            kube-proxy-86rws                             1/1     Running   0             22m   192.168.56.102   server2.domain.com              
kube-system            kube-proxy-9c2pr                             1/1     Running   0             20h   192.168.56.101   server1.domain.com              
kube-system            kube-scheduler-server1.domain.com            1/1     Running   0             20h   192.168.56.101   server1.domain.com              
kubernetes-dashboard   dashboard-metrics-scraper-856586f554-mwwzw   1/1     Running   0             20h   192.168.55.5     server1.domain.com              
kubernetes-dashboard   kubernetes-dashboard-67484c44f6-ggvj8        1/1     Running   0             20h   192.168.55.4     server1.domain.com              

To remove this node from the cluster use:

[root@server1 ~]# kubectl delete node server2.domain.com
node "server2.domain.com" deleted

Useful commands to debug container issues

List of commands that give containers status as well as container logs:

kubectl get pods --all-namespaces -o wide
kubectl -n kube-system  describe pod kube-flannel-ds-2922x
kubectl -n kube-system  logs -p kube-flannel-ds-2922x

References

The post Kubernetes on virtual machines hands-on – part 1 appeared first on IT World.

]]>
https://blog.yannickjaquier.com/linux/kubernetes-on-virtual-machines-hands-on-part-1.html/feed 1
Immutable tables and blockchain tables hands-on https://blog.yannickjaquier.com/oracle/immutable-tables-and-blockchain-tables-hands-on.html https://blog.yannickjaquier.com/oracle/immutable-tables-and-blockchain-tables-hands-on.html#respond Mon, 28 Mar 2022 08:28:03 +0000 https://blog.yannickjaquier.com/?p=5334 Preamble Immutable tables and blockchain tables are 21c new features that have been backported to 19c with those Release Update (RU): Immutable Tables 19.11 Blockchain Tables 19.10 Immutable table and blockchain tables are free in all database editions. Even if through the game of Release Update (RU) Oracle has made blockchain tables available before immutable […]

The post Immutable tables and blockchain tables hands-on appeared first on IT World.

]]>

Table of contents

Preamble

Immutable tables and blockchain tables are 21c new features that have been backported to 19c with those Release Update (RU):

  • Immutable Tables 19.11
  • Blockchain Tables 19.10

Immutable table and blockchain tables are free in all database editions. Even if through the game of Release Update (RU) Oracle has made blockchain tables available before immutable tables, blockchain tables is an extension of immutable table.

From official documentation:

Immutable tables are read-only tables that prevent unauthorized data modifications by insiders and accidental data modifications resulting from human errors.

Blockchain tables are insert-only tables that organize rows into a number of chains. Each row in a chain, except the first row, is chained to the previous row in the chain by using a cryptographic hash.

Blockchain tables are a step further from immutable tables to prevent unauthorized table modifications from stolen credentials as well as modifications from outside the database and/or using security breaches not yet patched. The subject that obviously comes to mind with blockchain tables is cryptographic money like bitcoin where each digital money transaction is stored in a blockchain table to prevent modification by an attacker of past transactions.

In my opinion the features are not yet 100% mature in 19c, even with 19.12, as it is quite easy to generate an ORA-00600 when playing with them.

My testing has been done on a 19c (19.12) Enterprise Edition release running on bare metal server running RHEL 7.9 with 12 cores and 64GB of memory.

Immutable tables

To use them in your 19.12 (pluggable) database you need to change the COMPATIBLE parameter:

ORA-00406: COMPATIBLE parameter needs to be 19.11.0.0.0 or greater
ORA-00722: Feature "Immutable table"

This is funny because in the COMPATIBLE official documentation Oracle says:

The value of the COMPATIBLE parameter should not be changed for a Release Update (RU) or Release Update Revision (RUR). For example, assume you are running Oracle Database 19c and the value of COMPATIBLE is 19.0.0. You then apply Oracle Database Release Update 19.6.0.0.0. Do not set the value of COMPATIBLE to 19.6.0; leave it set to 19.0.0.

So changed on my CDB and bounced the instance:

SQL> alter system set compatible='19.12.0' scope=spfile;

System altered.

You create an immutable table with below command. The minimum number of days after which a row can be deleted after its insertion is 16 days:

SQL> create immutable table test01(id number, descr varchar2(20)) no drop until 1 days idle no delete until 16 days after insert;

Table created.

SQL> select row_retention, row_retention_locked, table_inactivity_retention from dba_immutable_tables where table_name = 'TEST01';

ROW_RETENTION ROW TABLE_INACTIVITY_RETENTION
------------- --- --------------------------
           16 NO                           1

Be very carreful with the parameters you use as below command is creating an immutable table where rows can never be deleted (really ?) and that can never be dropped (at least I’m still searching how I could):

SQL> create immutable table test01(id number, descr varchar2(20)) no drop no delete;

Table created.

SQL> col table_name for a10
SQL> select table_name, row_retention, row_retention_locked, table_inactivity_retention from user_immutable_tables;

TABLE_NAME ROW_RETENTION ROW TABLE_INACTIVITY_RETENTION
---------- ------------- --- --------------------------
TEST01                   NO

The drop period inactivity cannot be decreased:

SQL> alter table test01 no drop until 0 days idle;
alter table test01 no drop until 0 days idle
*
ERROR at line 1:
ORA-05732: retention value cannot be lowered

The period after which rows can be deleted can also not be decreased:

SQL> alter table test01 no delete until 17 days after insert;

Table altered.

SQL> alter table test01 no delete until 16 days after insert;
alter table test01 no delete until 16 days after insert
*
ERROR at line 1:
ORA-05732: retention value cannot be lowered

You can also lock the row retention period with:

SQL> alter table test01 no delete until 17 days after insert locked;

Table altered.

SQL> select table_name, row_retention, row_retention_locked, table_inactivity_retention from user_immutable_tables;

TABLE_NAME ROW_RETENTION ROW TABLE_INACTIVITY_RETENTION
---------- ------------- --- --------------------------
TEST01                17 YES                          1

SQL> alter table test01 no delete until 18 days after insert;
alter table test01 no delete until 18 days after insert
*
ERROR at line 1:
ORA-05731: blockchain or immutable table TEST01 cannot be altered

For testing as expected:

SQL> insert into test01 values(1, 'One');

1 row created.

SQL> commit;

Commit complete.

SQL> select * from test01;

        ID DESCR
---------- --------------------
         1 One

SQL> set lines 200
SQL> col orabctab_creation_time$ for a40
SQL> select orabctab_creation_time$, orabctab_user_number$, a.* from test01 a;

ORABCTAB_CREATION_TIME$                  ORABCTAB_USER_NUMBER$         ID DESCR
---------------------------------------- --------------------- ---------- --------------------
20-OCT-21 10.17.40.761213 AM +00:00                        116          1 One
         
SQL> delete from test01 where id=1;
delete from test01 where id=1
            *
ERROR at line 1:
ORA-05715: operation not allowed on the blockchain or immutable table

Last but not least you have the DBMS_IMMUTABLE_TABLE package to automatically delete candidates rows that are older than a given date:

set serveroutput on size 9999
declare
  nb_rows number;
begin
  dbms_immutable_table.delete_expired_rows(schema_name => 'TEST01', table_name => 'TEST01', before_timestamp => systimestamp-1, number_of_rows_deleted => nb_rows);
  dbms_output.put_line('Number of rows deleted: ' || nb_rows);
end;
/

Number of rows deleted: 0

PL/SQL procedure successfully completed.

Blockchain tables

Basic testing

The syntax to create a blockchain table is very similar to immutable tables, except that there is a hashing algorithm that is apparently not (yet) possible to change:

SQL> create blockchain table transaction(source_iban varchar2(32), target_iban varchar2(32), deposit_date timestamp, deposit_amount number)
no drop until 0 days idle
no delete until 16 days after insert
hashing using "sha2_512" version "v1";

Table created.

SQL> set lines 200
SQL> col table_name for a15
SQL> select * from user_blockchain_tables;

TABLE_NAME      ROW_RETENTION ROW TABLE_INACTIVITY_RETENTION HASH_ALG
--------------- ------------- --- -------------------------- --------
TRANSACTION                16 NO                           0 SHA2_512

The table has more than 10 hidden columns on top of the one you have declared. Those columns contain, for example, hash of the row, sequence of the row in the chain, signature of the row and so on…

Inserting a row is as transparent as for a normal table:

SQL> insert into transaction values('FR7630001007941234567890185', 'FR1420041010050500013M02606', systimestamp, 100);

1 row created.

SQL> commit;

Commit complete.

If you try to select few of the available hidden columns. The hash column is one of the blockchain tables key column. For now the signature column is empty because I have not yet signed my insertions:

SQL> set lines 200
SQL> col orabctab_hash$ for a20
SQL> col orabctab_signature$ for a20
SQL> select orabctab_inst_id$,orabctab_chain_id$,orabctab_seq_num$,orabctab_hash$,orabctab_signature$,orabctab_creation_time$ from transaction;

ORABCTAB_INST_ID$ ORABCTAB_CHAIN_ID$ ORABCTAB_SEQ_NUM$ ORABCTAB_HASH$       ORABCTAB_SIGNATURE$  ORABCTAB_CREATION_TIME$
----------------- ------------------ ----------------- -------------------- -------------------- ----------------------------------------
                1                 16                 1 87EECD3739AF7DA12A02                      20-OCT-21 04.41.11.120982 PM +00:00
                                                       76A3ED85707E5463B249
                                                       0470515B30AD4EC00FCD
                                                       259A4EDA7A66763E8C23
                                                       2F436481F47098C4CF63
                                                       B294FDB95746D37DD171
                                                       C9A298CF

If I insert new rows, the tuple made of ORABCTAB_INST_ID$, ORABCTAB_CHAIN_ID$ and ORABCTAB_SEQ_NUM$ uniquely identifies a row in the blockchain table:

SQL> insert into transaction values('FR3214508000703188446194H85','FR3930003000704342572143H02', systimestamp, 150);

1 row created.

SQL> insert into transaction values('FR9017569000404619456186I04','FR4912739000306676995962C37', systimestamp, 250);

1 row created.

SQL> commit;

Commit complete.

SQL> set lines 200
SQL> select orabctab_inst_id$,orabctab_chain_id$,orabctab_seq_num$ from transaction;

ORABCTAB_INST_ID$ ORABCTAB_CHAIN_ID$ ORABCTAB_SEQ_NUM$
----------------- ------------------ -----------------
                1                 16                 1
                1                 21                 1
                1                 21                 2

Remark:
The chain id is apparently evolving as time pass.

All the rows are chained as in below schema:

blockchain01
blockchain01

Obviously, by design, you cannot alter the content of the table and you have the same properties as for immutable tables when you try to alter or change the retention period:

SQL> update transaction set deposit_amount=200 where source_iban='FR3214508000703188446194H85';
update transaction set deposit_amount=200 where source_iban='FR3214508000703188446194H85'
       *
ERROR at line 1:
ORA-05715: operation not allowed on the blockchain or immutable table

The DBMS_BLOCKCHAIN_TABLE has procedures to verify the itegrity of your blockchain table, for exmaple:

SQL> set serveroutput on size 9999
SQL>
DECLARE
  verify_rows NUMBER;
BEGIN
  DBMS_BLOCKCHAIN_TABLE.VERIFY_ROWS('test01','transaction', NULL, NULL, NULL, NULL, verify_rows, FALSE);
  DBMS_OUTPUT.PUT_LINE('Number of rows verified in table: ' || verify_rows);
END;
/

Number of rows verified in table: 3

PL/SQL procedure successfully completed.

Signing rows

In case a user credential got stolen you can also sign the inserted rows in blockchain tables using client private key that is never shared to the database. As such even if a thief try to insert rows in your blockchain table using a comprise user account he will lack the private client certificate and will not be able to sign rows. The Oracle official documentation is really not good at all and lots of steps are not well explained. I have finally succeeded to do sign rows using even blog post not written in English (so that I cannot share) where I have been able at least to understand the code.

Let’s start by creating the root CA certificate, obviously with openSSL:

[oracle@server01 ~]$ mkdir certs
[oracle@server01 ~]$ cd certs
[oracle@server01 certs]$ openssl genrsa -out MyRootCA.key 4096
Generating RSA private key, 4096 bit long modulus
.....++
.....................................++
e is 65537 (0x10001)
[oracle@server01 certs]$ openssl req -x509 -new -nodes -key MyRootCA.key -sha512 -days 1024 -out MyRootCA.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CH
State or Province Name (full name) []:Geneva
Locality Name (eg, city) [Default City]:Geneva
Organization Name (eg, company) [Default Company Ltd]:MyCompany
Organizational Unit Name (eg, section) []:DIT
Common Name (eg, your name or your server's hostname) []:MyServer
Email Address []:yannick.jaquier@gmail.com

Create a client certificate signed by our freshly generated root CA:

[oracle@server01 certs]$ openssl genrsa -out MyClient.key 4096
Generating RSA private key, 4096 bit long modulus
..........................++
.............................++
e is 65537 (0x10001)
[oracle@server01 certs]$ openssl req -new -key MyClient.key -out MyClient.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CH
State or Province Name (full name) []:Geneva
Locality Name (eg, city) [Default City]:Geneva
Organization Name (eg, company) [Default Company Ltd]:MyCompany
Organizational Unit Name (eg, section) []:DIT
Common Name (eg, your name or your server's hostname) []:MyServer
Email Address []:yannick.jaquier@gmail.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[oracle@server01 certs]$ openssl x509 -req -in MyClient.csr -CA MyRootCA.pem -CAkey MyRootCA.key -CAcreateserial -out MyClient.pem -days 365 -sha512
Signature ok
subject=/C=CH/ST=Geneva/L=Geneva/O=MyCompany/OU=DIT/CN=MyServer/emailAddress=yannick.jaquier@gmail.com
Getting CA Private Key

So we now get:

[oracle@server01 certs]$ ll
total 24
-rw-r----- 1 oracle dba 1752 Oct 28 15:19 MyClient.csr
-rw-r----- 1 oracle dba 3243 Oct 28 15:18 MyClient.key
-rw-r----- 1 oracle dba 2004 Oct 28 15:19 MyClient.pem
-rw-r----- 1 oracle dba 3243 Oct 28 15:16 MyRootCA.key
-rw-r----- 1 oracle dba 2122 Oct 28 15:17 MyRootCA.pem
-rw-r----- 1 oracle dba   17 Oct 28 15:19 MyRootCA.srl

Load the public certificate inside the database with (the write privilege on directory is to write row signature file), you need to note down the certificate id you will later use:

SQL> create or replace directory certs as '/home/oracle/certs';

Directory created.

SQL> grant read, write on directory certs to test01;

Grant succeeded.

SQL> set serveroutput on size 9999
SQL>
DECLARE
  file       BFILE;
	buffer     BLOB;
	amount     NUMBER := 32767;
	cert_id  RAW(16);
BEGIN
	file := BFILENAME('CERTS', 'MyClient.pem');
	DBMS_LOB.FILEOPEN(file);
	DBMS_LOB.READ(file, amount, 1, buffer);
	DBMS_LOB.FILECLOSE(file);
	DBMS_USER_CERTS.ADD_CERTIFICATE(buffer, cert_id);
	DBMS_OUTPUT.PUT_LINE('Certificate ID = ' || cert_id);
END;
/

Certificate ID = 38AE484868150D34451AFA8B99E4871E

PL/SQL procedure successfully completed.

SQL> set lines 200
SQL> col user_name for a10
SQL> col distinguished_name for a50
SQL> select user_name, distinguished_name, UTL_RAW.LENGTH(certificate_id) cert_id_len, DBMS_LOB.GETLENGTH(certificate) cert_len from user_certificates;

USER_NAME  DISTINGUISHED_NAME                                 CERT_ID_LEN   CERT_LEN
---------- -------------------------------------------------- ----------- ----------
TEST01     EMAIL=yannick.jaquier@gmail.com,CN=MyServer,OU=DIT          16       2004
           ,O=MyCompany,L=Geneva,ST=Geneva,C=CH

SQL> select certificate_id from user_certificates;

CERTIFICATE_ID
--------------------------------
38AE484868150D34451AFA8B99E4871E

Drop it with:

SQL> exec DBMS_USER_CERTS.DROP_CERTIFICATE('38AE484868150D34451AFA8B99E4871E');

PL/SQL procedure successfully completed.

Generate a signature file (signature.dat) for one row of your blockchain table:

SQL>
DECLARE
  row_data        BLOB;
  buffer          RAW(4000);
  inst_id         BINARY_INTEGER;
  chain_id        BINARY_INTEGER;
  sequence_no     BINARY_INTEGER;
  row_len         BINARY_INTEGER;
  l_file          UTL_FILE.FILE_TYPE;
BEGIN
  SELECT ORABCTAB_INST_ID$, ORABCTAB_CHAIN_ID$, ORABCTAB_SEQ_NUM$ INTO inst_id, chain_id, sequence_no FROM transaction
  WHERE ORABCTAB_INST_ID$=1 and ORABCTAB_CHAIN_ID$=16 and ORABCTAB_SEQ_NUM$=1;
  DBMS_BLOCKCHAIN_TABLE.GET_BYTES_FOR_ROW_SIGNATURE('test01','transaction', inst_id, chain_id, sequence_no, 1, row_data);
  row_len := DBMS_LOB.GETLENGTH(row_data);
  DBMS_LOB.READ(row_data, row_len, 1, buffer);
  l_file := UTL_FILE.fopen('CERTS','signature.dat','wb', 32767);
  UTL_FILE.put_raw(l_file, buffer, TRUE);
  UTL_FILE.fclose(l_file);
END;
/

PL/SQL procedure successfully completed.

Now I have this signature row file in the CERTS directory that I must sign with my client certificate using openSSL. For the sake of testing I have my client private key (MyClient.key) in same directory as my other certificates but clearly those private client certificates must not be on the database server and kept in a safe place:

[oracle@server01 certs]$ ll signature.dat
-rw-r----- 1 oracle dba 64 Oct 28 15:25 signature.dat
[oracle@server01 certs]$ openssl dgst -sha512 -sign MyClient.key -out row_signature.dat signature.dat

Finally sign the row with this row signature file (do not forget the commit as the PL/SQL procedure does not do it, wasted a bit of time on this):

SQL>
DECLARE
  inst_id binary_integer;
  chain_id binary_integer;
  sequence_no binary_integer;
  file BFILE;
  amount NUMBER := 2000;
  signature RAW(2000);
  cert_id RAW (16) := HEXTORAW('38AE484868150D34451AFA8B99E4871E');
BEGIN
  SELECT ORABCTAB_INST_ID$, ORABCTAB_CHAIN_ID$, ORABCTAB_SEQ_NUM$ INTO inst_id, chain_id, sequence_no FROM transaction
  WHERE ORABCTAB_INST_ID$=1 and ORABCTAB_CHAIN_ID$=16 and ORABCTAB_SEQ_NUM$=1;
  file := bfilename('CERTS', 'row_signature.dat');
  DBMS_LOB.FILEOPEN(file);
  dbms_lob.READ(file, amount, 1, signature);
  dbms_lob.FILECLOSE(file);
  DBMS_BLOCKCHAIN_TABLE.SIGN_ROW('test01','transaction', inst_id, chain_id, sequence_no, NULL, signature, cert_id, DBMS_BLOCKCHAIN_TABLE.SIGN_ALGO_RSA_SHA2_512);
END;
/

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

Now if you check the ORABCTAB_SIGNATURE$ hidden columns of your blockchain table you will see that the row has been signed:

SQL> set lines 200
SQL> set pages 1000
SQL> select orabctab_inst_id$, orabctab_chain_id$, orabctab_seq_num$, orabctab_signature$ from transaction;

ORABCTAB_INST_ID$ ORABCTAB_CHAIN_ID$ ORABCTAB_SEQ_NUM$ ORABCTAB_SIGNATURE$
----------------- ------------------ ----------------- ------------------------------
                1                 16                 1 AB2BA40938A89152F24A753DE5A925
                                                       C16263403C2B1F0675D9D2767A3AEB
                                                       27078473F96CAD89D47D33670B2038
                                                       7A0BDB2144A33FEF1F7E2BDDACE352
                                                       5DD4E5CEF44E20C5656A7DE3647D88
                                                       58020801A5C32F14ED1E306CDCC672
                                                       78567152E46772B10F1693F48E7C48
                                                       8BEF2BA7EBAF6D9F71CF07E1031B0D
                                                       1038D4E4D98150F

                1                 21                 1
                1                 21                 2

References

The post Immutable tables and blockchain tables hands-on appeared first on IT World.

]]>
https://blog.yannickjaquier.com/oracle/immutable-tables-and-blockchain-tables-hands-on.html/feed 0
MariaDB PAM LDAP authentication with newest SSSD daemon https://blog.yannickjaquier.com/mariadb/mariadb-pam-ldap-authentication-with-newest-sssd-daemon.html https://blog.yannickjaquier.com/mariadb/mariadb-pam-ldap-authentication-with-newest-sssd-daemon.html#respond Mon, 28 Feb 2022 10:19:02 +0000 https://blog.yannickjaquier.com/?p=5390 Preamble In an old MariaDB instance we had MariaDB PAM LDAP users’ authentication based on our corporate LDAP server. We recently migrated this MariaDB 10.0.13 running on Red Hat Enterprise Linux Server release 6.10 (Santiago) in MariaDB 10.5.9 running on Red Hat Enterprise Linux release 8.2 (Ootpa). And as you might guess nothing worked or […]

The post MariaDB PAM LDAP authentication with newest SSSD daemon appeared first on IT World.

]]>

Table of contents

Preamble

In an old MariaDB instance we had MariaDB PAM LDAP users’ authentication based on our corporate LDAP server. We recently migrated this MariaDB 10.0.13 running on Red Hat Enterprise Linux Server release 6.10 (Santiago) in MariaDB 10.5.9 running on Red Hat Enterprise Linux release 8.2 (Ootpa).

And as you might guess nothing worked or I would not have written this blog post and if you are here it’s most probably because you are in same situation…

At the end we discovered that the issue was 100% linked to the Operating System and had nothing to see with MariaDB instance where configuration is piece of cake…

I recall that in this blog post I will not build any LDAP server and our clear objective is to use our corporate server (Oracle Unified Directory, OUD) already containing all our employees.

This first part will preparation of Os packages and database required parameters second part about configuration of legacy services nslcd – local LDAP name service daemon and third part will be a migration to the latest way of working using System Security Services Daemon (SSSD) service.

Extracted from official Red Hat documentation SSSD is the new way of accessing remote directories and has below benefits:

  • Reduced load on identity and authentication servers
  • Offline authentication
  • A single user account: improved consistency of the authentication process

My testing has been done on Red Hat Enterprise Linux release 8.2 (Ootpa) and MariaDB Community 10.5.9.

MariaDB PAM LDAP Linux and database configuration

For required packages it is quite simple:

dnf install openldap-clients nss-pam-ldapd pam pam-devel

Ensure you have the PAM LDAP library:

[root@server1 ~]# ll /lib64/security/pam_ldap.so
-rwxr-xr-x. 1 root root 33328 Aug 12  2018 /lib64/security/pam_ldap.so

From MariaDB it’s as simple as:

MariaDB [(none)]> install soname 'auth_pam';
Query OK, 0 rows affected (0.000 sec)
MariaDB [(none)]> install plugin if not exists pam soname 'auth_pam';
Query OK, 0 rows affected, 1 warning (0.001 sec)

You can confirm the plugin is active with:

MariaDB [(none)]> show plugins;
+-------------------------------+----------+--------------------+-------------+---------+
| Name                          | Status   | Type               | Library     | License |
+-------------------------------+----------+--------------------+-------------+---------+
.
.
.
| pam                           | ACTIVE   | AUTHENTICATION     | auth_pam.so | GPL     |
+-------------------------------+----------+--------------------+-------------+---------+

In our my.cnf file we have added:

[client]
.
plugin_dir=/maria_10.5.9/software/10.5.9/lib/plugin

[mariadb]
plugin_dir=/maria_10.5.9/software/10.5.9/lib/plugin
.

This to avoid this below error:

[mariapoc@server1 ~]$ $MARIADB_HOME/bin/mysql --defaults-file=$MARIADB_HOME/conf/my.cnf --user=yjaquier -p
Enter password:
ERROR 1045 (28000): Plugin dialog could not be loaded: /usr/local/mysql/lib/plugin/dialog.so: cannot open shared object file: No such file or directory

Then I created and account for myself, account that is matching my corporate LDAP short login with below commands:

MariaDB [(none)]> CREATE USER 'yjaquier'@'%' IDENTIFIED VIA pam USING 'mariadb';
Query OK, 0 rows affected (0.029 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'yjaquier'@'%' ;
Query OK, 0 rows affected (0.029 sec)
MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.002 sec)

Connection did not work and checking in /var/log/secure file I have seen:

Nov 19 10:02:35 SERVER1 auth_pam_tool[931652]: pam_ldap(mariadb:auth): error reading from nslcd: Connection reset by peer

And using journalctl -xfl I have seen:

Nov 19 10:02:34 server1 nslcd[931632]: [8b4567]  no available LDAP server found, sleeping 1 seconds
Nov 19 10:02:35 server1 nslcd[931632]: [8b4567]  failed to bind to LDAP server ldap://127.0.0.1/: Can't contact LDAP server: Transport endpoint is not connected
Nov 19 10:02:35 server1 nslcd[931632]: [8b4567]  no available LDAP server found: Can't contact LDAP server: Transport endpoint is not connected
Nov 19 10:02:35 server1 auth_pam_tool[931652]: pam_ldap(mariadb:auth): error reading from nslcd: Connection reset by peer

From this output it was clear that nslcd daemon is the process doing the LDAP call and that I needed to configure but it did not go as expected…

Ldapsearch configuration

To start debugging LDAP accesses I wanted at least to be able to perform a ldapsearch to confirm I was able to query my LDAP server. The installation of openldap is one of the prerequisite of MariaDB PAM LDAP authentication if you want to build your own LDAP server. But as explained I obviously plan to use our corporate LDAP server…

In /etc/openldap/ldap.conf I have only configured BASE and URI parameters:

BASE ou=employee,dc=company,dc=com
URI ldaps://ldap-server.company.com

Remark:
Our LDAP server is using the LDAP secure protocol so the ldaps protocol keyword.

My first ldapsearch command was on my own account:

[root@server1 ~]# ldapsearch -H "ldaps://ldap-server.company.com/" -b "ou=employee,dc=company,dc=com" -s sub "(uid=yjaquier)"
ldap_sasl_interactive_bind_s: Can't contact LDAP server (-1)
        additional info: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (self signed certificate in certificate chain)

Okay my LDAP server has a certificate self signed by my company. Looks okay for an internal one, why woud we pay for an internal company service…

Changing the way to connect does not display certificate error any more:

[root@server1 ~]# ldapsearch -x -H "ldaps://ldap-server.company.com/" -b "ou=employee,dc=company,dc=com" -s sub "(uid=yjaquier)"
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

But (-d for debug option) error is still there in real:

[root@server1 ~]# ldapsearch -d 1 -x -H "ldaps://ldap-server.company.com/" -b "ou=employee,dc=company,dc=com" -s sub "(uid=yjaquier)"
ldap_url_parse_ext(ldaps://ldap-server.company.com/)
ldap_create
ldap_url_parse_ext(ldaps://ldap-server.company.com:636/??base)
ldap_sasl_bind
ldap_send_initial_request
ldap_new_connection 1 1 0
ldap_int_open_connection
ldap_connect_to_host: TCP ldap-server.company.com:636
ldap_new_socket: 3
ldap_prepare_socket: 3
ldap_connect_to_host: Trying 164.129.225.251:636
ldap_pvt_connect: fd: 3 tm: -1 async: 0
attempting to connect:
connect success
TLS trace: SSL_connect:before SSL initialization
TLS trace: SSL_connect:SSLv3/TLS write client hello
TLS trace: SSL_connect:SSLv3/TLS write client hello
TLS trace: SSL_connect:SSLv3/TLS read server hello
TLS certificate verification: depth: 2, err: 19, subject: /C=CH/O=STMicroelectronics International N.V./OU=Corporate Services/CN=STMicroelectronics Private Corporate Root CA1, issuer: /C=CH/O=STMicroelectronics International N.V./OU=Corporate Services/CN=STMicroelectronics Private Corporate Root CA1
TLS certificate verification: Error, self signed certificate in certificate chain
TLS trace: SSL3 alert write:fatal:unknown CA
TLS trace: SSL_connect:error in error
TLS: can't connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (self signed certificate in certificate chain).
ldap_err2string
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

To go further with this certificates story I extracted the complete chain of certification of my LDAP server with:

[root@server1 ~]# cd /etc/openldap/certs
[root@server1 certs]# openssl s_client -showcerts -verify 5 -connect ldap-server.company.com:636  < /dev/null | awk '/BEGIN/,/END/{ if(/BEGIN/)    {a++}; out="ldap-server-cert"a".pem"; print >out}'
verify depth is 5
depth=2 C = CH, O = STMicroelectronics International N.V., OU = Corporate Services, CN = STMicroelectronics Private Corporate Root CA1
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=2 C = CH, O = STMicroelectronics International N.V., OU = Corporate Services, CN = STMicroelectronics Private Corporate Root CA1
verify return:1
depth=1 C = CH, O = STMicroelectronics International N.V., OU = Corporate Services, CN = STMicroelectronics Corporate Server CA1
verify return:1
depth=0 CN = server05.domain.com, OU = DIT, O = STMicroelectronics International NV
verify return:1
DONE
[root@server1 certs]# ll
total 12
-rw-r--r--. 1 root root  548 Nov 18 12:12 ldap-server-cert1.pem
-rw-r--r--. 1 root root 2403 Nov 18 12:12 ldap-server-cert2.pem
-rw-r--r--. 1 root root 2183 Nov 18 12:12 ldap-server-cert3.pem

If you verify them we get more or less what we got with ldapsearch command (or what nslcd daemon is throwing):

[root@server1 certs]# for cert in /etc/openldap/certs/*.pem; do openssl verify -show_chain $cert ; done
CN = server05.domain.com, OU = DIT INFRASTRUCTURE & SERVICES, O = STMicroelectronics International NV
error 20 at 0 depth lookup: unable to get local issuer certificate
error /etc/openldap/certs/ldap-server-cert1.pem: verification failed
C = CH, O = STMicroelectronics International N.V., OU = Corporate Services, CN = STMicroelectronics Corporate Server CA1
error 20 at 0 depth lookup: unable to get local issuer certificate
error /etc/openldap/certs/ldap-server-cert2.pem: verification failed
C = CH, O = STMicroelectronics International N.V., OU = Corporate Services, CN = STMicroelectronics Private Corporate Root CA1
error 18 at 0 depth lookup: self signed certificate
error /etc/openldap/certs/ldap-server-cert3.pem: verification failed

I had to insert into my MariaDB server my company Certificate Authority files.

[root@server1 ~]# cd /etc/pki/ca-trust/source/anchors
[root@server1 anchors]# cp ~/certs/companycorporateRootCA.cer.txt .
[root@server1 anchors]# cp ~/certs/companycorporateserverCA.cer.txt .
[root@server1 anchors]# update-ca-trust extract

If you verify again the certificate chain all is good:

[root@server1 anchors]# for cert in /etc/openldap/certs/*.pem; do openssl verify -show_chain $cert ; done
/etc/openldap/certs/ldap-server-cert1.pem: OK
Chain:
depth=0: CN = server05.domain.com, OU = DIT INFRASTRUCTURE & SERVICES, O = STMicroelectronics International NV (untrusted)
depth=1: C = CH, O = STMicroelectronics International N.V., OU = Corporate Services, CN = STMicroelectronics Corporate Server CA1
depth=2: C = CH, O = STMicroelectronics International N.V., OU = Corporate Services, CN = STMicroelectronics Private Corporate Root CA1
/etc/openldap/certs/ldap-server-cert2.pem: OK
Chain:
depth=0: C = CH, O = STMicroelectronics International N.V., OU = Corporate Services, CN = STMicroelectronics Corporate Server CA1 (untrusted)
depth=1: C = CH, O = STMicroelectronics International N.V., OU = Corporate Services, CN = STMicroelectronics Private Corporate Root CA1
/etc/openldap/certs/ldap-server-cert3.pem: OK
Chain:
depth=0: C = CH, O = STMicroelectronics International N.V., OU = Corporate Services, CN = STMicroelectronics Private Corporate Root CA1

And now the ldapsearch command is working:

[root@server1 ~]# ldapsearch -x -H "ldaps://ldap-server.company.com/" -b "ou=employee,dc=company,dc=com" -s sub "(uid=yjaquier)"
# extended LDIF
#
# LDAPv3
# base  with scope subtree
# filter: (uid=yjaquier)
# requesting: ALL
#

You can also remove the -x option and perform an authenticated LDAP query with (at the prompt supply your LDAP password):

[root@server1 ~]# ldapsearch -D "eduid=ed999999,ou=employee,dc=company,dc=com" -W -H "ldaps://ldap-server.company.com/" -b "ou=employee,dc=company,dc=com" -s sub "(uid=yjaquier)"
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base  with scope subtree
# filter: (uid=yjaquier)
# requesting: ALL
#

My LDAP corporate server is working very well (I had no doubt about it to be honest) and LDAP queries are working well so from this technical point nothing forbid my MariaDB server to use it to authenticate database users.

The certificate configuration was also a pre-requisite for the nslcd daemon or you get error message like:

Nov 19 15:27:43 server1 nslcd[949124]: [8b4567]  failed to bind to LDAP server ldaps://ldap-server.company.com/: Can't contact LDAP server: error:1416F086:SSL routines:tls_process_server_certificate:certificate ver>
Nov 19 15:27:43 server1 nslcd[949124]: [8b4567]  no available LDAP server found, sleeping 1 seconds

You can also use a graphical free LDAP browser like Apache Directory Studio (http://directory.apache.org/studio/). With this tool you can automatically accept your server certificates. All search are graphical and much easier:

mariadb_pam_ldap01
mariadb_pam_ldap01

MariaDB PAM LDAP authentication with legacy nslcd configuration

For the MariaDB PAM LDAP authentication using legacy nslcd daemon configure to use pam_ldap.so library in /etc/pam.d/mariadb file:

[root@server1 ~]# cd /etc/pam.d/mariadb
auth    required pam_ldap.so
account required pam_ldap.so

Remark:
In our old configuration we had config=/etc/ldap_mariadb.conf keyword that is not working anymore…

You can add a useful debug option to have more verbose output to debug. Till you are not sure the MariaDB PAM LDAP module is effective it’s really a good idea to start in debug mode by default:

[root@server1 pam.d]# cat mariadb
auth    required pam_ldap.so debug
account required pam_ldap.so debug

As we have seen above MariaDB for PAM LDAP authentication is relying on nslcd daemon that is:

nslcd is a daemon that will do LDAP queries for local processes that want to do user, group and other naming lookups (NSS) or do user authentication, authorization or password modification (PAM).

So in /etc/nslcd.conf configuration file I configured uri and base parameters to match my corporate LDAP server and basedn:

uri ldaps://ldap-server.company.com/
base ou=employee,dc=company,dc=com

And restarted daemon with systemctl restart nslcd. To monitor what was going on you can use journalctl -xfl or tail the /var/log/secure file (you must activate the debug option of the MariaDB PAM LDAP configuration in /etc/pam.d/mariadb):.

[root@server1 etc]# journalctl -xfl
Nov 19 12:32:19 SERVER1 auth_pam_tool[960519]: pam_ldap(mariadb:auth): nslcd authentication; user=yjaquier
Nov 19 12:32:19 SERVER1 auth_pam_tool[960519]: pam_ldap(mariadb:auth): user not handled by nslcd
[root@server1 ~]# tail -f /var/log/secure
Nov 19 12:32:19 server1 auth_pam_tool[960519]: pam_ldap(mariadb:auth): nslcd authentication; user=yjaquier
Nov 19 12:32:19 server1 auth_pam_tool[960519]: pam_ldap(mariadb:auth): user not handled by nslcd

Honestly the debug information are quite poor and what has been key in my debugging session is to put nslcd service in debug mode. To do it use:

[root@server1 ~]# systemctl stop nslcd
[root@server1 ~]# /usr/sbin/nslcd -d > /tmp/nslcd.debug.log 2>&1

If you want to add a timestamp to log:

/usr/sbin/nslcd -d 2>&1 | awk '{ print strftime("%b %d %T"), $0}' > /tmp/nslcd.debug.log

With a failed authentication I got:

nslcd: [8b4567]  DEBUG: nslcd_pam_authc("yjaquier","mariadb","***")
nslcd: [8b4567]  DEBUG: myldap_search(base="ou=employee,dc=company,dc=com", filter="(&(objectClass=posixAccount)(uid=yjaquier))")
nslcd: [8b4567]  DEBUG: ldap_initialize(ldaps://ldap-server.company.com/)
nslcd: [8b4567]  DEBUG: ldap_set_rebind_proc()
nslcd: [8b4567]  DEBUG: ldap_set_option(LDAP_OPT_PROTOCOL_VERSION,3)
nslcd: [8b4567]  DEBUG: ldap_set_option(LDAP_OPT_DEREF,0)
nslcd: [8b4567]  DEBUG: ldap_set_option(LDAP_OPT_TIMELIMIT,0)
nslcd: [8b4567]  DEBUG: ldap_set_option(LDAP_OPT_TIMEOUT,0)
nslcd: [8b4567]  DEBUG: ldap_set_option(LDAP_OPT_NETWORK_TIMEOUT,0)
nslcd: [8b4567]  DEBUG: ldap_set_option(LDAP_OPT_REFERRALS,LDAP_OPT_ON)
nslcd: [8b4567]  DEBUG: ldap_set_option(LDAP_OPT_RESTART,LDAP_OPT_ON)
nslcd: [8b4567]  DEBUG: ldap_set_option(LDAP_OPT_X_TLS,LDAP_OPT_X_TLS_HARD)
nslcd: [8b4567]  DEBUG: ldap_simple_bind_s(NULL,NULL) (uri="ldaps://ldap-server.company.com/")
nslcd: [8b4567]  DEBUG: ldap_result(): end of results (0 total)
nslcd: [8b4567]  DEBUG: "yjaquier": user not found: No such object

The objectClass=posixAccount is an issue for me because we don’t have this property in our LDAP directory. So in my /etc/nslcd.conf file I have added a filter rule to change this objectClass=posixAccount by something we have in our directory (what you choose is really linked on how your LDAP server has been configured). The second option I have put in comment is also possible but I don’t like it too much:

# Yannick's filter
filter passwd (objectClass=company-login)
#filter passwd (uid=*)

And voila the next authentication trial is now working:

nslcd: [7b23c6]  DEBUG: nslcd_pam_authc("yjaquier","mariadb","***")
nslcd: [7b23c6]  DEBUG: myldap_search(base="ou=employee,dc=company,dc=com", filter="(&(objectClass=company-login)(uid=yjaquier))")
nslcd: [7b23c6]  DEBUG: ldap_initialize(ldaps://ldap-server.company.com/)
nslcd: [7b23c6]  DEBUG: ldap_set_rebind_proc()
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_PROTOCOL_VERSION,3)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_DEREF,0)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_TIMELIMIT,0)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_TIMEOUT,0)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_NETWORK_TIMEOUT,0)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_REFERRALS,LDAP_OPT_ON)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_RESTART,LDAP_OPT_ON)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_X_TLS,LDAP_OPT_X_TLS_HARD)
nslcd: [7b23c6]  DEBUG: ldap_simple_bind_s(NULL,NULL) (uri="ldaps://ldap-server.company.com/")
nslcd: [7b23c6]  DEBUG: ldap_result(): companyid=111111,ou=employee,dc=company,dc=com
nslcd: [7b23c6]  DEBUG: myldap_search(base="companyid=111111,ou=employee,dc=company,dc=com", filter="(objectClass=*)")
nslcd: [7b23c6]  DEBUG: ldap_initialize(ldaps://ldap-server.company.com/)
nslcd: [7b23c6]  DEBUG: ldap_set_rebind_proc()
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_PROTOCOL_VERSION,3)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_DEREF,0)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_TIMELIMIT,0)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_TIMEOUT,0)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_NETWORK_TIMEOUT,0)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_REFERRALS,LDAP_OPT_ON)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_RESTART,LDAP_OPT_ON)
nslcd: [7b23c6]  DEBUG: ldap_set_option(LDAP_OPT_X_TLS,LDAP_OPT_X_TLS_HARD)
nslcd: [7b23c6]  DEBUG: ldap_sasl_bind("companyid=111111,ou=employee,dc=company,dc=com","***") (uri="ldaps://ldap-server.company.com/") (ppolicy=yes)
nslcd: [7b23c6]  DEBUG: got LDAP_CONTROL_PASSWORDPOLICYRESPONSE (No error)
nslcd: [7b23c6]  DEBUG: myldap_search(base="companyid=111111,ou=employee,dc=company,dc=com", filter="(objectClass=*)")
nslcd: [7b23c6]  DEBUG: ldap_result(): companyid=111111,ou=employee,dc=company,dc=com
nslcd: [7b23c6]  DEBUG: ldap_unbind()
nslcd: [7b23c6]  DEBUG: bind successful
nslcd: [7b23c6]  DEBUG: myldap_search(base="ou=employee,dc=company,dc=com", filter="(&(objectClass=shadowAccount)(uid=yjaquier))")
nslcd: [7b23c6]  DEBUG: ldap_result(): end of results (0 total)
nslcd: [3c9869] DEBUG: connection from pid=961756 uid=0 gid=59831
nslcd: [3c9869]  DEBUG: nslcd_pam_authz("yjaquier","mariadb","","","")
nslcd: [3c9869]  DEBUG: myldap_search(base="ou=employee,dc=company,dc=com", filter="(&(objectClass=company-login)(uid=yjaquier))")
nslcd: [3c9869]  DEBUG: ldap_result(): companyid=111111,ou=employee,dc=company,dc=com
nslcd: [3c9869]  DEBUG: myldap_search(base="ou=employee,dc=company,dc=com", filter="(&(objectClass=shadowAccount)(uid=yjaquier))")
nslcd: [3c9869]  DEBUG: ldap_result(): end of results (0 total)

At MariaDB level:

[marmtrpq@server1 conf]$ /maria_10.5.9/software/10.5.9/bin/mariadb --defaults-file=/maria_10.5.9/software/10.5.9/conf/my.cnf --user=yjaquier -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 106
Server version: 10.5.9-MariaDB-log MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> select user(),current_user();
+--------------------+----------------+
| user()             | current_user() |
+--------------------+----------------+
| yjaquier@localhost | yjaquier@%     |
+--------------------+----------------+
1 row in set (0.001 sec)

MariaDB [(none)]>

MariaDB PAM LDAP authentication with System Security Services Daemon (SSSD)

For this part I had to request help from Red Hat support as the documentation and the available ressources around this new daemon are quite sparse… The sssd service should already be installed, if not do it with dnf, and the configuration files are in /etc/sssd directory.

The SSSD daemon requires an unique number to fetch into your LDAP directory. Looking at the output of various ldapseach command for my own account I have decided to use employeenumber. This attribute should be in plenty of enterprise LDAP directory. I initially planned to use our internal customized employee id but this attribute contains letter where I work so Red Hat confirmed it is not possible to use it. To test the LDAP authentication it is also strongly suggested to activate the OS authentication with LDAP to easily test the feature.

For debugging it is highly interesting to install SSSD tools with:

Install sssctl with:
[root@server1 ~]# dnf install sssd-tools

I have started by creating my own config file (/etc/sssdsssd.conf). I have kept pam_ldap as domain name to mimic that has been done with nslcd daemon. When you start your trails it is more than interesting to activate a decent debug level. For debugging it is also interesting to implement the Name Service Switch (NSS) SSSD and Pluggable Authentication Module (PAM) service modules to be able to ssh with your LDAP account locally on your server:

[root@server1 sssd]# cat sssd.conf
[sssd]
services = pam, nss
domains = pam_ldap
debug_level = 9

[domain/pam_ldap]
debug_level = 9
id_provider = ldap
auth_provider = ldap
ldap_uri = ldaps://ldap-server.company.com
ldap_search_base = ou=employee,dc=company,dc=com
ldap_user_uid_number = employeenumber
ldap_user_gid_number = employeenumber
ldap_user_object_class = company-login
ldap_user_name = uid
access_provider = permit
sudo_provider = ldap
chpass_provider = ldap
autofs_provider = ldap
resolver_provider = ldap

[pam]
debug_level = 9

[nss]
debug_level = 9

I had an error while creating my own config file:

[root@server1 sssd]# systemctl status sssd
● sssd.service - System Security Services Daemon
   Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2021-11-22 16:56:58 CET; 2min 19s ago
  Process: 1877030 ExecStart=/usr/sbin/sssd -i ${DEBUG_LOGGER} (code=exited, status=4)
 Main PID: 1877030 (code=exited, status=4)

Nov 22 16:56:58 server1 systemd[1]: sssd.service: Failed with result 'exit-code'.
Nov 22 16:56:58 server1 systemd[1]: Failed to start System Security Services Daemon.
Nov 22 16:56:58 server1 systemd[1]: sssd.service: Start request repeated too quickly.
Nov 22 16:56:58 server1 systemd[1]: sssd.service: Failed with result 'exit-code'.
Nov 22 16:56:58 server1 systemd[1]: Failed to start System Security Services Daemon.

Which can be solved with:

[root@server1 ~]# sssctl config-check
File ownership and permissions check failed. Expected root:root and 0600.

[root@server1 ~]# chmod 600 /etc/sssd/sssd.conf
[root@server1 ~]# chown root:root /etc/sssd/sssd.conf
[root@server1 ~]# sssctl config-check
Issues identified by validators: 0

Messages generated during configuration merging: 0

Used configuration snippet files: 0
[root@server1 sssd]# sssctl domain-list
implicit_files
pam_ldap

To debug the /var/log/sssd directory will become your best friend and more particularly sssd_pam_ldap.log file. You should see entries with the LDAP account you test. The LDAP search query used that can help you, with ldapsearch command, if this search query is supposed to work. As well as the result of this search command to understand if the query has returned a result or not…

Each time you change /etc/sssd/sssd.conf file use below command to stop/start SSSD daemon, purge log files and delete cached credentials:

[root@server1 sssd]# systemctl stop sssd; rm -f /var/log/sssd/*; rm -f /var/lib/sss/{db,mc}/*; systemctl start sssd; systemctl status sssd
● sssd.service - System Security Services Daemon
   Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2022-02-01 12:40:38 CET; 18ms ago
 Main PID: 4099413 (sssd)
    Tasks: 5 (limit: 100943)
   Memory: 49.3M
   CGroup: /system.slice/sssd.service
           ├─4099413 /usr/sbin/sssd -i --logger=files
           ├─4099414 /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0 --logger=files
           ├─4099415 /usr/libexec/sssd/sssd_be --domain pam_ldap --uid 0 --gid 0 --logger=files
           ├─4099416 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --logger=files
           └─4099417 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files

Feb 01 12:40:38 server1 sssd[4099413]: Starting up
Feb 01 12:40:38 server1 sssd_be[4099415]: Starting up
Feb 01 12:40:38 server1 sssd_be[4099414]: Starting up
Feb 01 12:40:38 server1 sssd_pam[4099416]: Starting up
Feb 01 12:40:38 server1 sssd_nss[4099417]: Starting up

While investigating with Red Hat support, we noticed that, there are certain attributes that can not be retrieved anonymously from our corporate LDAP directory. So, we added following options under [domain] section in sssd.conf ile:

ldap_default_bind_dn = companyid=111111,ou=employee,dc=company,dc=com
ldap_default_authtok_type = password
ldap_default_authtok = secure_password

For my initial testing I have used my own account and we then created a dedicated LDAP account with minimum privilege to be used on all our MariaDB server where LDAP authentication would be used.

Of course it is not acceptable to have password in clear in this configuration file. You have an option to obfuscate it. Change the two below parameters with those values in sssd.conf file (leave password blank):

ldap_default_authtok_type = obfuscated_password
ldap_default_authtok =

The encrypt password with sss_obfuscate tool:

[root@server1 sssd]# sss_obfuscate --domain=pam_ldap
Enter password:
Re-enter password:
[root@server1 sssd]# sss_obfuscate --domain=pam_ldap
Enter password:
Re-enter password:
[root@server1 sssd]# grep ldap_default sssd.conf
ldap_default_bind_dn = companyid=111111,ou=employee,dc=company,dc=com
ldap_default_authtok_type = obfuscated_password
ldap_default_authtok = AAAQACovNMxOkYUwdmdxA0Tgo+X7oOBikZbQ9zXFZqaWTgR0tE4EmHElZDVAJjHwa2uQTk2vHdLes0iWLXo3/JD8N3EAAQID

Last but not least activate LDAP authentication with:

[root@server1 ~]# authconfig --enableldap --enableldapauth --ldapserver="ldaps://ldap-server.company.com" --ldapbasedn="ou=employee,dc=company,dc=com" --update
Running authconfig compatibility tool.
The purpose of this tool is to enable authentication against chosen services with authselect and minimum configuration. It does not provide all capabilities of authconfig.

IMPORTANT: authconfig is replaced by authselect, please update your scripts.
See man authselect-migration(7) to help you with migration to authselect

Executing: /usr/bin/authselect check
Executing: /usr/bin/authselect select sssd --force
Executing: /usr/bin/systemctl enable sssd.service
Executing: /usr/bin/systemctl stop sssd.service
Executing: /usr/bin/systemctl start sssd.service
[root@server1 ~]# cat /etc/sysconfig/authconfig
USELDAP=yes
USELDAPAUTH=yes
[root@server1 ~]# authselect list
- nis            Enable NIS for system authentication
- sssd           Enable SSSD for system authentication (also for local users only)
- winbind        Enable winbind for system authentication
[root@server1 ~]# authselect show sssd
.
.
[root@server1 ~]# authselect list-features sssd
.
.
[root@server1 sssd]# sssctl domain-status pam_ldap
Online status: Online

Active servers:
LDAP: ldap-server.company.com

Discovered LDAP servers:
- ldap-server.company.com

At that stage, as a trial, you should be able to ssh to localhost using your LDAP account and password:

[root@server1 ~]# ssh yjaquier@localhost
yjaquier@localhost's password:
Last login: Tue Feb  1 15:08:12 2022 from 127.0.0.1
Could not chdir to home directory : No such file or directory
/usr/bin/id: cannot find name for group ID 111111
[yjaquier@server1 ~]$ id yjaquier
uid=111111(yjaquier) gid=111111 groups=111111
[yjaquier@server1 ~]$

The user creation remains exactly the same and now /etc/pam.d/mariadb file must match the new SSSD configuration:

[root@server1 ~]# cat /etc/pam.d/mariadb
auth    required pam_sss.so domains=pam_ldap
account required pam_sss.so domains=pam_ldap

And finally LDAP authentication using SSSD worked !!

References

The post MariaDB PAM LDAP authentication with newest SSSD daemon appeared first on IT World.

]]>
https://blog.yannickjaquier.com/mariadb/mariadb-pam-ldap-authentication-with-newest-sssd-daemon.html/feed 0
Database Resident Connection Pooling (DRCP) hands-on https://blog.yannickjaquier.com/oracle/database-resident-connection-pooling-drcp-hands-on.html https://blog.yannickjaquier.com/oracle/database-resident-connection-pooling-drcp-hands-on.html#respond Fri, 28 Jan 2022 08:10:07 +0000 https://blog.yannickjaquier.com/?p=5319 Preamble Database Resident Connection Pooling (DRCP) is not more than a connection pool inside the database. Connection pooling is usually a feature provided by middle tier processes like I have already done in a pure Java program or in one of the well know servlet containers / application servers like Tomcat/JBoss or else. So why […]

The post Database Resident Connection Pooling (DRCP) hands-on appeared first on IT World.

]]>

Table of contents

Preamble

Database Resident Connection Pooling (DRCP) is not more than a connection pool inside the database. Connection pooling is usually a feature provided by middle tier processes like I have already done in a pure Java program or in one of the well know servlet containers / application servers like Tomcat/JBoss or else.

So why implementing this connection pooling directly inside the database ? First to be able to use for application where it is not available, typical example provided by Oracle is web application where you often need small and quick database access to retrieve information. Ultimately we can also imagine clubbing multiple middle tier pools inside the same DRCP pool.

The clear added value is a decrease in database memory footprint as well as a decrease in CPU consumption. Each client process is using CPU ! One also very interesting added value is to keep the session open and your web application will not have the burden of creating a session each time it needs to connect to the database.

Each connection to the pool is handle by a Connection Broker that then delegate the client process to the chosen server process. Once the client disconnect or after a timeout the server process is released to the pool:

drcp01
drcp01

In 19.10 they have added two new initialization parameters to control number of processes to handle session authentication (different from the number of minimum and maximum sessions inside the pool):

SQL> show parameter min_auth_servers

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
min_auth_servers                     integer     1
SQL> show parameter max_auth_servers

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
max_auth_servers                     integer     25

In 19.11 they have added DRCP_DEDICATED_OPT parameter to have the pool behave like a dedicated server (so as a one-to-one client/server) in case you are below the maximum number of pool connection. This is again a trade between performance and resources consumption (the V$AUTHPOOL_STATS view is NOT existing, probably a bug):

SQL> show parameter drcp_dedicated_opt

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
drcp_dedicated_opt                   string      YES

My testing has been done on a 19.12 Oracle database running on a bare metal server with 12 cores and 64GB of memory.

Database Resident Connection Pooling creation

I have started to create my pool directly in the pluggable database I’m using but the message is clear, you must create the pool in the root container:

SQL> exec dbms_connection_pool.start_pool(pool_name => 'pool01');
BEGIN dbms_connection_pool.start_pool(pool_name => 'pool01'); END;

*
ERROR at line 1:
ORA-56515: DRCP: Operation not allowed from a Pluggable Database
ORA-06512: at "SYS.DBMS_CONNECTION_POOL", line 4
ORA-06512: at line 1

You also cannot (yet) change the pool name and only the default name (SYS_DEFAULT_CONNECTION_POOL) is accepted. As there is a parameter pool_name I have no doubt that one day we will be able to create multiple pools. So currently all you pooled database connection will go in same pool, cannot split them and create a bigger pool for a more important application:

SQL> exec dbms_connection_pool.start_pool(pool_name => 'pool01');
BEGIN dbms_connection_pool.start_pool(pool_name => 'pool01'); END;

*
ERROR at line 1:
ORA-56501: DRCP: Pool startup failed
ORA-56500: DRCP: Pool not found
ORA-06512: at "SYS.DBMS_CONNECTION_POOL", line 4
ORA-06512: at line 1

Before (or after) starting the pool you can modify its parameters with either DBMS_CONNECTION_POOL.ALTER_PARAM or DBMS_CONNECTION_POOL.CONFIGURE_POOL:

SQL> exec dbms_connection_pool.configure_pool(pool_name => 'SYS_DEFAULT_CONNECTION_POOL', minsize => 1, maxsize => 10);

PL/SQL procedure successfully completed.

SQL> exec dbms_connection_pool.alter_param(pool_name => 'SYS_DEFAULT_CONNECTION_POOL', param_name => 'incrsize', param_value => 1);

PL/SQL procedure successfully completed.

Thne start the pool with:

SQL> exec dbms_connection_pool.start_pool(pool_name => 'SYS_DEFAULT_CONNECTION_POOL');

PL/SQL procedure successfully completed.
SQL> set lines 200
SQL> col connection_pool for a30
SQL> select connection_pool, status, minsize, maxsize, incrsize from dba_cpool_info;

CONNECTION_POOL                STATUS              MINSIZE    MAXSIZE   INCRSIZE
------------------------------ ---------------- ---------- ---------- ----------
SYS_DEFAULT_CONNECTION_POOL    ACTIVE                    1         10          1

At Linux level it will create background processes, called connection broker to handle client request and free server processes:

[oracle@server01 ~]$ ps -ef | grep ora_l00 | grep -v grep
oracle   10516     1  0 15:24 ?        00:00:00 ora_l000_orcl
oracle   10522     1  0 15:24 ?        00:00:00 ora_l001_orcl

Database Resident Connection Pooling testing

Even if the pool has been created in the root container you can obviously use it to connect to your pluggable database. You can either use easy connect string or define an TNS string in your tnsnames.ora:

pdb1_pool =
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)
                      (HOST=server01.domain.com)
                      (PORT=1531))
                      (CONNECT_DATA=(SERVICE_NAME=pdb1)
                      (SERVER=POOLED)))

To test it I have used the Oracle client I have on my Windows desktop as well as my own account (YJAQUIER) and a test account (TEST01):

PS C:\Users\yjaquier> sqlplus yjaquier/secure_password@server01.domain.com:1531/pdb1:pooled

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Oct 6 15:58:54 2021
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Last Successful login time: Wed Oct 06 2021 15:37:45 +02:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL>

Or

PS C:\Users\yjaquier> tnsping pdb1_pool

TNS Ping Utility for 64-bit Windows: Version 19.0.0.0.0 - Production on 12-OCT-2021 15:06:55

Copyright (c) 1997, 2019, Oracle.  All rights reserved.

Used parameter files:
C:\app\client\product\19.0.0\client_1\network\admin\sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp) (HOST=server01.domain.com) (PORT=1531)) (CONNECT_DATA=(SERVICE_NAME=pdb1) (SERVER=POOLED)))
OK (110 msec)
PS C:\Users\yjaquier> sqlplus test01/test01@pdb1_pool

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Oct 12 15:06:59 2021
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Last Successful login time: Tue Oct 12 2021 14:57:58 +02:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.12.0.0.0

SQL>

Finally you have few views to follow connections and their respective pool usage (feel free to describe them to see all available columns):

SQL> set lines 200
SQL> col username for a15
SQL> col cclass_name for a15
SQL> select username, cclass_name, process_id, program, machine from v$cpool_conn_info;

USERNAME        CCLASS_NAME     PROCESS_ID               PROGRAM                                          MACHINE
--------------- --------------- ------------------------ ------------------------------------------------ ----------------------------------------------------------------
YJAQUIER        YJAQUIER.SHARED 21904:9308               sqlplus.exe                                      COMPANY\DESKTOP007
TEST01          TEST01.SHARED   9860:2396                sqlplus.exe                                      COMPANY\DESKTOP007

SQL> col pool_name for a30
SQL> select pool_name, num_open_servers, num_busy_servers, num_requests from v$cpool_stats;

POOL_NAME                      NUM_OPEN_SERVERS NUM_BUSY_SERVERS NUM_REQUESTS
------------------------------ ---------------- ---------------- ------------
SYS_DEFAULT_CONNECTION_POOL                   3                0            7

SQL> select * from v$cpool_cc_info;

POOL_NAME                      CCLASS_NAME         CON_ID
------------------------------ --------------- ----------
SYS_DEFAULT_CONNECTION_POOL    YJAQUIER.SHARED          0
SYS_DEFAULT_CONNECTION_POOL    TEST01.SHARED            0

SQL> select cclass_name, num_requests, num_hits, num_misses from v$cpool_cc_stats;

CCLASS_NAME     NUM_REQUESTS   NUM_HITS NUM_MISSES
--------------- ------------ ---------- ----------
YJAQUIER.SHARED            5          0          5
TEST01.SHARED              4          0          4

Simply stop the pool with:

SQL> exec dbms_connection_pool.stop_pool(pool_name => 'SYS_DEFAULT_CONNECTION_POOL');

PL/SQL procedure successfully completed.

References

The post Database Resident Connection Pooling (DRCP) hands-on appeared first on IT World.

]]>
https://blog.yannickjaquier.com/oracle/database-resident-connection-pooling-drcp-hands-on.html/feed 0