Kubernetes on virtual machines hands-on – part 3
Table of contents
Preamble
After the creation of our first Kubernetes Nginx stateless pod let see why stateless pod is an issue (not only) for database pod. In this third article I will focus on why you need stateful container for certain types of workload.
As it has kept me busy for some time we will also see how to create the share storage.
In next blog post we will see which Kubernetes storage plugin to use to access it.
PostgreSQL stateless deployment creation
I will obviously use the official PostgreSQL image that can be find on Docker Hub:

For this new deployment I have decided to create my own YAML file that I would be able to use over and over to add new functionalities. I have started by the one of my Nginx deployment as a skeleton:
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- image: postgres:latest
name: postgres
ports:
- containerPort: 5433
restartPolicy: Always
schedulerName: default-scheduler
Let’s load this new deployment:
[root@server1 ~]# kubectl apply -f postgres.yaml deployment.apps/postgres created [root@server1 ~]# kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE httpd 1/1 1 1 22h nginx 1/1 1 1 6d23h postgres 0/1 1 0 16s
After a while my deployement failed:
[root@server1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE httpd-757fb56c8d-7cdj5 1/1 Running 0 22h nginx-6799fc88d8-xg5kd 1/1 Running 0 23h postgres-74b5d46bcb-tvv8v 0/1 ContainerCreating 0 59s [root@server1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE httpd-757fb56c8d-7cdj5 1/1 Running 0 22h nginx-6799fc88d8-xg5kd 1/1 Running 0 23h postgres-74b5d46bcb-tvv8v 0/1 Error 2 (18s ago) 82s
Get the log with:
[root@server1 ~]# kubectl logs postgres-74b5d46bcb-tvv8v
Error: Database is uninitialized and superuser password is not specified.
You must specify POSTGRES_PASSWORD to a non-empty value for the
superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
connections without a password. This is *not* recommended.
See PostgreSQL documentation about "trust":
https://www.postgresql.org/docs/current/auth-trust.html
So added to my YAML file:
env: - name: POSTGRES_PASSWORD value: secure_password - name: POSTGRES_DB value: testdb
The options I have changed are the postgres superuser password with POSTGRES_PASSWORD and I have asked the creation of a default database called testdb with POSTGRES_DB. The Docker Hub PostgresSQL default page gives a clear explanation and a list of all possible environment variables.
Delete and re-deploy:
[root@server1 ~]# kubectl delete deployment postgres deployment.apps "postgres" deleted [root@server1 ~]# kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE httpd 1/1 1 1 22h nginx 1/1 1 1 6d23h [root@server1 ~]# kubectl apply -f postgres.yaml deployment.apps/postgres created [root@server1 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE httpd-757fb56c8d-7cdj5 1/1 Running 0 22h nginx-6799fc88d8-xg5kd 1/1 Running 0 23h postgres-6d7fcf96b5-gfpxf 1/1 Running 0 5s
You can now connect to the PostgreSQL database inside the pod with (we see that the port is not 5433 as expected but still the default 5432):
[root@server1 ~]# kubectl exec -it postgres-6d7fcf96b5-gfpxf -- /bin/bash
root@postgres-6d7fcf96b5-gfpxf:/# su - postgres
postgres@postgres-6d7fcf96b5-gfpxf:~$ psql
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
testdb | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
(4 rows)
postgres=#
Or directly from the server where is running the pod after I got the IP address of the pod:
[root@server2 ~]# psql --host=192.168.55.19 --port=5432 --username=postgres
Password for user postgres:
psql (13.4, server 14.0 (Debian 14.0-1.pgdg110+1))
WARNING: psql major version 13, server major version 14.
Some psql features might not work.
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
testdb | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
(4 rows)
postgres=#
It finally appears that the – containerPort: 5433 is not providing the expected result. The solution to change default port (5432) is:
args: ["-c", "port=5433"]
The stateless issue with database containers
Let’s create a new table in my tesdb database and insert a new row into it:
[root@server1 ~]# kubectl exec -it postgres-5594494b8f-2wsvh -- /bin/bash
root@postgres-5594494b8f-2wsvh:/# su - postgres
postgres@postgres-5594494b8f-2wsvh:~$ psql --port=5433
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
testdb | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
(4 rows)
postgres=# \c testdb
You are now connected to database "testdb" as user "postgres".
testdb=# create table test01(id integer, descr varchar(20));
CREATE TABLE
testdb=# insert into test01 values(1,'One');
INSERT 0 1
testdb=# select * from test01;
id | descr
----+-------
1 | One
(1 row)
testdb=#
Now I delete the pod, like it would happen in a real life k8s cluster. The pod is automatically recreated by the deployment and I then try to get my test table figures:
[root@server1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-757fb56c8d-7cdj5 1/1 Running 0 23h 192.168.55.17 server2.domain.com <none> <none>
nginx-6799fc88d8-xg5kd 1/1 Running 0 24h 192.168.55.16 server2.domain.com <none> <none>
postgres-5594494b8f-2wsvh 1/1 Running 0 17m 192.168.55.21 server2.domain.com <none> <none>
[root@server1 ~]# kubectl delete pod postgres-5594494b8f-2wsvh
pod "postgres-5594494b8f-2wsvh" deleted
[root@server1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
httpd-757fb56c8d-7cdj5 1/1 Running 0 23h 192.168.55.17 server2.domain.com <none> <none>
nginx-6799fc88d8-xg5kd 1/1 Running 0 24h 192.168.55.16 server2.domain.com <none> <none>
postgres-5594494b8f-p88h9 1/1 Running 0 5s 192.168.55.22 server2.domain.com <none> <none>
[root@server1 ~]# kubectl exec -it postgres-5594494b8f-p88h9 -- /bin/bash
root@postgres-5594494b8f-p88h9:/# su - postgres
postgres@postgres-5594494b8f-p88h9:~$ psql --port=5433 --dbname=testdb
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.
testdb=# select * from test01;
ERROR: relation "test01" does not exist
LINE 1: select * from test01;
^
testdb=#
Oups, well as expected I would say, the information has gone and again in the case of a database this is clearly not acceptable. To make persistent the content we need to work a little more with persistent volume and persistent volume claim.
Creation of the cluster filesystem between Kubernetes nodes
Kubernetes has a lot of available persistent volumes plugins to allow you to mount an incredible number of different storage types on your k8s nodes. On my trial k8s cluster made of two virtual machines I have decided to use a shared disk same as we have already seen in my Oracle Real Application Cluster (RAC) configuration trial. Once this shared storage and cluster filesystem will be created the idea is to use the local Kubernetes storage plugin.
Once I have attached the shared disk let’s create a new partition:
[root@server1 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xde5a83e7.
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p):
Using default response p.
Partition number (1-4, default 1):
First sector (2048-2097151, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-2097151, default 2097151):
Created a new partition 1 of type 'Linux' and of size 1023 MiB.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
[root@server1 ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 : 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xde5a83e7
Device Boot Start End Sectors Size Id Type
/dev/sdb1 2048 2097151 2095104 1023M 83 Linux
[root@server1 ~]# mkfs -t xfs /dev/sdb1
meta-data=/dev/sdb1 isize=512 agcount=4, agsize=65472 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=261888, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=1566, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
On second node to make the shared partition available I had to trick a bit fdisk:
[root@server2 ~]# blkid /dev/sdb1 [root@server2 ~]# fdisk /dev/sdb Welcome to fdisk (util-linux 2.32.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): w The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks.
Then as I will not have Oracle ASM you have to decide to use a shared filesystem…
GFS2
This cluster FS is sponsored by RedHat and installation is simple as:
[root@server1 ~]# dnf install gfs2-utils.x86_64
Then create the FS with:
[root@server1 ~]# mkfs -t gfs2 -p lock_dlm -t cluster01:postgres -j 8 /dev/sdb1 It appears to contain an existing filesystem (xfs) This will destroy any data on /dev/sdb1 Are you sure you want to proceed? [y/n] y Discarding device contents (may take a while on large devices): Done Adding journals: Done Building resource groups: Done Creating quota file: Done Writing superblock and syncing: Done Device: /dev/sdb1 Block size: 4096 Device size: 1.00 GB (261888 blocks) Filesystem size: 1.00 GB (261886 blocks) Journals: 8 Journal size: 8MB Resource groups: 12 Locking protocol: "lock_dlm" Lock table: "cluster01:postgres" UUID: 90f456e8-cf74-43af-a838-53b129682f7d
But when I tried to mount it I got:
[root@server1 ~]# mount -a mount: /mnt/shared: mount(2) system call failed: Transport endpoint is not connected.
From the RedHat official solution I have discovered that lock_dlm modules was not loaded:
[root@server1 ~]# lsmod |grep lock [root@server1 ~]# modprobe lock_dlm modprobe: FATAL: Module lock_dlm not found in directory /lib/modules/4.18.0-305.19.1.el8_4.x86_64 [root@server1 ~]# dmesg | grep gfs [ 8255.885092] gfs2: GFS2 installed [ 8255.896751] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres" [ 8255.899671] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107 [ 8291.542025] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres" [ 8291.542186] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107 [ 8376.146357] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres" [ 8376.156197] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107 [ 8442.132982] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres" [ 8442.137871] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107 [12479.923651] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres" [12479.924713] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107 [12861.644565] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres" [12861.644663] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107 [13016.278584] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres" [13016.279004] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107 [13042.852965] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres" [13042.866282] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107 [13362.619425] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres" [13362.631850] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
I have tried to install kernel extra modules:
[root@server1 ~]# dnf install kernel-modules-extra.x86_64
To finally realized that GFS2 is delivered as extra cost add-ons in Red Hat Enterprise Linux, such as the High Availability Add-On for clustering and the Resilient Storage Add-On for GFS2. I thought the FS was free but apparently GFS was free but GFS2 is not…
OCFS2
As I’m using the Oracle Linux OCFS2 sounds like a good idea and for sure OCFS2 is released under the GNU General Public License. Install it with:
[root@server1 ~]# dnf install ocfs2-tools.x86_64
On one of your node create the OCFS2 cluster with:
[root@server1 ~]# o2cb add-cluster k8socfs2
[root@server1 ~]# o2cb add-node --ip 192.168.56.101 --port 7777 --number 1 k8socfs2 server1.domain.com
[root@server1 ~]# o2cb add-node --ip 192.168.56.102 --port 7777 --number 2 k8socfs2 server2.domain.com
[root@server1 ~]# o2cb register-cluster k8socfs2
[root@server1 ~]# o2cb start-heartbeat k8socfs2
[root@server1 ~]# cat /etc/ocfs2/cluster.conf
cluster:
heartbeat_mode = local
node_count = 2
name = k8socfs2
node:
number = 1
cluster = k8socfs2
ip_port = 7777
ip_address = 192.168.56.101
name = server1.domain.com
node:
number = 2
cluster = k8socfs2
ip_port = 7777
ip_address = 192.168.56.102
name = server2.domain.com
Copy this configuration file (/etc/ocfs2/cluster.conf) on all nodes of your OCFS2 cluster.
Initialize the OCFS2 cluster stack (O2CB) with:
[root@server1 ~]# o2cb.init --help
Usage: /usr/sbin/o2cb.init {start|stop|restart|force-reload|enable|disable|configure|load|unload|online|offline|force-offline|status|online-status}
[root@server1 /]# o2cb.init configure
Configuring the O2CB driver.
This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting
<enter> without typing an answer will keep that current value. Ctrl-C
will abort.
Load O2CB driver on boot (y/n) [n]: y
Cluster stack backing O2CB [o2cb]:
Cluster to start on boot (Enter "none" to clear) [ocfs2]: k8socfs2
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
checking debugfs...
Loading filesystem "ocfs2_dlmfs": Unable to load filesystem "ocfs2_dlmfs"
Failed
[root@server1 /]#
[root@server1 /]# lsmod |egrep -i "ocfs|o2"
[root@server1 /]# modprobe ocfs2_dlmfs
modprobe: FATAL: Module ocfs2_dlmfs not found in directory /lib/modules/4.18.0-305.19.1.el8_4.x86_64
[root@server1 /]# o2cb.init status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Driver for "ocfs2_dlmfs": Not loaded
Checking O2CB cluster "ociocfs2": Offline
stat: cannot read file system information for '/dlm': No such file or directory
Debug file system at /sys/kernel/debug: mounted
I realized that all issues I had (including mount.ocfs2: Unable to access cluster service while trying initialize cluster) were linked to the kernel I was using. Not the UEK Oracle kernel. All issues have been resolved at the moment I switch to Oracle UEK kernel !
Do not forget to start and enable o2cb service with:
[root@server2 ~]# systemctl status o2cb ● o2cb.service - Load o2cb Modules Loaded: loaded (/usr/lib/systemd/system/o2cb.service; disabled; vendor preset: disabled) Active: inactive (dead) [root@server2 ~]# systemctl start o2cb [root@server2 ~]# systemctl status o2cb ● o2cb.service - Load o2cb Modules Loaded: loaded (/usr/lib/systemd/system/o2cb.service; disabled; vendor preset: disabled) Active: active (exited) since Mon 2021-10-18 14:04:15 CEST; 1s ago Process: 73099 ExecStart=/sbin/o2cb.init enable (code=exited, status=0/SUCCESS) Main PID: 73099 (code=exited, status=0/SUCCESS) Oct 18 14:04:14 server2.domain.com systemd[1]: Starting Load o2cb Modules... Oct 18 14:04:15 server2.domain.com o2cb.init[73099]: checking debugfs... Oct 18 14:04:15 server2.domain.com o2cb.init[73099]: Setting cluster stack "o2cb": OK Oct 18 14:04:15 server2.domain.com o2cb.init[73099]: Cluster ociocfs2 already online Oct 18 14:04:15 server2.domain.com systemd[1]: Started Load o2cb Modules. [root@server2 ~]# systemctl enable o2cb Created symlink /etc/systemd/system/multi-user.target.wants/o2cb.service → /usr/lib/systemd/system/o2cb.service.
Create the FS with:
[root@server1 ~]# man mkfs.ocfs2 [root@server1 ~]# mkfs -t ocfs2 --cluster-name=k8socfs2 --fs-feature-level=max-features --cluster-stack=o2cb -N 4 /dev/sdb1 mkfs.ocfs2 1.8.6 Cluster stack: o2cb Cluster name: k8socfs2 Stack Flags: 0x0 NOTE: Feature extended slot map may be enabled Overwriting existing ocfs2 partition. Proceed (y/N): y Label: Features: sparse extended-slotmap backup-super unwritten inline-data strict-journal-super metaecc xattr indexed-dirs usrquota grpquota refcount discontig-bg Block size: 2048 (11 bits) Cluster size: 4096 (12 bits) Volume size: 1072693248 (261888 clusters) (523776 blocks) Cluster groups: 17 (tail covers 7936 clusters, rest cover 15872 clusters) Extent allocator size: 4194304 (1 groups) Journal size: 33554432 Node slots: 4 Creating bitmaps: done Initializing superblock: done Writing system files: done Writing superblock: done Writing backup superblock: 0 block(s) Formatting Journals: done Growing extent allocator: done Formatting slot map: done Formatting quota files: done Writing lost+found: done mkfs.ocfs2 successful
Get the id of the device with:
[root@server1 ~]# blkid /dev/sdb1 /dev/sdb1: UUID="ea6e9804-105d-4d4c-96e8-bd54ab5e93d2" BLOCK_SIZE="2048" TYPE="ocfs2" PARTUUID="de5a83e7-01" [root@server1 ~]# echo "ea6e9804-105d-4d4c-96e8-bd54ab5e93d2" >> /etc/fstab [root@server1 ~]# vi /etc/fstab [root@server1 ~]# tail -n 2 /etc/fstab # Shared storage UUID="ea6e9804-105d-4d4c-96e8-bd54ab5e93d2" /mnt/shared ocfs2 defaults 0 0 [root@server1 ~]# mount -a
With Oracle UEK kernel I now have:
[root@server3 postgres]# o2cb.init status Driver for "configfs": Loaded Filesystem "configfs": Mounted Stack glue driver: Loaded Stack plugin "o2cb": Loaded Driver for "ocfs2_dlmfs": Loaded Filesystem "ocfs2_dlmfs": Mounted Checking O2CB cluster "ociocfs2": Online Heartbeat dead threshold: 31 Network idle timeout: 30000 Network keepalive delay: 2000 Network reconnect delay: 2000 Heartbeat mode: Local Checking O2CB heartbeat: Active Debug file system at /sys/kernel/debug: mounted [root@server3 ~]# df /mnt/shared Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdb1 1047552 78956 968596 8% /mnt/shared
And I can share file from all nodes of my k8s cluster…
I had an issue where the FS was automatically unmounted by the system:
Oct 18 15:10:50 server1 kernel: o2dlm: Joining domain E80332E942C649EB942623C43D2B35DC Oct 18 15:10:50 server1 kernel: ( Oct 18 15:10:50 server1 kernel: 1 Oct 18 15:10:50 server1 kernel: ) 1 nodes Oct 18 15:10:50 server1 kernel: ocfs2: Mounting device (8,17) on (node 1, slot 0) with ordered data mode. Oct 18 15:10:50 server1 systemd[1]: mnt-postgres.mount: Unit is bound to inactive unit dev-disk-by\x2duuid-b7f61498\x2da4f0\x2d4570\x2da0ed\x2dcb50caa98165.device. Stopping, too. Oct 18 15:10:50 server1 systemd[1]: Unmounting /mnt/shared... Oct 18 15:10:50 server1 systemd[10949]: mnt-postgres.mount: Succeeded.
Solved with:
[root@server1 /]# systemctl daemon-reload
I also had an issue where o2cb was not able to register my cluster name:
[root@server2 ~]# o2cb register-cluster k8socfs2 o2cb: Internal logic failure while registering cluster 'k8socfs2'
Old trials not completely removed and solved it with:
[root@server2 /]# o2cb cluster-status Cluster 'ociocfs2' is online [root@server2 /]# o2cb unregister-cluster ociocfs2
References
- Docker PostgreSQL Official Image
- A Simple Guide to Oracle Cluster File System (OCFS2) using iSCSI on Oracle Cloud Infrastructure
- Creating the Configuration File for the Cluster Stack
- GFS2 file system creation





