Kubernetes on virtual machines hands-on – part 3

Preamble

After the creation of our first Kubernetes Nginx stateless pod let see why stateless pod is an issue (not only) for database pod. In this third article I will focus on why you need stateful container for certain types of workload.

As it has kept me busy for some time we will also see how to create the share storage.

In next blog post we will see which Kubernetes storage plugin to use to access it.

PostgreSQL stateless deployment creation

I will obviously use the official PostgreSQL image that can be find on Docker Hub:

kubernetes07
kubernetes07

For this new deployment I have decided to create my own YAML file that I would be able to use over and over to add new functionalities. I have started by the one of my Nginx deployment as a skeleton:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - image: postgres:latest
        name: postgres
        ports:
        - containerPort: 5433
      restartPolicy: Always
      schedulerName: default-scheduler

Let’s load this new deployment:

[root@server1 ~]# kubectl apply -f postgres.yaml
deployment.apps/postgres created
[root@server1 ~]# kubectl get deployment
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
httpd      1/1     1            1           22h
nginx      1/1     1            1           6d23h
postgres   0/1     1            0           16s

After a while my deployement failed:

[root@server1 ~]# kubectl get pod
NAME                        READY   STATUS              RESTARTS   AGE
httpd-757fb56c8d-7cdj5      1/1     Running             0          22h
nginx-6799fc88d8-xg5kd      1/1     Running             0          23h
postgres-74b5d46bcb-tvv8v   0/1     ContainerCreating   0          59s
[root@server1 ~]# kubectl get pod
NAME                        READY   STATUS    RESTARTS      AGE
httpd-757fb56c8d-7cdj5      1/1     Running   0             22h
nginx-6799fc88d8-xg5kd      1/1     Running   0             23h
postgres-74b5d46bcb-tvv8v   0/1     Error     2 (18s ago)   82s

Get the log with:

[root@server1 ~]# kubectl logs postgres-74b5d46bcb-tvv8v
Error: Database is uninitialized and superuser password is not specified.
       You must specify POSTGRES_PASSWORD to a non-empty value for the
       superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
 
       You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
       connections without a password. This is *not* recommended.
 
       See PostgreSQL documentation about "trust":
       https://www.postgresql.org/docs/current/auth-trust.html

So added to my YAML file:

env:
- name: POSTGRES_PASSWORD
  value: secure_password
- name: POSTGRES_DB
  value: testdb

The options I have changed are the postgres superuser password with POSTGRES_PASSWORD and I have asked the creation of a default database called testdb with POSTGRES_DB. The Docker Hub PostgresSQL default page gives a clear explanation and a list of all possible environment variables.

Delete and re-deploy:

[root@server1 ~]# kubectl delete deployment postgres
deployment.apps "postgres" deleted
[root@server1 ~]# kubectl get deployment
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
httpd   1/1     1            1           22h
nginx   1/1     1            1           6d23h
[root@server1 ~]# kubectl apply -f postgres.yaml
deployment.apps/postgres created
[root@server1 ~]# kubectl get pod
NAME                        READY   STATUS    RESTARTS   AGE
httpd-757fb56c8d-7cdj5      1/1     Running   0          22h
nginx-6799fc88d8-xg5kd      1/1     Running   0          23h
postgres-6d7fcf96b5-gfpxf   1/1     Running   0          5s

You can now connect to the PostgreSQL database inside the pod with (we see that the port is not 5433 as expected but still the default 5432):

[root@server1 ~]# kubectl exec -it postgres-6d7fcf96b5-gfpxf -- /bin/bash
root@postgres-6d7fcf96b5-gfpxf:/# su - postgres
postgres@postgres-6d7fcf96b5-gfpxf:~$ psql
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.
 
postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 testdb    | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
(4 rows)
 
postgres=#

Or directly from the server where is running the pod after I got the IP address of the pod:

[root@server2 ~]# psql --host=192.168.55.19 --port=5432 --username=postgres
Password for user postgres:
psql (13.4, server 14.0 (Debian 14.0-1.pgdg110+1))
WARNING: psql major version 13, server major version 14.
         Some psql features might not work.
Type "help" for help.
 
postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 testdb    | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
(4 rows)
 
postgres=#

It finally appears that the – containerPort: 5433 is not providing the expected result. The solution to change default port (5432) is:

args: ["-c", "port=5433"]

The stateless issue with database containers

Let’s create a new table in my tesdb database and insert a new row into it:

[root@server1 ~]# kubectl exec -it postgres-5594494b8f-2wsvh -- /bin/bash
root@postgres-5594494b8f-2wsvh:/# su - postgres
postgres@postgres-5594494b8f-2wsvh:~$ psql --port=5433
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.
 
postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 testdb    | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
(4 rows)
 
postgres=# \c testdb
You are now connected to database "testdb" as user "postgres".
testdb=# create table test01(id integer, descr varchar(20));
CREATE TABLE
testdb=# insert into test01 values(1,'One');
INSERT 0 1
testdb=# select * from test01;
 id | descr
----+-------
  1 | One
(1 row)
 
testdb=#

Now I delete the pod, like it would happen in a real life k8s cluster. The pod is automatically recreated by the deployment and I then try to get my test table figures:

[root@server1 ~]# kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP              NODE                 NOMINATED NODE   READINESS GATES
httpd-757fb56c8d-7cdj5      1/1     Running   0          23h   192.168.55.17   server2.domain.com   <none>           <none>
nginx-6799fc88d8-xg5kd      1/1     Running   0          24h   192.168.55.16   server2.domain.com   <none>           <none>
postgres-5594494b8f-2wsvh   1/1     Running   0          17m   192.168.55.21   server2.domain.com   <none>           <none>
[root@server1 ~]# kubectl delete pod postgres-5594494b8f-2wsvh
pod "postgres-5594494b8f-2wsvh" deleted
[root@server1 ~]# kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP              NODE                 NOMINATED NODE   READINESS GATES
httpd-757fb56c8d-7cdj5      1/1     Running   0          23h   192.168.55.17   server2.domain.com   <none>           <none>
nginx-6799fc88d8-xg5kd      1/1     Running   0          24h   192.168.55.16   server2.domain.com   <none>           <none>
postgres-5594494b8f-p88h9   1/1     Running   0          5s    192.168.55.22   server2.domain.com   <none>           <none>
[root@server1 ~]# kubectl exec -it postgres-5594494b8f-p88h9 -- /bin/bash
root@postgres-5594494b8f-p88h9:/# su - postgres
postgres@postgres-5594494b8f-p88h9:~$ psql --port=5433 --dbname=testdb
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.
 
testdb=# select * from test01;
ERROR:  relation "test01" does not exist
LINE 1: select * from test01;
                      ^
testdb=#

Oups, well as expected I would say, the information has gone and again in the case of a database this is clearly not acceptable. To make persistent the content we need to work a little more with persistent volume and persistent volume claim.

Creation of the cluster filesystem between Kubernetes nodes

Kubernetes has a lot of available persistent volumes plugins to allow you to mount an incredible number of different storage types on your k8s nodes. On my trial k8s cluster made of two virtual machines I have decided to use a shared disk same as we have already seen in my Oracle Real Application Cluster (RAC) configuration trial. Once this shared storage and cluster filesystem will be created the idea is to use the local Kubernetes storage plugin.

Once I have attached the shared disk let’s create a new partition:

[root@server1 ~]# fdisk /dev/sdb
 
Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
 
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xde5a83e7.
 
Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p):
 
Using default response p.
Partition number (1-4, default 1):
First sector (2048-2097151, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-2097151, default 2097151):
 
Created a new partition 1 of type 'Linux' and of size 1023 MiB.
 
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
 
[root@server1 ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xde5a83e7
 
Device     Boot Start     End Sectors  Size Id Type
/dev/sdb1        2048 2097151 2095104 1023M 83 Linux
[root@server1 ~]# mkfs -t xfs /dev/sdb1
meta-data=/dev/sdb1              isize=512    agcount=4, agsize=65472 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=261888, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=1566, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

On second node to make the shared partition available I had to trick a bit fdisk:

[root@server2 ~]# blkid /dev/sdb1
[root@server2 ~]# fdisk /dev/sdb
 
Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
 
 
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Then as I will not have Oracle ASM you have to decide to use a shared filesystem…

GFS2

This cluster FS is sponsored by RedHat and installation is simple as:

[root@server1 ~]# dnf install gfs2-utils.x86_64

Then create the FS with:

[root@server1 ~]# mkfs -t gfs2 -p lock_dlm -t cluster01:postgres -j 8 /dev/sdb1
It appears to contain an existing filesystem (xfs)
This will destroy any data on /dev/sdb1
Are you sure you want to proceed? [y/n] y
Discarding device contents (may take a while on large devices): Done
Adding journals: Done
Building resource groups: Done
Creating quota file: Done
Writing superblock and syncing: Done
Device:                    /dev/sdb1
Block size:                4096
Device size:               1.00 GB (261888 blocks)
Filesystem size:           1.00 GB (261886 blocks)
Journals:                  8
Journal size:              8MB
Resource groups:           12
Locking protocol:          "lock_dlm"
Lock table:                "cluster01:postgres"
UUID:                      90f456e8-cf74-43af-a838-53b129682f7d

But when I tried to mount it I got:

[root@server1 ~]# mount -a
mount: /mnt/shared: mount(2) system call failed: Transport endpoint is not connected.

From the RedHat official solution I have discovered that lock_dlm modules was not loaded:

[root@server1 ~]# lsmod |grep lock
[root@server1 ~]# modprobe lock_dlm
modprobe: FATAL: Module lock_dlm not found in directory /lib/modules/4.18.0-305.19.1.el8_4.x86_64
[root@server1 ~]# dmesg | grep gfs
[ 8255.885092] gfs2: GFS2 installed
[ 8255.896751] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[ 8255.899671] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[ 8291.542025] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[ 8291.542186] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[ 8376.146357] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[ 8376.156197] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[ 8442.132982] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[ 8442.137871] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[12479.923651] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[12479.924713] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[12861.644565] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[12861.644663] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[13016.278584] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[13016.279004] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[13042.852965] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[13042.866282] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107
[13362.619425] gfs2: fsid=cluster01:postgres: Trying to join cluster "lock_dlm", "cluster01:postgres"
[13362.631850] gfs2: fsid=cluster01:postgres: dlm_new_lockspace error -107

I have tried to install kernel extra modules:

[root@server1 ~]# dnf install kernel-modules-extra.x86_64

To finally realized that GFS2 is delivered as extra cost add-ons in Red Hat Enterprise Linux, such as the High Availability Add-On for clustering and the Resilient Storage Add-On for GFS2. I thought the FS was free but apparently GFS was free but GFS2 is not…

OCFS2

As I’m using the Oracle Linux OCFS2 sounds like a good idea and for sure OCFS2 is released under the GNU General Public License. Install it with:

[root@server1 ~]# dnf install ocfs2-tools.x86_64

On one of your node create the OCFS2 cluster with:

[root@server1 ~]# o2cb add-cluster k8socfs2
[root@server1 ~]# o2cb add-node --ip 192.168.56.101 --port 7777 --number 1 k8socfs2 server1.domain.com
[root@server1 ~]# o2cb add-node --ip 192.168.56.102 --port 7777 --number 2 k8socfs2 server2.domain.com
[root@server1 ~]# o2cb register-cluster k8socfs2
[root@server1 ~]# o2cb start-heartbeat k8socfs2
[root@server1 ~]# cat /etc/ocfs2/cluster.conf
cluster:
        heartbeat_mode = local
        node_count = 2
        name = k8socfs2
 
node:
        number = 1
        cluster = k8socfs2
        ip_port = 7777
        ip_address = 192.168.56.101
        name = server1.domain.com
 
node:
        number = 2
        cluster = k8socfs2
        ip_port = 7777
        ip_address = 192.168.56.102
        name = server2.domain.com

Copy this configuration file (/etc/ocfs2/cluster.conf) on all nodes of your OCFS2 cluster.

Initialize the OCFS2 cluster stack (O2CB) with:

[root@server1 ~]# o2cb.init --help
Usage: /usr/sbin/o2cb.init {start|stop|restart|force-reload|enable|disable|configure|load|unload|online|offline|force-offline|status|online-status}
[root@server1 /]# o2cb.init configure
Configuring the O2CB driver.
 
This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot.  The current values will be shown in brackets ('[]').  Hitting
<ENTER> without typing an answer will keep that current value.  Ctrl-C
will abort.
 
Load O2CB driver on boot (y/n) [n]: y
Cluster stack backing O2CB [o2cb]:
Cluster to start on boot (Enter "none" to clear) [ocfs2]: k8socfs2
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
checking debugfs...
Loading filesystem "ocfs2_dlmfs": Unable to load filesystem "ocfs2_dlmfs"
Failed
[root@server1 /]#
[root@server1 /]# lsmod |egrep -i "ocfs|o2"
[root@server1 /]# modprobe ocfs2_dlmfs
modprobe: FATAL: Module ocfs2_dlmfs not found in directory /lib/modules/4.18.0-305.19.1.el8_4.x86_64
[root@server1 /]# o2cb.init status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Driver for "ocfs2_dlmfs": Not loaded
Checking O2CB cluster "ociocfs2": Offline
stat: cannot read file system information for '/dlm': No such file or directory
Debug file system at /sys/kernel/debug: mounted

I realized that all issues I had (including mount.ocfs2: Unable to access cluster service while trying initialize cluster) were linked to the kernel I was using. Not the UEK Oracle kernel. All issues have been resolved at the moment I switch to Oracle UEK kernel !

Do not forget to start and enable o2cb service with:

[root@server2 ~]# systemctl status o2cb
● o2cb.service - Load o2cb Modules
   Loaded: loaded (/usr/lib/systemd/system/o2cb.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
[root@server2 ~]# systemctl start o2cb
[root@server2 ~]# systemctl status o2cb
● o2cb.service - Load o2cb Modules
   Loaded: loaded (/usr/lib/systemd/system/o2cb.service; disabled; vendor preset: disabled)
   Active: active (exited) since Mon 2021-10-18 14:04:15 CEST; 1s ago
  Process: 73099 ExecStart=/sbin/o2cb.init enable (code=exited, status=0/SUCCESS)
 Main PID: 73099 (code=exited, status=0/SUCCESS)
 
Oct 18 14:04:14 server2.domain.com systemd[1]: Starting Load o2cb Modules...
Oct 18 14:04:15 server2.domain.com o2cb.init[73099]: checking debugfs...
Oct 18 14:04:15 server2.domain.com o2cb.init[73099]: Setting cluster stack "o2cb": OK
Oct 18 14:04:15 server2.domain.com o2cb.init[73099]: Cluster ociocfs2 already online
Oct 18 14:04:15 server2.domain.com systemd[1]: Started Load o2cb Modules.
[root@server2 ~]# systemctl enable o2cb
Created symlink /etc/systemd/system/multi-user.target.wants/o2cb.service → /usr/lib/systemd/system/o2cb.service.

Create the FS with:

[root@server1 ~]# man mkfs.ocfs2
[root@server1 ~]# mkfs -t ocfs2 --cluster-name=k8socfs2 --fs-feature-level=max-features --cluster-stack=o2cb -N 4 /dev/sdb1
mkfs.ocfs2 1.8.6
Cluster stack: o2cb
Cluster name: k8socfs2
Stack Flags: 0x0
NOTE: Feature extended slot map may be enabled
Overwriting existing ocfs2 partition.
Proceed (y/N): y
Label:
Features: sparse extended-slotmap backup-super unwritten inline-data strict-journal-super metaecc xattr indexed-dirs usrquota grpquota refcount discontig-bg
Block size: 2048 (11 bits)
Cluster size: 4096 (12 bits)
Volume size: 1072693248 (261888 clusters) (523776 blocks)
Cluster groups: 17 (tail covers 7936 clusters, rest cover 15872 clusters)
Extent allocator size: 4194304 (1 groups)
Journal size: 33554432
Node slots: 4
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 0 block(s)
Formatting Journals: done
Growing extent allocator: done
Formatting slot map: done
Formatting quota files: done
Writing lost+found: done
mkfs.ocfs2 successful

Get the id of the device with:

[root@server1 ~]# blkid /dev/sdb1
/dev/sdb1: UUID="ea6e9804-105d-4d4c-96e8-bd54ab5e93d2" BLOCK_SIZE="2048" TYPE="ocfs2" PARTUUID="de5a83e7-01"
[root@server1 ~]# echo "ea6e9804-105d-4d4c-96e8-bd54ab5e93d2" >> /etc/fstab
[root@server1 ~]# vi /etc/fstab
[root@server1 ~]# tail -n 2 /etc/fstab
# Shared storage
UUID="ea6e9804-105d-4d4c-96e8-bd54ab5e93d2"     /mnt/shared      ocfs2    defaults     0 0
[root@server1 ~]# mount -a

With Oracle UEK kernel I now have:

[root@server3 postgres]# o2cb.init status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster "ociocfs2": Online
  Heartbeat dead threshold: 31
  Network idle timeout: 30000
  Network keepalive delay: 2000
  Network reconnect delay: 2000
  Heartbeat mode: Local
Checking O2CB heartbeat: Active
Debug file system at /sys/kernel/debug: mounted
[root@server3 ~]# df /mnt/shared
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/sdb1        1047552 78956    968596   8% /mnt/shared

And I can share file from all nodes of my k8s cluster…

I had an issue where the FS was automatically unmounted by the system:

Oct 18 15:10:50 server1 kernel: o2dlm: Joining domain E80332E942C649EB942623C43D2B35DC
Oct 18 15:10:50 server1 kernel: (
Oct 18 15:10:50 server1 kernel: 1
Oct 18 15:10:50 server1 kernel: ) 1 nodes
Oct 18 15:10:50 server1 kernel: ocfs2: Mounting device (8,17) on (node 1, slot 0) with ordered data mode.
Oct 18 15:10:50 server1 systemd[1]: mnt-postgres.mount: Unit is bound to inactive unit dev-disk-by\x2duuid-b7f61498\x2da4f0\x2d4570\x2da0ed\x2dcb50caa98165.device. Stopping, too.
Oct 18 15:10:50 server1 systemd[1]: Unmounting /mnt/shared...
Oct 18 15:10:50 server1 systemd[10949]: mnt-postgres.mount: Succeeded.

Solved with:

[root@server1 /]# systemctl daemon-reload

I also had an issue where o2cb was not able to register my cluster name:

[root@server2 ~]# o2cb register-cluster k8socfs2
o2cb: Internal logic failure while registering cluster 'k8socfs2'

Old trials not completely removed and solved it with:

[root@server2 /]# o2cb cluster-status
Cluster 'ociocfs2' is online
[root@server2 /]# o2cb unregister-cluster ociocfs2

References

Yannick Jaquier on LinkedinYannick Jaquier on RssYannick Jaquier on Twitter
Yannick Jaquier
Find more about me on social media.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>