Table of contents
Preamble
I have already played multiple time with OpenStack the now world famous public and private cloud control open source software.
I have never posted any post on it mainly because any try I have done have completed on a failure. The product itself is made of lots of components and as it is evolving very rapidly you often hit a situation where the latest release of of product is no more compatible with the others… At least all the time I have already spent on it have helped me to conclude this ultimate testing…
The only situation where I succeeded to have it working is when I downloaded a prepared VirtualBox image, easy ins’t it ?
So when I saw the Oracle announcement on their latest integration of OpenStack in their Oracle Enterprise Linux I have been quite exited to see how it could work and was already thinking of a 10 minutes working installation:
New Oracle OpenStack for Oracle Linux Now Available
First commercially available OpenStack implementation completely packaged as Docker instances
Yes they have packages all OpenStack components in Docker containers: how could it be nicer and simpler to implement.
Testing has been done using Oracle Enterprise Linux 7.2 64 bits and Oracle® OpenStack for Oracle Linux 2.0.2. I allocated 4 cores and 6GB RAM to this virtual machine. As it is just for testing I will deploy an all-in-one unique OpenStack server. My server will be called server1.domain.com with non-routable IP address 192.168.56.101. In fact I have created my guest with three LAN card, one for administrative connection, one bridged to be able to use Internet and a third one requested by Neutron component:
[root@server1 ~]# ip addr list 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 08:00:27:bc:4d:c3 brd ff:ff:ff:ff:ff:ff inet 192.168.56.101/24 brd 192.168.56.255 scope global enp0s3 valid_lft forever preferred_lft forever inet6 fe80::a00:27ff:febc:4dc3/64 scope link valid_lft forever preferred_lft forever 3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000 link/ether 08:00:27:20:2e:d6 brd ff:ff:ff:ff:ff:ff inet 10.10.10.10/24 brd 10.70.21.255 scope global dynamic enp0s8 valid_lft 338090sec preferred_lft 338090sec inet6 fe80::a00:27ff:fe20:2ed6/64 scope link valid_lft forever preferred_lft forever 4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovs-system state UP qlen 1000 link/ether 08:00:27:01:85:c5 brd ff:ff:ff:ff:ff:ff . . |
As the post would be really two long I have decided to split is in two parts: part1 and part2.
Installation
Preparing Oracle Linux Target Nodes
The sentence in installation guide is not for me 100% clear but you have to create /var/lib/docker (and /var/lib/registry) as a btrfs filesystems:
[root@server1 /]# lvcreate -n lvol10 -L 15G vg00 Logical volume "lvol10" created. [root@server1 ~]# lvcreate -n lvol11 -L 15G vg00 Logical volume "lvol11" created. [root@server1 ~]# mkfs -t btrfs /dev/vg00/lvol10 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information. Label: (null) UUID: 4154667b-f8d1-4105-9c21-d1373ec0072b Node size: 16384 Sector size: 4096 Filesystem size: 15.00GiB Block group profiles: Data: single 8.00MiB Metadata: DUP 1.01GiB System: DUP 12.00MiB SSD detected: no Incompat features: extref Number of devices: 1 Devices: ID SIZE PATH 1 15.00GiB /dev/vg00/lvol10 [root@server1 ~]# mkfs -t btrfs /dev/vg00/lvol11 btrfs-progs v4.2.2 See http://btrfs.wiki.kernel.org for more information. Label: (null) UUID: 2295fad1-f164-42f5-8e19-bedaef0c290d Node size: 16384 Sector size: 4096 Filesystem size: 15.00GiB Block group profiles: Data: single 8.00MiB Metadata: DUP 1.01GiB System: DUP 12.00MiB SSD detected: no Incompat features: extref Number of devices: 1 Devices: ID SIZE PATH 1 15.00GiB /dev/vg00/lvol11 |
Edit /etc/fstab and mount the recently created logical volume:
[root@server1 ~]# grep /var/lib /etc/fstab /dev/mapper/vg00-lvol10 /var/lib/docker btrfs defaults 0 0 /dev/mapper/vg00-lvol11 /var/lib/registry btrfs defaults 0 0 [root@server1 ~]# mkdir -p /var/lib/docker /var/lib/registry [root@server1 lib]# mount -a [root@server1 ~]# df |grep /var/lib /dev/mapper/vg00-lvol10 15728640 16896 13613824 1% /var/lib/docker /dev/mapper/vg00-lvol11 15728640 16896 13613824 1% /var/lib/registry |
If you do not do it (my first attempt where I did not understood well what to do) Docker service will fail with:
[root@server1 docker]# systemctl status docker.service -l ● docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/docker.service.d └─docker-sysconfig.conf Active: failed (Result: exit-code) since Fri 2016-02-19 17:41:35 CET; 5s ago Docs: https://docs.docker.com Process: 11436 ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $INSECURE_REGISTRY (code=exited, status=1/FAILURE) Main PID: 11436 (code=exited, status=1/FAILURE) Feb 19 17:41:35 server1.domain.com systemd[1]: Starting Docker Application Container Engine... Feb 19 17:41:35 server1.domain.com docker[11436]: Warning: '-d' is deprecated, it will be removed soon. See usage. Feb 19 17:41:35 server1.domain.com docker[11436]: time="2016-02-19T17:41:35+01:00" level=warning msg="please use 'docker daemon' instead." Feb 19 17:41:35 server1.domain.com docker[11436]: time="2016-02-19T17:41:35.264959961+01:00" level=fatal msg="Error starting daemon: error initializing graphdriver: prerequisites for driver not satisfied (wrong filesystem?)" Feb 19 17:41:35 server1.domain.com systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE Feb 19 17:41:35 server1.domain.com systemd[1]: Failed to start Docker Application Container Engine. Feb 19 17:41:35 server1.domain.com systemd[1]: Unit docker.service entered failed state. Feb 19 17:41:35 server1.domain.com systemd[1]: docker.service failed. |
You need to stop and disable Firewall (legacy iptables):
[root@server1 lib]# systemctl stop firewalld [root@server1 lib]# systemctl disable firewalld Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service. Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service. |
As well as SELinux (reboot needed):
[root@server1 lib]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of three two values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted |
Realized that the default repository file coming from fresh OEL 7.2 install is not up to date so got latest one from http://public-yum.oracle.com/public-yum-ol7.repo.
At first install try I got:
[root@server1 yum.repos.d]# yum install openstack-kolla-preinstall . . Error: Package: docker-engine-1.9.1-1.0.1.el7.x86_64 (ol7_addons) Requires: kernel-uek >= 4.1 . . . |
But I already updated my virtual machine, realized that Unbreakable Enterprise Kernel Release 4 repository was not activated, so changed it and downloaded latest kernel with yum udpate command. I also had to take latest repo file at http://public-yum.oracle.com/public-yum-ol7.repo even if my OEL 7.2 was fresh install:
[ol7_UEKR3] name=Latest Unbreakable Enterprise Kernel Release 3 for Oracle Linux $releasever ($basearch) baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/UEKR3/$basearch/ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle gpgcheck=1 enabled=0 [ol7_UEKR4] name=Latest Unbreakable Enterprise Kernel Release 4 for Oracle Linux $releasever ($basearch) baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/$basearch/ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle gpgcheck=1 enabled=1 |
Finally chnage /etc/sysconfig/docker file as instructed and restart docker service…
Preparing Oracle Linux Master Nodes
Install the Oracle OpenStack for Oracle Linux toolkit:
sudo yum install openstack-kollacli |
And added the group to my personal account, even if I will be using root so:
sudo usermod -aG kolla yjaquier |
Setting up a Docker Registry
From office I’m obviously behind a proxy:
[root@server1 ~]# cat /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=proxy_user:proxy_password@proxy_server:8080" Environment="HTTPS_PROXY=proxy_user:proxy_password@proxy_server:8080" |
Creating a self-signed certificate for my server:
[root@server1 ~]# mkdir -p /etc/docker/certs.d/server1.domain.com:5443 [root@server1 ~]# mkdir -p /var/lib/registry/conf.d [root@server1 ~]# cd /var/lib/registry/conf.d [root@server1 conf.d]# openssl req -newkey rsa:4096 -nodes -sha256 -keyout domain.key -x509 -days 365 -out domain.crt Generating a 4096 bit RSA private key ...............................................................++ .................................................++ writing new private key to 'domain.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]: State or Province Name (full name) []: Locality Name (eg, city) [Default City]: Organization Name (eg, company) [Default Company Ltd]: Organizational Unit Name (eg, section) []: Common Name (eg, your name or your server's hostname) []:server1.domain.com Email Address []: [root@server1 conf.d]# chmod 600 domain.key [root@server1 conf.d]# ll total 8 -rw-r--r-- 1 root root 1992 Feb 22 15:25 domain.crt -rw------- 1 root root 3272 Feb 22 15:25 domain.key [root@server1 conf.d]# cp /var/lib/registry/conf.d/domain.crt /etc/docker/certs.d/server1.domain.com\:5443/ca.crt |
First try to create the registry failed:
docker run -d -p 5443:5000 --name registry --restart=always \ -v /var/lib/registry:/registry_data \ -e REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry_data \ -e REGISTRY_HTTP_TLS_KEY=/registry_data/conf.d/domain.key \ -e REGISTRY_HTTP_TLS_CERTIFICATE=/registry_data/conf.d/domain.crt \ registry:2 Unable to find image 'registry:2' locally Pulling repository docker.io/library/registry Error while pulling image: Get https://index.docker.io/v1/repositories/library/registry/images: x509: certificate signed by unknown authority |
I also installed, following issue I already had on Docker, my proxy certificate:
[root@server1 conf.d]# cp /tmp/za.cer /etc/pki/ca-trust/source/anchors/ [root@server1 conf.d]# update-ca-trust extract |
Second time registry creation went fine:
[root@server1 conf.d]# docker run -d -p 5443:5000 --name registry --restart=always \ -v /var/lib/registry:/registry_data \ -e REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry_data \ -e REGISTRY_HTTP_TLS_KEY=/registry_data/conf.d/domain.key \ -e REGISTRY_HTTP_TLS_CERTIFICATE=/registry_data/conf.d/domain.crt \ registry:2 Unable to find image 'registry:2' locally 2: Pulling from library/registry f32095d4ba8a: Pull complete 9b607719a62a: Pull complete 973de4038269: Pull complete 2867140211c1: Pull complete 8da16446f5ca: Pull complete fd8c38b8b68d: Pull complete 136640b01f02: Pull complete e039ba1c0008: Pull complete c457c689c328: Pull complete Digest: sha256:339d702cf9a4b0aa665269cc36255ee7ce424412d56bee9ad8a247afe8c49ef1 Status: Downloaded newer image for registry:2 53d1962e4b47a43582ddef440e291c1cfe6936d142b33d6d86fa7bcfeda69246 |
Finally loading the downloaded Docker OpenStack images in local registry (yum install bzip2 unzip as well):
[root@server1 docker]# ./import_to_registry.sh server1.domain.com:5443 Checking file integrity: ol-openstack-images-2.0.2.tar.bz2: OK Please wait, this will take quite some time to complete! The push refers to a repository [server1.domain.com:5443/oracle/ol-openstack-swift-object-base] (len: 1) unable to ping registry endpoint https://server1.domain.com:5443/v0/ v2 ping attempt failed with error: Get https://server1.domain.com:5443/v2/: EOF v1 ping attempt failed with error: Get https://server1.domain.com:5443/v1/_ping: EOF ERROR: Pushing images to remote registry! |
Spent a lot of time on this and discovered I had to no_proxy for local addresse and my server name:
[root@server1 ~]# cat /etc/systemd/system/docker.service.d/http-proxy.conf [Service] Environment="HTTP_PROXY=proxy_user:proxy_password@proxy_server:8080" "NO_PROXY=localhost,127.0.0.1,server1.domain.com" Environment="HTTPS_PROXY=proxy_user:proxy_password@proxy_server:8080" "NO_PROXY=localhost,127.0.0.1,server1.domain.com" |
I also had to modify docker configuration to instruct I have an insecure registry because self-signed certificate:
[root@server1 docker]# cat /etc/sysconfig/docker # /etc/sysconfig/docker # Modify these options if you want to change the way the docker daemon runs #OPTIONS='--selinux-enabled' OPTIONS='--storage-driver btrfs --selinux-enabled=false' DOCKER_CERT_PATH=/etc/docker # Enable insecure registry communication by appending the registry URL # to the INSECURE_REGISTRY variable below and uncommenting it # INSECURE_REGISTRY='--insecure-registry ' INSECURE_REGISTRY='--insecure-registry server1.domain.com:5443' # On SELinux System, if you remove the --selinux-enabled option, you # also need to turn on the docker_transition_unconfined boolean. # setsebool -P docker_transition_unconfined # Location used for temporary files, such as those created by # docker load and build operations. Default is /var/lib/docker/tmp # Can be overriden by setting the following environment variable. # DOCKER_TMPDIR=/var/tmp # Controls the /etc/cron.daily/docker-logrotate cron job status. # To disable, uncomment the line below. # LOGROTATE=false # Allow creation of core dumps GOTRACEBACK=crash |
Import went fine and run for multiple hours:
[root@server1 docker]# ./import_to_registry.sh docker01.domain.com:5443 Checking file integrity: ol-openstack-images-2.0.2.tar.bz2: OK Please wait, this will take quite some time to complete! The push refers to a repository [docker01.domain.com:5443/oracle/ol-openstack-swift-object-base] (len: 1) 0df884636100: Pushed 9ec205e38555: Pushed c1822acbddff: Pushed 9e439a32f8f1: Pushed 6a3e66af746c: Pushed 7ddf3cc6745e: Pushed 97c1eb37e57f: Pushed c56ae83a93a7: Pushed ff19c266819e: Pushed e7acd23af511: Pushed a8567fef3a02: Pushed fada7a49741b: Pushed 5130d75ee7f2: Pushed 7cafc7a80c5e: Pushed . . . |
Deploying Oracle OpenStack for Oracle Linux
I have used below setting for kollacli commands:
kollacli setdeploy local kollacli host add server1.domain.com kollacli group addhost control server1.domain.com kollacli group addhost compute server1.domain.com kollacli group addhost database server1.domain.com kollacli group addhost network server1.domain.com kollacli group addhost storage server1.domain.com kollacli property set docker_registry server1.domain.com:5443 kollacli property set enable_haproxy no kollacli property set network_interface enp0s3 kollacli property set kolla_internal_address 192.168.56.101 kollacli property set neutron_external_interface enp0s9 kollacli deploy |
First try failed for:
TASK: [rabbitmq | Waiting for RabbitMQ server start-up] ***********************
failed: [server1.domain.com] => {"changed": true, "cmd": ["docker", "exec", "rabbitmq", "/wait_start.sh"], "delta": "0:00:05.825187", "end": "2016-02-25 15:41:56.789551", "rc": 137, "start": "2016-02-25 15:41:50.964364", "warnings": []}
stdout: Waiting for rabbit@server1 ...
pid is 31 ...
FATAL: all hosts have already failed -- aborting
PLAY RECAP ********************************************************************
to retry, use: --limit @/usr/share/kolla/site.retry
server1.domain.com : ok=27 changed=20 unreachable=0 failed=1
ERROR: Command Failed
I discovered it was because I did not add my server name and IP in /etc/hosts:
[root@server1 network-scripts]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.56.101 server1 server1.domain.com |
Second try was successful…
You can now access to http://192.168.56.101 and get this welcome screen:
Account and password, if you did not customize it is: admin / paswword:
BTRFS issue
Btrfs is a filesystems I’m not daily using, I used ext4 and now xfs. With all the testing I did I hit a strange bug where filesystems was supposed to be full while df command was saying the opposite. I was not planning to dig inside btrfs but had to:
[root@server1 ~]# df Filesystem 1K-blocks Used Available Use% Mounted on devtmpfs 1872432 0 1872432 0% /dev tmpfs 1891028 0 1891028 0% /dev/shm tmpfs 1891028 8568 1882460 1% /run tmpfs 1891028 0 1891028 0% /sys/fs/cgroup /dev/mapper/vg00-lvol00 10475520 2160332 8315188 21% / /dev/mapper/vg00-lvol03 508588 25776 482812 6% /home /dev/mapper/vg00-lvol02 4184064 2118292 2065772 51% /tmp /dev/sda1 508588 167808 340780 33% /boot tmpfs 378208 0 378208 0% /run/user/1000 /dev/mapper/vg00-lvol11 15728640 16896 13613824 1% /var/lib/registry /dev/mapper/vg00-lvol10 20971520 14663420 6049508 71% /var/lib/docker |
Digging a bit on internet I understood I add to balance my filesystem as well explained in this Fixing Btrfs Filesystem Full Problems blog post for example:/
[root@server1 ~]# btrfs balance start -dusage=55 /var/lib/docker ERROR: error during balancing '/var/lib/docker' - No space left on device There may be more info in syslog - try dmesg | tail [root@server1 ~]# btrfs balance start -dusage=55 /var/lib/registry Done, had to relocate 1 out of 3 chunks |
But again I was hitting the worst case where even the balance operation was not possible. So had to allocate temporary storage to perform the allocation::
[root@server1 ~]# btrfs fi show Label: none uuid: 25815922-dfe5-4454-9f63-fedc4ef8ead7 Total devices 1 FS bytes used 640.00KiB devid 1 size 15.00GiB used 3.02GiB path /dev/mapper/vg00-lvol11 Label: none uuid: 5e28b7ae-3ea6-41c7-946a-0ab01310f097 Total devices 1 FS bytes used 9.71GiB devid 1 size 20.00GiB used 20.00GiB path /dev/mapper/vg00-lvol10 btrfs-progs v4.2.2 [root@server1 ~]# df /var/tmp Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/vg00-lvol00 10475520 2160332 8315188 21% / [root@server1 ~]# dd if=/dev/zero of=/var/tmp/btrfs bs=1G count=1 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.303 s, 824 MB/s [root@server1 ~]# losetup -v -f /var/tmp/btrfs [root@server1 ~]# ll /dev/loop0 brw-rw---- 1 root disk 7, 0 Feb 23 10:06 /dev/loop0 [root@server1 ~]# btrfs device add /dev/loop0 /var/lib/docker Performing full device TRIM (1.00GiB) ... |
Now I can delete all filesystem snapshots to clean some space:
[root@server1 subvolumes]# btrfs subvolume list /var/lib/docker |head ID 258 gen 1522 top level 5 path btrfs/subvolumes/f32095d4ba8a9cbc622d10ae515b127cf863e47db3d63ed140f4ae85c7afb0cc ID 259 gen 1522 top level 5 path btrfs/subvolumes/9b607719a62aebe658894221e14778b3d3a3a410a06ff7b35a95be9f4e6be007 ID 260 gen 1522 top level 5 path btrfs/subvolumes/973de40382693bc26be15a6c521e2a0e863c757bbc12dfee99dc60c2ea79aa56 ID 261 gen 1522 top level 5 path btrfs/subvolumes/2867140211c1ef7c3bb91c9b86e3e2a14d6fd81c1b553f8a188bc70a19b8b307 ID 262 gen 1522 top level 5 path btrfs/subvolumes/8da16446f5ca365a71c72e1c990a373b84b513d8030fbffb3c38289b88a762b9 ID 263 gen 1522 top level 5 path btrfs/subvolumes/fd8c38b8b68d3287375bdb043fb7eb70bf0b5d7638919a7c11ea4f2835101b3d ID 264 gen 1522 top level 5 path btrfs/subvolumes/136640b01f0262489802762e9147b0d77ec36f2382a082ca61d744cea2d333db ID 265 gen 1522 top level 5 path btrfs/subvolumes/e039ba1c00081a3d813f9b070266117ebca4b1f3c1fb941632ae22762099af7c ID 266 gen 1522 top level 5 path btrfs/subvolumes/c457c689c328c1ec8a1b97a9c56616395afc5c7fb68ff171d03acf4e03740bc5 ID 1002 gen 1522 top level 5 path btrfs/subvolumes/f17a224ff5f8370f9a921ea27fd54e0951ab66b436c7031a425bd2ff90c6e607-init [root@server1 subvolumes]# for file in /var/lib/docker/btrfs/subvolumes/* > do > btrfs subvolume delete $file > done Delete subvolume (no-commit): '/var/lib/docker/btrfs/subvolumes/00459255da08dd64936527459d50d8fa382914b11be5050865314bfcb2bba832' Delete subvolume (no-commit): '/var/lib/docker/btrfs/subvolumes/0092ebc37eb0b80f70a755719e21b0268474b3d7a13b43b0ae783c4d6314ab10' Delete subvolume (no-commit): '/var/lib/docker/btrfs/subvolumes/00f34edf3dfba0bde08cac1dccf55289e9cc142a30054b98381d080cb8fd6ef3' Delete subvolume (no-commit): '/var/lib/docker/btrfs/subvolumes/012b911881ac4713d3355020d2d4f4adce0cffad6666eda4cfc554d22bbbd75f' Delete subvolume (no-commit): '/var/lib/docker/btrfs/subvolumes/0179007fa7566976ed0059da83247e9b96657a77c52a7f16ff9cbe546dc1263e' Delete subvolume (no-commit): '/var/lib/docker/btrfs/subvolumes/01a105841f2791655b69f25d1900de662ebded908ca6e5e66c9097dea412fe22' |
Temporary storage can be deleted and balance operation can be done:
[root@server1 subvolumes]# btrfs device delete /dev/loop0 /var/lib/docker [root@server1 ~]# losetup -D [root@server1 subvolumes]# btrfs balance start -v -dusage=5 /var/lib/docker Dumping filters: flags 0x1, state 0x0, force is off DATA (flags 0x2): balancing, usage=5 Done, had to relocate 7 out of 21 chunks [root@server1 subvolumes]# btrfs balance start -v -dusage=55 /var/lib/docker Dumping filters: flags 0x1, state 0x0, force is off DATA (flags 0x2): balancing, usage=55 Done, had to relocate 1 out of 15 chunks [root@server1 subvolumes]# btrfs balance start -v -dusage=100 /var/lib/docker Dumping filters: flags 0x1, state 0x0, force is off DATA (flags 0x2): balancing, usage=100 Done, had to relocate 2 out of 15 chunks |
References
