Creating and Managing Docker Volumes
HPE Cloud Volumes provides a Docker Volume plugin that enables you to create and manage Docker volumes on HPE Cloud Volumes using Docker tools. After you have installed the Docker Volume plugin, you can:
- Create Docker volumes backed by HPE Cloud volumes to persist application data and specify various volume options.
- Clone a Docker volume in order to have multiple copies of production data for development and testing.
- Import HPE Cloud Volumes from existing environments into a Docker environment.
- Import clones of HPE Cloud Volume snapshots into Docker.
- Make HPE Cloud Volumes Docker volumes available on any Docker container node.
- Remove HPE Cloud Volumes from Docker control.
- Delete HPE Cloud Volumes from Docker to reclaim space.
- Set CloudVolume defaults in
/etc/hpe-storage/volume-driver.json
for HPE Cloud Volumes based Docker volumes. File system defaults include:- Performance policy
- Protection schedule
- Encryption
- Cloud provider
- Cloud resource group
- CloudVolume tier
- CloudVolume size
- CloudVolume IOPS limit
- Filesystem permission
- Set volume overrides (
/etc/hpe-storage/volume-driver.json
) for HPE Cloud Volumes based Docker volumes. Volume overrides include the same options as volume defaults. Overrides can not be altered by options supplied through the Docker Volume API or CLI calls.
Installing the Docker Volume plugin
For information on installing the HPE Cloud Volumes Plugin for Docker (containerized v2 plugin), refer to Docker Managed Plugin on GitHub.
Customizing the Docker Volume Plugin
You can customize the Docker Volume plugin behavior by specifying options in /etc/hpe-storage/volume-driver.json
. When using a configuration management system, the options set in this configuration file support consistency across nodes.
The volume-driver.json
file is comprised of three sections: General Options, Create Options, and Create Override Options.
General Options
The first section of the file contains general options, which are used to control the general behavior of the plugin.
General Option | Description |
---|---|
volumeDir | Describes the location on the host that the Docker driver should make HPE Cloud Volumes available. |
nameSuffix | Specifies the suffix to add to volume names (created by the Docker driver) on the group. |
destroyOnRm | Controls the action taken by the driver when you remove a volume using the docker volume rm command. If this option is set to true, the volume and all of its snapshots are deleted from the HPE Cloud Volumes portal and are no longer available. If this option is set to false, the volume is removed from Docker control, but remains in the group as a regular volume. |
snapPrefix | Specifies the prefix of the name of the snapshot used to clone a Docker volume. |
mountConflictDelay | Specifies the delay after which a mount request is processed on a volume that is already mounted to a different node. |
The following shows the content of this section of the configuration file with the default values:
"global": {
"volumeDir": "/opt/nimble/",
"nameSuffix": ".docker",
"snapPrefix": "BaseFor",
"mountConflictDelay": 30,
"destroyOnRm": true
}
Create Options
The second section of volume-driver.json
contains create options. This section is used to set your own default options to use with the create
command. These options are used by default when a you do not specify values when creating Docker volumes. To set your own defaults, replace the default values with your own values and save your changes. .
Create Option | Description |
---|---|
sizeInGiB | The size of the volume in GiB |
fsOwner | The user ID and group ID that should own the root directory of the file system, in the form of [userId:groupId]. |
fsMode | 1 to 4 octal digits that represent the file mode to apply to the root directory of the file system. |
encryption | Indicates whether the volume should be encrypted. |
limitIOPS | The IOPS limit of the volume. The IOPS limit should be in the range [300, 50000], or -1 for unlimited. |
protectionTemplate | The name of the protection template. Only the following values are allowed for protection template: daily:3 , daily:7 , daily:14 ,hourly:6 , hourly:12 , hourly , twicedaily:4 ,twicedaily:8 , twicedaily:14 , weekly:2 ,weekly:4 ,weekly:8 , monthly:3 ,monthly:6 , monthly:12 or none . |
volumeType | Indicates HPE Cloud Volume type. This can be either GPF or PF. |
region | Specifies public cloud provider region name |
privateCloud | Specifies VPC ID for AWS or VNET name for Azure. |
privateCloudResourceGroup | Specifies Azure private cloud resource group |
cloudComputeProvider | Specifies public cloud provider name (Amazon AWS ) or (Microsoft Azure ). |
cloudAccountName | Specifies the HPE CV cloud account name. To retrieve the cloud account name from the portal, log into HPE CV and click Cloud Accounts from the main menu. |
automatedConnection | Specifies whether the connection between HPE Cloud Volumes and the public cloud provider network is automated. |
perfPolicy | Specifies the name of the policy to use as the default performance policy. Supported performance policies include: Other ,Exchange , Oracle , SharePoint , SQL , Windows File Server . |
The following is the content of this section of the configuration file with the default values:
"defaults": {
"sizeInGiB": 1,
"limitIOPS": 500,
"fsOwner": "0:0",
"fsMode": "600",
"perfPolicy": "Other",
"protectionTemplate": "twicedaily:4",
"encryption": true,
"volumeType": "GPF",
"initiators": ["<IP address for iSCSI>"],
"automatedConnection": true,
"existingCloudSubnet": "<cloud subnet>",
"region": "<public cloud region of the instance>",
"privateCloud": "<vpc/vnet id>",
"cloudComputeProvider": "Amazon AWS",
"cloudAccountName": "My Account Name"
}
The third section of volume-driver.json
is used to set override (forced) options for use with the create
command. These options override the default options.
Create Override Options | Description |
---|---|
sizeInGiB | The size of the volume in GiB. |
fsOwner | The user ID and group ID that should own the root directory of the files ystem in the form of [userId:groupId]. |
fsMode | 1 to 4 octal digits that represent the file mode to apply to the root directory of the file system. |
perfPolicy | The name of the performance policy assigned to the volume. |
encryption | Indicates whether the volume should be encrypted. |
limitIOPS | The IOPS limit of the volume. The IOPS limit should be in the range [300, 50000] |
protectionTemplate | The name of the protection template. |
By default there are no override settings in the configuration file.
"overrides": {
}
You can obtain cloud instance details, such as VPC/VNET, ResourceGroup, subnet, and region using the following commands on the instance itself, as documented by AWS and Azure.
AWS
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
GET http://169.254.169.254/latest/meta-data/
Azure
https://docs.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service
$ curl -H Metadata:true "http://169.254.169.254/metadata/instance?api-version=2017-12-01"
Provisioning Docker Volumes
There are several ways to provision a Docker volume depending on what tools are used:
- Docker Engine (CLI)
- Docker Compose file with either Docker UCP or Docker Engine
The Docker Volume plugin leverages the existing Docker CLI and APIs. Consequently, all native Docker tools can be used to provision a volume.
Note: The plugin applies a set of default volume create options specified in /etc/hpe-storage/volume-driver.json
. Unless you override the default options using the volume option flags, the defaults are applied when you create volumes. For example, the default volume size is 1GiB. Refer to the The volume-driver.json File for information about how to change the default options to suit your needs.
Displaying Help for Docker Commands
Use the docker
command to manage docker volumes. For generic help on docker volume commands, refer to https://docs.docker.com/engine/reference/commandline/volume/.
To display help for a specific docker volume command, type the -o
flag in the command, as shown in the following example:
$ docker volume create -d cvblock -o help
Universal options:
-o mountConflictDelay=X X is the number of seconds to delay a mount request when there is a conflict (default is 0)
-o destroyOnRm indicates that the cloud volume (including snapshots) backing this volume should be destroyed when this volume is deleted
-o protectionTemplate=X X is the protection template in the form of schedule:retention
Protection Templates: daily:3, daily:7, daily:14, hourly:6, hourly:12, hourly:24
twicedaily:4, twicedaily:8, twicedaily:14, weekly:2, weekly:4, weekly:8, monthly:3, monthly:6, monthly:12 or none
Create options:
-o sizeInGiB=X X is the size of volume specified in GiB
-o size=X X is the size of volume specified in GiB (short form of sizeInGiB)
-o fsOwner=X X is the user id and group id that should own the root directory of the filesystem, in the form of [userId:groupId]
-o fsMode=X X is 1 to 4 octal digits that represent the file mode to be applied to the root directory of the filesystem
-o perfPolicy=X X is the name of the performance policy (optional)
Performance Policies: Other, Exchange, Oracle, SharePoint, SQL, Windows File Server
-o encryption indicates that the volume should be encrypted (optional)
-o volumeType=X X indicates the cloud volume type(optional)
Volume Types: PF, GPF
-o limitIOPS=X X is the IOPS limit of the volume. IOPS limit should be in range [300, 50000]
Clone options:
-o cloneOf=X X is the name of Docker Volume to create a clone of
-o snapshot=X X is the name of the snapshot to base the clone on (optional, if missing, a new snapshot is created)
-o createSnapshot indicates that a new snapshot of the volume should be taken and used for the clone. Snapshot name can be specified using -o snapshot option (optional)
-o destroyOnDetach indicates volume to be destroyed on last unmount request
Import Volume options:
-o importVol=X X is the name of the existing Cloud Volume to import
-o forceImport forces the import of the volume. Note that overwrites application metadata (optional)
Import Clone of Snapshot options:
-o importVolAsClone=X X is the name of the Cloud Volume to clone and import. If -o replStore option is specified, then this indicates a replica volume to clone and import
-o replStore=X X is the name of the replication store where replica volume to be cloned is located
-o snapshot=X X is the name of the snapshot to clone and import (optional, if missing, will use the most recent snapshot)
-o createSnapshot indicates that a new snapshot of the volume should be taken and used for the clone (optional)
-o destroyOnDetach indicates that the Cloud Volume (including snapshots) backing this volume should be destroyed when this volume is unmounted or detached
Creating a Docker Volume
The Docker Volume plugin uses a set of default options when you create a volume unless you override the options using the volume create
option flags.
-
Create a Docker volume with the default options:
$ sudo docker volume create -d cvblock --name docker_volume_name
-
(Optional) Inspect the new volume.
$ docker volume inspect docker_volume_name
-
(Optional) Attach the volume to an interactive container:
$ sudo docker run -it --rm -v docker_volume_name:/mountpoint docker_image_name
The volume is mounted inside the container specified in the
volume:/mountpoint
Example
Create a docker volume with the name myvol1
:
$ sudo docker volume create -d cvblock myvol1
Inspect the volume named myvol1
:
$ docker volume inspect myvol1
[
{
"Driver": "cvblock",
"Labels": {},
"Mountpoint": "/",
"Name": "myvol1",
"Options": {},
"Scope": "global",
"Status": {
"BlockSize": 4096,
"CVRegion": "us-test",
"ConnectionProvisioning": "automated",
"Encryption": true,
"LimitIOPS": 300,
"LimitMBPS": 3,
"MarkedForDeletion": false,
"PerfPolicy": "Other Workloads",
"PrivateCloud": "vpc-1f354a7b",
"ProtectionSchedule": "snap:twicedaily:4:retention",
"Serial": "1281513fb602b1a56c9ce90060514147",
"VolSizeMiB": 1024,
"VolumeName": "myvol1.docker",
"VolumeType": "GPF",
"delayedCreate": true,
"destroyOnDetach": false,
"destroyOnRm": false,
"filesystem": "xfs",
"fsMode": "600",
"fsOwner": "0:0",
"mountConflictDelay": 30
}
}
]
Attach the volume to an interactive container:
$ sudo docker run -it --rm -v myvol1:/data alpine
Cloning a Docker Volume
$ sudo docker volume create \
-d cvblock \
-o cloneOf=docker_volume_name \
-o snapshot=cvblock_snapshot_name \
docker_clone_name
Example
Clone a Docker volume named myvol1
to a new Docker volume named myvol1-clone
:
$ sudo docker volume create -d cvblock -o cloneOf=myvol1 myvol1-clone
Select the snapshot on which to base the clone:
$ sudo docker volume create -d cvblock \
-o cloneOf=docker_volume_name \
-o snapshot=cvblock_snapshot_name \
docker_clone_name
Importing a Volume to Docker
Before you import a volume to Docker:
- Make sure no initiators are attached to the volume
- Use the
create
command with theimportVol
option to import an HPE Cloud Volume to Docker and name it.
$ sudo docker volume create -d cvblock -o importVol=cvblock_volume_name docker_volume_name
Example
To import an HPE Cloud Volume named myoldvol1
as a Docker volume named myimportvol1
:
$ sudo docker volume create -d cvblock -o importVol=myoldvol1 myimportvol1
Importing a Volume Snapshot to Docker
Use the create
command with the importVolAsClone
option to import a HPE Cloud Volume snapshot as a Docker volume. Optionally, specify a particular snapshot on the HPE Cloud Volume using the snapshot option.
To import an HPE Nimble Storage volume snapshot to Docker:
$ sudo docker volume create
-d cvblock \
-o importVolAsClone=cvblock_volume_name \
-o snapshot=cvblock_snapshot_name \
docker_imported_snapshot
Note: If no snapshot is specified, the latest snapshot on the volume is imported
Example
Import the HPE Cloud Volume myoldvol1
using snapshot mysnap1
as a Docker volume named myimportsnap1
$ sudo docker volume create -d cvblock -o importVolAsClone=myoldvol1 -o snapshot=mysnap1 myimportsnap1
myimportsnap1
Importing a Replica Volume Snapshot to Docker
Use the create
command with the importVolAsClone
on the volume option to import a replica volume snapshot as a Docker volume. Optionally, specify a particular snapshot on the HPE Cloud Volume using the snapshot option. If no snapshot option is provided, then the most recent snapshot is used.
$ sudo docker volume create \
-d cvblock \
-o importVolAsClone=cvblock_replica_volume \
-o snapshot=cvblock_snapshot_name \
-o replStore=cvblock_repl_store \
docker_volume_name
Example
Import the most recent HPE Cloud Volumes snapshot on the replica volume myreplvol1
as a Docker volume named myreplclone1
from the Replication Store myreplstore1
:
$ sudo docker volume create \
-d cvblock \
-o importVolAsClone=myreplvol1 \
-o snapshot=mysnap1 \
-o replStore=myreplstore1 \
myreplclone1
Listing Volumes
$ docker volume ls
Example
$ docker volume ls
DRIVER VOLUME NAME
cvblock myvol1
cvblock myvol1-clone
cvblock-local local
Running a Container Using a Docker Volume
Run a container dependent on a HPE Cloud Volume backed Docker volume.
$ sudo docker run -it --rm -v docker_volume_name:/mountpoint_inside_container image_name
Example
Start an alpine container using the volume myvol1
on /data
:
$ sudo docker run -it --rm -v myvol1:/data alpine
Removing a Docker Volume
When you remove volumes from Docker control, the volumes are still left intact in the HPE Cloud Volumes portal. You can reestablish access to the volumes and related snapshots using the Docker Volume plugin.
$ sudo docker volume rm docker_volume_name
Note: To delete volumes from the HPE Cloud Volumes portal using the remove
command, edit /opt/hpe-storage/volume-driver.json
and change the destroyOnRm
option to true
.
Important: When destroyOnRm
is set to true
, volumes and all related snapshots are deleted from the portal and can no longer be accessed by the Docker Volume plugin.
Example
Remove the Docker volume named myvol1-clone
:
$ sudo docker volume rm myvol1-clone
Docker Swarm Considerations
If you are considering using any Docker clustering technologies for your Docker deployment, you need to understand the fencing mechanism used to protect data. Attaching the same Docker volume to multiple containers on the same host is fully supported. However, mounting the same volume on multiple hosts is not supported.
Because Docker does not provide a fencing mechanism for nodes that have become disconnected from the Docker Swarm, isolated nodes continue to run their containers. When the containers are rescheduled on a surviving node, the Docker Engine will request that the Docker Volume(s) be mounted. In order to prevent data corruption, the Docker Volume plugin will stop serving the Docker Volume to the original node before mounting it on the newly requested node.
During a mount request, the Docker Volume plugin inspects the volume for connected Cloud VMs. If the Cloud VM does not match the initiator requesting to mount the volume, the Cloud VM is forcefully removed and the iSCSI session disconnected. The volume is now fenced off and other nodes are unable to access any data in the volume
The volume will then reassign a new Cloud VM matching the requesting initiator, and it is mounted for the container requesting the volume. This is done because the volumes are formatted with XFS, which is not a clustered file system and will become corrupted if the same volume is mounted to multiple hosts
The side effect of a fenced node is that I/O hangs indefinitely, and the initiator is rejected during login. If the fenced node rejoins the Docker Swarm, the swarm tries to shut down the services that were rescheduled elsewhere to maintain the desired replica set for the service. This operation will also hang indefinitely waiting for I/O.
It is recommend that you run a dedicated Docker host that does not host any other critical applications besides the Docker engine. Doing this supports a safe way to reboot a node after a grace period and have it start cleanly when a hung task is detected. Otherwise, the node can remain in the hung state indefinitely.
The following kernel parameters control the system behavior when a hung task is detected:
+
# Reset after these many seconds after a panic
kernel.panic = 5
# I do consider hung tasks reason enough to panic
kernel.hung_task_panic = 1
# To not panic in vain, I'll wait these many seconds before I declare
a hung task
kernel.hung_task_timeout_secs = 150
Add these parameters to the /etc/sysctl.d/99-hung_task_timeout.conf
file and reboot the system.
Note: Docker Swarm declares a node as failed after five (5) seconds. Services are then rescheduled and up and running again in less than ten (10) seconds. The parameters noted above provide the system a way to manage other tasks that may appear to be hung and avoid a system panic. By default, the Docker Volume plugin waits up to 30 seconds before forcefully remapping the volume. You can configure the delay with the mountConflictDelay
option.