Creating and Managing Docker Volumes

HPE Cloud Volumes provides a Docker Volume plugin that enables you to create and manage Docker volumes on HPE Cloud Volumes using Docker tools. After you have installed the Docker Volume plugin, you can:

• Create Docker volumes backed by HPE Cloud volumes to persist application data and specify various volume options.
• Clone a Docker volume in order to have multiple copies of production data for development and testing.
• Import HPE Cloud Volumes from existing environments into a Docker environment.
• Import clones of HPE Cloud Volume snapshots into Docker.
• Make HPE Cloud Volumes Docker volumes available on any Docker container node.
• Remove HPE Cloud Volumes from Docker control.
• Delete HPE Cloud Volumes from Docker to reclaim space.
• Set CloudVolume defaults in /etc/hpe-storage/volume-driver.json for HPE Cloud Volumes based Docker volumes. File system defaults include:
  • Performance policy
  • Protection schedule
  • Encryption
  • Cloud provider
  • Cloud resource group
  • CloudVolume tier
  • CloudVolume size
  • CloudVolume IOPS limit
  • Filesystem permission
• Set volume overrides (/etc/hpe-storage/volume-driver.json) for HPE Cloud Volumes based Docker volumes. Volume overrides include the same options as volume defaults. Overrides can not be altered by options supplied through the Docker Volume API or CLI calls.

Installing the Docker Volume plugin

For information on installing the HPE Cloud Volumes Plugin for Docker (containerized v2 plugin), refer to Docker Managed Plugin on GitHub.

Customizing the Docker Volume Plugin

You can customize the Docker Volume plugin behavior by specifying options in /etc/hpe-storage/volume-driver.json. When using a configuration management system, the options set in this configuration file support consistency across nodes.

The volume-driver.json file is comprised of three sections: General Options, Create Options, and Create Override Options.

General Options

The first section of the file contains general options, which are used to control the general behavior of the plugin.

The following shows the content of this section of the configuration file with the default values:

"global": {
    "volumeDir": "/opt/nimble/",
    "nameSuffix": ".docker",
    "snapPrefix": "BaseFor",
    "mountConflictDelay": 30,
    "destroyOnRm": true
 }

Create Options

The second section of volume-driver.json contains create options. This section is used to set your own default options to use with the create command. These options are used by default when a you do not specify values when creating Docker volumes. To set your own defaults, replace the default values with your own values and save your changes. .

The following is the content of this section of the configuration file with the default values:

"defaults": {
         "sizeInGiB": 1,
         "limitIOPS": 500,
         "fsOwner": "0:0",
         "fsMode": "600",
         "perfPolicy": "Other",
         "protectionTemplate": "twicedaily:4",
         "encryption": true,
         "volumeType": "GPF",
         "initiators": ["<IP address for iSCSI>"],
         "automatedConnection": true,
         "existingCloudSubnet": "<cloud subnet>",
         "region": "<public cloud region of the instance>",
         "privateCloud": "<vpc/vnet id>",
         "cloudComputeProvider": "Amazon AWS",
         "cloudAccountName": "My Account Name"
}

The third section of volume-driver.json is used to set override (forced) options for use with the create command. These options override the default options.

By default there are no override settings in the configuration file.

"overrides": {

}

You can obtain cloud instance details, such as VPC/VNET, ResourceGroup, subnet, and region using the following commands on the instance itself, as documented by AWS and Azure.

AWS

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

GET http://169.254.169.254/latest/meta-data/

Azure

https://docs.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service

$ curl -H Metadata:true "http://169.254.169.254/metadata/instance?api-version=2017-12-01"

Provisioning Docker Volumes

There are several ways to provision a Docker volume depending on what tools are used:

• Docker Engine (CLI)
• Docker Compose file with either Docker UCP or Docker Engine

The Docker Volume plugin leverages the existing Docker CLI and APIs. Consequently, all native Docker tools can be used to provision a volume.

Note: The plugin applies a set of default volume create options specified in /etc/hpe-storage/volume-driver.json. Unless you override the default options using the volume option flags, the defaults are applied when you create volumes. For example, the default volume size is 1GiB. Refer to the The volume-driver.json File for information about how to change the default options to suit your needs.

Displaying Help for Docker Commands

Use the docker command to manage docker volumes. For generic help on docker volume commands, refer to https://docs.docker.com/engine/reference/commandline/volume/.

To display help for a specific docker volume command, type the -o flag in the command, as shown in the following example:

$ docker volume create -d cvblock -o help

Universal options:
-o mountConflictDelay=X X is the number of seconds to delay a mount request when there is a conflict (default is 0)
-o destroyOnRm            indicates that the cloud volume (including snapshots) backing this volume should be destroyed when this volume is deleted
-o protectionTemplate=X   X is the protection template in the form of schedule:retention
                          Protection Templates: daily:3, daily:7, daily:14, hourly:6, hourly:12, hourly:24
                          twicedaily:4, twicedaily:8, twicedaily:14, weekly:2, weekly:4, weekly:8, monthly:3, monthly:6, monthly:12 or none

Create options:
-o sizeInGiB=X            X is the size of volume specified in GiB
-o size=X                 X is the size of volume specified in GiB (short form of sizeInGiB)
-o fsOwner=X              X is the user id and group id that should own the root directory of the filesystem, in the form of [userId:groupId]
-o fsMode=X               X is 1 to 4 octal digits that represent the file mode to be applied to the root directory of the filesystem
-o perfPolicy=X           X is the name of the performance policy (optional)
                          Performance Policies: Other, Exchange, Oracle, SharePoint, SQL, Windows File Server
-o encryption             indicates that the volume should be encrypted (optional)
-o volumeType=X           X indicates the cloud volume type(optional)
                          Volume Types: PF, GPF
-o limitIOPS=X            X is the IOPS limit of the volume. IOPS limit should be in range [300, 50000]

Clone options:
-o cloneOf=X              X is the name of Docker Volume to create a clone of
-o snapshot=X             X is the name of the snapshot to base the clone on (optional, if missing, a new snapshot is created)
-o createSnapshot         indicates that a new snapshot of the volume should be taken and used for the clone. Snapshot name can be specified using -o snapshot option (optional)
-o destroyOnDetach        indicates volume to be destroyed on last unmount request

Import Volume options:
-o importVol=X            X is the name of the existing Cloud Volume to import
-o forceImport            forces the import of the volume.  Note that overwrites application metadata (optional)

Import Clone of Snapshot options:
-o importVolAsClone=X     X is the name of the Cloud Volume to clone and import. If -o replStore option is specified, then this indicates a replica volume to clone and import
-o replStore=X            X is the name of the replication store where replica volume to be cloned is located
-o snapshot=X             X is the name of the snapshot to clone and import (optional, if missing, will use the most recent snapshot)
-o createSnapshot         indicates that a new snapshot of the volume should be taken and used for the clone (optional)
-o destroyOnDetach        indicates that the Cloud Volume (including snapshots) backing this volume should be destroyed when this volume is unmounted or detached

Creating a Docker Volume

The Docker Volume plugin uses a set of default options when you create a volume unless you override the options using the volume create option flags.

• Create a Docker volume with the default options:

  $ sudo docker volume create -d cvblock --name docker_volume_name
  

• (Optional) Inspect the new volume.

  $ docker volume inspect docker_volume_name
  

• (Optional) Attach the volume to an interactive container:

  $ sudo docker run -it --rm -v docker_volume_name:/mountpoint docker_image_name
  

  The volume is mounted inside the container specified in the volume:/mountpoint

Example

Create a docker volume with the name myvol1:

$ sudo docker volume create -d cvblock myvol1

Inspect the volume named myvol1:

$ docker volume inspect myvol1
[
    {
        "Driver": "cvblock",
        "Labels": {},
        "Mountpoint": "/",
        "Name": "myvol1",
        "Options": {},
        "Scope": "global",
        "Status": {
            "BlockSize": 4096,
            "CVRegion": "us-test",
            "ConnectionProvisioning": "automated",
            "Encryption": true,
            "LimitIOPS": 300,
            "LimitMBPS": 3,
            "MarkedForDeletion": false,
            "PerfPolicy": "Other Workloads",
            "PrivateCloud": "vpc-1f354a7b",
            "ProtectionSchedule": "snap:twicedaily:4:retention",
            "Serial": "1281513fb602b1a56c9ce90060514147",
            "VolSizeMiB": 1024,
            "VolumeName": "myvol1.docker",
            "VolumeType": "GPF",
            "delayedCreate": true,
            "destroyOnDetach": false,
            "destroyOnRm": false,
            "filesystem": "xfs",
            "fsMode": "600",
            "fsOwner": "0:0",
            "mountConflictDelay": 30
        }
    }
]

Attach the volume to an interactive container:

$ sudo docker run -it --rm -v myvol1:/data alpine

Cloning a Docker Volume

$ sudo docker volume create \
    -d cvblock \
    -o cloneOf=docker_volume_name \
    -o snapshot=cvblock_snapshot_name \
    docker_clone_name

Example

Clone a Docker volume named myvol1 to a new Docker volume named myvol1-clone:

$ sudo docker volume create -d cvblock -o cloneOf=myvol1 myvol1-clone

Select the snapshot on which to base the clone:

$ sudo docker volume create -d cvblock \
    -o cloneOf=docker_volume_name \
    -o snapshot=cvblock_snapshot_name \
    docker_clone_name

Importing a Volume to Docker

Before you import a volume to Docker:

• Make sure no initiators are attached to the volume
• Use the create command with the importVol option to import an HPE Cloud Volume to Docker and name it.

$ sudo docker volume create -d cvblock -o importVol=cvblock_volume_name docker_volume_name

Example

To import an HPE Cloud Volume named myoldvol1 as a Docker volume named myimportvol1:

$ sudo docker volume create -d cvblock -o importVol=myoldvol1 myimportvol1

Importing a Volume Snapshot to Docker

Use the create command with the importVolAsClone option to import a HPE Cloud Volume snapshot as a Docker volume. Optionally, specify a particular snapshot on the HPE Cloud Volume using the snapshot option.

To import an HPE Nimble Storage volume snapshot to Docker:

$ sudo docker volume create
    -d cvblock \
    -o importVolAsClone=cvblock_volume_name \
    -o snapshot=cvblock_snapshot_name \
    docker_imported_snapshot

Note: If no snapshot is specified, the latest snapshot on the volume is imported

Example

Import the HPE Cloud Volume myoldvol1 using snapshot mysnap1 as a Docker volume named myimportsnap1

$ sudo docker volume create -d cvblock -o importVolAsClone=myoldvol1 -o snapshot=mysnap1 myimportsnap1
myimportsnap1

Importing a Replica Volume Snapshot to Docker

Use the create command with the importVolAsClone on the volume option to import a replica volume snapshot as a Docker volume. Optionally, specify a particular snapshot on the HPE Cloud Volume using the snapshot option. If no snapshot option is provided, then the most recent snapshot is used.

$ sudo docker volume create \
    -d cvblock \
    -o importVolAsClone=cvblock_replica_volume \
    -o snapshot=cvblock_snapshot_name \
    -o replStore=cvblock_repl_store \
    docker_volume_name

Example

Import the most recent HPE Cloud Volumes snapshot on the replica volume myreplvol1 as a Docker volume named myreplclone1 from the Replication Store myreplstore1:

$ sudo docker volume create \
    -d cvblock \
    -o importVolAsClone=myreplvol1 \
    -o snapshot=mysnap1 \
    -o replStore=myreplstore1 \
    myreplclone1

Listing Volumes

$ docker volume ls

Example

$ docker volume ls
DRIVER              VOLUME NAME
cvblock              myvol1
cvblock              myvol1-clone
cvblock-local        local

Running a Container Using a Docker Volume

Run a container dependent on a HPE Cloud Volume backed Docker volume.

$ sudo docker run -it --rm -v docker_volume_name:/mountpoint_inside_container image_name

Example

Start an alpine container using the volume myvol1 on /data:

$ sudo docker run -it --rm -v myvol1:/data alpine

Removing a Docker Volume

When you remove volumes from Docker control, the volumes are still left intact in the HPE Cloud Volumes portal. You can reestablish access to the volumes and related snapshots using the Docker Volume plugin.

$ sudo docker volume rm docker_volume_name

Note: To delete volumes from the HPE Cloud Volumes portal using the remove command, edit /opt/hpe-storage/volume-driver.json and change the destroyOnRm option to true.

Important: When destroyOnRm is set to true, volumes and all related snapshots are deleted from the portal and can no longer be accessed by the Docker Volume plugin.

Example

Remove the Docker volume named myvol1-clone:

$ sudo docker volume rm myvol1-clone

Docker Swarm Considerations

If you are considering using any Docker clustering technologies for your Docker deployment, you need to understand the fencing mechanism used to protect data. Attaching the same Docker volume to multiple containers on the same host is fully supported. However, mounting the same volume on multiple hosts is not supported.

Because Docker does not provide a fencing mechanism for nodes that have become disconnected from the Docker Swarm, isolated nodes continue to run their containers. When the containers are rescheduled on a surviving node, the Docker Engine will request that the Docker Volume(s) be mounted. In order to prevent data corruption, the Docker Volume plugin will stop serving the Docker Volume to the original node before mounting it on the newly requested node.

During a mount request, the Docker Volume plugin inspects the volume for connected Cloud VMs. If the Cloud VM does not match the initiator requesting to mount the volume, the Cloud VM is forcefully removed and the iSCSI session disconnected. The volume is now fenced off and other nodes are unable to access any data in the volume

The volume will then reassign a new Cloud VM matching the requesting initiator, and it is mounted for the container requesting the volume. This is done because the volumes are formatted with XFS, which is not a clustered file system and will become corrupted if the same volume is mounted to multiple hosts

The side effect of a fenced node is that I/O hangs indefinitely, and the initiator is rejected during login. If the fenced node rejoins the Docker Swarm, the swarm tries to shut down the services that were rescheduled elsewhere to maintain the desired replica set for the service. This operation will also hang indefinitely waiting for I/O.

It is recommend that you run a dedicated Docker host that does not host any other critical applications besides the Docker engine. Doing this supports a safe way to reboot a node after a grace period and have it start cleanly when a hung task is detected. Otherwise, the node can remain in the hung state indefinitely.

The following kernel parameters control the system behavior when a hung task is detected:

+

# Reset after these many seconds after a panic
kernel.panic = 5

# I do consider hung tasks reason enough to panic
kernel.hung_task_panic = 1

# To not panic in vain, I'll wait these many seconds before I declare
a hung task
kernel.hung_task_timeout_secs = 150

Add these parameters to the /etc/sysctl.d/99-hung_task_timeout.conf file and reboot the system.

Note: Docker Swarm declares a node as failed after five (5) seconds. Services are then rescheduled and up and running again in less than ten (10) seconds. The parameters noted above provide the system a way to manage other tasks that may appear to be hung and avoid a system panic. By default, the Docker Volume plugin waits up to 30 seconds before forcefully remapping the volume. You can configure the delay with the mountConflictDelay option.