Efficient volume copy for generic volume migration
Currently Cinder uses dd command for data copy of volume migration,
but the copy always copy full blocks even if the source data contains
many null and zero blocks. The dd command has an option conv=sparse
to skip null or zero blocks for more efficient data copy.
However, if the destination volume is not zero cleared beforehand,
we should copy full block from source to dest volume to cleanup dest
volume in order to avoid security issue.
If the volume pre-initilization(zero cleared) is ensured beforehand,
we can skip copy of null and zero blocks to destination volume by
using sparse copy.
In order to use this option properly, we have to check
sparse_copy_volume capability for destination backend driver via
RPC API before volume copy.
This patch also adds sparse_copy_volume capability flag into volume
stats of LVM and NFS drivers to enable efficient copy for these
backends.
Vincent Hou [Thu, 28 May 2015 03:18:09 +0000 (11:18 +0800)]
Volume status management during migration
This patch proposes a new implementation for the status and
the migration_status for volumes.
* The initial migration_status is None, meaning no migration has been
done; Migration_status 'error' means the previous migration failed.
Migration_status 'success' means the previous migration succeeded.
* If the key 'lock_volume' is set to True from the request, the volume
status should be set to 'maintenance' during migration and goes
back to its original status after migration. Otherwise, if the
key 'lock_volume' is set to False, the volume status will remain the
same as its original status. The default value for lock_volume is
False and it applies to the available volume.
* From the REST's perspectives, all the create, update and delete
actions are not allowed if the volume is in 'maintenance', because
it means the volume is out of service. If it is not in maintenance
mode, the migration can be interrupted if other requests are
issued, e.g. attach. For the termination of migration, another
patch will target to resolve it.
DocImpact
APIImpact The key 'lock_volume' has been added into the API,
telling the volume to change the status to 'maintenance' or not.
The migration_status has been added into results returned
from volume list command, if the request is from an admin.
Jon Bernard [Thu, 27 Aug 2015 20:03:57 +0000 (16:03 -0400)]
Skip intermittent VMDK tests
The test 'test_create_backing_from_stream_optimized_file' is failing
intermittently in the gate, which causes delays in the review process of
other patches. Given the proximity to feature freeze, I think it's
appropriate to temporarily disable these tests until a proper solution
can be found.
Jordan Pittier [Fri, 17 Jul 2015 09:48:23 +0000 (11:48 +0200)]
Rework Scality SOFS driver to use RemoteFS class
This change is big but the logic is simple. The previous version of
this driver reimplemented most of what is now in the RemoteFS base
class. SOFS stands for Scale Out FileSystem and is based on FUSE, so
it's only natural to leverage the RemoteFS class.
This patch adds a base framework of capabilities reporting
feature. The get_capabilities RPC API returns dictionary
which is consisted of two parts.
First part includes static backend capabilities which are
obtained by get_volume_stats(). Second part is properties
which includes parameters correspond to extra specs.
This properties part is consisted of cinder standard
capabilities and vendor unique properties. These properties
are created via these two methods.
Since _init_standard_capabilities() only exposes cinder
standard capabilities into the properties dictionary,
each backend driver needs to expose their own properties
by overriding _init_vendor_properties().
Tom Swanson [Tue, 25 Aug 2015 20:13:31 +0000 (15:13 -0500)]
Dell SC: init_volume stale volume info fix
In init_volume the volume information becomes stale after the volume
is mapped to a server. The volume should be updated before being sent
to unmap.
The unmap function was testing for the volume being active. This
should not be required. It is unlikely an inactive volume will have
mappings but not impossible. This check has been removed.
Removed LOG.error from _find_mapping_profiles as the API returning
an error is fine if the volume is inactive. Plus we were not failing
on it. The LOG.debug is sufficient.
Cinder Nested Quota Driver patch adds hierarchical support. Quota
API's now can also handle hierarchical projects.
For the first pass only a user with admin role in a parent project
will be able to update and delete its children quotas.
Even after these changes are made, the existing DbQuotaDriver
functionalities will remain unchanged. The same driver will be able to
handle flat projects as well as hierarchical projects.
Co-Authored-By: Erickson Santos <erickson@lsd.ufcg.edu.br>
Change-Id: Ie669d7d74d40c0ff1f1e54f673e7f3ae96b3b950
Implements: bp cinder-nested-quota-driver
Sonia Ghanekar [Wed, 26 Aug 2015 17:08:53 +0000 (10:08 -0700)]
Add manage/unmanage volume support for Nimble
This patch adds support for volume manage/unmanage for a Nimble
ISCSI driver. It uses an additional field 'agent-type' on the
backend for distinguishing volumes managed by the Nimble driver.
While managing an existing volume it sets the agent-type to
Openstack and while unmanaging sets it back to None.
Since this field was not used in the earlier versions, in order
to be backward compatible, during the initialization of the driver,
the 'agent-type' is updated to Openstack for all the existing
Nimble volumes.
diem_tran [Wed, 12 Aug 2015 19:27:10 +0000 (15:27 -0400)]
Local cache feature of Oracle ZFSSA drivers
The feature enables ZFSSA drivers to create and reuse cache volumes
on ZFSSA backends. It takes advantage of ZFSSA snapshot and clone,
which can be created almost instantly from an existing volume.
Cache volumes will be removed if it has no clone. Custom schemas
and WebDAV properties are used to store cache volumes' properties.
This patch replaces urllib.unquote with urllib.parse.unquote
in the vmdk driver.
Commit d08c7ffe52e3ee211e425c363200998704726e58 replaced
urllib and urllib2 with six.moves.urllib and changed urllib.
unquote to urllib.parse.unquote but missed one call site
in the vmdk driver since the code which introduced it was
under review at the same time. Due to this, if we set an
optional parameter 'vmware_cluster_name', the driver init
will fail with AttributeError.
This patch also adds a unit test which would have caught
this error during gate check.
Update volume status AFTER terminate_connection is done
Current volume API terminate_connection() updates volume's status in
DB before volume driver finishes terminate_connection(). In cases
where volume driver needs some time complete terminate_connection(),
it means volume becomes 'available' before volume driver done cleaning
up. This fix changes the order of 'unreserve_volume' (updating DB)
and driver's terminate_connection to make sure when volume is marked
'available' the client has done cleaning connection.
Note that terminate_connection volume RPC API is a synchronized call,
but volume manager/driver doesn't return any value to caller. What's
more, when volume API extension calls volume terminate_connection API,
it doesn't expect return values, so this change removes unnecessary
return values from volume API.
wanghao [Thu, 4 Jun 2015 09:50:12 +0000 (17:50 +0800)]
Incremental backup improvements for L
1. Add 'is_incremental=True' and 'has_dependent_backups=True/False' to
response body of querying.
2. Add parent_id to notification system.
Since we need to get volume has_dependent_backups value when querying
volume detail list, to reduce the performance impact, add index to
parent_id column in backup table.
APIImpact
When showing backup detail it will return additional info
"is_incremental": True/False and "has_dependent_backups": True/False
wanghao [Wed, 31 Dec 2014 04:06:53 +0000 (12:06 +0800)]
Implement function to manage/unmanage snapshots
1. Add snapshots manage action in contrib api and unmanage
action in API extension.
2. Implement manage_existing_snapshot in manager and flow
to import existing snapshots.
3. Add manage/unmanage support in the LVM driver
Implements: blueprint support-import-export-snapshots
DocImpact
APIImpact
Similar to volume manage/unmanage.
Change-Id: Ib6cf8392b0bc99f803316991f31a75788677e9cf
Cory Stone [Wed, 26 Aug 2015 00:05:36 +0000 (19:05 -0500)]
Dont eager load volume type specs on volume list
Some drivers need extra specs loaded on create and the session could be
closed by the time they access them. However for a volume list, eager
loading these attributes causes a severe performance penalty.
Move the eager load to the single volume get case.
John Griffith [Wed, 19 Aug 2015 17:42:40 +0000 (17:42 +0000)]
Filter out extra-specs from type get for non-admin
Currently when a get_volume type call is made to the
db api, we don't check context and auto-fill the
extra-specs info for that type.
Extra specs are intended to be admin only info, but anybody
that calls the API directly to list or get volume-types is
also given this info which is not intended.
This patch just adds a context check to the db api's
private extra-specs builder method. In the case of
non-admin, we just skip adding the extra-specs.
Masaki Kanno [Tue, 25 Aug 2015 23:56:55 +0000 (08:56 +0900)]
Prevent that all backup objects are deleted
All backup objects of a project are deleted by this bug. The issue occurs
when Swift is a backend. The below is reproduction steps of the issue.
Step 1: Create a backup from a volume.
The creation of the backup may fail by some kind of causes. If the case is
a failure in put_container() in ChunkedBackupDriver._create_container(),
backup objects of the volume are not stored in Swift because 'volumebackups'
container is not created. In the case, the following information is recorded
in Cinder DB.
Step 2: Investigate and solve causes in which put_container() failed.
Step 3: Create another backup from a volume.
When the creation of the backup succeeded, 'volumebackups' container is
created in Swift, then backup objects of the volume are stored into the
container.
Step 4: Delete the backup created in step 1.
Only a record of the backup in Cinder DB should be deleted because the
backup objects were not stored in Swift. However, the backup objects created
in step 3 also are deleted by the bug.
Usually SwiftBackupDrier.get_container_entries() lists only object names of
the backup. However, when backup.service_metadata of the backup is None,
SwiftBackupDriver.get_container_entries() lists all object names in
'volumebackups' container. As a result, all the backup objects in the
container are deleted.
This fix prevents that all the backup objects are deleted.
When a backup is deleted, deletion of the backup objects by delete_object()
is not executed if backup.service_metadata of the backup is None.
Yuriy Nesenenko [Tue, 25 Aug 2015 10:55:56 +0000 (13:55 +0300)]
Parameter osapi_max_limit is always used by default
Default value max_limit is initialized before a value is read from
the config file. Default parameter values are always evaluated when
the “def” statement they belong to is executed. So parameter
osapi_max_limit is always used by default even it's set in the
configuration file.
Like other vendors, NetApp has published a storage controller
client library to PyPI for use by its OpenStack drivers. This
patch updates the NetApp Cinder drivers to use the new library.
The new library, when downloaded from PyPI, provides the REST
and ZAPI interface code that the driver references.
This code was previously part of the driver code tree in
Cinder: netapp/dataontap/client/api.py and netapp/eseries/client.py.
This patchset removes the relevant code from these files
and also modifies the unit tests to work in the absence
of the new library.
Tomoki Sekiyama [Fri, 14 Aug 2015 18:13:10 +0000 (14:13 -0400)]
TemporaryImages to inspect image before conversion
This intruduces TemporaryImages utility class which enables
download images to temporary file for inspection before they are
used for format conversion. Using this utility, we can get image
information such as virtual size, which is required to implement
image cache, before calling fetch_to_raw() etc., without
downloading the image twice nor changing APIs of image_utils.
Tomoki Sekiyama [Fri, 8 May 2015 23:52:19 +0000 (19:52 -0400)]
Efficient image transfer for Glance cinder store
This adds an implementation of clone_image which offloads
the image copy when the image is stored in Glance Cinder store.
It uses "create_clone_volume" method to create a new volume
from an image that is placed on a cinder volume.
To enable this feature, glance_api_version in cinder.conf must be
set to 2, and allowed_direct_url_schemes must contain 'cinder'.
In glance-api.conf, show_multiple_locations must be set to True.
Also the Cinder store must be enabled.
In addition, if image_upload_use_cinder_backend is set to True,
upload-to-image in raw format will create a cloned volume and
register its location to the image service.
If image_upload_use_internal_tenant is set to True, the image
volume is stored in the internal tenant. Otherwise it is placed
in the current context's tenant.
This also changes LVM driver implementation to enable
image upload. Especially for thin LVM, it modifies
create_cloned_volume method to create thin snapshot LV.
Note that the thin snapshot LV can be treated as the normal
LVs; it can be read/written and can survive after the parent
volume is deleted.
Only raw format image is supported. Otherwise, normal image
upload/download method is used.
Currently Glance Cinder store does not support upload and
download, so the image created by this feature cannot be used
by other projects than Cinder. Patches to provide access to
images stored on Cinder are proposed here:
glance-specs: https://review.openstack.org/183363
glance patch(adding rootwrap): https://review.openstack.org/186201
glance_store patch: https://review.openstack.org/166414
Note that this change works even without Glance patches.
Brent Roskos [Sun, 16 Aug 2015 12:41:48 +0000 (08:41 -0400)]
adds user_id to check_is_admin
A small tactical update to allow cinder to consider user_id
when checking for admin.
This is needed in the field until the larger changes around
admin scoping are completed. Checking for role only is not
sufficient in a multi-domain configuration.
Gorka Eguileor [Tue, 18 Aug 2015 09:29:15 +0000 (11:29 +0200)]
Fix backup list all_tenants=0 filtering for admin
When a user with admin role lists backups specifying all_tenants=0 in
the request, it will not behave as expected, it will return backups for
all tenants just as if all_tenants=1 had been specified.
This patch fixes this issue and will only return all tenants in backup
listings if all_tenants is set to a value meaning True (1, true, yes,
y).
Anton Arefiev [Fri, 10 Jul 2015 13:44:59 +0000 (16:44 +0300)]
Add Cinder API wsgi application
This change adds cinder API application, so cinder API can be run
under wsgi server(Apache, Nginx, etc). Eventlet will still be used
by default.
Cinder API with eventlet as a webserver and wsgi application
managing has some cons:
* Cinder API is deployed in other way as a common web application.
Apache/Nginx is generally used web servers for REST API application.
* Cinder API is run as a separate service. It means that cloud
operators need to configure some software to monitor that the API
is running.
* Apache/Nginx works better under the real heavy load than eventlet.
To use c-api deployment under Apache in devstack assign
devstack var CINDER_USE_MOD_WSGI="True".
Related changes in devstack:
https://review.openstack.org/#/c/204643/
Cinder documentation on how deploy cinder API under Apache:
https://review.openstack.org/#/c/207020/
Patch for Infra to make it tested:
https://review.openstack.org/#/c/208498/
John Griffith [Thu, 9 Jul 2015 21:11:54 +0000 (21:11 +0000)]
Cinder replication V2
This adds a scaled back replication implementation
that leaves the bulk of the work up to the driver.
We just provide basic admin API methods to do things
like enable/disable and fail-over. Set up and
specification of replication targets for a specific
back end are now intended to be part of the cinder.conf
in the driver section itself.
Replication targets are configured via the cinder.conf
file in their associated driver section. See the devref
doc included in this commit for details on the format.
The next step in configuration is to create a volume-type
with replication info in the extra-specs.
extra-specs = replication=enable, volume_backend_name=foo
This instructs the driver to utilize replication, default is
up to the driver but would suggest single way rep and in the case
of multiple targets, driver could choose or have a default.
If the back end doesn't report replication=enabled in it's stats
updates the scheduler will fail to place the volume due to invalid
host, or no hosts available.
Vendors can easily modify extra-specs or their own config settings
to modify this behavior, any vendor-unique adaptation can be
provided through the use of scoped keys. Suggested examples
will be published in docs.
See doc/source/devref/replication.rst for more info
kedar-vidvans [Fri, 7 Aug 2015 21:23:46 +0000 (17:23 -0400)]
Assisted volume migration for Oracle ZFSSA drivers
The Oracle ZFSSA drivers will migrate the volume directly from the
source host to the destination host if both hosts are configured
with Oracle ZFSSA. This type of volume migration will bypass the
cinder host and will directly transfer the volume to the destination
backend.
The ZFSSA iSCSI driver will use zfssa's remote replication service
to move the the volume to the destination zfssa.
nikeshm [Fri, 21 Aug 2015 20:39:51 +0000 (02:09 +0530)]
Add https options and minor code changes
Adds https options x_verify_certificate and x_verify_certificate_path
in DotHill and rebranded drivers.
Minor cosmetic changes like replacing 'realstor' with 'virtual'
and 'wbi' with 'api'. Adding default value for 'x_backend_type'
and 'x_api_protocol' options. Minor code optimization for
finding array iSCSI addresses.
Here 'x' stands for dothill or hpmsa or lenovo.
DocImpact
Change-Id: I7a5924bbd56053aee6d652842b48e913c54ae7dd
Closes-Bug: #1486160 Co-Authored-By: Lakshman <lakshminarayanat@vedams.com> Co-Authored-By: Chris Maio <chris.maio@dothill.com>
Code Review / openstack-build / cinder-build.git / log