Patrick East [Wed, 2 Sep 2015 20:51:42 +0000 (13:51 -0700)]
Get full volume model in Replication manager API’s
Currently we pass in a dictionary of the volume properties, but not the
actual volume model. This dictionary is missing some of the properties
like ‘name’ which backends may required for volume operations.
Fix problem of efficient volume copy for migration
After introducing commit f586043fa969b9d1dcf4933aacbf615f53691093,
new volume copy method _copy_volume_data() was added into
manager.py. Originally driver.py had this method and it was copied
into manager.py. However new _copy_volume_data() lost efficient
volume copy logic during the reimplementation.
This patch simply add efficient volume copy logic again into new
_copy_volume_data() to fix the problem.
Patrick East [Wed, 8 Jul 2015 00:23:58 +0000 (17:23 -0700)]
Generic image-volume cache
This introduces a new feature for backends to be able to use cached
glance images when creating volumes from images.
If enabled, and the cinder internal tenant is configured, it will
create a minimal sized clone of a volume the first time it is
downloaded from glance. A new db table ‘image_volume_cache’ tracks
these volumes, which are owned by the Cinder internal tenant. The
original will then be extended to full size. Any subsequent requests to
create a volume from an image will be able to do a volume clone from
the image-volume cache entry instead of downloading the image data from
glance again.
In the future we could create an entry upon creating an image from a
volume right before uploading the volume to glance. This version
however does not aim to do more than add the initial framework and
help with subsequent image downloads. There are certainly more
improvements that can be done over time on top of this.
These image-volumes are host specific, so each backend may end up with
its very own image-volume to do clones from.
The cache is limited in size by number of entries allowed and size in
gb. When creating a new entry if space is needed the last used entries
will be evicted to make room.
In the current implementation the image-volumes will be viewable by a
cloud admin in the volume list, and show up as owned by the Cinder
internal tenant. They can safely be deleted manually by an admin, this
will cause the entry to be removed from the cache. These volumes will
use quota for the internal tenant.
Cache actions will post notifications to Ceilometer. There are
notifications like ‘image_volume_cache.miss’, ‘image_volume_cache.hit’,
and ‘image_volume_cache.evict’. A change will be required to the
event_definitions.yaml to see them nicely. Until then you only need to
add a new event type 'image_volume_cache.*' and look for the ‘image_id’
and ‘host’ fields.
DocImpact: We probably want some instructions on restrictions of the
cache and how to use it. The three new config options should also be
documented somewhere: ‘image_volume_cache_enabled’,
’image_volume_cache_max_size_gb’, ‘image_volume_cache_max_size_count’
Michael Price [Wed, 5 Aug 2015 20:36:59 +0000 (15:36 -0500)]
Implement thin provisioning support for E-Series
Implement a new extra spec, 'netapp_thin_provisioned', that will allow
users to define thin provisioned Cinder volumes on E-Series storage,
alongside pre-existing extra specs such as
'netapp_eseries_data_assurance', 'netapp_eseries_flash_reach_cache',
'netapp_raid_type', etc.
We have a followup patch, https://review.openstack.org/#/c/215801/ ,
that reports 'thin_provisioning_support=True/False' and same for
'thick_provisioning_support', in accord with the scheduler-based
over-subscription support added in Kilo.
We are not yet attempting to implement the get_capabilities()
feature just merged into Liberty
(https://review.openstack.org/#/c/201243/)
but fully intend to do so in a way that conforms with that plan of
record.
The CloudByte driver is reading the parent's volume id from the
new volume's source_volid. There are some cases when this isn't set.
The source volume object/dict is passed in, so just access the id
from there.
Tom Barron [Mon, 17 Aug 2015 18:08:28 +0000 (14:08 -0400)]
Scheduler-based over-subscription for NFS drivers
Add support to the NFS drivers for over-subscription/overprovisioning.
This allows the Cinder scheduler to more accurately represent the
remaining space on a pool when thin-provisioned Cinder Volumes are in
use on an NFS backend.
DocImpact Co-Authored-By: Clinton Knight <cknight@netapp.com> Co-Authored-By: Mike Rooney <rooneym@netapp.com>
Implements: blueprint nfs-over-subscription-thin-provisioning
Xing Yang [Wed, 19 Aug 2015 02:16:03 +0000 (22:16 -0400)]
ScaleIO driver should use os-brick connector
Currently the ScaleIO Cinder volume driver contains the connect_volume logic
inside the driver itself. This code is redundant with the code in the os-brick
connector. This patch modifies the driver to use the os-brick connector and
removes the redundant code.
Thang Pham [Fri, 28 Aug 2015 14:02:50 +0000 (07:02 -0700)]
Sync volume versionedobject to ORM
The following patch syncs the volume object to match the
ORM. More specifically, it adds metadata, admin_metdata,
volume_attachment, and volume_type to the volume object.
These fields are required to convert cinder internals to
use volume versionedobject. It was originally part of
a large patch, but is broken up to make reviews more
bearable.
When importing backup metadata only required fields were being imported,
and all other fields were being ignored.
This led to some serious problems when importing backups back, since
you would lose not only the original Volume ID, but also relevant parent
id information needed to restore incremental backups.
This patch fixes this by importing everything back and creating backup
records with the right ID.
Michal Dulko [Wed, 8 Jul 2015 09:27:30 +0000 (11:27 +0200)]
Service object
Add versionedobjects abstraction layer to services. Distinguish time
zone aware DateTime fields. The object derives from
CinderObjectDictCompat, so it supports both object (obj.prop) and dict
(obj['prop']) syntax to access properties. Complete move to object
notation will be made in a follow up clean up patch.
Co-Authored-By: Michal Dulko <michal.dulko@intel.com>
Change-Id: I09f593f9f9aa8befa40d989b731159b78a429071
Partial-Implements: blueprint cinder-objects
Mitsuhiro Tanino [Mon, 10 Aug 2015 20:44:25 +0000 (16:44 -0400)]
Allow specified backend capabilities to be retrieved
This change adds a new admin-api extension to allow admin to
fetch specified backend capabilities.
With this extension, admin can obtain what the current deployed
backend in Cinder is able to do from the endpoint.
The extension takes "host" as an parameter.
GET http://CINDER_API_ENDPOINT/v2/TENANT_ID/capabilities/<host>
John Griffith [Sun, 23 Aug 2015 18:20:54 +0000 (12:20 -0600)]
Add cinder.conf.sample to gitignore
Now that we have some sort of conf checking again in the
tox ini file, we should make sure we don't inadvertenly
commit a generated sample (which I've done twice now).
Liu Xinguo [Tue, 14 Jul 2015 14:36:57 +0000 (22:36 +0800)]
Enhance FC zone support for Huawei driver
Currently, when using FC switch, there are manual operations
necessary to enable the Huawei driver to work. This change
makes it so those manual operations are no longer required
any more. Huawei driver can choose the appropriate
FC ports and manage FC zone automatically.
(Pulled from gate, cinder can no longer pass unit tests)
Jon Bernard [Mon, 1 Jun 2015 18:09:30 +0000 (14:09 -0400)]
Add support for file I/O volume migration
This patch changes the generic volume migration logic to support
non-attachable volumes. Non-attachable refers to volume drivers that do
not support attachment via the typical iSCSI or similar protocols where
a block device is made available on the host machine. Device drivers
such as RBD make volumes available to cinder via a file handle that
proxies read() and write() calls to the Ceph cluster.
This patch improves the generic migration logic to determine whether a
migration operation can proceed with dd using block device paths or file
operations on handles returned from the os-brick connectors.
Changes to the RBD driver are included to correctly rename the target
volume during the completion phase of a successful migration.
It appears there is still some work to be done for attached in-use
volume migration for certain configurations. Successful tests were seen
for:
LVM to LVM (available and in-use)
LVM to/from NFS (available and in-use)
LVM to/from Ceph (available)
Ceph to LVM (in-use)
NFS to/from Ceph (available)
Ceph to NFS (in-use)
Failures were seen (due to Nova) for the following:
LVM to Ceph (in-use)
NFS to Ceph (in-use)
(Pulled from gate, cinder can no longer pass unit tests)
Matt Riedemann [Sun, 30 Aug 2015 15:27:45 +0000 (08:27 -0700)]
Add debug logging before attaching volume in driver
It'd be helpful to log the volume/instance/mount/host
information in attach_volume before calling the volume
driver to do the attachment to know we have the right
values.
Tom Barron [Mon, 17 Aug 2015 10:36:24 +0000 (06:36 -0400)]
NetApp DOT block driver over-subscription support
Add support to the NetApp Data ONTAP block drivers for over-
subscription/overprovisioning. This allows the Cinder scheduler
to more accurately represent the remaining space on a pool when
thin-provisioned Cinder Volumes are in use on a Data ONTAP
backend.
DocImpact Co-Authored-By: Clinton Knight <cknight@netapp.com>
Partially implements: blueprint netapp-block--over-subscription-thin-provisioning
Daniel Tadrzak [Thu, 2 Jul 2015 13:44:39 +0000 (15:44 +0200)]
Cleanup for SnapshotObject
Mostly switched from dictionary notation (obj['prop']) to object
notation (obj.prop).
This patch doesn't contain changes for drivers. Drivers need to be
fixed in other patch.
Deleting volumes in CloudByte Storage is an asynchronous process.
So, the driver needs to wait till the job/process is completed.
The driver uses the jobid in a wait loop for a configured interval
of time, till the volume deletion is completed.
Anna Sortland [Thu, 27 Aug 2015 20:36:11 +0000 (15:36 -0500)]
Earlier authority check for create volume API
create() API in cinder/volume/api.py does the authority check
in cinder/volume/flows/api/create_volume.py.
This creates potential for disclosing information during error checking
prior to user authorization being checked.
This fix will do authority check to create() itself, so that
it is done before proceeding with the rest of code flow.
Efficient volume copy for generic volume migration
Currently Cinder uses dd command for data copy of volume migration,
but the copy always copy full blocks even if the source data contains
many null and zero blocks. The dd command has an option conv=sparse
to skip null or zero blocks for more efficient data copy.
However, if the destination volume is not zero cleared beforehand,
we should copy full block from source to dest volume to cleanup dest
volume in order to avoid security issue.
If the volume pre-initilization(zero cleared) is ensured beforehand,
we can skip copy of null and zero blocks to destination volume by
using sparse copy.
In order to use this option properly, we have to check
sparse_copy_volume capability for destination backend driver via
RPC API before volume copy.
This patch also adds sparse_copy_volume capability flag into volume
stats of LVM and NFS drivers to enable efficient copy for these
backends.
Vincent Hou [Thu, 28 May 2015 03:18:09 +0000 (11:18 +0800)]
Volume status management during migration
This patch proposes a new implementation for the status and
the migration_status for volumes.
* The initial migration_status is None, meaning no migration has been
done; Migration_status 'error' means the previous migration failed.
Migration_status 'success' means the previous migration succeeded.
* If the key 'lock_volume' is set to True from the request, the volume
status should be set to 'maintenance' during migration and goes
back to its original status after migration. Otherwise, if the
key 'lock_volume' is set to False, the volume status will remain the
same as its original status. The default value for lock_volume is
False and it applies to the available volume.
* From the REST's perspectives, all the create, update and delete
actions are not allowed if the volume is in 'maintenance', because
it means the volume is out of service. If it is not in maintenance
mode, the migration can be interrupted if other requests are
issued, e.g. attach. For the termination of migration, another
patch will target to resolve it.
DocImpact
APIImpact The key 'lock_volume' has been added into the API,
telling the volume to change the status to 'maintenance' or not.
The migration_status has been added into results returned
from volume list command, if the request is from an admin.
Code Review / openstack-build / cinder-build.git / log