Zhiteng Huang [Sat, 18 May 2013 14:21:28 +0000 (22:21 +0800)]
Implement QoS support for volumes
This patch is to implement generic Quality-of-Service(QoS) support for volumes.
The goal is to add an interface so that cloud/Cinder admins can use to set
volume QoS, which can be enforced either in hypervisor or on Cinder back-end
or both. QoS specifications are added as a standalone (only visible to admin)
entity. So admin can create/update/delete and associate/disassociate QoS
specifications to other entities, in this case volume types.
Note that while it's possible for Cinder to set the granularity of QoS control
to every single volume, this patch puts the control granularity to the level
of volumes of the same type to minimize the impact of other Cinder parts.
In other words, the design is to bond QoS with volume types. So Cinder admin
can associate volume types with QoS specifications, and volumes of same volume
type share the same QoS specifications.
QoS can mean a lot different things that it's unlikely we can come up with a
interpretation that all vendors can agree on. So the approach this
implementation takes is to make Quality-of-Service specs as free-from, i.e.
expressed as key/value pairs.
Changes:
- Add a quality_of_service_specs table, using adjacency list relation to store
a specs entry and its detailed specs in key/values. Note that to be able to
distinguish where should the QoS specs be consumed, each QoS specs entity
will have a 'consumer' (i.e. fixed key) with the value of where admin would
like the QoS policy to be enforced/consumed, currently these three values are
considered valid: 'front-end' (Nova Compute), 'back-end' (Cinder back-end),
'both'. The default value for 'consumer' is 'back-end';
- Add a new API extension 'qos_specs_manage' to allow list/create/update/
delete/associate/disassociate of QoS specs;
- Add volume/qos_specs internal API for qos specs manipulation;
- Add 'qos_specs' info to data structure when
initialize_connection() is called.
- Add 'qos_specs' to request_specs and filter properties for
a volume create request.
TODO
- Modify 'type_manage' API extension to be able to accept qos info.
- Modify volume_types.create() to accept qos info and do the checks.
Ronen Kat [Thu, 8 Aug 2013 09:12:37 +0000 (12:12 +0300)]
Backup driver for IBM Tivoli Storage manager (TSM)
An implementation of Cinder backup driver using TSM as a backend for
Cinder backups. The driver is a wrapper for the TSM command line
utility dsmc, and uses TSM image backup and restore.
John Griffith [Wed, 21 Aug 2013 19:05:30 +0000 (13:05 -0600)]
Dont crash service if sf cluster isnt available
Currently if the SolidFire driver is configured but the cluster
isn't available for some reason the ensure_export call will raise
an unhandled exception and crash the volume-service.
We should be able to handle things like loosing connectivity to
a single back-end without impacting the other volume-service
backends.
We'll wrap the ensure_export call in a try block here and
return None in the case that the connection can't be made.
This will keep the service from crashing and log an error
message that the connection timed out.
Additional work would include adding a periodic retry task
to the manager to try and start the backend service for us
on some regular interval incase the device comes back.
Kurt Martin [Thu, 22 Aug 2013 16:28:37 +0000 (09:28 -0700)]
3PAR driver add missing domain check on QOS cmd
The 3PAR drivers need to support CPGs that are not part of a
virtual domain on the 3PAR backend. This patch now checks
if the domain is set before calling the createvvset command.
This instance was overlooked in the following patch that has
already been merged: https://review.openstack.org/#/c/42697/
Julia Varlamova [Mon, 12 Aug 2013 08:49:25 +0000 (12:49 +0400)]
Refactor cinder/tests/test_volume.py
Refactor test_volume.py to remove code duplication.
- remove iterant definition of the 'expected' dict from
'test_create_delete_volume' and 'test_create_delete_snapshot'
method;
- move 'test_copy_volume_to_image_status_available',
'test_copy_volume_to_image_status_use' and
'test_copy_volume_to_image_exception' methods to new class
CopyVolumeToImageTestCase;
- move 'test_volume_get_active_by_window' and
'test_snapshot_get_active_by_window' to new class
GetActiveByWindowTestCase;
- create base class BaseVolumeTestCase for TestCase classes.
bp cinder-tests-improvement
This patch adds some cinder utils functions
that wrap calls into brick to automatically
populate cinder's root wrap helper. This
prevents propogating the recreation of the
sudo cinder-rootwrap string.
John Griffith [Mon, 12 Aug 2013 21:50:53 +0000 (21:50 +0000)]
Fix python 3 pep8 errors for print
The new auto-pulled requirements files exposed
python 3 compat issues in a number of modules.
We added these to tox.ini ignore temporarily,
this change updates the print routines in bin/cinder
and removes the debug cruft that was in the unit tests.
Will update requirements and test-requirements
appropriately in a follow up patch.
Jay S. Bryant [Wed, 21 Aug 2013 21:39:04 +0000 (16:39 -0500)]
Fix incorrect msgstr's to avoid translation errors
There are numerous msgstr's in the cinder.po files for en_US,
en_AU and en_GB that are incorrect translations of the associated
msgid. It appears that there are a number that are the result of
problems with past fuzzy translations gone wrong and some may also be the
result of copy/paste mistakes. Regardless of the origin it makes debug and
development very difficult if the string that is coming out in the
logs doesn't match what is expected. This patch fixes the numerous
bad msgstr's.
Once this commit is merged, Transifex will catch the change to
the Cinder po files and then spin a new Transifex Translations import.
Bill Owen [Fri, 16 Aug 2013 22:49:06 +0000 (15:49 -0700)]
GPFS use clone_image for creating volumes
If both source and target of gpfs create volume from image operation
are backed by gpfs storage use clone_image method for implementing
the move of image data to the new volume.
The copy_image_to_volume method is used only if this is not true,
and uses image_utils.fetch_to_raw to move image data to the new
volume.
Kurt Martin [Wed, 21 Aug 2013 16:38:54 +0000 (09:38 -0700)]
3PAR driver terminate connection host validation
The 3PAR backend does not allow FQDN host names (i.e. foo.rose.hp.com),
instead it requires just foo without the rose.hp.com. This patch will
now validate the host name in terminate connection by calling
_safe_host just as it was doing in initialize connection when creating
the host on the 3PAR backend.
Ken'ichi Ohmichi [Tue, 20 Aug 2013 03:26:36 +0000 (12:26 +0900)]
Allow to delete a volume in error_extending status
We must be able to remove "error_extending" volume, because we
cannot retrieve it without changing database status.
If trying to remove the volume, now it fails like the following:
$ cinder delete vol-test01
ERROR: Invalid volume: Volume status must be available or error,
but current status is: error_extending
$
"error_extending" also would be error status, so it is good to
remove the volume.
Luis A. Garcia [Tue, 20 Aug 2013 16:47:38 +0000 (16:47 +0000)]
Use system locale when Accept-Language header is not provided
Remove en_US as the default language when no header is provided, and use
None instead. Upon translation None will be defaulted to system as it
was before the translation changes.
Ronen Kat [Wed, 7 Aug 2013 12:37:07 +0000 (15:37 +0300)]
Generic backup_volume and restore_backup functions
Add implementation for backup_volume and restore_backup to the
VolumeDriver class, which uses brick for attach and detach.
Add default NotImplmeneted implmentation for non block drivers:
remotefs, coraid, gpfs, scality, sheepdog and sm.
Joel Coffman [Tue, 20 Aug 2013 17:02:24 +0000 (13:02 -0400)]
Relax policy so owner can access encryption info
The admin only policy is too restrictive to allow Nova to access a
volume's encryption metadata using the owner's request context. Hence,
this commit relaxes the policy for the volume encryption metadata
API extension so the metadata is accessible to the volume's owner.
Avishay Traeger [Tue, 20 Aug 2013 15:53:47 +0000 (18:53 +0300)]
Fix Fibre Channel attach for single WWN
The code allowed for only a string of the WWN or a list of them.
Unfortunately unicode is also returned, for which the attach fails.
This patch allows for unicode as well.
John Griffith [Fri, 16 Aug 2013 23:21:21 +0000 (17:21 -0600)]
Add accept_transfer to solidfire driver
The SolidFire cluster is tenant aware and as such
needs to change account association on volume transfer.
We use the project_id to build this account, so read that
in here and create a new account if necessary and re-assign
the existing volume to the new tenant account.
John Griffith [Fri, 16 Aug 2013 23:12:11 +0000 (17:12 -0600)]
Added need info to accept_transfer
Drivers that implement accept_transfer will need
things like the new user_id and project_id. We were
not including this in the original add, in order for
drivers that are tenant aware this information will be
necessary.
Also the get_volume call in the transfer was using
the new user context, so the volume would never be
found. We fix this here by providing an elevated
context to the get_volume call when accepting the
transfer.
Alan Jiang [Tue, 20 Aug 2013 05:22:44 +0000 (00:22 -0500)]
Allow volume create from source unless in error status
This patch restores the source volume status checking behavior
to allow create_volume function to proceed unless it is in
error status. In FibreChannel attached environment, it is typical
to have a volume attached before the volume clone starts. The
operating system owns the volume will ensure the consistency.
Luis A. Garcia [Mon, 19 Aug 2013 20:19:57 +0000 (20:19 +0000)]
Avoid serializing CinderExceptions before they are translated
CinderExceptions were being unicode()'d when being wrapped in an
HTTPException, and this was causing the delayed translation to fail for
those errors.
Also, CinderExceptions have a 'message' class attribute that holds the
generic error message template, e.g. "Backup %(backup_id)s is not
found", unfortunately, because the names are the same, it was
overshadowing the actual exception instance 'message', e.g. "Backup 1 is
not found", when translating. This patch puts the exception's actual
message in a new field called 'msg'.
This patch adds the ability to pass in a custom
root_helper for executing commands. This is needed
for other projects that need a custom root-wrapper,
such as nova.
John Griffith [Mon, 19 Aug 2013 21:04:15 +0000 (15:04 -0600)]
Standardize on ID for log messages
We have some places where logs use name to identify
a volume and others where we use ID. Let's standardize
on the UUID here as that's typically the unique identifier
we use in most places anyway. Even though name is a
derviative it seems better to be consistent with this
and use the UUID by itself.
Joshua Harlow [Mon, 19 Aug 2013 18:31:56 +0000 (11:31 -0700)]
Reduce hidden effects of sqlalchemy objects
The flows are currently holding onto sqlalchemy
due to bug #1214083 and this is causing a problem
with gettextutils which appears to be deepcopying
the raw objects, of which one of those is sqlalchemy
objects, which can't seem to be deepcopied.
Kurt Martin [Mon, 19 Aug 2013 18:14:42 +0000 (11:14 -0700)]
Removed need for domain in 3PAR drivers
The 3PAR drivers need to support CPGs that are not part of a
virtual domain on the 3PAR backend. This patch removes the need
for the CPG to be part of a domain and changes a few of the commands
that took the domain as an option.
Avishay Traeger [Sun, 11 Aug 2013 16:40:10 +0000 (19:40 +0300)]
Allow Cinder to call Nova client
This code allows Cinder to call Nova client functions. This will be used
for online migration and guest-assisted snapshots, which are both in
progress.
Avishay Traeger [Thu, 15 Aug 2013 11:31:03 +0000 (14:31 +0300)]
Storwize/SVC: More error logging
Some cases had exception raised without logging the error, and one
case where FC was not configured properly raised an IndexError rather
than presenting a clear message.
Kurt Martin [Fri, 16 Aug 2013 15:48:03 +0000 (08:48 -0700)]
Fixes SSH injection threat in 3PAR driver
The setqos ssh command was not built up correctly when the following
patch https://review.openstack.org/#/c/37697/ landed for cleaning up
the SSH calls from injection attacks in the 3PAR driver.
The command was in the following format causing the injection threat
due to the spaces in the second item in the list:
['setqos', '-io 5000 -bw 500M vvset:vvs-JOHB2Oj0QJ2UaWatwbe7Bg']
When it should actually be in the following format:
['setqos', '-io', '5000', '-bw', '500M', 'vvset:vvs-JOHB2Oj0QJ2UaWatwbe7Bg']
This patch fixes an append vs. extend that was introduced in patch
https://review.openstack.org/#/c/42241
Kurt Martin [Thu, 15 Aug 2013 23:22:31 +0000 (16:22 -0700)]
Fix SSH injection threat in 3PAR driver
The setqos ssh command was not built up correctly when the following
patch https://review.openstack.org/#/c/37697/ landed for cleaning up
the SSH calls from injection attacks in the 3PAR driver.
The command was in the following format causing the injection threat
due to the spaces in the second item in the list:
['setqos', '-io 5000 -bw 500M vvset:vvs-JOHB2Oj0QJ2UaWatwbe7Bg']
When it should actually be in the following format:
['setqos', '-io', '5000', '-bw', '500M', 'vvset:vvs-JOHB2Oj0QJ2UaWatwbe7Bg']
This patch updates each driver to provide
the same mechanism for reporting the version
of the driver. It also includes fixing the
reported driver version at get_volume_stats()
time to be the version set in the driver.
When the manager starts the driver it now
logs the driver's name and version in the log file.
Victor Rodionov [Fri, 9 Aug 2013 05:40:02 +0000 (09:40 +0400)]
Refactor Nexenta driver
Moving NEXENTA_OPTIONS out of nexenta/volume.py and splitting the
options. This change is to avoid duplication as the iSCSI and NFS driver
will use the same configuration options.
The usage of this option occurred only in that module.
Rename this options to num_volume_device_scan_tries,
according to discussion on IRC.
The old one marked as depricated option.
Luis A. Garcia [Wed, 10 Jul 2013 00:50:12 +0000 (00:50 +0000)]
Externalize error messages in the v2 API
This patch does more internationalization for the REST API error
messages that don't currently have it to take advantage of the new
support added by bp user-locale-api to show error messages in the locale
requested by the user through the Accept-Language HTTP header.
We only do v2 because consumers have used the response error message in
the past for error checks, so changing it in v1 too would break them.
Code Review / openstack-build / cinder-build.git / log