Anthony Lee [Sat, 5 Dec 2015 00:27:14 +0000 (16:27 -0800)]
3PAR: Implement un/manage snapshot support
Implements support for managing and unmanaging snapshots to
the HPE 3PAR FC and iSCSI drivers.
This patch now allows snapshots to be removed from OpenStack
management but still left on the 3PAR backend. Snapshots on
the 3PAR backend can also be managed by OpenStack.
Alex O'Rourke [Thu, 29 Oct 2015 22:04:32 +0000 (15:04 -0700)]
3PAR: Implement v2 replication (unmanaged)
This patch implements the unmanaged side of v2 replication in the HPE
3PAR driver.
Both sync and periodic replication modes are supported. Each
replication_device entry should have a replication_mode value set
to sync|periodic.
A volume type extra_spec value of replication:mode
should also be set. If replication:mode is periodic,
replication:sync_period should be set as well. Which replication_device
entry(s) are used is determined by the value of replication:mode set for
each volume type.
cinder.conf should have the replication config group:
Alex O'Rourke [Thu, 29 Oct 2015 22:04:32 +0000 (15:04 -0700)]
3PAR: Implement v2 replication (managed)
This patch implements the managed side of v2 replication in the HPE
3PAR driver.
Both sync and periodic replication modes are supported. Each
replication_device entry should have a replication_mode value set
to sync|periodic.
A volume type extra_spec value of replication:mode
should also be set. If replication:mode is periodic,
replication:sync_period should be set as well. Which replication_device
entry(s) are used is determined by the value of replication:mode set for
each volume type.
cinder.conf should have the replication config group (3parfcrep)
and at least one other target group (3parfc) as such:
Sean McGinnis [Sat, 5 Dec 2015 18:19:52 +0000 (12:19 -0600)]
Fix StrOpts with integer defaults
A couple config options sneaked in defined as strings but
really expecting integer values. This causes errors with the
latest oslo.config that expects the appropriate type.
This patch changes these instances to be the expected IntOpt.
Kendall Nelson [Fri, 4 Dec 2015 00:10:39 +0000 (18:10 -0600)]
Dynamically Pull Out Option Sections
Previously in the generation of the opts.py file, the option groups
were hard coded. Now, they are pulled out as the lines are parsed.
If a group doesn't already exist in the dictionary it gets added and
then the options are added to the section. If the group already exists,
the options are added to the proper section as you would expect, and if
the options being registered don't have a group, they are added to the
default section.
This patch also moves the check for a non-zero exit code from
oslo-config-generator to the right location in the file. This
logic had been put too late in the file and was not catching
oslo-config-generator failures as it should have been.
Chris Morrell [Wed, 2 Dec 2015 22:27:37 +0000 (22:27 +0000)]
Address potential races in SolidFire VAG
While building the initial volume access group solution I realized there were
race conditions present. This patch attempts to address those by being far
more careful during VAG creation and removal. Tracking multiattach volume
detaches around VAG volume removal introduced all sorts of future race
conditions so instead multiattach volumes are treated differently and are only
removed from VAGs on volume removal. Not ideal but it'll get the job done.
Think I covered all the new branches with unit tests. Complexity was introduced
but it should be worth it due to the safety that was also introduced.
The following patch updates migrate_volume,
migrate_volume_completion, and update_migrated_volume
APIs to use volume versionedobjects. Changes were
made to be backwards compatible with older RPC clients.
It only includes changes to the core cinder code.
Changes in the drivers are left to each driver
maintainer to update.
Note that this patch DOES NOT try to use object dot
notation everywhere, since it would increase the
size of the patch. Instead, it will be done in
subsequent patches.
Earlier, quotas used to authenticate to endpoint from not required
option located in keymgr.encryption_auth_url. Now, the required
auth_uri option from config file is used to authenticate to
keystone.
Co-Authored-By: Michal Dulko <michal.dulko@intel.com>
Change-Id: I1076527704f8def2c6755c060df49232e5ebe805
Closes-Bug: 1516085
With the current code, it's possible to end up with a zero-size
persistence file (or even corruption of the contents) if the
process gets killed or the system takes a power outage.
Switch to a "write to temp file and rename" model for writing the
persistence file. This will make it more robust against unclean
process termination or unclean system shutdown.
Kendall Nelson [Wed, 2 Dec 2015 20:59:33 +0000 (14:59 -0600)]
Adding devref about genconfig
This adds a description of the generation of opts.py and
cinder.conf.sample, the commands to generate them, and the scripts
that do the work generating them.
Eric Harney [Tue, 1 Dec 2015 23:14:17 +0000 (18:14 -0500)]
LIO: Handle initiator IQNs as case insensitive
If there is a mismatch in case between the stored
initiator IQN in the LIO ACL vs. the IQN provided
in a later call, rtstool would fail to find the
existing ACL.
Compare IQNs in a case insensitive manner to ensure
they will always match as intended. (Refer to RFC
3720 section 3.2.6.1 for details on this.)
This patch replaces usages of mox with mock in the VMDK driver unit tests.
It also refactors the unit tests for copy_volume_to_image so that those
tests test only a single method.
VMware: Unit test refactoring (image to vol - 2/2)
There are cases where a single test tests multiple methods. This
patch refactors the unit tests for the following methods in the
vmdk module to fix this issue:
* _create_virtual_disk_from_preallocated_image
* _create_virtual_disk_from_sparse_image
* _fetch_stream_optimized_image
There will be follow-up patches to fix the remaining unit tests.
VMware: Unit test refactoring (image to vol - 1/2)
There are cases where a single test tests multiple methods. This
patch refactors the unit tests for the following methods in the
vmdk module to fix this issue:
* copy_image_to_volume
* _create_volume_from_non_stream_optimized_image
There will be follow-up patches to fix the remaining unit tests.
Sean McGinnis [Fri, 27 Nov 2015 21:18:50 +0000 (15:18 -0600)]
Remove deprecated LVM ISCSI and ISER Drivers
The LVMISCSIDriver and LVMISERDrivers were deprecated in the Kilo
release. This removes both drivers and related test code.
Any existing users of LVMDriver should switch to using the
LVMVolumeDriver and set the iscsi_helper to the desire iSCSI helper
(tgtadm, lioadm, or iseradm).
Some unit tests under these driver test classes were not actually
specific to these drivers. Relevant unit tests were moved to a
different test class if they looked useful.
LisaLi [Thu, 12 Nov 2015 06:08:11 +0000 (06:08 +0000)]
Delete unused codes in rbd.retype
Before rbd.retype is called, encryption and host have
been checked so that encryptions are different and
host are same.
Meanwhile, as extra_spec is not used during the
rbd driver. As a result, even if they are different
rbd.retype can still return True which means no need
to do further migration.
Victor Stinner [Tue, 1 Dec 2015 10:00:03 +0000 (11:00 +0100)]
xio: fix regression in authentication
The change I75a5dee2fd1f8cf45dbf416b0353736a43efd8e8 replaced
base64.encodestring(auth_key)[:-1] with
oslo_serialization.base64.encode_as_text(auth_key)[:-1]. It's wrong
because base64.encodestring() adds '\n' to the output, wheras
oslo_serialization.base64.encode_as_text() doesn't add '\n'.
This change removes '[:-1]' to strip the newline, it's no more
needed.
Volume object was missing fields that were defined on
the SQLAlchemy model as relationships or backrefs such as
consistencygroup, glance_metadata and snapshots.
This commit adds fields representing these relationships.
LisaLi [Fri, 13 Nov 2015 02:09:03 +0000 (02:09 +0000)]
Add retype in lvm driver
During retype operation, it makes sure that volume
types have same encryption, and same backend qos
before moving on.
In lvm driver, no extra_spec and qos are used. As
a result, if retype a volume to same host, it just
needs to return True, and no needs to migrate the
volume.
Gorka Eguileor [Mon, 30 Nov 2015 11:04:58 +0000 (12:04 +0100)]
Take into consideration races in XtremIOClient3
When working with FC and xtremio is using XtremIOClient3 on FC's
terminate_connection we can get VolumeBackendAPIException saying a
resource could not be found when we shouldn't.
The cause is that Client3 is open to race conditions on 2 methods,
find_lunmap and num_of_mapped_volumes, where a list of mappings is first
retrieved and then we iterate this list to retrieved additional info on
the mappings.
The race would happen if one of the mappings is removed from the backend
in the time it takes to retrieve the additional info after we have
retrieved the list.
This patch fixes this issue by ignoring any mappings that have been
removed and are now NotFound when retrieving additional information for
a mapping in those 2 methods.
This patch also fixes this kind of race problems on volume creation
since it uses find_lumap method.
zhangsong [Fri, 13 Nov 2015 16:44:53 +0000 (00:44 +0800)]
Fix the bug of can't get the desired image info
When we use 'qemu-img info' command to get image info, some kinds
of image may show an 'unavailable' disk size. For example:
$qemu-img info sheepdog:volume-1af1afa0-7820-4063-9b90-4fa6161a74a9
image: sheepdog:volume-1af1afa0-7820-4063-9b90-4fa6161a74a9
file format: raw
virtual size: 10G (10737418240 bytes)
disk size: unavailable
$qemu-img info -f rbd rbd:data/foo
image: rbd:data/foo
file format: rbd
virtual size: 10G (10737418240 bytes)
disk size: unavailable
cluster_size: 4194304
The current code didn't consider 'unavailable' size, and it will
raise a ValueError in this case. This patch fixed this bug by
adding the judgment of 'unavailable' size.
Sean McGinnis [Fri, 27 Nov 2015 20:12:06 +0000 (14:12 -0600)]
Cleanup orphaned code in sqlalchemy API
There were several methods in the sqlalchemy code that were no longer
being used anywhere. While some of these have potential for being useful,
they are not in use and should not be sitting around the codebase waiting
for someone to need them.
Mitsuhiro Tanino [Fri, 13 Nov 2015 20:01:32 +0000 (15:01 -0500)]
Fix quotas issue during volume transfer
When admin transfers a volume with volume type from
one tenant to another, it correctly updates the quota
for the default values, but does not update for the
volume type values.
This patch fixes the problem to update quotas for
the volume type values.
wuyuting [Wed, 25 Nov 2015 02:00:44 +0000 (10:00 +0800)]
Declare multiattach is True in Dell Eqlx driver
An Eqlx volume can be attached to multi instances, however,
this feature is not declared in eqlx driver. As a result,
if a user create a multi-attached volume using eqlx as
backend, they will be failed.
In this commit, we declared multiattach is True in eqlx
driver.
Dirk Mueller [Thu, 29 Jan 2015 13:11:35 +0000 (14:11 +0100)]
RBD: Make snapshot_delete more robust
Since there is no equivalent of a force-snapshot-delete,
the normal snapshot-delete should be more robust to
also handle backend errors. In case the backend does not have
the image snapshot anymore, log an info message and succeed the
operation.
Kendall Nelson [Tue, 17 Nov 2015 18:39:58 +0000 (12:39 -0600)]
Hacking Checks for assertTrue/IsNone()
This patch adds a hacking check to make sure that assertEquals isn't
comparing the result value to True or None. When developers
use assertEquals(None, return_value) the check will catch it when
pep8 runs and suggest using assertIsNone(return_value) instead.
Similar situations will occur when trying to use assertTrue(True,
return_value).
This patch also makes the necessary changes that get caught by the
new hacking check.
Gorka Eguileor [Fri, 21 Aug 2015 17:06:07 +0000 (19:06 +0200)]
Improve metadata update operations
Currently our metadata operations in the DB and the API are less than
optimal, main issues are:
- To update metadata first we get all metadata in the API and add
requested update metadata, then on the DB we retrieve each of the
metadata keys and update one by one contents.
- When deletion of existing keys is requested in the DB we retrieve all
metadata contents and for those that are not in the new metadata we
retrieve the contents from the DB (again) and delete the entry.
This patch changes this so we no longer retrieve metadata in the API,
deletion does not retrieve any metadata from the DB and just makes 1
deletion request to the DB, and for changes we retrieve metadata once
and then issue a bulk change to create new entries and update existing
ones.
To allow atomic state changes across Cinder services we need to
implement a way to easily do compare-and-swap.
This patch adds methods to allow compare-and-swap on DB models and on
Cinder Versioned Objects as well.
Conditions for the compare part of the update can consist of:
- Inclusion: status == 'available'
- Exclusion: status != 'in-use'
- Multi-inclusion: status in ('available', 'error')
- Multi-exclusion: status not in ('attaching', 'in-use')
- Sqlalchemy filters
A complete example of usage would be the compare-and-swap used in volume
delete requests that has to take in consideration not only the status
but the attach and migration status as well as the volume not having
snapshots:
now = timeutils.utcnow()
expected = {'attach_status': db.Not('attached'),
'migration_status': None,
'consistencygroup_id': None}
good_status = ('available', 'error', 'error_restoring',
'error_extending')
if not force:
expected.update(status=good_status)
# Volume cannot have snapshots if we want to delete it
filters = [~sql.exists().where(models.Volume.id ==
models.Snapshot.volume_id)]
It can also be specified whether to save already dirtied fields from the
objects or not and by default (if no expected_values argument is
provided) it will make sure that the entry in the DB has the same values
as the objects we are saving.
We can select values based on conditions using Case objects in the
'values' argument. For example:
Exclusion and multi-exclusion will handle, by default, NULL values like
Python does instead of like SQL does, so NULL values will be considered
different than any non NULL values. That way if we search for something
not equal to 1 we will also get NULL values.
WARNING: SQLAlchemy does not allow selecting order of SET clauses, so
for now we cannot do things like
{'previous_status': model.status, 'status': 'retyping'}
because it will result in both previous_status and status being set to
'retyping'. Issue has been reported [1] and a patch to fix it [2] has
been submitted.
[1]: https://bitbucket.org/zzzeek/sqlalchemy/issues/3541
[2]: https://github.com/zzzeek/sqlalchemy/pull/200
Nicolas Trangez [Wed, 25 Nov 2015 13:45:09 +0000 (14:45 +0100)]
Revert "Add Scality SRB driver"
The Scality SRB kernel-driver is being re-designed to optimize for
specific workloads, which are (initially) not compatible with VM-style
block device access. Not to confuse users, we believe it's advisable to
remove the Cinder SRB driver in the meantime.
Code Review / openstack-build / cinder-build.git / log