John Griffith [Sat, 12 Dec 2015 18:32:57 +0000 (18:32 +0000)]
Fix invalid cache image-volume creation
During the volume ref creation in clone_image_volume
we were including a _name_id attribute which is invalid
in the base olso object. The result is silent failure
for a key error. Note that the comments for the column
in the db specifcally state "do not call directly".
The variable is also denoted as private by the _ prefix.
Fix this by just popping the "_name_id' attribute.
In addition there were a number of things that needed
some cleanup:
1. Don't iterate over list of tuples to get K/V for dict
Just use Dict(xxxx)
2. Don't use delete dict.key
This isn't safe, because if the key DNE you'll get a key exception
Instead use pop which is safe if key DNE
3. Add an error message when this fails, because currently the logs
don't give an operation a chance of knowing it failed, let alone
an opportunity to debug it
Finally the cache volume creation is still calling the db
directly and doing a resource create as opposed to
using the new volume object create handler. I went ahead
and converted this to use objects, however that resulted in
breaking two of the volume-->image unit tests that use cinder
as a glance backend.
I've opened bug 1525773 for those issues and added a skip
to the tests in question.
peter_wang [Tue, 8 Dec 2015 08:18:31 +0000 (03:18 -0500)]
VNX: Fix issue in deleting cg/cgsnapshot
If user deletes a cg/cgsnapshot quickly after its creation,
driver would report 'not found' warning, and return success
for the deletion, but actually, the object underlying the VNX
is not deleted.
This fix tries to wait the cg and cg snapshot to be available
to make sure it's eligible for deletion.
During copy_volume_to_image(), this tries to return
volume['volume_type'] attribute in a volume object, but
if the volume does not have a volume type, this would
trigger a db call which raise an 404 not found error.
Volume object should check volume_type_id at first and
then decide whether returning volume_type or None to
avoid VolumeTypeNotFound exception.
Also this change includes LOG.info messages fix to show
right behavior of copy_volume_to_image().
apoorvad [Wed, 9 Dec 2015 17:42:21 +0000 (09:42 -0800)]
Retype functionality in Tintri driver
At this point default retype behavior is as follows. If a volume changes
type on the same backend then that volume will be migrated. Since Tintri
VMstore does not differentiate between volume types, we can avoid this
migration by implementing retype functionality.
Ryan McNair [Wed, 9 Dec 2015 15:29:27 +0000 (15:29 +0000)]
Fix non-migration swap with error
The fix I242c2c7c4a7197bbce04d0b3d75688f989ea1fd5 for non-migration
swap volume caused drivers which don't support volume swapping,
e.g. Ceph, to put the volumes in an incorrect state (additional
details can be found in bug #1489744 report). This patch adds an
additional check to ensure no errors occurred during the swap before
it completes the volume status updates, as well a test-case for
the non-migration swap scenario.
Dave McCowan [Mon, 7 Dec 2015 19:25:24 +0000 (14:25 -0500)]
Check context before returning cached value
The key manager caches the value of barbican client to be reused,
saving an extra call to keystone. The cached value is only
applicable to the current context, so the context must be checked
before returning the cached value.
Alex O'Rourke [Thu, 29 Oct 2015 22:04:32 +0000 (15:04 -0700)]
3PAR: Implement v2 replication (managed)
This patch implements the managed side of v2 replication in the HPE
3PAR driver.
Both sync and periodic replication modes are supported. Each
replication_device entry should have a replication_mode value set
to sync|periodic.
A volume type extra_spec value of replication:mode
should also be set. If replication:mode is periodic,
replication:sync_period should be set as well. Which replication_device
entry(s) are used is determined by the value of replication:mode set for
each volume type.
cinder.conf should have the replication config group (3parfcrep)
and at least one other target group (3parfc) as such:
Most changes in this patch:
- make sure that Request.body is set to bytes;
- replace jsonutils.dumps with jsonutils.dump_as_bytes;
- replace json.loads with oslo_serialization.jsonutils.loads;
- replace dict.iteritems with dict.items.
Shay Halsband [Thu, 19 Nov 2015 14:42:46 +0000 (16:42 +0200)]
XtremIO: fix iscsi chap discovery bug
* The set and get parameters for discovery auth have different
names and caused a confusion that led to a bug, fixed in this
commit.
* Add unit test to make sure the right parameters are used
Sean McGinnis [Sat, 5 Dec 2015 18:19:52 +0000 (12:19 -0600)]
Fix StrOpts with integer defaults
A couple config options sneaked in defined as strings but
really expecting integer values. This causes errors with the
latest oslo.config that expects the appropriate type.
This patch changes these instances to be the expected IntOpt.
Kendall Nelson [Fri, 4 Dec 2015 00:10:39 +0000 (18:10 -0600)]
Dynamically Pull Out Option Sections
Previously in the generation of the opts.py file, the option groups
were hard coded. Now, they are pulled out as the lines are parsed.
If a group doesn't already exist in the dictionary it gets added and
then the options are added to the section. If the group already exists,
the options are added to the proper section as you would expect, and if
the options being registered don't have a group, they are added to the
default section.
This patch also moves the check for a non-zero exit code from
oslo-config-generator to the right location in the file. This
logic had been put too late in the file and was not catching
oslo-config-generator failures as it should have been.
Chris Morrell [Wed, 2 Dec 2015 22:27:37 +0000 (22:27 +0000)]
Address potential races in SolidFire VAG
While building the initial volume access group solution I realized there were
race conditions present. This patch attempts to address those by being far
more careful during VAG creation and removal. Tracking multiattach volume
detaches around VAG volume removal introduced all sorts of future race
conditions so instead multiattach volumes are treated differently and are only
removed from VAGs on volume removal. Not ideal but it'll get the job done.
Think I covered all the new branches with unit tests. Complexity was introduced
but it should be worth it due to the safety that was also introduced.
The following patch updates migrate_volume,
migrate_volume_completion, and update_migrated_volume
APIs to use volume versionedobjects. Changes were
made to be backwards compatible with older RPC clients.
It only includes changes to the core cinder code.
Changes in the drivers are left to each driver
maintainer to update.
Note that this patch DOES NOT try to use object dot
notation everywhere, since it would increase the
size of the patch. Instead, it will be done in
subsequent patches.
Earlier, quotas used to authenticate to endpoint from not required
option located in keymgr.encryption_auth_url. Now, the required
auth_uri option from config file is used to authenticate to
keystone.
Co-Authored-By: Michal Dulko <michal.dulko@intel.com>
Change-Id: I1076527704f8def2c6755c060df49232e5ebe805
Closes-Bug: 1516085
With the current code, it's possible to end up with a zero-size
persistence file (or even corruption of the contents) if the
process gets killed or the system takes a power outage.
Switch to a "write to temp file and rename" model for writing the
persistence file. This will make it more robust against unclean
process termination or unclean system shutdown.
Kendall Nelson [Wed, 2 Dec 2015 20:59:33 +0000 (14:59 -0600)]
Adding devref about genconfig
This adds a description of the generation of opts.py and
cinder.conf.sample, the commands to generate them, and the scripts
that do the work generating them.
Eric Harney [Tue, 1 Dec 2015 23:14:17 +0000 (18:14 -0500)]
LIO: Handle initiator IQNs as case insensitive
If there is a mismatch in case between the stored
initiator IQN in the LIO ACL vs. the IQN provided
in a later call, rtstool would fail to find the
existing ACL.
Compare IQNs in a case insensitive manner to ensure
they will always match as intended. (Refer to RFC
3720 section 3.2.6.1 for details on this.)
This patch replaces usages of mox with mock in the VMDK driver unit tests.
It also refactors the unit tests for copy_volume_to_image so that those
tests test only a single method.
VMware: Unit test refactoring (image to vol - 2/2)
There are cases where a single test tests multiple methods. This
patch refactors the unit tests for the following methods in the
vmdk module to fix this issue:
* _create_virtual_disk_from_preallocated_image
* _create_virtual_disk_from_sparse_image
* _fetch_stream_optimized_image
There will be follow-up patches to fix the remaining unit tests.
VMware: Unit test refactoring (image to vol - 1/2)
There are cases where a single test tests multiple methods. This
patch refactors the unit tests for the following methods in the
vmdk module to fix this issue:
* copy_image_to_volume
* _create_volume_from_non_stream_optimized_image
There will be follow-up patches to fix the remaining unit tests.
Sean McGinnis [Fri, 27 Nov 2015 21:18:50 +0000 (15:18 -0600)]
Remove deprecated LVM ISCSI and ISER Drivers
The LVMISCSIDriver and LVMISERDrivers were deprecated in the Kilo
release. This removes both drivers and related test code.
Any existing users of LVMDriver should switch to using the
LVMVolumeDriver and set the iscsi_helper to the desire iSCSI helper
(tgtadm, lioadm, or iseradm).
Some unit tests under these driver test classes were not actually
specific to these drivers. Relevant unit tests were moved to a
different test class if they looked useful.
LisaLi [Thu, 12 Nov 2015 06:08:11 +0000 (06:08 +0000)]
Delete unused codes in rbd.retype
Before rbd.retype is called, encryption and host have
been checked so that encryptions are different and
host are same.
Meanwhile, as extra_spec is not used during the
rbd driver. As a result, even if they are different
rbd.retype can still return True which means no need
to do further migration.
Victor Stinner [Tue, 1 Dec 2015 10:00:03 +0000 (11:00 +0100)]
xio: fix regression in authentication
The change I75a5dee2fd1f8cf45dbf416b0353736a43efd8e8 replaced
base64.encodestring(auth_key)[:-1] with
oslo_serialization.base64.encode_as_text(auth_key)[:-1]. It's wrong
because base64.encodestring() adds '\n' to the output, wheras
oslo_serialization.base64.encode_as_text() doesn't add '\n'.
This change removes '[:-1]' to strip the newline, it's no more
needed.
Volume object was missing fields that were defined on
the SQLAlchemy model as relationships or backrefs such as
consistencygroup, glance_metadata and snapshots.
This commit adds fields representing these relationships.
LisaLi [Fri, 13 Nov 2015 02:09:03 +0000 (02:09 +0000)]
Add retype in lvm driver
During retype operation, it makes sure that volume
types have same encryption, and same backend qos
before moving on.
In lvm driver, no extra_spec and qos are used. As
a result, if retype a volume to same host, it just
needs to return True, and no needs to migrate the
volume.
Gorka Eguileor [Mon, 30 Nov 2015 11:04:58 +0000 (12:04 +0100)]
Take into consideration races in XtremIOClient3
When working with FC and xtremio is using XtremIOClient3 on FC's
terminate_connection we can get VolumeBackendAPIException saying a
resource could not be found when we shouldn't.
The cause is that Client3 is open to race conditions on 2 methods,
find_lunmap and num_of_mapped_volumes, where a list of mappings is first
retrieved and then we iterate this list to retrieved additional info on
the mappings.
The race would happen if one of the mappings is removed from the backend
in the time it takes to retrieve the additional info after we have
retrieved the list.
This patch fixes this issue by ignoring any mappings that have been
removed and are now NotFound when retrieving additional information for
a mapping in those 2 methods.
This patch also fixes this kind of race problems on volume creation
since it uses find_lumap method.
zhangsong [Fri, 13 Nov 2015 16:44:53 +0000 (00:44 +0800)]
Fix the bug of can't get the desired image info
When we use 'qemu-img info' command to get image info, some kinds
of image may show an 'unavailable' disk size. For example:
$qemu-img info sheepdog:volume-1af1afa0-7820-4063-9b90-4fa6161a74a9
image: sheepdog:volume-1af1afa0-7820-4063-9b90-4fa6161a74a9
file format: raw
virtual size: 10G (10737418240 bytes)
disk size: unavailable
$qemu-img info -f rbd rbd:data/foo
image: rbd:data/foo
file format: rbd
virtual size: 10G (10737418240 bytes)
disk size: unavailable
cluster_size: 4194304
The current code didn't consider 'unavailable' size, and it will
raise a ValueError in this case. This patch fixed this bug by
adding the judgment of 'unavailable' size.
Sean McGinnis [Fri, 27 Nov 2015 20:12:06 +0000 (14:12 -0600)]
Cleanup orphaned code in sqlalchemy API
There were several methods in the sqlalchemy code that were no longer
being used anywhere. While some of these have potential for being useful,
they are not in use and should not be sitting around the codebase waiting
for someone to need them.
Code Review / openstack-build / cinder-build.git / log