Patrick East [Fri, 11 Sep 2015 21:12:03 +0000 (14:12 -0700)]
Fix Pure get pgroup volume snapshot name
When creating a volume from a consistency group snapshot we have to go
look up the volume level snapshot from the group. Previously we could
just directly get to the cgsnapshot model from the snapshot. After we
switched to using the snapshot objects however we cannot do that (at
least not until we have cgsnapshot objects too).
As an interim solution we did a query to the REST API on the array to
just list all of the snapshots. Unfortunately that doesn’t work on all
versions of the Purity REST API. To have a more usable solution we need
to do a db lookup to get the pgsnapshot.
It’s not ideal, and will be removed ASAP, but for now we need to at
least not regress in functionality.
The VMDK driver configuration uses invalid product name "VC"
instead of "vCenter". This patch replaces "VC" with "vCenter"
in help messages and code documentation. This change is
consistent with the rest of driver documentation.
DocImpact
Replaced "VC" with "vCenter" in driver config option help
messages
Xing Yang [Thu, 10 Sep 2015 19:31:39 +0000 (15:31 -0400)]
Add ScaleIO Cinder driver commands
ScaleIO Cinder driver commands are currently missing from
Cinder's rootwrap filters. Since we are not using os-brick's
rootwrap filters in Liberty, these missing commands need to be
added to Cinder's rootwrap filters.
John Griffith [Thu, 10 Sep 2015 00:59:38 +0000 (00:59 +0000)]
SolidFire provider_id for snapshots on init
We've added the provider_id field to the database for both
volumes and snapshots. This however couldn't be used because
there was no upgrade/migration path included for backends to
populate these fields (ie upgrade from K to L).
This patch fixes the provider_id columns for snapshots and volumes
both, and updates these fields on backend init. This fixes the
issue of not being able to "rely" on the database entries, and it
also fixes some issues with inefficiencies in how we were deleting
resources.
Now we can just use the provider_id info from the reference passed
in on the delete calls.
Tom Barron [Mon, 24 Aug 2015 17:21:48 +0000 (13:21 -0400)]
Fix NetApp loop in clone of NFS backed images
If there is a list of locations for glance images and an unsuitable
location is encountered when iterating through the list, current code
breaks out of the loop, failing to find later, valid locations.
Here we fix this issue by ignoring invalid locations and continuing
through the loop rather than breaking out when we find invalid locations.
Tom Barron [Wed, 26 Aug 2015 20:07:00 +0000 (16:07 -0400)]
backup init_host cleanup exception handling
Current cleanup of leftover backups, volumes, and temporary
volumes and snapshots is not resilient in the face of exceptions
such as failures in detach operations.
This commit adds appropriate try/except logic to this section
of code, thereby ensuring that the backup service can start even
if exceptions are encountered when attempting to cleanup detritus
from earlier failed backup operations.
Kendall Nelson [Wed, 9 Sep 2015 13:21:25 +0000 (08:21 -0500)]
Making opt names consistent
Wherever there are lists of options being registered, the name of
the list needs to be plural to be consistent with all other opt lists
being registered elsewhere. This change is necessary for the dynamic
generation of cinder.sample.conf.
Co-Authored-By: Jacob Gregor <jgregor@us.ibm.com>
Change-Id: I1434a195016a2a595d0205d73a78a60cd94137f8
Michal Dulko [Wed, 9 Sep 2015 15:20:18 +0000 (17:20 +0200)]
Add ConsistencyGroup object entries to linstack.py
Change I3febb95bb2b3dd8f10da4f01ffa705e39cb20520 wasn't passing pylint
gate because of missing objects_ignore_messages entries in linstack.py.
This commits adds these entries.
John Griffith [Wed, 9 Sep 2015 15:15:31 +0000 (15:15 +0000)]
Pass in snapshot refs for host on provider_update
Now that snapshot_get_by_host is fixed we can and should
pass in a list of snapshot-refs to the drivers update_provider
method (just like we do with volumes).
John Griffith [Tue, 8 Sep 2015 21:29:05 +0000 (15:29 -0600)]
Return a tuple from SolidFire update_provider_info
The SolidFire update_provider_info is only returning
a list of updated volumes, but the manager call (and
the method def in the reference) calls for a tuple of
volumes and snapshots.
This patch fixes the return to be a tuple, a follow on
will be needed to add snapshot updates.
Eric Harney [Tue, 8 Sep 2015 20:23:45 +0000 (16:23 -0400)]
Add unmanage default implementation to VolumeDriver
Currently, only drivers that inherit from ManageableVD
get the default unmanage() implementation.
This is not correct, because drivers can support unmanage
without driver code. Fix unmanage() in VolumeDriver to
support the LVM driver's unmanage operation.
(Regression from the refactoring done when introducing
abc classes.)
Eric Harney [Tue, 8 Sep 2015 17:31:31 +0000 (13:31 -0400)]
Add migrate_volume to BaseVD
migrate_volume() only exists in the deprecated VolumeDriver()
class and not BaseVD. This means that drivers inheriting from
*VD hit attribute errors when trying to call driver.migrate_volume().
I suspect this indicates that we should rework some of our
abc class structure a little bit, but this seems like a
reasonable fix for now.
This patch disables the 3PAR drivers if an admin tries to
enable the generic image volume cache capability.
The 3PAR drivers don't work with this currently because of the way volume
cloning works in the drivers. When a volume is cloned, it's done so
in the background, which makes the clone available immediately for attaching.
The problem is, a background cloning also locks the original volume from
being modified. The generic image cache does a clone, and then an extend
volume, which fails for the 3PAR drivers currently.
Sean McGinnis [Sat, 5 Sep 2015 18:04:05 +0000 (13:04 -0500)]
Add os-brick's scsi_id command to Cinder rootwrap
FC drivers are failing on calls to scsi_id. This command
is in the os-brick rootwrap filters, but that file is not
being pulled in to Cinder yet.
As a temporary fix until that work can be completed, this
patch adds the missing command directly to volume.filters
to allow this command to run. Follow up work will be needed
to remove this and other legacy brick commands from the
volume.filters file and instead pull in os-brick's
os-brick.filters values.
Jacob Gregor [Fri, 4 Sep 2015 21:30:12 +0000 (16:30 -0500)]
Error message in update_migrated_volume was incorrect
If an error was encountered in self.vg.rename_volume the error
message tried to use volume['name'], which is an invalid key.
The invalid key error would then cause the retype/migration to
abort in the middle of the process and leave original and temporary
volume in place. This patch fixes this bug by providing a valid key.
Fix HDS HNAS driver logging password as plain text
HDS HNAS driver logs every command that is sent to HNAS. Some
commands need the HNAS password, so the driver ends up logging
this password as plain text. This patch changes these commands
syntax so that oslo_utils.strutils.mask_password can successfully
mask this password.
John Griffith [Wed, 2 Sep 2015 20:08:49 +0000 (20:08 +0000)]
Add mechanism to update snapshot provider_id
During the L release we added the ability to store
provider_id info for volumes and snapshots. We did
not however add a mechanism for this info to be updated
on upgrades.
This patch adds on to the volume update mechanism and
gives a driver the opportunity to set provider_id columns
that aren't set on snapshots just as it does for volumes.
Commit 212aff327a36f925be3be69b9b54b946dbdd5c2a replaced
'cinder.openstack.common.fileutils.file_open' with 'open'
from builtins. It also updated unit tests to patch 'six.
moves.builtins.open' instead of 'cinder.openstack.common.
fileutils.file_open'. This resulted in intermittent failure
of test 'test_create_backing_from_stream_optimized_file'
in VMwareVcVmdkDriverTestCase. The failure happens because
the test patches 'open' from builtins and tests code which
throws an exception in a 'with open(...)' block. This
exception is handled in excutils.save_and_reraise_exception
which uses 'with open(...)' and the exception handling
fails due to the global patching of 'open'.
This patch replaces the global patching of builtins 'open'
with a patch decorator which binds 'open' to the namespace
of the code under testing.
Volume creation will fail if the VMDK driver selects
an ESX host in maintenance mode for volume creation.
This may happen even if there are usable ESX hosts
in vCenter. This patch fixes the problem by ignoring
ESX hosts in maintenance mode during volume creation.
The 3PAR drivers were missing the method
update_migrated_volume, which would then cause
all sorts of problems after a volume was migrated
into a 3PAR array.
Switch SVC driver to use lsportfc to determine FC target WWPNS
Switched the startup code within SVC driver to use the
SVC lsportfc command rather than the lsnode command to determine the
target WWPNs for an SVC node, and updated SVC testcases.
The SVC lsportfc command returns all of the FC ports so that
all of the FC ports can be zoned and thus utilized by
the VM that a volume is attached to.
Where the lsnode command is documented to only report the
first 4 FC ports.
Patrick East [Wed, 2 Sep 2015 20:58:02 +0000 (13:58 -0700)]
Use consolidated update for failover_replication
Right now it’s using a special built volume_updates dictionary that
contains some fields *and* a model update. We then glom all of them
together into an update dictionary for the volume model.
This change simplifies them into a single model update more in-line with
how the other volume API’s work for drivers.
It also fixes a bug where the model_update property of the volume_update
was not being applied to the db model.
Currently some of the exceptions defined in the VMware driver
emit unformatted error messages. This is because they are
not overriding the correct message format variable. This
patch fixes the issue by overriding the correct variable
defined in the exception super class.
Edmund Rhudy [Thu, 27 Aug 2015 17:49:58 +0000 (13:49 -0400)]
Adds allow_availability_zone_fallback option to Cinder
This change adds allow_availability_zone_fallback, which allows Cinder
to fall back to an already configured availability zone if the
requested availability zone is unavailable.
Michael Price [Sun, 16 Aug 2015 14:58:41 +0000 (10:58 -0400)]
NetApp E-Series over-subscription support
Add support to the E-Series driver for
over-subscription/overprovisioning. This allows the Cinder scheduler
to more accurately represent the remaining space on a pool when
thin provisioned Cinder volumes are in use on an E-Series backend.
Co-Authored-By: Tom Barron <tpb@dyncloud.net>
DocImpact
Partially-implements: blueprint netapp-block--over-subscription-thin-provisioning
Change-Id: I5fb8f480168b9195221412d70f187c5ddfc4f527
Abhiram Moturi [Mon, 10 Aug 2015 14:23:09 +0000 (14:23 +0000)]
ZFSSA driver to return project 'available' space
This fix allows the iSCSI driver to return the 'available' space
property at project level instead of the pool level which is more
accurate in cases when storage is not thin provisioned.
Patrick East [Wed, 2 Sep 2015 20:51:42 +0000 (13:51 -0700)]
Get full volume model in Replication manager API’s
Currently we pass in a dictionary of the volume properties, but not the
actual volume model. This dictionary is missing some of the properties
like ‘name’ which backends may required for volume operations.
Fix problem of efficient volume copy for migration
After introducing commit f586043fa969b9d1dcf4933aacbf615f53691093,
new volume copy method _copy_volume_data() was added into
manager.py. Originally driver.py had this method and it was copied
into manager.py. However new _copy_volume_data() lost efficient
volume copy logic during the reimplementation.
This patch simply add efficient volume copy logic again into new
_copy_volume_data() to fix the problem.
Patrick East [Wed, 8 Jul 2015 00:23:58 +0000 (17:23 -0700)]
Generic image-volume cache
This introduces a new feature for backends to be able to use cached
glance images when creating volumes from images.
If enabled, and the cinder internal tenant is configured, it will
create a minimal sized clone of a volume the first time it is
downloaded from glance. A new db table ‘image_volume_cache’ tracks
these volumes, which are owned by the Cinder internal tenant. The
original will then be extended to full size. Any subsequent requests to
create a volume from an image will be able to do a volume clone from
the image-volume cache entry instead of downloading the image data from
glance again.
In the future we could create an entry upon creating an image from a
volume right before uploading the volume to glance. This version
however does not aim to do more than add the initial framework and
help with subsequent image downloads. There are certainly more
improvements that can be done over time on top of this.
These image-volumes are host specific, so each backend may end up with
its very own image-volume to do clones from.
The cache is limited in size by number of entries allowed and size in
gb. When creating a new entry if space is needed the last used entries
will be evicted to make room.
In the current implementation the image-volumes will be viewable by a
cloud admin in the volume list, and show up as owned by the Cinder
internal tenant. They can safely be deleted manually by an admin, this
will cause the entry to be removed from the cache. These volumes will
use quota for the internal tenant.
Cache actions will post notifications to Ceilometer. There are
notifications like ‘image_volume_cache.miss’, ‘image_volume_cache.hit’,
and ‘image_volume_cache.evict’. A change will be required to the
event_definitions.yaml to see them nicely. Until then you only need to
add a new event type 'image_volume_cache.*' and look for the ‘image_id’
and ‘host’ fields.
DocImpact: We probably want some instructions on restrictions of the
cache and how to use it. The three new config options should also be
documented somewhere: ‘image_volume_cache_enabled’,
’image_volume_cache_max_size_gb’, ‘image_volume_cache_max_size_count’
Michael Price [Wed, 5 Aug 2015 20:36:59 +0000 (15:36 -0500)]
Implement thin provisioning support for E-Series
Implement a new extra spec, 'netapp_thin_provisioned', that will allow
users to define thin provisioned Cinder volumes on E-Series storage,
alongside pre-existing extra specs such as
'netapp_eseries_data_assurance', 'netapp_eseries_flash_reach_cache',
'netapp_raid_type', etc.
We have a followup patch, https://review.openstack.org/#/c/215801/ ,
that reports 'thin_provisioning_support=True/False' and same for
'thick_provisioning_support', in accord with the scheduler-based
over-subscription support added in Kilo.
We are not yet attempting to implement the get_capabilities()
feature just merged into Liberty
(https://review.openstack.org/#/c/201243/)
but fully intend to do so in a way that conforms with that plan of
record.
John Griffith [Wed, 2 Sep 2015 15:56:15 +0000 (15:56 +0000)]
Remove useless response checks in SolidFire driver
The main issue_api_request method in the SolidFire driver
has had a mechanism to check validity of response data for
some time now. It checks the response and if there isn't
a valid result in the response it raises.
The problem is that we still have some cruft from the old
code that does local checking/verfication. These checks
won't ever actually be called because of the exception, and
they create some confusion.
This patch just removes those useless checks and cleans up
the api_request calls a bit.
Code Review / openstack-build / cinder-build.git / log