Victor Stinner [Mon, 15 Feb 2016 15:51:20 +0000 (16:51 +0100)]
Port netapp dataontap driver to Python 3
With this change, all unit tests on volume drivers pass on Python 3.
Changes:
* Replace a/b with a//b to get an integer on Python 3.
* Replace map(...) and filter(...) with a list-comprehension to get a
list on Python 3.
* Replace dict.keys() with list(dict.keys()) to get a list
on Python 3.
* Replace (str, int, float, long) with
six.integer_types + (str, float): long type was removed
from Python 3.
* decode_base32_to_hex(): on Python 3, decode encode_hex_to_base32()
to return a Unicode string.
* convert_es_fmt_to_uuid(): encode/decode on Python 3 to pass the
right types to base64.b32decode() (bytes) and uuid.UUID()
(Unicode).
* fakes.py: Change type of XML document from Unicode to bytes: add
b prefix to literal strings (b'...').
* Replace range(a, b) with list(range(a, b)) to get a list
on Python 3.
* tests-py3.txt: add cinder.tests.unit.volume.drivers, all volume
drivers tests now pass on Python 3.
LisaLi [Fri, 19 Feb 2016 08:28:36 +0000 (09:28 +0100)]
Scalable backup service - Liberty compatibility
To support rolling upgrades we need to make sure that Mitaka's services
are running fine with Liberty's. It gets complicated with backups as
we've strongly reworked them. Main difference is that Mitaka c-bak can
handle backup/restore of any volume and Liberty was restricted to
operate only on volumes placed on the same node.
Now when running in version heterogeneous environment we need to use old
way of backup jobs scheduling and switch to new one (round robin) only
when everything is running Mitaka.
This commit implements that by adding a dummy backup RPC API version
(1.3) that marks the beginning of scalable backups era. Jobs are
scheduled the new way only if every c-bak reports that (or higher)
version.
There are also small changes to volume.rpcapi - to fail fast if some
c-vol services aren't supporting new calls required by scalable backups
feature. This allows us to error out backups with proper message when
upgrade was done in an improper way (in Mitaka we require c-vols to be
upgraded before c-baks).
This commit also includes small changes to CinderObjectSerializer to
block tries to "forwardport" an object when sending it over RPC. If a
service receives an older object it should handle it explicitly.
Related-Blueprint: scalable-backup-service Co-Authored-By: Michal Dulko <michal.dulko@intel.com>
Change-Id: I45324336ba00726d53cfa012e8bd498868919a8c
Tom Barron [Tue, 23 Feb 2016 02:17:25 +0000 (21:17 -0500)]
Fix invalid uuid warnings in object unit tests
Currently 'tox -epy27 -- --regex cinder.tests.unit.objects' emits
74 FutureWarnings from oslo.versionedobjects about invalid uuids
as documented here[1].
This commit changes these tests to use valid uuids so that when
they are run these warnings are no longer emitted.
We had the call in the manager as part of a periodic,
it did nothing at all. Some drivers had comments "no idea
what to do with this", even better we had unit tests that
faked out a call that returned nothing and asserted it was
nothing. ?!?
Mitsuhiro Tanino [Fri, 22 Jan 2016 16:31:25 +0000 (11:31 -0500)]
[LVM] Restore target config during ensure_export
If server crash or reboot happened, LIO target configuration
will be initialized after boot up a server. Currently, Cinder
has a functionality to save LIO target configuration to save
file at several checkpoint.
If LIO target service is configured properly, LIO target
configuration is restored by this service during boot up
a server, but if not, existing in-use volumes would become
inconsistent status after c-vol service starts.
If there is no iSCSI target configuration during
ensure_export, LIO dirver should restore the saved
configuration file to avoid the problem.
Kurt Martin [Wed, 17 Feb 2016 21:58:41 +0000 (13:58 -0800)]
3PAR get host by WWN now handles mixed cases
The 3PAR driver comparison of WWNs will be able to handle
WWNs with upper/lower case differences. For example, the
driver code can now locate a 3PAR host with a WWN of 123abc6789012345 or 123ABD6789012345.
Michael Price [Fri, 23 Oct 2015 20:52:12 +0000 (13:52 -0700)]
NetApp: E-Series remove snapshot limitations
The E-Series backend does not allow snapshots defined under a snapshot
group to be deleted out of order. Only the oldest snapshot image may
be deleted. In order to work around this behavior, we would only define
a single snapshot on each snapshot group. Unfortunately, this would
limit the total number of snapshots to a total of 4 per volume.
In order to remove this limitation without causing deletion errors
when a newer snapshot is deleted, a soft-delete was implemented that
will remove the snapshot from Cinder management and purge the
snapshot from the backend once other dependent snapshots have been
deleted.
In order to ensure that our clone operations have a snapshot group
available, one of the 4 snapshot groups that can be defined on a
volume will be reserved for clone operations and will not be
allowed to be utilized for snapshot creation. This will limit the
total number of snapshots supported per volume to a maximum of 96.
The new 1.4 version of the WebServices proxy is required in order
to support the full snapshot behavior. Since snapshots are such
an integral part of the driver, 1.4 is now required for running
the driver with this release.
Co-Authored-By: Erik Johannes <erik.johannes@netapp.com>
Change-Id: Ie097743b167c1086ecf0536ad696786bdcd241b2
Michał Dulko [Fri, 5 Feb 2016 14:41:01 +0000 (15:41 +0100)]
Add SIGHUP handlers to reset RPC version pins
Adding SIGHUP handlers (by implementing reset from oslo.service) to
cinder-scheduler, cinder-backup and cinder-volume that reset cached RPC
version pins. This is to avoid the need to restart all the services when
upgrade of the deployment is completed.
Some changes go a little deep into the stack, because to reload all the
pins we need to recreate <service>.rpcapi.<service>API objects that are
stored in the memory.
Please note that SIGHUP signal is handled by oslo.service only when
service runs in daemon mode (without tty attached). To test this commit
in DevStack you need to add "&" to the end of the command that starts
the service.
Situation is more complicated with the API service, so we're leaving it
with restart required for now. In the deployments with HA cinder-api is
typically behind a load balancer, so restarting individual nodes
one-by-one should be easy.
DocImpact: Add information on rolling upgrades procedures to the docs.
Implements: blueprint rpc-object-compatibility
Gorka Eguileor [Thu, 18 Feb 2016 12:51:40 +0000 (13:51 +0100)]
Rtstool sets wrong exception message on save
Rtstool does not properly set the message on the RtstoolError exception
on the save_to_file method (save operation) because it's passing
formatting arguments as if it were a LOG call instead of formatting
those same arguments with
%.
There are 2 cases where this can happen, when saving to the default
location on a system that has no target.service installed and the
creation of the default directory fails, and when saving the file
actually fails.
This patch properly formats the message before passing it to the
instantiation of RtstoolError exception.
Victor Stinner [Tue, 24 Nov 2015 16:49:21 +0000 (17:49 +0100)]
Test middleware test_faults to Python 3
* XMLDictSerializer: sort dictionary items to get a reliable XML
output. On Python 3, dictionary items are iterated in a random
order, because the hash function is randomized by default.
* Use byte strings for HTTP body
* On Python 3, decode serialized data to get Unicode
* Add a base TestCase class to factorize the _prepare_xml() method
* tests-py3.txt: add cinder.tests.unit.api.middleware.test_faults
Ryan McNair [Sat, 30 Jan 2016 16:24:32 +0000 (16:24 +0000)]
Split out NestedQuotas into a separate driver
Fixes the following issues with NestedQuotas:
* Requires conf setting change to use nested quota driver
* Enforces default child quota value with volume creation
* Disables the use of -1 to be set for child quotas
* Adds an admin only API command which can be used to validate
the current setup for nested quotas, and can update existing
allocated quotas in the DB which have been incorrectly set
by previous use of child limits with -1
There will be follow-up patches with the following improvements:
* make -1 limits functional for child projects
* cache the Keystone project heirarchies to improve efficiency
Note: ideally validation of nested quotas would occur in the setup
of the nested quota driver, but doing the validation requires a
view of ALL projects present in Keystone, so unless we require Keystone
change to allow "cinder" service user to be able to list/get projects,
we need the admin-only API for validation that should be called by
cloud-admin.
Gorka Eguileor [Tue, 16 Feb 2016 18:03:49 +0000 (19:03 +0100)]
Wait for periodic tasks to stop on exit
When a Service in Cinder exits it stops all operations and waits for
them to complete, and it also stops all periodica tasks created with
FixedIntervalLoop, but it doesn't wait until these are completed.
This patch will wait for all periodic tasks that received the stop
request without raising an exception before exiting.
Nate Potter [Mon, 4 Jan 2016 23:10:19 +0000 (23:10 +0000)]
Roll back reservations quota in RPC if necessary
Currently the latest version of the volume rpcapi passes along
old_reservations in retype whereas older versions do not. In the
case where old_reservations aren't passed, they will be checked again
in c-vol. This patch rolls back the quotas that were checked for
old_reservations if the client can't send the latest version in
the RPCAPI so that they aren't reserved twice.
This is an amendment to this recently merged patch:
Iba24edd8ad824837028353b52c90742df55c9173
LisaLi [Wed, 30 Dec 2015 07:03:10 +0000 (15:03 +0800)]
Scaling backup service
Currently the cinder backup service is tightly coupled to the cinder
volume service in ways that prevent scaling out backup services
horizontally across multiple physical nodes.
This patch is to loosen this coupling to enable backup processes
to run on multiple nodes without having to be colocated with volume
services.
The following works are not included in this patch:
1. Remote attach snapshot.
2. Vendor specific work.
3. Remove current backup_volume in driver.
4. Rolling upgrades.
DocImpact
Change-Id: I743e676372703e74178c79683dd622d530981e04
Partial-Implements: bp scalable-backup-service Co-Authored-By: Tom Barron <tpb@dyncloud.net>
* InfortrendCommon._create_partition_with_pool(): cast gi_to_mi()
result to int. On Python 3, gi_to_mi() returns a float (a/b always
return a float, even for int/int).
* test_retype_with_migrate(): build create_params using a dictionary
to create parameter in the same order than the driver.
* tests-py3.txt: cinder.tests.unit.test_infortrend_common
Victor Stinner [Mon, 15 Feb 2016 17:27:17 +0000 (18:27 +0100)]
Fix last Python 3 issues in zonemanager
* Replace auth_string.encode("base64", strict") with
oslo_serialization.base64.encode_as_text(auth_string).
* tests-py3.txt: add cinder.tests.unit.zonemanager, add zonemanager
unit tests now pass on Python 3
Victor Stinner [Tue, 9 Feb 2016 13:36:45 +0000 (14:36 +0100)]
Port API v1 and v2 to Python 3
* Replace jsonutils.dumps() with jsonutils.dump_as_bytes() since
output is used for the HTTP body, and HTTP body type is bytes, not
Unicode. On Python 3, jsonutils.dumps() returns Unicode.
* FakeHttplibSocket: replace io.StringIO with io.BytesIO, HTTP body
type is bytes. Encode Unicode to UTF-8.
* Fix FakeHttplibSocket.makefile() API: only the first parameter is
mandatory. On Python 3, it's only called with the first 'mode'
parameter.
* Use bytes strings to test HTTP bodies (JSON or XML).
* Replace dict.values() with list(dict.values()) to get a list on
Python 3.
* Replace error.message with encodeutils.exception_to_unicode(error),
exceptions loose their message attribute in Python 3.
* Update JSON serializer test in test_wsgi.py to use bytes.
* tests-py3.txt: add cinder.tests.unit.api.v1 and
cinder.tests.unit.api.v2
Victor Stinner [Tue, 9 Feb 2016 17:40:21 +0000 (18:40 +0100)]
Port API contribs to Python 3
* Replace dict.iteritems() with dict.items(), dict.iteritems() was
removed in Python 3.
* Replace dict.values() with list(dict.values()) to get a list on
Python 3.
* HTTP body type must be bytes:
- Fix unit tests to use bytes strings for HTTP body
- Encode XML to UTF-8 on Python 3.
* Use "%r" instead of "%s" to format request and response in test to
avoid BytesWarning on Python 3.
* tests-py3.txt: add cinder.tests.unit.api.contrib
Victor Stinner [Mon, 15 Feb 2016 13:43:32 +0000 (14:43 +0100)]
Port test_emc_vnx to Python 3
* Configuration.__getattr__(): replace self.local_conf with
object.__getattribute__(self, 'local_conf') to avoid reentrant call
to __getattr__() in copy.copy(conf) on Python 3.
* test_emc_vnx: Don't compare directly provider_location strings but
set of provider_location items to support hash randomization. On
Python 3, hash randomization is enabled by default and so
dictionary items are rendered in an random order.
* test_emc_vnx: use assertIn() to check if the provider_location
contains 'type^smp' substring. Before, the test failed if the
substring was a the beginning of the string (position 0).
* tests-py3.txt: add cinder.tests.unit.test_emc_vnx
Yuriy Nesenenko [Mon, 4 Jan 2016 13:14:15 +0000 (15:14 +0200)]
Fix service-list filter
Currently cinder service list --host=<hostname> will not show any
results for the volume services. This is because the volume-services
append an internal pool name to the hostname and as a result there
won't be a match for the specific filter. This patch fixes the
incorrect work of service-list with '--host' argument. Also it filters
data on the DB side to improve performance.
LisaLi [Fri, 15 Jan 2016 08:12:45 +0000 (16:12 +0800)]
Update quota when volume type renames
When customers rename volume type, the corresponding quota_usages
should be changed. Or else quota_usage shows incorrect data.
Meanwhile, it deletes quota_usages whose corresponding volume
types don't exist. These invalid data are left when renaming
volume types in old versions.
LisaLi [Thu, 24 Dec 2015 03:16:26 +0000 (11:16 +0800)]
Add restore_volume_id in backup
This patch is to add restore_volume_id in backup object.
When restoring a volume from a backup, it saves the
volume in backup object.
Currently volume service and backup service are in same host.
When backup service starts, it does cleanup tasks on both
backups and volumes on current host.
But with bp scalable-backup-service, backup service and
volume services can run on different hosts. When doing cleanup
tasks, we need to find out backing-up and restoring volumes
related to the backups on current host. Backing-up volumes can
be found with field backup.volume_id. Restoring volumes are found
by new field backup.restore_volume_id.
Change-Id: I757be7a5e47fc366c181400587b5a61fe3709a0b
Partial-Implements: bp scalable-backup-service Co-Authored-By: Tom Barron <tpb@dyncloud.net>
Tom Barron [Fri, 12 Feb 2016 16:48:16 +0000 (11:48 -0500)]
Fix 'asert' typo in unit test
The dellscapi unit tests contain two occurences of an
apparent typo, 'mock_get_volume_list.asert_called_once_with'.
Because mock will magically create an 'asert_called_once_with'
method, this typo masks a problem with the test_manage_unmanage
test where the wrong set of arguments is expected.
This commit substitutes 'assert' for 'asert' in these occurences
and fixes up the test_manage_unmanage test.
Cory Stone [Mon, 8 Feb 2016 17:22:59 +0000 (11:22 -0600)]
Fix dynamic import of CONF.volume_api_class
Modifies the import to actually happen when API() is called instead of
when the module is loaded.
This is necessary because CONF.volume_api_class is always evaluated as
the default value since the config file hadn't been parsed yet when the
module is loaded.
Nova fixed this problem a while ago for the compute driver in
https://review.openstack.org/#/c/14353/.
Clinton Knight [Sat, 23 Jan 2016 00:02:00 +0000 (19:02 -0500)]
Filter & goodness functions in NetApp drivers
This commit adds a performance module that utilizes various
performance APIs to measure storage controller node utilization
for each pool, and to report those metrics to the Cinder scheduler
via the standard filter & goodness functions. Both Clustered and
7-mode Data ONTAP driver flavors are supported across NFS and
iSCSI/FC protocols.
Adriano Rosso [Fri, 5 Feb 2016 17:41:01 +0000 (15:41 -0200)]
HNAS driver: Fix SSH and cluster_admin_ip0 bug
HNAS driver has 2 configuration options:
- SSC (ssh_enabled: False ): requires a local utility package
installed to run the commands
- SSH (ssh_enabled: True): Runs the commands using the utility
package installed on HNAS
Currently, the HNAS driver is considering the "ssh_enabled" and the
"cluster_admin_ip0" tags to decide if the command to get the utility
package version should be ran locally or not. This is not correct
because the "cluster_admin_ip0" tag is not mandatory even if the SSH
mode is enabled.
So, if we have a situation that SSH is enabled and the
"cluster_admin_ip0" does not exist in the configuration, the driver
tries to run the command to get the utility package version locally
and its initialization breaks.
This patch makes the driver consider only the "ssh_enabled" tag
value to decide if the command to get the utility package version
is necessary.
Javeme [Thu, 11 Feb 2016 12:09:41 +0000 (20:09 +0800)]
Don't use Mock.called_once_with that does not exist
class mock.Mock does not exist method called_once_with, it just exists
method assert_called_once_with. Currently there are still ome places
where we use called_once_with method, this patch let's correct it.
NOTE: called_once_with() does nothing because it's a mock object.
Code Review / openstack-build / cinder-build.git / log