Ollie Leahy [Thu, 15 Oct 2015 15:20:28 +0000 (15:20 +0000)]
Use project id from volume when retyping volumes
Use the project_id from the volume being retyped to reserve quota in
get_volume_type_reservation(). Previously the project_id from the
context was used, which could cause reservations to be made and never
cleared.
This is a null-merge of the 7.0.0 release tag back into the master
branch so that the 7.0.0 tag will appear in the git commit history of
the master branch. It contains no actual changes to the master branch,
regardless of how our code review system's UI represents it. Please
ask in #openstack-infra if you have any questions, and otherwise try
to merge this as quickly as possible to avoid later conflicts on the
master branch.
Sean McGinnis [Mon, 12 Oct 2015 21:07:55 +0000 (16:07 -0500)]
Only use LOG.exception in exception handler
There were a few instances of the code calling LOG.exception
in places where there was not an exception in context. This
can cause errors and should just be LOG.error as it effectively
ends up being.
Some instances in vhdutils.py are being left as they are being
addressed in Ib3113f2c4752d37e890f97d259da5d51cbfcfb96 under a
different bug report.
Victor Stinner [Wed, 7 Oct 2015 21:44:46 +0000 (23:44 +0200)]
Port API admin action tests to Python 3
* Replace ast.literal_eval(response.body) with response.json
* Use oslo_serialization.jsonutils.dump_as_bytes() to encode the
response to JSON as bytes for the HTTP body.
* tests-py3.txt: add cinder.tests.unit.api.contrib.test_admin_actions
Victor Stinner [Wed, 7 Oct 2015 16:41:48 +0000 (18:41 +0200)]
Port API types extra specs to Python 3
* Replace dict.iteritems() with dict.items(). The iteritems() method
of dictionaries was removed in Python 3.
* Add cinder.tests.unit.api.contrib tests to tests-py3.txt:
Victor Stinner [Wed, 7 Oct 2015 16:12:48 +0000 (18:12 +0200)]
Port API to Python 3
* test_make_flat_dict(): only use one dictionary key. On Python 3,
the hash function is now randomized by default and so dictionary
keys are returned in a random order.
* Replace "not cmp(a, b) == 0" with "a != b"
* test_common: replace range(n) with list(range(n)) to get a list on
Python 3. On Python 3, range() now returns an iterator.
* Replace dict.items()[0] with list(dict.items()[0]). On Python 3,
dict.items() now returns a view which is not indexable.
* VolumeTypesController.index(): replace dict.values() with
list(dict.values()) to get a list on Python 3.
* test_xmlutil: use byte strings for XML
* test_extensions: use byte strings for HTTP body
* Add the following tests to tests-py3.txt:
Matt Riedemann [Mon, 12 Oct 2015 13:44:14 +0000 (06:44 -0700)]
windows: don't use LOG.exception if not logging an exception
There is a bug in python 3.4 such that if you call LOG.exception and
you're not in a logging handler context it fails. That bug is fixed
inadvertantly in python 3.5 under:
http://bugs.python.org/issue17911
This fixes the instance in the windows vhdutils module to use LOG.error
rather than LOG.exception to avoid that logging error.
Since LOG.exception was being used I'm assuming the original author wanted
traceback given this is a utility method, so we use the exc_info kwarg which
will log a traceback if there is one.
When updating only volume type is_public, error will be raised: `ERROR:
Specify either volume type name and/or description`. Fixed this by
adding a check that is_public is None, the same way name and description
do.
Ollie Leahy [Thu, 1 Oct 2015 17:21:52 +0000 (18:21 +0100)]
encryption_api_url requires a version
If the value configured for encryption_api_url does not include the
barbican API version, then some calls from cinder will fail. This can
mean that encrypted volumes cannot be deleted. To prevent this happening
raise an exception if the configured value for encryption_api_url does not
include the barbican version.
Jacob Gregor [Thu, 1 Oct 2015 18:52:34 +0000 (13:52 -0500)]
Cleanup/move code in Storwize Driver
This patch updates the name of helper.py to storwize_svc_common.py.
In addition, ssh.py has been collapsed into storwize_svc_common.py
to reduce redundant code. Unit tests were also updated to reflect
these updates.
This patch is the first of a series of patches with the aim of
bringing the storwize_svc driver in line with other san based
Cinder drivers.
Co-Authored By: Kendall Nelson <kjnelson@us.ibm.com>
Co-Authored By: Slade Baumann <baumann@us.ibm.com>
Victor Stinner [Wed, 7 Oct 2015 15:38:12 +0000 (17:38 +0200)]
Port WSGI tests to Python 3
* Replace dict.keys()[0] with list(data.keys())[0]. On Python 3,
dict.keys() now returns a view which is not indexable.
* Skip SSL tests on Python 3. Tests hang for an unknown reason, they
must be fixed later.
* Fix unit tests: HTTP body type is bytes, not Unicode.
* Debug.print_generator(): on Python 3, write into sys.stdout.buffer
instead of sys.stdout, because HTTP body type is bytes not Unicode.
* ResponseObject: encode serializer output to UTF-8 if it's Unicode.
* tox.ini: add the following tests to Python 3.4
Note: Ignore pylint error E1101 on sys.stdout.buffer. pylint on
Python 2 complains that the buffer attribute doesn't exist, whereas
the code is only executed on Python 3 and the attribute exists on
Python 3.
Dmitry Guryanov [Tue, 13 Oct 2015 11:53:13 +0000 (14:53 +0300)]
Fix method VolumeTypeList.get_all
Function volume_types.get_all_types returns dict, then
VolumeTypeList.get_all passes it to base.obj_make_list, which, in turn,
passes it to oslo_versionedobjects.base.obj_make_list.
And this function iterates over 4th argument and calls
item_cls._from_db_object for each item. So if it will iterate over dict
from volume_types.get_all_types, it'll get strings instead of orm
objects.
By the way, VolumeTypeList.get_all is not used anywhere in cinder code.
I tried to use it in unit test for my new patch and found this problem.
Use lvm_conf_file directory for LVM_SYSTEM_DIR value.
Currently when LVM_SYSTEM_DIR is added to LVM.LVM_CMD_PREFIX it is
hardcoded as /etc/cinder.
However the configuration option lvm_conf_file could specify a file
located in an arbitrary directory, not just /etc/cinder, so we should
be using the path to the directory containing the specified lvm.conf
file as the value for LVM_SYSTEM_DIR.
WebOb 1.5 was released at 2015-10-11. With this new version,
webob.exc.WSGIHTTPException() constructor now fails with a KeyError
when the HTTP status code is 0.
test_exceptions_raise() of test_misc tries to instanciate all
exceptions of cinder.exception. The problem is that
ConvertedException uses a default HTTP status code of 0.
Modify the default HTTP status code of ConvertedException to 400 to
fix the unit test. The bug is only in the test,
cinder/api/openstack/wsgi.py copies an existing HTTP code:
John Griffith [Fri, 9 Oct 2015 23:31:15 +0000 (17:31 -0600)]
Update config format for replication_devices
The first pass at setting up config entries for
replication targets was a hacky custom string parser
with its own unique syntax.
A better option is to use oslo.cfg's MultiOpt in conjunction
with Dict types so we can eliminate quite a bit of custom
parsing and most importantly use a standard syntax for conf
entries to make things easy for admins and reduce the
probability of syntax errors and mistakes in config setttings.
There is a strict rule about constructing status line for HTTP:
'...Status-Line, consisting of the protocol version followed by a
numeric status code and its associated textual phrase, with each
element separated by SP characters'
(http://www.faqs.org/rfcs/rfc2616.html)
This patch coerces filling associated textual phrase.
Also removed unused code from cinder/tests/unit/test_exception.py
Change-Id: Ia9099fb5020cee02bfee2cd0e8e111845918c36c
Closes-Bug: #1496055 Co-Authored-By: Marian Horban <mhorban@mirantis.com>
Volume extension API does not throw HTTPBadRequest response,
when input to the API is invalid.
This patch handles invalid input to the volume extension API
and return HTTPBadRequest.
Victor Stinner [Mon, 12 Oct 2015 08:35:25 +0000 (10:35 +0200)]
Fix test_misc for WebOb 1.5
WebOb 1.5 was released at 2015-10-11. With this new version,
webob.exc.WSGIHTTPException() constructor now fails with a KeyError
when the HTTP status code is 0.
test_exceptions_raise() of test_misc tries to instanciate all
exceptions of cinder.exception. The problem is that
ConvertedException uses a default HTTP status code of 0.
Modify the default HTTP status code of ConvertedException to 400 to
fix the unit test. The bug is only in the test,
cinder/api/openstack/wsgi.py copies an existing HTTP code:
Alex O'Rourke [Fri, 9 Oct 2015 18:10:38 +0000 (11:10 -0700)]
Fix error string format for replication API calls
enable_replication, disable_replication, and failover_replication all
raise exceptions if the volume they are being called on are not in
valid states. When that is done, the error message is not properly
formatted and looks as such:
"ERROR: Invalid volume: (u'Invalid status to enable replication. valid
states are: %(valid_states)s, current replication-state is:
%(curr_state)s.', {'curr_state': u'enabled', 'valid_states':
['disabled']}) (HTTP 400)"
The reason for this is the msg string is using a ',' instead of a '%'
to evaluate the string. When fixed, the error is displayed correctly:
"ERROR: Invalid volume: Invalid status to enable replication. valid
states are: ['disabled'], current replication-state is: enabled.
(HTTP 400)"
Victor Stinner [Wed, 7 Oct 2015 16:02:43 +0000 (18:02 +0200)]
Port IBM flashsystem to Python 3
* Replace dict.keys()[0] with list(dict.keys())[0]. On Python 3,
dict.keys() now returns a view which is not indexable.
* tests-py3.txt: add cinder.tests.unit.test_ibm_flashsystem
Victor Stinner [Wed, 7 Oct 2015 16:07:38 +0000 (18:07 +0200)]
Port ceph driver to Python 3
* Replace xrange() with range(). cinder/volume/utils.py uses
"from six.moves import range", so "range" is xrange on Python 2.
* Replace '' with b'' for image content to get a bytes string on
Python 3.
Note: _transfer_data() of cinder.volume.utils is tested by
test_backup_ceph which is already run on Python 3, but the test only
failed with python3 run with -bb. This bug is now fixed.
Tom Barron [Thu, 8 Oct 2015 01:10:48 +0000 (21:10 -0400)]
Remove references to Swift in chunked driver
After the chunked/Swift backup driver refactor in commit 2f701a39208f9396893cd8a57165e37cfeb6fc2f
some references to 'Swift' remained in code comments and
log messages in code that no longer lives in the Swift driver
proper.
Tom Barron [Fri, 2 Oct 2015 20:49:46 +0000 (16:49 -0400)]
Add insecure option for swift backup
When running backups with the swift backup driver and an
https url for the swift endpoint, the config option
backup_swift_ca_cert_file must point to a valid server side
cert.
For test purposes, it is useful to be able to ignore
cert errors while still making a SSL/TLS connection, the way
one can with https:// urls and the curl command and the
swift client itself.
This commit adds a boolean CONFIG option, 'backup_swift_auth_insecure',
whose default value is False, in which case there is no change from
current behavior. If this option is set to True, however, cert validation
is skipped for SSL/TLS.
Xiaoqin Li [Sun, 4 Oct 2015 17:59:37 +0000 (10:59 -0700)]
Add fast format option for thick volume creation
Add option for user to disable fast format at thick
provisioned volume creation in IBM Storwize cinder.
The default value is False and clients who do not want
to do Fast Format for thick volume, they can set this
parameter to True.
DocImpact: Add fast format option
Change-Id: Ib627696b91c336d09de6066ee70ee518fca5a7a9
Closes-Bug: 1502404
Xi Yang [Tue, 1 Sep 2015 10:05:15 +0000 (06:05 -0400)]
Retype enhancement for EMC VNX cinder driver
VNX driver will trigger lun migration when retyping a thin/thick
volume to a compressed volume and it is time-consuming. To enhance
the performance, this patch will directly turn on the compression
on a volume instead of migrating it.
John Griffith [Tue, 6 Oct 2015 23:15:58 +0000 (17:15 -0600)]
Verify volume is replication capable
The V1 implementation of replication added a replication_status
field to the Volume object and set it to disabled by default on
volume creation.
This is problematic, because there's no way for the API or any
caller to know if the status is disabled because it was
explicitly set that way, or if it is because the back end or
volume-type do not support replication.
This results in cases like enable replication (which does a status
check on disabled) to be called inappropriately on devices that
don't support replication.
This patch adds a decorator to the new replication methods which will
check that the volume is of type with replication_enabled=True before
attempting any replication related operations.
Victor Stinner [Wed, 7 Oct 2015 17:01:57 +0000 (19:01 +0200)]
Port violin driver to Python 3
* Replace dict.items()[0] with list(dict.items())[0], same for
dict.values()
* Replace dict.iteritems() with dict.items()
* Replace dict.itervalues() with dict.values()
* Replace a/b with a//b to get integer on Python 3.
* test_get_active_fc_targets(): ignore order when comparing active
FC targets. On Python 3, the hash function is randomized and so
dictionaries values are returned in a random order.
Victor Stinner [Wed, 7 Oct 2015 15:57:00 +0000 (17:57 +0200)]
Port EMC scaleio to Python 3
* Import urllib modules using six.moves.urllib
* MockHTTPSResponse: encode HTTP body to UTF-8 if it's Unicode,
replace six.string_types with (bytes, six.text_type).
* _id_to_base64(): catch also binascii.Error when calling
b16decode(); encode name to UTF-8 before calling b64encode();
decode b64encode() from ASCII on Python 3 to get Unicode.
* tox.ini: add cinder.tests.unit.volume.drivers.emc.scaleio
to Python 3.4
Jay S Bryant [Tue, 6 Oct 2015 09:03:53 +0000 (04:03 -0500)]
Remove extra register_opts() calls in netapp eseries
The eseries/library.py file is importing netapp's
options file. The options file already registers
all of the configuration options. For some reason
the library.py file was re-registering the options
and also properly appending them to the configuration list.
This patch removes the extra register_opts() calls.
John Griffith [Tue, 6 Oct 2015 23:39:24 +0000 (17:39 -0600)]
Fix SolidFire target composition
We added the target composition a while back but never
really finished it. We created a set of interface methods
and had them call the target class mostly thinking we'd
want this when adding FC.
It turns out that doing a regular composition and using
get_attr will actually work fine when the FC target is
added, so re factor this and make it a true composition
where both objects inherit attributes from each other.
Code Review / openstack-build / cinder-build.git / log