Victor Stinner [Mon, 29 Jun 2015 20:20:16 +0000 (22:20 +0200)]
Port hitachi driver to Python 3
* On Python 3, shlex expects Unicode, not bytes: don't encode
* Replace map() with list-comprehension where a list is expected.
* tox.ini: add cinder.tests.unit.test_hitachi_hbsd_horcm_fc to Python 3.4
Tom Barron [Thu, 11 Jun 2015 17:07:07 +0000 (13:07 -0400)]
Remove oslo logging from backup unit tests
Many of the unit tests for backup services import oslo_log and
invoke LOG.debug, LOG.info, etc. There appears to be no current
reason to do this and, more generally, unit tests for everything
but oslo_log itself should not be doing logging themselves.
This commit removes oslo logging from the backup unit tests.
Closes-bug: 1464772
Tom Barron [Wed, 17 Jun 2015 18:54:01 +0000 (14:54 -0400)]
Fix 'no actual-pathname' NetApp API error
Cinder volume logs sometimes show this error
NetApp API failed. Reason - 13114:
No actual-pathname for 10.1.0.9:/vol/whatever
with Kilo code and 7-mode DOT system.
The issue is due to our Cinder driver passing the
entire share to the relevant API instead of just
the export path portion of the share. This only
happens when the NFS image cache is full, and cached
files need to be removed.
Cedric Zhuang [Thu, 21 May 2015 11:26:23 +0000 (04:26 -0700)]
Use symbol for error code in VNX cinder driver
Some of the error code of VNX CLI is scattered in the code and is
not coded as a constant.
Gather all VNX error codes in the VNXError class and compose a new
method to check the error in the CLI output. If no specific error
is supplied, check all available errors in the output.
Yuriy Nesenenko [Tue, 19 May 2015 15:17:01 +0000 (18:17 +0300)]
Filter snapshots data on the DB side
It filters snapshots data on the DB side to improve performance.
Some tests are removed from test_snapshots.py because their function
implemented on the DB side.
Jay S. Bryant [Wed, 24 Jun 2015 19:35:38 +0000 (14:35 -0500)]
Fix library includes for config generator
Noticed today while working on fixing issues
with the config generator that oslo_policy
had been added in the wrong location for
pulling the config options in to the config
generator.
This fix moves the items that had been put in
oslo.config.generator.rc but were missing in generate_sample.sh
over to the appropriate location. I also removed the option for
importing additional libraries in oslo.config.generator.rc
given that it will not work.
I am still working on a longer term solution, but
we really should get this fixed so we can generate sample
config files again.
Anton Arefiev [Wed, 24 Jun 2015 15:45:35 +0000 (18:45 +0300)]
Fix cinder-manage volume delete cmd
Since pools was introduced in cinder, field 'host' in resources
contains pool part. So when we are trying to delete volume with
cinder-manage, 'host' with pool part passed to rpc target. As
result message isn't delivered to host, and we can't delete volume.
Victor Stinner [Tue, 23 Jun 2015 12:08:25 +0000 (14:08 +0200)]
Fix Python 3 issues in the blockbridge driver
* Replace httplib import with six.moves.http_client
* Replace urllib imports with six.moves.urllib, update urllib.
For example, replace urllib.quote() with urllib.parse.quote().
* test_blockbridge: try get the mock module from unittest.mock, part of
the Python stdlib since Python 3.3. The third-party mock module has a
bug on Python 3.4.
* On Python 3, base64.encodestring() expects bytes and returns bytes.
Encode credentials to UTF-8 and decode base64 from ASCII to get the
final string as Unicode on Python 3.
* test_cfg_api_auth_scheme_password: parse the connection URL to compare
it. Comparing directly the URL fails on Python 3 because URL
parameters are rendered in a random order because of the hash
randomization.
* Replace pools.values()[0] with list(pools.values())[0]. On Python 3,
dict.values() returns an iterator.
* tox.ini: add cinder.tests.unit.test_blockbridge to Python 3.4.
Victor Stinner [Tue, 23 Jun 2015 11:37:42 +0000 (13:37 +0200)]
Fix Python 3 issues in the swift backup driver
* On Python 3, encode/decode JSON to/from UTF-8
* Use byte strings for volume content
* Replace dict.keys()[0] and dict.value()[0] with list(dict.items())[0].
Get the key and the value at once to ensure that they are consistent.
On Python 3, items() returns an iterator: create a list to use the [0]
operator.
* SwiftBackupDriver: use a bytearray instead of a string because
bytes += bytes is inefficient on Python 3, whereas str += str is
optimized on Python 2. Replace also StringIO with BytesIO.
* BytesIO has no more len attribute on Python 3: get the length of the
content instead.
* Replace buffer(bytearray(128)) with b'\0' * 128 to create a string of
128 zeroed bytes
* Replace "rw" file mode with "w+" ("rw" raises an exception
on Python 3).
* tox.ini: add the following tests to Python 3.4
Victor Stinner [Tue, 23 Jun 2015 09:55:40 +0000 (11:55 +0200)]
Fix Python 3 issues in ceph and rbd drivers
* Add cinder.utils.convert_str() function: helper to convert a string to
a native string: convert to bytes on Python 2, convert to Unicode on
Python 3. Function needed by ceph and rbd drivers.
* Replace encodeutils.safe_encode() with utils.convert_str()
* test_backup_ceph: get the size of the volume file using os.fstat() to
fix a test. Before the volume size was a mock, but comparison between
integers and mocks raise a TypeError on Python 3.
* Fix RBDDriver._update_volume_stats(): replace a/b with a//b to get an integer
on Python 2 and Python 3 (a/b always returns a float on Python 3)
* Replace map() with a list-comprehension in RBDDriver because a list is
expected. On Python 3, map() returns an iterator.
* tox.ini: add to following tests to Python 3.4
Victor Stinner [Tue, 23 Jun 2015 09:20:47 +0000 (11:20 +0200)]
Fix Python 3 issues in backup
* Replace (int, long) with six.integer_types: the "long" type has been
removed in Python 3
* Replace str.encode("bas64") with base64.encodestring(str), base64
text codec has been removed in Python 3. Same change for decode
(base64.decodestring)
* On Python 3, encode JSON to UTF-8
* tox.ini: add the following tests to Python 3
git-harry [Wed, 24 Jun 2015 08:46:45 +0000 (09:46 +0100)]
Remove generate_glance_url
The function generate_glance_url is part of the inital fork from nova.
It is only called once and that is by a unit test designed to test the
function.
This commit removes generate_glance_url and its associated test because
they are not used by cinder.
Michal Dulko [Tue, 23 Jun 2015 13:47:53 +0000 (15:47 +0200)]
Harden scheduler.rpcapi unit tests
Tests for scheduler.rpcapi are very weak. Even changing keyword
arguments names passed to call and cast methods in this module
doesn't trigger unit tests failures. This commit adds keyword
arguments validation to prevent problems similar to ones caused
by bug 1467841 in backup.rpcapi. I've also fixed argument order
in assertEqual usages there.
Michal Dulko [Tue, 23 Jun 2015 13:42:00 +0000 (15:42 +0200)]
Fix backups.rpcapi to pass objects over RPC
Commit Icff37261b367463b71a1268be16f9c97f595bf0c brought bugs in
backup.rpcapi. For export_record, import_record and reset_status
backup.id is passed instead of whole backup object - which
backup.manager expects. This causes severe failures. This commit
changes the rpcapi to pass objects and adds missing
backup.test_rpcapi unit tests that will prevent such failures in
the future.
wanghao [Tue, 26 May 2015 09:25:26 +0000 (17:25 +0800)]
Fix weird change of volume status in re-scheduling
When create a volume failed as some exception, cinder will re-schedule the
creation request, the retry number is 3 in default.
But when re-scheduling, this volume's status was changed like this:
This is weird to user, since if this volume is in re-scheduling process,
it's status should always be creating, and then become error or available
at last.
Victor Stinner [Tue, 23 Jun 2015 09:11:04 +0000 (11:11 +0200)]
Fix tox -e py34
Add a py34 test environment to tox.ini running a subset of tests which
pass on Python 3.4. With this change, the py34 check job should pass and
so it will become possible to make the job voting to avoid further
Python 3 regressions.
Jay S. Bryant [Mon, 22 Jun 2015 16:43:37 +0000 (11:43 -0500)]
Add exception catch in report_state for DBError
We discovered while testing Cinder in an HA
environment that transient DB errors can be
encountered that are not currently covered by
the exception catch in service.py report_state().
The uncaught exceptions were causing the thread
for report_state to prematurely exit and causing
services to no longer update the DB when, in fact,
they were still usable.
This change adds an exception catch for DBError
so that the thread can continue to function.
Sean McGinnis [Thu, 18 Jun 2015 21:56:33 +0000 (16:56 -0500)]
Dell SC: Enable use of Storage Profiles
The Storage Center array allows volumes to be associated with
different Storage Profiles. These profiles allow setting tiering
and RAID types per volume. The driver currently only uses the
default Storage Profile configured for the Cinder user account.
This patch adds the capability to use different Storage Profiles
by using extra specs to allow the creation of different volume
types for the desired profiles.
Ivan Kolodyazhny [Sat, 20 Jun 2015 20:02:41 +0000 (23:02 +0300)]
Use elevated context for backup destroy
db.backup_destroy requires admin context to mark backup record as
deleted. Commit Icff37261b367463b71a1268be16f9c97f595bf0c removed admin
context passed to backup_destroy method.
This patch uses elevated context to destroy backup.
Change-Id: I75b9e1fff48569a8aa320f2f02914fc7b6665d79
Closes-Bug: #1467167
Ivan Kolodyazhny [Sun, 21 Jun 2015 08:33:05 +0000 (11:33 +0300)]
Fix Cinder Objects unit tests
Due to the import order all tests passed on CI. But you can not
run cinder.unit.objects.test_backup only. This patch fixes it
adding base class and invokes objects.register_all().
Matt Riedemann [Sat, 20 Jun 2015 19:43:34 +0000 (12:43 -0700)]
rbd: add volume_id to connection_info in initialize_connection
Most other volume drivers have a 'volume_id' key in the
connection_info['data'] dict returned from initialize_connection, this
adds it to the rbd volume driver.
This is part of a bigger effort to standardize the connection_info dict
so that consumers like Nova can rely on a common set of keys when
processing connection_info.
Mike Perez [Sat, 20 Jun 2015 02:12:15 +0000 (19:12 -0700)]
Fix Datera driver export call
A previous commit 3fabed9d64696a204263fd7e00b65115a8ecce87 broke the
export call for the Datera driver. This allows the information to
correctly be accessed again.
Pavel Boldin [Wed, 13 May 2015 22:01:36 +0000 (01:01 +0300)]
Add iscsi_target_flags configuration option
Newly introduced `iscsi_target_flags' allows to specify `tgtd'
backing storage flags. This can be used to instruct `tgtd' to
open backing device file using `O_DIRECT' flag ensuring direct
access without cache.
John Griffith [Fri, 19 Jun 2015 18:32:01 +0000 (12:32 -0600)]
Remove the hardcoded concurrency limit for ostestr
ostestr by default uses concurrency = ncpu.
This caused all sorts of things to blow up in Cinder
when one got over 6 cpus. Since the initial merge a
patch has landed that appears to fix the issues, and
we now run reliably at 8 and even 12 cpus.
This patch removes the hard coded concurrency=6 setting
from tox.ini
Vilobh Meshram [Fri, 19 Jun 2015 17:17:07 +0000 (10:17 -0700)]
Nested Quota : Create allocated column in cinder.quotas
Create allocated column in cinder.quotas. This allocated
column will be the sum of "hard_limits" values for the
immediate child projects. This will be needed to track
allocated quota to child projects.
Change-Id: Ia80d9a6cdbbc7e86cf9f11979f5e80c53f0fac1f
Implements: bp cinder-nested-quota-driver
Yuriy Nesenenko [Thu, 4 Jun 2015 15:50:14 +0000 (18:50 +0300)]
Handle incorrect '--config-dir' param
This patch fixes the error that occurs if the --config-dir directory
doesn't exist. The exception ConfigDirNotFoundError should be caught
in this case.
Heat stack deletion followed by an unsuccessful heat stack delete
attempt resulted in calling begin_detaching on a volume which was
already in detaching state. It was observed that the API returned
success instead of failing the operation. There is a check which
verifies the volume status, but it is not using the updated volume
status. This patch fixes the problem by getting updated volume
status from DB. But this fix won't eliminate the possibility of
above mentioned inconsistent behavior if multiple begin_detaching
happens concurrently.
Eric Harney [Thu, 18 Jun 2015 22:06:54 +0000 (18:06 -0400)]
Tests: Make fake_notifier per-instance
Tests have been failing because multiple threads modify the
same global NOTIFICATIONS list. Move the notification storage
to be per-test instance, which is what the tests assume.
This has been broken for a while, but made more obvious by
recent changes to parallelize tests.
Also remove all the extra reset calls we added to work around
this problem over time.
Dmitry Guryanov [Thu, 18 Jun 2015 17:14:34 +0000 (20:14 +0300)]
Add Virtuozzo Storage Volume Driver
Add a volume driver which can use Virtuozzo Storage, which
has filesystem interface and so volume driver has a similar
workflow to NFS and SMBFS drivers.
At this point the driver contain minimal set of features. Because
I think some refactoring should be done in RemoteFS drivers before
further development. For example code, which deals with image formats
should go to the RemoteFS class. So I don't add qcow2 images support
in this patch.
Matt Riedemann [Thu, 18 Jun 2015 15:16:52 +0000 (08:16 -0700)]
Use a hard-coded project_id in racy cinder.tests.unit.test_volume tests
Since concurrent unit test runs have been turned on with ostestr there
are races with the notifications causing the assertions in
test_volume.py tests to fail.
A few attempts have been made at writing an elastic-recheck query to
identify the failures but they are too generic, e.g. querying for
"fake_notifier.NOTIFICATIONS" or "MismatchError 0 != ", which could hit
on anything that fails with those, e.g. anything that uses
"self.assertEqual(0, " in the code could hit that kind of query - and we
want to avoid those types of queries running against changes in the
check queue since they could mask legitimate test failures.
So we add something unique in these tests to identify them from other
test failures in Cinder by using a hard-coded uuid for the project_id so
that if we have an assertion failure with notifications, the
notification payload is dumped (due to commit 4bbe1c87e26150c62e1dc8a2a5b8979896c1ddd9) and we can query on the
assertion failure plus the hard-coded project_id used only in these
tests.
Rajesh Tailor [Wed, 17 Jun 2015 11:35:35 +0000 (04:35 -0700)]
Incorrect exception caught in qos-specs create api
Caught InvalidQoSSpecs exception instead of InvalidInput exception
in qos-specs create api, as _verify_prepare_qos_specs method raises
InvalidQoSSpecs exception if invalid value is passed.
Michal Dulko [Wed, 3 Jun 2015 12:25:26 +0000 (14:25 +0200)]
Backup object
This patch adds VersionedObjects abstraction layer to volume backups.
The object derives from CinderObjectDictCompat, so it supports both
object (obj.prop) and dict (obj['prop']) syntax to access properties.
Complete move to object notation will be made in a follow up clean up
patch.
Co-Authored-By: Grzegorz Grasza <grzegorz.grasza@intel.com>
Change-Id: Icff37261b367463b71a1268be16f9c97f595bf0c
Partial-Implements: blueprint cinder-objects
Michal Dulko [Thu, 18 Jun 2015 07:27:26 +0000 (09:27 +0200)]
Add missing Jinja2 to requirements.txt
Recently merged commit I5bbdc0f97db9b0ebd7b48e50ab7869e2ca33aead got in
a dependency on Jinja2 library without adding it to requirements.txt.
The gate was running fine, because Nova already has it in it's
requirements. Problem uncovers when we install DevStack with
--use-venv=True option.
Code Review / openstack-build / cinder-build.git / log