Tom Barron [Thu, 11 Jun 2015 17:07:07 +0000 (13:07 -0400)]
Remove oslo logging from backup unit tests
Many of the unit tests for backup services import oslo_log and
invoke LOG.debug, LOG.info, etc. There appears to be no current
reason to do this and, more generally, unit tests for everything
but oslo_log itself should not be doing logging themselves.
This commit removes oslo logging from the backup unit tests.
Closes-bug: 1464772
Yuriy Nesenenko [Tue, 19 May 2015 15:17:01 +0000 (18:17 +0300)]
Filter snapshots data on the DB side
It filters snapshots data on the DB side to improve performance.
Some tests are removed from test_snapshots.py because their function
implemented on the DB side.
Victor Stinner [Tue, 23 Jun 2015 11:37:42 +0000 (13:37 +0200)]
Fix Python 3 issues in the swift backup driver
* On Python 3, encode/decode JSON to/from UTF-8
* Use byte strings for volume content
* Replace dict.keys()[0] and dict.value()[0] with list(dict.items())[0].
Get the key and the value at once to ensure that they are consistent.
On Python 3, items() returns an iterator: create a list to use the [0]
operator.
* SwiftBackupDriver: use a bytearray instead of a string because
bytes += bytes is inefficient on Python 3, whereas str += str is
optimized on Python 2. Replace also StringIO with BytesIO.
* BytesIO has no more len attribute on Python 3: get the length of the
content instead.
* Replace buffer(bytearray(128)) with b'\0' * 128 to create a string of
128 zeroed bytes
* Replace "rw" file mode with "w+" ("rw" raises an exception
on Python 3).
* tox.ini: add the following tests to Python 3.4
Victor Stinner [Tue, 23 Jun 2015 09:55:40 +0000 (11:55 +0200)]
Fix Python 3 issues in ceph and rbd drivers
* Add cinder.utils.convert_str() function: helper to convert a string to
a native string: convert to bytes on Python 2, convert to Unicode on
Python 3. Function needed by ceph and rbd drivers.
* Replace encodeutils.safe_encode() with utils.convert_str()
* test_backup_ceph: get the size of the volume file using os.fstat() to
fix a test. Before the volume size was a mock, but comparison between
integers and mocks raise a TypeError on Python 3.
* Fix RBDDriver._update_volume_stats(): replace a/b with a//b to get an integer
on Python 2 and Python 3 (a/b always returns a float on Python 3)
* Replace map() with a list-comprehension in RBDDriver because a list is
expected. On Python 3, map() returns an iterator.
* tox.ini: add to following tests to Python 3.4
Victor Stinner [Tue, 23 Jun 2015 09:20:47 +0000 (11:20 +0200)]
Fix Python 3 issues in backup
* Replace (int, long) with six.integer_types: the "long" type has been
removed in Python 3
* Replace str.encode("bas64") with base64.encodestring(str), base64
text codec has been removed in Python 3. Same change for decode
(base64.decodestring)
* On Python 3, encode JSON to UTF-8
* tox.ini: add the following tests to Python 3
Michal Dulko [Tue, 23 Jun 2015 13:47:53 +0000 (15:47 +0200)]
Harden scheduler.rpcapi unit tests
Tests for scheduler.rpcapi are very weak. Even changing keyword
arguments names passed to call and cast methods in this module
doesn't trigger unit tests failures. This commit adds keyword
arguments validation to prevent problems similar to ones caused
by bug 1467841 in backup.rpcapi. I've also fixed argument order
in assertEqual usages there.
Michal Dulko [Tue, 23 Jun 2015 13:42:00 +0000 (15:42 +0200)]
Fix backups.rpcapi to pass objects over RPC
Commit Icff37261b367463b71a1268be16f9c97f595bf0c brought bugs in
backup.rpcapi. For export_record, import_record and reset_status
backup.id is passed instead of whole backup object - which
backup.manager expects. This causes severe failures. This commit
changes the rpcapi to pass objects and adds missing
backup.test_rpcapi unit tests that will prevent such failures in
the future.
wanghao [Tue, 26 May 2015 09:25:26 +0000 (17:25 +0800)]
Fix weird change of volume status in re-scheduling
When create a volume failed as some exception, cinder will re-schedule the
creation request, the retry number is 3 in default.
But when re-scheduling, this volume's status was changed like this:
This is weird to user, since if this volume is in re-scheduling process,
it's status should always be creating, and then become error or available
at last.
Victor Stinner [Tue, 23 Jun 2015 09:11:04 +0000 (11:11 +0200)]
Fix tox -e py34
Add a py34 test environment to tox.ini running a subset of tests which
pass on Python 3.4. With this change, the py34 check job should pass and
so it will become possible to make the job voting to avoid further
Python 3 regressions.
Jay S. Bryant [Mon, 22 Jun 2015 16:43:37 +0000 (11:43 -0500)]
Add exception catch in report_state for DBError
We discovered while testing Cinder in an HA
environment that transient DB errors can be
encountered that are not currently covered by
the exception catch in service.py report_state().
The uncaught exceptions were causing the thread
for report_state to prematurely exit and causing
services to no longer update the DB when, in fact,
they were still usable.
This change adds an exception catch for DBError
so that the thread can continue to function.
Sean McGinnis [Thu, 18 Jun 2015 21:56:33 +0000 (16:56 -0500)]
Dell SC: Enable use of Storage Profiles
The Storage Center array allows volumes to be associated with
different Storage Profiles. These profiles allow setting tiering
and RAID types per volume. The driver currently only uses the
default Storage Profile configured for the Cinder user account.
This patch adds the capability to use different Storage Profiles
by using extra specs to allow the creation of different volume
types for the desired profiles.
Ivan Kolodyazhny [Sat, 20 Jun 2015 20:02:41 +0000 (23:02 +0300)]
Use elevated context for backup destroy
db.backup_destroy requires admin context to mark backup record as
deleted. Commit Icff37261b367463b71a1268be16f9c97f595bf0c removed admin
context passed to backup_destroy method.
This patch uses elevated context to destroy backup.
Change-Id: I75b9e1fff48569a8aa320f2f02914fc7b6665d79
Closes-Bug: #1467167
Ivan Kolodyazhny [Sun, 21 Jun 2015 08:33:05 +0000 (11:33 +0300)]
Fix Cinder Objects unit tests
Due to the import order all tests passed on CI. But you can not
run cinder.unit.objects.test_backup only. This patch fixes it
adding base class and invokes objects.register_all().
Matt Riedemann [Sat, 20 Jun 2015 19:43:34 +0000 (12:43 -0700)]
rbd: add volume_id to connection_info in initialize_connection
Most other volume drivers have a 'volume_id' key in the
connection_info['data'] dict returned from initialize_connection, this
adds it to the rbd volume driver.
This is part of a bigger effort to standardize the connection_info dict
so that consumers like Nova can rely on a common set of keys when
processing connection_info.
Mike Perez [Sat, 20 Jun 2015 02:12:15 +0000 (19:12 -0700)]
Fix Datera driver export call
A previous commit 3fabed9d64696a204263fd7e00b65115a8ecce87 broke the
export call for the Datera driver. This allows the information to
correctly be accessed again.
Pavel Boldin [Wed, 13 May 2015 22:01:36 +0000 (01:01 +0300)]
Add iscsi_target_flags configuration option
Newly introduced `iscsi_target_flags' allows to specify `tgtd'
backing storage flags. This can be used to instruct `tgtd' to
open backing device file using `O_DIRECT' flag ensuring direct
access without cache.
John Griffith [Fri, 19 Jun 2015 18:32:01 +0000 (12:32 -0600)]
Remove the hardcoded concurrency limit for ostestr
ostestr by default uses concurrency = ncpu.
This caused all sorts of things to blow up in Cinder
when one got over 6 cpus. Since the initial merge a
patch has landed that appears to fix the issues, and
we now run reliably at 8 and even 12 cpus.
This patch removes the hard coded concurrency=6 setting
from tox.ini
Vilobh Meshram [Fri, 19 Jun 2015 17:17:07 +0000 (10:17 -0700)]
Nested Quota : Create allocated column in cinder.quotas
Create allocated column in cinder.quotas. This allocated
column will be the sum of "hard_limits" values for the
immediate child projects. This will be needed to track
allocated quota to child projects.
Change-Id: Ia80d9a6cdbbc7e86cf9f11979f5e80c53f0fac1f
Implements: bp cinder-nested-quota-driver
Yuriy Nesenenko [Thu, 4 Jun 2015 15:50:14 +0000 (18:50 +0300)]
Handle incorrect '--config-dir' param
This patch fixes the error that occurs if the --config-dir directory
doesn't exist. The exception ConfigDirNotFoundError should be caught
in this case.
Heat stack deletion followed by an unsuccessful heat stack delete
attempt resulted in calling begin_detaching on a volume which was
already in detaching state. It was observed that the API returned
success instead of failing the operation. There is a check which
verifies the volume status, but it is not using the updated volume
status. This patch fixes the problem by getting updated volume
status from DB. But this fix won't eliminate the possibility of
above mentioned inconsistent behavior if multiple begin_detaching
happens concurrently.
Eric Harney [Thu, 18 Jun 2015 22:06:54 +0000 (18:06 -0400)]
Tests: Make fake_notifier per-instance
Tests have been failing because multiple threads modify the
same global NOTIFICATIONS list. Move the notification storage
to be per-test instance, which is what the tests assume.
This has been broken for a while, but made more obvious by
recent changes to parallelize tests.
Also remove all the extra reset calls we added to work around
this problem over time.
Dmitry Guryanov [Thu, 18 Jun 2015 17:14:34 +0000 (20:14 +0300)]
Add Virtuozzo Storage Volume Driver
Add a volume driver which can use Virtuozzo Storage, which
has filesystem interface and so volume driver has a similar
workflow to NFS and SMBFS drivers.
At this point the driver contain minimal set of features. Because
I think some refactoring should be done in RemoteFS drivers before
further development. For example code, which deals with image formats
should go to the RemoteFS class. So I don't add qcow2 images support
in this patch.
Matt Riedemann [Thu, 18 Jun 2015 15:16:52 +0000 (08:16 -0700)]
Use a hard-coded project_id in racy cinder.tests.unit.test_volume tests
Since concurrent unit test runs have been turned on with ostestr there
are races with the notifications causing the assertions in
test_volume.py tests to fail.
A few attempts have been made at writing an elastic-recheck query to
identify the failures but they are too generic, e.g. querying for
"fake_notifier.NOTIFICATIONS" or "MismatchError 0 != ", which could hit
on anything that fails with those, e.g. anything that uses
"self.assertEqual(0, " in the code could hit that kind of query - and we
want to avoid those types of queries running against changes in the
check queue since they could mask legitimate test failures.
So we add something unique in these tests to identify them from other
test failures in Cinder by using a hard-coded uuid for the project_id so
that if we have an assertion failure with notifications, the
notification payload is dumped (due to commit 4bbe1c87e26150c62e1dc8a2a5b8979896c1ddd9) and we can query on the
assertion failure plus the hard-coded project_id used only in these
tests.
Rajesh Tailor [Wed, 17 Jun 2015 11:35:35 +0000 (04:35 -0700)]
Incorrect exception caught in qos-specs create api
Caught InvalidQoSSpecs exception instead of InvalidInput exception
in qos-specs create api, as _verify_prepare_qos_specs method raises
InvalidQoSSpecs exception if invalid value is passed.
Michal Dulko [Wed, 3 Jun 2015 12:25:26 +0000 (14:25 +0200)]
Backup object
This patch adds VersionedObjects abstraction layer to volume backups.
The object derives from CinderObjectDictCompat, so it supports both
object (obj.prop) and dict (obj['prop']) syntax to access properties.
Complete move to object notation will be made in a follow up clean up
patch.
Co-Authored-By: Grzegorz Grasza <grzegorz.grasza@intel.com>
Change-Id: Icff37261b367463b71a1268be16f9c97f595bf0c
Partial-Implements: blueprint cinder-objects
Michal Dulko [Thu, 18 Jun 2015 07:27:26 +0000 (09:27 +0200)]
Add missing Jinja2 to requirements.txt
Recently merged commit I5bbdc0f97db9b0ebd7b48e50ab7869e2ca33aead got in
a dependency on Jinja2 library without adding it to requirements.txt.
The gate was running fine, because Nova already has it in it's
requirements. Problem uncovers when we install DevStack with
--use-venv=True option.
Doug Hellmann [Tue, 16 Jun 2015 19:49:14 +0000 (19:49 +0000)]
Update version for Liberty
Update the version for Liberty, switching from date-based versioning
to pre-versioning using SemVer. See
http://lists.openstack.org/pipermail/openstack-dev/2015-May/065211.html
and
http://lists.openstack.org/pipermail/openstack-dev/2015-June/067082.html
for details.
Ivan Kolodyazhny [Thu, 11 Jun 2015 10:55:28 +0000 (13:55 +0300)]
Ceph driver support retries on rados_connect_timeout
Added retry feature when connection to Ceph cluster failed. This patch
introduces new config options rados_connection_retries and
rados_connection_interval to configure retries due to
the rados_connect_timeout.
This patch adds reporting of the multiattach capability
in the lvm driver. Currently the Cinder scheduler does filtering
on hosts at create time for the --allow-multiattach option being
passed into the create request from the client. In order for someone
to be able to create an LVM volume that can do multiattach, the LVM
driver needs to report the capability.
This patch's purpose is to allow LVM volumes to be multiattachable
for the folks working on the Nova patches to get Nova to do multiattach.
Without this patch, they can't do end to end testing.
Lee [Fri, 24 Apr 2015 07:50:20 +0000 (15:50 +0800)]
Add volume drivers for Infortrend Storage
Infortrend implement ISCSI and FC volume drivers for
EonStor DS product.
It manages storage by Infortrend CLI tool.
common_cli.py implements the basic Cinder Driver API.
infortrend_fc_cli.py and infortrend_iscsi_cli.py use them to
provide FC and iSCSI specific support.
Support features:
- Volume Create/Delete
- Volume Attach/Detach
- Snapshot Create/Delete
- Create Volume from Snapshot
- Get Volume Stats
- Copy Image to Volume
- Copy Volume to Image
- Clone Volume
- Extend Volume
Shay Halsband [Sun, 8 Mar 2015 12:34:53 +0000 (14:34 +0200)]
XtremIO Volume driver requests, multipath
* introduce new driver parameter to configure ssl CA validation
* replace urllib2 with requests
* changes to support iSCSI multipath
* fix missing logging for get requets
* query cluster version without exception
DocImpact: added the option driver_ssl_cert_verify to specify whether
a volume backends verify SSL certificats when quering https urls.
Code Review / openstack-build / cinder-build.git / log