Victor Stinner [Mon, 29 Jun 2015 13:33:12 +0000 (15:33 +0200)]
Port dothill to Python 3
* Replace urllib2 with six.moves.urllib
* On Python 3, encode string to UTF-8 to hash it using MD5
* On Python 3, decode base64 from ASCII to get Unicode
* Replace "pattern in exc" with "pattern in exc.args"
* test_initialize_connection(): fix get_active_iscsi_target_portals()
mock. Use the return_value attribute to return a dictionary, instead
of returning a single IP address.
* tox.ini: add cinder.tests.unit.test_dothill to Python 3.4
Victor Stinner [Mon, 29 Jun 2015 13:53:44 +0000 (15:53 +0200)]
Port drbdmanagedrv driver to Python 3
* Replace map() with a list-comprehension where a list is expected
* Replace apply(fn, args) with fn(*args)
* Use str.replace() to remove "{" and "}" characters in _clean_uuid(),
instead of using str.translate()
* Use literal syntax to create a new dictionary instead of using
dict() + dict.items()
* tox.ini: add cinder.tests.unit.test_drbdmanagedrv to Python 3.4
Kazumasa Nomura [Thu, 25 Jun 2015 18:33:16 +0000 (03:33 +0900)]
Fix cinder.conf.sample generation
Introduction of the hp_xp driver broke cinder.conf.sample generation due to
the way the driver was registering the configuration items. Originally the
options were being included into the FC driver file and registered there.
When the config generator found the options in the separate hp_xp_opts.py
file it did not know what group to include the options in since they were
not registered at all.
This fix moves registration of the options to the hp_xp_opts.py file so
that the config generation will work. Since cfg.CONF is a global, the
hp_xp driver will still have access to the options in there after they are
set by importing the opts file. An additional problem that was uncovered
when creating this fix is the fact that the old config generator we are
using from oslo-incubator doesn't support using cfg.Opt. To temporarily
workaround this I have changed the cfg.Opts to be cfg.IntOpts or
cfg.ListOpts(as appropriate) and documented the valid values for those
options.
The last change that was necessary was to add config values for several of
the options in cinder/tests.py as the options were marked as required by
the driver. If values were not set this caused oslo config to error out
reporting that they values needed to be set.
Change-Id: Ie47d38c2f8fde675cf00ea2ec9abf60691c57ef9
Closes-Bug: 1466198 Signed-off-by: Jay S. Bryant<jsbryant@us.ibm.com>,
Kazumasa Nomura <kazumasa.nomura.rx@hitachi.com>
Tom Barron [Thu, 11 Jun 2015 17:07:07 +0000 (13:07 -0400)]
Remove oslo logging from backup unit tests
Many of the unit tests for backup services import oslo_log and
invoke LOG.debug, LOG.info, etc. There appears to be no current
reason to do this and, more generally, unit tests for everything
but oslo_log itself should not be doing logging themselves.
This commit removes oslo logging from the backup unit tests.
Closes-bug: 1464772
Jay S. Bryant [Fri, 26 Jun 2015 20:56:47 +0000 (15:56 -0500)]
Sync the latest fileutils module from oslo-incubator
As part of our normal development in a release we should get
our openstack/common directory code synced up with the latest
code in oslo-incubator. This is the sync for fileutils:
Current HEAD in OSLO:
---------------------
commit 2e41adeb4c19114fd3504eb69891e9149dba7072
Date: Wed Jun 24 14:45:25 2015 +0000
Updated from global requirements
Changes merged with this patch:
--------------------- 16eb642eb -- Clean up logging to conform to guidelines
Tom Barron [Wed, 17 Jun 2015 18:54:01 +0000 (14:54 -0400)]
Fix 'no actual-pathname' NetApp API error
Cinder volume logs sometimes show this error
NetApp API failed. Reason - 13114:
No actual-pathname for 10.1.0.9:/vol/whatever
with Kilo code and 7-mode DOT system.
The issue is due to our Cinder driver passing the
entire share to the relevant API instead of just
the export path portion of the share. This only
happens when the NFS image cache is full, and cached
files need to be removed.
Cedric Zhuang [Thu, 21 May 2015 11:26:23 +0000 (04:26 -0700)]
Use symbol for error code in VNX cinder driver
Some of the error code of VNX CLI is scattered in the code and is
not coded as a constant.
Gather all VNX error codes in the VNXError class and compose a new
method to check the error in the CLI output. If no specific error
is supplied, check all available errors in the output.
Yuriy Nesenenko [Tue, 19 May 2015 15:17:01 +0000 (18:17 +0300)]
Filter snapshots data on the DB side
It filters snapshots data on the DB side to improve performance.
Some tests are removed from test_snapshots.py because their function
implemented on the DB side.
PranaliDeore [Tue, 26 May 2015 07:12:27 +0000 (00:12 -0700)]
Change generic NotFound to specific exception
1. Changed exception type from NotFound to specific NotFound
exception for better readability.
e.g. VolumeNotFound, SnapshotNotFound etc.
2. Error messages returned to the user are not consistent across
all apis in case of all exceptions derived from NotFound exception.
Instead of returning custom error messages return message
defined in exception class itself.
NOTE: Doesn't applied to 'HostBinaryNotFound' as host information to
the tenant should not be leaked.
3. Added explanation to HTTPNotFound wherever appropriate.
e.g. raise webob.exc.HTTPNotFound() to
raise webob.exc.HTTPNotFound(explanation=err.msg)
Jay S. Bryant [Wed, 24 Jun 2015 19:35:38 +0000 (14:35 -0500)]
Fix library includes for config generator
Noticed today while working on fixing issues
with the config generator that oslo_policy
had been added in the wrong location for
pulling the config options in to the config
generator.
This fix moves the items that had been put in
oslo.config.generator.rc but were missing in generate_sample.sh
over to the appropriate location. I also removed the option for
importing additional libraries in oslo.config.generator.rc
given that it will not work.
I am still working on a longer term solution, but
we really should get this fixed so we can generate sample
config files again.
Anton Arefiev [Wed, 24 Jun 2015 15:45:35 +0000 (18:45 +0300)]
Fix cinder-manage volume delete cmd
Since pools was introduced in cinder, field 'host' in resources
contains pool part. So when we are trying to delete volume with
cinder-manage, 'host' with pool part passed to rpc target. As
result message isn't delivered to host, and we can't delete volume.
Victor Stinner [Tue, 23 Jun 2015 12:08:25 +0000 (14:08 +0200)]
Fix Python 3 issues in the blockbridge driver
* Replace httplib import with six.moves.http_client
* Replace urllib imports with six.moves.urllib, update urllib.
For example, replace urllib.quote() with urllib.parse.quote().
* test_blockbridge: try get the mock module from unittest.mock, part of
the Python stdlib since Python 3.3. The third-party mock module has a
bug on Python 3.4.
* On Python 3, base64.encodestring() expects bytes and returns bytes.
Encode credentials to UTF-8 and decode base64 from ASCII to get the
final string as Unicode on Python 3.
* test_cfg_api_auth_scheme_password: parse the connection URL to compare
it. Comparing directly the URL fails on Python 3 because URL
parameters are rendered in a random order because of the hash
randomization.
* Replace pools.values()[0] with list(pools.values())[0]. On Python 3,
dict.values() returns an iterator.
* tox.ini: add cinder.tests.unit.test_blockbridge to Python 3.4.
Victor Stinner [Tue, 23 Jun 2015 11:37:42 +0000 (13:37 +0200)]
Fix Python 3 issues in the swift backup driver
* On Python 3, encode/decode JSON to/from UTF-8
* Use byte strings for volume content
* Replace dict.keys()[0] and dict.value()[0] with list(dict.items())[0].
Get the key and the value at once to ensure that they are consistent.
On Python 3, items() returns an iterator: create a list to use the [0]
operator.
* SwiftBackupDriver: use a bytearray instead of a string because
bytes += bytes is inefficient on Python 3, whereas str += str is
optimized on Python 2. Replace also StringIO with BytesIO.
* BytesIO has no more len attribute on Python 3: get the length of the
content instead.
* Replace buffer(bytearray(128)) with b'\0' * 128 to create a string of
128 zeroed bytes
* Replace "rw" file mode with "w+" ("rw" raises an exception
on Python 3).
* tox.ini: add the following tests to Python 3.4
Victor Stinner [Tue, 23 Jun 2015 09:55:40 +0000 (11:55 +0200)]
Fix Python 3 issues in ceph and rbd drivers
* Add cinder.utils.convert_str() function: helper to convert a string to
a native string: convert to bytes on Python 2, convert to Unicode on
Python 3. Function needed by ceph and rbd drivers.
* Replace encodeutils.safe_encode() with utils.convert_str()
* test_backup_ceph: get the size of the volume file using os.fstat() to
fix a test. Before the volume size was a mock, but comparison between
integers and mocks raise a TypeError on Python 3.
* Fix RBDDriver._update_volume_stats(): replace a/b with a//b to get an integer
on Python 2 and Python 3 (a/b always returns a float on Python 3)
* Replace map() with a list-comprehension in RBDDriver because a list is
expected. On Python 3, map() returns an iterator.
* tox.ini: add to following tests to Python 3.4
Victor Stinner [Tue, 23 Jun 2015 09:20:47 +0000 (11:20 +0200)]
Fix Python 3 issues in backup
* Replace (int, long) with six.integer_types: the "long" type has been
removed in Python 3
* Replace str.encode("bas64") with base64.encodestring(str), base64
text codec has been removed in Python 3. Same change for decode
(base64.decodestring)
* On Python 3, encode JSON to UTF-8
* tox.ini: add the following tests to Python 3
git-harry [Wed, 24 Jun 2015 08:46:45 +0000 (09:46 +0100)]
Remove generate_glance_url
The function generate_glance_url is part of the inital fork from nova.
It is only called once and that is by a unit test designed to test the
function.
This commit removes generate_glance_url and its associated test because
they are not used by cinder.
Michal Dulko [Tue, 23 Jun 2015 13:47:53 +0000 (15:47 +0200)]
Harden scheduler.rpcapi unit tests
Tests for scheduler.rpcapi are very weak. Even changing keyword
arguments names passed to call and cast methods in this module
doesn't trigger unit tests failures. This commit adds keyword
arguments validation to prevent problems similar to ones caused
by bug 1467841 in backup.rpcapi. I've also fixed argument order
in assertEqual usages there.
Michal Dulko [Tue, 23 Jun 2015 13:42:00 +0000 (15:42 +0200)]
Fix backups.rpcapi to pass objects over RPC
Commit Icff37261b367463b71a1268be16f9c97f595bf0c brought bugs in
backup.rpcapi. For export_record, import_record and reset_status
backup.id is passed instead of whole backup object - which
backup.manager expects. This causes severe failures. This commit
changes the rpcapi to pass objects and adds missing
backup.test_rpcapi unit tests that will prevent such failures in
the future.
wanghao [Tue, 26 May 2015 09:25:26 +0000 (17:25 +0800)]
Fix weird change of volume status in re-scheduling
When create a volume failed as some exception, cinder will re-schedule the
creation request, the retry number is 3 in default.
But when re-scheduling, this volume's status was changed like this:
This is weird to user, since if this volume is in re-scheduling process,
it's status should always be creating, and then become error or available
at last.
Victor Stinner [Tue, 23 Jun 2015 09:11:04 +0000 (11:11 +0200)]
Fix tox -e py34
Add a py34 test environment to tox.ini running a subset of tests which
pass on Python 3.4. With this change, the py34 check job should pass and
so it will become possible to make the job voting to avoid further
Python 3 regressions.
Jay S. Bryant [Mon, 22 Jun 2015 16:43:37 +0000 (11:43 -0500)]
Add exception catch in report_state for DBError
We discovered while testing Cinder in an HA
environment that transient DB errors can be
encountered that are not currently covered by
the exception catch in service.py report_state().
The uncaught exceptions were causing the thread
for report_state to prematurely exit and causing
services to no longer update the DB when, in fact,
they were still usable.
This change adds an exception catch for DBError
so that the thread can continue to function.
Sean McGinnis [Thu, 18 Jun 2015 21:56:33 +0000 (16:56 -0500)]
Dell SC: Enable use of Storage Profiles
The Storage Center array allows volumes to be associated with
different Storage Profiles. These profiles allow setting tiering
and RAID types per volume. The driver currently only uses the
default Storage Profile configured for the Cinder user account.
This patch adds the capability to use different Storage Profiles
by using extra specs to allow the creation of different volume
types for the desired profiles.
Ivan Kolodyazhny [Sat, 20 Jun 2015 20:02:41 +0000 (23:02 +0300)]
Use elevated context for backup destroy
db.backup_destroy requires admin context to mark backup record as
deleted. Commit Icff37261b367463b71a1268be16f9c97f595bf0c removed admin
context passed to backup_destroy method.
This patch uses elevated context to destroy backup.
Change-Id: I75b9e1fff48569a8aa320f2f02914fc7b6665d79
Closes-Bug: #1467167
Ivan Kolodyazhny [Sun, 21 Jun 2015 08:33:05 +0000 (11:33 +0300)]
Fix Cinder Objects unit tests
Due to the import order all tests passed on CI. But you can not
run cinder.unit.objects.test_backup only. This patch fixes it
adding base class and invokes objects.register_all().
Matt Riedemann [Sat, 20 Jun 2015 19:43:34 +0000 (12:43 -0700)]
rbd: add volume_id to connection_info in initialize_connection
Most other volume drivers have a 'volume_id' key in the
connection_info['data'] dict returned from initialize_connection, this
adds it to the rbd volume driver.
This is part of a bigger effort to standardize the connection_info dict
so that consumers like Nova can rely on a common set of keys when
processing connection_info.
Mike Perez [Sat, 20 Jun 2015 02:12:15 +0000 (19:12 -0700)]
Fix Datera driver export call
A previous commit 3fabed9d64696a204263fd7e00b65115a8ecce87 broke the
export call for the Datera driver. This allows the information to
correctly be accessed again.
Pavel Boldin [Wed, 13 May 2015 22:01:36 +0000 (01:01 +0300)]
Add iscsi_target_flags configuration option
Newly introduced `iscsi_target_flags' allows to specify `tgtd'
backing storage flags. This can be used to instruct `tgtd' to
open backing device file using `O_DIRECT' flag ensuring direct
access without cache.
John Griffith [Fri, 19 Jun 2015 18:32:01 +0000 (12:32 -0600)]
Remove the hardcoded concurrency limit for ostestr
ostestr by default uses concurrency = ncpu.
This caused all sorts of things to blow up in Cinder
when one got over 6 cpus. Since the initial merge a
patch has landed that appears to fix the issues, and
we now run reliably at 8 and even 12 cpus.
This patch removes the hard coded concurrency=6 setting
from tox.ini
Code Review / openstack-build / cinder-build.git / log