Alex Meade [Tue, 7 Jul 2015 19:04:01 +0000 (15:04 -0400)]
NetApp E-Series: Add debug tracing
Add method and api level debug tracing to the NetApp
E-Series drivers using the utils.trace_method
decorator. This is enabled in the driver via the
'trace_flags' configuration option.
Alex Meade [Mon, 6 Jul 2015 14:17:43 +0000 (10:17 -0400)]
Add tracing facility for drivers
It is useful for engineering to be able to see the API traffic
to and from the storage controllers, as well as to see the
progression of method calls through the driver.
This work will add a simple config option to enable various
debug tracing types to be written to the debug log at runtime.
Use of the decorator has been added to NetApp drivers.
Drivers can make use of this facility by adding the
@cinder.utils.trace_method decorator to any method
calls they would like to allow tracing for or use trace_flags
for any other developer targeted debug logs. The trace_method
decorator can easily be added to every function of a class by
using the cinder.utils.TraceWrapperMetaclass or
cinder.utils.TraceWrapperWithABCMetaclass metaclasses. It is
advised to use the added decorators as the outer most decorator
for a function in order to avoid side-effects with other
decorators.
The @cinder.utils.trace_api method should be used to log
requests to storage backends by wrapping a method that
has the request as an argument and returns the response.
For example:
@six.add_metaclass(cinder.utils.TraceWrapperMetaclass)
class MyClass(object):
pass
Vincent Hou [Thu, 7 May 2015 05:53:25 +0000 (13:53 +0800)]
Implement the update_migrated_volume for the drivers
This patch implements update_migrated_volume for LVM, Storwize
and updates Dell, StorPool and Infortrend drivers accordingly. It
makes sure that after a successful volume migration, the cinder
volume name(id) is the same as the backend volume name(id). Other
back-end drivers can take this patch as a reference to implement
update_migrated_volume.
PS: Not applicable to multi-attached volumes, since we need to wait
until the multi-attach lands in Nova.
This patch also adds a unit test for the StorPool driver's
update_migrated_volume() implementation.
Co-Authored-By: Peter Penchev <openstack-dev@storpool.com>
Change-Id: I69707340ddf2b55286ff0d84319529b2f502cefa
Partial-Bug: #1450649
Victor Stinner [Tue, 30 Jun 2015 14:47:15 +0000 (16:47 +0200)]
Fix Python 3 issues in cmd
* Replace filter() with a list-comprehension using if to get a list on
Python 3.
* Get the mock module from the stdlib unittest module on Python 3.3 and
newer, or fallback to the third-party mock module.
* Replace __builtin__ with six.moves.builtins.
* tox.ini: add the following tests for Python 3.4
Victor Stinner [Tue, 30 Jun 2015 14:06:28 +0000 (16:06 +0200)]
Port image/glance.py to Python 3
* Fix usage of six.reraise(): new_exc is the exception value, not the
exception type
* Replace __builtin__ with six.moves.builtins.
* test_extracting_v2_boot_properties(): set config.glance_num_retries to
fix the test on Python 3 (comparison between mock and int now raises
a TypeError on Python 3).
* TestGlanceSerializer: use dictionaries with only one key to have a
reliable output even if the hash is randomized (hash randomization is
now enabled by default in Python 3).
* tox.ini: add cinder.tests.unit.image.test_glance to Python 3.4.
Commit Ibaf43858d60e1320c339f2523b5c09c7f7c7f91e caused new problem with
cross thread communication. According to Python documentation, code can
lead to a deadlock if the spawned thread directly or indirectly attempts
to import a module. python-rados spawns new thread to connect to
cluster. So I removed new spawning new thread to connect to rados. All
long-running operations calls whith python-rbd are still implemented in
native Python threads to block eventlet loop.
Victor Stinner [Mon, 29 Jun 2015 13:33:12 +0000 (15:33 +0200)]
Port dothill to Python 3
* Replace urllib2 with six.moves.urllib
* On Python 3, encode string to UTF-8 to hash it using MD5
* On Python 3, decode base64 from ASCII to get Unicode
* Replace "pattern in exc" with "pattern in exc.args"
* test_initialize_connection(): fix get_active_iscsi_target_portals()
mock. Use the return_value attribute to return a dictionary, instead
of returning a single IP address.
* tox.ini: add cinder.tests.unit.test_dothill to Python 3.4
Victor Stinner [Tue, 30 Jun 2015 14:14:09 +0000 (16:14 +0200)]
Fix Python 3 issues in targets unit tests
* Replace StandardError with ZeroDivisionError: StandardError was
removed in Python 3.
* Replace __builtin__ with six.moves.builtins.
* Open text mode with "w" mode (instead of "wb").
* tox.ini: add targets unit tests to Python 3.4
Note: cinder.tests.unit.targets.test_iet_driver still fails on Python 3,
it will be fixed in a different change.
Victor Stinner [Mon, 29 Jun 2015 13:53:44 +0000 (15:53 +0200)]
Port drbdmanagedrv driver to Python 3
* Replace map() with a list-comprehension where a list is expected
* Replace apply(fn, args) with fn(*args)
* Use str.replace() to remove "{" and "}" characters in _clean_uuid(),
instead of using str.translate()
* Use literal syntax to create a new dictionary instead of using
dict() + dict.items()
* tox.ini: add cinder.tests.unit.test_drbdmanagedrv to Python 3.4
Kazumasa Nomura [Thu, 25 Jun 2015 18:33:16 +0000 (03:33 +0900)]
Fix cinder.conf.sample generation
Introduction of the hp_xp driver broke cinder.conf.sample generation due to
the way the driver was registering the configuration items. Originally the
options were being included into the FC driver file and registered there.
When the config generator found the options in the separate hp_xp_opts.py
file it did not know what group to include the options in since they were
not registered at all.
This fix moves registration of the options to the hp_xp_opts.py file so
that the config generation will work. Since cfg.CONF is a global, the
hp_xp driver will still have access to the options in there after they are
set by importing the opts file. An additional problem that was uncovered
when creating this fix is the fact that the old config generator we are
using from oslo-incubator doesn't support using cfg.Opt. To temporarily
workaround this I have changed the cfg.Opts to be cfg.IntOpts or
cfg.ListOpts(as appropriate) and documented the valid values for those
options.
The last change that was necessary was to add config values for several of
the options in cinder/tests.py as the options were marked as required by
the driver. If values were not set this caused oslo config to error out
reporting that they values needed to be set.
Change-Id: Ie47d38c2f8fde675cf00ea2ec9abf60691c57ef9
Closes-Bug: 1466198 Signed-off-by: Jay S. Bryant<jsbryant@us.ibm.com>,
Kazumasa Nomura <kazumasa.nomura.rx@hitachi.com>
Xi Yang [Tue, 23 Jun 2015 09:42:18 +0000 (05:42 -0400)]
Fix timeout issue in EMC VNX driver unit test
This patch is going to fix the timeout issue in EMC VNX
driver unit test.
Currently, the timeout value of a looping call in EMC unit test
is set to 0.000002s which is too small. The execution time of
a looping call may exceed this timeout value and cause the test
case failure. This fix is changing the timeout to the default
value of VNX driver.
Tom Barron [Thu, 11 Jun 2015 17:07:07 +0000 (13:07 -0400)]
Remove oslo logging from backup unit tests
Many of the unit tests for backup services import oslo_log and
invoke LOG.debug, LOG.info, etc. There appears to be no current
reason to do this and, more generally, unit tests for everything
but oslo_log itself should not be doing logging themselves.
This commit removes oslo logging from the backup unit tests.
Closes-bug: 1464772
Jay S. Bryant [Fri, 26 Jun 2015 20:56:47 +0000 (15:56 -0500)]
Sync the latest fileutils module from oslo-incubator
As part of our normal development in a release we should get
our openstack/common directory code synced up with the latest
code in oslo-incubator. This is the sync for fileutils:
Current HEAD in OSLO:
---------------------
commit 2e41adeb4c19114fd3504eb69891e9149dba7072
Date: Wed Jun 24 14:45:25 2015 +0000
Updated from global requirements
Changes merged with this patch:
--------------------- 16eb642eb -- Clean up logging to conform to guidelines
Tom Barron [Wed, 17 Jun 2015 18:54:01 +0000 (14:54 -0400)]
Fix 'no actual-pathname' NetApp API error
Cinder volume logs sometimes show this error
NetApp API failed. Reason - 13114:
No actual-pathname for 10.1.0.9:/vol/whatever
with Kilo code and 7-mode DOT system.
The issue is due to our Cinder driver passing the
entire share to the relevant API instead of just
the export path portion of the share. This only
happens when the NFS image cache is full, and cached
files need to be removed.
Cedric Zhuang [Thu, 21 May 2015 11:26:23 +0000 (04:26 -0700)]
Use symbol for error code in VNX cinder driver
Some of the error code of VNX CLI is scattered in the code and is
not coded as a constant.
Gather all VNX error codes in the VNXError class and compose a new
method to check the error in the CLI output. If no specific error
is supplied, check all available errors in the output.
Yuriy Nesenenko [Tue, 19 May 2015 15:17:01 +0000 (18:17 +0300)]
Filter snapshots data on the DB side
It filters snapshots data on the DB side to improve performance.
Some tests are removed from test_snapshots.py because their function
implemented on the DB side.
PranaliDeore [Tue, 26 May 2015 07:12:27 +0000 (00:12 -0700)]
Change generic NotFound to specific exception
1. Changed exception type from NotFound to specific NotFound
exception for better readability.
e.g. VolumeNotFound, SnapshotNotFound etc.
2. Error messages returned to the user are not consistent across
all apis in case of all exceptions derived from NotFound exception.
Instead of returning custom error messages return message
defined in exception class itself.
NOTE: Doesn't applied to 'HostBinaryNotFound' as host information to
the tenant should not be leaked.
3. Added explanation to HTTPNotFound wherever appropriate.
e.g. raise webob.exc.HTTPNotFound() to
raise webob.exc.HTTPNotFound(explanation=err.msg)
Jay S. Bryant [Wed, 24 Jun 2015 19:35:38 +0000 (14:35 -0500)]
Fix library includes for config generator
Noticed today while working on fixing issues
with the config generator that oslo_policy
had been added in the wrong location for
pulling the config options in to the config
generator.
This fix moves the items that had been put in
oslo.config.generator.rc but were missing in generate_sample.sh
over to the appropriate location. I also removed the option for
importing additional libraries in oslo.config.generator.rc
given that it will not work.
I am still working on a longer term solution, but
we really should get this fixed so we can generate sample
config files again.
Anton Arefiev [Wed, 24 Jun 2015 15:45:35 +0000 (18:45 +0300)]
Fix cinder-manage volume delete cmd
Since pools was introduced in cinder, field 'host' in resources
contains pool part. So when we are trying to delete volume with
cinder-manage, 'host' with pool part passed to rpc target. As
result message isn't delivered to host, and we can't delete volume.
Victor Stinner [Tue, 23 Jun 2015 12:08:25 +0000 (14:08 +0200)]
Fix Python 3 issues in the blockbridge driver
* Replace httplib import with six.moves.http_client
* Replace urllib imports with six.moves.urllib, update urllib.
For example, replace urllib.quote() with urllib.parse.quote().
* test_blockbridge: try get the mock module from unittest.mock, part of
the Python stdlib since Python 3.3. The third-party mock module has a
bug on Python 3.4.
* On Python 3, base64.encodestring() expects bytes and returns bytes.
Encode credentials to UTF-8 and decode base64 from ASCII to get the
final string as Unicode on Python 3.
* test_cfg_api_auth_scheme_password: parse the connection URL to compare
it. Comparing directly the URL fails on Python 3 because URL
parameters are rendered in a random order because of the hash
randomization.
* Replace pools.values()[0] with list(pools.values())[0]. On Python 3,
dict.values() returns an iterator.
* tox.ini: add cinder.tests.unit.test_blockbridge to Python 3.4.
Victor Stinner [Tue, 23 Jun 2015 11:37:42 +0000 (13:37 +0200)]
Fix Python 3 issues in the swift backup driver
* On Python 3, encode/decode JSON to/from UTF-8
* Use byte strings for volume content
* Replace dict.keys()[0] and dict.value()[0] with list(dict.items())[0].
Get the key and the value at once to ensure that they are consistent.
On Python 3, items() returns an iterator: create a list to use the [0]
operator.
* SwiftBackupDriver: use a bytearray instead of a string because
bytes += bytes is inefficient on Python 3, whereas str += str is
optimized on Python 2. Replace also StringIO with BytesIO.
* BytesIO has no more len attribute on Python 3: get the length of the
content instead.
* Replace buffer(bytearray(128)) with b'\0' * 128 to create a string of
128 zeroed bytes
* Replace "rw" file mode with "w+" ("rw" raises an exception
on Python 3).
* tox.ini: add the following tests to Python 3.4
Victor Stinner [Tue, 23 Jun 2015 09:55:40 +0000 (11:55 +0200)]
Fix Python 3 issues in ceph and rbd drivers
* Add cinder.utils.convert_str() function: helper to convert a string to
a native string: convert to bytes on Python 2, convert to Unicode on
Python 3. Function needed by ceph and rbd drivers.
* Replace encodeutils.safe_encode() with utils.convert_str()
* test_backup_ceph: get the size of the volume file using os.fstat() to
fix a test. Before the volume size was a mock, but comparison between
integers and mocks raise a TypeError on Python 3.
* Fix RBDDriver._update_volume_stats(): replace a/b with a//b to get an integer
on Python 2 and Python 3 (a/b always returns a float on Python 3)
* Replace map() with a list-comprehension in RBDDriver because a list is
expected. On Python 3, map() returns an iterator.
* tox.ini: add to following tests to Python 3.4
Victor Stinner [Tue, 23 Jun 2015 09:20:47 +0000 (11:20 +0200)]
Fix Python 3 issues in backup
* Replace (int, long) with six.integer_types: the "long" type has been
removed in Python 3
* Replace str.encode("bas64") with base64.encodestring(str), base64
text codec has been removed in Python 3. Same change for decode
(base64.decodestring)
* On Python 3, encode JSON to UTF-8
* tox.ini: add the following tests to Python 3
git-harry [Wed, 24 Jun 2015 08:46:45 +0000 (09:46 +0100)]
Remove generate_glance_url
The function generate_glance_url is part of the inital fork from nova.
It is only called once and that is by a unit test designed to test the
function.
This commit removes generate_glance_url and its associated test because
they are not used by cinder.
Michal Dulko [Tue, 23 Jun 2015 13:47:53 +0000 (15:47 +0200)]
Harden scheduler.rpcapi unit tests
Tests for scheduler.rpcapi are very weak. Even changing keyword
arguments names passed to call and cast methods in this module
doesn't trigger unit tests failures. This commit adds keyword
arguments validation to prevent problems similar to ones caused
by bug 1467841 in backup.rpcapi. I've also fixed argument order
in assertEqual usages there.
Michal Dulko [Tue, 23 Jun 2015 13:42:00 +0000 (15:42 +0200)]
Fix backups.rpcapi to pass objects over RPC
Commit Icff37261b367463b71a1268be16f9c97f595bf0c brought bugs in
backup.rpcapi. For export_record, import_record and reset_status
backup.id is passed instead of whole backup object - which
backup.manager expects. This causes severe failures. This commit
changes the rpcapi to pass objects and adds missing
backup.test_rpcapi unit tests that will prevent such failures in
the future.
wanghao [Tue, 26 May 2015 09:25:26 +0000 (17:25 +0800)]
Fix weird change of volume status in re-scheduling
When create a volume failed as some exception, cinder will re-schedule the
creation request, the retry number is 3 in default.
But when re-scheduling, this volume's status was changed like this:
This is weird to user, since if this volume is in re-scheduling process,
it's status should always be creating, and then become error or available
at last.
Victor Stinner [Tue, 23 Jun 2015 09:11:04 +0000 (11:11 +0200)]
Fix tox -e py34
Add a py34 test environment to tox.ini running a subset of tests which
pass on Python 3.4. With this change, the py34 check job should pass and
so it will become possible to make the job voting to avoid further
Python 3 regressions.
Code Review / openstack-build / cinder-build.git / log