xiaolei hu [Thu, 25 Jun 2015 02:22:50 +0000 (10:22 +0800)]
Separate FlashSystem FC and iSCSI common code
The patch is mainly to split original FC driver code into common
code and FC driver for IBM FlashSystem. iSCSI driver merged in L-1
has been inherited from original FC driver for common functions
already.
In this patch,
* Separate FC driver into new driver and common code.
* Modify iSCSI driver to inherite the common code.
* Common function _get_node_data in iSCSI driver is moved into
common code.
* Add locks in initialize_connection and terminate_connection
for iSCSI driver.
* Remove flashsystem_multipath_enabled to use Cinder iSCSI
multipath support.
Alberto Murillo [Tue, 30 Jun 2015 21:12:55 +0000 (16:12 -0500)]
Update expected error message from lvs
drivers/lvm.py uses _volume_not_present() to avoid deleting a snapshot
when it does not exist in the lvm backend. This functions uses brick/
local_dev/lvm.py get_lv_info() which expects "not found" as error message.
Since LVM2 2.102.112, The error message returned by lvs changed
From:
One or more specified logical volume(s) not found
To:
Failed to find logical volume \"%s/%s\"
Vincent Hou [Thu, 7 May 2015 05:53:25 +0000 (13:53 +0800)]
Implement the update_migrated_volume for the drivers
This patch implements update_migrated_volume for LVM, Storwize
and updates Dell, StorPool and Infortrend drivers accordingly. It
makes sure that after a successful volume migration, the cinder
volume name(id) is the same as the backend volume name(id). Other
back-end drivers can take this patch as a reference to implement
update_migrated_volume.
PS: Not applicable to multi-attached volumes, since we need to wait
until the multi-attach lands in Nova.
This patch also adds a unit test for the StorPool driver's
update_migrated_volume() implementation.
Co-Authored-By: Peter Penchev <openstack-dev@storpool.com>
Change-Id: I69707340ddf2b55286ff0d84319529b2f502cefa
Partial-Bug: #1450649
Victor Stinner [Tue, 30 Jun 2015 14:47:15 +0000 (16:47 +0200)]
Fix Python 3 issues in cmd
* Replace filter() with a list-comprehension using if to get a list on
Python 3.
* Get the mock module from the stdlib unittest module on Python 3.3 and
newer, or fallback to the third-party mock module.
* Replace __builtin__ with six.moves.builtins.
* tox.ini: add the following tests for Python 3.4
Victor Stinner [Tue, 30 Jun 2015 14:06:28 +0000 (16:06 +0200)]
Port image/glance.py to Python 3
* Fix usage of six.reraise(): new_exc is the exception value, not the
exception type
* Replace __builtin__ with six.moves.builtins.
* test_extracting_v2_boot_properties(): set config.glance_num_retries to
fix the test on Python 3 (comparison between mock and int now raises
a TypeError on Python 3).
* TestGlanceSerializer: use dictionaries with only one key to have a
reliable output even if the hash is randomized (hash randomization is
now enabled by default in Python 3).
* tox.ini: add cinder.tests.unit.image.test_glance to Python 3.4.
Commit Ibaf43858d60e1320c339f2523b5c09c7f7c7f91e caused new problem with
cross thread communication. According to Python documentation, code can
lead to a deadlock if the spawned thread directly or indirectly attempts
to import a module. python-rados spawns new thread to connect to
cluster. So I removed new spawning new thread to connect to rados. All
long-running operations calls whith python-rbd are still implemented in
native Python threads to block eventlet loop.
Victor Stinner [Mon, 29 Jun 2015 13:33:12 +0000 (15:33 +0200)]
Port dothill to Python 3
* Replace urllib2 with six.moves.urllib
* On Python 3, encode string to UTF-8 to hash it using MD5
* On Python 3, decode base64 from ASCII to get Unicode
* Replace "pattern in exc" with "pattern in exc.args"
* test_initialize_connection(): fix get_active_iscsi_target_portals()
mock. Use the return_value attribute to return a dictionary, instead
of returning a single IP address.
* tox.ini: add cinder.tests.unit.test_dothill to Python 3.4
Michal Dulko [Wed, 1 Jul 2015 06:41:03 +0000 (08:41 +0200)]
Mock socket.gethostbyaddr in test_v7000_fcp
Currently test_v7000_fcp doesn't mock socket.gethostbyaddr and
hardcodes expected response to be google-public-dns-a.google.com
(as it is querying for 8.8.8.8 as IP). This introduces dependency on
external DNS lookup to pass unit tests and causes problems with
running tests behind a proxy.
This commit mocks socket.gethostbyaddr in the problematic test.
Dmitry Guryanov [Wed, 10 Jun 2015 20:01:55 +0000 (23:01 +0300)]
smbfs: fix invalid check for smbfs_used_ratio correctness
The check allowed to specify values greater than 1
because of a error in condition. So I've fixed the
condition and updated unit test, so it would catch
the bug.
Victor Stinner [Tue, 30 Jun 2015 14:14:09 +0000 (16:14 +0200)]
Fix Python 3 issues in targets unit tests
* Replace StandardError with ZeroDivisionError: StandardError was
removed in Python 3.
* Replace __builtin__ with six.moves.builtins.
* Open text mode with "w" mode (instead of "wb").
* tox.ini: add targets unit tests to Python 3.4
Note: cinder.tests.unit.targets.test_iet_driver still fails on Python 3,
it will be fixed in a different change.
Victor Stinner [Mon, 29 Jun 2015 13:53:44 +0000 (15:53 +0200)]
Port drbdmanagedrv driver to Python 3
* Replace map() with a list-comprehension where a list is expected
* Replace apply(fn, args) with fn(*args)
* Use str.replace() to remove "{" and "}" characters in _clean_uuid(),
instead of using str.translate()
* Use literal syntax to create a new dictionary instead of using
dict() + dict.items()
* tox.ini: add cinder.tests.unit.test_drbdmanagedrv to Python 3.4
Victor Stinner [Mon, 29 Jun 2015 13:11:12 +0000 (15:11 +0200)]
Port test_db_api to Python 3
* _dict_from_object(): on dictionaries, call the items() method instead
of iteritems() method. oslo.db objects have no items() method yet, so
check the object type to decide which method should be called.
* test_db_api: fix usage of db.quota_reserve(), quotas dictionary values
must be integers (hard limit of quotas), not Quota instances of
cinder.db.sqlalchemy.models.
* tox.ini: add the following tests to Python 3.4
Victor Stinner [Mon, 29 Jun 2015 20:20:16 +0000 (22:20 +0200)]
Port hitachi driver to Python 3
* On Python 3, shlex expects Unicode, not bytes: don't encode
* Replace map() with list-comprehension where a list is expected.
* tox.ini: add cinder.tests.unit.test_hitachi_hbsd_horcm_fc to Python 3.4
wanghao [Tue, 23 Jun 2015 08:38:29 +0000 (16:38 +0800)]
Do not allow to modify access for public volume type
Now public volume type can be added/removed project access, but when
listing access, cinder returns 'Access list not available for public
volume types.' It's weird for users experience.
We should check if a type is public and do not allow public volume type
to modify project access.
APIImpact
When modifying access for public volume type, API will return 400 and message:
"Invalid volume type: Type access modification is not applicable to public
volume type."
Kazumasa Nomura [Thu, 25 Jun 2015 18:33:16 +0000 (03:33 +0900)]
Fix cinder.conf.sample generation
Introduction of the hp_xp driver broke cinder.conf.sample generation due to
the way the driver was registering the configuration items. Originally the
options were being included into the FC driver file and registered there.
When the config generator found the options in the separate hp_xp_opts.py
file it did not know what group to include the options in since they were
not registered at all.
This fix moves registration of the options to the hp_xp_opts.py file so
that the config generation will work. Since cfg.CONF is a global, the
hp_xp driver will still have access to the options in there after they are
set by importing the opts file. An additional problem that was uncovered
when creating this fix is the fact that the old config generator we are
using from oslo-incubator doesn't support using cfg.Opt. To temporarily
workaround this I have changed the cfg.Opts to be cfg.IntOpts or
cfg.ListOpts(as appropriate) and documented the valid values for those
options.
The last change that was necessary was to add config values for several of
the options in cinder/tests.py as the options were marked as required by
the driver. If values were not set this caused oslo config to error out
reporting that they values needed to be set.
Change-Id: Ie47d38c2f8fde675cf00ea2ec9abf60691c57ef9
Closes-Bug: 1466198 Signed-off-by: Jay S. Bryant<jsbryant@us.ibm.com>,
Kazumasa Nomura <kazumasa.nomura.rx@hitachi.com>
Xi Yang [Tue, 23 Jun 2015 09:42:18 +0000 (05:42 -0400)]
Fix timeout issue in EMC VNX driver unit test
This patch is going to fix the timeout issue in EMC VNX
driver unit test.
Currently, the timeout value of a looping call in EMC unit test
is set to 0.000002s which is too small. The execution time of
a looping call may exceed this timeout value and cause the test
case failure. This fix is changing the timeout to the default
value of VNX driver.
Tom Barron [Thu, 11 Jun 2015 17:07:07 +0000 (13:07 -0400)]
Remove oslo logging from backup unit tests
Many of the unit tests for backup services import oslo_log and
invoke LOG.debug, LOG.info, etc. There appears to be no current
reason to do this and, more generally, unit tests for everything
but oslo_log itself should not be doing logging themselves.
This commit removes oslo logging from the backup unit tests.
Closes-bug: 1464772
Jay S. Bryant [Fri, 26 Jun 2015 20:56:47 +0000 (15:56 -0500)]
Sync the latest fileutils module from oslo-incubator
As part of our normal development in a release we should get
our openstack/common directory code synced up with the latest
code in oslo-incubator. This is the sync for fileutils:
Current HEAD in OSLO:
---------------------
commit 2e41adeb4c19114fd3504eb69891e9149dba7072
Date: Wed Jun 24 14:45:25 2015 +0000
Updated from global requirements
Changes merged with this patch:
--------------------- 16eb642eb -- Clean up logging to conform to guidelines
Tom Barron [Wed, 17 Jun 2015 18:54:01 +0000 (14:54 -0400)]
Fix 'no actual-pathname' NetApp API error
Cinder volume logs sometimes show this error
NetApp API failed. Reason - 13114:
No actual-pathname for 10.1.0.9:/vol/whatever
with Kilo code and 7-mode DOT system.
The issue is due to our Cinder driver passing the
entire share to the relevant API instead of just
the export path portion of the share. This only
happens when the NFS image cache is full, and cached
files need to be removed.
Sergey Vilgelm [Fri, 26 Jun 2015 14:35:11 +0000 (16:35 +0200)]
Use right oslo.service entry points
Use oslo.service.service and periodic_task entry points to generate
a sample configuration file. oslo.service library doesn't have
an oslo.service entry point.
Cedric Zhuang [Thu, 21 May 2015 11:26:23 +0000 (04:26 -0700)]
Use symbol for error code in VNX cinder driver
Some of the error code of VNX CLI is scattered in the code and is
not coded as a constant.
Gather all VNX error codes in the VNXError class and compose a new
method to check the error in the CLI output. If no specific error
is supplied, check all available errors in the output.
Yuriy Nesenenko [Tue, 19 May 2015 15:17:01 +0000 (18:17 +0300)]
Filter snapshots data on the DB side
It filters snapshots data on the DB side to improve performance.
Some tests are removed from test_snapshots.py because their function
implemented on the DB side.
PranaliDeore [Tue, 26 May 2015 07:12:27 +0000 (00:12 -0700)]
Change generic NotFound to specific exception
1. Changed exception type from NotFound to specific NotFound
exception for better readability.
e.g. VolumeNotFound, SnapshotNotFound etc.
2. Error messages returned to the user are not consistent across
all apis in case of all exceptions derived from NotFound exception.
Instead of returning custom error messages return message
defined in exception class itself.
NOTE: Doesn't applied to 'HostBinaryNotFound' as host information to
the tenant should not be leaked.
3. Added explanation to HTTPNotFound wherever appropriate.
e.g. raise webob.exc.HTTPNotFound() to
raise webob.exc.HTTPNotFound(explanation=err.msg)
Jay S. Bryant [Wed, 24 Jun 2015 19:35:38 +0000 (14:35 -0500)]
Fix library includes for config generator
Noticed today while working on fixing issues
with the config generator that oslo_policy
had been added in the wrong location for
pulling the config options in to the config
generator.
This fix moves the items that had been put in
oslo.config.generator.rc but were missing in generate_sample.sh
over to the appropriate location. I also removed the option for
importing additional libraries in oslo.config.generator.rc
given that it will not work.
I am still working on a longer term solution, but
we really should get this fixed so we can generate sample
config files again.
Anton Arefiev [Wed, 24 Jun 2015 15:45:35 +0000 (18:45 +0300)]
Fix cinder-manage volume delete cmd
Since pools was introduced in cinder, field 'host' in resources
contains pool part. So when we are trying to delete volume with
cinder-manage, 'host' with pool part passed to rpc target. As
result message isn't delivered to host, and we can't delete volume.
Victor Stinner [Tue, 23 Jun 2015 12:08:25 +0000 (14:08 +0200)]
Fix Python 3 issues in the blockbridge driver
* Replace httplib import with six.moves.http_client
* Replace urllib imports with six.moves.urllib, update urllib.
For example, replace urllib.quote() with urllib.parse.quote().
* test_blockbridge: try get the mock module from unittest.mock, part of
the Python stdlib since Python 3.3. The third-party mock module has a
bug on Python 3.4.
* On Python 3, base64.encodestring() expects bytes and returns bytes.
Encode credentials to UTF-8 and decode base64 from ASCII to get the
final string as Unicode on Python 3.
* test_cfg_api_auth_scheme_password: parse the connection URL to compare
it. Comparing directly the URL fails on Python 3 because URL
parameters are rendered in a random order because of the hash
randomization.
* Replace pools.values()[0] with list(pools.values())[0]. On Python 3,
dict.values() returns an iterator.
* tox.ini: add cinder.tests.unit.test_blockbridge to Python 3.4.
Victor Stinner [Tue, 23 Jun 2015 11:37:42 +0000 (13:37 +0200)]
Fix Python 3 issues in the swift backup driver
* On Python 3, encode/decode JSON to/from UTF-8
* Use byte strings for volume content
* Replace dict.keys()[0] and dict.value()[0] with list(dict.items())[0].
Get the key and the value at once to ensure that they are consistent.
On Python 3, items() returns an iterator: create a list to use the [0]
operator.
* SwiftBackupDriver: use a bytearray instead of a string because
bytes += bytes is inefficient on Python 3, whereas str += str is
optimized on Python 2. Replace also StringIO with BytesIO.
* BytesIO has no more len attribute on Python 3: get the length of the
content instead.
* Replace buffer(bytearray(128)) with b'\0' * 128 to create a string of
128 zeroed bytes
* Replace "rw" file mode with "w+" ("rw" raises an exception
on Python 3).
* tox.ini: add the following tests to Python 3.4
Victor Stinner [Tue, 23 Jun 2015 09:55:40 +0000 (11:55 +0200)]
Fix Python 3 issues in ceph and rbd drivers
* Add cinder.utils.convert_str() function: helper to convert a string to
a native string: convert to bytes on Python 2, convert to Unicode on
Python 3. Function needed by ceph and rbd drivers.
* Replace encodeutils.safe_encode() with utils.convert_str()
* test_backup_ceph: get the size of the volume file using os.fstat() to
fix a test. Before the volume size was a mock, but comparison between
integers and mocks raise a TypeError on Python 3.
* Fix RBDDriver._update_volume_stats(): replace a/b with a//b to get an integer
on Python 2 and Python 3 (a/b always returns a float on Python 3)
* Replace map() with a list-comprehension in RBDDriver because a list is
expected. On Python 3, map() returns an iterator.
* tox.ini: add to following tests to Python 3.4
Victor Stinner [Tue, 23 Jun 2015 09:20:47 +0000 (11:20 +0200)]
Fix Python 3 issues in backup
* Replace (int, long) with six.integer_types: the "long" type has been
removed in Python 3
* Replace str.encode("bas64") with base64.encodestring(str), base64
text codec has been removed in Python 3. Same change for decode
(base64.decodestring)
* On Python 3, encode JSON to UTF-8
* tox.ini: add the following tests to Python 3
Code Review / openstack-build / cinder-build.git / log