Victor Stinner [Mon, 25 May 2015 15:08:00 +0000 (17:08 +0200)]
Replace xrange() with six.moves.range()
Add "from six.moves import range" to replace the builtin range()
function with six.moves.range() to always create an iterator, instead of
creating a temporary list.
Replace "xrange" with "range".
Don't add the import for ranges of 1024 items or less.
wanghao [Thu, 14 May 2015 07:42:44 +0000 (15:42 +0800)]
Introduce Guru Meditation Reports into Cinder
This commit integrates functionality from the
`openstack.common.report` module into Cinder.
This enables Cinder services to receive SIGUSR1
and print a Guru Meditation Report to stderr.
The required modules were added to
'openstack-common.conf' as well.
It is essentially a copy from implementation of
nova side.
Co-Authored-By: Gauvain Pocentek<gauvain.pocentek@objectif-libre.com> Co-Authored-By: Walter A. Boring IV<walter.boring@hp.com>
Change-Id: Iaf70ade96d4ad4234cc7e88277ed7b52cf458c2a
Kurt Martin [Thu, 11 Jun 2015 20:29:11 +0000 (13:29 -0700)]
Update 3PAR user config help strings
The 3PAR drivers no longer requires a user to have the 'super'
access role. This patch simply updates the help strings for the
3PAR username and password to reflect that an 'edit' role will
be sufficient.
Victor Stinner [Fri, 12 Jun 2015 11:28:38 +0000 (13:28 +0200)]
Fix Python 3 issues in utils
* Replace string.split(key_file, ',') with key_file.split(','):
the string.split() function was removed in Python 3, use the
str.split() method instead
* Fix conf_fixture: import backup_enable_progress_timer and
backup_swift_enable_progress_timer options. This change is required
to run tests using testtools (it's now possible yet to run tests on
Python 3 using testr).
* hash_file(): use b'' (bytes) for the sentinel, not '' (unicode), when
reading the input binary file
* test_hash_file() requires a binary file (bytes), not a text file
(unicode)
* Get the builtin open() function from six.moves.builtins instead of
getting it from __builtin__ to mock the open() function. The
__builtin__ module was renamed to builtins in Python 3.
* sanitize_hostname(): on Python 3, decode back the hostname from Latin1
to work on a native string (Unicode)
* replace reduce() with six.moves.reduce(): the Python 2 builtin
reduce() function was moved to the functools module in Python 3
* convert_version_to_str(): fix integer division, use a//b, not a/b
* convert_version_to_str(): replace reduce() with ''.join() with map(str)
Mitsuhiro Tanino [Sat, 13 Jun 2015 04:23:40 +0000 (00:23 -0400)]
Wait until service thread is done on service stop
In order to be able to perform graceful shutdown of services, we need
to wait for all the current service threads to finish before exiting
from the service process.
Oslo.service provides this facility through it's wait() method, so we
need to call it in Cinder Service stop() method which gets called when
graceful shutdown is requested by sending the process SIGINT or SIGTERM.
This patch enables the reporting of the
multiattach capability in get_volume_stats.
The scheduler currently filters on the capability
for drivers that support/don't support that ability
to do more than one attachment for a volume.
By reporting multiattach = True, it lets the scheduler
know this driver allows attaching a volume more than once.
Victor Stinner [Fri, 12 Jun 2015 12:55:37 +0000 (14:55 +0200)]
Replace dict(obj.iteritems() with dict(obj)
Since oslo.db 1.10, it's possible to convert directly a SQLAlchemy
object to a dictionary using dict(obj) instead of having to write
dict(obj.iteritems()).
dict(obj) works also on regular dictionaries to duplicate a dictionary.
Victor Stinner [Fri, 12 Jun 2015 12:53:05 +0000 (14:53 +0200)]
Replace dict.iteritems() with dict.items()
This change adds Python 3 compatibility to the modified code.
Replace also six.iteritems(obj) with obj.items().
The iteritems() method of Python 2 dictionaries was renamed to items()
on Python 3. As discussed on the openstack-dev mailing list, iteritems()
must be replaced with items(), six.iteritems() should not be used. In
OpenStack, the overhead of creating a temporary list with dict.items()
on Python 2 is negligible.
Enables native support for HGST Solutions software Spaces as
Cinder volumes or snapshots.
Each Cinder Volume or Snapshot is mapped to a single HGST Space.
This space may be named differently from the actual volume/snap ID
and so we store the Space Name<->ID mapping in the volume provider_id.
Snapshots are not supported with the current HGST Solutions
software, so they are implemented as heavyweight copies in the
driver.
All Spaces are made visible on the Cinder host for speed of access,
and only the spaces requested by Nova instances are actually made
visible on other members of the cluster.
Not all nodes need SSD storage to take advantage of these volumes,
cinder.conf entries specify which nodes share their local SSDs.
Prerequisites:
HGST Solutions is a software-SAN-like package which allows local
SSDs in a cluster to be combined into a single storage pool.
The driver has a series of configuration options which must be set
in the cinder.conf, prefixed with hgst_*.
Nodes should have the HGST software installed and connected to the
HGST domain prior to rolling out Nova nodes using this storage.
Additional patches required for full functionality (being tracked
under the same blueprint):
OS-brick patch @ https://review.openstack.org/#/c/186588/
Nova patch @ https://review.openstack.org/#/c/186594/
Nova patch required until os-brick<->nova connection finalized
in https://review.openstack.org/#/c/175569/
Daniel Wilson [Wed, 10 Jun 2015 21:32:49 +0000 (14:32 -0700)]
Volume manager should set filter_function and goodness_function
If a user defines a filter_function or goodness_function in their cinder.conf
file the scheduler should use that function. This change appends those
functions to volume_stats if the driver has not specified them.
Victor Stinner [Wed, 10 Jun 2015 12:55:52 +0000 (14:55 +0200)]
Replace it.next() with next(it) for py3 compat
The Python 2 next() method of iterators was renamed to __next__() on
Python 3. Use the builtin next() function instead which works on Python
2 and Python 3.
This patch was generated by the next operation of the sixer tool:
https://pypi.python.org/pypi/sixer
Manual changes:
* cinder/tests/unit/test_cmd.py: use "tpgs = iter([tpg])" instead of
using a magic mock for tpgs
This patch was generated by the six_moves operation of the sixer tool
version 0.4:
https://pypi.python.org/pypi/sixer
Manual changes:
* Disable pylint warnings on the HTTPSConnection classes because pylint
doesn't support importing from six.moves yet, see:
https://bitbucket.org/logilab/pylint/issue/550/
Alex Meade [Fri, 1 May 2015 16:56:48 +0000 (12:56 -0400)]
NetApp E-Series: Add Fibre Channel Support
The NetApp driver for E-series product lines currently support iSCSI. This
patch adds Fibre Channel support to the E-Series driver. This driver reuses
the same functionality as the E-Series iSCSI driver except for the
initialize_connection and terminate_connection driver methods which make
full use of Cinder's FibreChannel zone manager.
Alex Meade [Tue, 28 Apr 2015 01:19:36 +0000 (21:19 -0400)]
NetApp E-Series: Refactor class structure for FC
This patch moves classes to mimic the NetApp ONTAP drivers class
structure by creating a library module, making the iSCSI driver
a thin layer. This way the E-Series FC driver can be added in a
similar manner.
Alex Meade [Sun, 15 Feb 2015 22:01:54 +0000 (17:01 -0500)]
NetApp E-Series driver: Remove caching logic
This patch removes unnecessary API caching logic from the NetApp
E-Series driver. The cache, represented as the _objects dictionary,
is refreshed nearly every API call and therefore provides no value.
This refactor is also preparation for refactoring the class structure
so we can easily add an E-Series Fibre Channel driver.
John Griffith [Mon, 1 Jun 2015 20:04:15 +0000 (20:04 +0000)]
Add secondary account capability to SolidFire
The SolidFire Cluster has a limit of 2K volumes per SolidFire
account. Given that the SolidFire account maps directly to the
OpenStack tenant this can be a limitation for some large scale
deployments that may have tenants that wish to create more than
2K volumes in their OpenStack cloud.
This patch introduces the concept of a secondary account for
an OpenStack tenant on the SolidFire backend.
If an OpenStack tenant has more than 2K volumes, and wishes to
create additional volumes, we can create a secondary/supplemental
SolidFire account that is owned by the same OpenStack tenant.
If they wish to go beyond 4K volumes... well, SolidFire V3
driver will deal with that, for now 4K is a line in the sand.
While working on unit tests, we again move to using mock
opportunistically. In addition, a few tests that either
weren't actually testing anything, or were just a fake-loop
that didn't actually work have been removed:
NOTE:
This patch is a temporary for backporting purposes, we are
still planning a refresh in Liberty of the driver.
Victor Stinner [Tue, 9 Jun 2015 13:37:02 +0000 (15:37 +0200)]
Replace urllib and urllib2 with six.moves.urllib
The Python 2 urllib and urllib2 modules changed a lot in Python 3. On
Python 3, the urllib functions are now grouped into submodules of
urllib: urllib.parse, urllib.error, etc. The urllib2 has been removed.
Replace urllib and urllib2 with six.moves.urllib to make the Cinder code
compatible with Python 2 and Python 3.
The new code now looks like Python 3 code (ex: "urllib.parse.urlsplit"),
but it also works on Python 2.
Replace also "import six.moves.urllib.parse as urlparse" with "from
six.moves import urllib" to uniformize the code:
"urlparse.urlsplit(href)" becomes "urllib.parse.urlsplit(href)".
This patch was generated by the urllib operation of the sixer tool version 0.4:
https://pypi.python.org/pypi/sixer
Manual changes:
* Add a try/except ImportError to get the parse_http_list function
in cinder/api/urlmap.py
* Import the socket module instead of using urllib2.socket
in cinder/volume/drivers/huawei/rest_common.py
* test_nimble.py: replace NIMBLE_URLLIB2 value with
'six.moves.urllib.request'
* Remove "from six.moves.urllib import parse as urlparse", it was
replaced with urllib.parse from six.moves.
* Reformat to respect the 80 columns constraint
Victor Stinner [Wed, 10 Jun 2015 12:39:37 +0000 (14:39 +0200)]
Replace unicode with six.text_type
The "unicode" type was renamed to "str" in Python 3. Use six.text_type
to make Cinder compatible with Python 3.
The initial patch was generated by the unicode operation of sixer tool:
https://pypi.python.org/pypi/sixer
Manual changes:
* cinder/volume/drivers/san/hp/hp_3par_iscsi.py:
replace "isinstance(value, str) or isinstance(value, unicode)"
with "isinstance(value, six.string_types)"
* cinder/volume/drivers/san/hp/hp_3par_iscsi.py:
bump the version to 2.0.17
* Revert changes on hacking: hacking explicitly searchs the "unicode"
pattern for Python 2. Hacking may require more work to be ported to
Python 3.
* Revert changes on docstrings and strings
Gorka Eguileor [Wed, 10 Jun 2015 09:57:07 +0000 (11:57 +0200)]
Use correct rtslib namespace for newer versions
For rtslib-fb versions 2.1.52 and newer its own namespace (rtslib_fb) is
recommended instead of the older one (rtslib).
On versions with this new namespace a deprecation warning will be
thrown: "'rtslib' package name for rtslib-fb is deprecated, please
instead import 'rtslib_fb'"
This patch uses this new namespace when possible and falls back to the
older namespace if not.
Jay S. Bryant [Tue, 9 Jun 2015 22:10:18 +0000 (17:10 -0500)]
Dispose DB connections between backend proc starts
Currently we do not dispose the existing DB pools before
forking off additional storage backend processes. This can
result in timing issues and DB errors.
This change simply ensures that we do a dispose_engine() in
the loop where we are forking off processes for the storage
backends.
Tom Swanson [Tue, 2 Jun 2015 19:02:33 +0000 (14:02 -0500)]
Dell SC: Expanded comments and update var names
Additional comments have been added to help explain what is happening
in the driver. This revolves mostly around dell_storagecenter_api.py
which is not an obvious file for a reviewer to review. Hopefully
additional comments will aid this.
Updated single character variable names except for r which is the
REST calls return throughout the code by convention.
Reworked create_volume for clarity. No functional changes.
Changed two obvious error messages from LOG.debug to LOG.error.
Jeremy Stanley [Tue, 9 Jun 2015 17:37:43 +0000 (17:37 +0000)]
Merge tag '2015.1.0'
This is a null-merge of the 2015.1.0 release tag back into the master
branch so that the 2015.1.0 tag will appear in the git commit history of
the master branch. It contains no actual changes to the master branch,
regardless of how our code review system's UI represents it. Please
ask in #openstack-infra if you have any questions, and otherwise try
to merge this as quickly as possible to avoid later conflicts on the
master branch.
Jeremy Stanley [Tue, 9 Jun 2015 16:31:41 +0000 (16:31 +0000)]
Merge tag '2014.2'
This is a null-merge of the 2014.2 release tag back into the
master branch. It contains no actual changes to the master branch,
regardless of how our code review system's UI represents it. Please
ask in #openstack-infra if you have any questions, and otherwise try
to merge this as quickly as possible to avoid later conflicts on the
master branch.
Declare the existing StorPool volume driver functionality by
deriving from the appropriate driver.*VD classes.
6. Implement volume retyping and migrations
initialize_connection(): store the StorPool client ID in the connection
data to be used by the "StorPool" Brick connector.
validate_connector(): actually check the connector data for a valid
hostname defined in the StorPool configuration.
_storpool_client_id(): new method to get the numeric ID of a host in
the StorPool cluster from the /etc/storpool.conf file.
_attach_volume(), _detach_volume(): if the "remote" flag is set, pass
the request on to the default Cinder driver implementation; it knows
how to make remote volumes accessible.
_attach_volume(), _detach_volume(): support the case when these methods
are invoked from a method that is not overridden in the StorPool Cinder
driver - create a storpool.spopenstack.AttachDB attachment structure
and destroy it upon detaching.
ensure_export(): add a stub method, the actual work is handled by
the Nova volume attachment driver.
retype(): new method: handle a StorPool-to-StorPool retyping where
at most the volume's StorPool template or replication is changed.
For anything else, return False to indicate an unsupported request so
that a full volume migration is triggered.
update_migrated_volume(): rename the StorPool volume to correspond to
the real Cinder volume's name.
Add the StorPoolConfigurationMissing and StorPoolConfigurationInvalid
exceptions.
Flesh out the validate_connector() and initialize_connection() unit tests.
Edwin Wang [Sat, 13 Dec 2014 20:42:09 +0000 (04:42 +0800)]
Add iSCSI protocol support for IBM FlashSystem
This change implements iSCSI protocol for Cinder driver.
It is inherited from driver.ISCSIDriver and flashsystem.py.
We will separate flashsystem.py into common and FC protocol.
iSCSI test result:
https://bugs.launchpad.net/cinder/+bug/1406442
Kurt Martin [Mon, 8 Jun 2015 17:26:20 +0000 (10:26 -0700)]
Fixes 3PAR snapshot failure with optional params
Create snapshot fails when HP3PAR hp3par_snapshot_retention and/or
hp3par_snapshot_expiration parameters were set in cinder.conf. The
params were being sent to the backend as strings instead on integers.
peter_wang [Fri, 29 May 2015 00:38:56 +0000 (20:38 -0400)]
Add 'source-id' and 'source-name' support in VNX driver
Currently, VNX driver only supports 'id' as the id-type in manage/unmanage volume.
With release of new version python-cinderclient, 'source-id' and 'source-name' are
preferred id-type when using cinder manage.
This fix is to add 'source-id' and 'source-name' support
and obsolete the self-defined 'id' support.
Tom Swanson [Wed, 13 May 2015 17:00:54 +0000 (12:00 -0500)]
Dell SC: Added support for alternate iscsi portals
Added support for returning alternate target portals,
luns and iqns in a non multipath setup. This was done
per https://review.openstack.org/#/c/140877/ Tests
were modified to support this change.
The iscsi initialize_connection and find_iscsi_properties
functions took the brunt of this. The full list of portals,
luns and iqns are returned along with the preferred portal,
lun and iqn rather than a single iqn, lun and portal.
Some minor log message changes for clarity. (Removed the
one character vars.)
Code Review / openstack-build / cinder-build.git / log