2b966f9 Fix deletion of cached file for policy enforcer 8202a96 Merge "Make policy debug logging less verbose" 238e601 Make policy debug logging less verbose 9c88dc3 file_open: fixed docstring to refer to open() instead of file() 6c7407b fileutils: port to Python 3 fe3389e Improve help strings 33a2cee save_and_reraise_exception: make logging respect the reraise parameter 15722f1 Adds a flag to determine whether to reload the rules in policy dacc065 Merge "Update oslo log messages with translation domains" 5d1f15a Documenting policy.json syntax fcf517d Update oslo log messages with translation domains b59cfd9 Merge "Allow policy.json resource vs constant check" e038d89 Fix policy tests for parallel testing 0da5de6 Allow policy.json resource vs constant check e4b2334 Replaces use of urlutils with six in policy module e71cd1a Merge "Trivial: Make vertical white space after license header consistent" 8b2b0b7 Use hacking import_exceptions for gettextutils._ 6d0a6c3 Correct invalid docstrings 6fa29ae Trivial: Make vertical white space after license header consistent 0d8f18b Use urlutils functions instead of urllib/urllib2 12bcdb7 Remove vim header 9ef9fec Use six.string_type instead of basestring 4bfb7a2 Apply six for metaclass 1538c80 ConfigFileNotFoundError with proper argument 477bf7a Add utils for creating tempfile 33533b0 Keystone user can't perform revoke_token d602070 Merge "excutils: replace unicode by six.u" 2ad95e4 parameterize fileutils removal functions d3b6e97 excutils: replace unicode by six.u e35e166 excutils: use six.reraise to re-raise 14ba138 Merge "Fix wrong argument in openstack common policy" 64bb5e2 Fix wrong argument in openstack common policy b7edc99 Fix missing argument bug in oslo common policy 96d1f88 Merge "BaseException.message is deprecated since Python 2.6" f58c936 Merge "Fix policy default_rule issue" 3626b6d Fix policy default_rule issue df3f2ba BaseException.message is deprecated since Python 2.6 7bf8ee9 Allow use of hacking 0.6.0 and enable new checks d74ac1d Merge "Fix missing argument bug in oslo common policy" e4ac367 Fix missing argument bug in oslo common policy 1a2df89 Enable H302 hacking check 323e465 Add conditional exception reraise 22ec8ff Make AMQP based RPC consumer threads more robust 7119e29 Enable hacking H404 test. 4246ce0 Added common code into fileutils and strutils. 21ee25f Add common code for fileutils. 6d27681 Enable H306 hacking check. 1091b4f Reduce duplicated code related to policies a514693 Removes len() on empty sequence evaluation fde1e15 Convert unicode for python3 portability e700d92 Replaces standard logging with common logging 65e3d8c update OpenStack, LLC to OpenStack Foundation 547ab34 Fix Copyright Headers - Rename LLC to Foundation 9e5912f Fix pep8 E125 errors. 6d102bc Provide i18n to those messages without _() 9a8c1d7 Move nova's util.synchronized decorator to openstack common. f182936 Merge "Revert "Add support for finer-grained policy decisions"" 76751a6 Revert "Add support for finer-grained policy decisions" 8b585cb Remove an unneeded 'global' 3fc4689 Add support for finer-grained policy decisions 21b69d8 Add a 'not' operator to the policy langage fa7dc58 Add a new policy language 8c6e7a7 Remove deprecated policy engine APIs
Change-Id: Iddca4243d312c9cd768588753af49dde068d5e4b Co-Authored-By: Jordan Pittier <jordan.pittier@cloudwatt.com>
XtremIO Direct Driver has been contributed to Juno release.
support all minimum required features by Juno for both iSCSI and FibreChannel
Certificate Test Results
https://bugs.launchpad.net/cinder/+bug/1336844
Xing Yang [Wed, 27 Aug 2014 03:53:34 +0000 (23:53 -0400)]
Consistency Groups
This patch enables Consistency Groups support in Cinder.
It will be implemented for snapshots for CGs in phase 1.
Design
------------------------------------------------
The workflow is as follows:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.
This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.
Reverts the changes to cinder/volume/manager.py added in
commit b868ae707f9ecbe254101e21d9d7ffa0b05b17d1 as calling
remove_export in terminate_connection causes Nova live
migration to fail when volumes are attached.
Anthony Lee [Thu, 21 Aug 2014 22:57:06 +0000 (15:57 -0700)]
Fixing 3PAR excessive FC port usage
Updating the 3PAR FC driver so that it can detect if there is
only a single FC path available. When a single FC path is
detected only a single VLUN will be created instead of one for
every available NSP on the host. This will prevent a host
from using extra FC ports that are not needed. If multiple
FC paths are available all the ports will still be used.
Jay S. Bryant [Mon, 25 Aug 2014 18:09:14 +0000 (13:09 -0500)]
Mock processutils.execute properly in test_ibmnas
test_delete_snapfiles and test_delete_snapfiles_nas_gpfs were
not properly mocking out cinder.openstack.common.processutils.execute .
This was causing the unittests to prompt for a sudo password during
unit test on some systems.
This change mocks out processutils.execute to return what is expected
for this test case.
Juan Zuluaga [Thu, 10 Jul 2014 17:15:27 +0000 (13:15 -0400)]
Add Oracle ZFS Storage Appliance ISCSI Driver
ZFSSA ISCSI Driver is designed for ZFS Storage Appliance product
line (ZS3-2, ZS3-4, ZS3-ES, 7420 and 7320).
It uses REST API to communicate out of band with the storage controller
to perform the following:
* Create/Delete Volume
* Extend Volume
* Create/Delete Snapshot
* Create Volume from Snapshot
* Delete Volume Snapshot
* Attach/Detach Volume
* Get Volume Stats
* Clone Volume
Update cinder.conf.sample to include ZFS Storage Appliance
properties.
Certification test results:
https://bugs.launchpad.net/cinder/+bug/1356075
Ronen Kat [Sat, 26 Jul 2014 14:06:52 +0000 (17:06 +0300)]
Add support in Cinder for volume replication - driver approach
This is take #2 for managing replicaiton in Cinder.
This patch provides the foundation in Cinder to make volume
replication available to the cloud admin. It makes Cinder aware
of volume replicas, and allows the cloud admin to define storage
policies (volume types) that will enable replication.
In this version Cinder delegates most the work on replication
to the driver itself.
This includes:
1. Driver exposes replication capabilities via volume type convention.
2. Extend volume table to include columns to support replicaion.
3. Create replicas in the driver, making it transparant to Cinder.
4. Volume manager code to handle API, updates to create_volume to
support creating test replicas.
5. Driver methods to expose per replication functions
Cinder-specs available at https://review.openstack.org/#/c/98308/
Volume replication use-case: Simplified disaster recovery
The OpenStack cloud is deployed across two metro distance data centers.
Storage backends are available in both data ceneters. The backends
are managed by either a single Cinder host or two, depending on the
storage backend requirements.
Storage admin configures the Cinder volume driver to support
replication.
Cloud admin creates a volume type "replicated" with extra-specs:
capabilities:replication="<is> True"
Every volume created in type "replicated" has a copy on both
backends.
In case of data center failure in first data center, the cloud admin
promotes the replica, and redeploy the VMs - they will now run on
a host in the secondary data center using the storage on the
secondary data center.
Xing Yang [Thu, 3 Jul 2014 21:50:45 +0000 (17:50 -0400)]
EMC VMAX Driver Juno Update
This driver is an enhancement from the EMC SMI-S driver.
In Juno, VNX support will be removed from this driver.
Moving forward, this driver will support VMAX only.
The following features are added for VMAX:
* Extend volume
* Create volume from snapshot
* Dynamically creating masking views, storage groups,
and initiator groups
* Striped volumes
* FAST policies
Tempest test results from CI system:
https://bugs.launchpad.net/cinder/+bug/1337840
Tomoki Sekiyama [Thu, 21 Aug 2014 15:13:54 +0000 (11:13 -0400)]
Revert test_rootwrap_filter to avoid python2.6 test failure
With oslo.rootwrap 1.3.0.0a1 which has a bug #1340792, that some filter
rules are dependent on its evaluation order, test_rootwrap_filter randomly
fail for a patch adding to etc/cinder/rootwrap.d/volume.filters with
python2.6, as evaluation order is randomly changes.
The bug in oslo.rootwrap is fixed in master branch, but is not yet
released and not available in CI. So this until new oslo.rootwrap is
released, this patch reverts the unit test to avoid to block reviews.
Boris Pavlovic [Sun, 29 Jun 2014 16:03:28 +0000 (20:03 +0400)]
Integrate OSprofiler and Cinder
*) Add osprofiler wsgi middleware
This middleware is used for 2 things:
1) It checks that person who want to trace is trusted and knows
secret HMAC key.
2) It start tracing in case of proper trace headers
and add first wsgi trace point, with info about HTTP request
*) Add initialization of osprofiler at start of serivce
Set's olso.messaging notifer instance (to send notifications to Ceilometer)
Zhiteng Huang [Tue, 19 Aug 2014 14:27:26 +0000 (22:27 +0800)]
Honor volume:get policy
The fix for bug 1356368 hard-coded a policy check (same as
rule:admin_or_owner) for volume:get. While in most cases this is
what people want, it'd be good we honor policy setting.
Note that before commit 0505bb268942534ad5d6ecd5e34a4d9b0e7f5c04,
DB query volume_get() actually acted as the policy checker for
volume:get, and it raised VolumeNotFound if context.project_id didn't
match volume['project_id']. The check_policy() in volume:get didn't
get a chance to raise PolicyNotAuthorized exception. So in this
change we keep the same behavor.
Extending IBMNAS driver to support NFS based GPFS storage system
- Adds support for hardware platform "NAS based on IBM GPFS
deployments".
- This will enable IBM GPFS-NFS based storage systems to be used
as a backend in OpenStack environments.
Driver certification test results:
https://bugs.launchpad.net/cinder/+bug/1358590
DocImpact
Needs an update in ibmnas driver documentation as this change
proposes a new configuration option (ibmnas_platform_type)
to cinder.conf
Csaba Henk [Sat, 16 Aug 2014 15:49:00 +0000 (16:49 +0100)]
GlusterFS: Use image_utils for tempfile creation
Besides the aesthetical aspect of replacing ad-hoc code
with an utility function, it's more correct
to perform image conversions in CONF.image_conversion_dir,
which is taken care by the image_utils.temporary_file()
context guard.
Rick Chen [Tue, 19 Aug 2014 16:12:23 +0000 (00:12 +0800)]
Modify error code compatible with Mac OS.
Mac OS builtin python errno object has no
attribute 'ENAVAIL'.
Changed
errno.ENAVAIL --> errno.ENODATA
Another when backend storage service not ready, the
error code "ENAVAIL" change to "ENOPROTOOPT".
The extended snapshots extension makes a separate db request for
snapshots which is horribly inefficent. In addition it causes an
issue when using all_tenants=1, because the second request doesn't
pass in the filter. Fix this by caching the snapshot in the request
object like we do for volumes and retrieve the cached object in the
extension.
Jon Bernard [Mon, 18 Aug 2014 20:58:30 +0000 (16:58 -0400)]
Remove redundant temporary_chown from IetAdm
I may have missed something but it appears this routine is effectively
the same as temporary_chown defined in utils.py. It seems reasonable to
use that one instead.
Rick Chen [Sat, 16 Aug 2014 06:46:16 +0000 (14:46 +0800)]
Failed to initialize connection
Failed to initialize connection when the storage server heavy loading.
Changed related on:
dpl_iscsi and dpl_fc
initialize_connection
terminate_connection
When storage server busy, the request api will receive httplib response
code accepted and the response body include event uuid. The operation
request use the event uuid to comfirm the job is completed or not.
John Griffith [Sat, 16 Aug 2014 00:18:33 +0000 (18:18 -0600)]
Mock out image source file in image_utils tests
Change 202ebc2d987c86984ba9cf8e8e36c32899baaa3b
added some metrics gathering around time to fetch
and convert images. To do this it uses os.stat,
and when it was written and tested on Linux everything
was cool.
Turns out however there were a few more methods that needed
mocks for the stat command. This patch adds them.
Also, it was pointed out that for this specific set of tests
we don't actually need to set the class stat_results, mock
will just work for us here since we're not checking an explicit
value. So this patch also removes those class decl's from
previous methods as they're not needed and we should keep
things consistent.
John Griffith [Mon, 18 Aug 2014 21:52:29 +0000 (15:52 -0600)]
Provide a quick way to run flake8
"run_tests.sh -p" always checks every file for errors even though you
have probably only changed a handful. This patch adds "run_tests.sh -8"
which only checks the files that were modified in the HEAD commit or
the current working tree.
Copied from Matthew's work in Nova here:
https://review.openstack.org/#/c/110746/15
Change-Id: I84f6a522da5352a703caa59975643043ec1911fe Co-Authored-By: Matthew Gilliard <matthew.gilliard@gmail.com>
Adds a barbican keymgr wrapper to the key manager interface in
cinder. This allows barbican to be configured as the key manager
for encryption keys in cinder. The wrapper translates calls from
the existing key manager interface to python-barbicanclient.
There are two new configuration variables: encryption_auth_url
(the authentication URL for the encryption service) and
encryption_api_url (the api URL for the encryption service).
John Griffith [Fri, 15 Aug 2014 23:26:15 +0000 (17:26 -0600)]
Catch vol not found in SolidFire transfer
The accept_transfer call in SolidFire wasn't catching
the volume_not found exception like it should be (and
like other methods do).
This patch just adds a check for volume is None on the
get_volume call during accept_transfer and raises an
appropriate exception if we run into that case.
While we're at it, add some more detail to the "NotFound"
error messages to include the driver operation as per
request of reviewers.
John Griffith [Sat, 16 Aug 2014 15:11:54 +0000 (15:11 +0000)]
Fix LOG string formatting in image_utils
My recent change to report stats on image fetch and
conversion used an incorrectly formatted log message.
The result is Trace messages with Key errors in the c-vol
log files.
This change fixes those messages and gets rid of the key error
exception messages in the Logs.
Rick Chen [Sat, 16 Aug 2014 03:56:01 +0000 (11:56 +0800)]
Change the froce delete volume flage to True
The ProphetStor storage volume unable to remove if this has
assignee on other service or application. We make sure the volume
created by the cinder, and the cinder has the privileges to delete
volume.
John Griffith [Tue, 29 Jul 2014 13:37:07 +0000 (07:37 -0600)]
Update ref used for notifications
When we update a volume status in the db, we need to
use the newly updated volume-ref object in the following
notification method call. Otherwise we're sending a
notification message with the old/outdated status
information.
John Griffith [Thu, 10 Jul 2014 00:00:31 +0000 (00:00 +0000)]
Add iSCSI Target objects as independent objects
This patch is a step in decoupling the target
methods and the Volume Driver's Control methods.
This adds the targets directory and the new target objects
that we use with the exception of IET (follow up for that later).
TgtAdm and LIO drivers have been tested with the new LVM object.
All existing drivers are still able to be specified and use the
same objects and code-path they were using previously. New
connector objects are only used when specifying the new driver.
Next step will be mapping current ref LVM driver to the new
LVM object and target model and working on the unit-tests.
After that mark the "old" methods and objects as deprecated
and we can then begin working on some other improvements.
Tomoki Sekiyama [Wed, 25 Jun 2014 20:50:18 +0000 (16:50 -0400)]
Rewrite ionice command filter using ChainingRegExpFilter
Currently, the ionice command prepended to a dd command is allowed by three
rootwrap RegExpFilter's that cover 3 arguments patterns. However, this
doesn't support if either 'iflag=direct' or 'oflag=direct' is omitted.
Because of this problem, deletion of volumes may fail if volume_clear_ionice
is set, as 'iflag=direct' is omitted.
This commit fixes this problem by replacing the filters with
ChainingRegExpFilter's, which allow to execute ionice to be combined with the
other allowed commands, including 'dd'.
Originally '-c[0-3]( -n[0-7])?' was allowed as an ionice option, but it is
invalid to specify -n[0-7] in a single option (for example, when '-c2 -n7'
is specified, ionice causes an error "invalid class argument: '2 -n7'").
In this patch, 2 filters are provided to cover the case only with -c option
and the case with both -c and -n options.
Marc Koderer [Tue, 22 Jul 2014 11:53:45 +0000 (13:53 +0200)]
Use abstract class for the backup driver interface
Instead of using NotImplementedError exceptions to define the interface
it's better to use python abc class [1]. The advantage that it fails
faster if a class doesn't implement the needed interface.
See also [2].
Joshua Harlow [Fri, 15 Aug 2014 00:02:32 +0000 (17:02 -0700)]
Put result in quotes
When a result is the empty string is is hard/confusing
to look at the logs and determine what the task result
was, to avoid this put the result in quotes so that it
is clear in the log message what is the result and what
is not.
Victor A. Ying [Thu, 14 Aug 2014 22:03:58 +0000 (15:03 -0700)]
Fix exception handling in PureISCSIDriver
PureISCSIDriver had some error logging messages that make use of empty
replacement fields "{}" in strings on which the .format() method is
called, as allowed in Python 2.7+. Python 2.6 requires explicitly
naming or enumerating fields, i.e., "{}" must be replaced with "{0}".
This change fixes this so PureISCSIDriver is Python 2.6 compatible.
PureISCSIDriver.terminate_connection() also changed to catch
errors raised by _get_host_name(). Exception handling generally changed
to be more correct.
This change also adds testing of these code path to the unit tests,
to make sure it's actually correct.
'iscsi_target_create_safe' method didn't catch DB exceptions
correct.
Need to catch both db_exc.DBError and db_exc.DBDuplicateEntry
due to current oslo.db limitations: it doesn't handle correct
DBDuplicateEntry with different SQLite versions
John Griffith [Wed, 13 Aug 2014 21:25:49 +0000 (15:25 -0600)]
Actually encode the SolidFire json dump result
The SolidFire driver intends to do a encode on the json data to utf-8,
unfortunatly it's not really doing anything because the syntax of the
calls is wrong.
This patch changes things to actually perform the encoding.
Jay S. Bryant [Wed, 13 Aug 2014 17:16:49 +0000 (12:16 -0500)]
Sync latest oslo-incubator log for debug fixes
This patch pulls in the changes in openstack/common/log.py that
fix the problems that popped up after we removed _() from
around LOG.debug messages. The change in log ensures that any
text that is not of six.text_type is converted to six.text_type.
Merge "Set keystonemiddleware and routes.middleware to log on WARN level"
---------------------
Additional changes being merged (newest to oldest):
--------------------- 759bd879 -
Merge "Set keystonemiddleware and routes.middleware to log on WARN level 71d072f1 -
Merge "Except socket.error if syslog isn't running" 37c00918 -
Add unicode coercion of logged messages to ContextFormatter 66144135 -
Correct coercion of logged message to unicode 1188d88a -
Except socket.error if syslog isn't running ac995bec -
Fix E126 pep8 errors
Note that this change required an update to config.sample due to change 759bd879 - Merge "Set keystonemiddleware and routes.middleware to log
on WARN level"
Code Review / openstack-build / cinder-build.git / log