Anthony Lee [Mon, 8 Dec 2014 14:52:18 +0000 (06:52 -0800)]
Add driver filter and evaluator for scheduler
This patch adds a new filter for the cinder scheduler that
can interpret two new properties provided by backends,
'filter_function' and 'goodness_function'. A driver can rely
on cinder.conf entries to define these properties for a backend
or the driver can generate them some other way. An evaluator is
used by the filter to parse the properties. The 'goodness_function'
property is used to weigh qualified backends in case multiple ones
pass the filter. More details can be found in the spec:
https://review.openstack.org/#/c/129330/
Implements: blueprint filtering-weighing-with-driver-supplied-functions
DocImpact: New optional backend properties in cinder.conf.
New filter and weigher available for scheduler.
Change-Id: I38408ab49b6ed869c1faae746ee64a3bae86be58
John Griffith [Tue, 6 Jan 2015 21:40:00 +0000 (14:40 -0700)]
Remove import of private _lazy module
New version of oslo.i18n released and some things
moved around (internal private modules in the lib).
This should be fine and shouldn't matter to us, BUT
it seems we had some hackery going on in our unit tests
that were being lazy and importing and manipulating the
private library.
This patch removes those cases and fixed up the cinder.i18n
helper method for enable_lazy to accept a bool (which is how
the libraries method works to begin with).
There were several tests in test_faults that were actually
performing and comparing translations of messages. These
tests aren't quite working now because they had a number
of things they imported from private variables and methods
in the i18n module. Honestly I'm not sure of the value of
testing those things here anyway, but for now I've just
added a skip to those and we can sort out long term fixes
and plans later.
John Griffith [Tue, 18 Nov 2014 00:46:54 +0000 (00:46 +0000)]
Transition LVM Driver to use Target Objects
This patch refactors the LVM Driver to take a
seperate Target object instead of mixing the
control and data path implementations inside the
driver itself.
It removes the volume/iscsi.py and brick/iscsis/*
files which were duplicating code and actually
very messy in terms of where calls were actually
being implemented.
Fix handling of serialized data in filtering of volumes
Commit 4aaf40ba1aab4d7c347b05750d0fe21f8d1bcc68 has introduced a bug.
'ast.literal_eval(v)' throws exception 'SyntaxError' if 'v' is an expression
such as UUID '920da701-93c1-4178-9f1a-ef1c7a8a384d', 'd-', or 'd+'.
Catch the 'SyntaxError' exception in addition to 'ValueError' and
assume 'v' is a string. So the API can handle the request successfully
rather than returning a '500' error code.
Currently Cinder makes all requests to other services (Nova, Swift,
etc.) with current user context. Sometimes Cinder needs privileged
rights for external queries (e.g. asking Nova where an instance is
hosted); there is no way to do it yet.
This patch adds to ability to configure an account with special rights
in the configuration ('os_privileged_user_name',
'os_privileged_user_password' and 'os_privileged_user_tenant' options).
Then, requests that need special permissions can be achieved by creating
a client(privileged_user=True).
Note: This user does not necessarily need to have an admin role
associated with it. For instance, policies can be changed to allow a
specific user (without any roles) to perform special actions.
DocImpact: New configuration options to set a privileged user account
Change-Id: I61d8a6de1c5db5ee2ecce124997f9b6447b04e47
Patrick East [Fri, 5 Dec 2014 02:51:06 +0000 (18:51 -0800)]
Add support to PureISCSIDriver for Consistency Groups
This change adds implementations for the required driver methods for
Consistency Groups to be supported with the PureISCSIDriver. There is a
nice direct mapping between Consistency Groups and Purity Protection
Groups which makes the implementation pretty straightforward.
Tom Fifield [Wed, 24 Dec 2014 14:38:32 +0000 (23:38 +0900)]
Expand the description of extra_capabilities
The existing description of extra_capabilities was a bit terse,
meaning deployers couldn't easily gauge the power of this option.
This patch expands the textual description using information from
the blueprint and original commit message, so it can be picked
up automatically by the documentation.
John Griffith [Tue, 23 Dec 2014 00:05:01 +0000 (17:05 -0700)]
Fix broken StorPool driver
The newly added StorPool driver has a number of issues;
It imports it's own 3'rd party libs which are not in the
requirements file, it declares conf options but never registers
them, and as a result of these two things it break the ability
to generate a configuration file.
This patch adds a try_import around the import storpool calls
like we do in other drivers, and it registers the config options
properly.
We also move the api setting out of init and into the check_setup
so the service doesn't crash if somebody tries to load the driver
without the required storpool modules.
Eric Harney [Mon, 17 Nov 2014 20:01:02 +0000 (15:01 -0500)]
Brick LVM: Remove self.lv_list
This member isn't used anywhere, and now that requests for
specific LVs are overwriting self.lv_list with a subset of
the LVs that are present, it doesn't contain coherent info
anyway.
This commit broke the ability to create Iscsi targets w/
cinder-rtstool. Everything seems to be working fine except there
are issues with regards to how rtstool output is parsed in that
any output (regardless of exit status) causes the command
to be marked as failed.
Suggesting this as a quick revert until we can investigate
the proper fix for error handling with cinder-rtstool.
Clinton Knight [Fri, 12 Dec 2014 20:30:26 +0000 (15:30 -0500)]
Replace the NetApp driver proxy layer with a proper factory.
The proxy implementation required a hack in the Abstract Base
Classes project in Cinder core. This factory implementation
removes the need for the hack. It also improves logging, as
the actual driver object is provided to the volume manager
instead of the driver proxy.
Xing Yang [Fri, 19 Dec 2014 17:45:45 +0000 (12:45 -0500)]
Handle the volume not found case in the VMAX driver
When a volume cannot be found on the array, VMAX driver throws
an exception. This brings a volume into error_deleting state during
the volume deletion operation. This patch fixed it by returning None
when a volume cannot be found.
John Griffith [Fri, 19 Dec 2014 19:25:30 +0000 (19:25 +0000)]
Fix format errors in brick/iscsi LOG messages
Currently in cinder.brick.iscsi.iscs:LioADM.create_iscsi_target
The Log message for the exception is:
LOG.error("%s" % e),
This rightfully results in:
*** UnicodeError: UnicodeError(u'Message objects
do not support str() because they may contain
non-ascii characters. Please use unicode() or
translate() instead.',)
In some cases this causes the Volume service to stop and
doesn't help a ton with debug. While looking at this also
noticed a number of other similar cases where invalid LOG
messages were set up. Following the i8n guidelines here:
http://docs.openstack.org/developer/oslo.i18n/guidelines.html
Went ahead and cleaned the bulk of the LOG messages in this
file up to adhere to the guidelines as well as fixing the
UnicodeError described in the bug.
Andrew Kerr [Tue, 25 Nov 2014 20:47:01 +0000 (15:47 -0500)]
Add unit tests for NetApp do_setup methods
We need to ensure that our do_setup methods are properly calling check_flags.
This patch adds tests to that effect. We also need to ensure that our config
documentation is accurate; this also fixes a minor issue for that.
John Griffith [Sat, 20 Dec 2014 00:45:16 +0000 (17:45 -0700)]
Set iet_conf to nonexistent file in unit test
The Iet test in test_iscsi isn't setup to deal with
an actual iet.conf file. The result is that if you happen
to be on a system that has an iet.conf file the test will fail
when it gets to the os.stat check.
This patch just uses a tempdir with a bogus file name that is
set as the iet_conf option to make sure the test never runs into
a situation where the file is actually there.
Avishay Traeger [Thu, 18 Dec 2014 13:55:26 +0000 (14:55 +0100)]
Fix issue with passing lists in filters
The Cinder db code already allows searching for a value in a list,
for example, id in ['id1', 'id2', 'id3']. The problem is when we
get the value at the API it comes as a string (e.g.
"['id1', 'id2', 'id3']", and therefore the filter fails. This was
already fixed for metadata filters, so just apply it to all filters.
We do a try/except because if it really is a string, then the
literal_eval will fail because we don't pass double quotes, such as
"'string'". The current code therefore will now cover lists and
fall back to the original behavior for strings.
John Griffith [Fri, 19 Dec 2014 22:14:22 +0000 (15:14 -0700)]
Add a provider_id column to Volumes and Snapshots
There are a number of cases where there's some really ugly mapping
work in drivers to try and map the backend device-id to the
Cinder ID of a Volume or Snapshot. Most drivers seem to work around this in
their own creative ways using their own mechanisms (metadata,
naming schemes etc).
It seems like it would be useful for a number of reasons however to
go ahead and add a simple column to the Volumes table that could be
used for storing a backend ID for the Volume and Snapshot.
Mike Perez [Fri, 19 Dec 2014 23:44:04 +0000 (15:44 -0800)]
Mock leaked _execute() calls in driver tests
Some drivers have _execute() calls being linked, causing tests to prompt
for password and eventual fail for time out. This prevents those
_execute calls from happening.
Jay S. Bryant [Fri, 19 Dec 2014 23:24:58 +0000 (17:24 -0600)]
Sync request_utils module from oslo-incubator
The request_utils module hasn't had a sync done since shortly
after the module was added. This patch brings us up to date
with the latest oslo-incubator code.
This sync is needed to enable removal of the old gettextutils
and i18n code.
Current HEAD in OSLO:
---------------------
commit 36b0e8570b449129d6d474c03b02ceb62edb78df
Date: Thu Dec 11 11:27:08 2014 +0100
We shouldn't replace `oslo-incubator` in comments
Changes being merged with this patch:
--------------------- 5d40e143 - Remove code that moved to oslo.i18n 39625e18 - Set pbr 'warnerrors' option for doc build
Jay S. Bryant [Fri, 19 Dec 2014 22:30:31 +0000 (16:30 -0600)]
Sync periodic_task module from oslo-incubator
The periodic_task module hasn't been sync'd since the middle of the
Juno release cycle. This patch brings it up to date with the
latest oslo-incubator code.
Current HEAD in OSLO:
---------------------
commit 36b0e8570b449129d6d474c03b02ceb62edb78df
Date: Thu Dec 11 11:27:08 2014 +0100
We shouldn't replace `oslo-incubator` in comments
Changes merged with this patch:
--------------------- 5d40e143 - Remove code that moved to oslo.i18n a3220c51 - add list_opts to all modules with configuration options
Jay S. Bryant [Fri, 19 Dec 2014 21:06:19 +0000 (15:06 -0600)]
Sync the latest middleware module from oslo-incubator
Middleware hasn't had a sync from oslo-incubator since the
middle of the icehouse development cycle. This sync
brings us up to date with the latest changes.
Current HEAD in OSLO:
---------------------
commit 36b0e8570b449129d6d474c03b02ceb62edb78df
Date: Thu Dec 11 11:27:08 2014 +0100
We shouldn't replace `oslo-incubator` in comments
Changes being merged with this patch by file:
---------------------
catch_errors.py
- ce8f8fa4 - Add middleware.catch_errors shim for Kilo
- 4504e4f4 - Remove middleware
- 5d40e143 - Remove code that moved to oslo.i18n
- 76183592 - add deprecation note to middleware
- 463e6916 - remove oslo log from middleware
- fcf517d7 - Update oslo log messages with translation domains
- fdcae242 - Middleware to catch all error in WSGI pipeline
request_id.py
- 4ffc4c87 - Add middleware.request_id shim for Kilo
- 4504e4f4 - Remove middleware
- 76183592 - add deprecation note to middleware
Mitsuhiro Tanino [Thu, 18 Dec 2014 14:07:24 +0000 (09:07 -0500)]
LVM: Volume is deleted unexpectedly during volume migration
When using LVMISCSIDriver, a volume is unexpectedly deleted during
volume migration operation. This problem only occurs when the
volume_group of backend A and backend B is same value such as
misconfiguration. Even if the configuration is wrong, cinder should
not delete the volume unexpectedly for user.
Jay S. Bryant [Fri, 19 Dec 2014 20:54:54 +0000 (14:54 -0600)]
Sync the latest loopingcall module from oslo-incubator
We have not done a sync of loopingcall since its dependency
upon the old i18n module was removed. To get that old code
removed we need to do this sync.
Current HEAD in OSLO:
---------------------
commit 36b0e8570b449129d6d474c03b02ceb62edb78df
Date: Thu Dec 11 11:27:08 2014 +0100
We shouldn't replace `oslo-incubator` in comments
Change being merged with this patch:
--------------------- 5d40e143 - Remove code that moved to oslo.i18n
Jay S. Bryant [Fri, 19 Dec 2014 20:37:45 +0000 (14:37 -0600)]
Sync install_venv_common from oslo-incubator
The install_venv_common modules hasn't had a sync done
since early in the icehouse release. This sync brings
Cinder's version up to date with the latest code.
Current HEAD in OSLO:
---------------------
commit 36b0e8570b449129d6d474c03b02ceb62edb78df
Date: Thu Dec 11 11:27:08 2014 +0100
We shouldn't replace `oslo-incubator` in comments
Change being merged with this patch:
--------------------- fe3389e5 - Improve help strings
Jay S. Bryant [Fri, 19 Dec 2014 20:10:33 +0000 (14:10 -0600)]
Sync latest imageutils from oslo-incubator
Performing a sync as imageutils has not been updated since late
in the Icehouse release cycle.
Current HEAD in OSLO:
---------------------
commit 36b0e8570b449129d6d474c03b02ceb62edb78df
Date: Thu Dec 11 11:27:08 2014 +0100
We shouldn't replace `oslo-incubator` in comments
Change being merged with this patch:
--------------------- b2d35eec - Use list.pop(0) to keep the code simpler
John Griffith [Fri, 19 Dec 2014 19:32:02 +0000 (19:32 +0000)]
rtstool on Ubuntu installs in /usr/local/bin
The cinder-rtstool which is required to use LIO iscsi_helper
installs in /usr/local/bin on Ubuntu and /usr/bin on RHEL
variants. Currently the rootwrap.conf only has /usr/bin
configured, as a result LIO can't be used on Ubuntu without
modifying the rootwrap file.
This patch just adds /usr/local/bin to the rootwrap.conf; Honestly
I'm not sure if this is good, bad or doesn't matter. May be able
to address this via documentation, or upstream in the packaging of
cinder-rtstool instead? Not sure what's preferred.
Tom Barron [Tue, 9 Dec 2014 09:46:29 +0000 (04:46 -0500)]
Mock calls to rpm and dpkg from NetApp unit tests
This patch fixes an issue wherein several NetApp unit tests ran
OS rpm or dpkg commands because the callouts to these commands
were not mocked out during driver initialization.
It also replaces 'rpm -qa' with 'rpm -q' when that command is
invoked since the latter also works and is faster.
Jay S. Bryant [Fri, 19 Dec 2014 15:37:19 +0000 (09:37 -0600)]
Fix files in Cinder with execute bit set
I noticed there were a number of files in the cinder/volume
directory tree that have the execute bit set for no apparent
reason. I think in all cases it was because that was just
they way the permissions on the file were originally set based
on the user's umask settings.
This patch changes the permissions to be consistent with the other
python files in the cinder/volume directory.
Daniel Wilson [Wed, 17 Dec 2014 20:05:44 +0000 (12:05 -0800)]
Add error handling to _connect function in PureISCSIDriver
The driver was throwing an exception when trying to connect a volume
to a host when they were already connected. This change catches the
exception and looks up the existing connection information to use as
a return value.
Kurt Martin [Thu, 18 Dec 2014 19:50:52 +0000 (11:50 -0800)]
Fix 3PAR host persona mapping to match WSAPI
The HP 3PAR drivers had mapped it's extra specs host personas to match
the 3PAR CLI/IMC, the WSAPI server has a different mapping for the
same personas. This patch corrects the mapping.
The Configuration Reference Guide will need to be updated to document
the correct mappings to match the code.
Closes-Bug: 1403997
DocImpact
Anthony Lee [Tue, 9 Dec 2014 01:12:27 +0000 (17:12 -0800)]
Fixing 3PAR connection name cache error
The 3PAR driver had a connection name cache in 3PAR common. If
Cinder was restarted or the common object was destroyed the cache
was lost. Instead of having the cache in 3PAR common, the 3PAR
driver can query the backend to see if there is a host already
available. Replacing the name cache with a backend query had minimal
impact on performance of the driver and eliminates the problem of the
cache being destroyed/lost.
Code Review / openstack-build / cinder-build.git / log