Rushi Agrawal [Fri, 1 Feb 2013 18:16:58 +0000 (23:46 +0530)]
Fix inability to delete volumes in error state for NetApp driver
While using NetApp 7-mode storage box to create volumes, if the volume
is not created on the storage box (possibly due to wrong/missing
options in cinder.conf file), the volume goes in error state, and when
one tries to delete the volume, an exception was raised which made
the volume go in error_deleting state, which is unrecoverable. This
commit quiesces that exception, resulting in successful removal of the
volume entry from the volume table.
Thierry Carrez [Tue, 29 Jan 2013 13:58:07 +0000 (14:58 +0100)]
Import Oslo's common rootwrap to Cinder
Import oslo-incubator's rootwrap, which contains all the new
features and bugfixes that were pushed to nova-rootwrap earlier
in this cycle, including logging support and path search.
Michael J Fork [Mon, 4 Feb 2013 14:51:29 +0000 (14:51 +0000)]
Mark password config options with secret
Config object supports masking values when writing out if the secret
flag is set on the option definition. This change flags all cinder
options containing a password.
Chuck Short [Sun, 3 Feb 2013 01:22:18 +0000 (19:22 -0600)]
Skip tests if cinder is not installed
The test_capacity_filter_passes_infinite and test_capacity_filter_passes_unknown
tests were failing when cinder is not installed. Skip the tests like the other
tests for test_host_filters.
Change-Id: I5ebfa2dca05b6e89a12e3153598bf4699da888de Signed-off-by: Chuck Short <chuck.short@canonical.com>
john-griffith [Sat, 2 Feb 2013 02:21:18 +0000 (19:21 -0700)]
Fix provider_location column add for PSQL
Migration 006 (commit 690cae58e6bbac5758ea2f7b60774c797d28fba5)
didn't work properly for postgres,
this patch corrects the upgrade by ensuring the execute
is performed and the value is initialized to None.
Since we haven't released a milestone etc with this migration in the
code it should be safe to just fix it here and submit.
john-griffith [Fri, 1 Feb 2013 05:38:51 +0000 (22:38 -0700)]
Fix the generalized copy_image_to_volume operation.
The Generis iSCSI copy volume<->image patch
(change: Iff097629bcce9154829a7eb5aee0ea6302338b26) did
not account for the fact that the existing volume reference
passed in to the copy method would not have the updated
provider_location and iSCSI info available.
To address this, we simply just get a new ref from the DB
after creation if a copy image is requested.
Also, the list of acceptable formats was not fully inclusive,
not only that, but there's no reason to not let the qemu-convert
handle any errors here.
Zhiteng Huang [Wed, 19 Dec 2012 06:56:11 +0000 (14:56 +0800)]
Pull cfg module from Oslo and update cinder-manage accordingly
New cfg module abandons previous disable_interspersed_args() and
recommand to use add_subparsers from argparser module instead.
This patch pull cfg module from Oslo and update the affected
cinder-manage utils.
When we are cloning a volume, we now set the source volume's status
to 'in use'. This prevents a user from deleting the source volume while the
backend copies/clones the contents to the new volume.
When the cloning is complete, we reset the status to it's original
state.
Zhiteng Huang [Mon, 28 Jan 2013 17:12:31 +0000 (01:12 +0800)]
Allow volume back-end to report 'infinite' or 'unknown' as capacity
For some reason, some volume back-ends cannot report actual capacity
that is available or even total capacity (e.g. thin-provisioning),
Cinder should allow such capability report and make placement/schedule
decision based on these status.
In particular, we agree to use 'infinite' to mark unlimited space and
'unknown' to mark unknown/unclear space. Host manager and CapacityFilter
are updated accordingly to allow such capacity. For those back-ends
report 'infinite' or 'unknown' capacity, CapacityFilter will let them
pass to give them a chance to try. Even if failure happened, the retry
mechanism should take care of the case.
Dan Prince [Mon, 28 Jan 2013 15:44:57 +0000 (10:44 -0500)]
Update osapi_volume_extension default.
Updates the Cinder config default for osapi_volume_extension
so that it only uses the new cinder.api.contrib.standard_extensions
loader.
Previously both the new and old extension loader path were listed
which can cause extensions to be loaded twice thus causing
log WARNINGs to appear in the Cinder API log file.
Also, regenerates the cinder.conf.sample to reflect this change.
John Griffith [Fri, 11 Jan 2013 21:38:23 +0000 (14:38 -0700)]
Implement LVM thin provisioning support.
As of LVM2 version 2.02.89 the ability to do thin provisioning
was made available in LVM, this provides some cool new features
but also addresses some problems with things like terrible
LVM snapshot performance.
Currently the version of LVM in Ubuntu 12.04 does NOT support LVM thin,
however an experimental PPA from brightbox which is a backport from
Quantal has been proposed to Cannonical to be pulled in. For some
users the experimental PPA is a better option than dealing with some of
the current issues in the standard LVM2 version of Precise (including
the dd hangs on secure delete). See BP for more info.
This change will create the thin pool if it doesn't exist
using pool_size flag, or by default using entire VG space.
Zhiteng Huang [Thu, 24 Jan 2013 17:21:44 +0000 (01:21 +0800)]
Fix hosts extension and enable its tests
This patch corrects the import in hosts.py, fix the DB API
service_get_by_host_and_topic raises ServiceNotFound exception with
right parameter and move tests to the right place so that they are
enabled.
John Griffith [Wed, 23 Jan 2013 04:15:40 +0000 (04:15 +0000)]
Get updated vol status in volume.api.reserve.
A race condtion was discovered where a nova volume attach
could easily be performed twice for the same volume. Although
the cinder attach update would fail, this occured after the BDM
updates were made and in essence the attach to the compute instance
had already been issued.
This change simply forces a get from the DB of the volume-ref in the
reserve call and checks if an attach is already in progress.
Xing Yang [Thu, 17 Jan 2013 21:33:47 +0000 (16:33 -0500)]
Update EMC SMI-S iSCSI Driver
Refactor EMC SMI-S based iSCSI driver. There are lots of common code
that can be shared between iSCSI and FC drivers using SMI-S. Break
the original emc.py into two files: emc_smis_common.py and
emc_smis_iscsi.py. Also added support for get_volume_stats() and
create_cloned_volume(). get_volume_stats() will be revisited after
winston-d's changes. copy volume<->image will be revisited after
avishay-il's changes.
Pádraig Brady [Wed, 16 Jan 2013 16:23:48 +0000 (16:23 +0000)]
ensure zeros are written out when clearing volumes
Note O_DIRECT is _not_ used when copying from /dev/zero
and there are issues with enabling that (see 444cd542).
Therefore we arrange to have dd issue an fdatasync()
to ensure the data is persisted, lest it be discarded
from the write cache when the device is unprovisioned.
* cinder/volume/drivers/lvm.py (_copy_volume): Add 'conv=fdatasync'
to the dd option list if O_DIRECT isn't used when clearing
(which it won't as descrived above).
john-griffith [Fri, 18 Jan 2013 17:07:10 +0000 (10:07 -0700)]
Clean up IPV6 config checks in test_wsgi.
Two tests in test_wsgi require a configured IPV6 interface.
We added a check/skip for this but it wasn't very graceful and
didn't work on OS-X or platforms other than Linux.
This patch implements the checks a bit cleaner using the skip_if
decorator, and combines the linux/path check and configured interface
check.
john-griffith [Wed, 16 Jan 2013 06:38:11 +0000 (23:38 -0700)]
Add capability to update volume metadata.
This addresses some cleanup and bugs with volume metadata updates
on the cinder side. Mostly this implements v1/volume_metadata api
and adds things like propery checks to cinder.volume.api.update_volume_meta.
NOTE: This is only for api V1, a seperate patch will be provided for
V2 once this lands.
The remainder of the bp will be implemented in a cinderclient patch to follow.
Implement cinder side of blueprint update-vol-metadata
John Griffith [Thu, 17 Jan 2013 23:30:32 +0000 (16:30 -0700)]
Check for configured IPV6 before running tests
Some folks don't/can't have IPV6 configured on their systems,
the new IPV6 tests in test_wsgi require that you do, so
let's add a check and skip if there are no IPV6 configured interfaces.
john-griffith [Thu, 17 Jan 2013 01:49:07 +0000 (18:49 -0700)]
Move iscsi flags back to driver.py
During the lvm class create the iscsi flags were moved.
This is NOT correct, the flags belong with the ISCSI class
and should have been left in driver.py where ISCSIDriver
is defined.
Moving these breaks the inheritance structure for all those that
inherit from ISCSIDriver.
Mate Lakat [Mon, 31 Dec 2012 15:52:18 +0000 (15:52 +0000)]
Snapshot support for XenAPINFS
Related to blueprint xenapinfs-snapshots
Add support for snapshots, by implementing it with deep copies. Each
snapshot is a copy of the volume. Snapshot object has a new,
provider_location key, to store volume parameters.
Add a clear_volume() method that is called from delete_method().
This is only implemented at present for the base LVM driver.
This new clear_volume() method now supports these config vars:
'zero' is the default method and unchanged from previously.
'none' is used if security isn't a concern, or there is
independent volume scrubbing.
'shred' is for more security conscious situations where
an overwrite count is required. Currently this defaults to 3 passes.
size_in_MiB can be used to limit the cleared area to the first
part of the volume, which can greatly speed up the operation,
and can be useful with encrypted volumes for example where
overwritting the encryption keys at the start is sufficient.
Sascha Peilicke [Tue, 15 Jan 2013 13:53:20 +0000 (14:53 +0100)]
Relax various version constraints.
The current 3rd-party module requirements for cinder are still from the
nova git import. Usually, having an upper bound for requirements makes
only sense where API-incompatible changes are expected (like WebOb). For
the others it is better the only require a minimum version because
usually distributions differ on the shipped versions of these Python
modules. For instance, openSUSE has newer versions for almost all of
those and running the testsuite there works flawlessly.
Davanum Srinivas [Sat, 12 Jan 2013 21:16:19 +0000 (16:16 -0500)]
Support for SSL in wsgi.Server
SSL are entirely optional. Support for SSL as well using code from glance. We
have some new options for configuring the SSL support. There are tests for
accessing a sample app w/o ipv6 or ssl, one with just ssl and one with
ipv6 and ssl
Davanum Srinivas [Thu, 10 Jan 2013 16:26:03 +0000 (11:26 -0500)]
Enhance wsgi to listen on ipv6 address
Check if the hostname is ipv6 and set the family appropriately.
Picking up the snippet from glance.
Picked up some code from nova as well to get the test case running
properly
Avishay Traeger [Wed, 9 Jan 2013 07:31:19 +0000 (09:31 +0200)]
Factor out LVM code.
Currently volume drivers inherit from VolumeDriver and ISCSIDriver,
which contain LVM-specific code. The LVM-specific functions are
generally overridden. The problem is that there is no place for
generic code for all drivers to inherit, which the LVM driver can
override. This patch basically makes the VolumeDriver and ISCSIDriver
classes mostly empty and moves the LVM-specific code to lvm.py. Also,
moved the global _iscsi_location() and _iscsi_authentication()
functions into the new LVMISCSIDriver class since nobody else used
them.
Zhiteng Huang [Fri, 5 Oct 2012 16:08:09 +0000 (00:08 +0800)]
Implement filter scheduler
In order to do more sophisticated scheduling (e.g. schedule based on volume
type), filter scheduler is introduced. Several changes are made to make this
possible, some of them are similar to the counterpart in Nova:
- add HostState class to host_manager in order to store volume capabilities
- implement get_volume_stats() method of iSCSIDriver as an example to
demonstrate how volume backend driver reports capabilities as well as status
- add scheduler_options.py and 'scheduler_json_config_location' flag to be
allow loading json configuration file for scheduler at run time
- port common filters/weights from oslo
- add capacity weigher (scheduler/weights/capacity.py) for picking up
target volume backend by weighing free capacity of the host. The default
behavior is to spread volumes across hosts; by changing the
'capacity_weight_multiplier' to negative number, volume placing behavior will
become stacking.
- add capacity filter which filters those hosts have insufficient storage space
to serve the request.
- add 'reserved_percentage' config option to indicate how much space is
reserved. Capacity reservation is needed when volume resize is enabled.
- add 'publish_service_capabilities()' method to volume RPC API to allow
scheduler to have volume service report capabilities. This bumps volume RPC
API to version 1.2
- remove 'volume_force_update_capabilities' config option, volume status will be
report to scheduler in every checking.
The implication of this change to storage/backend driver developer:
- implementation of get_volume_stats() of the driver is now a *MUST*, filter
scheduler heavily relies on the status/capabilities reported by backend driver
to makeplacement decision. To ensure Cinder works seamlessly on the storage
system, driver should at least report following capabilities/status:
----------------------+---------------------------+---------------------------
Capability/Status | Description | Example
----------------------+---------------------------+---------------------------
'volume_backend_name'| back-end name, string | 'Example_Storage_Backend'
----------------------+---------------------------+---------------------------
'vendor_name' | vendor name, string | 'OpenStackCinder'
----------------------+---------------------------+---------------------------
'driver_version' | version, string | '1.0a'
----------------------+---------------------------+---------------------------
'storage_protocol' | supported protocols, | 'iSCSI', 'RBD', 'FC', 'NFS'
| string or list of strings | ['iSCSI', 'NFS', 'FC']
----------------------+---------------------------+---------------------------
'total_capacity_gb' | capacity in GB, integer | 102400
----------------------+---------------------------+---------------------------
'free_capacity_gb' | available capacity in GB, | 1000
| integer |
----------------------+---------------------------+---------------------------
'reserved_percentage' | reserved space in | 0, 10
| percentage, integer |
----------------------+---------------------------+---------------------------
The implication of this change to Cinder administrator:
- the default setting for filter scheduler should work well with the benefits
of:
* being able to fully utilize capacity of backends (driver now has to report
actul total space and space utilization and scheduler uses these info) not
limited by the 'max_gigabytes' config option any more;
* being able to choose placement policy between spreading & stacking for
volume creation (by modifying the 'capacity_weight_multiplier' in
CapacityWeigher)
- with filter scheduler, Cinder is now able to utilize the advanced features/
capabilities provided by different storage back-ends via: defining different
volume types with proper extra_specs. Volume types can be considered as sets
of back-end capabilities requirement.
For example, a volume type which has 'storage_protocol':'FC' key/value pair
definition in its extra_spec can only be served by those back-ends who report
they support FiberChannel protocol. Another example is volume type has 'QoS'
requirement can only be served by back-ends support QoS.
Note/TODO:
* Currently scheduler makes its decision based on the status and capabilities
information reported by volume nodes, and these information is stored in memory
of scheduler process. More sophisticated way may be add on table in DB to
record status/capabilities of all volume nodes, like Nova does for compute nodes.
john-griffith [Mon, 31 Dec 2012 23:16:19 +0000 (16:16 -0700)]
Update SolidFire Volume driver
Adds support for clone volumes and set QoS use volume type / extra_specs.
Also includes some minor cleanup and some hardening for known pre-release
XDB issue in SolidFire Cluster API.
We have the driver broken into 2 files:
hp_3par_common.py and
hp_3par_iscsi.py
The reason we do this is because we have a fibre channel driver
that will be submitted shortly after this is committed. The
fibre channel driver and the iscsi driver share a lot of the same
code that talks to the 3PAR array for provisioning. So,
it made sense not to have duplicate code. The fibre channel driver
will be dependent on the fibre channel support I am actively working
on for nova/cinder grizzly release.
The driver uses a 2 mechanisms to talk to the 3PAR array:
1) a python REST client (hp3parclient) that lives in the pypi
repository here:
http://pypi.python.org/pypi/hp3parclient
2) SSH. We had to pull in some of the ssh code from the base san
driver to help fix an issue with executing commands on the 3PAR
array. The 3PAR has the ability to turn on CSV output for command
results, which makes this easier to parse. Unfortunately, there
is no way to turn CSV mode on permanently for all ssh requests.
So, we have to turn on the CSV output for every single ssh command
issued. Since we use ssh as well, we require the san_* options
to be set.
We use a dual mechianism because the REST API that ships with the 3.1.2
firmware doesn't support all of the capabilities a cinder driver needs
to export volumes.
When a newer version of the firmware comes out that supports host
management on the 3PAR array, then we will get rid of the SSH code.
Dan Prince [Tue, 8 Jan 2013 18:23:15 +0000 (13:23 -0500)]
Update exceptions to pass correct kwargs.
Updates a variety of Cinder exceptions so that they pass the correct
kwargs for proper exception message formatting. Previously
these exceptions would work but have incorrect error messages
because of how CinderException handled exception formatting errors.
Dan Prince [Tue, 8 Jan 2013 18:20:57 +0000 (13:20 -0500)]
Add option to make exception format errors fatal.
Adds a new fatal_exception_format_errors config option which
defaults to False. This option is use to control how the base
CinderException class handles errors which can occur when it
formats error messages.
The motivation for this change is to be able to enable exception
format checking in our tests by setting fatal_exception_format_errors=True.
Zhiteng Huang [Fri, 5 Oct 2012 16:08:09 +0000 (00:08 +0800)]
Implement filter scheduler
In order to do more sophisticated scheduling (e.g. schedule based on volume
type), filter scheduler is introduced. Several changes are made to make this
possible, some of them are similar to the counterpart in Nova:
- add HostState class to host_manager in order to store volume capabilities
- implement get_volume_stats() method of iSCSIDriver as an example to
demonstrate how volume backend driver reports capabilities as well as status
- add scheduler_options.py and 'scheduler_json_config_location' flag to be
allow loading json configuration file for scheduler at run time
- port common filters/weights from oslo
- add capacity weigher (scheduler/weights/capacity.py) for picking up
target volume backend by weighing free capacity of the host. The default
behavior is to spread volumes across hosts; by changing the
'capacity_weight_multiplier' to negative number, volume placing behavior will
become stacking.
- add capacity filter which filters those hosts have insufficient storage space
to serve the request.
- add 'reserved_percentage' config option to indicate how much space is
reserved. Capacity reservation is needed when volume resize is enabled.
- change default scheduler to 'filter.FilterScheduler'.
- add 'publish_service_capabilities()' method to volume RPC API to allow
scheduler to have volume service report capabilities. This bumps volume RPC
API to version 1.2
- remove 'volume_force_update_capabilities' config option, volume status will be
report to scheduler in every checking.
The implication of this change to storage/backend driver developer:
- implementation of get_volume_stats() of the driver is now a *MUST*, filter
scheduler heavily relies on the status/capabilities reported by backend driver
to makeplacement decision. To ensure Cinder works seamlessly on the storage
system, driver should at least report following capabilities/status:
----------------------+---------------------------+---------------------------
Capability/Status | Description | Example
----------------------+---------------------------+---------------------------
'volume_backend_name'| back-end name, string | 'Example_Storage_Backend'
----------------------+---------------------------+---------------------------
'vendor_name' | vendor name, string | 'OpenStackCinder'
----------------------+---------------------------+---------------------------
'driver_version' | version, string | '1.0a'
----------------------+---------------------------+---------------------------
'storage_protocol' | supported protocols, | 'iSCSI', 'RBD', 'FC', 'NFS'
| string or list of strings | ['iSCSI', 'NFS', 'FC']
----------------------+---------------------------+---------------------------
'total_capacity_gb' | capacity in GB, integer | 102400
----------------------+---------------------------+---------------------------
'free_capacity_gb' | available capacity in GB, | 1000
| integer |
----------------------+---------------------------+---------------------------
'reserved_percentage' | reserved space in | 0, 10
| percentage, integer |
----------------------+---------------------------+---------------------------
The implication of this change to Cinder administrator:
- the default setting for filter scheduler should work well with the benefits
of:
* being able to fully utilize capacity of backends (driver now has to report
actul total space and space utilization and scheduler uses these info) not
limited by the 'max_gigabytes' config option any more;
* being able to choose placement policy between spreading & stacking for
volume creation (by modifying the 'capacity_weight_multiplier' in
CapacityWeigher)
- with filter scheduler, Cinder is now able to utilize the advanced features/
capabilities provided by different storage back-ends via: defining different
volume types with proper extra_specs. Volume types can be considered as sets
of back-end capabilities requirement.
For example, a volume type which has 'storage_protocol':'FC' key/value pair
definition in its extra_spec can only be served by those back-ends who report
they support FiberChannel protocol. Another example is volume type has 'QoS'
requirement can only be served by back-ends support QoS.
Note/TODO:
* Currently scheduler makes its decision based on the status and capabilities
information reported by volume nodes, and these information is stored in memory
of scheduler process. More sophisticated way may be add on table in DB to
record status/capabilities of all volume nodes, like Nova does for compute nodes.
john-griffith [Mon, 7 Jan 2013 17:08:16 +0000 (10:08 -0700)]
Use tempdir for lock_path in tests.
Commit: 2331d3336a6adf4fc13a3b187e91a5d1b1f7c723
introduced use of lockutils but wasn't setting a lock_path, in the tests.
As a result we were polluting cinder/openstack/ with cinder-xxxx lockfiles.
This patch adds a tempfile and cleanup to the test so we don't leave any
junk behind.
Doug Hellmann [Thu, 3 Jan 2013 16:09:36 +0000 (11:09 -0500)]
Upgrade WebOb to 1.2.3
The version of WebOb being used in OpenStack was more than
1 year old. This change updates to the latest stable release.
Upgrading WebOb resolves a version conflict between OpenStack
and Pecan, the web framework used by the Ceilometer team for
version 2 of the ceilometer API.
Refer to http://docs.webob.org/en/latest/news.html
for the list of changes between 1.0.8 and 1.2.3.
Doug Hellmann [Mon, 7 Jan 2013 15:52:39 +0000 (10:52 -0500)]
Make WebOb version specification more flexible
Change the WebOb version to >=1.0.8 as an temporary
measure to allow the actual version update to 1.2.3
to roll out across the projects one at a time without
breaking the integration tests.
Code Review / openstack-build / cinder-build.git / log