Sasikanth [Mon, 23 Jun 2014 14:10:12 +0000 (19:40 +0530)]
Update _resize_volume_file() to support appropriate permissions
ibmnas driver inherits _set_rw_permissions_for_all() from nfs.py
which sets 666 to the volumes. Changes are expected to be made in
nfs.py such that it sets 600 or 660 permissions, by doing so
ibmnas driver fails performing the operation _resize_volume_file()
The current create backing methods do not support specifying an adapter
type for the backing VM. These methods always create a backing VM with
a single disk and LSI logic adapter. This change adds optional parameters
to create backing methods so that a backing VM can be created without a
disk or with a specific adapter type.
Eric Guo [Mon, 30 Dec 2013 11:36:49 +0000 (19:36 +0800)]
Use oslo-incubator module units
There is Cinder's version units module, that looks good. Considering
consistency with other projects like Nova, Glance, Cinder would be
better to use oslo-incubator's version.
Joshua Harlow [Sat, 21 Jun 2014 01:54:55 +0000 (18:54 -0700)]
Attach log listeners to other engines
Attach the created task/flow/engine listener to
the other usages of taskflow that exist in cinder
so that those locations can also benefit from
the same logging of state and activity.
John Griffith [Wed, 11 Jun 2014 22:43:41 +0000 (22:43 +0000)]
Remove global conf settings from iscsi helper
This "intermediate" iscsi helper pulls all of it's
config settings from the global config. This is fine
if you only have a single backend, but if you do
multi-backend it puts things in a bad state where some
of the backend specific settings are picked up but
others are not (for example iscsi_ip_address).
This change modifies methods like create_export in
the volume/iscsi helper to take the drivers version
of the config settings as a parameter and use those
instead of setting off of the global values.
Long term there's a lot of cleanup surrounding our
inheritance model and especially the iscsi helpers.
We can address that going forward but here we just want
to fix the bug in the safest way possible.
Michael Berlin [Mon, 19 May 2014 15:28:54 +0000 (17:28 +0200)]
GlusterFS: Various unit test improvements
Fixed bug that "_ensure_share_writable" was not mocked and therefore
"GlusterfsException" was thrown earlier than expected with an unexpected
error message.
To better guard against situations like this, I've added a helper method
"assertRaisesAndMessageMatches" to test for the thrown exception AND its
message.
Added "mox.VerifyAll()" where it was missing and cleaned up unused
mocks.
Fixed test for "create_volume_from_snapshot" (which actually tested
"create_cloned_volume").
Added tests for driver function "copy_volume_to_image".
Overall, code coverage for volume/drivers/glusterfs.py improved from 74%
to 82%.
vmware: Force chunked transfer for upload-to-image
The upload-to-image operation downloads (using stream-optimized HTTP NFC
transfer) the virtual disk corresponding to the volume to a pipe from
where the glance image client reads the data for upload. Due to a recent
change in the glance image client, success in seeking the input file
object results in forgoing the chunked transfer in favor of regular
transfer. The glance image client expects an IOError if the input file
object is a pipe. Currently the pipe seek() is a NOP and this results in
upload-to-image failure. This changes fixes such failures by throwing
an IOError indicating an illegal seek.
Victor Sergeyev [Fri, 28 Feb 2014 09:20:37 +0000 (11:20 +0200)]
Sync the latest common db code from oslo
This sync brings the latest openstack/common/db code from
olso-incubator.
Changes in application code caused by API changes in openstack.common.db -
oslo.db no longer stores SQLAlchemy Engine and sessionmaker instances
globally and it's up to applications to create them. So we should add
methods for work with engine and session to Cinder.
List of changes:
54f7e7f Prevent races in opportunistic db test cases 8a0f581 Use oslotest instead of common test module 4a591ea Start ping listener also for postgresql f0e50ed Add a warning to not use get_table for working with ForeignKeys 2fd457b Ignore migrate versioning tables in utf8 sanity check 9fed4ed Fix Keystone doc build errors with SQLAlchemy 0.9 f7705f3 Make table utf-8 charset checking be optional for DB migration 5b7e61c Dispose db connections pool on disconnect 295fcd9 Do not use the 'extend' method on a dict_items object d1988b9 Set sql_mode callback on connect instead of checkout a1a8280 Fix excessive logging from db.sqlalchemy.session dc2d829 Add lockutils fixture to OpportunisticTestCase d10f871 Adapt DB provisioning code for CI requirements 5920bed Make db utils importable without migrate 9933bdd Get mysql_sql_mode parameter from config 96a2217 Prevent incorrect usage of _wrap_db_error() 6cab37c Python3: define a __next__() method for ModelBase 20a7510 Add from_config() method to EngineFacade 7959826 db: move all options into database group fea119e Drop special case for MySQL traditional mode, update unit tests a584166 Make TRADITIONAL the default SQL mode dda24eb Introduce mysql_sql_mode option, remove old warning 0b5af67 Introduce a method to set any MySQL session SQL mode 8dccc7b Handle ibm_db_sa DBDuplicateEntry integrity errors 5b9e9f4 Fix doc build errors in db.sqlalchemy 0f24d82 Fix migration.db_version when no tables ac84a40 Update log translation domains c0d357b Add model_query() to db.sqlalchemy.utils module 84254fc Fix a small typo in api.py b8a676c Remove CONF.database.connection default value 86707cd Remove None for dict.get() 0545121 Fix duplicating of SQL queries in logs fcf517d Update oslo log messages with translation domains fa05b7c Restore the ability to load the DB backend lazily 630d395 Don't use cfg.CONF in oslo.db ce69e7f Don't store engine instances in oslo.db 35dc1d7 py3kcompat: remove b4f72b2 Don't raise MySQL 2013 'Lost connection' errors 271adfb Format sql in db.sqlalchemy.session docstring 0334cb3 Handle exception messages with six.text_type eff69ce Drop dependency on log from oslo db code 7a11a04 Automatic retry db.api query if db connection lost 11f2add Clean up docstring in db.sqlalchemy.session 1b5147f Only enable MySQL TRADITIONAL mode if we're running against MySQL 39e1c5c Move db tests base.py to common code 986dafd Fix parsing of UC errors in sqlite 3.7.16+/3.8.2+ 9a203e6 Use dialect rather than a particular DB API driver 1779029 Move helper DB functions to db.sqlalchemy.utils bcf6d5e Small edits on help strings ae01e9a Transition from migrate to alembic 70ebb19 Fix mocking of utcnow() for model datetime cols 7aa94df Add a db check for CHARSET=utf8 aff0171 Remove "vim: tabstop=4 shiftwidth=4 softtabstop=4" from headers fa0f36f Fix database connection string is secret 8575d87 Removed copyright from empty files d08d27f Fix the obsolete exception message 8b2b0b7 Use hacking import_exceptions for gettextutils._ 9bc593e Add docstring for exception handlers of session 855644a Removal of _REPOSITORY global variable. ea6caf9 Remove string.lowercase usage a33989e Remove eventlet tpool from common db.api e40903b Database hook enabling traditional mode at MySQL f2115a0 Replace xrange in for loop with range c802fa6 SQLAlchemy error patterns improved 1c1f199 Remove unused import 6d0a6c3 Correct invalid docstrings 135dd00 Remove start index 0 in range() 28f8fd5 Make _extra_keys a property of ModelBase 45658e2 Fix violations of H302:import only modules bb4d7a2 Enables db2 server disconnects to be handled pessimistically 915f8ab db.sqlalchemy.session add [sql].idle_timeout e6494c2 Use six.iteritems to make dict work on Python2/3 48cfb7b Drop dependency on processutils from oslo db code 4c47d3e Fix locking in migration tests c2ee282 Incorporating MIT licensed code c5a1088 Typos fix in db and periodic_task module fb0e86a Use six.moves.configparser instead of ConfigParser 1dd4971 fix typo in db session docstring 8a01dd8 The ability to run tests at various backend 0fe4e28 Use log.warning() instead of log.warn() in oslo.db 12bcdb7 Remove vim header 4c22556 Use py3kcompat urlutils functions instead of urlparse ca7a2ab Don't use deprecated module commands 6603e8f Remove sqlalchemy-migrate 0.7.3 patching 274c7e2 Drop dependency on lockutils from oslo db code 97d8cf4 Remove lazy loading of database backend 2251cb5 Do not name variables as builtins 3acd57c Add db2 communication error code when check the db connection c2dcf6e Add [sql].connection as deprecated opt for db 001729d Modify SQLA session due to dispose of eventlet 4de827a Clean up db.sqla.Models.extra_keys interface 347f29e Use functools.wrap() instead of custom implementation 771d843 Move base migration test classes to common code 9721129 exception: remove 56ff3b3 Use single meta when change column type 3f2f70e Helper function to sanitize db url credentials df3f2ba BaseException.message is deprecated since Python 2.6 c76be5b Add function drop_unique_constraint() d4d8126 Change sqlalchemy/utils.py mode back to 644 cf41936 Move sqlalchemy migration from Nova 5758360 Raise ValueError if sort_dir is unknown 31c1995 python3: Add python3 compatibility support 3972c3f Migrate sqlalchemy utils from Nova 1a2df89 Enable H302 hacking check 3f503fa Add a monkey-patching util for sqlalchemy-migrate 7ba5f4b Don't use mixture of cfg.Opt() deprecated args 489e2b7 Ensure that DB configuration is backward compatible 7119e29 Enable hacking H404 test. ebaa578 Enable user to configure pool_timeout 30cb47e Changed processing unique constraint name. 6d27681 Enable H306 hacking check. 444bdbc Add a slave db handle for the SQLAlchemy backend. 484a1df Enable hacking H403 test 4ff33b0 Specify database group instead of DEFAULT
Zhiteng Huang [Fri, 20 Jun 2014 03:48:28 +0000 (11:48 +0800)]
Fix order dependency of admin metadata
Some test cases in VolumeTestCase depends on the order of admin metadata
returned from db.volume_get() call, which causes jenkins test failure in
recent reviews. This change extracts the key/value we are intertested in
and compare them with AssertDictMatch() to remove the order dependency.
Eric Harney [Thu, 19 Jun 2014 16:57:51 +0000 (12:57 -0400)]
GlusterFS tests: Mock out compute, don't load novaclient
Mock out the compute library for these tests, which prevents loading
the novaclient library.
This makes these tests more reliable, as currently the novaclient
lib interferes with our mocking of os.path.exists in these tests
in some test configurations.
This should also make the GlusterFS tests slightly more efficient
since they were previously loading the novaclient python library
but did not really need to.
According to the OpenStack translation policy available at
https://wiki.openstack.org/wiki/LoggingStandards debug messages
should not be translated. Like mentioned in several changes in
Nova by garyk this is to help prioritize log translation.
This patch adds a new hacking check - N319 - that ensures all
debug log messages don't have translations.
Joshua Harlow [Mon, 9 Jun 2014 22:32:17 +0000 (15:32 -0700)]
Handle the case where az is disabled/removed
Instead of retaining a local in-memory list of
availability zones that is never purged and therefore
never updates itself if availability zones are removed
we should instead cache those zones for a given period
of time (defaulting to one hour) and on cache expiry
refetch the enabled availability zones.
DocImpact: adds a new configuration that defines the
default availability zone cache expiry time in seconds.
This will enable the engine that runs the create volume
request to in a future change remove the need to have
a functor which proxies the validation logic around
availability zones.
glusterfs: Honor mount options when restarting cinder service
When restarting cinder service (cinder-volume specifically),
ensure that the gluster mounts are unmounted and remounted so
that any new mount options added to the shares config file
is taken effect, post service restart.
abhishekkekane [Tue, 10 Jun 2014 06:29:20 +0000 (06:29 +0000)]
Made provision for providing optional arguments
The 'quota_committed' attribute of 'RequestContext' object is
a transient property, so it will not be saved in the taskflow
persistent storage. The updated value of 'quota_committed'
attribute will not be available while resuming/reverting the
flow, if cinder api-service is down/stopped after committing
the quota.
Since this 'quota_committed' attribute is not used anywhere
in cinder project other than in create-volume taskflow api, so
removed 'quota_committed' from RequestContext and made
provision to pass it as an optional argument which will be
passed to api-flow via create_what dictionary, in order to
make it persistent and use it as and when needed.
Joshua Harlow [Thu, 12 Jun 2014 21:35:35 +0000 (14:35 -0700)]
Add task/flow listener support around the volume api flow
This will add in logging of what is occurring around the
actions taken by taskflow when taskflow executes the initial
create_volume workflow.
Since the oslo logging module will include the request_id of
the currently active request (per greenthread) this will make
it easier to understand and debug the actions occurring inside
taskflow's engine concept.
Dirk Mueller [Thu, 12 Jun 2014 22:24:23 +0000 (00:24 +0200)]
Retry lvremove with ignore_suspended_devices
A lvremove -f might leave behind suspended devices
when it is racing with udev or other processes
still accessing any of the device files. The previous
solution of using lvchange -an on the LV had the
side-effect of deactivating origin LVs alongway in
the thick volume case, which was undesired.
It turns out retrying the deactivation twice and
ignoring the suspended devices on the second iteration
avoids the hang of all LVM operations after an initial
failure.
The reset-state API call only sets that main status
column on the volume object. There's also a method
to set attach status, however it turns out that due
to the implementation of the status validation method
in the parent class, this was impossible to use because
it only checked and accepted "status" NOT "attach_status".
This patch fixes that attach_status problem, it also implements
the ability to update the migration status. A cinderclient change
will be needed to expose these calls as well.
Unmesh Gurjar [Mon, 2 Jun 2014 13:14:55 +0000 (18:44 +0530)]
vmware: Fixes VMDK volume incompatibility issue
VMDK volume created in a cluster having ESXi 5.1 host has a high
hardware version i.e. 9; hence, fails when attached to instance
on an ESXi 5.0 host. This patch sets the hardware version to 8,
so that the volume can be attached to instances on any ESXi host
starting version 5.0 and above.
Remove unused oslo module and adjust opentstack-common.conf
The issues are detected by new change about update.py
in I68c0108304be8a315bcefe939c22019caab9bc98. This patch
doesn't depend the oslo-incubator patch.
* remove unused module crypto
* add missing module loopingcall in openstack-common.conf
* sort the modules in openstack-common.conf
PranaliDeore [Tue, 6 May 2014 04:04:45 +0000 (04:04 +0000)]
Copy custom properties to image from volume
Presently after copying an image to volume, all properties
of the image are getting copied properly but while creating
image back from volume, it doesn't copy custom properties
to the image.
At present in volume-glance-metadata table all the properties
of volume are stored as key and value. Because of this it is
difficult to differentiate between core and custom properties.
To overcome this, I have added a new option
'glance_core_properties' in cinder.conf. This option defines
all core properties of an image. This way, it's easy to separate
core and custom properties from the glance_volume_metadata and add
custom property to the newly created image.
Joshua Harlow [Tue, 10 Jun 2014 02:02:42 +0000 (19:02 -0700)]
Fix log capturing fixture
Instead of nuking the unit test logs capture them according
to how tempest (and oslotest) capture them so that the tests
when ran locally provide the full logging output instead of
a miniaturized version.
This adds in the following environment variables which can
be used to control testr captured logging:
- OS_LOG_CAPTURE (turns on log capturing so that the testr
repository will have log output, at a INFO level)
- OS_DEBUG (changes the captured logging level from INFO
to DEBUG, only useful when OS_LOG_CAPTURE is enabled)
It also creates a lambda that is used to test whether an
environment is enabled (avoiding recreating the same set
of checks in multiple places).
Fix Brocade FC driver's use of the fc_fabric_names
The Brocade Fibre Channel Zone Manager driver wasn't
stripping out whitespace in the fc_fabric_names
prior to using them. This would cause breakage later on
in the driver.
Eric Harney [Tue, 10 Jun 2014 15:25:01 +0000 (11:25 -0400)]
LIO: Don't add ACL for local initiator name
Currently cinder-rtstool requires /etc/iscsi/initiatorname.iscsi
to be present in order to create a LUN/ACL/portal successfully.
However, this is not required since Cinder will create the
required ACLs dynamically at attach time as well.
Removing this path means that iscsi-initiator-utils no longer
needs to be installed locally when using a remote Nova compute
node.
On upload-to-image failure before initiating the data transfer or during
data transfer, the source volume status is restored properly whereas the
image created remains in queued or saving state. This change deletes the
image during such failures.
This patch flushes IO for each SCSI device that
brick removes from the system. The patch also
flushes a multipath device prior to removing
each underlying LUN associated with the
multipath device. We were seeing kernel
IO errors in the syslog without calling flush
on the block device, during a detach for a
multipath FibreChannel copy image to volume
action.
Rahul Verma [Tue, 3 Jun 2014 22:26:48 +0000 (15:26 -0700)]
Remove Quota Exception Stack Traces from Cinder Logs
Added an if clause to the Cinder API FaultWrapper so that
a warning is printed rather than an error and stack trace
if an exception of type QuotaError is thrown.
Nikola Dipanov [Fri, 25 Apr 2014 16:39:43 +0000 (18:39 +0200)]
Remove check_{attach,detach} from volumes API
These methods are not called from anywhere and are likely just a relic
from splitting Cinder out from Nova. Moreover their usage from the API
should really be discouraged as they are inherently racy (the result is
immediately outdated since it is not atomic with anything that might
follow).
Nikola Dipanov [Fri, 25 Apr 2014 11:38:31 +0000 (13:38 +0200)]
Make begin_detaching fail if volume not "in-use"
Like it's counterpart from Nova's volume-attach functionality standpoint
- reserve_volume, begin_detaching should fail if the volume is not in
the correct state to be detached. This will prevent nova from
attempting to check and then detach, which is inherently racy.
Mark Sturdevant [Tue, 27 May 2014 21:49:34 +0000 (14:49 -0700)]
3PAR volume detach with host in a host set
When deleting the last VLUN for a host, try to
delete the host. If the host is in a host set
this will fail. When that happens, log a nice
message including the reason, but don't fail
or hang the delete VLUN. Leave hosts in host
sets.
John Griffith [Fri, 30 May 2014 15:59:52 +0000 (15:59 +0000)]
Update cinder.conf
This is mostly just a sanity check, we're revmoving
this whole check from the gate, but we've had
conflicting versions of the truth here and I'm trying
to figure out why.
This patch is a fresh run of generate_sample, which differs
from https://review.openstack.org/#/c/96784/
Matt Riedemann [Thu, 29 May 2014 15:45:55 +0000 (08:45 -0700)]
Sync periodic_task from oslo-incubator
This is more or less to get commit c63fd5a from oslo into the core
projects which have several periodic tasks. Cinder has periodic tasks
to sync up volume state with the server and they don't have specific
spacing values set which can lead to non-deterministic spacing of when
the tasks run.
Note that this does not include the gettextutils and log dependencies
since there are not functional changes in those modules needed for the
periodic_task changes synced in *and* more importantly, the changes
to gettextutils and log require pervasive changes to cinder which
should happen when cinder integrates with the oslo-i18n library for
blueprint i18n--messages.
Further note that this does not include jsonutils due to some
issues introduced with a change for python 2.6 that impacts how strings
are encoded with simplejson. The details for that issue are in bug 1314129. The jsonutils changes are not related to the periodic_task
changes being synced in so the dependency is not functionally required.
cinder.db.base.Base is also updated otherwise multiple inheritance
involving that class will not work, which impacts all of the classes
that extend cinder.manager.Manager which extends both cinder.db.Base
and PeriodicTasks, and commit 47c9d60 adds attributes to the __init__
for PeriodicTasks. Nova had the same change with commit 5ae97ea.
Changes:
c63fd5a Make unspecified periodic spaced tasks run on default interval f0dd798 Remove rendundant parentheses of cfg help strings fcf517d Update oslo log messages with translation domains 051b9f3 Refactor unnecessary arithmetic ops in periodic_task 674cdaf Refactor if logic in periodic_task b6b82c5 Use timestamp in periodic tasks 47c9d60 Don't share periodic_task instance data in a class attr 8b2b0b7 Use hacking import_exceptions for gettextutils._ c5a1088 Typos fix in db and periodic_task module 12bcdb7 Remove vim header
John Griffith [Thu, 29 May 2014 18:18:12 +0000 (18:18 +0000)]
Remove second get call to list/show volumes
Recently it was observed that with large numbers of volumes
that things like "cinder list" could take extremely long to
return.
For example, running cinder list on a system with 1000 volumes took
greater than 30 seconds to return. It turns out that the cause of
this is the addition of visible admin_metadata.
There's two problems with this:
1. The original patch probably shouldn't have gone in
data is either admin data or it's not, selectively picking
pieces of admin data out to provide to the user just creates
complications and introduces confusion.
2. The REAL issue here is that since the standar gets are made
with the standard user context, the add_visible_admin_metadata
would go through and do an elevated context get on every single
volume individually. This is what caused the horrible performance
issue on cinder list with large numbers of volumes.
Running as admin, or removing the second call drops this down to about 3
seconds for the same 1000 volume list.
This patch removes the secondary admin context get_call. Instead where
we expect to do display the visible admin_meta, we pass in a flag
requesting that the volume object we're getting has the appropriate
metadata. This way we can elevate the context if needed and avoid
iterating through the gets again.
This patch also cleans up the get_visible_admin_meta methods, and
consolidates both V1 and V2 to use the utils method.
Juan Manuel Olle [Thu, 22 May 2014 13:24:36 +0000 (10:24 -0300)]
Fix a message format error in migration cleanup
In some situations, the volume migration failed. After the cleanup
begin, code was trying to format a string value as an int,
this caused "copy_error = True" not being invoked, resulted
in UnboundLocalError exception.
Yi Chun, Huang [Thu, 29 May 2014 10:10:50 +0000 (06:10 -0400)]
Add support for z/VM driver.
Add 'FCP' into connection_info in initialize_connection
in order to support IBM z/VM hypervisors. This change only
because z/VM need FCP(fibre channel port) info while other
platform don't need it. It will not have side effect
to other platforms since they can safely ignore it.
Code Review / openstack-build / cinder-build.git / log