Thang Pham [Thu, 29 May 2014 03:07:49 +0000 (23:07 -0400)]
GlusterFS: Handle deletion of snapshot with no backing file
If a glusterfs volume is in-use, nova is called to delete a
volume snapshot. It is possible for a timeout to occur, since
cinder is polling for nova task progress. In such case, a
GlusterfsException is thrown, but nova continues to process the
snapshot delete and commits the changes. Cinder is not aware of
this, and the actual snapshot entry in cinder is prevented from
being deleted, since no backing file exists for the out-of-sync
snapshot. This patch allows a snapshot with no backing file to
be deleted from cinder and updates the volume info file.
Eiichi Aikawa [Mon, 19 May 2014 23:43:13 +0000 (08:43 +0900)]
Fixed data copy issue of volume/driver.py
I found data copy issue on __init__ of ISERDriver class.
Some settings for iser should be copied to iscsi's.
But iser_port value was copied not to iscsi_port but to iser_port.
This patch will fix this issue as to be copied to iscsi_port.
Joshua Harlow [Mon, 9 Jun 2014 23:20:31 +0000 (16:20 -0700)]
Remove check_volume_az_zone functor and associated passing
To make it easier to understand why a volume is being rejected
when its availability zone is not allowed remove the functor
that does this check and use the set of valid availability zones
as the master list of valid zones that can be used (and use this
inside the task that validates this data).
Eric Harney [Mon, 14 Apr 2014 18:27:06 +0000 (14:27 -0400)]
Minor cleanups in test_volume
Fix incorrect parameter order for mock decorators. This
just happened to work because they were used in the same way
in test_reserve_volume_success().
Remove incorrect comments. These seem to mix up 'error' versus
'error_deleting' for expected states -- but the assertEqual args
are sufficiently explanatory by themselves.
Sasikanth [Mon, 23 Jun 2014 14:10:12 +0000 (19:40 +0530)]
Update _resize_volume_file() to support appropriate permissions
ibmnas driver inherits _set_rw_permissions_for_all() from nfs.py
which sets 666 to the volumes. Changes are expected to be made in
nfs.py such that it sets 600 or 660 permissions, by doing so
ibmnas driver fails performing the operation _resize_volume_file()
While creating a volume from stream-optimized image, the adapter type
in the image meta-data and the profile in the volume type extra spec
are ignored. This change fix those missing parameters.
The current create backing methods do not support specifying an adapter
type for the backing VM. These methods always create a backing VM with
a single disk and LSI logic adapter. This change adds optional parameters
to create backing methods so that a backing VM can be created without a
disk or with a specific adapter type.
When creating a volume from an image, after the volume is created,
qemu-img is supposed to copy the downloaded image to the volume.
This fails as the disk is enabled and is not accessible.
Also, Windows does not support using dynamic vhds as iSCSI disks
and as qemu-img cannot create fixed vhd images, there must be an
intermediate conversion before importing the disk.
In order to be able to modify an iSCSI disk, it must be disabled
first.
Eric Guo [Mon, 30 Dec 2013 11:36:49 +0000 (19:36 +0800)]
Use oslo-incubator module units
There is Cinder's version units module, that looks good. Considering
consistency with other projects like Nova, Glance, Cinder would be
better to use oslo-incubator's version.
Joshua Harlow [Sat, 21 Jun 2014 01:54:55 +0000 (18:54 -0700)]
Attach log listeners to other engines
Attach the created task/flow/engine listener to
the other usages of taskflow that exist in cinder
so that those locations can also benefit from
the same logging of state and activity.
John Griffith [Wed, 11 Jun 2014 22:43:41 +0000 (22:43 +0000)]
Remove global conf settings from iscsi helper
This "intermediate" iscsi helper pulls all of it's
config settings from the global config. This is fine
if you only have a single backend, but if you do
multi-backend it puts things in a bad state where some
of the backend specific settings are picked up but
others are not (for example iscsi_ip_address).
This change modifies methods like create_export in
the volume/iscsi helper to take the drivers version
of the config settings as a parameter and use those
instead of setting off of the global values.
Long term there's a lot of cleanup surrounding our
inheritance model and especially the iscsi helpers.
We can address that going forward but here we just want
to fix the bug in the safest way possible.
Michael Berlin [Mon, 19 May 2014 15:28:54 +0000 (17:28 +0200)]
GlusterFS: Various unit test improvements
Fixed bug that "_ensure_share_writable" was not mocked and therefore
"GlusterfsException" was thrown earlier than expected with an unexpected
error message.
To better guard against situations like this, I've added a helper method
"assertRaisesAndMessageMatches" to test for the thrown exception AND its
message.
Added "mox.VerifyAll()" where it was missing and cleaned up unused
mocks.
Fixed test for "create_volume_from_snapshot" (which actually tested
"create_cloned_volume").
Added tests for driver function "copy_volume_to_image".
Overall, code coverage for volume/drivers/glusterfs.py improved from 74%
to 82%.
vmware: Force chunked transfer for upload-to-image
The upload-to-image operation downloads (using stream-optimized HTTP NFC
transfer) the virtual disk corresponding to the volume to a pipe from
where the glance image client reads the data for upload. Due to a recent
change in the glance image client, success in seeking the input file
object results in forgoing the chunked transfer in favor of regular
transfer. The glance image client expects an IOError if the input file
object is a pipe. Currently the pipe seek() is a NOP and this results in
upload-to-image failure. This changes fixes such failures by throwing
an IOError indicating an illegal seek.
Victor Sergeyev [Fri, 28 Feb 2014 09:20:37 +0000 (11:20 +0200)]
Sync the latest common db code from oslo
This sync brings the latest openstack/common/db code from
olso-incubator.
Changes in application code caused by API changes in openstack.common.db -
oslo.db no longer stores SQLAlchemy Engine and sessionmaker instances
globally and it's up to applications to create them. So we should add
methods for work with engine and session to Cinder.
List of changes:
54f7e7f Prevent races in opportunistic db test cases 8a0f581 Use oslotest instead of common test module 4a591ea Start ping listener also for postgresql f0e50ed Add a warning to not use get_table for working with ForeignKeys 2fd457b Ignore migrate versioning tables in utf8 sanity check 9fed4ed Fix Keystone doc build errors with SQLAlchemy 0.9 f7705f3 Make table utf-8 charset checking be optional for DB migration 5b7e61c Dispose db connections pool on disconnect 295fcd9 Do not use the 'extend' method on a dict_items object d1988b9 Set sql_mode callback on connect instead of checkout a1a8280 Fix excessive logging from db.sqlalchemy.session dc2d829 Add lockutils fixture to OpportunisticTestCase d10f871 Adapt DB provisioning code for CI requirements 5920bed Make db utils importable without migrate 9933bdd Get mysql_sql_mode parameter from config 96a2217 Prevent incorrect usage of _wrap_db_error() 6cab37c Python3: define a __next__() method for ModelBase 20a7510 Add from_config() method to EngineFacade 7959826 db: move all options into database group fea119e Drop special case for MySQL traditional mode, update unit tests a584166 Make TRADITIONAL the default SQL mode dda24eb Introduce mysql_sql_mode option, remove old warning 0b5af67 Introduce a method to set any MySQL session SQL mode 8dccc7b Handle ibm_db_sa DBDuplicateEntry integrity errors 5b9e9f4 Fix doc build errors in db.sqlalchemy 0f24d82 Fix migration.db_version when no tables ac84a40 Update log translation domains c0d357b Add model_query() to db.sqlalchemy.utils module 84254fc Fix a small typo in api.py b8a676c Remove CONF.database.connection default value 86707cd Remove None for dict.get() 0545121 Fix duplicating of SQL queries in logs fcf517d Update oslo log messages with translation domains fa05b7c Restore the ability to load the DB backend lazily 630d395 Don't use cfg.CONF in oslo.db ce69e7f Don't store engine instances in oslo.db 35dc1d7 py3kcompat: remove b4f72b2 Don't raise MySQL 2013 'Lost connection' errors 271adfb Format sql in db.sqlalchemy.session docstring 0334cb3 Handle exception messages with six.text_type eff69ce Drop dependency on log from oslo db code 7a11a04 Automatic retry db.api query if db connection lost 11f2add Clean up docstring in db.sqlalchemy.session 1b5147f Only enable MySQL TRADITIONAL mode if we're running against MySQL 39e1c5c Move db tests base.py to common code 986dafd Fix parsing of UC errors in sqlite 3.7.16+/3.8.2+ 9a203e6 Use dialect rather than a particular DB API driver 1779029 Move helper DB functions to db.sqlalchemy.utils bcf6d5e Small edits on help strings ae01e9a Transition from migrate to alembic 70ebb19 Fix mocking of utcnow() for model datetime cols 7aa94df Add a db check for CHARSET=utf8 aff0171 Remove "vim: tabstop=4 shiftwidth=4 softtabstop=4" from headers fa0f36f Fix database connection string is secret 8575d87 Removed copyright from empty files d08d27f Fix the obsolete exception message 8b2b0b7 Use hacking import_exceptions for gettextutils._ 9bc593e Add docstring for exception handlers of session 855644a Removal of _REPOSITORY global variable. ea6caf9 Remove string.lowercase usage a33989e Remove eventlet tpool from common db.api e40903b Database hook enabling traditional mode at MySQL f2115a0 Replace xrange in for loop with range c802fa6 SQLAlchemy error patterns improved 1c1f199 Remove unused import 6d0a6c3 Correct invalid docstrings 135dd00 Remove start index 0 in range() 28f8fd5 Make _extra_keys a property of ModelBase 45658e2 Fix violations of H302:import only modules bb4d7a2 Enables db2 server disconnects to be handled pessimistically 915f8ab db.sqlalchemy.session add [sql].idle_timeout e6494c2 Use six.iteritems to make dict work on Python2/3 48cfb7b Drop dependency on processutils from oslo db code 4c47d3e Fix locking in migration tests c2ee282 Incorporating MIT licensed code c5a1088 Typos fix in db and periodic_task module fb0e86a Use six.moves.configparser instead of ConfigParser 1dd4971 fix typo in db session docstring 8a01dd8 The ability to run tests at various backend 0fe4e28 Use log.warning() instead of log.warn() in oslo.db 12bcdb7 Remove vim header 4c22556 Use py3kcompat urlutils functions instead of urlparse ca7a2ab Don't use deprecated module commands 6603e8f Remove sqlalchemy-migrate 0.7.3 patching 274c7e2 Drop dependency on lockutils from oslo db code 97d8cf4 Remove lazy loading of database backend 2251cb5 Do not name variables as builtins 3acd57c Add db2 communication error code when check the db connection c2dcf6e Add [sql].connection as deprecated opt for db 001729d Modify SQLA session due to dispose of eventlet 4de827a Clean up db.sqla.Models.extra_keys interface 347f29e Use functools.wrap() instead of custom implementation 771d843 Move base migration test classes to common code 9721129 exception: remove 56ff3b3 Use single meta when change column type 3f2f70e Helper function to sanitize db url credentials df3f2ba BaseException.message is deprecated since Python 2.6 c76be5b Add function drop_unique_constraint() d4d8126 Change sqlalchemy/utils.py mode back to 644 cf41936 Move sqlalchemy migration from Nova 5758360 Raise ValueError if sort_dir is unknown 31c1995 python3: Add python3 compatibility support 3972c3f Migrate sqlalchemy utils from Nova 1a2df89 Enable H302 hacking check 3f503fa Add a monkey-patching util for sqlalchemy-migrate 7ba5f4b Don't use mixture of cfg.Opt() deprecated args 489e2b7 Ensure that DB configuration is backward compatible 7119e29 Enable hacking H404 test. ebaa578 Enable user to configure pool_timeout 30cb47e Changed processing unique constraint name. 6d27681 Enable H306 hacking check. 444bdbc Add a slave db handle for the SQLAlchemy backend. 484a1df Enable hacking H403 test 4ff33b0 Specify database group instead of DEFAULT
Zhiteng Huang [Fri, 20 Jun 2014 03:48:28 +0000 (11:48 +0800)]
Fix order dependency of admin metadata
Some test cases in VolumeTestCase depends on the order of admin metadata
returned from db.volume_get() call, which causes jenkins test failure in
recent reviews. This change extracts the key/value we are intertested in
and compare them with AssertDictMatch() to remove the order dependency.
Eric Harney [Thu, 19 Jun 2014 16:57:51 +0000 (12:57 -0400)]
GlusterFS tests: Mock out compute, don't load novaclient
Mock out the compute library for these tests, which prevents loading
the novaclient library.
This makes these tests more reliable, as currently the novaclient
lib interferes with our mocking of os.path.exists in these tests
in some test configurations.
This should also make the GlusterFS tests slightly more efficient
since they were previously loading the novaclient python library
but did not really need to.
According to the OpenStack translation policy available at
https://wiki.openstack.org/wiki/LoggingStandards debug messages
should not be translated. Like mentioned in several changes in
Nova by garyk this is to help prioritize log translation.
This patch adds a new hacking check - N319 - that ensures all
debug log messages don't have translations.
Joshua Harlow [Mon, 9 Jun 2014 22:32:17 +0000 (15:32 -0700)]
Handle the case where az is disabled/removed
Instead of retaining a local in-memory list of
availability zones that is never purged and therefore
never updates itself if availability zones are removed
we should instead cache those zones for a given period
of time (defaulting to one hour) and on cache expiry
refetch the enabled availability zones.
DocImpact: adds a new configuration that defines the
default availability zone cache expiry time in seconds.
This will enable the engine that runs the create volume
request to in a future change remove the need to have
a functor which proxies the validation logic around
availability zones.
Tomoki Sekiyama [Fri, 13 Jun 2014 22:49:24 +0000 (18:49 -0400)]
I/O rate limit for volume copy with qemu-img convert
Currently, volume copy operations consumes disk I/O bandwidth heavily and
may slow down the other guest instances. This patch limits bandwidth for
volume copy to mitigate interference to other instance performance.
In this implementation, 'qemu-img' is put in a blkio cgroup for throttling,
when CONF.volume_copy_bps_limit is set to non-zero.
Tomoki Sekiyama [Wed, 7 May 2014 20:52:20 +0000 (16:52 -0400)]
I/O rate limit for volume copy with dd
Currently, volume copy operations consumes disk I/O bandwidth heavily and
may slow down the other guest instances. This patch limits bandwidth for
volume copy to mitigate interference to other instance performance.
In this implementation, 'dd' is put in a blkio cgroup for throttling,
when CONF.volume_copy_bps_limit is set to non-zero.
glusterfs: Honor mount options when restarting cinder service
When restarting cinder service (cinder-volume specifically),
ensure that the gluster mounts are unmounted and remounted so
that any new mount options added to the shares config file
is taken effect, post service restart.
abhishekkekane [Tue, 10 Jun 2014 06:29:20 +0000 (06:29 +0000)]
Made provision for providing optional arguments
The 'quota_committed' attribute of 'RequestContext' object is
a transient property, so it will not be saved in the taskflow
persistent storage. The updated value of 'quota_committed'
attribute will not be available while resuming/reverting the
flow, if cinder api-service is down/stopped after committing
the quota.
Since this 'quota_committed' attribute is not used anywhere
in cinder project other than in create-volume taskflow api, so
removed 'quota_committed' from RequestContext and made
provision to pass it as an optional argument which will be
passed to api-flow via create_what dictionary, in order to
make it persistent and use it as and when needed.
Joshua Harlow [Thu, 12 Jun 2014 21:35:35 +0000 (14:35 -0700)]
Add task/flow listener support around the volume api flow
This will add in logging of what is occurring around the
actions taken by taskflow when taskflow executes the initial
create_volume workflow.
Since the oslo logging module will include the request_id of
the currently active request (per greenthread) this will make
it easier to understand and debug the actions occurring inside
taskflow's engine concept.
Dirk Mueller [Thu, 12 Jun 2014 22:24:23 +0000 (00:24 +0200)]
Retry lvremove with ignore_suspended_devices
A lvremove -f might leave behind suspended devices
when it is racing with udev or other processes
still accessing any of the device files. The previous
solution of using lvchange -an on the LV had the
side-effect of deactivating origin LVs alongway in
the thick volume case, which was undesired.
It turns out retrying the deactivation twice and
ignoring the suspended devices on the second iteration
avoids the hang of all LVM operations after an initial
failure.
The reset-state API call only sets that main status
column on the volume object. There's also a method
to set attach status, however it turns out that due
to the implementation of the status validation method
in the parent class, this was impossible to use because
it only checked and accepted "status" NOT "attach_status".
This patch fixes that attach_status problem, it also implements
the ability to update the migration status. A cinderclient change
will be needed to expose these calls as well.
Unmesh Gurjar [Mon, 2 Jun 2014 13:14:55 +0000 (18:44 +0530)]
vmware: Fixes VMDK volume incompatibility issue
VMDK volume created in a cluster having ESXi 5.1 host has a high
hardware version i.e. 9; hence, fails when attached to instance
on an ESXi 5.0 host. This patch sets the hardware version to 8,
so that the volume can be attached to instances on any ESXi host
starting version 5.0 and above.
Remove unused oslo module and adjust opentstack-common.conf
The issues are detected by new change about update.py
in I68c0108304be8a315bcefe939c22019caab9bc98. This patch
doesn't depend the oslo-incubator patch.
* remove unused module crypto
* add missing module loopingcall in openstack-common.conf
* sort the modules in openstack-common.conf
PranaliDeore [Tue, 6 May 2014 04:04:45 +0000 (04:04 +0000)]
Copy custom properties to image from volume
Presently after copying an image to volume, all properties
of the image are getting copied properly but while creating
image back from volume, it doesn't copy custom properties
to the image.
At present in volume-glance-metadata table all the properties
of volume are stored as key and value. Because of this it is
difficult to differentiate between core and custom properties.
To overcome this, I have added a new option
'glance_core_properties' in cinder.conf. This option defines
all core properties of an image. This way, it's easy to separate
core and custom properties from the glance_volume_metadata and add
custom property to the newly created image.
Joshua Harlow [Tue, 10 Jun 2014 02:02:42 +0000 (19:02 -0700)]
Fix log capturing fixture
Instead of nuking the unit test logs capture them according
to how tempest (and oslotest) capture them so that the tests
when ran locally provide the full logging output instead of
a miniaturized version.
This adds in the following environment variables which can
be used to control testr captured logging:
- OS_LOG_CAPTURE (turns on log capturing so that the testr
repository will have log output, at a INFO level)
- OS_DEBUG (changes the captured logging level from INFO
to DEBUG, only useful when OS_LOG_CAPTURE is enabled)
It also creates a lambda that is used to test whether an
environment is enabled (avoiding recreating the same set
of checks in multiple places).
Fix Brocade FC driver's use of the fc_fabric_names
The Brocade Fibre Channel Zone Manager driver wasn't
stripping out whitespace in the fc_fabric_names
prior to using them. This would cause breakage later on
in the driver.
Eric Harney [Tue, 10 Jun 2014 15:25:01 +0000 (11:25 -0400)]
LIO: Don't add ACL for local initiator name
Currently cinder-rtstool requires /etc/iscsi/initiatorname.iscsi
to be present in order to create a LUN/ACL/portal successfully.
However, this is not required since Cinder will create the
required ACLs dynamically at attach time as well.
Removing this path means that iscsi-initiator-utils no longer
needs to be installed locally when using a remote Nova compute
node.
On upload-to-image failure before initiating the data transfer or during
data transfer, the source volume status is restored properly whereas the
image created remains in queued or saving state. This change deletes the
image during such failures.
This patch flushes IO for each SCSI device that
brick removes from the system. The patch also
flushes a multipath device prior to removing
each underlying LUN associated with the
multipath device. We were seeing kernel
IO errors in the syslog without calling flush
on the block device, during a detach for a
multipath FibreChannel copy image to volume
action.
Code Review / openstack-build / cinder-build.git / log