Matt Riedemann [Fri, 27 Jun 2014 16:50:56 +0000 (09:50 -0700)]
Sync processutils from oslo with deps
The main target for the sync is to pick up the get_worker_count()
method in commit 85f178489a128a04a7ee3ed44018403caa109ef0 so that we can
set osapi_volume_workers equal to the number of CPUs on the host.
Changes:
processutils
------------ 85f1784 Move nova.utils.cpu_count() to processutils module cdcc19c Mask passwords that are included in commands 8a0f567 Remove str() from LOG.* and exceptions 51778f9 Allow passing environment variables to execute() fcf517d Update oslo log messages with translation domains af41592 Catch OSError in processutils f773ea2 Fix i18n problem in processutils module 8b2b0b7 Use hacking import_exceptions for gettextutils._ 3b71f46 Fixed misspellings of common words 12bcdb7 Remove vim header a4dab73 Correct execute() to check 0 in check_exit_code
importutils
----------- 1173e46 Remove ValueError when accessing sys.modules
jsonutils
--------- 0d7296f Add kwargs to jsonutils.load(s) functions
log
--- 109e325 Use oslo.messaging to publish log errors de4adbc pep8: fixed multiple violations eac71f5 Fix common.log.ContextFormatter for Python 3 d78b633 Fixes a simple spelling mistake 621d831 always log a traceback in the sys.excepthook 90ae24b Remove redundant default=None for config options af36c2a Fix logging setup for Python 3.4 cdcc19c Mask passwords that are included in commands
strutils
-------- 8a0f567 Remove str() from LOG.* and exceptions fd18c28 Fix safe_encode(): return bytes on Python 3 302c7c8 strutils: Allow safe_{encode,decode} to take bytes as input
timeutils
--------- 250cd88 Fixed a new pep8 error and a small typo
The following changes are included with this patch:
-> 18f2bc1 Enforce unicode json output for jsonutils.load[s]()
--> cinder/openstack/common/jsonutils.py
Zhiteng Huang [Tue, 24 Jun 2014 07:59:10 +0000 (15:59 +0800)]
Bump minimum hacking version to 0.9.2
Take advantage of newer version hacking to automate a few style checkings.
A change included this hacking update was proposed by OpenStack Proposal Bot
in earlier patch sets (before 19) but somehow it was missed in latter
proposals. (https://review.openstack.org/#/c/96206/)
Modified/New rules in 0.9.2:
Rule changes:
* Report E129 instead of E125 for visually indented line with same indent
* as next logical line.
* Report E265 for space before block comment.
* Report E713 and E714 when operators ``not in`` and ``is not`` are
recommended (taken from hacking).
* Report E131 instead of E121 / E126 if the hanging indent is not consistent
within the same continuation block. It helps when error E121 or E126 is
in the ``ignore`` list.
* Report E126 instead of E121 when the continuation line is hanging with
extra indentation, even if indentation is not a multiple of 4.
- Dropped H901,H902 as those are now in pep8 and enforced by E713 and E714
New rules:
+ H104 File contains nothing but comments
+ H305 imports not grouped correctly
+ H307 like imports should be grouped together
+ H405 multi line docstring summary not separated with an empty line
+ H904 Wrap long lines in parentheses instead of a backslash
These rules (including changed rules) are all disabled in this change if they
caues any pep8 checking failure, and will be turned on in follow-up patches
if considered useful.
Original commit message in openstack/requirements:
Hacking 0.9.1 was out for a while (hacking 0.9.0 had a minor bug in it, so its
blacklisted), and we want to start supporting projects to migrate over
to 0.9.x. Expand the upper bound to allow 0.9.x while still
supporting 0.8.x series so we keep the old version in our mirrors as to
not break anything.
Eric Harney [Thu, 10 Apr 2014 16:57:50 +0000 (12:57 -0400)]
Fix dropped exception for create_export in vol manager
If self.driver.create_export() fails, model_update is
False and this exception is therefore not logged,
causing things to break later in the execution path in
unexpected ways.
Tomoki Sekiyama [Wed, 25 Jun 2014 21:22:32 +0000 (17:22 -0400)]
Add cgroups related commands to rootwrap filters
Adds rootwrap filters to allow commands to create/setup cgroups and
to put commands into cgroups in order to limit bandwidth usage,
that are used in setup_blkio_cgroup() (volume/utils.py).
Joshua Harlow [Wed, 25 Jun 2014 18:24:02 +0000 (11:24 -0700)]
Use a task subclass instead of a functor + task wrapper
Instead of using a functor that activates the scheduler driver
instead incorporate and use a task subclass, this avoids the less
meaningful information that is presented when a functor is used
as well as allows for a more straightforward path for future
subclassing and derivations of work.
Thang Pham [Thu, 29 May 2014 03:07:49 +0000 (23:07 -0400)]
GlusterFS: Handle deletion of snapshot with no backing file
If a glusterfs volume is in-use, nova is called to delete a
volume snapshot. It is possible for a timeout to occur, since
cinder is polling for nova task progress. In such case, a
GlusterfsException is thrown, but nova continues to process the
snapshot delete and commits the changes. Cinder is not aware of
this, and the actual snapshot entry in cinder is prevented from
being deleted, since no backing file exists for the out-of-sync
snapshot. This patch allows a snapshot with no backing file to
be deleted from cinder and updates the volume info file.
Eiichi Aikawa [Mon, 19 May 2014 23:43:13 +0000 (08:43 +0900)]
Fixed data copy issue of volume/driver.py
I found data copy issue on __init__ of ISERDriver class.
Some settings for iser should be copied to iscsi's.
But iser_port value was copied not to iscsi_port but to iser_port.
This patch will fix this issue as to be copied to iscsi_port.
Joshua Harlow [Mon, 9 Jun 2014 23:20:31 +0000 (16:20 -0700)]
Remove check_volume_az_zone functor and associated passing
To make it easier to understand why a volume is being rejected
when its availability zone is not allowed remove the functor
that does this check and use the set of valid availability zones
as the master list of valid zones that can be used (and use this
inside the task that validates this data).
Eric Harney [Mon, 14 Apr 2014 18:27:06 +0000 (14:27 -0400)]
Minor cleanups in test_volume
Fix incorrect parameter order for mock decorators. This
just happened to work because they were used in the same way
in test_reserve_volume_success().
Remove incorrect comments. These seem to mix up 'error' versus
'error_deleting' for expected states -- but the assertEqual args
are sufficiently explanatory by themselves.
Sasikanth [Mon, 23 Jun 2014 14:10:12 +0000 (19:40 +0530)]
Update _resize_volume_file() to support appropriate permissions
ibmnas driver inherits _set_rw_permissions_for_all() from nfs.py
which sets 666 to the volumes. Changes are expected to be made in
nfs.py such that it sets 600 or 660 permissions, by doing so
ibmnas driver fails performing the operation _resize_volume_file()
While creating a volume from stream-optimized image, the adapter type
in the image meta-data and the profile in the volume type extra spec
are ignored. This change fix those missing parameters.
The current create backing methods do not support specifying an adapter
type for the backing VM. These methods always create a backing VM with
a single disk and LSI logic adapter. This change adds optional parameters
to create backing methods so that a backing VM can be created without a
disk or with a specific adapter type.
When creating a volume from a snapshot on Windows, a shadow copy
volume is exported as an iSCSI disk. The issue is that this export
is readonly and cannot be mounted to instances.
As this export cannot be modified, to make the new volume usable,
the according image must be moved to a new path and imported .
Fixes cinder volume create on Windows Server 2012 R2
Windows Server 2012 R2 does not support vhd images as iSCSI disks,
requiring VHDX images. For this reason, the cinder driver fails to
create volumes. For the moment, the default format is vhd.
On WSS 2012 R2, we should use vhdx images as default.
This patch introduces vhd/vhdx related methods for both v1 and v2
wmi virtualization namespaces.
When creating a volume from an image, after the volume is created,
qemu-img is supposed to copy the downloaded image to the volume.
This fails as the disk is enabled and is not accessible.
Also, Windows does not support using dynamic vhds as iSCSI disks
and as qemu-img cannot create fixed vhd images, there must be an
intermediate conversion before importing the disk.
In order to be able to modify an iSCSI disk, it must be disabled
first.
Eric Guo [Mon, 30 Dec 2013 11:36:49 +0000 (19:36 +0800)]
Use oslo-incubator module units
There is Cinder's version units module, that looks good. Considering
consistency with other projects like Nova, Glance, Cinder would be
better to use oslo-incubator's version.
Joshua Harlow [Sat, 21 Jun 2014 01:54:55 +0000 (18:54 -0700)]
Attach log listeners to other engines
Attach the created task/flow/engine listener to
the other usages of taskflow that exist in cinder
so that those locations can also benefit from
the same logging of state and activity.
Juan Manuel Olle [Mon, 28 Apr 2014 18:09:29 +0000 (15:09 -0300)]
Adding filter options to backup list
This patch adds filter options to backup list.
Like cinder list, now cinder backup-list accept
parameters.
This blueprint is complimented with the client part
John Griffith [Wed, 11 Jun 2014 22:43:41 +0000 (22:43 +0000)]
Remove global conf settings from iscsi helper
This "intermediate" iscsi helper pulls all of it's
config settings from the global config. This is fine
if you only have a single backend, but if you do
multi-backend it puts things in a bad state where some
of the backend specific settings are picked up but
others are not (for example iscsi_ip_address).
This change modifies methods like create_export in
the volume/iscsi helper to take the drivers version
of the config settings as a parameter and use those
instead of setting off of the global values.
Long term there's a lot of cleanup surrounding our
inheritance model and especially the iscsi helpers.
We can address that going forward but here we just want
to fix the bug in the safest way possible.
Michael Berlin [Mon, 19 May 2014 15:28:54 +0000 (17:28 +0200)]
GlusterFS: Various unit test improvements
Fixed bug that "_ensure_share_writable" was not mocked and therefore
"GlusterfsException" was thrown earlier than expected with an unexpected
error message.
To better guard against situations like this, I've added a helper method
"assertRaisesAndMessageMatches" to test for the thrown exception AND its
message.
Added "mox.VerifyAll()" where it was missing and cleaned up unused
mocks.
Fixed test for "create_volume_from_snapshot" (which actually tested
"create_cloned_volume").
Added tests for driver function "copy_volume_to_image".
Overall, code coverage for volume/drivers/glusterfs.py improved from 74%
to 82%.
vmware: Force chunked transfer for upload-to-image
The upload-to-image operation downloads (using stream-optimized HTTP NFC
transfer) the virtual disk corresponding to the volume to a pipe from
where the glance image client reads the data for upload. Due to a recent
change in the glance image client, success in seeking the input file
object results in forgoing the chunked transfer in favor of regular
transfer. The glance image client expects an IOError if the input file
object is a pipe. Currently the pipe seek() is a NOP and this results in
upload-to-image failure. This changes fixes such failures by throwing
an IOError indicating an illegal seek.
Victor Sergeyev [Fri, 28 Feb 2014 09:20:37 +0000 (11:20 +0200)]
Sync the latest common db code from oslo
This sync brings the latest openstack/common/db code from
olso-incubator.
Changes in application code caused by API changes in openstack.common.db -
oslo.db no longer stores SQLAlchemy Engine and sessionmaker instances
globally and it's up to applications to create them. So we should add
methods for work with engine and session to Cinder.
List of changes:
54f7e7f Prevent races in opportunistic db test cases 8a0f581 Use oslotest instead of common test module 4a591ea Start ping listener also for postgresql f0e50ed Add a warning to not use get_table for working with ForeignKeys 2fd457b Ignore migrate versioning tables in utf8 sanity check 9fed4ed Fix Keystone doc build errors with SQLAlchemy 0.9 f7705f3 Make table utf-8 charset checking be optional for DB migration 5b7e61c Dispose db connections pool on disconnect 295fcd9 Do not use the 'extend' method on a dict_items object d1988b9 Set sql_mode callback on connect instead of checkout a1a8280 Fix excessive logging from db.sqlalchemy.session dc2d829 Add lockutils fixture to OpportunisticTestCase d10f871 Adapt DB provisioning code for CI requirements 5920bed Make db utils importable without migrate 9933bdd Get mysql_sql_mode parameter from config 96a2217 Prevent incorrect usage of _wrap_db_error() 6cab37c Python3: define a __next__() method for ModelBase 20a7510 Add from_config() method to EngineFacade 7959826 db: move all options into database group fea119e Drop special case for MySQL traditional mode, update unit tests a584166 Make TRADITIONAL the default SQL mode dda24eb Introduce mysql_sql_mode option, remove old warning 0b5af67 Introduce a method to set any MySQL session SQL mode 8dccc7b Handle ibm_db_sa DBDuplicateEntry integrity errors 5b9e9f4 Fix doc build errors in db.sqlalchemy 0f24d82 Fix migration.db_version when no tables ac84a40 Update log translation domains c0d357b Add model_query() to db.sqlalchemy.utils module 84254fc Fix a small typo in api.py b8a676c Remove CONF.database.connection default value 86707cd Remove None for dict.get() 0545121 Fix duplicating of SQL queries in logs fcf517d Update oslo log messages with translation domains fa05b7c Restore the ability to load the DB backend lazily 630d395 Don't use cfg.CONF in oslo.db ce69e7f Don't store engine instances in oslo.db 35dc1d7 py3kcompat: remove b4f72b2 Don't raise MySQL 2013 'Lost connection' errors 271adfb Format sql in db.sqlalchemy.session docstring 0334cb3 Handle exception messages with six.text_type eff69ce Drop dependency on log from oslo db code 7a11a04 Automatic retry db.api query if db connection lost 11f2add Clean up docstring in db.sqlalchemy.session 1b5147f Only enable MySQL TRADITIONAL mode if we're running against MySQL 39e1c5c Move db tests base.py to common code 986dafd Fix parsing of UC errors in sqlite 3.7.16+/3.8.2+ 9a203e6 Use dialect rather than a particular DB API driver 1779029 Move helper DB functions to db.sqlalchemy.utils bcf6d5e Small edits on help strings ae01e9a Transition from migrate to alembic 70ebb19 Fix mocking of utcnow() for model datetime cols 7aa94df Add a db check for CHARSET=utf8 aff0171 Remove "vim: tabstop=4 shiftwidth=4 softtabstop=4" from headers fa0f36f Fix database connection string is secret 8575d87 Removed copyright from empty files d08d27f Fix the obsolete exception message 8b2b0b7 Use hacking import_exceptions for gettextutils._ 9bc593e Add docstring for exception handlers of session 855644a Removal of _REPOSITORY global variable. ea6caf9 Remove string.lowercase usage a33989e Remove eventlet tpool from common db.api e40903b Database hook enabling traditional mode at MySQL f2115a0 Replace xrange in for loop with range c802fa6 SQLAlchemy error patterns improved 1c1f199 Remove unused import 6d0a6c3 Correct invalid docstrings 135dd00 Remove start index 0 in range() 28f8fd5 Make _extra_keys a property of ModelBase 45658e2 Fix violations of H302:import only modules bb4d7a2 Enables db2 server disconnects to be handled pessimistically 915f8ab db.sqlalchemy.session add [sql].idle_timeout e6494c2 Use six.iteritems to make dict work on Python2/3 48cfb7b Drop dependency on processutils from oslo db code 4c47d3e Fix locking in migration tests c2ee282 Incorporating MIT licensed code c5a1088 Typos fix in db and periodic_task module fb0e86a Use six.moves.configparser instead of ConfigParser 1dd4971 fix typo in db session docstring 8a01dd8 The ability to run tests at various backend 0fe4e28 Use log.warning() instead of log.warn() in oslo.db 12bcdb7 Remove vim header 4c22556 Use py3kcompat urlutils functions instead of urlparse ca7a2ab Don't use deprecated module commands 6603e8f Remove sqlalchemy-migrate 0.7.3 patching 274c7e2 Drop dependency on lockutils from oslo db code 97d8cf4 Remove lazy loading of database backend 2251cb5 Do not name variables as builtins 3acd57c Add db2 communication error code when check the db connection c2dcf6e Add [sql].connection as deprecated opt for db 001729d Modify SQLA session due to dispose of eventlet 4de827a Clean up db.sqla.Models.extra_keys interface 347f29e Use functools.wrap() instead of custom implementation 771d843 Move base migration test classes to common code 9721129 exception: remove 56ff3b3 Use single meta when change column type 3f2f70e Helper function to sanitize db url credentials df3f2ba BaseException.message is deprecated since Python 2.6 c76be5b Add function drop_unique_constraint() d4d8126 Change sqlalchemy/utils.py mode back to 644 cf41936 Move sqlalchemy migration from Nova 5758360 Raise ValueError if sort_dir is unknown 31c1995 python3: Add python3 compatibility support 3972c3f Migrate sqlalchemy utils from Nova 1a2df89 Enable H302 hacking check 3f503fa Add a monkey-patching util for sqlalchemy-migrate 7ba5f4b Don't use mixture of cfg.Opt() deprecated args 489e2b7 Ensure that DB configuration is backward compatible 7119e29 Enable hacking H404 test. ebaa578 Enable user to configure pool_timeout 30cb47e Changed processing unique constraint name. 6d27681 Enable H306 hacking check. 444bdbc Add a slave db handle for the SQLAlchemy backend. 484a1df Enable hacking H403 test 4ff33b0 Specify database group instead of DEFAULT
Zhiteng Huang [Fri, 20 Jun 2014 03:48:28 +0000 (11:48 +0800)]
Fix order dependency of admin metadata
Some test cases in VolumeTestCase depends on the order of admin metadata
returned from db.volume_get() call, which causes jenkins test failure in
recent reviews. This change extracts the key/value we are intertested in
and compare them with AssertDictMatch() to remove the order dependency.
Eric Harney [Thu, 19 Jun 2014 16:57:51 +0000 (12:57 -0400)]
GlusterFS tests: Mock out compute, don't load novaclient
Mock out the compute library for these tests, which prevents loading
the novaclient library.
This makes these tests more reliable, as currently the novaclient
lib interferes with our mocking of os.path.exists in these tests
in some test configurations.
This should also make the GlusterFS tests slightly more efficient
since they were previously loading the novaclient python library
but did not really need to.
According to the OpenStack translation policy available at
https://wiki.openstack.org/wiki/LoggingStandards debug messages
should not be translated. Like mentioned in several changes in
Nova by garyk this is to help prioritize log translation.
This patch adds a new hacking check - N319 - that ensures all
debug log messages don't have translations.
Joshua Harlow [Mon, 9 Jun 2014 22:32:17 +0000 (15:32 -0700)]
Handle the case where az is disabled/removed
Instead of retaining a local in-memory list of
availability zones that is never purged and therefore
never updates itself if availability zones are removed
we should instead cache those zones for a given period
of time (defaulting to one hour) and on cache expiry
refetch the enabled availability zones.
DocImpact: adds a new configuration that defines the
default availability zone cache expiry time in seconds.
This will enable the engine that runs the create volume
request to in a future change remove the need to have
a functor which proxies the validation logic around
availability zones.
Tomoki Sekiyama [Fri, 13 Jun 2014 22:49:24 +0000 (18:49 -0400)]
I/O rate limit for volume copy with qemu-img convert
Currently, volume copy operations consumes disk I/O bandwidth heavily and
may slow down the other guest instances. This patch limits bandwidth for
volume copy to mitigate interference to other instance performance.
In this implementation, 'qemu-img' is put in a blkio cgroup for throttling,
when CONF.volume_copy_bps_limit is set to non-zero.
Tomoki Sekiyama [Wed, 7 May 2014 20:52:20 +0000 (16:52 -0400)]
I/O rate limit for volume copy with dd
Currently, volume copy operations consumes disk I/O bandwidth heavily and
may slow down the other guest instances. This patch limits bandwidth for
volume copy to mitigate interference to other instance performance.
In this implementation, 'dd' is put in a blkio cgroup for throttling,
when CONF.volume_copy_bps_limit is set to non-zero.
glusterfs: Honor mount options when restarting cinder service
When restarting cinder service (cinder-volume specifically),
ensure that the gluster mounts are unmounted and remounted so
that any new mount options added to the shares config file
is taken effect, post service restart.
KIYOHIRO ADACHI [Fri, 27 Dec 2013 03:16:37 +0000 (12:16 +0900)]
Add keyword argument missing at some exc.HTTPError subclass
Some exc.HTTPError subclass needs keyword argument ``explanation``.
If there is no ``explanation=``, the message of the argument is
recognized as ``detail``.
abhishekkekane [Tue, 10 Jun 2014 06:29:20 +0000 (06:29 +0000)]
Made provision for providing optional arguments
The 'quota_committed' attribute of 'RequestContext' object is
a transient property, so it will not be saved in the taskflow
persistent storage. The updated value of 'quota_committed'
attribute will not be available while resuming/reverting the
flow, if cinder api-service is down/stopped after committing
the quota.
Since this 'quota_committed' attribute is not used anywhere
in cinder project other than in create-volume taskflow api, so
removed 'quota_committed' from RequestContext and made
provision to pass it as an optional argument which will be
passed to api-flow via create_what dictionary, in order to
make it persistent and use it as and when needed.
Joshua Harlow [Thu, 12 Jun 2014 21:35:35 +0000 (14:35 -0700)]
Add task/flow listener support around the volume api flow
This will add in logging of what is occurring around the
actions taken by taskflow when taskflow executes the initial
create_volume workflow.
Since the oslo logging module will include the request_id of
the currently active request (per greenthread) this will make
it easier to understand and debug the actions occurring inside
taskflow's engine concept.
Code Review / openstack-build / cinder-build.git / log