Seif Lotfy [Fri, 9 Aug 2013 13:37:38 +0000 (13:37 +0000)]
Fix ratelimiting
Current master does now respect ratelimiting, since parsing of the
api-paste.ini was faulty. api-paste.ini limited user limiting by
setting a line as follows:
user:<user-id>:(GET, *, ".*", 4, minute) which was passed to the
Limiter as kwargs with "user" as a key. Thus multiple user limiting
was not possible as well as extracting the id of the user was bound
to fail, since we checked on the key with startswith("user:")
An example config in the api-paste.ini has to look as follows:
limits = (POST, "*", .*, 10, MINUTE)
limits.<user-id1>:(GET, "*", .*, 4, minute)
limits.<user-id2>:(GET, "*", .*, 2, minute)
Kurt Martin [Thu, 8 Aug 2013 21:14:56 +0000 (14:14 -0700)]
3PAR drivers creating incorrect comment data
The 3PAR volumes created from a shapshot had incorrect comment data
that gets added in the comment section of the volumes on the backend.
This patch fixes the display name and description by getting the correct
data when the volume is created. This patch also fixes the inconsistency
in how the keys (name vs. display_name) were used in the comments.
Joshua Harlow [Thu, 1 Aug 2013 19:08:04 +0000 (12:08 -0700)]
Refactoring of create_volume to use taskflow.
Move the create_volume workflow to using taskflow and
split that workflow into three major pieces (each with
there own workflow) and create tasks that perform the
individual required actions to accomplish the pieces
desired outcome.
1. An api workflow composed of the following tasks:
- Extracting volume request (which checks types, values) and creates a
standard output for other tasks to work on (allowing further tasks to be
plugged in the chain without having to worry about other tasks output
formats).
- Quota reservation (rolled back on failure).
- Database entry creation.
- Quota committing.
- Volume RPC casting to volume scheduler or to targeted volume manager.
2. A scheduler workflow composed of the following tasks:
- Extracting scheduler request specification for further tasks to use.
- Change status & notify (activated only on failure).
- Create volume scheduler driver call (which will itself RPC cast to a
targeted volume manager).
3. A manager workflow composed of the following tasks:
- Extract volume request specification from incoming request for
further tasks to use. This also breaks up the incoming request into the 4
volume types that can be created later.
- Change status & notify on failure or reschedule on failure, this is
dependent on if rescheduling is enabled *and* which exception types are
thrown from the volume creation code.
- Create volume from specification
- This contains the code to create from image, create raw volume, create
from source volume, create from snapshot using the extracted volume
specification.
- Change status & notify success.
Key benefits:
- Handled exceptions in a easier to understand, easier to review and more
reliable way than they are currently being handled.
- Rescheduling is now easier to understand.
- Easier to understand structure with tasks that consume inputs, take some
action on them and produce outputs and revert on subsequent failure using
whatever they produced to know how to revert.
- Ability to add new unit tests that can test individual task actions by
providing mock task inputs and validating expected task outputs.
Future additions:
- Eventual addition of resumption logic to recover from operations stopped
halfway through.
- Ability to centrally orchestrate the tasks and pick and choice how
reconciliation of failures based on code or policies.
Lakhinder Walia [Thu, 1 Aug 2013 23:43:45 +0000 (16:43 -0700)]
Add minimum features in HDS driver (for Havana & Icehouse)
* Add create_cloned_volume() api.
* Add extend_volume() api.
* Reorganized some connection state keeping into _loc_info() function.
Earlier this logic was spread out in various calls.
* New self tests for #1 and #2 above.
Kurt Martin [Tue, 6 Aug 2013 20:25:36 +0000 (13:25 -0700)]
3PAR volumes created from snaps failed to attach
An error would occur if you attempted to attach a volume that
was created from a snapshot. The 3PAR backends does not have a
'userCPG' entry in the volume data on the backend, it instead
has a 'snapCPG' field. This patch will now look for the 'snapCPG'
entry and use that CPG when attaching the volume created from a
snapshot.
John Griffith [Tue, 6 Aug 2013 20:58:02 +0000 (14:58 -0600)]
Move volume_clear and clear_size opts up to driver
The block_device driver pulled in the clear and clear_size
options for it's own use and broke the inheritance model needed
for multiple backend configs.
This change moves the two configs up higher in the heirarchy to
driver.py so that it can be shared/used by all drivers if they desire.
This means that they are now set properly in volume_opts. At some
point we shoudl restructure this a bit to allow multi-backends to
specify independent settings for these configs.
Kurt Martin [Mon, 5 Aug 2013 21:23:52 +0000 (14:23 -0700)]
Set the concurrent connections on the 3PAR array
Currently, the 3PAR backend has a limit to the number of concurrent
connections that can be made to the WS API Server. This patch is
setting this limit to 15 connections, the highest possible.
Joel Coffman [Tue, 30 Jul 2013 14:06:28 +0000 (10:06 -0400)]
Create key manager interface
This interface provides a thin wrapper around an underlying key
management implementation such as Barbican or a KMIP server. The key
manager interface is used by the volume encryption code to retrieve
keys for volumes.
This change is a copy-paste of the key manager interface accepted by
Nova. The major modifications are to the module imports (i.e.,
nova -> cinder).
The RBD driver supports two methods of image cloning;
thin copy i.e. copy-on-write (requires v2 Glance API)
and full copy (default if v2 Glance API is not available).
clone_image() was failing in the Glance V1 API case because
it was not returning a tuple, as expected by the volume
manager. Glance V2 support works fine.
Bill Owen [Mon, 5 Aug 2013 18:32:27 +0000 (11:32 -0700)]
GPFS Verify min release level for mmclone command
The gpfs driver makes heavy use of mmclone command. This command
was added fairly recently. Verify that the GPFS cluster is operating
at a level that supports mmclone at startup during prerequisite
checking in check_for_setup_error.
Need to verify:
1. That gpfs cluster is at required level
2. That filesystem(s) are also operating at required level.
Luis A. Garcia [Tue, 23 Jul 2013 18:53:38 +0000 (18:53 +0000)]
Sync gettextutils from oslo
The oslo changes necessary for delayed translation were refactored in
oslo. This patch set brings in the refactored changes, implemented
under the same change-id mentioned below.
Kurt Martin [Wed, 31 Jul 2013 20:56:29 +0000 (13:56 -0700)]
Remove unnecessary metadata from the 3PAR drivers
Currently, both the HP 3PAR iSCSI and FC drivers are populating
the volume and snapshot metadata field with unnecessary data that
should actually be stored on the backend. This data should not be
accessible by the user which it is today and could be updated as well.
This patch will remove all custom 3PAR data from the metadata fields.
Kurt Martin [Fri, 2 Aug 2013 22:06:31 +0000 (15:06 -0700)]
Add new persona value in the 3PAR driver
A new persona type, '12 - OpenVMS' was added to the list of valid
personas in the 3PAR drivers. This persona was introduced in the
3.1.2 MU2 firmware update on the 3PAR storage array.
This patch includes bumping the driver versions for both
Fibre Channel and iSCSI to 1.1. Included a history change
that describes the changes since 1.0.
This patch also changes the way we do clone volume. We no
longer have to check the status of the volume clone on the 3par
as it's instantly done now since 3.1.2 MU2 firmware.
Roman Podolyaka [Wed, 24 Jul 2013 14:13:07 +0000 (17:13 +0300)]
Call get_session() only when necessary
A Session instance should be retrieved only when it's
really needed, e. g. to make a few DB queries inside a
single transaction or to pass the session to a private
DB API method, etc. At the same time, many queries are
simple and can be expressed by the means of auxiliary
model_query() function.
Roman Podolyaka [Wed, 24 Jul 2013 13:37:48 +0000 (16:37 +0300)]
Fix volume_create()/snapshot_create() DB methods
volume_create() and snapshot_create() DB API methods create
entities and return information about them by calling
_volume_get() and snapshot_get() methods. There is no need
to create a new transaction to call the corresponding 'getter'.
Unfortunately, fixing this breaks two existing tests, which rely
on the current behaviour of these DB API methods: both volume_create()
and snapshot_create() raise exceptions, nevertheless new entities are
created.
Roman Podolyaka [Mon, 22 Jul 2013 15:27:51 +0000 (18:27 +0300)]
Execute DB API methods in a single transaction
Many DB API methods do a few queries to a DB (e. g. SELECT
and then UPDATE, or SELECT and then DELETE, etc). By default,
a Session instance is used with autocommit=True, which means,
that each query to a DB is done in a separate transaction. This
is error-prone (as it may lead to race conditions or returning
of unexpected results) and makes rollbacks harder (if one of
transactions fail, the previous ones can not be rolled back).
This patch ensures that all DB API methods, which do a few
queries (or call private DB API methods), are executed inside
a single transaction.
Code Review / openstack-build / cinder-build.git / log