Merge remote-tracking branch 'origin/feature/qos' into merge-branch

Note to reviewers: gerrit diff for merge patches is very limited, and
leaving comments in global section won't scale, so please comment here:

https://etherpad.openstack.org/p/qos-merge-back-review

This merge commit introduces QoS feature into Liberty release of
Neutron.

The feature is documented in: doc/source/devref/quality_of_service.rst
included with the merge patch.

It includes:

- QoS API service plugin with QoS policy and QoS bandwidth limit
  (egress) rule support;
- core plugin mechanism to determine supported rule types, with its ML2
  implementation;
- new agent extension manager;
- QoS agent extension with pluggable backend QoS drivers (Open vSwitch
  and SR-IOV support is included).

To extend network and port core resources with qos_policy_id attribute,
a new ML2 extension driver (qos) was introduced that relies on the QoS
core resource extension (the idea is that eventually we'll get a core
resource extension manager that can be directly reused by core plugins).

Agent-server interaction is based on:

- get_device_details() method that is extended with qos_policy_id;
- a new push/pull mechanism that allows agents and servers to
  communicate using oslo.versionedobjects based objects sent on the
  wire.

The merge includes the following types of test coverage:

- unit tests;
- functional tests for OVS agent, QoS agent extension, and low level
  ovs_lib changes;
- API tests to cover port/network qos_policy_id attribute and new QoS
  resources.

The client changes can be found at:

* https://review.openstack.org/189655
* https://review.openstack.org/198277

The team also prepared fullstack test but it needs to wait for client
merge before it can pass in the gate:

* https://review.openstack.org/202492

Gerrit does not show diff for merge changes that did not result in any
conflict, so to facilitate review, rely on the following steps:

- fetch the patch locally
- git fetch origin
- git diff origin/master...

This merge also disables qos extension API tests until the service is
enabled in master gate.

Local changes apart from conflicts:
- updated down_revision for qos migration to reflect master expand head;
- disabled qos API tests with gate_hook.sh until we have it enabled in
  master gate;
- bumped oslo.versionedobjects requirement to reflect what is in
  openstack/requirements' global-requirements.txt

DocImpact
APIImpact
Partially-Implements: blueprint quantum-qos-api
Partially-Implements: blueprint ml2-qos
Partially-Implements: blueprint ml2-qos-ovs-bwlimiting
Partially-Implements: blueprint ml2-sriov-qos-with-bwlimiting
Change-Id: I92916d0e391791187e9a25ff172fb4b3504857b1

diff --cc doc/source/devref/index.rst
Simple merge

diff --cc etc/neutron.conf
Simple merge

diff --cc neutron/common/exceptions.py
Simple merge

diff --cc neutron/common/utils.py
Simple merge

diff --cc neutron/db/db_base_plugin_common.py
Simple merge

diff --cc neutron/db/migration/alembic_migrations/versions/liberty/expand/48153cb5f051_qos_db_changes.py
index 0000000000000000000000000000000000000000,99f4f1f12b2e5e3bd96f573ad546a07f52158ef6..a692b9553381f681ed14d7bc733c1928d51bc6ac
mode 000000,100755..100755
--- /dev/null
--- 2/neutron/db/migration/alembic_migrations/versions/liberty/expand/48153cb5f051_qos_db_changes.py
+++ b/neutron/db/migration/alembic_migrations/versions/liberty/expand/48153cb5f051_qos_db_changes.py
@@@ -1,0 -1,69 +1,69 @@@
 -Revises: 1c844d1677f7
+ # Copyright 2015 Huawei Technologies India Pvt Ltd, Inc
+ #
+ #    Licensed under the Apache License, Version 2.0 (the "License"); you may
+ #    not use this file except in compliance with the License. You may obtain
+ #    a copy of the License at
+ #
+ #         http://www.apache.org/licenses/LICENSE-2.0
+ #
+ #    Unless required by applicable law or agreed to in writing, software
+ #    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ #    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ #    License for the specific language governing permissions and limitations
+ #    under the License.
+ #
+ 
+ """qos db changes
+ 
+ Revision ID: 48153cb5f051
 -down_revision = '1c844d1677f7'
++Revises: 1b4c6e320f79
+ Create Date: 2015-06-24 17:03:34.965101
+ 
+ """
+ 
+ # revision identifiers, used by Alembic.
+ revision = '48153cb5f051'
++down_revision = '1b4c6e320f79'
+ 
+ from alembic import op
+ import sqlalchemy as sa
+ 
+ from neutron.api.v2 import attributes as attrs
+ 
+ 
+ def upgrade():
+     op.create_table(
+         'qos_policies',
+         sa.Column('id', sa.String(length=36), primary_key=True),
+         sa.Column('name', sa.String(length=attrs.NAME_MAX_LEN)),
+         sa.Column('description', sa.String(length=attrs.DESCRIPTION_MAX_LEN)),
+         sa.Column('shared', sa.Boolean(), nullable=False),
+         sa.Column('tenant_id', sa.String(length=attrs.TENANT_ID_MAX_LEN),
+                   index=True))
+ 
+     op.create_table(
+         'qos_network_policy_bindings',
+         sa.Column('policy_id', sa.String(length=36),
+                   sa.ForeignKey('qos_policies.id', ondelete='CASCADE'),
+                   nullable=False),
+         sa.Column('network_id', sa.String(length=36),
+                   sa.ForeignKey('networks.id', ondelete='CASCADE'),
+                   nullable=False, unique=True))
+ 
+     op.create_table(
+         'qos_port_policy_bindings',
+         sa.Column('policy_id', sa.String(length=36),
+                   sa.ForeignKey('qos_policies.id', ondelete='CASCADE'),
+                   nullable=False),
+         sa.Column('port_id', sa.String(length=36),
+                   sa.ForeignKey('ports.id', ondelete='CASCADE'),
+                   nullable=False, unique=True))
+ 
+     op.create_table(
+         'qos_bandwidth_limit_rules',
+         sa.Column('id', sa.String(length=36), primary_key=True),
+         sa.Column('qos_policy_id', sa.String(length=36),
+                   sa.ForeignKey('qos_policies.id', ondelete='CASCADE'),
+                   nullable=False, unique=True),
+         sa.Column('max_kbps', sa.Integer()),
+         sa.Column('max_burst_kbps', sa.Integer()))

diff --cc neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
Simple merge

diff --cc neutron/plugins/ml2/plugin.py
Simple merge

diff --cc neutron/tests/contrib/gate_hook.sh
index 0db93b3f32093ae6ae88fb86dca912a7e88c6d7a,3da5775c03299418d9e8a70dd15d8b3b07fa85e5..57dbc4a6319b2246a42ed46c139111dfa3eb89b0
--- 1/neutron/tests/contrib/gate_hook.sh
--- 2/neutron/tests/contrib/gate_hook.sh
+++ b/neutron/tests/contrib/gate_hook.sh
@@@ -24,22 -26,10 +24,24 @@@ the
      configure_host_for_func_testing
  elif [ "$VENV" == "api" ]
  then
 -    if [[ -z "$DEVSTACK_LOCAL_CONFIG" ]]; then
 -        export DEVSTACK_LOCAL_CONFIG="enable_plugin neutron-vpnaas git://git.openstack.org/openstack/neutron-vpnaas"
 -    else
 -        export DEVSTACK_LOCAL_CONFIG+=$'\n'"enable_plugin neutron-vpnaas git://git.openstack.org/openstack/neutron-vpnaas"
 -    fi
 +    cat > $DEVSTACK_PATH/local.conf <<EOF
 +[[post-config|/etc/neutron/neutron_lbaas.conf]]
 +
 +[service_providers]
 +service_provider=LOADBALANCER:Haproxy:neutron_lbaas.services.loadbalancer.drivers.haproxy.plugin_driver.HaproxyOnHostPluginDriver:default
 +
 +[[post-config|/etc/neutron/neutron_vpnaas.conf]]
 +
 +[service_providers]
 +service_provider=VPN:openswan:neutron_vpnaas.services.vpn.service_drivers.ipsec.IPsecVPNDriver:default
 +
 +EOF
 +
 +    export DEVSTACK_LOCAL_CONFIG+="
 +enable_plugin neutron-vpnaas git://git.openstack.org/openstack/neutron-vpnaas
 +"
 +
++    export DEVSTACK_LOCAL_CONFIG+="DISABLE_NETWORK_API_EXTENSIONS=qos
++"
      $BASE/new/devstack-gate/devstack-vm-gate.sh
  fi

diff --cc neutron/tests/tempest/services/network/json/network_client.py
index 4badd9623465d03318226483864287885dddd841,9c5ef4aa1a22000bc674e05e0f6a22b984d75a12..3fb233e98a72b5abd6d0e8257e18fe8af51b3856
--- 1/neutron/tests/tempest/services/network/json/network_client.py
--- 2/neutron/tests/tempest/services/network/json/network_client.py
+++ b/neutron/tests/tempest/services/network/json/network_client.py
@@@ -10,9 -10,11 +10,11 @@@
  #    License for the specific language governing permissions and limitations
  #    under the License.
  
 -import json
  import time
+ import urllib
+ 
  
 +from oslo_serialization import jsonutils as json
  from six.moves.urllib import parse
  from tempest_lib.common.utils import misc
  from tempest_lib import exceptions as lib_exc

diff --cc neutron/tests/unit/plugins/ml2/drivers/openvswitch/agent/test_ovs_neutron_agent.py
Simple merge

diff --cc neutron/tests/unit/plugins/ml2/test_plugin.py
Simple merge

diff --cc requirements.txt
index f0bbf740ad1ea3e7d1424af7838347e0db817265,7ea850029ecf84e5d1eb6405df1eedccc485ea6b..e9de05ec3745d6b1c65afd138352d7b78edd8fce
--- 1/requirements.txt
--- 2/requirements.txt
+++ b/requirements.txt
@@@ -34,10 -34,11 +34,11 @@@ oslo.middleware>=2.4.0 # Apache-2.
  oslo.policy>=0.5.0 # Apache-2.0
  oslo.rootwrap>=2.0.0 # Apache-2.0
  oslo.serialization>=1.4.0 # Apache-2.0
 -oslo.service>=0.1.0 # Apache-2.0
 -oslo.utils>=1.9.0 # Apache-2.0
 -oslo.versionedobjects>=0.3.0,!=0.5.0
 +oslo.service>=0.6.0 # Apache-2.0
 +oslo.utils>=2.0.0 # Apache-2.0
++oslo.versionedobjects>=0.6.0
  
 -python-novaclient>=2.22.0
 +python-novaclient>=2.26.0
  
  # Windows-only requirements
  pywin32;sys_platform=='win32'

diff --cc setup.cfg
Simple merge