Miguel Angel Ajo [Thu, 20 Aug 2015 13:57:19 +0000 (15:57 +0200)]
Fix qos api-tests after policy changes
The policy.json update in change
Ide1cd30979f99612fe89dddf3dc0e029d3f4d34a breaks the qos api-tests
due to actions which the default policy won't allow, like qos
rules or policies creation by non-admins.
We removed test_rule_association_nonshared_policy which
is not possible with the default policy.json in favor of
test_policy_create_forbidden_for_regular_tenants.
This commit unblocks the qos api-test re-enablement.
Jakub Libosvar [Thu, 20 Aug 2015 16:02:11 +0000 (16:02 +0000)]
fullstack: use migration scripts to create db schema
Previously, we used create_all() based on models. We don't use
create_all() in production code and there is no guarantee models and
scripts are in sync even though we have a good functional test that
validates that. There are still pieces that can't be compared by
alembic.
Jakub Libosvar [Thu, 20 Aug 2015 12:33:59 +0000 (12:33 +0000)]
Sync FK constraints in db models with migration scripts
We do have a functional test that compares Neutron's db models with
migration scripts. The comparison is based on alembic library that had a
bug which is gonna be solved in the next release [1]. Once we start
using newer alembic, functional test mentioned above will start failing
due to models and scripts are not in sync.
This patch adds needed constraints discovered by running functional test
locally with dev version of alembic.
Note: There is already a patch [2] that fixes QoS.
John Schwarz [Sun, 9 Aug 2015 14:00:57 +0000 (17:00 +0300)]
Add EnvironmentDescription, pass it down
* The EnvironmentDescription class describes an entire fullstack
environment (as opposed to the currently implemented host-only
descriptions). This will allow future patches to signify that a test
should set up an environment that supports tunneling, l2pop, QoS and
more.
* Now, most fullstack fixtures (config and process ones, at least),
expect both the EnvironmentDescription for the current test and the
HostDescription for the 'host' the config/process is on. This allows
for easier and most robust future changes, as now adding a new
parameter to one of the description objects doesn't mean adding that
argument to a number of other objects which are using it.
* Changed HostDescription's default argument of l3_agent to False, since
adding new configurations and defualting them to True forces the
author to go through ALL the tests and explicitly turn them on/off.
However, defaulting new configurations to False only requires
explicitly turning them on, which we ought to do anyway.
Ann Kamyshnikova [Thu, 20 Aug 2015 08:27:39 +0000 (11:27 +0300)]
Split DRIVER_TABLES in external.py
Split DRIVER_TABLES into separate lists for each driver.
This is needed for easier implementation of ModelMigrationSyncTest
in driver/plugin repositoties that were split out from Neutron.
In a case when first attempt to fetch default security group
fails and attempt to add it fails too due to a concurrent insertion,
later attempt to fetch the same default sg may fail due to
REPEATABLE READ transaction isolation level.
For this case RetryRequest should be issued to restart the
whole transaction and be able to see default group.
The patch also removes 'while True' logic as it's unsafe
Eugene Nikanorov [Sun, 10 May 2015 23:10:29 +0000 (03:10 +0400)]
Graceful ovs-agent restart
When agent is restarted it drops all existing flows. This
breaks all networking until the flows are re-created.
This change adds an ability to drop only old flows.
Agent_uuid_stamp is added for agents. This agent_uuid_stamp is set as
cookie for flows and then flows with stale cookies are deleted during
cleanup.
Co-Authored-By: Ann Kamyshnikova<akamyshnikova@mirantis.com>
Closes-bug: #1383674
* A note from the legal team: These tests in no way replace
any existing tests. I would never dream of such a thing. Nor
would anyone ever consider calling these 'unit' tests. That
would be mad!
Change-Id: I73c2b2096e767575a196bf08e7d4cc7ec52fdfa3 Co-Authored-By: Lynn Li <lynn.li@hp.com>
Assaf Muller [Fri, 12 Jun 2015 19:07:17 +0000 (15:07 -0400)]
Add a fullstack fake VM, basic connectivity test
* Full stack tests' fake VMs are represented via a namespace,
MAC, IP address and default gateway. They're plugged to an OVS
bridge via an OVS internal port. As opposed to the current
fake machine class used in functional testing, this new fake
machine also creates a Neutron port via the API and sets the
IP and MAC according to it. It also sets additional attributes
on the OVS port to allow the OVS agent to do its magic.
* The functional fake machine and the full stack fake machine
should continue to share commonalities.
* The fullstack fake machine currently takes the IP address
from the port and statically assigns it to the namespace
device. Later when I'll add support for the DHCP agent
in full stack testing this assignment will look for the dhcp
attribute of the subnet and either assign the IP address
via 'ip' or call a dhcp client.
* Added a basic L2 connectivity test between two such machines
on the same Neutron network.
* OVSPortFixture now uses OVSInterfaceDriver to plug the port
instead of replicate a lot of the code. I had to make a
small change to _setup_arp_spoof_for_port since all OVS ports
are now created with their external-ids set.
Sandhya Dasu [Mon, 17 Aug 2015 10:26:53 +0000 (06:26 -0400)]
Final decomposition of ML2 Cisco UCSM driver
The ML2 Cisco UCSM driver's entry point is being switched to the
networking-cisco vendor repo. The definition of the driver's db
file and all references to it in the neutron branch are removed.
Ann Kamyshnikova [Wed, 19 Aug 2015 11:19:11 +0000 (14:19 +0300)]
Fix query in get_reservations_for_resources
For PostgreSQL if you're using GROUP BY everything in the SELECT
list must be an aggregate SUM(...) or used in the GROUP BY.
For reference:
http://www.postgresql.org/message-id/200402271700.28133.dev@archonet.com
Closes-bug: #1486467
Miguel Angel Ajo [Tue, 18 Aug 2015 06:35:00 +0000 (08:35 +0200)]
Fix tenant access to qos policies
fix policy.json to not allow tenants to create policies or rules
by default and allow tenants attach ports and networks to policies,
please note that policy access is checked in the QoSPolicy neutron
object in such case.
The reservation engine is subject to failures due to concurrency;
the switch to pymysql is likely to also have a part in observed
failures. While no gate failures have been observed so far, this
is a time bomb waiting to explode and must be addressed.
For this reason this patch acts conservatively by ensuring the
API controllers do not use anymore reservation. The code for
reservation management is preserved, and will wired again on the
controller when these issues are sorted.
The devref for neutron quotas is updated accordingly as a part
of this patch.
The patch makes L3 agent aware of possible SNAT role
rescheduling to/from it.
The gist is to compare gw_port host change.
If it was changed and agent is not on target host then
it needs to clear snat namespace if one exists. If agent
is on target host it needs to create snat namespace from
scratch if it doesn't exist.
Host field was excluded from gw_port comparison on
agent side as part of HA Router feature implementation.
This code was moved to corresponding module.
Doug Hellmann [Fri, 14 Aug 2015 22:30:46 +0000 (22:30 +0000)]
Add logging to debug oslo.messaging failure
It looks like recent changes to oslo.messaging master are conflicting
with changes in neutron master with the way RPC services are started
when the rpc_workers value == 0.
We can skip trying to setup firewall filters for ports which are
having port_security_enabled as False or which are not associated
to any security group.
This patch does a simple fix to the quota DB driver in order
to ensure its compatibility with python3 and adds the quota
enforcement unit tests to the list of those executed as a part
of the py34 test environment.
Add the concept of resource reservation in neutron.
Usage tracking logic is also updated to support reservations.
Reservations are not however available with the now deprecated
configuration-based quota driver.
The base API controller will now use reservations to perform
quota checks rather than counting resource usage and then
invoking the limit_check routine.
The limit_check routine however has not been removed and
depreacated as a part of this patch. In order to ensure all
quota drivers expose a consistent interface, a
make_reservation method has been added to the configuration
based driver as well. This method simply performs "old-style"
limit checks by counting resource usage and then invoking
limit_check.
Doug Wiegley [Mon, 17 Aug 2015 15:17:46 +0000 (09:17 -0600)]
Don't fatal error during initialization for missing service providers
Sometime during the split, code was added to fixup driver paths,
which imports service providers even for plugins which are not
in use. That, combined with neutron including default service
providers for VPN and LOADBALANCER, resulted in a really messy
mess in terms of removing VPN from the main neutron test suites.
This change stops the imports, so that if one of the services is
missing, neutron server can still start. It likely breaks the driver
path fixup, which can be fixed outside of this gate blockage.
This merge commit introduces QoS feature into Liberty release of
Neutron.
The feature is documented in: doc/source/devref/quality_of_service.rst
included with the merge patch.
It includes:
- QoS API service plugin with QoS policy and QoS bandwidth limit
(egress) rule support;
- core plugin mechanism to determine supported rule types, with its ML2
implementation;
- new agent extension manager;
- QoS agent extension with pluggable backend QoS drivers (Open vSwitch
and SR-IOV support is included).
To extend network and port core resources with qos_policy_id attribute,
a new ML2 extension driver (qos) was introduced that relies on the QoS
core resource extension (the idea is that eventually we'll get a core
resource extension manager that can be directly reused by core plugins).
Agent-server interaction is based on:
- get_device_details() method that is extended with qos_policy_id;
- a new push/pull mechanism that allows agents and servers to
communicate using oslo.versionedobjects based objects sent on the
wire.
The merge includes the following types of test coverage:
- unit tests;
- functional tests for OVS agent, QoS agent extension, and low level
ovs_lib changes;
- API tests to cover port/network qos_policy_id attribute and new QoS
resources.
This merge also disables qos extension API tests until the service is
enabled in master gate.
Local changes apart from conflicts:
- updated down_revision for qos migration to reflect master expand head;
- disabled qos API tests with gate_hook.sh until we have it enabled in
master gate;
- bumped oslo.versionedobjects requirement to reflect what is in
openstack/requirements' global-requirements.txt
shihanzhang [Wed, 12 Aug 2015 09:12:27 +0000 (17:12 +0800)]
Rename function '_update_port_down'
The function _update_port_down is renamed to _get_agent_fdb
because it generates the fdb entries which are send to
related l2 agents, but the old name is hard to understand.
The idea here was to remove redundant unit tests.
The approach here has been that if the function being tested does not
implement any custom logic (apart from calling ovsdb), the unit test
does not help.
Refer to the bug description for more details of the specific tests
removed.
Kevin Benton [Sun, 16 Aug 2015 09:32:39 +0000 (02:32 -0700)]
Get rid of exception converter in db/api.py
The exception converter was necessary because the exceptions the
oslo db decorator looked for before were statically defined. The
retry decorator now accepts an exception_checker argument that takes
a function to call on exceptions to determine if they should be
caught.
This patch gets rid of the converted and replaces the one use case
with the new exception_checker argument.
Kevin Benton [Fri, 31 Jul 2015 01:07:03 +0000 (18:07 -0700)]
Use a conntrack zone per port in OVS
Conntrack zones per network are not adequate because VMs
on the same host communicating with each other cross iptables
twice. If conntrack is sharing the same zone for each cross,
the first one can remove the connection from the table on a RST
and then the second one marks the RST as invalid.
This patch adjusts the logic to use a conntrack zone per port
instead of per network. In order to avoid interrupting upgrades
or restarts, the initial zone map is built from the existing
iptables rules so existing port->zone mappings are maintained.
Code Review / openstack-build / neutron-build.git / log