Remove auto-generation of db schema from models at startup
This patch removes the Neutron capability of creating database tables
from sqlalchemy models for all those model classes for which
a table is not found in the database schema.
Migrations should be the official and only solution for creating and
managing the Neutron db schema.
This patch also adapts unit tests in order to ensure test schemas
are still correctly created.
Carl Baldwin [Thu, 10 Jul 2014 19:41:40 +0000 (19:41 +0000)]
Implement namespace cleanup for new DVR namespaces
DVR introduces a new namespace type called snat-<router_id>. These
namespaces are not properly cleaned up when found stale after an agent
restart. This patch fixes that.
Akihiro Motoki [Thu, 14 Aug 2014 06:53:47 +0000 (15:53 +0900)]
Skip FWaaS config mismatch check if RPC method is unsupported
In this commit FWaaS config check skip is skipped
if neutron server does not support this API.
Commit d6f014d introduced FWaaS config mismatch check between
server and agent. It added a new RPC method get_service_plugin_list
and bumped l3-agent RPC version to 1.3, but this version RPC is
only supported by L3 router service plugin and
it breaks existing plugins using L3 router mixin.
Bumping l3-agent RPC version requires detailed investigation on all
affected plugins and it can be done by plugin maintainer later.
Ihar Hrachyshka [Thu, 7 Aug 2014 22:04:44 +0000 (00:04 +0200)]
Use jsonutils instead of stdlib json
jsonutils provides multiple benefits in comparison to pure stdlib json
(like using simplejson on Python 2.6).
Similar patch was already merged before [1], but since it lacked hacking
rule to enforce jsonutils usage, new occurrences of stdlib json module
usage were introduced.
This patch switches all the code to using jsonutils and adds a hacking
rule to enforce the rule.
The hacking rule requires that jsonutils module does not mimic as 'json'
thru using import renames, so the code was updated not to rename the
module when doing import.
The hacking rule was shamelessly copied from the corresponding nova
review [2].
rajeev [Tue, 12 Aug 2014 00:42:18 +0000 (20:42 -0400)]
Fix interface add for dvr with gateway
when an interface is added after router gateway set, external
connectivity using snat fails. Instead of just adding the snat port for
the new subnet, method internal_network_added(..) incorrectly re-adds
all the snat ports with wrong cidr.
Stephen Ma [Wed, 6 Aug 2014 22:33:32 +0000 (22:33 +0000)]
Delete DVR namespaces on node after removing last VM
After removing the last VM using a distributed router,
the router's namespaces are still present on the VM host
The problem is that the neutron API server sent the router
remove notification to the L3 agent using the name of the
host running the L3 agent instead of the agent's uuid. This
caused an error when sending the notification. So the L3
agent never had the chance to cleanup the namespace.
This problem is fixed here.
Afterwards, it was found that the notification was still not
sent. The reason is that the router/L3-agent binding has
already been deleted before the routine that sends the
router removed notification was called. The notifier routine
errored out when it tried to delete the same router/L3 agent
binding. This problem is fixed in this patch by removing the
binding removal step from the DVR scheduler.
Fix PortNotFound error during update_device_up for DVR
An agent's request to update the ARP entry for a VM port
may come after a deletion request has been processed,
resulting in a PortNotFound exception being raised.
This patch takes care of this condition. A test has
been added, which required a minor refactoring of the
test case class, in order to accommodate the use of
side effects for the objects being mocked.
Kevin Benton [Wed, 30 Jul 2014 22:49:59 +0000 (15:49 -0700)]
Option to remove routers from dead l3 agents
Add a configuration-enabled periodic check to examine the
status of all L3 agents with routers scheduled to them and
admin_state_up set to True. If the agent is dead, the router
will be rescheduled to an alive agent.
Neutron considers and agent 'dead' when the server doesn't
receive any heartbeat messages from the agent over the
RPC channel within a given number of seconds (agent_down_time).
There are various false positive scenarios where the agent may
fail to report even though the node is still forwarding traffic.
This is configuration driven because a dead L3 agent with active
namespaces forwarding traffic and responding to ARP requests may
cause issues. If the network backend does not block the dead
agent's node from using the router's IP addresses, there will be
a conflict between the old and new namespace.
This conflict should not break east-west traffic because both
namespaces will be attached to the appropriate networks and
either can forward the traffic without state. However, traffic
being overloaded onto the router's external network interface
IP in north-south traffic will be impacted because the matching
translation for port address translation will only be present
on one router. Additionally, floating IPs associated to ports
after the rescheduling will not work traversing the old
namespace because the mapping will not be present.
Remove SELECT FOR UPDATE use in ML2 tunnel driver add_endpoint
SELECT FOR UPDATE expression, which is triggered with the use of the
SQLAlchemy Query object's with_lockmode('update') method, is
detrimental to performance and scalability of the database
performance code in Neutron due to the lock contention it produces.
SELECT FOR UPDATE can be entirely avoided in add_endpoint methods
with the use of single-shot SELECT and INSERT expressions and the
correction of VxlanEndpoint primary key: indeed previously it was not
possible to create multiple endpoints with the same ip, now the model
primary key constraint ensures it.
Method sync_routers is used by the L3 agent to query
routers it knows about. Routers and GW ports lists
are populated in two different times, which means that
they can be interleaved by a delete request which
results in gateway ports being missing in one of the
two data structures.
Paul Michali [Wed, 13 Aug 2014 01:26:27 +0000 (21:26 -0400)]
VPNaaS: Cisco fix validation for GW IP
The validation to check that the router has a GW IP
for the Cisco service driver was trying to directly
get the gw_port information, which is not available, as
the validator only has a dict of the vpn_service info.
Modified validator to get the router object (not a
dict representation), using the vpn_service's router_id
and then check the attributes for the router directly.
Carl Baldwin [Fri, 25 Jul 2014 03:57:40 +0000 (03:57 +0000)]
Preserve link local IP allocations for DVR fip ns across restart
The L3 agent allocates link local address pairs used in connecting the
routers to the floating ip namespace. When those allocations are
forgetten by restarting the L3 agent they all get rewired on restart.
This change preserves the allocations using a file in the local file
system. Storing them in the database would be overkill and would
affect system performance.
The metadata agent was unable to find networks
attached to the DVR router because it was only
filtering ports for 'centralized' routers.
To fix the issue, this patch expands the search
filters to include DVR router interfaces during
the network lookup operation.
The extra filter cause no evident performance
loss while serving the request; a different
approach would require to pass the router type
around to narrow down the search filter, but it
sounds like an overkill.
Praneet Bachheti [Thu, 29 May 2014 23:11:39 +0000 (16:11 -0700)]
Opencontrail plug-in implementation for core resources
Adds Opencontrail plug-in implementation with unit tests
This patch has no dependency on any other blueprints
The link below describes how to install VIF driver for opencontrail
https://github.com/Juniper/contrail-controller/wiki/
OpenContrail-bring-up-and-provisioning
- The contrail_plugin_core.py is the main interface for neutron common
infrastructure. It relays API requests to the opencontrail controller
Note: There are several other unrelated unit tests that also break with a
randomized PYTHONHASHSEED, but they are not addressed here. They will be
addressed in separate patches.
Make plugin and l3plugin available as mixin's properties
This makes the code simpler and slightly more efficient.
This is a minor refactoring related to a race condition
observed when the agent is processing a router while its
interfaces are being removed.
Isaku Yamahata [Mon, 4 Aug 2014 14:44:18 +0000 (23:44 +0900)]
add auth token to context
As discussed at
http://lists.openstack.org/pipermail/openstack-dev/2014-July/040644.html
SerivceVM project (and other routervm plugins) need auth token in context.
The first user will be l3 routervm plugin.
Michael Smith [Wed, 6 Aug 2014 22:02:35 +0000 (15:02 -0700)]
Fixes an issue with FIP re-association
When the last FIP is disassociated, the namespace and
interfaces should be removed. The internal interface
wasn't removed before without problems, but now the
namespace cannot be removed with that interface present.
The fix is to remove the internal FIP interface before
removing the namespace.
This is done by adopting the router_extra_attributes
models. However, the NSX 'distributed' extension is
slightly different in that it is visible to tenants
and prevents router conversion (allow_put=false).
PUT requests are made return the correct HTTP code;
The access control misalignments will need to be
adjusted via rules in policy.json. This will be
properly documented.
This patch clarifies that in order to use NSX distributed
routers, the replication_mode parameter should be set to
'service'. This is already its default value, so no further
change is needed beyond clarifying wording.
Roey Chen [Tue, 20 May 2014 14:10:17 +0000 (17:10 +0300)]
MLNX Agent: ensure removed ports get treated on resyncs
This patch ensures that removed ports will be cleaned up properly if exception
occurs in the daemon_loop.
It does the same for the mlnx-agent as was done for the lb-agent in
Ieada34ad315c0c29aa8462ebf041a448fde007b8.
Refactoring the daemon_loop and scan_ports in a way that -
If there's an exception, the ports that were flagged for cleaning will
be retreated in the following iteration,
unlike previous exception handling, where only existing ports
were treated in the following iteration.
Cédric Ollivier [Thu, 29 May 2014 10:01:28 +0000 (12:01 +0200)]
Add delete operations for the ODL MechanismDriver
This commit adds delete operations (networks, subnets and ports) for the ODL MechanismDriver.
It also modifies sync_single_resource to reduce db operations.
Sam Betts [Tue, 5 Aug 2014 23:05:05 +0000 (00:05 +0100)]
Predictable field and filter ordering
This fixes the fields and filters units tests that break with a
randomized PYTHONHASHSEED (see the bug report).
The RESOURCE_ATTRIBUTE_MAP is stored as a dict leading to an
unpredictable output order. Values in kvp strings are being stored as
sets underpinned by dicts when converted, leading to unpredictable
ordering of values when read.
Discovered with PYTHONHASHSEED = 2455351445 on these tests:
test_api_v2.APIv2TestCase.test_fields
test_api_v2.APIv2TestCase.test_fields_multiple
test_api_v2.FiltersTestCase.test_attr_info_with_convert_list_to
test_api_v2.APIv2TestCase.test_filters_with_fields
test_api_v2.APIv2TestCase.test_fields_multiple_with_empty
There are 3 parts to this fix:
1. Update the APIv2TestCase _do_field_list function to construct
field list in the same order as the controller constructs its list.
2. Ensure the APIv2TestCase _get_collection_kwargs maintains order
throughout.
3. Use new assertOrderedEqual function to sort values before assertion
in test_attr_info_with_convert_list_to
ronak [Wed, 6 Aug 2014 00:40:24 +0000 (17:40 -0700)]
Fixing neutron-db-manage with some options other than upgrade/downgrade
"mysql-engine" argument was added to upgrade and downgrade option of
neutron-db-manage.
Reference commit: http://tinyurl.com/mzepbmq
migration environment's run_migration_offline/online() gets called
even for other neutron-db-manage options as well such as current,
history, stamp, branches etc. For those options since the argument
can not be set, it throws oslo.config.cfg.NoSuchOptError.
This fix tries to catch it and set the value accordingly.
Sam Betts [Mon, 11 Aug 2014 09:57:40 +0000 (10:57 +0100)]
Removes extra indents from TestSubresourcePlugin
Removed extra indents from in front of each function in the
TestSubresourcePlugin class in test_api_v2.py so that it
meets pep8 requirements and matches up with the rest of the code.
Roey Chen [Sun, 18 May 2014 08:44:00 +0000 (11:44 +0300)]
MLNX Agent: Process port_update notifications in the main agent loop
This patch changes the way mlnx agent process port_update notifications.
It does the same for the mlnx agent as was done for the ovs-agent in
I219c6bdf63b0b5e945b655677f9e28fa591f03cd.
Processing a port_update notification directly in the RPC
call may cause competition with the main RPC loop.
To prevent this problem, the actual process of ports updates is done
in the main RPC loop, whereas the RPC call merely adds the updated
port MAC address to a set of updated ports.
port_update notifications received within a single main loop iteration
will be coalesced and processed only once.
Fix session's InvalidRequestError because of nested rollback
This patch addresses the issue by removing the extra nesting
(which is effectively redundant).
The longer story about this patch is the following:
Race conditions may cause DBDuplicateEntry exceptions that
require a transaction to be rollbacked back, and yet make
the whole operation succeed. A classic example is what has
been solved in commit fbc6b99. If the rollback is done in a
nested transaction, the above mentioned exception is raised.
To address the problem, we could use savepoints by means of
sqlalchemy's begin_nested(); Even though this approach is
preferable, it causes quite a bit of changes in the unit
tests (because of sqlite); it may also require that certain
DBMS, or certain DB backend configurations, support savepoints.
Carl Baldwin [Fri, 8 Aug 2014 17:31:59 +0000 (17:31 +0000)]
Remove unneeded device_owner field from l2pop tuple
The DVR development added this device_owner to the middle of this
tuple during early development because it was thought to be needed.
Over the course of development, it was found to be unnecessary and
much of the code that read it from this value was removed or
obsoleted. That job went unfinished and so this commit completes it.
This essentially restores the code to what it was before and fixes the
regression that was caused.
Removing local_vlan_map entry here makes later port_unbound no-op.
While it isn't a problem right now, it will be when per-port cleanup
is introduced.
This commit simply removes network_delete method. The local_vlan_map
will be cleaned up when the last port on the network is removed.
Sudipta Biswas [Tue, 5 Aug 2014 14:10:06 +0000 (19:40 +0530)]
Fix to throw correct error code for bad attribute
Currently the neutron network API throws up error code 500 for
the extended attribute for segmentation id. This can be reproduced
if the user types in a random string in place of an integer value
for the segmentation id. The proper behavior should throw an error
code 400 with the appropriate failure message. This patch fixes the
same issue and covers it with a test case.
Once gateway is set, external_gateway_added() was getting called every time
a router update was received. The check for change in external
gateway compared previously cached copy of gateway port (ri.ex_gw_port) with
the one passed in through update router (ri.router['gw_port']).
The cached copy was already being modified by code so the two values would
always appear to be different.
Making the change to compare correctly and remove actions not required
for gateway update.
Code Review / openstack-build / neutron-build.git / log