This patch introduces the RPC contract changes
required for both the server (plugin) and agent
to propagate and retrieve additional information
about Distributed Routers, like MAC addresses
and Port Bindings.
This patch introduces the models, the DB migrations
and the config options required by the L2 layer to
support DVR east/west traffic.
These changes will be used by the control-plane made
of ML2, L2pop and L2 agent.
Two new configuration options have been introduced:
'dvr_base_mac' is used to set DVR MAC addresses apart
from tenant ones (every distributed router will have
ports being created on compute hosts) and
'enable_distributed_routing' is used to enable dvr
support in the L2 agent. This gives the capability of
rolling out the dvr functionality in stages.
Kevin Benton [Mon, 9 Jun 2014 19:20:18 +0000 (12:20 -0700)]
Big Switch: Remove consistency hash on full sync
Clears the consistency hash before performing a full
backend synchronization. Since all of the data is being
replaced, there is no reason to include this and it
currently requires special-casing to handle.
Paul Michali [Mon, 23 Jun 2014 20:42:41 +0000 (16:42 -0400)]
VPNaaS: separate out validation logic for ref impl
Modify the persistence logic for VPNaaS to separate out the
validation for the reference implmentation. This allows
providers to override/extend the validation, as needed.
Additional commits will address the separation for other
providers and for other L3 services.
In addition, the logic that sets default values for optional
attributes is also moved into separate methods to allow
providers to override the defaults used.
Only attributes that have multiple validations are considered
at this time, but this can be extended, as needed.
Currently, one UT fails due to an oslo.messaging bug that
has been fixed, but not yet released for Neutron use.
As a workaround, the affected test case has been commented
out. This test verifies that the right (only, in this case)
device driver is used for validation. Once the issue with
releasing oslo.messaging 1.4.0.0a2 or newer is resolved,
this can be uncommented.
In the future, TaskFlow may be used for the L3 services workflow,
and these changes will provide validation modularization for that
effort.
This patch introduces the model and extension
framework for implementing distributed virtual
routing on top of Open vSwitch.
A new admin-only 'distributed' (as opposed to a
'centralized' or legacy-mode) attribute is added
to the API router resource. It is possible to convert
an existing (centralized) router to a distributed
one; the opposite conversion, even though allowed by
the API, may not be honored by the underlying
plugin implementation and an appropriate error will
be reported.
When creating a router (regardless of the user role),
Neutron will rely on a system wide configuration, whose
default currently allows to create 'centralized' routers.
Tests are added for basic unit coverage; when the first
building blocks for neutron-testing-refactor
are complete, functional testing will be added.
This is because we should be moving away from how
extension tests have been done up until now.
Paul Michali [Mon, 7 Jul 2014 22:57:43 +0000 (18:57 -0400)]
VPNaaS Cisco REST client enhance CSR create
For the create of the REST client object that represents a Cisco CSR,
all of the info needed were passed in as separate parameters. This
change just uses a dict instead, so that additional parameters can
be added w/o changing the API.
Updated the currently unused UT module, just so that it can be used
locally and stays up-to-date until it can be converted to use the
new requests-mock package.
Fixes port update failure when device ID is not updated
The updation was failing because the changed_device_id variable
was undeclared.This fix declares the variable with a default value
at the beginning of the method.
Carl Baldwin [Thu, 6 Mar 2014 23:57:11 +0000 (23:57 +0000)]
L3 agent prefers RPC messages over full sync
When the L3 agent starts up and runs the sync task it doesn't process
any incoming RPC events until the sync task is complete.
This change combines the work from _rpc_loop and _sync_routers_task in
to a single loop called _process_routers_loop. This loop spawns
threads that pull from a priority queue. The queue ensures that RPC
messages are handled before _process_routers_loop. The latter is
generally maintenance tasks triggered by the agent rather than user
triggered tasks.
Synchronization between RPC and sync routers loops is no longer
necessary since they both feed in to a single queue. There were
places where it was necessary to reorder some things to allow for the
lack of synchronization. For example, it is necessary to list
namespaces before fetching the full list of routers to ensure that it
doesn't delete a new namespace that gets created after listing
namespaces. The lack of the need for synchronization between loops is
probably the main strength of this patch.
With multiple worker threads, need to handle the case where an RPC
message came in while a thread was working on a router. Another
thread should not handle the same router that is already in progress.
Adds a mechanism to signal to the working thread that an update came
in for the router it is working on. The original thread will repeat
processing the router when it is finished to get the update.
Multiple rapid updates to the same router will be consolidated.
Essentially, there is still synchronization of work for a given router
but not between routers. Much better than before.
Maruti [Sat, 7 Jun 2014 12:42:45 +0000 (05:42 -0700)]
Fixed audit notifications for l3-agent-router ops
l3-agent-router-add and l3-agent-router-remove do not
generate audit notifications which are used for security
compliance.
CRUD operations of core network resources are
handled by neutron/api/v2/base.py. In base.py, each of
create(), update(), delete() methods makes calls to
oslo.messaging.Notifier.info() to generate these notifications.
In the proposed fix, it is fixed in a similar fashion in
extensions/l3agentscheduler.py inside create() and delete()
methods by introducing info() method calls inside them.
Steven Hillman [Tue, 8 Jul 2014 22:35:10 +0000 (15:35 -0700)]
Send network name and uuid to subnet create
Added the network segment name and uuid parameters to the infomation sent to
Cisco N1kV during subnet creation to allow for proper association of the
subnet and network segment.
Robert Kukura [Wed, 12 Mar 2014 01:54:35 +0000 (21:54 -0400)]
ML2: Bind ports outside transactions
The ML2 plugin now calls the bind_port() operation on the registered
mechanism drivers outside of any enclosing DB transaction. Ports are
created or updated in one transaction, then a binding is established
if possible, and finally a second transaction commits the binding
result.
With [re]binding moved outside the DB transaction that triggered it,
it is now possible that multiple threads or processes will
concurrently try to bind the same port, or that the port will be
updated between transactions. Concurrent attempts to bind the same
port are allowed to proceed, which results are used is resolved in the
second transaction, and binding is retried if necessary.
Improvements to the Cisco Nexus driver and unit tests from Rich Curran
needed due to the binding changes are also included.
Jakub Libosvar [Thu, 26 Jun 2014 14:56:32 +0000 (16:56 +0200)]
Freeze models for healing migration
Take a snapshot of all models from the code base at the time when the
healing migration merges. The healing migration needs this frozen view
of the models to be available (even as the models change in the future)
to compare with the current DB schema. The healing migration will use
this comparison to heal the schema.
While adding iptables rule, cidr is added as destination for both ingress
and egress directions. Modified code to add -s for egress and -d for ingress.
Assaf Muller [Sun, 4 May 2014 15:26:43 +0000 (18:26 +0300)]
Change all occurences of no_delete to do_delete
Previously, ports, networks and subnets had a do_delete=True
parameter. By default, these resources were deleted at the
end of the context manager scope. All other resources used
a different semantic: no_delete=False.
This caused confusing situations such as:
with self.subnet(network, do_delete=False) as subnet:
with self.security_group(no_delete=True) as sg:
pass
Now all resources use the same do_delete semantic.
Joe Gordon [Fri, 11 Jul 2014 18:02:54 +0000 (18:02 +0000)]
Revert "VPNaaS REST Client UT Broken"
I8badc249ad021fdbdb2367b5416c72435ed58994 causes anything importing
neutron/tests/unit/services/vpn/device_drivers/_test_cisco_csr_rest.py
to exit since httmock isn't a dependency.
Fix all hacking issues in addition to the revert, as this patch fixes 'tox -epep8'.
And just a reverting the patch will cause pep8 to fail.
If a model mentions 'default' it is better to set 'server_default'
as well because this 'defaut' can be reached only via code. This
change adds 'server_default' parameter to models and in database,
so if someone will use SQL query it will also use default value.
For model Router added parameter server_default. In migration it was
added by this change https://review.openstack.org/105212.
For model MeteringLabelRule added parameter server_default.
In migration it will be added by heal script due to problems
with Metering plugin and grenade.
Kevin Benton [Thu, 10 Jul 2014 19:18:54 +0000 (12:18 -0700)]
Add BSN plugin to agent migration script
Adds the Big Switch plugin to the migration script that creates
the agents table. A recent commit (d3be7b040eaa61a4d0ac617026cf5c9132d3831e)
added it to the agent tables but it missed this one so an error was thrown
during migration when it reached a table that depended on the agents table.
db_base_plugin_v2 imports too much modules that are not necessary
usually, so extract CommonDBMixin in different file.
Plus using db_base_plugin_v2 for some types of modules can lead to
cycles in imports, this refactoring should resolve the issue.
Shiv Haris [Fri, 20 Jun 2014 22:05:10 +0000 (15:05 -0700)]
Added support for NOS version 4.1.0, 5.0.0 and greater
NETCONF temaplates for NOS version greater than 4.1.0 are slightly
different (argh). An init time check of the NOS version is done
to enable selection of the correct NETCONF templates.
NSX: neutron router-interface-add should clear security-groups
NSX does not support security groups on router ports so in the case
where someone uses a port that has a security group on it as the router
port we need to clear the security group off the port.
Currently treat_devices_added_or_updated in the OVS agent skips
processing devices which disappeared from the integration bridge
during the agent loop.
This is fine, however the agent should not mark these devices as
processed. Otherwise they won't be processed, should they appear
again on the bridge.
This patch ensures these devices are not added to the current
device set.
The patch also changes treat_devices_added_or_updated. The
function now will return the list of skipped devices and not
anymore a flag signalling whether a resync is required.
With the current logic a resync would be required if retrieval
of device details fails. With this change, the function
treat_devices_added_or_updated will raise in this case and the
exception will be handled in process_network_ports.
For the sake of consistency, this patch also updates the
similar function treat_ancillary_devices_added in order to
use the same logic.
Finally, this patch amends an innaccurate related comment.
Ann Kamyshnikova [Fri, 21 Mar 2014 08:33:30 +0000 (12:33 +0400)]
Fix 'server_default' parameter usage in models
In ml2 models parameter 'default' is used for vnic_type, profile and
vif_details, but in migrations 27cc183af192_ml2_vnic_type,
157a5d299379_ml2_binding_profile and
50d5ba354c23_ml2_binding_vif_details is used 'server_default' parameter.
Usage 'default' and 'server_default' should be equal in models and
migration. So models in models is added 'server_default' parameter.
Paul Michali [Mon, 30 Jun 2014 22:31:42 +0000 (18:31 -0400)]
VPNaaS REST Client UT Broken
When the H302 work was done, it broke two modules used for UT of
Cisco CSR VPNaaS REST client. Since the UT is currently disabled
(because the httmock library was not approved for test-requirements),
the error was not detected.
This fixes the files, so that they can still be used manually (by
locally importing httmock or its source file, and renaming the
module to remove the "_" prefix).
In the long term, the UT needs to be reworked to use httpretty,
which uses a register based mechanism, instead of a context
manager based mechanism, so this will take some effort. In addition.
there is discussion of switching to a new mock-request package,
instead of httpretty.
Ihar Hrachyshka [Wed, 18 Jun 2014 14:56:25 +0000 (16:56 +0200)]
Avoid notifying while inside transaction opened in delete_port()
delete_port() calls to disassociate_floatingips() while in transaction.
The latter method sends RPC notification which may result in eventlet
yield. If yield switches a thread to another one that tries to access
the same floating IP object in db as disassociate_floatingips() method
does, we're locked and get db timeout.
We should avoid calling to notifier while under transaction.
To achieve this, I introduce a do_notify argument that controls whether
notification is done by disassociate_floatingips() itself or delegated
to caller. Callers that call to disassociate_floatingips() from under
transactions should handle notifications on their own. For this,
disassociate_floatingips() returns a set of routers that require
notification.
Updated drivers to reflect new behaviour. Added unit test.
CANARY_TABLE is for br-int, not for br-tun. Grouping it with
constants for br-tun tables together is misleading. Move it into
a separate section and comment appropriately. No functional changes.
Fix UnboundLocalError raised during L3 router sync task
This can be fixed in a number of ways: a) consolidating the
two except clauses into one; b) adding a 'return' after the
last except clause c) by calling the cleanup method only on
success; d) initializing 'routers' before usage.
Approach c) has the benefit of stating the developer's intent
more explicitly and minimize chances of regression.
self.assertIsInstance is preferred over assert(isinstance(..)). See feedback in:
https://review.openstack.org/#/c/102754/1/neutron/tests/unit/test_extension_security_group.py
Previously, sudo-requiring functional tests hardcoded the use of
'sudo' as the root helper. Devstack gate jobs do not allow
password-less invocation of 'sudo', though, so such tests were unable
to run in the gate. This patch adds the ability to configure the
rootwrap command installed by devstack by setting the OS_ROOTWRAP_CMD
environment variable in the test execution environment, allowing
sudo-requiring tests to run.
Code Review / openstack-build / neutron-build.git / log