Russell Bryant [Thu, 20 Nov 2014 21:01:14 +0000 (21:01 +0000)]
Show progress output while running unit tests
This patch imports some magic to enable showing test output as the
tests run. Seeing output showing tests as they execute is reassuring
that the test run is continuing to make progress. This originated in
tempest and was then brought over to Nova in this commit:
https://review.openstack.org/#/c/121914/
Note the justification in that commit to just copy it instead of move
it to a shared repo.
To get the old behavior where you only get output when something
fails, you can set the TRACE_FAILONLY env var. That originated in
this commit in Nova:
abhishek.talwar [Tue, 18 Nov 2014 09:50:51 +0000 (15:20 +0530)]
Check for default sec-group made case insensitive
Currently creating another security group with name
"default" is not allowed, however we can create
another security group with name "DEFAULT"
(or any other CASE pattern).
When trying to boot a VM without specifying
a security group it should always pick the
"default" security group. However, if another
security group of the name DEFAULT is present,
the VM gets associated with the wrong
security group (i.e DEFAULT and not default).
So, updated the code so that a security group
with name DEFAULT (or any other case pattern)
is not allowed.
Closes-Bug: #1384505
Brian Haley [Wed, 19 Nov 2014 17:10:57 +0000 (12:10 -0500)]
Fix a race condition adding a security group rule
setup_port_filters() needs to grab self.devices_to_refilter
before it calls prepare_devices_filter(), else it could skip
processing a device if an RPC arrives while it's processing
new devices. That device will now be handled the next time
it's called.
Russell Bryant [Tue, 18 Nov 2014 14:46:06 +0000 (14:46 +0000)]
Drop RpcProxy usage from FWaaS code
This patch removes the usage of the RpcProxy compatibility class from
the FWaaS code. The equivalent direct usage of oslo.messaging APIs
are now used instead.
Russell Bryant [Fri, 14 Nov 2014 14:21:05 +0000 (14:21 +0000)]
Drop RpcProxy usage from neutron.agent.rpc.PluginApi
This patch removes the usage of the RpcProxy compatibility class from
the neutron.agent.rpc.PluginApi class. The equivalent use of
oslo.messaging APIs have been put in place instead. This simple
conversion had a pretty wide impact on unit tests, as well.
The security groups API was converted in this patch as well. It was
necessary because the security group class is used as a mixin, so it
must be implemented the same way. Unfortunately, the way this is used
as a mix-in is not consistent, so for now it's only conditionally
converted.
Finally, some other miscellaneous plugin specific interfaces were
converted as well. Again, these were methods mixed-in for certain
plugins.
Note that there's one very minor functional difference in this patch.
The previous code set the base version to be '1.1'. The right pattern
is for this to be set to '1.0'. This version is the default version
specified by the client, telling the server that it must implement at
least this version to satisfy the request. The default should be
'1.0' and methods that require higher than that should specify it.
From looking at other parts of the code, '1.0' vs '1.1' is not
actually important, as '1.1' was actually the addition of some
security group methods defined elsewhere. The correction is more
about establishing the right pattern to follow.
Russell Bryant [Mon, 17 Nov 2014 21:33:06 +0000 (21:33 +0000)]
Fix a copy/pasted test mistake
Fix a minor test mistake that was copied around a few times. In mock
setup, the intention was to only return a value of the rpc method used
was a 'call', not 'cast'. Unfortunately, the variable checked was
method instead of rpc_method, so this expected return was always set
to None.
This also points out that what's being tested here isn't what *really*
was intended. It ends up testing that the code returns what we mocked
it to return, which isn't all that useful. However, at least with
this fix it's a bit more clear what it's trying to do.
Russell Bryant [Thu, 13 Nov 2014 21:31:10 +0000 (21:31 +0000)]
Drop several uses of RpcCallback
This patch drops several uses of the RpcCallback compatibility class.
All of these were trivial and straight forward conversions so I
batched them up.
There are still several other uses of RpcCallback, but the conversions
are were not necessarily trivial, and may be broken in one way or
another, so I wanted to address them separately. In particular, the
use of mixin classes means that there could be cases where the version
declaration is being stepped on, so they need to be investigated more
closely.
Russell Bryant [Wed, 12 Nov 2014 21:37:24 +0000 (21:37 +0000)]
Add some basic rpc api docs
The devref docs had a placeholder file for rpc API docs. Now that
both a client side and server side interface have been converted from
the rpc compat layer to oslo.messaging APIs, add some docs that give
an overview of what the client and server sides look like.
Also include a section that describes the code layout of where you can
find client and server api implementations in the neutron code base.
It starts by discussing the DHCP agent related APIs.
Russell Bryant [Tue, 11 Nov 2014 19:23:15 +0000 (14:23 -0500)]
Drop RpcProxy usage from PluginReportStateAPI
Drop the usage of the RpcProxy compatibility class from the
PluginReportStateAPI rpc client class. It now uses the appropriate
oslo.messaging APIs directly, instead.
John Perkins [Mon, 6 Oct 2014 21:24:57 +0000 (16:24 -0500)]
Fix hostname regex pattern
Current hostname_pattern regex complexity grows exponentially
when given a string of just digits, which can be exploited to
cause neutron-server to freeze.
Eugene Nikanorov [Mon, 17 Nov 2014 16:49:09 +0000 (20:49 +0400)]
Catch DBReferenceError exception during binding a router
In some cases (Concurrent API tests) router can be deleted
before it is bound to an agent by scheduler.
That may lead to traces on L3 agent side returned via RPC.
Need to handle this case in more graceful way.
Itsuro Oda [Wed, 29 Oct 2014 23:00:07 +0000 (08:00 +0900)]
Enable default SNAT from networks connected to a router indirectly
Make outgoing packets to an external interface SNATed regardless
of source address of the packets. As a result of deep review,
any problem was not found with this change.
Kevin Benton [Fri, 26 Sep 2014 04:42:39 +0000 (21:42 -0700)]
BSN: Optimistic locking strategy for consistency
Summary:
Adds an optimistic locking strategy for the Big Switch
server manager so multiple Neutron servers wanting to
communicate with the backend do not receive the consistency
hash for use simultaneously.
The bsn-rest-call semaphore is removed because serialization
is now provided by the new locking scheme.
A new DB engine is added because the consistency hashes
need a life-cycle with rollbacks and other DB operations
than cannot impact or be impacted by database operations
happening on the regular Neutron objects.
Unit tests are included for each of the new branches
introduced.
Problem Statement:
Requests to the Big Switch controllers must contain the
consistency hash value received from the previous update.
Otherwise, an inconsistency error will be triggered which
will force a synchronization. Essentially, a new backend
call must be prevented from reading from the consistency
hash table in the DB until the previous call has updated
the table with the hash from the server response.
This can be addressed by a semaphore around the rest_call
function for the single server use case and by a table lock
on the consistency table for multiple Neutron servers.
However, both solutions are inadequate because a single
Neutron server does not scale and a table lock is not
supported by common SQL HA deployments (e.g. Galera).
This issue was previously addressed by deploying servers
in an active-standby configuration. However, that only
prevented the problem for HTTP API calls. All Neutron
servers would respond to RPC messages, some of which would
result in a port update and possible backend call which
would trigger a conflict if it happened at the same time
as a backend call from another server. These unnecessary
syncs are unsustainable as the topology increases beyond
~3k VMs.
Any solution needs to be back-portable to Icehouse so new
database tables, new requirements, etc. are all out of the
question.
Solution:
This patch stores the lock for the consistency hash as a part
of the DB record. The guaruntees the database offers around
atomic insertion and constrained atomic updates offer the
primitives necessary to ensure that only one process/thread
can lock the record at once.
The read_for_update method is modified to not return the hash
in the database until an identifier is inserted into the
current record or added as a new record. By using an UPDATE
query with a WHERE clause restricting to the current state,
only one of many concurrent callers to the DB will successfully
update the rows. If a caller sees that it didn't update any
rows, it will start the process over of trying to get the
lock.
If a caller observes that the same ID has the lock for
more than 60 seconds, it will assume the holder has
died and will attempt to take the lock. This is also done
in a concurrency-safe UPDATE call since there may be many
other callers may attempt to do the same thing. If it
fails and the lock was taken by someone else, the process
will start over.
Some pseudo-code resembling the logic:
read_current_lock
if no_record:
insert_lock
sleep_and_retry if constraint_violation else return
if current_is_locked and not timer_exceeded:
sleep_and_retry
if update_record_with_lock:
return
else:
sleep_and_retry
This patch adds a check for validating the devices for a network
gateway are defined in the neutron DB before the request for
creating the network gateway is sent to the NSX backend.
Russell Bryant [Fri, 7 Nov 2014 15:30:15 +0000 (16:30 +0100)]
Drop usage of RpcProxy from L3PluginApi
Drop the usage of the RpcProxy compatibility class from the
L3PluginApi. The equivalent direct usage of the oslo.messaging APIs
are now being used instead.
Kevin Benton [Tue, 18 Nov 2014 16:34:56 +0000 (08:34 -0800)]
Prevent an iteration through ports on IPv6 slaac
A recent change[1] allowed subnets to be deleted even if they
had active IPv6 SLAAC allocations on them. The updated check
was inefficient because it would check every port in the subnet
even if the subnet was a SLAAC subnet. This patch just shortcuts
out that check.
Kevin Benton [Tue, 18 Nov 2014 15:51:08 +0000 (07:51 -0800)]
Use a string multiplier instead of 59 repetitions
A recently added test case added a long string of repeated 1's
which required a #noqa flag. A string multiplier represents the
same thing in a much shorter fashion.
Ann Kamyshnikova [Mon, 10 Nov 2014 14:05:54 +0000 (17:05 +0300)]
Convert all incoming protocol numbers to string
PostgreSQL is more sensitive for types than MySQL when it selects
something from columns in database. So it fails when it tries to
select from string field comparing with integer value.
Added unit test to verify conversion of protocol numbers to
strings.
Cedric Brandily [Tue, 18 Nov 2014 09:59:31 +0000 (10:59 +0100)]
Correct raw table regex in test_security_groups_rpc
Currently in test_security_groups_rpc, raw table regex uses nat table
chains regex but raw table chains[1] is a strict subset of nat table
chains[2] so a more strict regex should be used as raw table chains
regex.
Dane LeBlanc [Mon, 17 Nov 2014 15:40:04 +0000 (10:40 -0500)]
Subnet delete for IPv6 SLAAC should not require prior port disassoc
With the current Neutron implementation, a subnet cannot be deleted
until all associated IP addresses have been remove from ports (via
port update) or the associated ports/VMs have been deleted.
In the case of SLAAC-enabled subnets, however, it's not feasible to
require removal of SLAAC-generated addresses individually from each
associated port before deleting a subnet because of the multicast
nature of RA messages. For SLAAC-enabled subnets, the processing of
subnet delete requests needs to be changed so that these subnets will
be allowed to be deleted, and all ports get disassociated from their
corresponding SLAAC IP address, when there are ports existing on
the SLAAC subnet.
Russell Bryant [Fri, 7 Nov 2014 14:31:15 +0000 (15:31 +0100)]
Fix client side versions in dhcp rpc API
The dhcp rpc API has two version (1.0 and 1.1). The proper way to use
versioning for this is to only specify '1.1' from the client side when
you require that the remote side implements at least version '1.1' for
the method to work. Otherwise, '1.0' should still be specified. The
previous code specified '1.1' always.
Russell Bryant [Fri, 7 Nov 2014 14:11:42 +0000 (15:11 +0100)]
Drop usage of RpcProxy from DhcpPluginApi
Drop the usage of the RpcProxy compatibility class from the
DhcpPluginApi RPC client class. The implementation has been updated
to use the appropariate APIs from oslo.messaging directly.
Romil Gupta [Sun, 16 Nov 2014 17:35:35 +0000 (09:35 -0800)]
Moved out common testcases from test_type_vxlan.py
This patch-set will refactor the test_type_vxlan.py and test_type_gre.py.
We need to add testcases for TunnelRpcCallbackMixin() so better we
should move common test cases from test_type_vxlan.py to test_type_tunnel.py and
add testcases for TunnelRpcCallbackMixin() into it.
Code Review / openstack-build / neutron-build.git / log