Kevin Benton [Sat, 3 Oct 2015 14:25:19 +0000 (07:25 -0700)]
Use diffs for iptables restore instead of all rules
This patch changes our iptables logic to generate a delta of
iptables commands (inserts + deletes) to get from the current
iptables state to the new state. This will significantly reduce
the amount of data that we have to shell out to iptables-restore
on every call (and reduce the amount of data iptables-restore has
to parse).
We no longer have to worry about preserving counters since
we are adding and deleting specific rules, so the rule modification
code got a nice cleanup to get rid of the old rule matching.
This also gives us a new method of functionally testing that we are
generating rules in the correct manner. After applying new rules
once, a subsequent call should always have no work to do. The new
functional tests added leverage that property heavily and should
protect us from regressions in how rules are formed.
The _apply_synchronized times seem to converge when dealing
with ~85k rules. In the profile I can see that both approaches
seem to wait on iptables-restore for approximately the same
amount of time so it could be hitting the performance limits
of iptables-restore.
Kevin Benton [Mon, 5 Oct 2015 14:36:39 +0000 (07:36 -0700)]
Fix iptables modules references in rule generation
The way we were generating rules with module references
for TCP, UDP, and ICMP was not matching the output of
iptables-save so all of the counters for those rules
were being destroyed on each iptables reload.
This patch corrects the generation so it's in line with
iptables-save output. It uses the matching module name only
when a specific port number or port range is specified.
It also uses the full 'ipv6-icmp' protocol name that shows
up in the output rather than 'icmpv6'.
Kevin Benton [Mon, 5 Oct 2015 14:57:54 +0000 (07:57 -0700)]
Only lock in set_members on mutating operations
ipset was locking on every set_members call with an external
filesystem lock. This was expensive when lots of ports that
were a part of the same security group were on the same agent.
This patch adjusts it to check if it needs to make a change before
acquiring the semaphore.
Kevin Benton [Mon, 5 Oct 2015 14:06:54 +0000 (07:06 -0700)]
Remove excessive fallback iptables ACCEPT rules
The previous code was generating a fallback ACCEPT rule for every
port when there should only be one at the very end. The reason that
this wasn't causing a bug is because we have a duplicate rule remover
that was silently throwing away the extras and it happened to get them
in the right order.
This will remove the need of import random in unit/objects/test_base.py
as its already done in tests/tools.py and some of the functions are
called from there.
Ihar Hrachyshka [Thu, 8 Oct 2015 12:56:30 +0000 (14:56 +0200)]
Cleaned up remaining incorrect usage for LOG.exception
- callers should not explicitly pass exceptions into LOG.exception
because it's already implicitly included in the message by stdlib
logging module.
- callers should not call to LOG.exception when there is no exception to
log about (known to fail in Python 3.x < 3.5).
Kevin Benton [Wed, 7 Oct 2015 03:16:15 +0000 (20:16 -0700)]
Fixed multiple py34 gate issues
1. Scope mock of 'open' to module
By mocking 'open' at the module level, we can avoid affecting
'open' calls from other modules.
2. Stop using LOG.exception in contexts with no sys.exc_info set
Python 3.4 logger fills in record.exc_info with sys.exc_info() result
[1], and then it uses it to determine the current exception [2] to
append to the log message. Since there is no exception, exc_info[1] is
None, and we get AttributeError inside traceback module.
It's actually a bug in Python interpreter that it attempt to access the
attribute when there is no exception. It turns out that it's fixed in
latest master of cPython [3] (the intent of the patch does not seem
relevant, but it removes the offending code while reshuffling the code).
Note that now cPython correctly checks the exception value before
accessing its attributes [4].
The patch in cPython that resulted in the failure is [5] and is present
since initial Python 3k releases.
Ihar Hrachyshka [Mon, 5 Oct 2015 15:46:33 +0000 (17:46 +0200)]
Removed neutronclient option from metadata agent
The new RPC interface has proved itself for two cycles, I don't
recollect any serious issues with it, so let's just clean up the
obsolete neutronclient based fallback mechanism.
The metadata agent configuration documentation should be updated
to not require API configuration values for the agent to talk to
neutron-server.
The Drivers team realized that with the current model, RFE
bugs were never off their radar even though RFE proposals were
approved and ready to be worked on. As a result it was becoming
tricky to handle the RFE backlog during the drivers meeting.
These changes are aimed at keeping the list of RFE bugs to a
manageable size, irrespective of how fast code gets submitted
and merged: the responsibilities of vetting feature proposals
and reviewing code are very much different, and this policy
changes try to reflect that.
Michael Smith [Fri, 5 Dec 2014 00:15:43 +0000 (16:15 -0800)]
L3 Agent support for routers with HA and DVR
The main difference for DVR HA routers is where
the VRRP/keepalived logic is run and which ports
fall in the HA domain for DVR. Instead of running
in the qrouter namespace, keepalived will run inside
the snat-namespace. Therefore only snat ports will
fall under the control of the HA domain.
Change-Id: If2962580397d39f72fd1fbbc1188a6958f00ff0c Co-Authored-By: Michael Smith <michael.smith6@hp.com> Co-Authored-By: Hardik Italia <hardik.italia@hp.com> Co-Authored-By: Adolfo Duarte <adolfo.duarte@hp.com> Co-Authored-By: John Schwarz <jschwarz@redhat.com>
Kevin Benton [Mon, 5 Oct 2015 13:37:40 +0000 (06:37 -0700)]
Correct MAC representation to match iptables output
We were previously using the netaddr's mac_unix format
(which leaves off leading 0's) to generate iptables rules
based on MAC addresses. While iptables accepts this format,
it's not returned this way in the output so the iptables
rule matching code would never find the match for these
rules, causing the loss of counters on these rules on every
reload.
This patch corrects this with a custom dialect that matches
the iptables format.
This plugin didn't decompose in the last two cycles, I failed
to spot a functional CI, and there hasn't been any meaningful
activity done in the subtree for the past couple of cycles
Code Review / openstack-build / neutron-build.git / log