From 41d0498dc4bf617fc87edb74bea4d2b33c487f69 Mon Sep 17 00:00:00 2001 From: Assaf Muller Date: Mon, 16 Nov 2015 15:31:43 -0500 Subject: [PATCH] Moved fullstack test doc content to TESTING.rst * Removed long term goals documentation (I don't see a need to document these). * Added and rearranged short term goals. Change-Id: If494533cb6507f18b84a41b3f1daf42cd10d9f51 --- TESTING.rst | 87 ++++++++++++++++-- doc/source/devref/fullstack_testing.rst | 112 ++---------------------- neutron/tests/fullstack/README | 2 +- 3 files changed, 90 insertions(+), 111 deletions(-) diff --git a/TESTING.rst b/TESTING.rst index d71fa220e..0267925fd 100644 --- a/TESTING.rst +++ b/TESTING.rst @@ -56,11 +56,82 @@ properly cleaned up both on test success and failure. Fullstack Tests ~~~~~~~~~~~~~~~ -Fullstack tests (neutron/tests/fullstack/) target Neutron as a whole. -The test infrastructure itself manages the Neutron server and its agents. -Fullstack tests are a form of integration testing and fill a void between -unit/functional tests and Tempest. More information may be found -`here. `_ +Why? +++++ + +The idea behind "fullstack" testing is to fill a gap between unit + functional +tests and Tempest. Tempest tests are expensive to run, and target black box API +tests exclusively. Tempest requires an OpenStack deployment to be run against, +which can be difficult to configure and setup. Full stack testing addresses +these issues by taking care of the deployment itself, according to the topology +that the test requires. Developers further benefit from full stack testing as +it can sufficiently simulate a real environment and provide a rapidly +reproducible way to verify code while you're still writing it. + +How? +++++ + +Full stack tests set up their own Neutron processes (Server & agents). They +assume a working Rabbit and MySQL server before the run starts. Instructions +on how to run fullstack tests on a VM are available below. + +Each test defines its own topology (What and how many servers and agents should +be running). + +Since the test runs on the machine itself, full stack testing enables +"white box" testing. This means that you can, for example, create a router +through the API and then assert that a namespace was created for it. + +Full stack tests run in the Neutron tree with Neutron resources alone. You +may use the Neutron API (The Neutron server is set to NOAUTH so that Keystone +is out of the picture). VMs may be simulated with a container-like class: +neutron.tests.fullstack.resources.machine.FakeFullstackMachine. +An example of its usage may be found at: +neutron/tests/fullstack/test_connectivity.py. + +Full stack testing can simulate multi node testing by starting an agent +multiple times. Specifically, each node would have its own copy of the +OVS/DHCP/L3 agents, all configured with the same "host" value. Each OVS agent +is connected to its own pair of br-int/br-ex, and those bridges are then +interconnected. + +.. image:: images/fullstack_multinode_simulation.png + +Segmentation at the database layer is guaranteed by creating a database +per test. The messaging layer achieves segmentation by utilizing a RabbitMQ +feature called 'vhosts'. In short, just like a MySQL server serve multiple +databases, so can a RabbitMQ server serve multiple messaging domains. +Exchanges and queues in one 'vhost' are segmented from those in another +'vhost'. + +When? ++++++ + +1) You'd like to test the interaction between Neutron components (Server + and agents) and have already tested each component in isolation via unit or + functional tests. You should have many unit tests, fewer tests to test + a component and even fewer to test their interaction. Edge cases should + not be tested with full stack testing. +2) You'd like to increase coverage by testing features that require multi node + testing such as l2pop, L3 HA and DVR. +3) You'd like to test agent restarts. We've found bugs in the OVS, DHCP and + L3 agents and haven't found an effective way to test these scenarios. Full + stack testing can help here as the full stack infrastructure can restart an + agent during the test. + +Example ++++++++ + +Neutron offers a Quality of Service API, initially offering bandwidth +capping at the port level. In the reference implementation, it does this by +utilizing an OVS feature. +neutron.tests.fullstack.test_qos.TestQoSWithOvsAgent.test_qos_policy_rule_lifecycle +is a positive example of how the fullstack testing infrastructure should be used. +It creates a network, subnet, QoS policy & rule and a port utilizing that policy. +It then asserts that the expected bandwidth limitation is present on the OVS +bridge connected to that port. The test is a true integration test, in the +sense that it invokes the API and then asserts that Neutron interacted with +the hypervisor appropriately. API Tests ~~~~~~~~~ @@ -260,9 +331,11 @@ tools/configure_for_func_testing.sh is advised (As described above). When running full-stack tests on a clean VM for the first time, we advise to run ./stack.sh successfully to make sure all Neutron's dependencies are met. Full-stack based Neutron daemons produce logs to a -sub-folder in /tmp/fullstack-logs (for example, a test named -"test_example" will produce logs to /tmp/fullstack-logs/test_example/), +sub-folder in /tmp/dsvm-fullstack-logs (for example, a test named +"test_example" will produce logs to /tmp/dsvm-fullstack-logs/test_example/), so that will be a good place to look if your test is failing. +Fullstack test suite assumes 240.0.0.0/4 (Class E) range in root namespace of +the test machine is available for its usage. API Tests +++++++++ diff --git a/doc/source/devref/fullstack_testing.rst b/doc/source/devref/fullstack_testing.rst index 6e1db5c36..fca5d4341 100644 --- a/doc/source/devref/fullstack_testing.rst +++ b/doc/source/devref/fullstack_testing.rst @@ -24,112 +24,18 @@ Full Stack Testing ================== -Why? ----- - -The idea behind "fullstack" testing is to fill a gap between unit + functional -tests and Tempest. Tempest tests are expensive to run, and target black box API -tests exclusively. Tempest requires an OpenStack deployment to be run against, -which can be difficult to configure and setup. Full stack testing addresses -these issues by taking care of the deployment itself, according to the topology -that the test requires. Developers further benefit from full stack testing as -it can sufficiently simulate a real environment and provide a rapidly -reproducible way to verify code while you're still writing it. - -How? ----- - -Full stack tests set up their own Neutron processes (Server & agents). They -assume a working Rabbit and MySQL server before the run starts. Instructions -on how to run fullstack tests on a VM are available in our -`TESTING.rst. `_ - -Each test defines its own topology (What and how many servers and agents should -be running). - -Since the test runs on the machine itself, full stack testing enables -"white box" testing. This means that you can, for example, create a router -through the API and then assert that a namespace was created for it. - -Full stack tests run in the Neutron tree with Neutron resources alone. You -may use the Neutron API (The Neutron server is set to NOAUTH so that Keystone -is out of the picture). VMs may be simulated with a container-like class: -neutron.tests.fullstack.resources.machine.FakeFullstackMachine. -An example of its usage may be found at: -neutron/tests/fullstack/test_connectivity.py. - -Full stack testing can simulate multi node testing by starting an agent -multiple times. Specifically, each node would have its own copy of the -OVS/DHCP/L3 agents, all configured with the same "host" value. Each OVS agent -is connected to its own pair of br-int/br-ex, and those bridges are then -interconnected. - -.. image:: images/fullstack_multinode_simulation.png - -Segmentation at the database layer is guaranteed by creating a database -per test. The messaging layer achieves segmentation by utilizing a RabbitMQ -feature called 'vhosts'. In short, just like a MySQL server serve multiple -databases, so can a RabbitMQ server serve multiple messaging domains. -Exchanges and queues in one 'vhost' are segmented from those in another -'vhost'. - -When? +Goals ----- -1) You'd like to test the interaction between Neutron components (Server - and agents) and have already tested each component in isolation via unit or - functional tests. You should have many unit tests, fewer tests to test - a component and even fewer to test their interaction. Edge cases should - not be tested with full stack testing. -2) You'd like to increase coverage by testing features that require multi node - testing such as l2pop, L3 HA and DVR. -3) You'd like to test agent restarts. We've found bugs in the OVS, DHCP and - L3 agents and haven't found an effective way to test these scenarios. Full - stack testing can help here as the full stack infrastructure can restart an - agent during the test. - -Example -------- - -Neutron offers a Quality of Service API, initially offering bandwidth -capping at the port level. In the reference implementation, it does this by -utilizing an OVS feature. -neutron.tests.fullstack.test_qos.TestQoSWithOvsAgent.test_qos_policy_rule_lifecycle -is a positive example of how the fullstack testing infrastructure should be used. -It creates a network, subnet, QoS policy & rule and a port utilizing that policy. -It then asserts that the expected bandwidth limitation is present on the OVS -bridge connected to that port. The test is a true integration test, in the -sense that it invokes the API and then asserts that Neutron interacted with -the hypervisor appropriately. - -Prerequisites -------------- - -Fullstack test suite assumes 240.0.0.0/3 range in root namespace of the test -machine is available for its usage. - -Short Term Goals ----------------- - -* Multinode & Stability: - - Convert the L3 HA failover functional test to a full stack test - - Write a test for DHCP HA / Multiple DHCP agents per network +* Stabilize the job: + - Fix L3 HA failure + - Look in to non-deterministic failures when adding a large amount of + tests (Possibly bug 1486199). + - Switch to kill signal 15 to terminate neutron-server & agents (Bugs + 1487548 and 1494363). +* Convert the L3 HA failover functional test to a full stack test +* Write a test for DHCP HA / Multiple DHCP agents per network * Write DVR tests * Write additional L3 HA tests * Write a test that validates DVR + L3 HA integration after https://bugs.launchpad.net/neutron/+bug/1365473 is fixed. - -Long Term Goals ---------------- - -* How will advanced services use the full stack testing infrastructure? Full - stack tests infrastructure classes are expected to change quite a bit over - the next coming months. This means that other repositories may import these - classes and break from time to time, or copy them in their repositories - instead. Since changes to full stack testing infrastructure is a given, - XaaS repositories should be copying it and not importing it directly. -* Currently we configure the Neutron server with the ML2 plugin and the OVS - mechanism driver. We may modularize the topology configuration further to - allow to rerun full stack tests against different Neutron plugins or ML2 - mechanism drivers. -* Add OVS ARP responder coverage when the gate supports OVS 2.1+ diff --git a/neutron/tests/fullstack/README b/neutron/tests/fullstack/README index adbbd70f2..c172c70c8 100644 --- a/neutron/tests/fullstack/README +++ b/neutron/tests/fullstack/README @@ -1 +1 @@ -Please see neutron/doc/source/devref/fullstack_testing.rst. \ No newline at end of file +Please see neutron/TESTING.rst. \ No newline at end of file -- 2.45.2