Enhance TESTING.rst

Add detail about api testing and provide better visual separation
between the different types of testing.

The current testing guidelines are mainly about running tests, and
this change does little to fix that.  The intention is to add detail
about writing tests in subsequent changes.

Change-Id: I39d0439c91e5c6edb1d48d4da310443c99fb6d9e

diff --git a/TESTING.rst b/TESTING.rst
index a4f1388d020763f712c6110a43a92eae3e2dbe1e..8afe4ec95f7189f4c853f75308c9732de0c8ce52 100644 (file)
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -4,17 +4,47 @@ Testing Neutron
 Overview
 --------
 
-The unit tests (neutron/test/unit/) are meant to cover as much code as
+Neutron relies on different types of testing to ensure its quality, as
+described below.  In addition to in-tree testing, `Tempest`_ is
+responsible for validating Neutron's integration with other OpenStack
+components, and `Rally`_ is responsible for benchmarking.
+
+.. _Tempest: http://docs.openstack.org/developer/tempest/
+.. _Rally: http://rally.readthedocs.org/en/latest/
+
+Unit Tests
+~~~~~~~~~~
+
+Unit tests (neutron/test/unit/) are meant to cover as much code as
 possible and should be executed without the service running. They are
 designed to test the various pieces of the neutron tree to make sure
 any new changes don't break existing functionality.
 
-The functional tests (neutron/tests/functional/) are intended to
+Functional Tests
+~~~~~~~~~~~~~~~~
+
+Functional tests (neutron/tests/functional/) are intended to
 validate actual system interaction.  Mocks should be used sparingly,
 if at all.  Care should be taken to ensure that existing system
 resources are not modified and that resources created in tests are
 properly cleaned up.
 
+API Tests
+~~~~~~~~~
+
+API tests (neutron/tests/api/) are intended to ensure the function
+and stability of the Neutron API.  As much as possible, changes to
+this path should not be made at the same time as changes to the code
+to limit the potential for introducing backwards-incompatible changes.
+
+Since API tests need to be able to target a deployed Neutron daemon
+that is not necessarily test-managed, they should not depend on
+controlling the runtime configuration of the target daemon.  API tests
+should be black-box - no assumptions should be made about
+implementation.  Only the contract defined by Neutron's REST API
+should be validated, and all interaction with the daemon should be via
+a REST client.
+
 Development process
 -------------------
 
@@ -72,8 +102,8 @@ for DevStack at https://github.com/bcwaldon/vagrant_devstack.
     machine and develop from there.
 
 
-Running unit tests
-------------------
+Running tests
+-------------
 
 There are three mechanisms for running tests: run_tests.sh, tox,
 and nose2. Before submitting a patch for review you should always
@@ -142,11 +172,35 @@ versions of Python (2.6, 2.7, 3.3, etc).
 Testr handles the parallel execution of series of test cases as well as
 the tracking of long-running tests and other things.
 
-Running unit tests is as easy as executing this in the root directory of the
-Neutron source code::
+For more information on the standard Tox-based test infrastructure used by
+OpenStack and how to do some common test/debugging procedures with Testr,
+see this wiki page:
+
+  https://wiki.openstack.org/wiki/Testr
+
+.. _Testr: https://wiki.openstack.org/wiki/Testr
+.. _tox: http://tox.readthedocs.org/en/latest/
+.. _virtualenvs: https://pypi.python.org/pypi/virtualenv
+
+PEP8 and Unit Tests
+===================
+
+Running pep8 and unit tests is as easy as executing this in the root
+directory of the Neutron source code::
 
     tox
 
+To run only pep8::
+
+    tox -e pep8
+
+To run only the unit tests::
+
+    tox -e py27
+
+Functional Tests
+================
+
 To run functional tests that do not require sudo privileges or
 specific-system dependencies::
 
@@ -171,6 +225,9 @@ to install and configure all of Neutron's package dependencies.  It is
 not necessary to provide this option if devstack has already been used
 to deploy Neutron to the target host.
 
+Fullstack Tests
+===============
+
 To run all the full-stack tests, you may use: ::
 
     tox -e dsvm-fullstack
@@ -185,21 +242,20 @@ on the gate, running the dsvm-functional suite will also run all
 full-stack tests (and a new worker won't be assigned specifically for
 dsvm-fullstack).
 
-To run the api tests against a live Neutron daemon, deploy tempest and
-neutron with devstack and then run the following commands: ::
+API Tests
+=========
 
-    export TEMPEST_CONFIG_DIR=/opt/stack/tempest/etc
-    tox -e api
+To run the api tests, deploy tempest and neutron with devstack and
+then run the following command: ::
 
-For more information on the standard Tox-based test infrastructure used by
-OpenStack and how to do some common test/debugging procedures with Testr,
-see this wiki page:
+    tox -e api
 
-  https://wiki.openstack.org/wiki/Testr
+If tempest.conf cannot be found at the default location used by
+devstack (/opt/stack/tempest/etc) it may be necessary to set
+TEMPEST_CONFIG_DIR before invoking tox: ::
 
-.. _Testr: https://wiki.openstack.org/wiki/Testr
-.. _tox: http://tox.readthedocs.org/en/latest/
-.. _virtualenvs: https://pypi.python.org/pypi/virtualenv
+    export TEMPEST_CONFIG_DIR=[path to dir containing tempest.conf]
+    tox -e api
 
 
 Running individual tests