From fdd96e9a00cdb537217de51488aa59890564d0a8 Mon Sep 17 00:00:00 2001 From: Matthew Treinish Date: Fri, 14 Aug 2015 11:41:49 -0400 Subject: [PATCH] Fix some issues around tempest in fullstack testing doc The why section in the fullstack testing doc gives a good explanation of the rational behind the testing and where it fits in the testing pyramid. However, some of the drawbacks of tempest mentioned aren't accurate or are misleading. This commit attempts to reword that piece to clear up any potential sources of confusion. The difficulty in running tempest doesn't change depending on the nature of the deployment, since tempest is an external test suite that interacts with any deployment only through the api. The configuration and run mechanics do not change whether your cloud is 1 or multiple nodes. The real difficulty lies in setting up a multinode deployment. For the failure reporting, if you can't figure out why something failed from a tempest run it's the same for any end user of the API. It should be treated as a bug in the project if an end user can't figure out why something failed from logs and what gets returned by the API. But, since the fullstack tests are a bit lower level its not necessarily trying to catch bugs like that. This commit attempts to reword it to make that distinction clear. Change-Id: Ie5b01047412deb84fe1457100ecd4af48a1d7efc --- doc/source/devref/fullstack_testing.rst | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/doc/source/devref/fullstack_testing.rst b/doc/source/devref/fullstack_testing.rst index 565fda43c..68cb6a518 100644 --- a/doc/source/devref/fullstack_testing.rst +++ b/doc/source/devref/fullstack_testing.rst @@ -28,12 +28,15 @@ Why? ---- The idea behind "fullstack" testing is to fill a gap between unit + functional -tests and Tempest. Tempest tests are expensive to run, difficult to run in -a multi node environment, and are often very high level and provide little -indication to what is wrong, only that something is wrong. Developers further -benefit from full stack testing as it can sufficiently simulate a real -environment and provide a rapidly reproducible way to verify code as you're -still writing it. +tests and Tempest. Tempest tests are expensive to run, and operate only +through the REST API. So they can only provide an explanation of what went wrong +gets reported to an end user via the REST API, which is often too high level. +Additionally, Tempest requires an OpenStack deployment to be run against, which +can be difficult to configure and setup. The full stack testing addresses +these issues by taking care of the deployment itself, according to the topology +that the test requires. Developers further benefit from full stack testing as +it can sufficiently simulate a real environment and provide a rapidly +reproducible way to verify code as you're still writing it. How? ---- -- 2.45.2