Deal with tgt already exists errors

So there's a major problem in the iscsi code (and has been
for quite some time).  The tgt show command sometimes seems
to be corrupt and the issued command is actually a tgt create.
The result is that we already have a tgt and the call raises
which then causes the operation to fail.

An example of the issue:
  Stdout: Unexpected error while running command.Command:
  sudo cinder-rootwrap /etc/cinder/rootwrap.conf tgt-admin
  --update iqn.2010-10.org.openstack:volume-f055d3c5-db7a-
  484e-9d0d-b98495439413
  Exit code: 22
  Stdout:
  Command:tgtadm -C 0 --lld iscsi --op new --mode target
  --tid 1 -T iqn.2010-10.org.openstack:volume-f055d3c5-db7a-
  484e-9d0d-b98495439413
  exited with code: 22.
  Stderr: u'tgtadm: this target already exists

What's disturbing however is that in that section of code
we're sending a --op show!!  Could be something we're
doing with our member executor?  Or maybe something to
do with the new oslo concurrency code?

Regardless, his patch intends to provide a clear marker for
ER in the case that create export fails due to the
target entry already existing.

Also this patch will enable us to go ahead and just use
the existing target rather than bomb out and fail everything.
Root cause of why we're getting a second create is still
unknown and needs addressed, but this might help in getting
more info as well as keeping things stable until we address
the root issue.

Change-Id: I0faf4d49b2d3e631b08ec1dff4361ff2376e3308
Partial-Bug: #1398078

diff --git a/cinder/tests/targets/test_tgt_driver.py b/cinder/tests/targets/test_tgt_driver.py
index ecb36ffb3414576bf0930361b4bd701874addad5..9682b489180aa080f67f05f129d22b807eb76119 100644 (file)
--- a/cinder/tests/targets/test_tgt_driver.py
+++ b/cinder/tests/targets/test_tgt_driver.py
@@ -14,6 +14,7 @@ import tempfile
 
 import mock
 from oslo.utils import timeutils
+from oslo_concurrency import processutils as putils
 
 from cinder import context
 from cinder import test
@@ -197,6 +198,36 @@ class TestTgtAdmDriver(test.TestCase):
                 0,
                 self.fake_volumes_dir))
 
+    def test_create_iscsi_target_already_exists(self):
+        def _fake_execute(*args, **kwargs):
+            raise putils.ProcessExecutionError(
+                exit_code=1,
+                stdout='',
+                stderr='target already exists',
+                cmd='tgtad --lld iscsi --op show --mode target')
+
+        self.stubs.Set(self.target,
+                       '_execute',
+                       _fake_execute)
+
+        self.stubs.Set(self.target,
+                       '_get_target',
+                       lambda x: 1)
+
+        self.stubs.Set(self.target,
+                       '_verify_backing_lun',
+                       lambda x, y: True)
+
+        test_vol = 'iqn.2010-10.org.openstack:'\
+                   'volume-83c2e877-feed-46be-8435-77884fe55b45'
+        self.assertEqual(
+            1,
+            self.target.create_iscsi_target(
+                test_vol,
+                1,
+                0,
+                self.fake_volumes_dir))
+
     def test_create_create_export(self):
 
         def _fake_execute(*args, **kwargs):

diff --git a/cinder/volume/targets/tgt.py b/cinder/volume/targets/tgt.py
index eff730b3bd52edcdaf3ab02a1de87e7d11a08393..f5cf3b8466312e1b3483b2eb806d1728add1408f 100644 (file)
--- a/cinder/volume/targets/tgt.py
+++ b/cinder/volume/targets/tgt.py
@@ -232,13 +232,25 @@ class TgtAdm(iscsi.ISCSITarget):
                                        run_as_root=True)
             LOG.debug("Targets after update: %s" % out)
         except putils.ProcessExecutionError as e:
-            LOG.warning(_LW("Failed to create iscsi target for volume "
-                        "id:%(vol_id)s: %(e)s")
-                        % {'vol_id': vol_id, 'e': e})
-
-            # Don't forget to remove the persistent file we created
-            os.unlink(volume_path)
-            raise exception.ISCSITargetCreateFailed(volume_id=vol_id)
+            if "target already exists" in e.stderr:
+                LOG.warning(_LW('Could not create target because '
+                                'it already exists for volume: %s'), vol_id)
+                # NOTE(jdg): We've run into issues where the command being sent
+                # was not correct. This may be related to using the executor
+                # directly? Even though the above call specified is a show
+                # we see a new being called instead...
+
+                # Adding the additional Warning message above for a clear
+                # ER marker (Ref bug: #1398078).
+                pass
+            else:
+                LOG.warning(_LW("Failed to create iscsi target for volume "
+                            "id:%(vol_id)s: %(e)s")
+                            % {'vol_id': vol_id, 'e': e})
+
+                # Don't forget to remove the persistent file we created
+                os.unlink(volume_path)
+                raise exception.ISCSITargetCreateFailed(volume_id=vol_id)
 
         iqn = '%s%s' % (self.iscsi_target_prefix, vol_id)
         tid = self._get_target(iqn)