I noticed that ri.floating_ips was being accessed from the agent. It
didn't seem right. Also, noticed that this set gets passed around too
much between methods. It all boils down to saving the previous set
and then doing set difference to see which ones went away. This patch
set makes it simpler.