UploadNow is not safe to call asynchronously with FlushAndTeardown

[bliptec]

It's possible for UploadNow to execute while the system is half-shutdown in FlushAndTeardown. In Edge this manifested as a crash when trying to acquire the (now destroyed) lock for offline storage, which had already been shut down.

UploadNow does check that the log manager exists, but does not guarantee that the log manager is valid throughout the lifetime of the call. Max Golovanov (@maxgolov) was able to repro a similar crash by adding an artificial pause is FushAndTeardown

[maxgolov]

Current suggestion for Edge is:

• either avoid using it.
• or add an extra state check to invoke UploadNow only within active session, i.e. guaranteed in between the calls to Initialize and before FlushAndTeardown. Basically perform an extra atomic state check to see whether the app has called into F&TD. If it did, then don't call into UploadNow.

This has generally been the guidance w.r.t. all other SDK methods, including ILogger usage - you can't use these concurrently with shutting down the stack..

But I got a repro on my setup, so I believe we can make UploadNow more safe as follows:

• if invoked before Initialize or after Teardown via LogManagerBase.hpp wrapper -- it'd be a no-op.
• if invoked concurrently with FlushAndTeardown -- it'd be a no-op.
• so it'd only work if TelemetrySystem is alive, not paused and up and running.

[maxgolov]

Please verify the fixes integrated in latest master (3/9/2020). There were tests specifically added to stress-test this scenario. Please reopen if you get a repro on latest SDK. We discussed that the app should possibly apply the following changes:

• avoid calling UploadNow from callback while shutting down.
• avoid calling FlushAndTeardown if the callback is executing UploadNow, i.e. wait for UploadNow async call returning before proceeding with FlushAndTeardown.

Scenario has been addressed at core level: SDK respects the current Initialization state, and safely concludes / aborts UploadNow, also not letting UploadNow happen in case if FlushAndTeardown is running or if the instance of LOgManager is destroyed.