test(testkit): separate destructive tests from non-destructive ones #5527

User Story

I'm moving the e2e/rust tests onto testkit (#5281). Some of them only create and delete their own sandboxes. Others change the environment they run in: they set a gateway-wide setting, restart the gateway, or scale and roll gateway pods. I need a clear rule for which kind is which, so that a destructive test never runs against a gateway someone depends on and never runs beside tests it can break.

Description

Define how testkit tells destructive tests from non-destructive ones, and where each kind may run.

• Non-destructive: the test creates resources with run-unique names, such as sandboxes, templates and providers, and removes them on cleanup. It only reads gateway-wide state. It can run against any gateway, including a user's own, and beside other tests. Conformance is limited to these.
• Destructive: the test changes state that other tests or users of the same gateway can see. Known cases:
  • settings_management::settings_global_override_round_trip sets a global setting. That changes every sandbox on the gateway.
  • Gateway restart tests (PR6 in the test(e2e): decide which e2e/rust tests to port and in what order #5297 plan, decision A) stop and start the gateway.
  • Kubernetes HA rebalancing (PR8) scales and rolls the gateway pods.

Decide:

1. Marking. How a destructive test is marked. Options include a separate feature suite, a testkit attribute, or a nextest test group.
2. Opting in. How an environment allows destructive tests. For example, tmachine starts a dedicated gateway and publishes a variable, and testkit fails a destructive test without it. A developer running the suites against their own gateway must not change its state by accident.
3. Isolation. How destructive tests are kept from running beside other tests on the same gateway. This may use the serial test groups from test(runner): apply per-family nextest config and a thread count to runner rows #5295.
4. Config changes. Tests don't change gateway config files. The environment lays down any non-default config before the gateway starts, as discussed for drop-in config files (feat(gateway): merge drop-in config fragments over the base file #5489). Confirm that this rule also covers runtime settings, or say where it stops.

Context

• Came out of test(conformance): port sandbox lifecycle and API e2e tests to testkit #5431. Its sandbox_setting_round_trip used to clear and restore the global ocsf_json_enabled. It now only reads that value and fails if one is set.
• settings_global_override_round_trip stays in e2e/rust until this is decided.
• test(runner): apply per-family nextest config and a thread count to runner rows #5295 adds per-family nextest config and serial test groups. That covers isolation but not opting in.
• Part of test: retire the old e2e test system onto testkit #5281.

Definition of Done

• The rule for destructive and non-destructive tests, and where each may run, is written down in the testkit or tests README.
• A destructive test fails, rather than running, against an environment that hasn't opted in.
• The test(e2e): decide which e2e/rust tests to port and in what order #5297 plan and the affected PR issues name, for each destructive test, how it is marked and which environments run it.