Clean up pendingSegments table

[dclim]

The pendingSegments table should have entries purged when they're no longer required to determine sequences of segments.

See: https://groups.google.com/forum/#!topic/druid-user/O0yxORw92VM

[haoxiang47]

Ok, I will try to fix this issue

[dclim]

@haoxiang47 expressed interest in submitting a PR to fix this issue - @gianm do you have any thoughts about how best to do this?

[gianm]

Hmm, no immediate thoughts, other than we should make sure that we don't accidentally clean up pending segments that might be used by ingestions that are either long running or have been paused for a long time.

[dclim]

Do you see any issue with cleaning the entries up after the first task in a set of replicas successfully completes handoff in FiniteAppenderatorDriver.finish()? At least in the KafkaIndexTask case, this shouldn't be problematic because a) All the replicas should already be in publishing state and not allocating any more segments, and b) If one replica succeeds handoff, the rest of the replicas will be stopped shortly.

Alternately KafkaSupervisor could manage the entry removal once all the tasks associated with a sequenceName have been completed or stopped, but it feels cleaner doing it in FiniteAppenderatorDriver since it was the one who created the entry in the first place.

[gianm]

Ideally we don't rely on something acting at a single point in time for cleanup, because failures of that thing could leave rows dangling around forever. If we can remove all stale pending segments though, and not just the ones for our current sequence, that'd work.

[gianm]

(subject to being careful not to remove pending segments that are for still active sequences)

[dclim]

Hm alright, another way we could handle this is having a cleanup thread on the overlord that periodically compares the druid_pendingSegments table to the druid_tasks table and removes any entries in druid_pendingSegments with a sequenceName that doesn't have a corresponding 'active' entry in druid_tasks.

[haoxiang47]

well, in our system we have lots of datasources, so in overload it will create lots of tasks, when the times go, the table will biggger than before so that it will influence query from mysql. Now we just add index in druid_pendingSegments table, and it make the query fast a lot. So I think we can first simplely add indexes to this table and than cleanup the table automatically.

[jon-wei]

Addressed by #5149