[meilisearch] keep the incremental-update tracker in sync with the index

[mishig25]

Problem

docs-semantic-search contains 259 documents that aren't in the incremental-update tracker (hf-doc-build/doc-builder-embeddings-tracker). Incremental runs only delete tracked IDs, so these stale documents are never removed and keep showing up in search results (with outdated content and URLs). All tracked IDs are in the index, so the drift only goes one way. Two code paths can cause it:

• _run_incremental queued the stale-document deletions without waiting for them, then saved the tracker without those IDs. If a deletion task failed or never ran, its documents stayed in the index while the tracker forgot them.
• Full rebuilds (populate-search-engine without --incremental, then meilisearch-clean --swap) never updated the tracker.

Changes

• delete_documents_from_db waits for its deletion task and raises if it fails (like add_embeddings_to_db already does).
• _run_incremental always saves the tracker, but keeps the IDs whose deletion didn't succeed, so the next run retries them.
• meilisearch-clean --swap rebuilds the tracker from the main index after the swap. It now requires --hf_token (or HF_TOKEN) when swapping.

Incremental runs now also wait for their deletion tasks, which can take a while when the Meilisearch task queue is busy (they already wait for their additions).

Testing

Unit tests for the deletion wait, a failed deletion batch staying in the tracker, and the tracker rebuild after a swap (with and without --swap).

Existing stale documents

To clean up the 259 documents already in the index: run migrations/export_meili_ids_to_hf.py once to re-bootstrap the tracker from the index. The next incremental run then deletes every document that no current page produces.

This PR was generated with an AI coding agent.

🤖 Generated with Claude Code