feat(39): new option batchScanSize

[pp0rtal]

Issue: #39

Description

On long MDBM migrations, we can encounter socket timeout. This happens when there are no index (linear scan): Mongodb cursor does not filter out any document for minutes, because docs to update are rare or at the end of a collection. We avoid to create indexes for YOLO migrations.
For those long scans, we'd prefer to fetch by small batch (of 1 million for instance)

We need this as a new option batchScanSize:

new MongoBulkDataMigration({
    query: // unchanged (might specify _id here /!\)
    options: {
        batchScanSize: 100_000, // Will scan by 100K docs  
        
        // Specify index in case the one chosen by Mongo can be optimized
        batchScanSize: "company_1__id_1',
    }
})

Note Camille did this on a migration script code directly here I'm reimplementing the logic to make it a native MBDM feature.

Before

After

flowchart TD A["find(query)<br/>single cursor"] --> B{"next document?"} B -- yes --> C["update doc / flush bulk"] C --> B B -- no --> D("Finalize last bulk") Loading

find() is restricted to a lower/upper bound When the cursor is over, we iterate flowchart TD A["<pre>lower = last upper<br>upper = $gte lower _id + skip(batchScanSize) doc"] A --> C["<pre>find(lower ≤ _id < upper)</pre>(cursor)"] C --> D{"next document?"} D -- yes --> E["update + backup<br/>flush bulk every maxBulkSize"] E --> D D -- no --> F{"upper reached end?"}:::high_blue F -- no --> A F -- yes --> H["flush remaining bulk · end"] classDef high_blue fill:#2567F4,color:#fff,stroke-width:1,stroke:#fff Loading

Changes

• Call new method iterateByIdRanges() when options.batchScanSize is specified
  • Support find/aggregate
  • Support queries containing _id already (merge in $and:[...])
• Add support for options.hint, necessary to specify index now Mongo won't use a perfect matching index

[orca-security-eu]

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0   0   0   0	View in Orca
Passed	SAST	0   0   0   0	View in Orca
Passed	Secrets	0   0   0   0	View in Orca
Passed	Vulnerabilities	0   0   0   0	View in Orca

[aurmartin]

@pp0rtal Nice! Thanks a lot for this improvement, we will probably use it very soon!

[aurmartin]

@pp0rtal This will silently override any _id passed in the query. Should we have a warning or even throw an error in that case? Either here or when validation the options (if there is some kind of validation?), the batchScanSize option is not supported if the query is done on _id.

It's properly handled later 👌

[antoine-roth]

@pp0rtal 🚀

[JulienAmoros]

Nice!

I spent some time understanding the yield*, but now I get it, very elegant solution 👌

[JulienAmoros]

[nit] Wouldn't it be better to have a clearer interface for this function?

Suggested change

	let document = await nextDocument();
	let document = await nextDocument(documents);