Post-1.0: Define and handle files that exceed a configurable size threshold #134

Background

The ingestion pipeline currently reads every file unconditionally. Some media files (e.g. uncompressed RAW, long-form videos) can be orders of magnitude larger than a typical photo. These outliers affect hashing time, EXIF extraction memory, and proxy generation without a clear policy for what should happen to them.

Goal

• Decide on a default size threshold (e.g. Worker:MaxFileSizeBytes, default TBD).
• Define what the pipeline does when a file exceeds that threshold — options to evaluate:
  • Skip silently — log a warning and drop the file from the current crawl without recording it.
  • Skip and record — insert a MediaAsset with a skipped_oversized flag so the UI can surface it.
  • Process anyway, warn — proceed but emit a structured warning so operators can tune the threshold.
  • Configurable per media type — different limits for images vs. video.

Suggested approach (to be decided)

A FileSizeGuardMiddleware early in the pipeline (right after LoggingMiddleware, before hashing) that reads Worker:MaxFileSizeBytes from config and short-circuits via CanInvoke returning false when the file exceeds the limit. The OnCannotInvoke path (or a dedicated skip handler) records the outcome.

Open questions

• What is a sensible default threshold? (1 GB? 4 GB?)
• Should oversized files be surfaced in the UI or silently ignored?
• Does the threshold need to be per media type?

Notes

• Worker:MaxFileSizeBytes should be added to AppIniInitializer in HostApplicationBuilderExtensions with the chosen default.
• Related to the ContentHashingMiddleware streaming fix — large files are now hashed without loading fully into memory, but proxy generation (FFmpeg transcoding) may still be a concern for very large videos.