Skip to content

Add export history support#293

Open
bachuv wants to merge 3 commits into
vabachu/client-list-stream-historyfrom
vabachu/exporthistory-module
Open

Add export history support#293
bachuv wants to merge 3 commits into
vabachu/client-list-stream-historyfrom
vabachu/exporthistory-module

Conversation

@bachuv

@bachuv bachuv commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Issue describing the changes in this PR

Adds the :exporthistory module — durable, checkpointed export of terminal orchestration history to Azure Blob Storage. Stacked on #292.

Architecture

  • ExportJob (entity) — single source of truth: config, status, checkpoint cursor, progress counters. Owns the state machine (PENDING → RUNNING → COMPLETED/FAILED); signals a run on create.
  • ExportJobOrchestrator — pages terminal instances via ListTerminalInstancesActivity, fans out ExportInstanceHistoryActivity per instance, commits checkpoints back to the entity, handles BATCH vs CONTINUOUS, retries with bounded backoff, and continueAsNews periodically to keep history bounded. ExecuteExportJobOperationOrchestrator wraps entity operations.
  • ExportInstanceHistoryActivity — reads instance metadata (terminal check + completion timestamp), streams getOrchestrationHistory, serializes, and uploads via BlobExportWriter.
  • ClientExportHistoryClient / ExportHistoryJobClient + useExportHistory worker/client extensions.
  • API note: useExportHistory(...) takes an explicit DurableTaskClient (Java has no DI, unlike .NET's UseExportHistory).

Serialization (byte-for-byte .NET parity)

  • HistoryEventSerializer + HtmlSafeJsonEscapes reproduce the .NET export wire format exactly (field order, eventType/isPlayed, PascalCase enums, HTML-safe \uXXXX escaping). Pinned by HistoryEventSerializerParityTest against golden captured from Microsoft.Azure.DurableTask.Core (reference-history-events.jsonl).
  • A few outputs intentionally mirror .NET parity — e.g. TimerFired eventId:-1, HistoryState defaults 0001-01-01/size:0, ContinueAsNew as an ExecutionCompleted.
  • Entity events have no .NET wire equivalent, so they use a Java-native shape with an injected eventType discriminator (non-parity by design).

Testing

  • Unit: serializer parity (byte-for-byte), escaping, transitions, creation-option validation, blob naming.
  • Integration (@Tag("integration")): end-to-end BATCH export against the DTS emulator + Azurite — verifies job reaches COMPLETED and blobs are written. Wired into build-validation.yml.
  • SpotBugs (main + test) clean.

Pull request checklist

  • My changes do not require documentation changes
    • Otherwise: Documentation issue linked to PR
  • My changes are added to the CHANGELOG.md
  • I have added all required tests (Unit tests, E2E tests)

@bachuv bachuv requested a review from a team as a code owner July 1, 2026 23:46
@bachuv bachuv requested a review from Copilot July 1, 2026 23:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new :exporthistory module that enables durable, checkpointed export of terminal orchestration history to Azure Blob Storage, plus a sample and CI artifact reporting to support validation of the feature.

Changes:

  • Introduces the exporthistory Gradle subproject with entity/orchestrator/activity implementation for BATCH/CONTINUOUS export to Blob Storage.
  • Adds serializer + escaping utilities and golden parity tests to align exported history format with the .NET implementation (non-entity events).
  • Adds a runnable sample and integration/unit tests, plus workflow artifact upload for integration test reports.

Reviewed changes

Copilot reviewed 56 out of 56 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
settings.gradle Registers the new :exporthistory Gradle module.
samples/build.gradle Adds a runHistoryExportSample task and depends on :exporthistory.
samples/src/main/java/io/durabletask/samples/HistoryExportSample.java Demonstrates scheduling terminal instances and exporting their history to Blob Storage.
exporthistory/build.gradle Defines the new module build (deps, test/integrationTest tasks, publishing/signing, SpotBugs).
exporthistory/README.md Documents install/usage, modes, backend requirements, and format details.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/package-info.java Package-level docs describing the export history feature components.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportHistoryClientExtensions.java Provides client-side wiring helpers to create an ExportHistoryClient.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportHistoryWorkerExtensions.java Registers export entities/orchestrators/activities on a worker builder.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportHistoryClient.java Client wrapper to create/get/list export jobs backed by entity operations/reads.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportHistoryJobClient.java Per-job client for create/describe/delete via an operation orchestrator.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportHistoryStorageOptions.java Configures Blob destination auth + container/prefix/format settings.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJob.java Durable entity holding job configuration/status/checkpoint/progress and starting the run.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobOrchestrator.java Orchestrator that pages terminal instances, fans out exports, checkpoints, and continues-as-new.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExecuteExportJobOperationOrchestrator.java Orchestrator wrapper so clients can await entity operation completion/errors.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ListTerminalInstancesActivity.java Activity that calls DurableTaskClient.listInstanceIds to page terminal instances.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportInstanceHistoryActivity.java Activity that validates terminal state, reads history, serializes it, and uploads to Blob.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/BlobExportWriter.java Azure Blob upload implementation (gzip + content-type/encoding + metadata).
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/HistoryEventSerializer.java Serializes com.microsoft.durabletask.history events to the export wire format.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/HtmlSafeJsonEscapes.java Custom Jackson CharacterEscapes to match HTML-safe \\uXXXX escaping rules.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportBlobNaming.java Deterministic blob naming based on completion timestamp + instanceId (SHA-256).
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobCreationOptions.java Client-facing creation options and validation for export jobs.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobConfiguration.java Persisted job configuration used by orchestrator and entity.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobDescription.java Client-facing projection of entity state.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobTransitions.java Centralizes valid job-status transition rules + operation name constants.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobStatus.java Defines job lifecycle states (PENDING/ACTIVE/FAILED/COMPLETED).
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobState.java Entity state model for job status/config/checkpoint/progress fields.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobRunRequest.java Orchestrator input referencing the job entity and cycle counter.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobOperationRequest.java Operation request payload used by the operation orchestrator wrapper.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobQuery.java Filtering/paging options for listing jobs via entity queries.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobQueryResult.java Page result wrapper for listing jobs.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportHistoryConstants.java Defines orchestrator instance-id formatting conventions for export jobs.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobNotFoundException.java Exception for missing job entity reads.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobInvalidTransitionException.java Exception for invalid job state transitions.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobClientValidationException.java Exception for client-side validation/operation failures.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ListTerminalInstancesRequest.java Activity input model for paging terminal instances.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/InstancePage.java Activity output model containing instance IDs + next checkpoint.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/CommitCheckpointRequest.java Entity operation input for committing progress/checkpoints/failures.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportCheckpoint.java Cursor model used to resume paging across batches.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportFilter.java Completion-window + terminal-status filter configuration.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportDestination.java Blob destination (container + prefix) persisted in job config.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportMode.java Defines BATCH vs CONTINUOUS export modes.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportRequest.java Per-instance export activity input model.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportResult.java Per-instance export activity result model (success/blob or error).
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportFailure.java Captures per-instance failure details for job failure reporting.
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportFormat.java Defines export kind + schema version (value semantics).
exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportFormatKind.java Enumerates JSON vs JSONL export shapes.
exporthistory/src/test/resources/golden/reference-history-events.jsonl Golden reference output for serializer parity testing.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/HistoryEventSerializerTest.java Unit tests for JSON/JSONL shape, null omission, and content-type/extension helpers.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/HistoryEventSerializerParityTest.java Byte-for-byte parity tests against golden output and escaping behavior.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/ExportBlobNamingTest.java Tests deterministic naming, prefix handling, and timestamp formatting.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/ExportJobTransitionsTest.java Tests state transition validity rules.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/ExportJobOrchestratorTest.java Regression tests ensuring control-flow exceptions are rethrown by orchestrator.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/ExportJobDescriptionTest.java Tests projection correctness and ExportFormat semantics.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/ExportJobCreationOptionsTest.java Tests defaults, fluent setters, bounds checks, and mode-specific validation.
exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/ExportHistoryIntegrationTest.java End-to-end integration test against emulator + Azurite to verify blobs written.
.github/workflows/build-validation.yml Uploads exporthistory integration test report artifact for CI visibility.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants