Add export history support#293
Open
bachuv wants to merge 3 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new :exporthistory module that enables durable, checkpointed export of terminal orchestration history to Azure Blob Storage, plus a sample and CI artifact reporting to support validation of the feature.
Changes:
- Introduces the
exporthistoryGradle subproject with entity/orchestrator/activity implementation for BATCH/CONTINUOUS export to Blob Storage. - Adds serializer + escaping utilities and golden parity tests to align exported history format with the .NET implementation (non-entity events).
- Adds a runnable sample and integration/unit tests, plus workflow artifact upload for integration test reports.
Reviewed changes
Copilot reviewed 56 out of 56 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| settings.gradle | Registers the new :exporthistory Gradle module. |
| samples/build.gradle | Adds a runHistoryExportSample task and depends on :exporthistory. |
| samples/src/main/java/io/durabletask/samples/HistoryExportSample.java | Demonstrates scheduling terminal instances and exporting their history to Blob Storage. |
| exporthistory/build.gradle | Defines the new module build (deps, test/integrationTest tasks, publishing/signing, SpotBugs). |
| exporthistory/README.md | Documents install/usage, modes, backend requirements, and format details. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/package-info.java | Package-level docs describing the export history feature components. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportHistoryClientExtensions.java | Provides client-side wiring helpers to create an ExportHistoryClient. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportHistoryWorkerExtensions.java | Registers export entities/orchestrators/activities on a worker builder. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportHistoryClient.java | Client wrapper to create/get/list export jobs backed by entity operations/reads. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportHistoryJobClient.java | Per-job client for create/describe/delete via an operation orchestrator. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportHistoryStorageOptions.java | Configures Blob destination auth + container/prefix/format settings. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJob.java | Durable entity holding job configuration/status/checkpoint/progress and starting the run. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobOrchestrator.java | Orchestrator that pages terminal instances, fans out exports, checkpoints, and continues-as-new. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExecuteExportJobOperationOrchestrator.java | Orchestrator wrapper so clients can await entity operation completion/errors. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ListTerminalInstancesActivity.java | Activity that calls DurableTaskClient.listInstanceIds to page terminal instances. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportInstanceHistoryActivity.java | Activity that validates terminal state, reads history, serializes it, and uploads to Blob. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/BlobExportWriter.java | Azure Blob upload implementation (gzip + content-type/encoding + metadata). |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/HistoryEventSerializer.java | Serializes com.microsoft.durabletask.history events to the export wire format. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/HtmlSafeJsonEscapes.java | Custom Jackson CharacterEscapes to match HTML-safe \\uXXXX escaping rules. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportBlobNaming.java | Deterministic blob naming based on completion timestamp + instanceId (SHA-256). |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobCreationOptions.java | Client-facing creation options and validation for export jobs. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobConfiguration.java | Persisted job configuration used by orchestrator and entity. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobDescription.java | Client-facing projection of entity state. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobTransitions.java | Centralizes valid job-status transition rules + operation name constants. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobStatus.java | Defines job lifecycle states (PENDING/ACTIVE/FAILED/COMPLETED). |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobState.java | Entity state model for job status/config/checkpoint/progress fields. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobRunRequest.java | Orchestrator input referencing the job entity and cycle counter. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobOperationRequest.java | Operation request payload used by the operation orchestrator wrapper. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobQuery.java | Filtering/paging options for listing jobs via entity queries. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobQueryResult.java | Page result wrapper for listing jobs. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportHistoryConstants.java | Defines orchestrator instance-id formatting conventions for export jobs. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobNotFoundException.java | Exception for missing job entity reads. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobInvalidTransitionException.java | Exception for invalid job state transitions. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportJobClientValidationException.java | Exception for client-side validation/operation failures. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ListTerminalInstancesRequest.java | Activity input model for paging terminal instances. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/InstancePage.java | Activity output model containing instance IDs + next checkpoint. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/CommitCheckpointRequest.java | Entity operation input for committing progress/checkpoints/failures. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportCheckpoint.java | Cursor model used to resume paging across batches. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportFilter.java | Completion-window + terminal-status filter configuration. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportDestination.java | Blob destination (container + prefix) persisted in job config. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportMode.java | Defines BATCH vs CONTINUOUS export modes. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportRequest.java | Per-instance export activity input model. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportResult.java | Per-instance export activity result model (success/blob or error). |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportFailure.java | Captures per-instance failure details for job failure reporting. |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportFormat.java | Defines export kind + schema version (value semantics). |
| exporthistory/src/main/java/com/microsoft/durabletask/exporthistory/ExportFormatKind.java | Enumerates JSON vs JSONL export shapes. |
| exporthistory/src/test/resources/golden/reference-history-events.jsonl | Golden reference output for serializer parity testing. |
| exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/HistoryEventSerializerTest.java | Unit tests for JSON/JSONL shape, null omission, and content-type/extension helpers. |
| exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/HistoryEventSerializerParityTest.java | Byte-for-byte parity tests against golden output and escaping behavior. |
| exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/ExportBlobNamingTest.java | Tests deterministic naming, prefix handling, and timestamp formatting. |
| exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/ExportJobTransitionsTest.java | Tests state transition validity rules. |
| exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/ExportJobOrchestratorTest.java | Regression tests ensuring control-flow exceptions are rethrown by orchestrator. |
| exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/ExportJobDescriptionTest.java | Tests projection correctness and ExportFormat semantics. |
| exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/ExportJobCreationOptionsTest.java | Tests defaults, fluent setters, bounds checks, and mode-specific validation. |
| exporthistory/src/test/java/com/microsoft/durabletask/exporthistory/ExportHistoryIntegrationTest.java | End-to-end integration test against emulator + Azurite to verify blobs written. |
| .github/workflows/build-validation.yml | Uploads exporthistory integration test report artifact for CI visibility. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue describing the changes in this PR
Adds the
:exporthistorymodule — durable, checkpointed export of terminal orchestration history to Azure Blob Storage. Stacked on #292.Architecture
ExportJob(entity) — single source of truth: config, status, checkpoint cursor, progress counters. Owns the state machine (PENDING → RUNNING → COMPLETED/FAILED); signals a run on create.ExportJobOrchestrator— pages terminal instances viaListTerminalInstancesActivity, fans outExportInstanceHistoryActivityper instance, commits checkpoints back to the entity, handlesBATCHvsCONTINUOUS, retries with bounded backoff, andcontinueAsNews periodically to keep history bounded.ExecuteExportJobOperationOrchestratorwraps entity operations.ExportInstanceHistoryActivity— reads instance metadata (terminal check + completion timestamp), streamsgetOrchestrationHistory, serializes, and uploads viaBlobExportWriter.ExportHistoryClient/ExportHistoryJobClient+useExportHistoryworker/client extensions.Serialization (byte-for-byte .NET parity)
HistoryEventSerializer+HtmlSafeJsonEscapesreproduce the .NET export wire format exactly (field order,eventType/isPlayed, PascalCase enums, HTML-safe\uXXXXescaping). Pinned byHistoryEventSerializerParityTestagainst golden captured fromMicrosoft.Azure.DurableTask.Core(reference-history-events.jsonl).TimerFired eventId:-1,HistoryStatedefaults0001-01-01/size:0,ContinueAsNewas anExecutionCompleted.eventTypediscriminator (non-parity by design).Testing
@Tag("integration")): end-to-endBATCHexport against the DTS emulator + Azurite — verifies job reachesCOMPLETEDand blobs are written. Wired intobuild-validation.yml.Pull request checklist
CHANGELOG.md