Skip to content

Many test failures with System.IO.InvalidDataException : End of Central Directory record could not be found. #7662

New issue
New issue
Closed
#7663
Closed
Many test failures with System.IO.InvalidDataException : End of Central Directory record could not be found.#7662
#7663
Labels
Known Build ErrorUse this to report build issues in the .NET Helix tabblocking-clean-ciBlocking PR or rolling builds

Description

@tarekgh
tarekgh
opened on Jul 16, 2026

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1512421
Build error leg or test failing: Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.TensorFlowGettingSchemaMultipleTimes
Pull request: https://github.com/dotnet/machinelearning.git/pull/7661

System.IO.InvalidDataException : End of Central Directory record could not be found

   at System.IO.Compression.ZipArchive.ReadEndOfCentralDirectory()
   at System.IO.Compression.ZipArchive..ctor(Stream stream, ZipArchiveMode mode, Boolean leaveOpen, Encoding entryNameEncoding)
   at System.IO.Compression.ZipFile.Open(String archiveFileName, ZipArchiveMode mode, Encoding entryNameEncoding)
   at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName, String destinationDirectoryName, Encoding entryNameEncoding, Boolean overwriteFiles)
   at Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.UnZip(String gzArchiveName, String destFolder) in D:\a\_work\1\s\test\Microsoft.ML.TensorFlow.Tests\TensorflowTests.cs:line 2015
   at Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests.DownloadImageSet(String imagesDownloadFolder) in D:\a\_work\1\s\test\Microsoft.ML.TensorFlow.Tests\TensorflowTests.cs:line 1958
   at Microsoft.ML.TensorFlow.Scenarios.TensorFlowScenariosTests..ctor(ITestOutputHelper output) in D:\a\_work\1\s\test\Microsoft.ML.TensorFlow.Tests\TensorflowTests.cs:line 73
   at InvokeStub_TensorFlowScenariosTests..ctor(Object, Span`1)
   at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

DO NOT USE JSON BELOW IF THIS IS A BUILD BREAK otherwise build analysis will allow pull requests to merge that break the build worse. For a build break, do not use this issue form. Make a regular new issue.

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "System.IO.InvalidDataException : End of Central Directory record could not be found",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1512421
Error message validated: [System.IO.InvalidDataException : End of Central Directory record could not be found]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/16/2026 3:25:44 PM UTC

Report

Build Repository Test Pull Request
1517095 dotnet/machinelearning Microsoft.ML.AutoML.Test.AutoFitTests.AutoFitMultiClassification_Image_CV #7607
1517094 dotnet/machinelearning Microsoft.ML.AutoML.Test.AutoFeaturizerTests.AutoFeaturizer_image_test #7607
1517142 dotnet/machinelearning Microsoft.ML.AutoML.Test.AutoFeaturizerTests.AutoFeaturizer_image_test #7570
1513035 dotnet/machinelearning Microsoft.ML.AutoML.Test.AutoFitTests.AutoFitMultiClassification_Image_CV
1513034 dotnet/machinelearning Microsoft.ML.AutoML.Test.AutoFeaturizerTests.AutoFeaturizer_image_test
1512779 dotnet/machinelearning Microsoft.ML.AutoML.Test.AutoFitTests.AutoFitMultiClassification_Image_CV
1512778 dotnet/machinelearning Microsoft.ML.AutoML.Test.AutoFeaturizerTests.AutoFeaturizer_image_test
1512421 dotnet/machinelearning Microsoft.ML.AutoML.Test.AutoFeaturizerTests.AutoFeaturizer_image_test #7661

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 8 8

Activity

  1. tarekgh
    added
    blocking-clean-ciBlocking PR or rolling builds
    Known Build ErrorUse this to report build issues in the .NET Helix tab
    on Jul 16, 2026
  2. dotnet-policy-service
    added
    untriagedNew issue has not been triaged
    on Jul 16, 2026

  3. github-actions commented on Jul 16, 2026

    @github-actions
    github-actionsBot
    on Jul 16, 2026 – with GitHub Actions
    Contributor

    🔍 No similar issues found with high confidence.

    AI-powered similar issue detection

  4. ericstj commented on Jul 16, 2026

    @ericstj
    ericstj
    on Jul 16, 2026
    Member

    I think this is a download failure for the tests fetching a data set. https://aka.ms/mlnet-resources/datasets/flower_photos_tiny_set_for_unit_test.zip is failing. Looking at https://witty-sand-06796de1e.2.azurestaticapps.net/links?host=1&q=mlnet-resources I don't see a registration for this. I haven't been able to find the actual path but it should be on https://mlpublicassets.blob.core.windows.net.

  5. ericstj commented on Jul 16, 2026

    @ericstj
    ericstj
    on Jul 16, 2026
    Member

    Yep, confirmed those aka.ms links are gone. We need a directory listing of https://mlpublicassets.blob.core.windows.net/ to restore them.

  6. ericstj commented on Jul 16, 2026

    @ericstj
    ericstj
    on Jul 16, 2026
    Member

    ML.NET aka.ms/mlnet-resources audit

    Audited 2026-07-16 for dotnet/machinelearning issue #7662.

    Summary

    • 37 logical resources are referenced by the current repository.
    • 39 exact URL forms are generated because:
      • datasets/flower_photos_small_set.zip also appears with a trailing /.
      • Text/Sswe/sentiment.emd also appears as lowercase text/Sswe/sentiment.emd.
    • 24 logical resources still resolve to the expected asset.
    • 13 logical resources redirect to Bing HTML instead of an asset.
    • Eight of the broken aliases have clear matching blobs in the public assets container.
    • Five broken benchmark aliases have no matching blob anywhere in the 33-object public container. Their expected files are checked into test/data.

    Probe method: streamed GET with Range: bytes=0-63, following redirects. An alias was considered healthy only when its final response was non-HTML and its file signature/content type matched the expected asset.

    Broken aliases with recoverable blob targets

    Alias path under https://aka.ms/mlnet-resources/ Expected target Size Evidence
    datasets/flower_photos_small_set.zip https://mlpublicassets.blob.core.windows.net/assets/datasets/flower_photos_small_set.zip 11,709,383 ZIP signature PK; blob listing MD5 brq+jovspdD1r7EUYoyEVg==
    datasets/flower_photos_small_set.zip/ Same target as above; trailing-slash variant used by the performance benchmark 11,709,383 ZIP signature PK
    datasets/flower_photos_tiny_set_for_unit_test.zip https://mlpublicassets.blob.core.windows.net/assets/datasets/flower_photos_tiny_set_for_unit_tests.zip 2,324,722 Historical PR #4523 confirms singular alias to plural filename; ZIP signature PK; blob MD5 OBxMEK1IO9zlxzMZ5fZeYw==
    datasets/CatsVsDogs_tiny_for_unit_tests.zip https://mlpublicassets.blob.core.windows.net/assets/datasets/CatsVsDogs_tiny_for_unit_tests.zip 3,106,836 Historical PR #4523; ZIP signature PK; blob MD5 DLV9zSWlPVX18Cy9LtbRuQ==
    benchmarks/external/MSLRWeb10KTrain720kRows.tsv https://mlpublicassets.blob.core.windows.net/assets/benchmarks/MSLRWeb10KTrain720kRows.tsv 519,712,566 TSV content; blob MD5 L/9o53RKnfL94+E9OskucA==
    benchmarks/external/MSLRWeb10KValidate240kRows.tsv https://mlpublicassets.blob.core.windows.net/assets/benchmarks/MSLRWeb10KValidate240kRows.tsv 169,246,139 TSV content; blob MD5 Vc8t62qTYTOVNRU7sNqbNw==
    benchmarks/external/MSLRWeb10KTest240kRows.tsv https://mlpublicassets.blob.core.windows.net/assets/benchmarks/MSLRWeb10KTest240kRows.tsv 172,985,414 TSV content; blob MD5 L47StrmjXdvrWEEhd84tkA==
    benchmarks/external/digits.csv https://mlpublicassets.blob.core.windows.net/assets/benchmarks/digits.csv 264,712 CSV content; blob MD5 oDOcMKilMSobb55ccZ3M5Q==
    benchmarks/external/WikiDetoxAnnotated160kRows.tsv https://mlpublicassets.blob.core.windows.net/assets/benchmarks/wikiDetoxAnnotated160kRows.tsv 78,507,299 Alias uses Wiki...; blob uses wiki...; blob MD5 gq/IWOA/spmks/+h1um+Fg==

    Every alias above currently redirects to https://www.bing.com/?ref=aka&shorturl=... and returns HTML.

    Broken aliases without a matching public blob

    These files exist in test/data and are normally found locally before the fallback download is attempted. The public assets container contains no object with a matching basename.

    Alias path Existing asset identity Raw main candidate Raw size
    benchmarks/backcompat/MurmurHashV1.zip test/data/backcompat/MurmurHashV1.zip https://raw.githubusercontent.com/dotnet/machinelearning/main/test/data/backcompat/MurmurHashV1.zip 791
    benchmarks/adult.with-schema.txt test/data/adult.with-schema.txt https://raw.githubusercontent.com/dotnet/machinelearning/main/test/data/adult.with-schema.txt 3,656,858
    benchmarks/iris.txt test/data/iris.txt https://raw.githubusercontent.com/dotnet/machinelearning/main/test/data/iris.txt 2,757
    benchmarks/wikipedia-detox-250-line-data.tsv test/data/wikipedia-detox-250-line-data.tsv https://raw.githubusercontent.com/dotnet/machinelearning/main/test/data/wikipedia-detox-250-line-data.tsv 72,136
    benchmarks/breast-cancer.txt test/data/breast-cancer.txt https://raw.githubusercontent.com/dotnet/machinelearning/main/test/data/breast-cancer.txt 15,173

    For durable restoration, uploading immutable copies to mlpublicassets is safer than targeting the moving main branch.

    Healthy aliases

    Alias path Current final target Size
    image/AlexNetPrepOnnx/AlexNetPreprocess.onnx assets/image/AlexNetPrepOnnx/AlexNetPreprocess.onnx 618,820
    image/ResNetPrepOnnx/ResNetPreprocess.onnx assets/image/ResNetPrepOnnx/ResNetPreprocess.onnx 602,584
    image/AlexNetOnnx/AlexNet.onnx assets/image/AlexNetOnnx/AlexNet.onnx 233,133,636
    image/ResNet18Onnx/ResNet18.onnx assets/image/ResNet18Onnx/ResNet18.onnx 63,653,533
    image/ResNet50Onnx/ResNet50.onnx assets/image/ResNet50Onnx/ResNet50.onnx 94,288,003
    image/ResNet101Onnx/ResNet101.onnx assets/image/ResNet101Onnx/ResNet101.onnx 170,508,881
    WordVectors/glove.6B.50d.txt assets/wordvectors/glove.6B.50d.txt 171,350,079
    WordVectors/glove.6B.100d.txt assets/wordvectors/glove.6B.100d.txt 347,116,733
    WordVectors/glove.6B.200d.txt assets/wordvectors/glove.6B.200d.txt 693,432,828
    WordVectors/glove.6B.300d.txt assets/wordvectors/glove.6B.300d.txt 1,037,962,819
    WordVectors/glove.twitter.27B.25d.txt assets/wordvectors/glove.twitter.27B.25d.txt 257,699,726
    WordVectors/glove.twitter.27B.50d.txt assets/wordvectors/glove.twitter.27B.50d.txt 510,887,943
    WordVectors/glove.twitter.27B.100d.txt assets/wordvectors/glove.twitter.27B.100d.txt 1,021,669,379
    WordVectors/glove.twitter.27B.200d.txt assets/wordvectors/glove.twitter.27B.200d.txt 2,057,590,469
    WordVectors/wiki.en.vec assets/wordvectors/wiki.en.vec 6,597,238,061
    Text/Sswe/sentiment.emd assets/Text/Sswe/sentiment.emd 73,674,434
    text/Sswe/sentiment.emd Same target as above 73,674,434
    meta/resnet_v2_101_299.meta assets/meta/resnet_v2_101_299.meta 179,017,396
    meta/inception_v3.meta assets/meta/inception_v3.meta 95,664,402
    meta/mobilenet_v2.meta assets/meta/mobilenet_v2.meta 24,532,166
    meta/resnet_v2_50_299.meta assets/meta/resnet_v2_50_299.meta 102,616,931
    models/NasBert2000000.tsm assets/torchsharp/text/encoder_2000000.tsm 17,907,563
    models/pretrained_NasBert_14M_encoder.tsm assets/torchsharp/text/pretrained_NasBert_14M_encoder.tsm 54,767,471
    models/pretrained_Roberta_encoder.tsm assets/torchsharp/text/pretrained_Roberta_encoder.tsm 496,226,290
    models/autoformer_11m_torchsharp.bin assets/torchsharp/obj-detection/autoformer_11m_torchsharp.bin 77,547,537

    All healthy aliases return a 301 redirect to https://mlpublicassets.blob.core.windows.net/...; the target returned a valid partial-content response and the expected file signature.

    Repository source locations

    • Base URL and dynamic construction: src/Microsoft.ML.Core/Utilities/ResourceManagerUtils.cs:36,84,108
    • Image ONNX aliases: src/Microsoft.ML.DnnImageFeaturizer.ModelRedist/Microsoft.ML.DnnImageFeaturizer.ModelRedist.csproj:9-35
    • Flower dataset aliases:
      • docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/MulticlassClassification/ImageClassification/ImageClassificationDefault.cs:245-246
      • docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/MulticlassClassification/ImageClassification/ResnetV2101TransferLearningEarlyStopping.cs:233-234
      • docs/samples/Microsoft.ML.Samples/Dynamic/Trainers/MulticlassClassification/ImageClassification/ResnetV2101TransferLearningTrainTestSplit.cs:254-255
      • test/Microsoft.ML.AutoML.Tests/DatasetUtil.cs:187-188
      • test/Microsoft.ML.PerformanceTests/ImageClassificationBench.cs:141-142
      • test/Microsoft.ML.TensorFlow.Tests/TensorflowTests.cs:1953-1968
    • Word embeddings: src/Microsoft.ML.Transforms/Text/WordEmbeddingsExtractor.cs:605-634
    • TensorFlow meta-graphs: src/Microsoft.ML.Vision/ImageClassificationTrainer.cs:101-106,1312-1315
    • TorchSharp model aliases:
      • src/Microsoft.ML.TorchSharp/NasBert/TextClassificationTrainer.cs:115
      • src/Microsoft.ML.TorchSharp/NasBert/SentenceSimilarityTrainer.cs:115
      • src/Microsoft.ML.TorchSharp/NasBert/NerTrainer.cs:118
      • src/Microsoft.ML.TorchSharp/Roberta/QATrainer.cs:100
      • src/Microsoft.ML.TorchSharp/AutoFormerV2/ObjectDetectionTrainer.cs:119
    • Benchmark URL construction and call sites:
      • test/Microsoft.ML.PerformanceTests/BenchmarkBase.cs:32-56
      • test/Microsoft.ML.PerformanceTests/HashBench.cs:103
      • test/Microsoft.ML.PerformanceTests/KMeansAndLogisticRegressionBench.cs:16
      • test/Microsoft.ML.PerformanceTests/PredictionEngineBench.cs:36,75,110
      • test/Microsoft.ML.PerformanceTests/Numeric/Ranking.cs:25-26,75-77
      • test/Microsoft.ML.PerformanceTests/RffTransform.cs:22
      • test/Microsoft.ML.PerformanceTests/Text/MultiClassClassification.cs:23,100
      • test/Microsoft.ML.PerformanceTests/TextPredictionEngineCreation.cs:25
      • Dataset names: test/Microsoft.ML.TestFrameworkCommon/Datasets.cs:159-166,200-217

    Public container inventory

    Anonymous listing is enabled:

    https://mlpublicassets.blob.core.windows.net/assets?restype=container&comp=list

    The complete listing currently contains 33 blobs.

  7. ericstj
    closed this as completedin #7663on Jul 22, 2026
  8. dotnet-policy-service
    removed
    untriagedNew issue has not been triaged
    on Jul 22, 2026
  9. github-actions
    locked and limited conversation to collaborators on Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Known Build ErrorUse this to report build issues in the .NET Helix tabblocking-clean-ciBlocking PR or rolling builds

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions