Skip to content

Restore compatibility with spark-avro 1.0.0 - #114

Closed
JoshRosen wants to merge 17 commits into
masterfrom
compatibility-with-spark-avro-1.0
Closed

JoshRosen wants to merge 17 commits into
masterfrom
compatibility-with-spark-avro-1.0

Conversation

@JoshRosen

Copy link
Copy Markdown
Contributor

This patch addresses an issue where spark-redshift would run into "Mandatory url is not present in manifest file" errors when saving data back to Redshift if a pre-2.0.0 version of spark-avro was used at runtime. This problem should not arise for most users of spark-redshift, since the proper version of spark-avro should automatically be pulled in via Maven or Ivy; the goal of this patch is to provide compatibility for users who cannot upgrade spark-avro for other compatibility-related reasons.

Cause of the write path bug: In order to fix a crash that could occur when writing tables containing empty partitions, #99 modified the write path to use manifest files that instruct Redshift to load only the non-empty partitions' Avro files. The code that handled processing of filenames when generating the manifest made assumptions about filenames that hold for spark-avro 2.0.0+ but not for spark-avro 1.0.0.

Solution: use a different method to build the correct list of part file names when constructing the manifest.

Fixes #111.

@JoshRosen JoshRosen added the bug label Oct 28, 2015
@JoshRosen JoshRosen added this to the 0.5.3 milestone Oct 28, 2015
@codecov-io

Copy link
Copy Markdown

Current coverage is 95.07%

Merging #114 into master will increase coverage by +0.02% as of 3303608

@@            master    #114   diff @@
======================================
  Files           12      12       
  Stmts          506     508     +2
  Branches       122     122       
  Methods          0       0       
======================================
+ Hit            481     483     +2
  Partial          0       0       
  Missed          25      25       

Review entire Coverage Diff as of 3303608

Powered by Codecov. Updated on successful CI builds.

Comment thread .travis.yml

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This configuration acts as a regression test for this issue.

@JoshRosen JoshRosen closed this in 1293621 Oct 31, 2015
@JoshRosen
JoshRosen deleted the compatibility-with-spark-avro-1.0 branch October 31, 2015 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"Invalid operation: Mandatory url is not present in manifest file" error when writing data to Redshift while using an old version of spark-avro

3 participants