PARQUET-1300: [C++] Implement encrypted Parquet read and write support - #2555

Closed

thamht4190 wants to merge 125 commits into

apache:masterfrom

thamht4190:master

Commits on Jul 8, 2019

encryption (from apache/parquet-cpp github repo)
thamht4190
committed
79f8eb0
update thrift change and update encrypted footer
thamht4190
committed
138feeb
add encryption source files into CMakeLists.txt
thamht4190
committed
65fa020
add example from old PR of parquet-cpp
thamht4190
committed
859fe6b
change due to new update in crypto package
thamht4190
committed
60adabb
pass EncryptionProperties into parquet_encryption::Encrypt()/Decrypt() instead of pass it as many params
thamht4190
committed
4015657
fix issue of wrong column name in encryption-example and remove FileEncryptionProperties::uniform_encryption_
thamht4190
committed
700e71e
get column path from ColumnCryptoMetadata when column is encrypted with its own key
thamht4190
committed
73b936e
let encryption examples to be able to cover more cases
thamht4190
committed
015fd0a
footer plaintext mode
thamht4190
committed
c0cbac9
footer plaintext mode example
thamht4190
committed
2ba89f4
fix compiling issue
thamht4190
committed
331f94b
fix plaintext mode verification
thamht4190
committed
2f741e7
fix memory issue when serializing plaintext mode footer
thamht4190
committed
1fdd1e2
protection of sensitive metadata
thamht4190
committed
0bbe128
fix duplication of variable i
thamht4190
committed
23130e7
column metadata encryption: read algorithm, aad from FileCryptoMetadata or FileMetadata
thamht4190
committed
b221b96
keep redacted metadata version for old readers
thamht4190
committed
21b6dda
hidden column exception
thamht4190
committed
42e4b48
remove log
thamht4190
committed
593a454
add example for hidden column
thamht4190
committed
916b362
handle row group file_offset and total_compressed_size
thamht4190
committed
991cf35
Apply API changes

revit13
authored and
thamht4190
committed
f7b265f
Add AAD calculation

revit13
authored and
thamht4190
committed
c58af32
Fix parquet tests to work with the changes required to support AAD

revit13
authored and
thamht4190
committed
c8f1a8a
verify plaintext footer depends on config of decryption properties
thamht4190
committed
681209e
Fix code style

revit13
authored and
thamht4190
committed
0a2351d
Code style fixes in properties.h

revit13
authored and
thamht4190
committed
6fbfc52
revert change in parquet.thrift
thamht4190
committed
6187d40
Move all encrypted related classes from properties.h to new files: encryption_properties.h and encryption_properties.cc

Revital1 Eres
authored and
thamht4190
committed
c14ff0a
update crypto API change
thamht4190
committed
447a118
fix issue when column is encrypted in footer plaintext mode
thamht4190
committed
6d6a685
remove EncryptionProperties
thamht4190
committed
daec637
Change HiddenColumnExceptio message

Revital1 Eres
authored and
thamht4190
committed
1eb25e2
Fix indentation in encryption_properties.cc

Revital1 Eres
authored and
thamht4190
committed
3cc99fb
Rename functions in DecryptionKeyRetriever

Revital1 Eres
authored and
thamht4190
committed
0e29c54
Add check for aad_prefix to withoutAADPrefixStorage

Revital1 Eres
authored and
thamht4190
committed
cc73f2c
Add exception to FromThrift in thrift.h

Revital1 Eres
authored and
thamht4190
committed
cf6104f
Fix prefix aad calculation

Revital1 Eres
authored and
thamht4190
committed
24d304f
Remove fileAAD from ReaderProperties

Revital1 Eres
authored and
thamht4190
committed
eacef66
Remove column_map from ReaderProperties

Revital1 Eres
authored and
thamht4190
committed
4f3112c
Fix check for encryption and the existance of file_decryption in file_reader.cc

Revital1 Eres
authored and
thamht4190
committed
986331f
Save footer_key_metadata, algorithm, footer_decryptor and footer_signing_encryptor in InternalFileDecryptor and remove redundant AAD calculation in file_reader.cc

Revital1 Eres
authored and
thamht4190
committed
99c95b2
Rename file_decryption to file_decryption_properties in properties.h

Revital1 Eres
authored and
thamht4190
committed
7012745
Do not pass file_decryption as function parameter

Revital1 Eres
authored and
thamht4190
committed
bbec428
Rename is_plaintext_mode to is_encryption_algorithm_set

Revital Sur
authored and
thamht4190
committed
a73ebf1
fix function naming
thamht4190
committed
9b0dca7
fix const&
thamht4190
committed
250e1ed
make format
thamht4190
committed
a0385a5
Add plaintext_files_allowed

Revital Sur
authored and
thamht4190
committed
71a279b
Remove file_crypto_metadata_ field from SerializedRowGroup and SerializedFile classes

Revital Sur
authored and
thamht4190
committed
a844c10
Pass file_aad, algorithm and key_metadata to InternalFileDecryptor constructor

Revital Sur
authored and
thamht4190
committed
674866a
Fixes to previous commits

Revital Sur
authored and
thamht4190
committed
9826a6d
Put encryption_properties.h/cc content in encryption.h/cc

Revital Sur
authored and
thamht4190
committed
3ddcd8c
Remove encryption_properties.cc from CMakeLists.txt

Revital Sur
authored and
thamht4190
committed
5998317
Add column_metadata_map_, column_data_map_, footer_signing_encryptor_ and footer_encryptor_ to InternalFileEncryptor

Revital Sur
authored and
thamht4190
committed
b70b68f
Add column_data_map_, column_metadata_map_, footer_data_decryptor_ and footer_metadata_decryptor_ to InternalFileDecryptor

Revital Sur
authored and
thamht4190
committed
551649f
Rename aad to update_aad in Encryptor and Decryptor classes

Revital Sur
authored and
thamht4190
committed
9b5e5be
Move PARQUET_EMAGIC and PARQUET_MAGIC to file_writer.h and use it in file_reader.cc

Revital Sur
authored and
thamht4190
committed
60ef741
Rename file_encryption to file_encryption_properties in file_writer.cc

Revital Sur
authored and
thamht4190
committed
e5c2b48
Remove unused footer_decryptor_ from InternalFileDecryptor class and fix indentation in GetFooterSigningEncryptor

Revital Sur
authored and
thamht4190
committed
03d2495
Fix format

Revital1 Eres
authored and
thamht4190
committed
3537789
Change implementation of NULL_STRING

Revital1 Eres
authored and
thamht4190
committed
9e18a00
Change ParquetException message format in file_reader.cc

Revital1 Eres
authored and
thamht4190
committed
752e33c
Make format

Revital1 Eres
authored and
thamht4190
committed
a24f1d4
Add comments to encryption-reader-writer.cc example

Revital1 Eres
authored and
thamht4190
committed
1947381
Rename enable_plaintext_footer to set_plaintext_footer

Revital1 Eres
authored and
thamht4190
committed
30c886a
Rename aad variable in NextPage function

Revital1 Eres
authored and
thamht4190
committed
8f911c1
Change comment in GetColumnPageReader

Revital1 Eres
authored and
thamht4190
committed
18b598f
Change additional comments in GetColumnPageReader

Revital1 Eres
authored and
thamht4190
committed
8be2efe
Add comments in file_writer.cc

Revital1 Eres
authored and
thamht4190
committed
131cee6
Create both data and metadata decryptors to avoid redundant retrieval of key

Revital1 Eres
authored and
thamht4190
committed
cb552f6
Fix metadata parameter sent to parquet_encryption::AesDecryptor

Revital1 Eres
authored and
thamht4190
committed
8cc235f
Rename aad in GetFooterEncryptor and GetFooterSigningEncryptor

Revital1 Eres
authored and
thamht4190
committed
9662384
Rename verify to verify_signature

Revital1 Eres
authored and
thamht4190
committed
7bec5b3
Add comments to void WriteTo

Revital1 Eres
authored and
thamht4190
committed
4c1cfb3
Add additional comment in void WriteTo

Revital1 Eres
authored and
thamht4190
committed
1441ca0
Rename file_encryption to file_encryption_properties in WriterProperties

Revital1 Eres
authored and
thamht4190
committed
b219b8d
Use encrypted_footer instead of footer_signing_key when checking for encrypted footer

Revital1 Eres
authored and
thamht4190
committed
88687ad
Rename column_encryption_props to column_encryption_properties

Revital1 Eres
authored and
thamht4190
committed
65ab864
Add comments in thrift.h

Revital1 Eres
authored and
thamht4190
committed
49ff22b
Change parameters order in ColumnChunkMetaData::Make

Revital1 Eres
authored and
thamht4190
committed
aef0d7d
Change parameters order in PageReader::Open

Revital1 Eres
authored and
thamht4190
committed
b4debc4
Remove footer_encryption_key and footer_signing_key

Revital1 Eres
authored and
thamht4190
committed
11d6965
Remove ParquetException in GetFooterSigningEncryptor and GetFooterEncryptor

Revital1 Eres
authored and
thamht4190
committed
b4efaac
make format

Revital1 Eres
authored and
thamht4190
committed
9fd10b1
make format in thrift.h

Revital1 Eres
authored and
thamht4190
committed
068743c
fix rebase mistake in parquet.thrift
thamht4190
committed
cb11040
Fix aad settings in thrift.h

Revital1 Eres
authored and
thamht4190
committed
8ba072c
Port key erasure mechanism

Revital1 Eres
authored and
thamht4190
committed
04507d7
Fix columnMetaData

Revital1 Eres
authored and
thamht4190
committed
7636dd4
Minor fixes to previous code

Revital1 Eres
authored and
thamht4190
committed
ca2d8a6
fix build issue on MacOS
thamht4190
committed
56204ab
apply change from crypto package
thamht4190
committed
595716f
format code
thamht4190
committed
7336685
post-rebase change
thamht4190
committed
22a5c1e
add unit tests for encryption properties
thamht4190
committed
5697255
write unit tests for metadata
thamht4190
committed
b7bdc8a
Add encryption samples

Revital1 Eres
authored and
thamht4190
committed
fb826f5
fix lint and format issue
thamht4190
committed
d1aa339
fix metadata set, statistics set issues
thamht4190
committed
44f37b0
Various changes to encryption-reader-writer-all-crypto-options test and to AAD handling in file_reader.cc

Revital1 Eres
authored and
thamht4190
committed
72ca77f
Fix logging error

Revital1 Eres
authored and
thamht4190
committed
899a603
post-rebase change
thamht4190
committed
b59a3aa
fix isset of column chunk metadata and statistics
thamht4190
committed
afbb8bf
temporarily remove encryption-metadata-test
thamht4190
committed
1661ebb
fix windows compiling issue
thamht4190
committed
d604d10
fix issue of parquet-encryption-example
thamht4190
committed
00fbcf6
rename encryption-test.cc to encryption-properties-test.cc
thamht4190
committed
905067d
use isset instead of creating a copy of column chunk metadata
thamht4190
committed
479da8f
Address review comments

Revital1 Eres
authored and
thamht4190
committed
7052838
Fix SerializedPageReader initialization

Revital1 Eres
authored and
thamht4190
committed
b7f3ea3
Fix Format

Revital1 Eres
authored and
thamht4190
committed
80710e7
let parquet encryption be able to be off (when openssl is not found)
thamht4190
committed
b3e277e
Fix LogicalType

Revital1 Eres
authored and
thamht4190
committed
977ef11
keep encryption parameters at method declaration
thamht4190
committed
fb8ca61
add PARQUET_EXPORT into Builder class of encryption properties
thamht4190
committed
d7e024c
Change assert to ASSERT_EQ in encryption-configurations-test.cc

Revital1 Eres
authored and
thamht4190
committed
0c869dd
fix cmake format
thamht4190
committed
56b0455
Add MemoryPool field to Decryptors/Encryptors

Revital1 Eres
authored and
thamht4190
committed
d09bf9d
keep encryption parameters at method declaration (column_writer.cc/.h)
thamht4190
committed
4afcccb
Write to parquet stream to file in encryption test

Revital1 Eres
authored and
thamht4190
committed
afc7131
Add file reader and file writer Close to encryption-configurations-test.cc

Revital1 Eres
authored and
thamht4190
committed
c744cd2
Change encryption-configuration-test

Revital1 Eres
authored and
thamht4190
committed
76c2b02
Delete encryption-configuration-encrypted-columns-plaintext-footer.cc test

Revital1 Eres
authored and
thamht4190
committed
9011411

Uh oh!

Commits

Commits on Jul 8, 2019