appendRowGroup will loose pageIndex

[asfimport]

Currently, 
org.apache.parquet.hadoop.ParquetFileWriter#appendFile(org.apache.parquet.io.InputFile) uses appendRowGroup method to concate parquet row group. However, appendRowGroup method looses column index.

// code placeholder
  public void appendRowGroup(SeekableInputStream from, BlockMetaData rowGroup,
                             boolean dropColumns) throws IOException {
      ....
      // TODO: column/offset indexes are not copied
      // (it would require seeking to the end of the file for each row groups)
      currentColumnIndexes.add(null);
      currentOffsetIndexes.add(null);
  } 

 
https://github.com/apache/parquet-mr/blob/f8465a274b42e0a96996c76f3be0b50cf85ecf15/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L1033C19-L1033C19

 

Look forward to functionality that support append with page index.

 

Reporter: GANHONGNAN

_{Note: This issue was originally created as PARQUET-2340. Please see the migration documentation for further details.}

[asfimport]

GANHONGNAN:
@gszadovszky  Sorry to disturb. Do you have any plan to support it? Or any suggestion?

[asfimport]

Gabor Szadovszky / @gszadovszky:
[~NathanKan], I don't think these methods are used anymore. parquet-cli has another concept to merge files and that supports column indexes AFAIK. @wgtmac, could you confirm this? Maybe, we can close this jira?

[asfimport]

Gang Wu / @wgtmac:
[~NathanKan] You may be interested in the method ParquetFileWriter.appendColumnChunk. It is used internally by RewriteCommand to merge files without loosing their row group metadata.

[asfimport]

GANHONGNAN:
Thank you for your reply @ @gszadovszky @ @wgtmac. I am trying to merge row groups from different parquet files. So I use  org.apache.parquet.hadoop.ParquetFileWriter#appendFile(org.apache.parquet.io.InputFile) method which calls the deprecated method appendRowGroup. What do you think about implementing org.apache.parquet.hadoop.ParquetFileWriter#appendFile method using  ParquetFileWriter.appendColumnChunk?  @ @wgtmac   @ @gszadovszky

[asfimport]

Gang Wu / @wgtmac:
Do you have any special handling that ParquetRewriter cannot do? I'd suggest we stick to a single implementation.

Please check https://github.com/apache/parquet-mr/blob/f8465a274b42e0a96996c76f3be0b50cf85ecf15/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/rewrite/ParquetRewriter.java#L324

[asfimport]

GANHONGNAN:
@wgtmac  I will have a try with ParquetRewriter.java