I tried to make use of an ArrayPartition to prevent Arrow.jl arrays from being loaded into ram when concatenating tables.
The problem is that as soon as you want to actually materialise an array to do some operation the type constraints of copy are violated since a copied arrow array becomes a normal array (which I guess is due to the original array being a special type which is mmaped to disk).
I understand this is not the intended use case of this package, but I wonder if something like
function Base.copy(A::ArrayPartition{T}) where T
newx = copy.(A.x)
ArrayPartition{T, typeof(newx)}(newx)
endwould work instead of current implementation? This is still type stable (as long as copy.(A.x) is type stable) so it should not have any performance disadvantages.
I tried to make use of an
ArrayPartitionto prevent Arrow.jl arrays from being loaded into ram when concatenating tables.The problem is that as soon as you want to actually materialise an array to do some operation the type constraints of
copyare violated since a copied arrow array becomes a normal array (which I guess is due to the original array being a special type which is mmaped to disk).I understand this is not the intended use case of this package, but I wonder if something like
would work instead of current implementation? This is still type stable (as long as
copy.(A.x)is type stable) so it should not have any performance disadvantages.