Following Problem: the data.mdb files are sparce with file size 1TB - this causes a lot of troubles when backuping data - rsync fails to recognise the sparce file type and reads 1TB to transfer, other file related tools show 1TB as real filesize - this is causing a lot of problems along the way - most software tools using lmdb files had to patch this problem later on. (see e.g. caffe BVLC/caffe#3731)
please consider to fix this by using lmdb.open with writemap=True this avoids the wrong super large 1TB filesize for this normally small files. or limit map_size to much less than 1TB (see https://www.programcreek.com/python/example/106501/lmdb.open example 6)
regards
oliver
Following Problem: the data.mdb files are sparce with file size 1TB - this causes a lot of troubles when backuping data - rsync fails to recognise the sparce file type and reads 1TB to transfer, other file related tools show 1TB as real filesize - this is causing a lot of problems along the way - most software tools using lmdb files had to patch this problem later on. (see e.g. caffe BVLC/caffe#3731)
please consider to fix this by using lmdb.open with writemap=True this avoids the wrong super large 1TB filesize for this normally small files. or limit map_size to much less than 1TB (see https://www.programcreek.com/python/example/106501/lmdb.open example 6)
regards
oliver