Issue with curl version

[bu-ra3y]

When using the 2.3.2 and 2.4.0 images I get this error when using Rasterio to open a TIF on S3:
CPLE_AppDefined in GDAL was built against curl 7.59.0, but is running against 7.51.0. Runtime failure is likely !

I do not hit that error when using your old 2.2.2 image.

Does this have to do with the curl version you specify here?

amazonlinux-gdal/Dockerfile

Line 31 in b2ca47b

CURL_VERSION=7.59.0 \

[vincentsarago]

@bu-ra3y 👋 thanks for opening the issue.
I also noticed the log message on some AWS Lambda process.

Can you paste the full code to reproduce the message ?

Thanks

[vincentsarago]

Alright I figured this out @bu-ra3y

So on AWS Lambda the LD_LIBRARY_PATH is set to be /lib64:/usr/lib64:$LAMBDA_RUNTIME_DIR:$LAMBDA_RUNTIME_DIR/lib:$LAMBDA_TASK_ROOT:$LAMBDA_TASK_ROOT/lib:/opt/lib and on default aws lambda image we can found libcurl in /usr/lib64

	libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007ff117fdc000)

The problem is then we ship our own curl lib with GDAL located in /usr/task/lib with is in the LD_LIBRARY_PATH default but way after /usr/lib64 so by default AWS will always ignore our own curl lib.

To fix this I think you need to set LD_LIBRARY_PATH=/var/task/lib/ in the lambda environment options.

cc @mojodna and @fredliporace

[vincentsarago]

To fix this I think you need to set LD_LIBRARY_PATH=/var/task/lib/ in the lambda environment options.

This will need to be tested but it seems to me that when setting the LD_LIBRARY_PATH gdal is linked to less libraries (ldd output). I'm not linux expert so I'd love to have other 👀 on this matter.

Default env

$ ldd /var/task/lib/libgdal.so
tlinux-vdso.so.1 =>  (0x00007fff17bfe000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f6f42fb4000)
libproj.so.13 => /var/task/lib/libproj.so.13 (0x00007f6f42d3d000)
libgeos_c.so.1 => /var/task/lib/libgeos_c.so.1 (0x00007f6f42b0c000)
libwebp.so.7 => /var/task/lib/libwebp.so.7 (0x00007f6f4289f000)
libopenjp2.so.7 => /var/task/lib/libopenjp2.so.7 (0x00007f6f42645000)
libjpeg.so.62 => /usr/lib64/libjpeg.so.62 (0x00007f6f423f2000)
libzstd.so.1 => /var/task/lib/libzstd.so.1 (0x00007f6f42173000)
libz.so.1 => /lib64/libz.so.1 (0x00007f6f41f5d000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6f41d41000)
librt.so.1 => /lib64/librt.so.1 (0x00007f6f41b39000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f6f41935000)
libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007f6f416be000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f6f413b9000)
libm.so.6 => /lib64/libm.so.6 (0x00007f6f410b7000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6f40ea1000)
libc.so.6 => /lib64/libc.so.6 (0x00007f6f40add000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6f4424c000)
libgeos-3.6.2.so => /var/task/lib/libgeos-3.6.2.so (0x00007f6f4072f000)
libidn2.so.0 => /usr/lib64/libidn2.so.0 (0x00007f6f4050e000)
libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x00007f6f402e6000)
libpsl.so.0 => /usr/lib64/libpsl.so.0 (0x00007f6f40071000)
libssl3.so => /usr/lib64/libssl3.so (0x00007f6f3fe1f000)
libsmime3.so => /usr/lib64/libsmime3.so (0x00007f6f3fbf8000)
libnss3.so => /usr/lib64/libnss3.so (0x00007f6f3f8d3000)
libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f6f3f6a4000)
libplds4.so => /lib64/libplds4.so (0x00007f6f3f4a0000)
libplc4.so => /lib64/libplc4.so (0x00007f6f3f29b000)
libnspr4.so => /lib64/libnspr4.so (0x00007f6f3f05d000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f6f3ee0f000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f6f3eb28000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f6f3e8f6000)
libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f6f3e6f3000)
liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f6f3e4e4000)
libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f6f3e292000)
libunistring.so.0 => /usr/lib64/libunistring.so.0 (0x00007f6f3df7c000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f6f3dd0e000)
libicuuc.so.50 => /usr/lib64/libicuuc.so.50 (0x00007f6f3d998000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f6f3d789000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f6f3d586000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f6f3d36c000)
libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x00007f6f3d151000)
libicudata.so.50 => /usr/lib64/libicudata.so.50 (0x00007f6f3bb7e000)
libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f6f3b95d000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f6f3b726000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007f6f3b524000)

with LD_LIBRARY_PATH: /var/task/lib

$ ldd /var/task/lib/libgdal.so
linux-vdso.so.1 =>  (0x00007ffe08bc7000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f1c1ce2f000)
libproj.so.13 => /var/task/lib/libproj.so.13 (0x00007f1c1cbb8000)
libgeos_c.so.1 => /var/task/lib/libgeos_c.so.1 (0x00007f1c1c987000)
libwebp.so.7 => /var/task/lib/libwebp.so.7 (0x00007f1c1c71a000)
libopenjp2.so.7 => /var/task/lib/libopenjp2.so.7 (0x00007f1c1c4c0000)
libjpeg.so.62 => /var/task/lib/libjpeg.so.62 (0x00007f1c1c22f000)
libzstd.so.1 => /var/task/lib/libzstd.so.1 (0x00007f1c1bfb0000)
libz.so.1 => /lib64/libz.so.1 (0x00007f1c1bd9a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1c1bb7e000)
librt.so.1 => /lib64/librt.so.1 (0x00007f1c1b976000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f1c1b772000)
libcurl.so.4 => /var/task/lib/libcurl.so.4 (0x00007f1c1b501000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f1c1b1fc000)
libm.so.6 => /lib64/libm.so.6 (0x00007f1c1aefa000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f1c1ace4000)
libc.so.6 => /lib64/libc.so.6 (0x00007f1c1a920000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1c1e0c7000)
libgeos-3.6.2.so => /var/task/lib/libgeos-3.6.2.so (0x00007f1c1a572000)
libnghttp2.so.14 => /var/task/lib/libnghttp2.so.14 (0x00007f1c1a34c000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f1c1a0de000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f1c19e90000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f1c19ba9000)
libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f1c199a6000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f1c19774000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f1c19565000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f1c19362000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f1c19148000)
libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f1c18f27000)

[vincentsarago]

😢 just tried and it's not working

libgdal.so: cannot open shared object file: No such file or directory

[vincentsarago]

🎉 after spending the day compiling GDAL I finally found the solution 😄

amazonlinux-gdal/Dockerfile

Lines 104 to 105 in 358ce32

	&& touch config.rpath \
	&& LDFLAGS="-Wl,-R,$PREFIX/lib" CFLAGS="-O2 -Wl,-S" CXXFLAGS="-O2 -Wl,-S" ./configure \

The idea is to fix shared libraries path using rpath. Sadly there is one major downside of doing so is that gdal won't be able to run if libgdal.so is somewhere else /var/task/lib. I tried to use the $origin variable but didn't succeed yet.

[bu-ra3y]

Way to go, Vincent. When do you need libgdal.so in another location?

On Wed, Jan 23, 2019, 10:54 PM Vincent Sarago ***@***.***> wrote: 🎉 after spending the day compiling GDAL I finally found the solution 😄 https://github.com/RemotePixel/amazonlinux-gdal/blob/358ce32609cb64d31c3fd87b8e254e3e67b93db5/Dockerfile#L104-L105 The idea is to *fix* shared libraries path using *rpath*. Sadly there is one major downside of doing so is that gdal won't be able to run if libgdal.so is somewhere else /var/task/lib. I tried to use the $origin variable but didn't succeed yet. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIVAziPBXS69p-758kW1KBHYcGiLj5JIks5vGTx3gaJpZM4aNoGC> .

-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Will Barley Schedule a call <https://calendly.com/will-barley> +1.650.575.5445

[vincentsarago]

Hi @bu-ra3y

When do you need libgdal.so in another location?

I never so I'll publish this new version like that and then I'll try to play around.

Note: This also fix a previous problem we had with libjpeg being in lib64/. We first assumed we could put the .so in /lib/ but it appeared to me that AmazonLinux image is also providing a libjpeg.so in /usr/lib64 so by default with the old install, gdal was also fetching the libjpeg from AWS and not ours.
cc @mojodna

[bu-ra3y]

@vincentsarago , I really appreciate you addressing this.

How do I take advantage of your fix?
Looking at Docker Hub, it looks like your fix may be contained in tag 2.4.0rc.
So I am guessing that either

• I need to switch to use Docker Image 2.4.0rc
• Or, you I am guessing you might roll your fix into Image 2.4.0 in which case I will pull it again

Is there anything I need to do to change my paths/environment variables or do I leave that the same?

[vincentsarago]

@bu-ra3y no worry
I pushed a 2.4.0rc yesterday but I don't think it solved the problem.

I'll push a new 2.4.0 in a few minutes with the fixes

[vincentsarago]

sorry @bu-ra3y I'm still checking everything is fine, I've deploy some lambda package using the new image and it seems the code run slower than when using libcurl provided by AWS Lambda

[bu-ra3y]

In case this helps you:
I noticed a slowdown as well, prior to your libcurl fix.
I have been using your 2.2.2 image for a long time.
Going to 2.3.2 was a significant slowdown, and in some cases more than double.
2.4.0 sped things back up but still slower than 2.2.2

The main thing I am doing is reading TIFs from S3 using rasterio.open('s3://....').

[vincentsarago]

🤔 I may create a gdal2.4.0-light image without curl but I'll test it before

[vincentsarago]

here are some result

using gdal2.4.0-light (without curl and jpegturbo) HTTP/1

$ cat list_1000.txt| python lambda_bench_landsat.py LC08_L1TP_018034_20181017_20181031_01_T1 - --stage light
Nb Good: 1000 / 1000
Mean: 333.85487900000004
Median: 313.97
p5: 144.89600000000002 | p95: 562.0905
min: 40.465 | max: 1643.37

using gdal2.4.0 (with curl and jpegturbo) and HTTP/2 enable

$ cat list_1000.txt| python lambda_bench_landsat.py LC08_L1TP_018034_20181017_20181031_01_T1 - --stage http2

Nb Good: 1000 / 1000
Mean: 394.33198
Median: 356.125
p5: 251.35299999999998 | p95: 647.904
min: 84.51 | max: 2284.39

using gdal2.4.0 (with curl and jpegturbo) and HTTP/1.1 enable

$ cat list_1000.txt| python lambda_bench_landsat.py LC08_L1TP_018034_20181017_20181031_01_T1 - --stage http1

Nb Good: 1000 / 1000
Mean: 464.00965
Median: 396.115
p5: 255.80200000000002 | p95: 751.1659999999999
min: 82.68 | max: 10010.18

I kinda feel both options could be good because shipping gdal with its own CURL should reduce the number of HTTP call (GET) which could be a $ saver when working with requester-pays bucket.

I'll go with the plan of pushing both image gdal2.4.0 and gdal2.4.0-light

I could also backup the old gdal2.2.0 if you need it @bu-ra3y