geolibs

[tlpriest]

Anyone working on postgis geolibs?

[Miserlou]

Not that I know of, but that'd certainly a valuable addition for server-less GIS work!

[monkut]

I'm interested in using PostGIS with Postgres and Django.
I may try to get this going... So roughly I need to spin up amazon linux and install postgis? grab the resulting binaries and tar zip it up?

https://postgis.net/install/

[erindan]

I've been chatting with @monkut on the Slack channel and have had success setting up and running postgis extensions with Zappa. I'm happy to contribute a lambda-package in the next week or so, but may need a bit of guidance.

[erindan]

So, I've started looking into this and could use a bit of advice. It's my first attempt at providing a package and despite years in software development, am also quite new to making OSS contributions!

PostGIS extensions with Django have 3 separate dependencies: GEOS, GDAL and PROJ.4. I have all of them working in my own Zappa deployments by simply placing the *.so files in the root of my project (/var/task/ upon deployment) and setting GDAL_LIBRARY_PATH = '/var/task/libgdal.so'
GEOS_LIBRARY_PATH = '/var/task/libgeos_c.so.1' in the Django settings.py file. It is also necessary to include a GDAL data folder (it contains translations between coordinate systems) in the project root and to set GDAL_DATA = '/var/task/data/' in the AWS Environment Variables settings. When all the above is in place, Django knows how to use the dependencies if the project has specified 'ENGINE': 'django.contrib.gis.db.backends.postgis' and included the GIS apps 'django.contrib.gis' and 'rest_framework_gis'.

With all that in mind I have a few questions:

• Would it be better practice to create a single geodjango package with all the dependencies and the GDAL data folder or to create separate packages for each of GEOS, GDAL, and PROJ.4?
• Is there a way to automatically set an AWS ENV variable for GDAL_DATA when a deployment uses the package?
• Finally, I am unclear how Zappa deployments make use of lambda-packages. Are all packages slipstreamed into the deployment or is there something I've missed that determines which lambda-packages to include in any given deployment?

Thank you!

[taoru]

We implemented it in a bit different way: instead of creating a "fat" handler, we kept it "slim" (in Zappa terms) and added s3:// URL parsing capability to include handler. During cold startup, it downloads the libs and rewrites the paths.

[erindan]

@taoru I'd be interested in learning a bit more about how you did that. The GDAL and GEOS libs are pretty big - 70MB uncompressed. This could be a pain with my approach on a connection with slow upload speeds. I'm also a bit worried about adding stuff to lambda-packages that will bloat everyone's deployments per the last question in my previous post.

[taoru]

@erindan I'll try to find time this week for a better example or perhaps even a pull request, but the rough idea is to change LambdaHandler to check if library in the for library in included_libraries loop starts with s3:// and downloading it first, then use cdll.LoadLibrary to load it as if it were always there. Lambda to S3 speeds are very fast (even faster since a few days ago), so it takes just a few sec to load everything. Still, the developer needs to link the appropriate wheel or .so libs and upload them to S3 first.

My use case was similar to yours, our core team was in China, and every zappa update took 7-12 minutes at least, sometimes just hanging altogether halfway. Having postgis libs prebuilt in this repo would also be great, though you have more fine-grained control over features when building yourself (my lib archive ended up being just around 70MB).

Though I personally think it would be better to make the whole dependency optional directly in Django, and leave the heavy-lifting to Postgres. But it's considerably more work than building the libs. :)

[hammadzz]

Is the simplest current solution what @erindan is doing currently?

Is it good enough for small scale production use? If it isn't good enough maybe I will drop Zappa for now and come back to it in a couple of months.

[hammadzz]

@erindan any chance you can share more on how you generated the compatible files and the names of the ones you added to the root directory of the Zappa project. I assume you used an ec2 instance or docker image? Did you install the libraries there and copy them into your project?

[erindan]

@hammadzz Yes I used a Docker setup based on the instructions here: https://edgarroman.github.io/zappa-django-guide/setup/ and built them per the instructions in the Django docs https://docs.djangoproject.com/en/2.0/ref/contrib/gis/install/geolibs/

[erindan]

[erindan]

The above is what my resulting project structure roughly looks like after building the .so files and copying them into my project root (bcomes /var/task upon deployment).

[hammadzz]

@erindan any chance you have a script for it? As on the Amazon Linux AMI I can't manage to build GEOS. I will eventually figure it out. Also I have no idea where the libraries are being installed and where I would find the *.so files. I have been so used to using package managers to install these things.

EDIT: I will be back with build scripts