Google App Engine Vendoring Done Right

2 min read

This is my personal approach to using pure third-party python packages on Google App Engine.

Installing pure third-party python libs in GAE is very well documented.

With a little tweak you should even have zero issues working in a virtualenv, the modern and right way to work with python web applications. Virtualenvs create an isolated Python environment ensuring that application specific packages are isolated from system packages.

This approach(the one outlined in the docs) is quite easy and better. But occasionally I just want to be able to quickly test some functionality from my python console and be sure too that this will work in my GAE dev environment. Some prefer to use GAE's interactive console. I do too at times, but nothing beats cool terminal colors and stuff :D.

How I Vendor

One common path people follow is to install packages twice. One in their custom lib folder for the development server and another in their virtual environment . But I find it simpler to symlink the virtual environment's site packages to my third-party folder named lib. The concept is simple in that we essentially make our application's dependencies available in this folder and with a little vendoring awesomeness (or hell depending on whichever way you want to look at it) in the configuration of our GAE project, we're able to use these dependencies locally, in dev and in production. For what it's worth, I think it's awesome.

To vendor like I do follow these:

  1. Go into your project's root

  2. Create virtualenv with a desired name and activate it. I use env.

    $ virtualenv env
    $ source env/bin/activate
  3. Symlink the virtualenv's site packages to the lib folder

    $ ln -s env/lib/python2.7/site-packages lib

    After, you should have a top level structure resembling something like this:

    ├── app.yaml
    ├── env
    ├── lib -> env/lib/python2.7/site-packages
    └── requirements.txt
  4. pip install your packages. They get auto added into the lib folder as well For example to install Flask you can just use:

    $ pip install flask

    This works well with a requirement's file:

    $ pip install -r requirements.txt

    Even better this allows you to move about with your requirements file that I'm sure you've come to love and just can't live without.

  5. Edit or create to enable vendoring

    from google.appengine.ext import vendor
    # Add any libraries installed in the "lib" folder.
  6. In the skip files section of your app.yaml be sure to skip the upload of your virtual environment as shown on the last line in the following example

    - ^(.*/)?#.*#$
    - ^(.*/)?.*~$
    - ^(.*/)?.*\.py[co]$
    - ^(.*/)?.*/RCS/.*$
    - ^(.*/)?\..*$
    - ^env$ #virtual environment's folder

Once the packages are properly set up for App Engine to access, you can just fire up a python shell in your virtualenv to test stuff out. We can thus be pretty certain that any nifty stuff we try in the virtualenv will be the same thing we can put in our application's logic and expect the same results.

Tags: app engine | pip | python |