Setup your machine for the workshop

What you should have

Your machine should have the following installed : * Python (>2.7.10 at least for Python 2 or >3.4 for Python 3). If you are on OSX, installing homebrew and the homebrew python is highly recommended as well. * SQLite (It should be installed on most systems) * Upgrading pip is recommended.

pip install --user --upgrade pip

Automated setup

A script is provided to install most dependencies in this. Simply run scripts/setup.sh located here from the root of this git repository.

Manual Setup

Get Jupyter and virtualenv

I would recommend installing jupyter and virtualenv for testing.

pip install -U --user virtualenv jupyter

Virtualenvwrapper

For ease of use I recommend virtualenvwrapper, but this is completely optional.

pip install --user virtualenvwrapper
export WORKON_HOME=~/.virtualenvs
mkdir -p $WORKON_HOME
source /usr/local/bin/virtualenvwrapper.sh
rmvirtualenv airflow_workshop
mkvirtualenv airflow_workshop

Virtualenv

rm -rf airflow_workshop
virtualenv airflow_workshop
source airflow_workshop/bin/activate

Installing Airflow

The easiest way to install the latest stable version of Airflow is with pip:

pip install airflow

The current stable version is 1.8.0. You can install this version specifically by using

pip install airflow==1.8.0

As Airflow is still currently evolving a decent amount, I recommend pinning the version you use, if you do not feel like living on the bleeding edge.

You can also install Airflow with support for extra features like s3 or mysql:

#Install the latest with extras
pip install airflow[s3,mysql]
#Install the 1.8.0 with specific extras
pip install airflow[s3,mysql]==1.8.0

Note that this will only install the Python packages needed to talk to some external systems, but not these systems (MySQL needs to be installed separately by your method of choice.)

The current list of extras is available here and an older version can be found in the docs