Apache Airflow Workshop DataEngConf SF 2017

Contents:

• Setup your machine for the workshop
  • What you should have
  • Automated setup
  • Manual Setup
    • Get Jupyter and virtualenv
    • Virtualenvwrapper
    • Virtualenv
    • Installing Airflow
• Anatomy of a DAG
  • Example Pipeline definition
  • It’s a DAG definition file
  • Importing Modules
  • Default Arguments
  • Instantiate a DAG
  • Tasks
  • Templating with Jinja
  • PythonOperator
  • Separating Workflow definition and task definition
  • Setting up Dependencies
  • Recap
  • Testing
    • Running the Script
    • Command Line Metadata Validation
    • Testing
    • Backfill
  • What’s Next?

Indices and tables

• Index
• Search Page