Airflow

Airflow is a platform to programmatically author, schedule and monitor workflows

Important links

• AWS control panel: to login to AWS and access AP tools including the Airflow dev and prod UI
• Airflow dev UI: for running and monitoring development and training workflows on the Airflow UI (you will need to login to AWS first)
• Airflow prod UI: for running and monitoring production workflows on the Airflow UI (you will need to login to AWS first)
• Airflow repo: Github repo to store Airflow DAGs and roles
• Airflow template for Python: Github template repository for creating a Python image to run an Airflow pipeline
• Airflow template for R: Github template repository for creating an R image to run an Airflow pipeline
• Support: contact the Data Engineering team on #ask-data-engineering

To find out more

• Airflow pipeline concepts: What is Airflow and why you should use it
• Airflow pipeline instructions: Step by step guide for creating an example Airflow pipeline and related resources
• Troubleshooting Airflow pipelines: Common pitfalls

This page was last reviewed on 7 May 2022. It needs to be reviewed again on 7 May 2023 by the page owner #ask-data-engineering .

This page was set to be reviewed before 7 May 2023 by the page owner #ask-data-engineering. This might mean the content is out of date.