It presents very basic functionality and has a main limitation, which is the fact that it cannot execute tasks in parallel. For the sake of this tutorial, we will focus on the SequentialExecutor. There are many types of Executors in Apache Airflow, such as the SequentialExecutor, LocalExecutor, CeleryExecutor, DaskExecutor and others. The name is pretty self-explanatory: this component handles the coordination and execution of different tasks across multiple DAGs. # Run this from newly created directory to create the venv python3 -m venv venv # Activate your venv source venv/bin/activateīefore proceeding, it is important to discuss a bit about Airflow’s main component: the Executor. From your new directory, create and activate a new virtual environment for your Airflow project using venv.Create a new directory for your Airflow project (e.g.Linux/Mac OS - for those running Windows, activate and install Windows Subsystem for Linux (WSL), download Ubuntu 18 LTS from the Windows Marketplace and be happy :).In this introduction we will cover the easiest one, which is by installing it from the PyPi repository. There are multiple ways of installing Apache Airflow. Time to get our hands dirty and actually start with the tutorial. In fact, 2020 has seen individual contributions for Airflow at an all-time high. One might argue that Open Source projects always run the risk of dying at some point - but with a vibrant developer community we can say this risk is mitigated.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |