Apache Airflow graduation as Apache Top-Level
Today is a great day for Apache Airflow as it graduates from incubating status to a Top-Level Apache project. This is the next step of maturity for Airflow. For those unfamiliar, Airflow is an orchestration tool to schedule and orchestrate your data workflows. From ETL to training of models, or any other arbitrary tasks. Unlike other orchestrators, everything is written in Python, which makes it easy to use for both engineers and scientists. Having everything in code means that it is easy to version and maintain.
At the point of writing, Apache Airflow is in use at more than 200 organizations, including Adobe, Airbnb, Astronomer, Etsy, Google, ING, Lyft, Paypal, Qubole, Reddit, Reply, Square, Twitter, and United Airlines, and many more.
Within the cloud space, Airflow is gaining ground as well. Most notably, Airflow is being offered on the Google Cloud Platform as a managed service under the name of Cloud Composer. Furthermore, recent efforts by Amazon have resulted into seamless integration of Sagemaker within Airflow. Besides the cloud providers, also companies like Astronomer offer managed Airflow. Having this ecosystem of companies around Airflow make it a product that generalizes very well to many companies, and makes sure that there is a shared effort to continue development of Airflow.
For me personally, as a committer on the project, I find it truly amazing to see people from all over the world contributing code and documentation to Airflow to make it even better. I'm looking forward to what 2019 will bring to Airflow!
Interested in Apache Airflow Training?
A quick heads up that we also offer Apache Airflow Training as one of the public courses in our Academy. The next scheduled workshop is on February 4-5 2019 and you can register right here.
Can't make that one? Not to worry, you can check out our schedule to see the other times we offer this course throughout the year, or reach out below to inquire about booking in a private, in-company training session.
Follow us for more of this
Rhyme with AI
December 24, 2019
Schiphol Takeoff - Automate deployment without writing code!
December 09, 2019
Early Access of Apache Airflow book
October 30, 2019
IP whitelisting your Chalice application
October 26, 2019
GoDataDriven Open Source Contribution for Q3 2019
October 21, 2019
Azure container instance example
October 21, 2019