Apache Airflow graduation as Apache Top-Level
Today is a great day for Apache Airflow as it graduates from incubating status to a Top-Level Apache project. This is the next step of maturity for Airflow. For those unfamiliar, Airflow is an orchestration tool to schedule and orchestrate your data workflows. From ETL to training of models, or any other arbitrary tasks. Unlike other orchestrators, everything is written in Python, which makes it easy to use for both engineers and scientists. Having everything in code means that it is easy to version and maintain.
At the point of writing, Apache Airflow is in use at more than 200 organizations, including Adobe, Airbnb, Astronomer, Etsy, Google, ING, Lyft, Paypal, Qubole, Reddit, Reply, Square, Twitter, and United Airlines, and many more.
Within the cloud space, Airflow is gaining ground as well. Most notably, Airflow is being offered on the Google Cloud Platform as a managed service under the name of Cloud Composer. Furthermore, recent efforts by Amazon have resulted into seamless integration of Sagemaker within Airflow. Besides the cloud providers, also companies like Astronomer offer managed Airflow. Having this ecosystem of companies around Airflow make it a product that generalizes very well to many companies, and makes sure that there is a shared effort to continue development of Airflow.
For me personally, as a committer on the project, I find it truly amazing to see people from all over the world contributing code and documentation to Airflow to make it even better. I'm looking forward to what 2019 will bring to Airflow!
Interested in Apache Airflow Training?
A quick heads up that we also offer Apache Airflow Training as one of the public courses in our Academy. The next scheduled workshop is on February 4-5 2019 and you can register right here.
Can't make that one? Not to worry, you can check out our schedule to see the other times we offer this course throughout the year, or reach out below to inquire about booking in a private, in-company training session.
Follow us for more of this
Highlights from the new Apache Avro 1.9.0 release
May 14, 2019
May 12, 2019
GoDataDriven Open Source Contribution for March and April 2019
May 03, 2019
Dutch Data Science Week 2019
April 25, 2019
Migrating your Hadoop workloads to the Cloud
April 22, 2019
GoDataDriven announces Data Council NL community
April 19, 2019