Testing and debugging Apache Airflow
One of the questions I get asked the most about Apache Airflow is how to shorten the development cycle of pushing code, deploying, and manually triggering a DAG for verification to something that is locally testable without running on a live system. In this blog post I provide several pointers to testing and debugging Apache Airflow on your local machine.
The Zen of Python and Apache Airflow
Apache Airflow is a Python framework for programmatically creating workflows in DAGs. This allows for concise and flexible scripts but can also be the downside of Airflow; since it's Python code there are infinite ways to define your pipelines. The Zen of Python is a list of 19 Python design principles and in this blog post I point out some of these principles on four Airflow examples.
AWS Machine Learning Competency Status for GoDataDriven
GoDataDriven Open Source Contribution for January 2019, the Apache Edition
Our social responsibility as a company
Keras: multi-label classification with ImageDataGenerator
Multi-label classification is a useful functionality of deep neural networks. I recently added this functionality into Keras' `ImageDataGenerator` in order to train on data that does not fit into memory. This blog post shows the functionality and runs over a complete example using the VOC2012 dataset.