GoDataDriven Open Source Contribution: March 2017 Edition
Like announced last month, we are trying to collect all the contributions we do the the open source world, either to existing or to new projects.
- In Druid he updated the documentation with PR 3973, regarding ingesting parquet format into a Druid cluster;
- There he also fixed a faulty log line with PR 3970;
- In Druid Docker he fixed two issues in PR 33 and 34 by creating and setting correct directories and permissions;
- In Airflow PR 2042 he further extended the spark-submit operator/hook by adding YARN integration;
- In Flink PR 3280 he fixed the documentation by setting a correct reference.
- In the scalatra sample app, being Fokko a Scala aficionado, he got rid of
TypeParamSupport. The trait got deprecated because the functionality has been folded into the core. This resulted in PR 7 for the project.
On the other hand yours truly fixed in NiFi a wrong description of the UnpackContent processor in PR 1558 and open sourced a project to provision Google Cloud Engine instances to ease the classroom trainings deployment1. We are in fact often faced with a lot of challenges when delivering training where Spark is involved:
- If we use virtual machines (VM), the users can't never quite experience how powerful Spark is, as their machines are always so slow that it's not even funny. As an added chore, we need to create, maintain, and distribute several GBs around as these VMs are not small;
- If we use local mode, installing Spark in all configurations is incredibly cumbersome, especially if you want HDFS support; the slowness still applies, albeit in a less sever form;
- If you create a cluster, it's never nice to deploy it, install the packages, and make the keys available to everybody.
Since Google Cloud Engine makes it extremely easy to create clusters, the project kind of assumes that that's what you're using. That said, it should be easy enough to modify it. Personally I'm working on getting Anaconda + JupyterHub integrated so that users don't even need to have SSH access to the machine.
That's all for the second edition. As always, we're hiring Data Scientists and Data Engineers. Head up to our career page if you're interested. You get plenty of opportunities to give back to the community.
Huge props to Ron as he created the first working Ansible implementation! ↩
Follow us for more of this
August 15, 2018
Write less terrible code with Jupyter Notebook
August 05, 2018
GoDataDriven open source contribution: July 2018 edition
August 01, 2018
Dynniq presentation video at AI Expo Europe 2018
July 31, 2018
Working with multiple partition formats within a Hive table with Spark
July 31, 2018
Handling encoding issues with Unicode normalisation in Python
July 28, 2018