GoDataDriven Open Source Contribution: March 2017 Edition
Like announced last month, we are trying to collect all the contributions we do the the open source world, either to existing or to new projects.
- In Druid he updated the documentation with PR 3973, regarding ingesting parquet format into a Druid cluster;
- There he also fixed a faulty log line with PR 3970;
- In Druid Docker he fixed two issues in PR 33 and 34 by creating and setting correct directories and permissions;
- In Airflow PR 2042 he further extended the spark-submit operator/hook by adding YARN integration;
- In Flink PR 3280 he fixed the documentation by setting a correct reference.
- In the scalatra sample app, being Fokko a Scala aficionado, he got rid of
TypeParamSupport. The trait got deprecated because the functionality has been folded into the core. This resulted in PR 7 for the project.
On the other hand yours truly fixed in NiFi a wrong description of the UnpackContent processor in PR 1558 and open sourced a project to provision Google Cloud Engine instances to ease the classroom trainings deployment1. We are in fact often faced with a lot of challenges when delivering training where Spark is involved:
- If we use virtual machines (VM), the users can't never quite experience how powerful Spark is, as their machines are always so slow that it's not even funny. As an added chore, we need to create, maintain, and distribute several GBs around as these VMs are not small;
- If we use local mode, installing Spark in all configurations is incredibly cumbersome, especially if you want HDFS support; the slowness still applies, albeit in a less sever form;
- If you create a cluster, it's never nice to deploy it, install the packages, and make the keys available to everybody.
Since Google Cloud Engine makes it extremely easy to create clusters, the project kind of assumes that that's what you're using. That said, it should be easy enough to modify it. Personally I'm working on getting Anaconda + JupyterHub integrated so that users don't even need to have SSH access to the machine.
That's all for the second edition. As always, we're hiring Data Scientists and Data Engineers. Head up to our career page if you're interested. You get plenty of opportunities to give back to the community.
Huge props to Ron as he created the first working Ansible implementation! ↩
Follow us for more of this
Deep Learning Blimp
October 16, 2018
Big Data Expo 2018: Deep Learning, the Engine of the AI Revolution
October 05, 2018
How Blockchain Technology Will Make Platforms Obsolete
September 27, 2018
Wasting money with data science
September 21, 2018
How to Find Blockchain Use Cases: Part I
September 17, 2018
Opening up some training material
September 05, 2018