Datacenter Agility of Hadoop

An introduction to Apache Myriad on Mesos which provides the flexibility to increase or decrease Hadoop resources on the fly.

The current demands on the datacenter require a dynamic environment to succeed. The legacy approach of static clusters limits overall cluster utilization and does not allow for the growth demands of different versions of the analytic clusters. There is an increasing need to integrate big data and related Apache Hadoop applications with other datacenter services, ideally, by co-locating the data in HDFS/HBase with the services that need it.

In this talk, Ken (one of the committers on project) will show off Apache Myriad, which integrates Apache YARN with Apache Mesos. Mesos enables efficient resource sharing and isolation across a variety of distributed applications. This allows Apache Hadoop workloads to run alongside other datacenter applications, while sharing the same cluster, and hence breaking silos. This multi-tenancy strategy improves overall cluster utilization and operational efficiency.

About Ken Sipe

Ken is a distributed application engineer. Ken has worked with Fortune 500 companies to small startups in the roles of developer, designer, application architect and enterprise architect. Ken's current focus is on containers, container orchestration, high scale micro-service design and continuous delivery systems.

Ken is an international speaker on the subject of software engineering speaking at conferences such as JavaOne, JavaZone, Great Indian Developer Summit (GIDS), and The Strange Loop. He is a regular speaker with NFJS where he is best known for his architecture and security hacking talks. In 2009, Ken was honored by being awarded the JavaOne Rockstar Award at JavaOne in SF, California and the JavaZone Rockstar Award at JavaZone in Oslo, Norway as the top ranked speaker.

More About Ken »