In this webinar you will learn about:

• The people resources that are required to manage Hadoop
• Different ways to cross train employees or hire for the positions you need
• The different Hadoop management roles and how each functions
• Available resources that overcome hiring and training challenges

Time: 11am – 12pm PDT

Now in its second year in New York, the O’Reilly Strata Conference explores the change brought to technology and business by big data, data science, and pervasive computing. Joining forces this year with Hadoop World, Strata is at the heart of the big data industry.

Strata Conference brings together decision makers using the power of big data to drive business strategy and practitioners who collect, analyze, and manipulate the data — particularly in the worlds of finance, media, and government. The merger of Strata and Hadoop World will be the largest gathering of the Apache Hadoop community, with emphasis on hands-on and business sessions on the Hadoop ecosystem.

Strata Conference and Hadoop World is happening October 23-25 at the Hilton New York.

Co-presented by:
O’Reilly Media and Cloudera

Location: New York, New York

The two-day event will feature many of the leading thought leaders from the Apache Hadoop community who will showcase successful Hadoop use cases, share development and administration tips and tricks and educate organizations about how to leverage Apache Hadoop as a key component of their enterprise data platform. Hadoop Summit will also be an excellent network environment for developers, architects, administrators, data analysts, data scientists and vendors interested in advancing, extending or implementing Apache Hadoop.

Bayesian Counters

Alex Kozlov, Solutions Architect, Cloudera

Processing of large data requires new approaches to data mining: low, close to linear, complexity and stream processing. While in the traditional data mining the practitioner is usually presented with a static dataset, which might have just a timestamp attached to it, to infer a model for predicting future/takeout observations, in stream processing the problem is often posed as extracting as much information as possible on the current data to convert them to an actionable model within a limited time window. In this talk I present an approach based on HBase counters for mining over streams of data, which allows for massively distributed processing and data mining. I will consider overall design goals as well as HBase schema design dilemmas to speed up knowledge extraction process. I will also demo efficient implementations of Naive Bayes, Nearest Neighbor and Bayesian Learning on top of Bayesian Counters.

BranchReduce: Distributed Branch-and-Bound on YARN

Josh Wills, Director of Data Science, Cloudera

Branch-and-bound is a widely used technique for efficiently searching for solutions to combinatorial optimization problems. In this session, we will introduce BranchReduce, an open-source Java library for performing distributed branch-and-bound on a Hadoop cluster under YARN. Applications only need to write code that is specific to their optimization problem (namely the branching rule, the lower bound computation, and the upper bound computation), and BranchReduce handles deploying the application to the cluster, managing the execution, and periodically rebalancing the search space across the machines. We will give an overview of how BranchReduce works and then walk through an example that solves a scheduling problem with a near-linear speedup over a single machine implementation.

Improving HBase Availability and Repair

Joanthan Hsieh, Software Engineer, Cloudera
Jeff Bean, Training Program Lead, Cloudera

Apache HBase is a rapidly-evolving random-access distributed data store built on top of Apache Hadoop’s HDFS and Apache ZooKeeper. Drawing from real-world support experiences, this talk provides administrators insight into improving HBase’s availability and recovering from situations where HBase is not available. We share tips on the common root causes of unavailability, explain how to diagnose them, and prescribe measures for ensuring maximum availability of an HBase cluster. We discuss new features that improve recovery time such as distributed log splitting as well as supportability improvements. We will also describe utilities including new failure recovery tools that we have developed and contributed that can be used to diagnose and repair rare corruption problems on live HBase systems.

Optimizing MapReduce Job Performance

Todd Lipcon, Software Engineer, Cloudera

Optimizing MapReduce job performance is often seen as something of a black art. In order to maximize performance, developers need to understand the inner workings of the MapReduce execution framework and how they are affected by various configuration parameters and MR design patterns. The talk will illustrate the underlying mechanics of job and task execution, including the map side sort/spill, the shuffle, and the reduce side merge, and then explain how different job configuration parameters and job design strategies affect the performance of these operations. Though the talk will cover internals, it will also provide practical tips, guidelines, and rules of thumb for better job performance. The talk is primarily targeted towards developers directly using the MapReduce API, though will also include some tips for users of higher level frameworks.

No Elephant is an Island – Integrating Hadoop into the Enterprise

Jonathan Seidman, Solutions Architect, Cloudera

For years, Hadoop and associated tools like HBase and Hive have been providing companies with valuable storage and analysis capabilities. Increasingly though, organizations are looking to integrate Hadoop into their existing data infrastructures to support new data processing and analysis that was impractical with traditional data management systems. They’re also looking to go beyond command line interfaces to facilitate analysis of data through richer toolsets. Although still a young market, vendors have taken note of these trends, and we’re seeing rapid development of tools to provide data access, integration, and analysis for Hadoop. In this talk we’ll look at common patterns being applied to leverage Hadoop with traditional data management systems, and then look at the emerging landscape of tools which provide access and analysis of Hadoop data with existing systems such as data warehouses, relational databases, and business intelligence tools.

A new generation of data transfer tools for Hadoop: Sqoop 2

Bilung Lee, Software Engineer, Cloudera
Kathleen Ting, Customer Operations Engineer, Cloudera

Apache Sqoop (incubating) was created to efficiently transfer big data between Hadoop related systems (such as HDFS, Hive, and HBase) and structured data stores (such as relational databases, data warehouses, and NoSQL systems). The popularity of Sqoop in enterprise systems confirms that Sqoop does bulk transfer admirably. In the meantime, we have encountered many new challenges that have outgrown the abilities of the current infrastructure. To fulfill more data integration use cases as well as become easier to manage and operate, a new generation of Sqoop, also known as Sqoop 2, is currently undergoing development to address several key areas, including ease of use, ease of extension, and security. This session will talk about Sqoop 2 from both the development and operations perspectives.

HDFS NameNode High Availability

Aaron Myers, Software Engineer, Cloudera
Suresh Srinivas, Founder, Member of Technical Staff, Hortonworks

The HDFS NameNode is a robust and reliable service as seen in practice in production at Yahoo and other customers. However, the NameNode does not have automatic failover support. A hot failover solution called HA NameNode is currently under active development (HDFS-1623). This talk will cover the architecture, design and setup. We will also discuss the future direction for HA NameNode.

Why HBase? Or: How I learned to stop worrying and love consistency

Shaneal Manek, Software Engineer, Cloudera

There are a number of excellent databases that have proven invaluable when working with `big data` – including Cassandra, Riak, DynamoDB, MongoDB, and HBase. So, how do you decide which is the right solution for you? This talk will start by briefly discussing some of the theory involved with distributed databases – from the oft-cited (and almost-as-often misunderstood) CAP theorem, to vector clocks and the difficulties of eventual consistency, and much more. We will then compare how mainstream distributed databases make use of these concepts, and the tradeoffs they incur. Unfortunately, due to the nature of these tradeoffs, there is no one-size-fits all solution. So, we`ll discuss what classes of problems each of these systems is appropriate for – and include some real world benchmarks – to help you make an informed decisions about your distributed systems.

The four day event will feature concurrent tracks of over 250 presentations from leading US-based and international companies. In addition, breakout sessions, thematic panel discussions, keynote addresses and investor meetings will be offered during the conference. This global gathering of over 1000 institutional investors, private equity investors, VCs and leading executives will address trends and developments within the industries, as well as near- and long-term investment opportunities.

With multiplying data volumes and complexity, limited resources, and new technologies, it can be hard to gauge whether your business intelligence (BI) program is on the right track to meet changing business objectives. Does your team have the skills and knowledge it needs to make the most of your information assets? Is your BI program scalable and flexible, and is it headed in the right direction?

At the TDWI World Conference in Chicago, they provide essential BI/DW courses and go beyond to discuss evolving technologies such as Hadoop, MapReduce, big data analytics, and social media. We’ll also cover the important topics people are working on today, including data quality, data governance, ETL, and agile development.

What will you learn?

What is Apache Hadoop?
What are the components in the Hadoop ecosystem?
What is the motivation for using Apache Hadoop?
How is Hadoop being utilized with sensor data?
What are the benefits of analyzing data with Hadoop?

Who should attend?

This session is appropriate for developers, system administrators, managers, architects, or anyone interested learning about Apache Hadoop through the lens of sensors data. No previous Hadoop knowledge is required.

Berlin Buzzwords 2012 is a conference for developers and users of open source software projects, focussing on the issues of scalable search, data-analysis in the cloud and NoSQL-databases. Berlin Buzzwords presents more than 30 talks and presentations of international speakers specific to the three tags “search”, “store” and “scale”.

From Batch to Realtime with Hadoop

Lars George, Solutions Architect, Cloudera

In the early days of web applications, sites were designed to serve users and gather information along the way. With the proliferation of data sources and growing user bases, the amount of data generated required new ways for storage and processing. Hadoop’s HDFS and its batch oriented MapReduce opened new possibilities, yet it falls short of instant delivery of aggregate data to end users. Adding HBase and other layers, such as stream processing using Twitter’s Storm, can overcome this delay and bridge the gap to realtime aggregation and reporting. This presentation takes the audience from the beginning of web application design to the current architecture, which combines multiple technologies to be able to process vast amounts of data, while still being able to react timely and report near realtime statistics.

Hadoop Ecosystem

Kai Voigt, Senior Instructor, Cloudera

Hadoop is the core of a growing ecosystem of software components. Hadoop itself only consists of a distributed file system (HDFS) and a functional programming layer (MapReduce) to process stored data. The Hadoop Ecosystem developed tools around these basic components to interact with Hadoop more easily and integrate it into other environments.

This talk gives a brief introduction into Hadoop, and then covers the most important tools tools around it. When possible, demos will be presented. Sqoop & Flume – Getting Data into HDFS from Logs and Databases Hive & Pig – High Level Access to MapReduce HBase – A random access, low latency key/value store on top of Hadoop Mahout – A Machine Learning Library for MapReduce Whirr – Install a Hadoop Cluster on external cloud providers Oozie – A workflow engine to automatically submit Hadoop jobs and some more tools.

The target audience is new users of Hadoop to get an overview of the many available tools around.

Advanced HBase Schema Design

Lars George, Solutions Architect, Cloudera

While running a simple key/value based solution on HBase usually requires an equally simple schema, it is less trivial to operate a different application that has to insert thousands of records per second. This talk will address the architectural challenges when designing for either read or write performance imposed by HBase. It will include examples of real world use-cases and how they can be implemented on top of HBase, using schemas that optimize for the given access patterns.

Even as Open Source has gone mainstream, its influence on cloud computing, mobile technology, and Big Data is still in its infancy. Businesses are increasingly data-driven, as the Internet, cloud strategies, social media and mobile technologies have made it possible and imperative to collect massive quantities of data on customers, competitors, and more. Making sense of all that data, analyzing it in real time and using it wisely, all while deploying data-rich applications — leans heavily on both open-source software–like Cassandra, Hadoop, OpenStack, jQuery, and more—as well as the very principles of open-source development.

OSBC 2012 offers the chance to connect with the developers, users and companies behind the most significant open-source, mobile and cloud technologies, including HTML5 frameworks like jQuery and SproutCore, NoSQL databases like Mongo, Big Data technologies like Hadoop and Cassandra, and cloud platforms like OpenStack and Cloud Foundry. You’ll also learn key strategies so that you can make your business more effectively data-driven. Nowhere else will you hear such a comprehensive perspective on the open source industry. The conference program features content-rich sessions from visionary thought-leaders and industry executives on how open source technology is being used and what the implications are from a business perspective.

Cloudera Co-founder and Chief Scientist, Jeff Hammerbacher, will be presenting at Information 2012!

In 2011, The Economist hosted its inaugural Ideas Economy: Information 2011, bringing together path-breaking entrepreneurs with pioneers of data science to advance conversations about how the information revolution impacts governments, businesses, and individuals–and how to make sense of the deluge. With greater understanding of how big data is changing the world and why it matters, we now begin to apply sharp thinking to how data can be better leveraged to ensure lasting progress in the coming decades. Specifically, we are convening a conference to explore the next generation of smart systems—the networks of chips, sensors, wireless technologies and intelligent software—that are converging to link the physical and virtual worlds and transform the global economy.

The evolution of smart systems presents incredible opportunities—stronger companies, faster medicine, and safer neighborhoods—but just as many challenges. Privacy, security and intellectual-property protection are perpetual worries, made even more challenging as more data are sent all over the world and through ever more people and machines. Without intelligent planning and successful partnerships, these can create huge problems for individuals and companies as well as regulatory challenges for governments. And even well-designed systems are vulnerable to technical breakdowns. As we enter the era of smart systems, this conference will examine the promise as well as the perils, and propose intelligent solutions to each.