Transforming IT Blog
Join us in the Transforming IT HP Blog where we will discuss reinventing IT to overcome obstacles and take advantage of Instant on Enterprise opportunities.

Big Data — The plan comes together

By Donald Livengood, HP Distinguished Technologist/Strategist


I love it when a plan comes together…or, in the case I’m about to describe, when a plan is validated.


Sizing Hadoop: Don’t sweat it


A lot of work has gone into HP’s AppSystem for Apache Hadoop and into the Reference Architectures. The Reference Architectures were announced and discussed at HP Discover 2012 and, up until this year, many customer were still sweating bullets over sizing servers for their Hadoop clusters. For many this is a natural worry, since many of the MPP and BI systems, or other technologies they’ve worked with, really needed to be closely thought out and sized correctly; otherwise production problems would surface quickly.


Hadoop isn’t the same. It scales out beautifully in a linear fashion. However, based on past history and experience, many customers had not bought into the concept of “Need to speed up your tasks? Just add more nodes!” Instead, they wanted to get it right the first time. I completely understood that mindset, having had relatively recent experience in a space that did require thinking of the end-state up-front (VDI)…as well as a long history in the industry.


I say “had not bought into” because this year it seems to be changing. I’ve had multiple customers say that they aren’t getting wrapped around the axle with detailed sizing. Instead, they are taking our reference architecture(s) as-is, or slightly modified, and buying enough servers that they believe will handle their initial workloads; and if they don’t have it perfect, they’ll just add more nodes.


I had one customer tell me specifically that, in hind-sight, they probably wasted 10-12 weeks of work tweaking their own reference architecture and conducting performance testing of their design – only to find that their Hadoop cluster grew faster than they anticipated and they’d have been better off buying our AppSystem for Apache Hadoop or just implementing our reference architecture up front. They then could have focused more on the data ingestion and processing rather than spending so much time on the system architecture. Like I said, it’s nice when a plan comes together.


Clearly there is still a need to monitor a Hadoop cluster closely. Most implementations do make several tweaks related to Hadoop configuration files, JVM configuration, Operating System configuration/tuning, and even some BIOS changes. That’s going to be the case for most Hadoop implementations and will be driven by specific workloads in the environment.  


Consider combining technologies to create Big Data solutions


There was another “Ah-Ha” moment at a recent roadshow event. The event had a Hadoop focus, but during side meetings the topic of replacing everything with Hadoop or mixing-and-matching came up. As it turns out, for most of the customers, a combination of technologies is the way to go.


With Hadoop, customers have the option of loading up data from new, irregular data sets into Hadoop, analyzing them and, perhaps, transforming those datasets into structured results that can be consumed by other systems. A similar use of Hadoop is to load more structured datasets and perform exploratory analytics on them and then, once interesting results are obtained, move the on-going detailed analytics of that data to a specialized system.


In fact, these two use cases were discussed by one of the customers at the roadshow and, in this particular case, the “other system” was Vertica. The “Ah-Ha” for me here was around Vertica creating integration points to Hadoop. Autonomy has provided similar connectivity to Hadoop and, interestingly, Vertica and Autonomy also have connectors to each other. Another validation of the plan.


Don’t forget the surrounding infrastructure

Big Data technologies, whether Open Source or commercial products, all need to behave as expected and function as proper tenants of the datacenter. Governance of the data in the solution and ensuring protection and compliance are critical. Integration of the solutions into the operational model will need to take place. This will include integration into the Identity & Access Management systems as well as ensuring sensible and practical backup and recovery processes are in place. 


Monitoring of the performance and capacity of these new systems at the application, server, storage, and network layers is critical and essential for IT to consider as these new technologies enter the data center. As I stated earlier, these systems need to behave as proper tenants of the datacenter and should be assessed and analyzed with the same critical eye.


How do you cover all of these topics? It turns out that HP has announced services at Discover 2012 in this space to address some of these areas. (Read how HP can help you deploy a successful roadmap for Hadoop). Discover 2013 will likely unveil more services in this space. Again…more of the plan coming together.


 As HP Discover 2013 approaches, I expect you’ll see HP continue to provide hardware, software, and services that can be combined to solve Big Data challenges. The plan continues to evolve.


Feel free to chime in!


To learn more about me and how I can help your organization develop a Big Data plan, click below.


Donald Livengood, HP Expert.jpg

Tags: Big Data| Hadoop
Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the community guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
About the Author
About the Author(s)
  • I’m a Global Strategist, a certified (PMI) Project Manager, specializing in business to IT alignment, agility consulting, Infrastructure Transformation and Strategic Architecture for Big Data, Mobility, Private Cloud, Unified Communications and Collaboration. I drive the strategy, vision and content of strategic consulting services in the Big Data IT Infrastructure services area at HP. As part of this, I meet with senior level customers to understand their challenges, conduct workshops to determine future vision and roadmaps as well as presenting at industry and analyst events.
  • Craig Partridge is the WW strategy lead for HP Technology Services Networking group. His role now covers strategy for consulting, professional and support services. The major areas of focus are Mobility, UC, Cloud Networking and IPv6. All aligned to core HP networking solutions - simplified, secure, optimized and available.
  • Don has held roles with the business and marketing of consulting for HP. Currently he supports HP's Client and Microsoft Solutions and the emerging Mobility Consulting services. He holds a MBA from UCLA's Anderson School.
  • Over 12 years of consulting, new technology services development and marketing experience covering data center, IT infrastructure, cloud technology domains. Hande holds a M.B.A degree from Bentley College, MA.
  • Having joined HP in 2003 Ian Jagger is the world-wide marketing and program manager for HP Technology Consulting's Strategic Consulting Services, Critical Facilities Services and Energy and Sustainability Management Services, as well as emerging IT services Prior to his current role, he served as the HP Services Marketing Manager for Central and Eastern Europe, Middle East and Africa, having joined HP in a similar role in the Middle East. Prior to HP Jagger had a 15 year international sales career, culminating in being Sales and Marketing Director for Steelcase Inc addressing Northern Europe before focusing more specifically on marketing. His initial focus was consultancy and interim marketing management, primarily for small to mid-sized customers based or looking to expand in the Middle Eastern region. Immediately prior to joining HP he was a strategic marketing consultant addressing investment targets for a technology fund. Born in Rochdale, United Kingdom, Jagger holds an honors bachelor of science degree in economics and a degree in social psychology from Loughborough University, England. He also holds a Masters Diploma in Marketing from the Chartered Institute of Marketing, is a Member of the Chartered Institute of Marketing and a Chartered Marketer. He has one daughter and lives in Cary, North Carolina.
  • Jeff Enters works in the HP Technology Services Networking organization and consults with customers on their IT strategies. He has over 20 years of consulting, design and integration experience in multi-vendor Voice and Data environments.
  • Editor and writer with 12+ years experience in the corporate software and technology sectors.
  • Jordan Lee has over twenty years of consulting and industry experience, helping some of the world’s largest firms craft and implement their business and IT strategies. His priority is to advise Hewlett-Packard clients how to best prepare for and take full advantage of the dramatic shifts in the IT economy taking place today. Over his career, Mr. Lee has held executive positions both in industry and consulting organizations, where he has provided consulting to some of the largest firms in the US. Over the years, he has helped his clients redesign business processes and organizations, and implement strategic IT programs around ERP, System Integration, Business Intelligence, and IT Infrastructure.
  • Jordan owns the worldwide mobility portfolio for TS Network Consulting and is a OneHP Mobility ambassador. His expertise centers on mobility strategy, client and applications virtualization and enterprise mobility management technologies.
  • Eduardo Zepeda, WW TS Social Media Program Manager & Internal Communications for WW Technology Services Blogging on behalf of HP Technology Services (TS_Guest)
  • Ken Larson has over 30 years of experience in Information Technology aligning business to technology. As an Enterprise Architect, he has delivered many successful architecture related services across business and government sectors in manufacturing, insurance, banking, oil, utilities, US state and federal governments. He is certified in TOGAF and IT Service Management.
  • Laura Cunningham is a CPA and business consultant with HP Technology Services Consulting. She helps CIOs and their teams bridge the gap between what the CIO wants and what the CFO requires by building a comprehensive business case that can withstand financial scrutiny.
  • I am on the WW Cloud and Big Data Solutions Team. I help our customers adopt HP advanced solutions that are made up of products and services from across HP. I have over 30 years experience in the technology business including 17 years of business ownership.
  • Broad mix of experiences developed in more than 20 years of technology-driven innovation. Fascinated by changes triggered by mix of behavior, needs and technology. Bachelor in Theoretical Physics.
  • Working for EMEA TS Consulting, I am a Specialist in end to end management of customer data, from creation through consumption, to protection and preservation and ultimately (controlled) destruction. This includes, host, connectivity, storage, data protection and backup and archive, from a technical and more importantly, operational perspective. I have worked in the storage and data management industry for over 15 years, on both sides of the desk, as a customer and now as a consultant.
  • Patrick Lownds is a Senior Technology Consultant and is involved in designing and delivering both Client Virtualization and Cloud Computing solutions in the datacenter. Patrick co-authored “Mastering Hyper-V Deployment
  • I’m a Global Chief Engineer certified Exchange Architect and Master (MCA and MCM), specializing in Messaging, Mobility, Private Cloud, Unified Communications and Collaboration. This relates to all work to coordinate sales, pursuit and delivery readiness in all services that relate to HP's portfolio around Microsoft Exchange. Includes internal readiness as well as external events, analyst briefings. With 25+ year’s experience in the industry. Thomas has been involved with Microsoft products since 1993. Specialties: - Architecting complex public and private Cloud solutions for Exchange, SharePoint and Lync in standard, dedicated or hybrid scenarios. - Lead the HP specialists team unit to win and acheive our business targets and budget - Drive HP's Exchange Services for Private Cloud - Working with partners, vendors and internal teams to align, expand and grow HP's strategy.
  • Tim Swiader has twenty plus years in the Information Technology industry. He has worked primarily with the fortune 100 and legacy carriers transforming their applications, networks and data center facilities.
  • Tom Clement has over 30 years experience in the areas of adult learning, secondary education, and leadership development. During this time Tom has been a consistent champion of “non-traditional” training delivery methods, including blended learning, virtual delivery (self paced and instructor led), the use of training games and simulations, and experiential learning. Tom has spent the past 25 years of his career at Hewlett Packard, focused most recently on HP’s global Virtualization, Cloud, and Converged Infrastructure customer training programs. Tom manages the strategic direction and overall performance of these training programs, ensuring these worldwide programs help HP’s customers capitalize on the business opportunities made available by IT advancements in each of these subject areas. Tom and his global teammates utilize best in class instructors, course content and supporting equipment infrastructure to deliver these training programs to HP’s customers. The team prides itself on providing the Virtualization, Cloud, and Converged Infrastructure content customers need when and where they need it, anywhere in the world. Tom is based in the Washington, DC suburbs and can be reached at
  • Tari is a Distinguished Technologist with 30 years of IT and cyber security experience. He is dual board certified in information security/business continuity and is responsible for a wide range of management and technology consulting services encompassing information security, disaster recovery, privacy, and risk management. His problem-solving skills, knowledge of various technology platforms, compliance statutes, industries, as well as his experience in deploying defense-in-depth and InfoSec Program solution architectures is commonly applied when advising CIOs/CISOs as well as leveraged in numerous HP client engagements throughout the world. Tari has designed, built, and managed some of the world’s largest InfoSec programs allowing them to defend against even the most aggressive attackers.
  • I provide technical consulting services at all phases including analysis, planning, design and implementation. I have a wide range of experience in WAN and LAN technologies, as well as providing security solutions and deploying operating system infrastructure. Besides working directly with clients to deploy technology in their data centers, I also find myself architecting or discussing solutions with a business’s chief information officer, helping to lay out a roadmap for the coming years.
  • Bill is the Principal Data Center Energy Technologist for HP Technology Services. Kosik is a licensed professional engineer, LEED Accredited Professional, a Certified Energy Manager, and a Building Energy Modeling Professional. He is responsible for research and implementation of sustainable, energy-efficient, and environmentally responsible design strategies for data centers. He is currently a subject matter expert for the USGBC on the new LEED Data Centers, the EPA/DOE on unification of energy metrics, and the Green Grid on responding to the EPA’s Energy Star for Data Centers program. He has an engineering degree from the University of Illinois at Urbana-Champaign.

Follow Us