If there’s one thing serial entrepreneur Adam Skotnicky would warn organizations about, it’s data complexity. As VP of Engineering at Cloudera and founder of tcp.cloud and Taikun, which was recently acquired by Cloudera, Adam is an expert at capitalizing on emerging opportunities in the tech sector without letting complicated data structures hold him back.
Paul Muller, host of The AI Forecast Podcast, and Adam discuss how engineering teams can find their way back to simplicity while maintaining flexibility and control. They delve into why IT teams feel swamped by tooling and operational challenges, how platform engineering can make things easier for users, and what it really means to achieve that cloud-like agility in hybrid environments.
Here are a few of the main points from the discussion.
The Pitfall of Overengineering
Paul: Organizations today are managing data across multiple clouds, on-prem, and hybrid environments. From your perspective, what are the biggest challenges they face in that complexity?
Adam: The thing is that you need to focus on the core value of what you’re trying to build.
If you go all in, you might overengineer your solution. You don’t need to have all the features on the planet. It’s like a candy shop for engineers, right? They go crazy. Then you have the sugar rush, and then you have this huge fall after that. It’s exactly what it is.
The Future Is Workload-First, Invisible Infrastructure
Paul: What was the inspiration to try to create a more cloud-like experience in your data center? I think a lot of technologists would say that the issue with this promise of hybrid has always been that my on-premises stuff might have a little bit of automation, but it’s nowhere near as slick or as simple as when I’m using a public cloud service, where they spend a lot of engineering dollars to make it really feel like a catalog. Do you agree that that’s been the compromise in the past, and how did you get around that with what you were doing with Taikun?
Adam: If you want to build something similar, the cloud-like experience means removing people from the process. If you have any ticket between you and your application, or if I own this application, you log in, go to the catalog, and deploy things. That’s the ultimate goal. Beyond that, no people touch it; they observe, make sure it works, and ensure it’s performant and secure. They do this without you, without requiring anything from you, and that’s how public cloud works. That’s the experience; that’s what cloud-like means.
Paul: Talk to me about what you’re seeing in the marketplace as it comes to deploying these big data workloads. How does a self-service, flexible cloud experience empower teams to focus on insights rather than infrastructure?
Adam: I absolutely agree that it’s about workload and workload only. It’s not about the infrastructure, and that’s why we don’t want anyone to touch it. You want to abstract the infrastructure completely, but we still allow you to go and tinker with it. You can do that and explore, but in production environments, you shouldn’t touch it. You should follow best practices because then you can finally focus on the workload, and you shouldn’t go from the workload down. The infrastructure should be there. That’s what we’re doing at Taikun. We focus on the workload.
One Platform, Any Environment
Paul: What are people using workloads like the Cloudera platform going to notice that's different about this new way of working as they start to deploy?
Adam: We are now the abstraction layer for Cloudera services, so Cloudera services will be independent of that environment, so they can run on public or private cloud on your few servers or hundreds or thousands of servers and still have the same experience. You can now run as many of them as you want, connect them to as many endpoints as you want, choose where to combine, and then configure them. It’s not a public cloud or a hybrid cloud. You can use both. You can run your production environments, which you can scale on-prem because of data sovereignty, and you can play with technologies in the public cloud because you can scale up and down from zero to a hundred in minutes. You can combine these approaches.
Paul: Amazing. What do people need to do to start preparing for this new world? Is it something they can just instantly drop in, and it’s a technology problem, or how much of this is a people problem where you need to start to get people to think differently? What do I need to do to get ready to get the most out of hybrid?
Adam: You can choose your approach. You can go with my preferred way, which we call the “golden pot.” Everything is built in, so you can go one way or another or somewhere in between. You can still run your old, good virtual machine side by side with this environment. There are loads and loads of know-how built into the structures and processes already in place. Both approaches will be there, and in Cloudera products, if you choose not to interface with this new world, it’ll be embedded for you.
Catch the full conversation with Adam Skotnicky on The AI Forecast on Spotify, Apple Podcasts, and YouTube.
