Simplicity Is An Achievement
build it
Look at those two remote controllers. Are they familiar? I used a multi-button, universal remote for years. I mastered its programming sequences to power my devices on/off in the proper order. But, when I first tried an Amazon Fire TV remote, it blew me away. Within seconds, I was easily navigating movies, live broadcasts, and music. The controller was intuitive. It was simple. But this simplicity was only possible because of the enormous technological progress that gave me a tiny computer-on-a-stick that plugs into my television's HDMI port. Brilliant!
What happened? As we got more and better technology (DVD players, stereo systems, televisions), the complexity grew until some very smart people devised ways to combine and abstract away all this complexity into something simple, like Fire TV1. This is a massive achievement.
This same technical progress and creeping complexity is evident in the world of computer servers. The rise of complex, private data centers gave way to easier-to-operate, multi-tenant cloud providers (AWS, Azure, CloudFlare). Gone are the days of knowing how to wire up and cool down racks of servers, at least for most of us.
Have any of you programmed with machine code (assembly)? If so, have you done it lately? I’ll bet not. Higher-level programming languages have evolved to supplant our direct use of machine code. We write more complicated software faster than ever because of them.
A similar transformation has happened over the last eight years with "serverless," which provides us with simpler technologies, such as managed services, PaaS, FaaS, etc. The complex is abstracted away to allow businesses to focus on building value quickly. Ben Kehoe makes this point in his seminal essay "Serverless is a State of Mind" (linked below, in References).
While writing this article, I came across a post from AWS's Alexey Kuznetsov on LinkedIn where he implores us to "Know the perils of PaaS." Instead of writing a short comment under his original piece, I thought I'd use his arguments as a foil and fold them into this article. They do a great job stressing some of the complexity I sought to demonstrate. All block quotes are from Kuznetsov's article.
"I recently had a call with a CTO of a customer who is a huge serverless advocate. Among other things he shared that his developers don't understand VPC, don't know how to do a subnet math and always defer to some veteran old-schooler any time they face a need to do any of that. [sic]"2
I'd like to focus on the line "developers don't understand VPC" and agree with him. Developers, in general, do not understand VPC. And, in the near future, they won't need to.
The VPC will ultimately share the same fate as machine code. It will exist, but it will be abstracted away. Any knowledge you invest in learning VPCs will have an increasingly smaller payoff over time. That knowledge will be needed somewhere (e.g., working for AWS), but most CTOs will use higher-level tools to build their products.
When I talk to people about ownership, specifically engineering ownership, I use VPCs as an explicit example of what to avoid. Requiring your engineers to manage CIDR blocks, NACLs, and Subnets adds unnecessary complexity to your system. It takes away mental capacity that could be used to build and improve your product.
While using a VPC was once unavoidable, we can make smarter choices today:
RDS wants a VPC. Use DynamoDB.
OpenSearch wants a VPC. Use Algolia.
Fargate wants a VPC. Use Lambda.
Elasticache wants a VPC. Use Momento.
Kafka wants a VPC. Use EventBridge.3
This is the primary reason I advocate for serverless: It simplifies ownership, simplifies app running, and allows you to invest more time in things that give you a competitive advantage and less time in things that don't.
"If you take comfort of a managed platform, at some point your toolkit shrinks to one or two platforms and all you know is how to wire up some Lambdas and IAM policies. None of which are fundamental skills. [sic]"4
"Fundamental skills" to whom? To a CTO trying to stand up a product to secure market share? To an engineer teaching others how to build better applications faster? Using managed platforms may not be a fundamental skill for someone who has to build and maintain VPCs, CIDR blocks, NACLs, and the like, but I don't want to be that person. It's not necessary.
What if I re-phrase the previous quote using my machine code analogy from above? "If you take comfort in a modern programming language, at some point, your toolkit shrinks to one or two languages, and all you know is how to encapsulate some data in classes and interfaces. None of which are fundamental skills." Is this a bad thing? Not at all, no.
I love higher-level programming languages and do just fine without manipulating machine code directly. One long semester of assembly was plenty for me. If machine code is your passion, great! Do it. But don't advocate that everyone should know it as a “fundamental skill.”
In other words, VPCs have had an important, fundamental place in building and running apps. But, the future will largely ignore them.
Luca Rossi, from refactoring.fm, sent a recent email that is worth including in this discussion. He wrote:
If you want to focus on delivering value and staying one step ahead of your competitors, start with serverless. Wire up those Lambdas and IAM policies and build, baby, build! Let VPCs go the way of machine code.
Simplicity isn't a shortcoming; it's an achievement. Have fun.
References
Alexey Kuznetsov on LinkedIn: Know the perils of PaaS
Ben Kehoe: Serverless is a State of Mind
Algolia
Momento
Luca Rossi: Refactoring
Or Apple TV; its controller is also very intuitive
Or Kinesis, as your case may be. Kafka tries to do everything.