In the world of microservices, the promise of agility and independent deployments is alluring. Each team owns their domain, develops their services, and deploys them to a shared cloud cluster, forming a grand application symphony. But beneath this harmonious vision lies a common challenge: how do developers actually test their changes before merging and unleashing them into the wild? This is our story of trial, error, and ultimately, triumph, in finding the optimal approach for developer testing.
The testing conundrum in a distributed world
Our development teams, each meticulously crafting their piece of the microservices puzzle, faced a significant hurdle. Before any code could hit production, developers desperately needed a way to validate their changes in an environment that closely mirrored our cloud cluster, without stepping on anyone else's toes or breaking existing functionalities. The sheer number of interdependencies meant isolated unit tests weren't enough. We needed integration, end-to-end validation, and a sense of confidence before pushing forward.
Attempts at a solution: a series of unfortunate events
Like many, our journey began with hopeful experiments, each promising a solution, only to reveal new complexities.
The shared test cluster: a bottleneck nightmare
Our first instinct was simple: create a dedicated "test cluster" in the cloud. Developers could deploy their changes there, test, and iterate. What could go wrong? Well, everything. Soon, this shared resource became a chaotic battleground. Multiple teams deploying their service changes simultaneously led to instability, unpredictable behavior, and a constant state of flux where nothing truly worked for anyone. Not to mention, maintaining and scaling this shared environment came with a hefty price tag, constantly reminding us of its inefficiencies.
Virtual services: the header propagation headache
Next, we explored the concept of virtual services. The idea was clever: route requests based on specific headers to different namespaces of the same service within the cluster. This way, one service could have multiple versions running, and developers could direct traffic to their specific version.
The promise was segregation without duplication. The reality? A mess. Our microservices communicate through a complex blend of synchronous and asynchronous calls. Standardizing header propagation across every single communication path proved to be an insurmountable task. The load is also increased on cluster due to multiple duplicate services. The chaos of misrouted requests and broken dependencies quickly taught us that what looked good on paper didn't always translate to our mixed communication patterns.
Telepresence: a bridge too far for security
Telepresence offered an intriguing alternative, allowing developers to connect their local development environment directly to the remote cluster. It felt like a magic tunnel, letting local services interact with remote ones. However, this approach presented a significant security dilemma. Granting individual developers' direct access to the central cluster, even for testing, was deemed too risky. The potential attack surface and compliance concerns quickly put this promising solution out of reach.
Cluster per team: the costly dream
The idea of giving each team its own dedicated cloud cluster for testing briefly surfaced. It solved the contention problem and offered complete isolation. But the celebratory mood quickly deflated when we crunched the numbers. Scaling this approach to all our teams meant an astronomical cost that was simply unsustainable. It was a perfect solution, but one we couldn't afford.
The lightbulb moment: local is the new cloud
After these many detours, we began to pivot our thinking. What if the solution wasn't more cloud infrastructure, but less? What if we brought the cluster experience down to the developer's local machine?
An important point to note is that running a distributed system on a local machine may appear counterintuitive; however, advancements in hardware and solutions such as minikube and k3s have made this feasible.
Minikube: A promising start, then a grinding halt
Our initial foray into local clusters led us to Minikube. It was easy to set up and offered a great way to simulate a Kubernetes environment locally. We started deploying our services, and things looked promising for a small number of deployments. However, as the service count grew—reaching even a modest 5-6 services—our developers' machines began to groan. CPU and memory consumption soared, rendering their computers unresponsive. Minikube, while excellent for smaller projects, couldn't handle the scale of our interdependent microservices.
K3s: the lightweight champion rises
Just when we thought local clusters might be another dead end, we discovered K3s. Pitched as a "lightweight Kubernetes," K3s is turned out to be our perfect solution. Its minimal resource footprint and streamlined design were exactly what we needed.
With K3s, we successfully managed to bring up a local Kubernetes cluster with a good number of services running concurrently on a standard developer laptop! This was a game-changer. All services currently publish Helm charts for cloud cluster deployment. This method also utilizes those Helm charts to facilitate deployment in the local environment.
Another noteworthy aspect is that developers often use their laptops as a primary workspace. This environment allows them the flexibility to experiment with various approaches, providing new methods for testing and supporting their productivity.
Developer wins
With K3s and Local Launchpad:
- Isolate their work: Test their changes against a full ecosystem of services without impacting others.
- Iterate rapidly: Make changes, redeploy locally, and see the results instantly.
- Reduce cloud costs: Significantly decrease our reliance on expensive cloud test environments.
- Work offline: Develop and test even without an internet connection.
- Quick prototyping: This local setup helps developers to accelerate prototyping.
A few bumps
- Initial setup can be time-consuming (though automation helps).
- Developers now own their local environments—maintenance included.
The Future is Local
Our journey through the microservices testing landscape was fraught with challenges, but each failed attempt taught us valuable lessons. We learned that the "one size fits all" approach rarely works, especially in complex distributed systems. Our eventual embrace of K3s for local development has revolutionized our workflow, empowering our developers with unparalleled autonomy and efficiency. It's a testament to the fact that sometimes, the most elegant solutions are found not by building bigger, but by building smarter, and lighter.
About the Author
Lakshmi Billa is a seasoned software engineer who serves as the Manager of Services Engineering at Pegasystems. She is passionate about making life easier and more productive for developers who work in complex distributed systems.