A glimpse into our presentation from the Tech Leadership event
The short answer: Always. Tech infrastructure is constantly evolving, and keeping up with that change is crucial for businesses, especially SaaS companies. Failing to upgrade can lead to a drain on resources, with companies spending up to 15-25% of their annual revenue just to maintain outdated systems. These inefficiencies create operational bottlenecks, particularly when rising demand requires scalable solutions. So, what’s the key to success for a tech company looking to stay ahead?
This was the central theme of our first Tech Leadership event, hosted earlier this month at our Cape Town offices in Claremont. Our Senior Engineer, Dr. Stephan Schoeman, led the session, guiding engineers through the intricacies of infrastructure redesign. Stephan’s presentation was a candid dive into the realities of system overhauls, peppered with hard-earned lessons from Gathr’s own experiences and practical advice on how to avoid common pitfalls.
Navigating Infrastructure Overhauls
Stephan kicked off the session by underscoring the importance of proactively upgrading infrastructure. He emphasized that while the natural inclination is to put off major system redesigns, especially when things seem to be working fine, the truth is that waiting until a system breaks under pressure can be far costlier in the long run. His key message: infrastructure is not just a support function but a strategic asset that should evolve alongside your business needs.
Reflecting on Gathr’s journey, Stephan walked the audience through the company’s own tech infrastructure overhaul – from a manual AWS ECS setup to a fully automated Infrastructure-as-Code system running on Kubernetes. He highlighted how moving away from the manual, “clickops” style of infrastructure management toward a modern, scalable system was essential to maintaining Gathr’s growth trajectory.
What our new infrastructure looks like?
Lessons from the Frontline
Stephan shared the real-world challenges of inheriting a system built by an external team, one that relied on manual configurations and was increasingly difficult to manage. As demand on the system grew, so did the operational inefficiencies. Engineers were forced to spend more time managing hardware and resources instead of focusing on delivering innovation.
One of the most eye-opening moments in Stephan’s presentation was his breakdown of how Gathr’s existing infrastructure was strained under the pressure of scaling. The team initially solved some performance issues by implementing a concurrent queuing system and overprovisioning resources. But this led to another problem: rising costs from wasted resources, revealing a deeper flaw in their overall system design.
Finding the Right Path Forward
At this point, the company was faced with a critical decision: invest in internal development to optimize the system or engage an external consulting firm to rebuild it. Stephan explained how Gathr ultimately partnered with LSD Open, a consulting company with cloud migration expertise. This decision wasn’t made lightly, and Stephan took the audience through the pros and cons they had considered:
- Hiring an internal senior developer would give them more control but would require extensive training and time.
- Outsourcing to experts would speed up the process but meant they had to trust an external partner with their critical infrastructure.
Stephan’s honesty about the complexities of this decision resonated with many engineers in the room who have faced similar dilemmas. In the end, the partnership with LSD Open led to the successful migration to Kubernetes, enabling Gathr to achieve the scalability and performance they needed, while also bringing down costs.
Reflecting on the Challenges
As with any major project, the migration wasn’t without its hiccups. Stephan described how unforeseen latency issues during the database migration forced the team to roll back and rethink certain aspects of the system. He framed these setbacks not as failures, but as learning experiences that ultimately resulted in a more robust solution. The lessons learned during this process allowed Gathr to emerge with a more granularly scalable infrastructure, improved system monitoring, and faster deployment capabilities.
Stephan’s pragmatic approach to sharing these challenges and how the team overcame them made it clear that even the most well-thought-out plans can face obstacles. The key, he stressed, is to remain adaptable and continuously refine your approach.
Key Takeaways
As Stephan began to wrap up the talk, he turned to the broader takeaways for the engineers in the room. He emphasized that the success of a project like this is not just in the technical execution but in the questions you ask at the outset. If he could go back to the start of the migration, Stephan said there are several critical questions he would have asked to better prepare the team for the road ahead:
- What is the primary problem we are trying to solve?
- How important is standardization across our systems?
- How much groundwork needs to be laid before the migration?
- Do we have a thorough understanding of our current infrastructure’s weaknesses?
- How flexible are we willing to be in terms of altering our application layout?
- How quickly can our team adapt to rapid change?
- What specific business or performance gains will this migration bring us?
Stephan’s reflections provided the engineers with actionable insights on how to approach their own infrastructure challenges, equipping them with a framework for future projects. By the end of the session, it was clear that while there’s no one-size-fits-all approach to infrastructure overhauls, asking the right questions early on can set the foundation for success.