Performance Strategy for a High-Demand Ecommerce Business (Part One)
Performance is a fundamental part of software quality; ensuring the availability of the different components of a system is crucial to guaranteeing a smooth, secure, and reliable user experience. A resilient system not only guarantees reliability but also allows us to ensure (at least projectively) the achievement of business goals.
For a performance team, events like Cyber Day are—for many reasons—a real challenge. Ensuring that the multiple agents involved in an e-commerce platform (for example) interact smoothly and efficiently is vital to minimizing the risk of sales drops due to system unavailability. However, it's not enough to conduct performance tests using ideal scenarios only to then, during an event, encounter reality, or user behavior that isn't always linear (as is often the case in basic tests), nor does it simply scale gradually, nor does it involve tens of thousands of users waiting for the event to go live. Reality is much more complex.
At Haka Lab, we love a challenge, and some time ago we accepted a tremendous one: helping one of the largest e-commerce companies in Chile (with a presence in other countries like Mexico, the United States, and Canada) successfully navigate Cyber Day.
The challenges were performance-related, information-related, and business-related.
One might think that a robust testing plan and an infrastructure with astronomical expenses would be enough to ensure the minimum stability or performance objectives for a highly complex e-commerce platform, but nothing could be further from the truth.
Before becoming our partners, this e-commerce company experienced firsthand the complete collapse of its ecosystem during a Cyber Day event, registering losses of several billion pesos. This doesn't even include the exponential increase in complaints about issues such as canceled orders, failed payments, and lack of stock, among others.
And while performance tests had been conducted, the question remained: What went wrong?
For us at Haka Lab, the initial challenge was understanding the context, and when we refer to this "context," we're not just talking about infrastructure; we're talking about systems, their interactions, relationships, dependencies, and boundaries.
However, if we wanted to succeed, software and hardware weren't the only areas to consider. Had the performance tests taken into account the business projections for the event? Were there records or projections about user behavior? Did the sending of offer and campaign notifications consider the ecosystem's limitations? As you can see from this brief summary, the problem can be grouped into six key points.
We will review each point in question to understand how these come together in the performance.
Software: We need to understand how the software we're testing "works," including service requirements, error handling, configured timeouts, rate limits, and the impact of its execution on the infrastructure, among other relevant points. At this stage, we can find some curious aspects—especially in large e-commerce platforms—such as a lack of understanding about "other services" or, to be more technical, their specific functions. One of the most striking is the difference in wait times between two directly related services. Let me give you an example (typical in Back-to-Front (BFF) applications):
Service A has a 10-second timeout, and service B has a 20-second timeout. During normal operation, these services run smoothly; the ecosystem's infrastructure ensures that both services respond as expected, or rather, within the parameters of service A.
However, during a high-demand event (please bear with me), service B typically bears the brunt of the workload, as it's the service that actually queries other cloud services, on-premises systems, databases, etc.
Meanwhile, service A, being a best-for-all (BFF), (normally) only makes calls and waits for responses for the front end. Therefore, the response times of service B—driven by the high demand—far exceed the threshold of service A, resulting in problems that are difficult to debug. Why?
In our initial tests of this e-commerce platform, we discovered that this disparity between the services meant that when we looked at the logs or monitoring tools, we saw errors in service A, but when querying service B, we saw correct responses. This disconnect is not only costly to debug, since the error isn't immediately obvious, but it also reveals a much deeper problem stemming from poor communication between teams or a lack of understanding of service contracts.
Additionally, hasty conclusions drawn from performance tests, without proper analysis, could lead us to request infrastructure adjustments for service "A" due to its timeouts, which are often directly linked to overload.
This simplified example (as simplified as possible to avoid causing drowsiness) demonstrates how a more in-depth review can uncover underlying issues, allowing us to address the root causes. This translates directly into reduced risk, cost savings, and greater reliability across the platform's services.
We invite you to stay tuned to our social media channels; we'll be publishing part two soon!