Resolved: 6:30 PM Sun, February 3, 2013 (UTC)
Late yesterday our cloud host isolated the root cause of the error rates and this morning deployed a fix to all of our load balancer instances. As expected during normal operation our external metrics now show no errors occurring and we believe the situation to be resolved. We're monitoring the situation and ask that anyone experiencing any difficulties please contact support@librato.com. We thank you for your patience during this degradation and continue to strive to provide the most reliable/robust metrics solution possible.
Update: 4:50 PM Sat, February 2, 2013 (UTC)
Continuing to test the assumption that the issue is at least partially exacerbated by the number of concurrent connections, yesterday we created additional load balancer instances and used DNS round-robin to spread our API connections evenly across them. As best we can tell from our external metrics, this appears to have somewhat reduced the rate of errors. We will be spawning additional load balancer instances today and continue reducing the connections served by any one instance until our external metrics indicate error rates are either more manageable or returns have otherwise diminished.
We remain in communication with our cloud host as they continue their efforts to uncover the root cause of the issue and in parallel continue work on implementing an alternative solution.
Initial Issue
We've been investigating reports that some of our users have been experiencing elevated rates of 503 errors off/on this week as they've made requests to our API. We wanted to provide some preliminary results of the investigation, inform users of our progress towards resolving the issue, and suggest basic workarounds.Current Progress
Prior to reports of degraded service, our architecture leveraged a load balancer maintained by our cloud host that routes all incoming connections to a tier of Nginx instances. These Nginx instances then proxy requests to the appropriate services (e.g. web application, API, etc). The user-reported errors were not showing up in either our Nginx/individual service logs or the load balancer metrics provided by our cloud host. As the metrics at our disposal did not match the growing amount of anecdotal evidence that error rates were elevated, we've put a system in place that makes synthetic requests of different types to our API from an external location. These new metrics now reliably reproduce an unacceptably high percentage of 503 errors silently occurring at our load balancer without showing up in the cloud host-provided metrics. We have provided our cloud host with a test account on our service and all the tooling they need to reproduce these results. They have confirmed their ability to reproduce the errors and have escalated the support request.Pending the results of their investigation and absent valid metrics/insight into the actual behavior of the load balancer, our best guess was that this had something to do with the number of connections being serviced by the load balancer. So we took two steps:
- Split each service onto it's own load balancer instance.
- Increased the capacity of the service tiers directly adjacent to the load balancer by 33%.
The first step has the obvious effect of lessening the number of concurrent connections serviced by any single load balancer. The second step was taken to completely rule out any chance that there were capacity issues under our direct control at fault. Our service metrics have in no way indicated that we were under-provisioned, but we wanted to completely rule that out.
After taking these steps we ran the same tests directly against our API again and additionally through a reverse-proxy route in our web application (that is now handled by a separate load balancer instance). Our testing shows the same high error rate to our API service and little or no errors occurring through the reverse-proxy. This indicates that the specific load balancer instance in front of our API service is either defective in some way, or generally unable to handle the number of connections it's receiving. We have forwarded these results to our cloud host and are currently awaiting further guidance. In parallel we have already started working towards implementing an alternative load balancing solution in unlikely case our cloud host ultimately cannot provide a load balancer that meets our performance requirements.
Current Guidance
The one step our users can take to alleviate the effect of the issues we're currently addressing is to take reasonable steps to retry failed requests, e.g. 1-3 attempts, when an API request fails with a 503. This is not a guarantee
We take full ownership of our responsibility to deliver a robust, reliable experience to our users. Please do not interpret any of the details/guidance provided above as an attempt to deflect culpability and follow us on Twitter for subsequent updates.