Heroku Admits To Performance Degradation Over The Past 3 Years After Criticism From Rap Genius

Heroku, the popular cloud application platform, may not be quite as fast today as it was three years ago. Yesterday, Rap Genius‘ James Somers , posted a widely read blog post, arguing that Heroku had quietly changed the way it distributes tasks from Ruby on Rails apps across the Amazon EC2 machines it makes available to its users at some point in the last few years without alerting developers of this change. Instead of intelligently routing requests to the next available server, as Heroku did in its early days, Somers argued that it now distributes requests randomly, resulting in increased queuing times. Today, Heroku’s general manager Oren Teich admitted that this is indeed the case.

“We failed to explain how our product works,” Teich wrote this evening. “We failed to help our customers scale. We failed our community at large. I want to personally apologize, and commit to resolving this issue.” According to him, Heroku users have indeed “experienced a degradation in performance over the past 3 years as we have scaled.” Heroku’s goal, he wrote, is to be the best platform for all developers. “In this case, we did not succeed. But we will make it right.”

Tomorrow, Teich promises, the company will post an in-depth technical review and the company will also soon provide developers with a clearer picture of what their apps’ queue of web requests really looks like. He also promises that Heroku will update its documentation and give developers tools to understand how to improve the performance of their apps.

The degraded performance is especially visible for sites like Rap Genius, which use a large amount of so-called Heroku “dynos,” which are essentially customized on-demand Amazon EC2 instances (there are web and worker dynos, but the problem here only seems to affect the web dynos). In its current documentation, Heroku implies that it uses “intelligent scaling” to route web requests to the next available machine. As Somers discovered, however, the distribution is actually random, which means that many requests can end up in a queue even though there are dynos available that are currently idle.

Heroku charges $35 per month for each of these and the quickly growing Rap Genius currently pays about $20,000 per month to run most of its critical its infrastructure on the platform. For a very in-depth look at how Rap Genius uses Heroku and how Somers figured out that Heroku, take some time out of your busy schedule and read his post here.