Trying to get the best performance out of a legacy application can be very challenging. It can be particularly hard when that legacy application is based around .NET 4.x Web Forms. Unfortunately, we don’t always have the ability to update our frameworks. This was the case with one of my clients a short time back.
The client needed to be able to deal with high traffic volumes on a .NET Web Forms application. When traffic bursts would arrive, they would find 10,000 concurrent users suddenly hitting the system at once. The client had hoped that by moving to Azure, they could take advantage of auto-scaling in Azure App Services. They were surprised when they continued to see the same problem. In fact, when the systems did scale, they often had little or no traffic on the additional nodes.
Most of their issues were solved with a few changes to web.config and a couple of NuGet packages.
Web Forms provided an incredible model (for its time) that made it easy to build and deploy scalable applications quickly. Out of the box, it had some reliable defaults for developers or single servers. This made it quite popular, despite its limitations. Microsoft also invested significant time to make the framework extensible. Despite that, most developers always use the default configuration:
- Stateful data (session state) stored in-memory
- User profiles and roles (membership) in SQL Server
- ViewState persisted to page with details about the state of any controls.
While this can work for smaller systems and individual servers, it tends to fail in cloud environments. There are two serious issues with this approach:
Using in-memory sessions requires leaving Application Request Routing ( ARR) enabled. ARR creates a special ARRAffinity cookie that ensures all requests for a specific client are sent to the same server each time for processing. In theory, this means the code can use an in-memory session state. This solution is often chosen as an easy, inexpensive path to move legacy .NET Web Forms apps to Azure. Or so they think!
Note: If you’re using App Services, make sure ARR Affinity is disabled if possible! The more stateless your servers, the better they perform and scale.
The default view state persistance created a hidden field with large amounts of data. The kind of data that sets off warnings on Next-Generation Firewalls.
Most of their performance issues related to the choice to use in-memory sessions and ARR. As the loads increased, the auto-scaler would provision more App Service instances. However, ARR would ensure that requests from all existing clients continue to go to the same server. This ensured they could use in-memory sessions. While new requests would be sent to new App Service instances, the large number of existing clients would remain on the original, overloaded server. This pattern would repeat each time the system scaled out. This created a mix of App Service instances, with some under extreme load and others seeing hardly any traffic.
Thinking in-memory sessions will work this way in the cloud is a common mistake. I’ve even seen it on App Services with minimal loads. Believe it or not, ARR Affinity doesn’t actually guarantee your session will remain available in these cases or that you’ll return to the same server. Cloud resources are elastic. This means that they are created and destroyed when needed. Azure upgrades and replaces systems at any time. Even if you’re not needing to scale out, you can still find the server is restarted or replaced at any time. At their core, App Services are meant to be stateless. In short, even if ARR is enabled, you have no guarantee that you will return to the same server or that the in-memory state will still exist.
The customer first tried to do get around this by changing to a session state provider which persisted the data in Azure SQL. This would allow them to eliminate ARR. While this helped increase the supported number of users, it created a new issue. Under load, pages would stop responding or fail suddenly. Behind the scenes, database deadlocks were creating problems for hte requests. It turns out that the session state provider needs to frequently read and write the database. This created a large amount of contention for the records. They tried to scale up the database to keep up with the requests, but continued to see the performance issues. As the load increased, the time required to receive a response grew significantly.
To have this level of scale, systems need at least two aspects. First, it needs to utilize memory to ensure that reads can happen quickly without contention. This is what the in-memory session was providing. Second, it needs to make it available for read and write across all web servers. This distributed cache approach is what they were attempting with Azure SQL. They needed aspects of both.
To solve this, Microsoft released an open-source Redis Session State Provider. You can read about this on their site. By configuring the session store to use Redis, it’s possible to have an in-memory, distributed cache. We get the benefits of both approaches. It does require an instance of Redis, such as the fully managed Azure Cache for Redis.
Azure’s Redis systems have been notorious for rejecting some requests, so retry logic is very important in this configuration. Because of this, Microsoft provided two important settings:
- retryTimeoutInMilliseconds: By setting this to a value greater than zero (default: 5000), the provider will retry failing requests for a specified period of time. After that, it will make one last attempt before throwing an exception.
- throwOnError: By default (true), exceptions are raised when the timeout occurs. If this is set to false, exceptions will not be thrown when the retry attempts fail.
By replacing the original provider with this one, we were able to eliminate the need for ARR and enable them to scale to far more than 10,000 users. How we tested and verified this solution is a story for another post. 🤔
Two more things to be aware of if you’re experiencing these issues. First, if you are using Output Caching in Web Forms, consider the Redis Output Cache provider to get similar benefits from a distributed output cache. Second, if you need to eliminate the View State to avoid WAF issues, you can either create a custom provider or utilize the session state provider to assist you. Web Forms provides an entity called PageStatePersister to determine how that information is persisted. By overriding the returned instance in a PageAdapter, you can use the built-in SessionPageStatePersister instead of the default HiddenFieldPageStatePersister. The view state will now be tracked in memory as part of the associated session.
Hopefully, this post gives you some ways to make the most of a legacy codebase on Azure until you can move to .NET 6 (and the MVC framework).