Thursday, January 10, 2013

Tips to improve Performance/Scalability of WCF Service hosted in IIS7.5


This is nothing new. In fact there are numerous posts/articles scattered all over hence in this post, my aim is to consolidate and organize the tips already published by well known authors to make life bit easier when you need to throttle your WCF Service using .Net 4.0

But before we start, i would suggest its better have a good understanding on entire request handling pipeline right from IIS in order to understand what settings below do when you change them.
System.net Changes

If your ASP.NET application is using web services (WFC or ASMX) or System.Net to communicate with a backend over HTTP you may need to increase connectionManagement/maxconnection. For ASP.NET applications, this is limited to 12 * #CPUs by the autoConfig feature. This means that on a quad-proc, you can have at most 12 * 4 = 48 concurrent connections to an IP end point. Because this is tied to autoConfig, the easiest way to increase maxconnection in an ASP.NET application is to set System.Net.ServicePointManager.DefaultConnectionLimit programatically, from Application_Start, for example. Set the value to the number of concurrent System.Net connections you expect your application to use. I've set this to Int32.MaxValue and not had any side effects, so you might try that--this is actually the default used in the native HTTP stack, WinHTTP. If you're not able to set System.Net.ServicePointManager.DefaultConnectionLimit programmatically, you'll need to disable autoConfig , but that means you also need to set maxWorkerThreads and maxIoThreads.


ASP.NET Pipeline Optimization

There are several ASP.NET default HttpModules which sit in the request pipeline and intercept each and every request. For example, SessionStateModule intercepts each request, parses the session cookie and then loads the proper session in the HttpContext. Not all of these modules are always necessary. For example, if you aren't using Membership and Profile provider, you don't need FormsAuthentication module. If you aren't using Windows Authentication for your users, you don't need WindowsAuthentication. These modules are just sitting in the pipeline, executing some unnecessary code for each and every request.

The default modules are defined in machine.config file (located in the $WINDOWS$\Microsoft.NET\Framework\$VERSION$\CONFIG directory).

<httpModules>
  <add name="OutputCache" type="System.Web.Caching.OutputCacheModule" />
  <add name="Session" type="System.Web.SessionState.SessionStateModule" />
  <add name="WindowsAuthentication"
        type="System.Web.Security.WindowsAuthenticationModule" />
  <add name="FormsAuthentication"
        type="System.Web.Security.FormsAuthenticationModule" />
  <add name="PassportAuthentication"
        type="System.Web.Security.PassportAuthenticationModule" />
  <add name="UrlAuthorization" type="System.Web.Security.UrlAuthorizationModule" />
  <add name="FileAuthorization" type="System.Web.Security.FileAuthorizationModule" />
  <add name="ErrorHandlerModule" type="System.Web.Mobile.ErrorHandlerModule,
                             System.Web.Mobile, Version=1.0.5000.0,
                             Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
</httpModules>

You can remove these default modules from your Web application by adding <remove> nodes in your site's web.config. For example:

<httpModules>
         <!-- Remove unnecessary Http Modules for faster pipeline -->
         <remove name="Session" />
         <remove name="WindowsAuthentication" />
         <remove name="PassportAuthentication" />
         <remove name="AnonymousIdentification" />
         <remove name="UrlAuthorization" />
         <remove name="FileAuthorization" />
</httpModules>

The above configuration is suitable for websites that use database based Forms Authentication and do not need any Session support. So, all these modules can safely be removed.


ASP.NET Process Model Changes

Process Model configuration defines some process level properties like how many number of threads ASP.NET uses, how long it blocks a thread before timing out, how many requests to keep waiting for IO works to complete and so on. The default is in most cases too limiting. So, the process model configuration can be tweaked to make ASP.NET process consume more system resources and gain better scalability from each server.

A regular ASP.NET installation will create machine.config with the following configuration:

<system.web>
   <processModel autoConfig="true" /> 

You need to tweak this auto configuration and use some specific values for different attributes in order to customize the way ASP.NET worker process works. For example:

<processModel
   enable="true"
   timeout="Infinite"
   idleTimeout="Infinite"
   shutdownTimeout="00:00:05"
   requestLimit="Infinite"
   requestQueueLimit="5000"
   restartQueueLimit="10"
   memoryLimit="60"
   webGarden="false"
   cpuMask="0xffffffff"
   userName="machine"
   password="AutoGenerate"
   logLevel="Errors"
   clientConnectedCheck="00:00:05"
   comAuthenticationLevel="Connect"
   comImpersonationLevel="Impersonate"
   responseDeadlockInterval="00:03:00"
   responseRestartDeadlockInterval="00:03:00"
   autoConfig="false"
   maxWorkerThreads="100"
   maxIoThreads="100"
   minWorkerThreads="40"
   minIoThreads="30"
   serverErrorMessageFile=""
   pingFrequency="Infinite"
   pingTimeout="Infinite"
   maxAppDomains="2000"
/>

Here all the values are default values except for the following ones:

  • maxWorkerThreads - This is default to 100 per CPU. On a dual core computer, there'll be 200 threads allocated for ASP.NET. This means at a time ASP.NET can process 200 requests in parallel on a dual core server. If you have an application which is not that CPU intensive and can easily take in more requests per second, then you can increase this value. Especially if your Web application uses a lot of Web service calls or downloads/uploads a lot of data which does not put pressure on the CPU. When ASP.NET runs out of worker threads, it stops processing more requests that come in. Requests get into a queue and keep waiting until a worker thread is freed. This generally happens when a site starts receiving much more hits than you originally planned. In that case, if you have CPU to spare, increase the worker threads count per process.

  • maxIOThreads - This is default to 100 per CPU. On a dual core computer, there'll be 200 threads allocated for ASP.NET for I/O operations. This means at a time ASP.NET can process 200 I/O requests in parallel on a dual core server. I/O requests can be file read/write, database operations, web service calls, HTTP requests generated from within the Web application and so on.
  • minWorkerThreads - When a number of free ASP.NET worker threads fall below this number, ASP.NET starts putting incoming requests into a queue. So, you can set this value to a low number in order to increase the number of concurrent requests. However, do not set this to a very low number because Web application code might need to do some background processing and parallel processing for which it will need some free worker threads.
  • minIOThreads - Same as minWorkerThreads but this is for the I/O threads. However, you can set this to a lower value than minWorkerThreads because there's no issue of parallel processing in case of I/O threads.
  • memoryLimit - Specifies the maximum allowed memory size, as a percentage of total system memory, that the worker process can consume before ASP.NET launches a new process and reassigns existing requests. If you have only your Web application running in a dedicated box and there's no other process that needs RAM, you can set a high value like 80. However, if you have a leaky application that continuously leaks memory, then it's better to set it to a lower value so that the leaky process is recycled pretty soon before it becomes a memory hog and thus keep your site healthy. Especially when you are using COM components and leaking memory. However, this is a temporary solution, you of course need to fix the leak.


Things Improved in WCF 4

In WCF 4, the default settings are much relaxed and carefully calculated to make best balance of CPU and threads. Here's what the default configuration is according to WCF team member's blog post:

In WCF 4, we have revised the default values of these settings so that people don’t have to change the defaults in most cases. Here are the main changes:

  • MaxConcurrentSessions: default is 100 * ProcessorCount

  • MaxConcurrentCalls: default is 16 * ProcessorCount

  • MaxConcurrentInstances: default is the total of the above two, which follows the same pattern as before.

Yes, we use the multiplier “ProcessorCount” for the settings. So on a 4-proc server, you would get the default of MaxConcurrentCalls as 16 * 4 = 64. The basic consideration is that, when you write a WCF service and you use the default settings, the service can be deployed to any system from low-end one-proc server to high-end such as 24-way server without having to change the settings. So we use the CPU count as the multiplier.

We also bumped up the value for MaxConcurrentSessions from 10 to 100. Based on customer’s feedback, this change fits the need for most applications.

Please note, these changes are for the default settings only. If you explicitly set these settings in either configuration or in code, the system would use the settings that you provided. No “ProcessCount” multiplier would be applied.


 HTTP.sys kernel queue limit

  • Increase the HTTP.sys queue limit, which has a default of 1000. If the operating system is x64 and you have 2 GB of RAM or more, setting it to 5000 should be fine. If it is too low, you may see HTTP.sys reject requests with a 503 status. Open IIS Manager and the Advanced Settings for your Application Pool, then change the value of "Queue Length".


processModel/requestQueueLimit

configuration limits the maximum number of requests in the ASP.NET system for IIS 6.0, IIS 7.0, and IIS 7.5. This number is exposed by the "ASP.NET/Requests Current" performance counter, and when it exceeds the limit (default is 5000) we reject requests with a 503 status (Server Too Busy).

1 comment:

  1. Did you know that you can generate money by locking selected areas of your blog or site?
    Simply open an account with AdWorkMedia and add their content locking plug-in.

    ReplyDelete

There was an error in this gadget