In the previous blog we were discussing about the basics of app domain recycling and the steps we can take to confirm if app domain recycling is happening. We might come across a situation where from the steps explained previously we will come to know that app domain recycling is happening but we are not sure what triggered the app domain recycling. In this blogs we will discuss about the advanced debugging techniques to identify that.
Below are some of the data that we need to collect.
Steps to collect Procmon:
1. Download process monitor from https://technet.microsoft.com/en-us/library/bb896645.aspx
2. Run procmon.exe
3. Wait till the issue reoccurs and after the issue reoccurrence please stop the trace.
Note: Process monitor collects all the system level events and a lot of data. Even though this is a very good trace, if the issue you are facing is very intermittent avoid going for procmon data collection as this might consume more space and cause disk space issues. |
Steps to Create Debug Diag Rule:
1. We will use Debug Diagnostic 1.2 version to troubleshoot this issue further. Please install the Debug Diagnostic 2.0 update 2 tool fromhttps://www.microsoft.com/en-us/download/details.aspx?id=49924(Please un-install all other versions of Debug Diagnostic tool before installing the latest version)
2. Before proceeding with the rule creation we need to download the symbols needed for the automated rule creation. Take the below steps to download the symbols for your w3wp process.
a. Open Debug Diag and go to processes tab and select the appropriate w3wp process specific to your application pool which we are investigating b. Right click on the process and create full user dump. c. Once you have the dump, open debug diag analysis tool and select the w3wp.dmp file you just created and run crash/hang analysis on that dump. d. After the analysis completes the public symbols will be downloaded to C:\Symbols folder and we are good to proceed with the rule creation |
Note: to run the automated analysis you need internet connection on the server. If you don’t have internet connection on the server then capture the dump on the server and take the dump to the machine where you have internet connection and proceed with step c. copy over the symbols in the C:\symbols folder to the C:\symbols folder on the server. |
3. Create a Crash Rule in Debug Diagnostic 2.0 with the following steps:
a. Open DebugDiag (Start -> Programs -> Debug Diagnostic Tool 1.2)
b. Select "Crash" and click Next
c. If the crashing process is w3wp.exe then choose the option A specific IIS application pool and click Next and choose the application pool that is crashing from the list and click Next. (Note: The Application pool list may be empty if the IIS 6 Metabase Compatibility is not installed. In such a case you can always type the name of the application pool and click Next)
d. If the crashing process is not w3wp.exe then choose the option A specific process and click Next and choose the process name from the list and click Next.
e. Click Next in Advanced Configuration (Optional), click on Breakpoints and then click on Add Breakpoint
f. If it is a 4.x app then Add clr!AppDomainNative::Unload and change the Action Type to Custom and Action Limit to 5 and click on Ok. If it is a 4.x app then Add mscorwks!AppDomain::UnloadWait and change the Action Type to Custom and Action Limit to 5 and click on Ok.
g. Select Action Type as Full User Dump.
h. Click on Save and Close
i. Click Next for "Rule Name". The "Userdump Location" can be changed here.
j. Select "Activate the rule now" and click Finish
3. Notice the Status is Active. The Userdump Count will increase each time a dump file is created.
4. Wait for the issue to happen.
If everything goes well you will process monitor trace and the process dump at the time of the issue.
How to analyze process monitor trace?
As we had discussed earlier we know that application domain recycles whenever there is some change to either the framework directory or to your application directory. So we will be interested to look what exactly is getting changed at the time of the issue and who is the culprit doing it.
I will show you some of the shortcuts that you can use to analyse this trace.
As I said most likely we will be interested in the writes to any directory.
Step 1: Go to Tools->Count Occurrences->Select Operation and then click on Count.
Step 2: We will be interested in WriteFile as below. Click on Filter. Click on close.
Step 3: Next we are interested in changes to either the framework directory or application directory. Go to Filter->Filter.
 |
Step 4: In the first drop down select Path and in second drop down select contains and in the 3 text box, type in your application directory. For example in my case it would be C:\inetpub\wwwroot\TestUserIdentity452. Click on Add
Step 5: You will get a list of write file attempts as below.
Now you can double click on any one of the above by matching the time stamp of the app domain recycle in the event log. Here you will see which process is trying to write the file and the account used by the process.
 |
Repeat from step 4 and try the above steps for the path contains “Framework” and see if there are any writes happening.
How to analyse the dumps captured using debug diag?
Sometimes I have seen that just running debug diag analysis will give you information on the app domain unload exception. If not I will discuss some tricks as to how we can debug this using windbg.
WinDBG basic setup instructions:
If you don’t have windbg you can go ahead and install windbg from https://msdn.microsoft.com/en-us/windows/hardware/hh852365.aspx
Select and download Standalone debugging tools for windows specific to your operating systems.
You can setup the symbols for your debugger or windbg by opening windbg and going to File->Symbol File Path and entering the below path
SRV*C:\symcache* http://msdl.microsoft.com/download/symbols |
Prerequisites:
Download the netext.dll from http://netext.codeplex.com/ which will be using for analysis.
Copy the appropriate version dll’s inside your debugger installation folder.
How to analyse the dumps?
Open the dump in debugger.
Load the netext.dll in the debugger by typing in the below command.
.load netext |
Before you can run any commands you need to run indexing for the dumps you have.
!windex |
I have worked on 3 of the scenarios which might cause application domain recycles and below are the commands and the outputs you might get
Scenario 1: configuration change
0:035> !wfrom -type System.Web.FileMonitor where ($enumname(_lastAction) == "Modified") $a("==================\nPath",DirectoryMonitor.Directory+"\\"+_fileNameLong),$a("Change Time:",$tickstodatetime(_utcLastCompletion.dateData)) ================== Path: C:\inetpub\wwwroot\web.config Change Time:: 06-10-2015 20:20:39 1 Object(s) listed 14 Object(s) skipped by filter |
Scenario 2: Bin directory change
0:036> !wruntime Runtime Settings per Application Pool ========================================================================= Address : 000000F585279150 First Request : 07-10-2015 01:25:51 App Pool User : IIS APPPOOL\DefaultAppPool Trust Level : Full App Domnain Id : /LM/W3SVC/1/ROOT/TestUserIdentity452-1-130886547510004207 Debug Enabled : False Active Requests : 0n0 Path : C:\inetpub\wwwroot\TestUserIdentity452\ (local disk) Temp Folder : C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files Compiling Folder: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\testuseridentity452\cd5dfb1f\82daaaa9 Shutdown Reason : BinDirChangeOrDirectoryRename at 07-10-2015 01:26:49 Change Notification for critical directories. bin dir change or directory rename HostingEnvironment initiated shutdown Change Notification for critical directories. bin dir change or directory rename Change Notification for critical directories. bin dir change or directory rename HostingEnvironment caused shutdownn at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at System.Environment.get_StackTrace() at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal() at System.Web.HttpRuntime.ShutdownAppDomain(String stackTrace) at System.Web.HttpRuntime.OnCriticalDirectoryChange(Object sender, FileChangeEvent e) at System.Web.FileChangesMonitor.OnCriticaldirChange(Object sender, FileChangeEvent e) at System.Web.DirectoryMonitor.FireNotifications() at System.Web.Util.WorkItem.CallCallbackWithAssert(WorkItemCallback callback) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch() |
Scenario 3: Page change
0:033> !wfrom -type System.Web.FileMonitor where ($enumname(_lastAction) == "Modified") $a("==================\nPath",DirectoryMonitor.Directory+"\\"+_fileNameLong),$a("Change Time:",$tickstodatetime(_utcLastCompletion.dateData)) ================== Path: C:\inetpub\wwwroot\TestUserIdentity452\Default.aspx Change Time:: 09-10-2015 00:11:40 ================== Path: C:\inetpub\wwwroot\TestUserIdentity452\Default.aspx Change Time:: 09-10-2015 00:11:17 ================== Path: C:\inetpub\wwwroot\TestUserIdentity452\Default.aspx Change Time:: 09-10-2015 00:11:34 3 Object(s) listed 15 Object(s) skipped by filter |
Reference: http://netext.codeplex.com/#wfrom
Hope this helps J