Issue
When using Central Administration to view content sources for the Search Service Application the content source history and content source pages takes a very long time to load. Crawls might hang with WCF related messages in the ULS logs or will try to run Rebalance Crawl store partition job.
Similar issue has been claimed to have been resolved in April 2011 CU for SharePoint 2010 via this Knowledge Base article
Symptoms:
As the content in the content source increases, the time it takes for the content sources page will increase. This is a good litmus test.
We can also query the Crawl Store database against the MSSCRAWLURL for the number of links against the content sources and Host ID
The ULS Logs will indicate the following sort of message in the Database category with event ID fa43
04/16/2015 08:25:14.47 mssearch.exe (0x3CAC) 0x1678 SharePoint Server Search Crawler:Common ad9s8 Assert location: search\libs\utill\hangrecoverer.cxx(189) condition: !"Crawl hangs" StackTrace: at Microsoft.Office.Server.Native.dll: (sig=c3cf111a-d6fc-4ec5-ba1c-c9b8ebb47e41|2|microsoft.office.server.native.pdb, offset=1F074) at MSSrch.dll: (sig=bcc79b69-8dc0-4f7b-b42b-469030237946|2|mssrch.pdb, offset=22759) at MSSrch.dll: (offset=226FD) at MSSrch.dll: (offset=1DE54) at ntdll.dll: (sig=9d04eb0a-a387-494f-bd81-ed062072b99c|2|ntdll.pdb, offset=16CEB) at ntdll.dll: (offset=16405) at ntdll.dll: (offset=16BF2) at kernel32.dll: (sig=c4d1d906-5632-4196-99a8-b2f25b62381d|2|kernel32.pdb, offset=1652D) at ntdll.dll: (sig=9d04eb0a-a387-494f-bd81-ed062072b99c|2|ntdll.pdb, offset=2C541)
One will also see the rebalance job running against owstimer for the crawl partitions.
04/16/2015 09:07:12.28 OWSTIMER.EXE (0x390C) 0x2164 SharePoint Foundation Monitoring nasq Medium Entering monitored scope (Timer Job Rebalance crawl store partitions for bf01da64-8d00-441e-add5-79ab3741a479). Parent No 4a85179d-420b-e0df-4af4-51bc9809ff48
Solution
In SharePoint 2013 however we need to balance the SharePoint Crawl Store links content every now and then. Adding a new Crawl store DB doesn’t help the situation as well. Crawl stores maintain an indicative threshold called Imbalance threshold and a method that replies with a Boolean as to whether or not the Crawl Store is Imbalanced.
$SSA= Get-SPEnterpriseSearchServiceApplication
$CrawlStorePartitionManager= new-Object Microsoft.Office.Server.Search.Administration.CrawlStorePartitionManager($SSA)
$CrawlStoreParititonManager.CrawlStoresAreUnbalanced()
True
The Search Service Application has a Method called CrawlStoresAreUnbalanced that gives us a boolean indication of whether or not the Crawl Store Links are full enough to do this rebalancing at all.
$CrawlStoreParititonManager.CrawlStoresAreUnbalanced()
False
Note: 1 Million is the default value for CrawlStoreImbalanceThreshold. In case they are unbalanced, we would know immediately.
It’s Property bag also has a Property called CrawlStoreImbalanceThreshold which can be obtained and modified using the GetProperty and SetProperty methods.
$SSA.GetProperty("CrawlStoreImbalanceThreshold")
$SSA.SetProperty("CrawlStoreImbalanceThreshold",10000)
Increasing Crawl stored for various hosts also helps to improve performance
Just increasing the number of crawl stores does not automatically balance the Links in the MSSCrawlURL across the hosts. Fortunately for us, Sharepoint gives us a Property to manage this as well. There is a job called “Rebalance crawl store partitions” that runs to perform this for us.
This runs automatically when some thresholds are attained called CrawlPartitionSplitThreshold. The default value for rebalancing to start is 10 Million items but it can be done earlier based on our needs.
When we make the above change, we can test this value against the registry key under CatalogNames with the same name CrawlPartitionSplitThreshold
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\15.0\Search\Applications\<SSAGUID>-crawl-0\CatalogNames
We can call a method called BeginCrawlStoreRebalancing from the CrawlStorePartitionManager Object we created some time back
$SSA= Get-SPEnterpriseSearchServiceApplication
$CrawlStorePartitionManager= new-Object Microsoft.Office.Server.Search.Administration.CrawlStorePartitionManager($SSA)
$SSA.GetProperty("CrawlPartitionSplitThreshold")
10000000
$SSA.SetProperty("CrawlPartitionSplitThreshold",50000)
$CrawlStorePartitionManager.BeginCrawlStoreRebalancing()
Note: The Balance job doesn’t run unless the threshold is crossed. Even if we force the rebalance to run, the actual rebalancing happens only if the threshold is met. So setting the value for CrawlPartitionSplitThreshold is critical.
We can achieve the same effect by forcing the Rebalance crawl store partitions job to run by following powershell commands.
(Get-SPTimerJob|where{$_.Name-match "Rebalance"} ).RunNow()