Background:
I had the privilege assisting a customer with some sync issues. We have a custom OU that we have customer security for user accounts that leave the company. We had some users that left the company. They were disabled, hidden from the GAL and moved the to the "bDelete" OU. They let the syncs and crawls run over night and in the morning they discovered that the user accounts were still showing up in search results. We looked in the User Profile Service Application and searched for the affected users. We still had user profiles for the user who left the company. These users also had an associated my site still. We performed a search in the user info table in the profile database and the user account was still present. The user wasn't removed from SharePoint. This turned our attention to the Syncs themselves. Looking in the miisclient.exe I noticed we haven't been successfully completing our syncs for a while now with the error "stopped-extension-dll-exception" with no details. As you can see this doesn't help us at all :(. Since the error seems to be with the FIM sync service and not SharePoint our attention turned now to the application event viewer for windows. At last some headway!!! In the application event logs for windows I saw this error:
Log Name: Application
Source: FIMSynchronizationService
Date: 7/10/2015 11:14:17 AM
Event ID: 6801
Task Category: (3)
Level: Error
Keywords: Classic
User: N/A
Computer: SP2010.Contoso.com
System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
at System.Net.WebClient.DownloadData(Uri address)
at Microsoft.Office.Server.UserProfiles.ManagementAgent.ProfileImportExportExtension.DownloadPictures(ProfileChangeData[] profiles)
at Microsoft.Office.Server.UserProfiles.ManagementAgent.ProfileImportExportExtension.Microsoft.MetadirectoryServices.IMAExtensibleFileImport.GenerateImportFile(String fileName, String connectTo, String user, String password, ConfigParameterCollection configParameters, Boolean fFullImport, TypeDescriptionCollection types, String& customData)
Forefront Identity Manager 4.0.2450.47
Cause:
During our Moss Import phase the FIM service is accessing the profile photo exactly as it's recorded… http://mysite.contoso.com/personal/User+Photos/Profile+Pictures/contoso_username_MThumb.jpg
Which means the FIM service will perform an HTTP GET command for each profile picture. This means a few things:
The Sync machine must have access to the My Site Host from a network perspective
The profile picture must be able to be displayed with a direct call via a browser
Combine this with the application event error and we have our issue for the failed Syncs. The question now is, we have thousands of users how do we determine which users are affected? The issue we have here is that the sync will fail as soon as it errors out on one profile photo. Even if it is successful for the first 1500 users as soon as it hits one bad profile picture the whole thing fails. Well we have a few options….
Resolution:
- We could look through the IIS logs on our WFE servers and attempt to find all the 401's for the profile photo.
- I'm assuming this would take forever
- select RecordID, Bdeleted, NTName, PreferredName, Email, PictureUrl from UserProfile_Full where PictureUrl like '%http%'
order by PictureURL
- You may find the culprit but you also may not. This also may be too time consuming.
- We could also run a script that would perform an HTTP GET command against every user profile and then log the results
- RUN AT YOUR OWN RISK
- See Scripts section below
- Analyze the log file produced and simply remove the affected users profile picture and run another sync
- Don't forget to run the update-spphotostore commands
- Update-SPProfilePhotoStore –CreateThumbnailsForImportedPhotos $true
Scripts:
#RUN AT YOUR OWN RISK
# You need to specify the mysite URL and path for log file. You run it like so:
# .\CheckProfilePicturesV7.ps1 -url http://mysite -filepath c:\logs -full
# When it's done, it will automatically open the log file ex:"c:\logs\ProblemPicLog.txt" in Notepad (if it finds problems)
# If it finds any profiles with problem picture URLs, it will prompt you to decide if
# you want to automatically remove the picture for those profiles. Just answer Y or N.
#IF YOU ANSWER YES, OR Y, THERE WILL BE DATA LOSS TO THAT PROFILE AS THE PICTURE WILL BE REMOVED!!!!!!!!!!!!!!!
Param(
[Parameter(Mandatory=$True,Position=1)]
[string]$URL,
[Parameter(Mandatory=$True)]
[string]$filePath,
[Parameter(Mandatory=$False)]
[switch]$Full
)
$logfile = "$filePath\ProblemPicLog.txt"
[array]$problemProfiles = @()
[array]$problemProfileReport = @()
[array]$ProfilesWithPics = @()
$site = Get-SPSite $url
if ($site) {Write-Host "Successfully obtained site reference"}
else {Write-Host "Failed to obtain site reference. Check the URL you entered"}
$serviceContext = Get-SPServiceContext($site)
if ($serviceContext) {Write-Host "Successfully obtained service context"}
else {Write-Host "Failed to obtain service context"}
$upm = new-object Microsoft.Office.Server.UserProfiles.UserProfileManager($serviceContext)
if ($upm) {Write-Host "Successfully obtained user profile manager"}
else {Write-Host "Failed to obtain user profile manager. Make sure the site you entered is associated with a UPA"}
# Go get all the profiles with a picture set and store their login name and picture URL in an array
$numpics = 0
Write-Host "Working on grabbing the profiles. This may take a while if you have a large number of profiles with pictures..." -ForegroundColor green
$profiles = $upm.GetEnumerator()
foreach($user in $profiles)
{
if ($user["PictureURL"].Value)
{$numpics++
$pic = $user["PictureURL"].Value
$ProfilesWithPics += ,@($user.MultiloginAccounts, $pic)
}
}
Write-Host "There appears to be " $numpics " profiles with pictures set" -ForegroundColor green
Function Hit-AllProfilePictures-Quick
{
Write-Host "Doing a QUICK check for all mthumb and lthumbs using Object Model." -ForegroundColor green
Write-Host "Notes:"
Write-Host "This will only work when run from the mysite farm."
Write-Host "It only checks to see if there is a file with matching name in the picture library. It checks for both the medium and large thumbnails."
Write-Host "It does not account for networking / environmetnal issues. To check for that, run with the -Full parameter."
$date = Get-Date -Format U
"Started Quick picture check at " + $date + " (UTC time)" | out-file $logfile -append
"===============================================" | out-file $logfile -append
$progressCounter = 0
$countProblemProfiles = 0
# Get all picture file names into an array so we can more easily access them
[array]$PictureFileNames = @()
$msh = get-spweb $upm.MySiteHostUrl
$list = $msh.lists["User Photos"]
foreach($listitem in $list.items)
{
$PictureFileNames += $listitem.name
}
#Loop through the array and do the file name comparison
for ($a=0; $a -lt $ProfilesWithPics.length; $a++)
{ $progressCounter++
Write-Progress -Activity "Checking Picture Thumbnails" -status "Checking $progressCounter" -percentComplete ($progressCounter / $ProfilesWithPics.length*100)
$UName = $ProfilesWithPics[$a][0]
$PUrl = $ProfilesWithPics[$a][1]
"PictureURL for " + $UName + " = " + $PUrl | out-file $logfile -append
$mthumbURL = $PUrl
$lthumbURL = $mthumbURL.Replace("MThumb", "LThumb")
# Check the file name array that I got from the picture library to see if there is a file in there with the same name
"Checking for Medium thumbnail " + $mthumbURL | out-file $logfile -append
Try
{
$pieces = $mthumbURL.split("/")
$numberofpieces=$pieces.Count
$Fname=$pieces[$NumberOfPieces-1]
if($PictureFileNames -contains $FName)
{"Found " +$Fname +" in the picture library" | out-file $logfile -append}
else{"ERROR!!! for user: " + $UName + " --- could not find " +$FName +" in the picture library" | out-file $logfile -append
$problemProfiles += $UName
$problemProfileReport += $UName, $mthumbURL
$countProblemProfiles++
}
}
catch [Exception]
{"ERROR!!! for user: " + $UName + " --- " + $_.Exception.Message | out-file $logfile -append
" " | out-file $logfile -append
$problemProfiles += $UName
$problemProfileReport += $UName, $mthumbURL, $_.Exception, " "
$countProblemProfiles++
}
"Checking for Large thumbnail " + $lthumbURL | out-file $logfile -append
Try
{$pieces = $lthumbURL.split("/")
$numberofpieces=$pieces.Count
$Fname=$pieces[$NumberOfPieces-1]
if($PictureFileNames -contains $Fname)
{"Found " +$Fname +" in the picture library" | out-file $logfile -append}
else{ "ERROR!!! for user: " + $UName + " --- could not find " +$Fname +" in the picture library" | out-file $logfile -append
$problemProfiles += $UName
$problemProfileReport += $UName, $lthumbURL
$countProblemProfiles++
}
}
catch [Exception]
{"ERROR!!! for user: " + $UName + " --- " + $_.Exception.Message | out-file $logfile -append
" " | out-file $logfile -append
$problemProfiles += $UName
$problemProfileReport += $UName, $lthumbURL, $_.Exception, " "
$countProblemProfiles++
}
" " | out-file $logfile -append
}
if ($problemProfileReport)
{"In summary, these are your problem profiles: " | out-file $logfile -append
"----------------------------------------------" | out-file $logfile -append
ForEach ($profileReport in $problemProfileReport)
{$profileReport | out-file $logfile -append
}
Write-Host "Done. " $countProblemProfiles "missing thumbnails were identified. The log file you specified in" $logfile "should open in Notepad. Please review the results before proceeding. A summary of all the errors is at the bottom." -foregroundcolor red
Write-Host "Total number of profiles with pictures processed = " $progressCounter -foregroundcolor green
Start-Process notepad -WindowStyle normal -FilePath $logfile
[String]$YN = Read-Host "Do you want to remove the picture for the problem profiles that were identified? Y\N"
if($YN -ieq "y" -or $YN -ieq "yes" )
{write-host "You chose Yes" -foregroundcolor red
write-host "Removing problematic profile picture links..." -foregroundcolor red
Remove-ProblemPictures
}
else{write-host "You chose No" -foregroundcolor red
write-host "No pictures were automatically removed. You should manually verify the profile picture URLs for the identified problem profiles are correct. The Sync service needs to be able to access these with an HTTP GET." -foregroundcolor red
}
}
else
{Write-Host "Good News! There were no problem profiles detected" -foregroundcolor green
Write-Host "Total number of profiles with pictures processed = " $progressCounter -foregroundcolor green
Write-Host "You can have a look at the log file you specified in" $logfile -foregroundcolor green
}
}
Function Hit-AllProfilePictures-Full
{
Write-Host "Doing a FULL check for all Mthumbs and Lthumnbs by doing an HTTP GET request" -ForegroundColor green
Write-Host "Working on it..." -ForegroundColor green
$date = Get-Date -Format U
"Started Full picture check at " + $date + " (UTC time)" | out-file $logfile -append
"===============================================" | out-file $logfile -append
$progressCounter = 0
$countProblemProfiles = 0
#Loop through the array and do the file name comparison
for ($a=0; $a -lt $ProfilesWithPics.length; $a++)
{ $progressCounter++
Write-Progress -Activity "Checking Picture Thumbnails" -status "Checking $progressCounter" -percentComplete ($progressCounter / $ProfilesWithPics.length*100)
$UName = $ProfilesWithPics[$a][0]
$PUrl = $ProfilesWithPics[$a][1]
$mthumbURL = $PUrl
$lthumbURL = $mthumbURL.Replace("MThumb", "LThumb")
# Do a web request to the mthumb URL we found for the profile
"Hitting picture " + $mthumbURL | out-file $logfile -append
$date = Get-Date -Format U
$date + " (UTC time)" | out-file $logfile -append
Try
{ $request = [System.Net.WebRequest]::Create($mthumbURL)
$request.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
$request.proxy = [System.Net.WebRequest]::DefaultWebProxy
$request.ContentType = "application/x-www-form-urlencoded"
$request.Method = "GET"
"Response = " + $request.GetResponse().StatusCode | out-file $logfile -append
" " | out-file $logfile -append
}
catch [Exception]
{ "ERROR!!! for user: " + $UName + " --- " + $_.Exception.Message | out-file $logfile -append
" " | out-file $logfile -append
$problemProfiles += $UName
$problemProfileReport += $UName, $mthumbURL, $_.Exception, " "
$countProblemProfiles++
}
# Now do a web request to the Lthumb URL we found for the profile
"Hitting picture " + $LthumbURL | out-file $logfile -append
$date = Get-Date -Format U
$date + " (UTC time)" | out-file $logfile -append
Try
{ $request = [System.Net.WebRequest]::Create($LthumbURL)
$request.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
$request.proxy = [System.Net.WebRequest]::DefaultWebProxy
$request.ContentType = "application/x-www-form-urlencoded"
$request.Method = "GET"
"Response = " + $request.GetResponse().StatusCode | out-file $logfile -append
" " | out-file $logfile -append
}
catch [Exception]
{ "ERROR!!! for user: " + $UName + " --- " + $_.Exception.Message | out-file $logfile -append
" " | out-file $logfile -append
$problemProfiles += $UName
$problemProfileReport += $UName, $LthumbURL, $_.Exception, " "
$countProblemProfiles++
}
}
if ($problemProfileReport)
{"In summary, these are your problem profiles: " | out-file $logfile -append
"----------------------------------------------" | out-file $logfile -append
ForEach ($profileReport in $problemProfileReport)
{$profileReport | out-file $logfile -append
}
Write-Host "Done. " $countProblemProfiles "missing thumbnails were identified. The log file you specified in" $logfile "should open in Notepad. Please review the results before proceeding. A summary of all the errors is at the bottom." -foregroundcolor red
Write-Host "Total number of profiles with pictures processed = " $progressCounter -foregroundcolor green
Start-Process notepad -WindowStyle normal -FilePath $logfile
[String]$YN = Read-Host "Want to remove the picture for the problem profiles that were identified? Y\N"
if($YN -ieq "y" -or $YN -ieq "yes" )
{write-host "You chose Yes" -foregroundcolor red
write-host "Removing problematic profile picture links..." -foregroundcolor red
Remove-ProblemPictures
}
else{write-host "You chose No" -foregroundcolor red
write-host "No pictures were automatically removed. You should manually verify the profile picture URLs for the identified problem profiles are correct. The Sync service needs to be able to access these with an HTTP GET." -foregroundcolor red
}
}
else
{Write-Host "Good News! There were no problem profiles detected." -foregroundcolor green
Write-Host "Total number of profiles with pictures processed = " $progressCounter -foregroundcolor green
Write-Host "You can have a look at the log file you specified in" $logfile -foregroundcolor green
}
}
Function Remove-ProblemPictures()
{if ($problemprofiles)
{
ForEach ($problemprofile in $problemProfiles)
{write-host "Removing the picture link for profile: " $problemprofile
$userProfile = $upm.GetUserProfile($problemprofile)
$ProfilePictureUrl = ""
$userProfile["PictureURL"].Value = $ProfilePictureUrl
$userProfile.Commit()
}
Write-Host "Done. The broken picture links should be cleared up. You can wait few minutes (it takes a minute or two to remove the values from cache) and then try running this script again, or just run a Full Sync." -foregroundcolor green
}
Else{Write-Host "No problem picture links to remove!" -foregroundcolor red}
}
If($Full -eq $true)
{Hit-AllProfilePictures-Full($url)}
Else{Hit-AllProfilePictures-Quick($url)}
More Information:
All Scripts are provided as is. Use at your Own Risk