Problems and symptoms

I came across an interesting problem today, one which illustrates a point I’ve always tried to get across when discussing troubleshooting

The issue being reported may just be a symptom of the actual problem. Never assume you have the full picture.

Onto the problem :

This was a Citrix Presentation server 4.5 environment with Internet Explorer , Outlook and some industry specific applications. Virtual IPs were being used in a hosted environment to present IPs to Internet Explorer to allow external filtered internet access. Some users began to experience the following error and sessions were failing to launch.

So what could be the problem? The only symptom we had so far was that the users could not log on because the pool of virtual IPs was exhausted. Not much to go on so far so the best first step is check the VIP config as per the following article : http://support.citrix.com/article/CTX111898

To sum up the steps carried out here:

  1. Checked the VIPs were configured correctly in the Farm and assigned to the servers
  2. Checked the mentioned registry keys :
    HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\VIP\HookProcessesVIP
    HKLM\SOFTWARE\Citrix\CtxHook\AppInit_Dlls\VIPHook\<Process name>
  3. Check the viphook.dll was loaded into the processes by the use of Process monitor

Nothing was wrong however so it was time to look at the problem a different way. Looking at the Access management console for the affected servers and grouping the users by username displayed the following

 

Although the usernames are obscured each Red box in the image is a single user. As you can see each user is getting 2 IPs rather than one thus exhausting the IP pool faster. But why?

There is a common misconception with Virtual IPs that they only apply to the application they are assigned to in the Console. This isn’t the case, the IPs can be assigned to applications but in practice they are assigned to each session the user opens on the server. The configuration in the console merely allows that application to communicate with the Network using the Virtual IP address

Looking at the session list gave a clue as to where the problem was coming from. All the users who had the session sharing problem had Outlook open. Session depends on the applications being published with identical settings and the session sharing key matching. The key is composed of the following settings : Color depth, Screen Size, Access Control Filters (for SmartAccess), Sound, Drive Mapping, Printer Mapping

Checking the settings for Outlook confirmed that the session sharing problem was being caused by the Colour Depth being different to the other applications. Changing this back resolved the issue and the sessions started sharing again which eliminate the secondary issue of the virtual IP pool being exhausted and preventing logons.

By making no assumptions and treating all of the problems as a symptom what initially appeared to be a complicated and obtuse problem was actually rather simple to resolve.

Shane

Configuring application Crash dumps with Powershell

In a windows environment applications crash for many reasons and the best way to troubleshoot them is to collect a application crash dump. An application crash dump is a snapshot of what the application was doing when it crashed.

From Windows Vista and Windows Server 2008 onwards Microsoft introduced Windows Error Reporting or WER . This allows the server to be configured to automatically enable the generation and capture of Application Crash dumps. The configuration of this is discussed here . The main problem with the default configuration is the dump files are created and stored in the  %APPDATA%\crashdumps folder running the process which can make it awkward to collect dumps as they are spread all over the server. There are additional problems with this as but the main problem I always had with it was that its a simple task that is very repetitive but easy to do incorrectly. Therefore is it a perfect task to be automated.

I wrote this little script in Powershell

app_crashdump.ps1

This script does 3 things :

  1. Creates a folder to put the crash dumps in
  2. Gives the appropriate accounts access to this folder
  3. Configures the registry with the appropriate settings

Part 1 : Creating the folder

[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.VisualBasic') | Out-Null
$Folder=[Microsoft.VisualBasic.Interaction]::InputBox("Specify where to store crashdumps (not network location)", "Path", "c:\Crashdump")

New-Item $Folder -Type Directory -ErrorAction SilentlyContinue

### Verify the folder the user specified was a valid folder. Else failback to c:\Crashdump

$validatepath=Test-Path $Folder
	if ($validatepath -eq $false)
	{
	New-Item C:\Crashdump -Type Directory
	Set-Variable -Name Folder -value C:\Crashdump -Scope Script
	}

This piece of code asks the user for input as to where to put the folder, makes the folder. If it cannot make the folder the user suggested then it has a default path of c:\Crashdump

Part 2 : Specifying the permissions

$Acl= get-acl $Folder
$machinename = hostname
$querydomain = [System.DirectoryServices.ActiveDirectory.Domain]::GetCurrentDomain()
$domain = $querydomain.name

#Setting ACLs

$Acl.SetAccessRuleProtection($true, $false)
$acl.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule("Network","FullControl", "ContainerInherit, ObjectInherit", "None", "Allow")))
$acl.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule("Network Service","FullControl", "ContainerInherit, ObjectInherit", "None", "Allow")))
$acl.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule("Local Service","FullControl", "ContainerInherit, ObjectInherit", "None", "Allow")))
$acl.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule("System","FullControl", "ContainerInherit, ObjectInherit", "None", "Allow")))
$acl.AddAccessRule((New-Object System.Security.AccessControl.FileSystemAccessRule("Everyone","FullControl", "ContainerInherit, ObjectInherit", "None", "Allow")))

Set-Acl $folder $Acl

This code just defines some variables and assigns the following user accounts permissions to write to the folder

Network, Network Service, Local Service, System  and the domain account everyone. It then writes the ACL back to the folder.

Part 3: Actually configuring WER

$verifydumpkey = Test-Path "HKLM:\Software\Microsoft\windows\Windows Error Reporting\LocalDumps"

	if ($verifydumpkey -eq $false )
	{
	New-Item -Path "HKLM:\Software\Microsoft\windows\Windows Error Reporting\" -Name LocalDumps
	}

##### adding the values

$dumpkey = "HKLM:\Software\Microsoft\Windows\Windows Error Reporting\LocalDumps"

New-ItemProperty $dumpkey -Name "DumpFolder" -Value $Folder -PropertyType "ExpandString" -Force
New-ItemProperty $dumpkey -Name "DumpCount" -Value 10 -PropertyType "Dword" -Force
New-ItemProperty $dumpkey -Name "DumpType" -Value 2 -PropertyType "Dword" -Force

This Script checks if the Local Dumps Registry key exists, if it doesnt it creates it and then adds the necessary values. You have probably noticed a potential gotcha with powershell and registry entries. Powershell treats registry values as properties of the Key that they are in as discussed here

The full script can be downloaded from here . To run this script you need to allow unsigned scripts by running the following command from a administrative powershell window.

Set-ExecutionPolicy -ExecutionPolicy Unrestricted

alternatively you can sign the script, which isn’t very difficult but has a number of steps. The Technet Scripting Guy blog has a very good guide here and part 2 here .

I use this script in a couple of ways but mainly to sumplify the task of enabling users/admins with collecting application crash dumps for analysis.

Hope this helps

Shane