Avantgarde Technologies

<a href="http://www.avantgardetechnologies.com.au">Avantgarde Technologies</a>
Perth's IT Experts

Thursday, October 29, 2015

Active Directory Issues - Network Drives Not Mapping

A customer of mine raised an issue in regards network drives not being mapped for users.  This includes drives mapped via Group Policy and Home Drives mapped via the NT4 Home Drive option of the Active Directory user account.

When users attempt to navigate to the UNC paths manually or map a drive manually it works as expected, however mapping network drives automatically upon logon simply did not work.

Also users were unable to navigate to the domain name "\\domain.local".  However users could navigate to "\\domain.local\netlogon" and "\\domain.local\sysvol".  Navigating to the domain root resulted in this error being generated:

\\domain.local is not accessible.  You might not have permissions to use this network resource.  Contact the administrator of this server to find out if you have access permissions.

Logon Failure: The target account name is incorrect.

This issue with not being able to navigate to the domain root UNC share occurred on all member workstations and servers throughout the organisation.  Domain Controllers were not effected by the issue.

The following three event logs were also found throughout the SYSTEM event log on all client workstations throughout the companies domain.

Log Name:      System
Source:        NETLOGON
Date:          26/10/2015 7:56:15 AM
Event ID:      5719
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      COMPUTER.domain.local
This computer was not able to set up a secure session with a domain controller in domain DOMAIN due to the following: 
There are currently no logon servers available to service the logon request. 
This may lead to authentication problems. Make sure that this computer is connected to the network. If the problem persists, please contact your domain administrator.  

Log Name:      System
Source:        Microsoft-Windows-Security-Kerberos
Date:          23/10/2015 4:02:46 PM
Event ID:      4
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      COMPUTER.domain.local
The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server candc1$. The target name used was cifs/domain.local. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (DOMAIN.LOCAL) is different from the client domain (DOMAIN.LOCAL), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

Log Name:      System
Source:        Microsoft-Windows-GroupPolicy
Date:          28/10/2015 6:18:58 PM
Event ID:      1006
Task Category: None
Level:         Error
User:          DOMAIN\UserAccount
Computer:      COMPUTER.domain.local
The processing of Group Policy failed. Windows could not authenticate to the Active Directory service on a domain controller. (LDAP Bind function call failed). Look in the details tab for error code and description.

Some computers were also receiving:

The processing of Group Policy failed. Windows attempted to read the file \\domain.local\sysvol\domain.local\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\gpt.ini from a domain controller and was not successful. Group Policy settings may not be applied until this event is resolved. This issue may be transient and could be caused by one or more of the following: 
a) Name Resolution/Network Connectivity to the current domain controller. 
b) File Replication Service Latency (a file created on another domain controller has not replicated to the current domain controller). 
c) The Distributed File System (DFS) client has been disabled.

This GUID referenced the Default Domain Policy (the first policy in the domain).  There was nothing wrong with the Default Domain Policy on the customers network, policy was simply not applying to domain members due an issue in Active Directory.

One issue which I focused on was the Kerberos error "KRB_AP_ERR_MODIFIED".  I have seen this issue before when the same SPN was registered on at least two accounts. For example, a SPN was registered on two accounts: A and B. What happens is that KDC will generate a service ticket that may be encrypted with password of account A. Then, when the client sends that ticket to the service during authentication, the service may try to decrypt this using account B.

I searched the customers domain for duplicated SPN's using "setspn.exe -X" and found some however they were not related to the error received.  Also I have never seen the "KRB_AP_ERR_MODIFIED" error generated on EVERY domain member workstation/server in an Active Directory environment.

Looking further into the SPN's we decided to dump the entire domain with every object/attribute to a text file using:

ldifde -f out.txt -d dc=domain,dc=local

Generally SPN against a user account reference a member server on the network running a particular service account such as SQL  However as this issue was affecting the entire domain and KRB_AP_ERR_MODIFIED refers to duplicated SPN records, we looked to see if there were any SPNs set at domain level on an account by searching our output from ldifde for "host/domain.local".

We found an SPN set to the root of the domain from the search results.  Important content from output below is blurred to protect the privacy of the customer.

We went and removed the incorrectly setup SPN record from the problematic service account svc_adfs using Active Directory Users and Computers with Advanced Features turned on then forced replication with "repadmin /syncall /APeD".

After removing the incorrectly configured SPN, we purged the kerberos tickets off a workstation then attempted to start explorer at the root of the domain "\\domain.local".  We were able to navigate to this share successfully.

The issues with network drives not mapping on logon were also resolved.

The SVC_ADFS account was created as part of an AD FS deployment for federation with applications and Microsoft Cloud Services.  AD FS backend roll was installed on two corporate domain controllers and two proxy servers were deployed in a DMZ setup to process the authentication requests from external services.  This is the Microsoft Best Practice for corporate organisations under 1000 seats as it reduces the amount of servers required and provides high redundancy by leveraging NLB on both the backend and frontend AD FS servers as per:


I went over the engineers build documentation who was in charge of implementing AD FS and could not see how the SPN was set, he did not manually set it.

Hope this post helps someone who experiences this same issue.

Monday, October 26, 2015

Enable Firewall Logging on Windows

Are your packets being dropped by Windows Firewall?  Want an insight into what is going on?  Simply open local group policy on a workstation / server (gpedit.msc) or configure a GPO in Group Policy Management Console (GPMC).  Under Windows Firewall with Advanced Security, go to the general properties.  Select the profile --> Logging and enable Logging on the set profile.  The log file by default goes to:


Very handy for troubleshooting.

Exchange 2013 POP3 Proxy Inactive

A customer complained that POP3 was no longer working.  After looking into this, it turned out that PopProxy was Inactive on the Exchange 2013 server.  As to why it was inactive is unknown.

To start the PopProxy was challanging, generally you change ServerComponentState using the maintenance requester for most components.  However running the following command did nothing:

Set-ServerComponentState -State Active -Requester Maintenance -Component PopProxy -Identity AB-EXCH-01

To start the PopProxy component, I needed to use the Exchange 2013 Health API as a requester.

Set-ServerComponentState -State Active -Requester HealthAPI -Component PopProxy -Identity AB-EXCH-01

As shown below:

Exchange 2013 421 4.3.2 Service not active

A customer of mine upgraded an Exchange 2013 cluster node from Exchange 2013 CU7 to Exchange 2010 CU10.  After the upgrade, emails failed to come in on the cluster node with the following SMTP error being generated "421 4.3.2 Service not active".

This can be reproduced by simply telneting the faulty Exchange 2013 server.  After entering MAIL FROM: into the SMTP syntax, the error occurs and is shown numerous times throughout the receive connector protocol logs on the frontend transport stack.

After further investigation I found out that majority of the Server Components for the faulty Exchange Server were in an inactive state.

To bring the server back to an active state, ServerWideOffline was set to Active which resumes all services using a requester of Maintenance.  This was done with the following command:

Set-ServerComponentState -State Active -Requester Maintenance -Identity AB-EXCH-02 -Component ServerWideOffline

Note: ForwardSyncDaemon and ProvisioningRps is Inactive by default.

After running this command all Exchange 2013 components were back to an active state apart from components disabled by default.

Exchange 2013 Cluster Issues 0x80071736

A customer of mine went upgraded one node of a two node DAG from Exchange 2013 CU7 to Exchange 2013 CU10.  After installing the update on the first cluster node, they contacted us complaining complaining that the DAG was no longer available online in Failover Cluster Manager.

The following error was presented when attempting to bring the cluster online:

Failed to bring the resource 'Cluster Name' online.
Error code 0x80071736
The resource failed to come online due to the failure of one or more provider resources.

The resource it was complaining about was the Cluster IP address was unavailable (

In addition to this error, the following errors were logged in event viewer on a regular basis.

Log Name:      System
Source:        Microsoft-Windows-DistributedCOM
Date:          26/10/2015 8:45:01 AM
Event ID:      10028
Task Category: None
Level:         Error
Keywords:      Classic
User:          DOMAIN\administrator
Computer:      Exchange1.domain.local

DCOM was unable to communicate with the computer DAG1.domain.local using any of the configured protocols; requested by PID 6bc4 (C:\Windows\system32\ServerManager.exe).

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          26/10/2015 9:58:05 AM
Event ID:      1223
Task Category: IP Address Resource
Level:         Error
User:          SYSTEM
Computer:      AB-EXCH-01.domain.local
Cluster IP address resource 'Cluster IP Address' cannot be brought online because the cluster network 'Cluster Network 1' is not configured to allow client access. Please use the Failover Cluster Manager snap-in to check the configured properties of the cluster network.

These cluster nodes both had two interfaces:
  • Mapi Interface
  • Replication Interface

After patching one of the Exchange 2013 servers the MapiDagNetwork got disabled on the with IgnoreNetwork set to true on the DagGroupNetwork.  As this DAGNetwork contained the DAG cluster IP address (within the subnet), it forced the cluster offline.  To re-enable the cluster we simply needed to set IgnoreNetwork to false.

After setting IgnoreNetwork to false, we were able to manually start the DAG in FailoverManager by right clicking and clicking Bring Online.

Monday, October 12, 2015

Removing Large Amounts of Spam from Exchange Server Queue Database

A customer running Exchange 2010 experienced a large number of spam emails in the submission queue (over 80,000).

All the Spam Emails started with "ID#ALERT#", as a result we ran the following command to clean up any emails with "ID#ALERT#" in the subject.

Get-Message -ResultSize unlimited | Where-Object {$_.Subject -match "ID#ALERT#"} | Remove-Message

Due to the significant amount of spam, the command failed to remove the messages and just hung for hours.  However we could remove messages individually.

We could however list the emails with the following command:

Get-Message -ResultSize unlimited | Where-Object {$_.Subject -match "ID#ALERT#"}

This gave us an idea to write a foreach command to remove each email individually from a CSV file.  First we needed a detailed list with the identity of each spam email so we ran the following command:

Get-Message -ResultSize unlimited | Where-Object {$_.Subject -match "ID#ALERT#"} | select Identity > C:\output.txt

We then formatted the CSV file to ensure we specified a name for the Identity column.  As you see we added "Identity" to the top of the file.

We then ran the Remove-Message command multiple times for each Identity in the CSV file using the following command:

Import-Csv "C:\output.txt" | ForEach-Object {Remove-Message -Identity $_.Identity -confirm:$false}

This ran the Remove-Message command over 80,000 times for each message in the queue and was able to clean it up successfully.

Sunday, October 4, 2015

Websense Appliance Services Not Starting after Hard Shutdown

When a Websense v5000 or v10000 appliance is forcefully shutdown due to power loss or hard system failure, upon restart the Filtering service, Policy service, User service and Usage monitor can fail to start in the Websense Appliance Manager portal.  This is an issue I have seen more then once and as a result decided to do a write-up.

This issue occurs due to a number of temporary files which are not cleaned up (a process that occurs during a graceful shutdown).  To remove these files manually, we must connect to the appliance using an SSH shell session.

To connect to a shell session you need to login to Websense Appliance Manager first then under Administration --> Toolbox, click Technical Support Tools and find the passcode under "Websense Remote Access".  This is the password used for SSH which is randomly generated by the appliance.

Next login to the IP address of the Websense Appliance (the same IP you used for the Appliance Manager web interface).
The username is "websense-ts" and the password is the one obtained above.
Navigate to /opt/Websense/bin and remove all temporary p12 files which were not deleted due to an incorrect shutdown.
rm -f *.p12
 Also remove the journal.dat file under the /opt/Websense/bin using the following command:
rm -f 'journal.dat

After the temporary files have been removed, restart all websense services.  This can be done from the Websense Appliance Manager website or from the shell by running the following command:

./WebsenseAdmin restart

Now all the Websense services on the appliance have returned to a running healthy state.


Tuesday, September 29, 2015

VawTrak Trojan

Today I was diagnosing why a clients Internet was running so slow.  After tracing the traffic I found it was one Windows 7 PC which was infected with a virus.  The following processes were running on the machine all communicating with various Internet IP addresses.
  • conhost.exe
  • cmd.exe
  • ctfmon.exe
  • dllhost.exe
  • msiexec.exe
  • notepad.exe
  • presentationhost.exe
Note: Use Windows Resource Monitor and navigate to the Network tab to find out which processes are communicating with Internet resources.

When killing one of these processes, they would simply respawn.  The computer was also running very slow and sluggish with web browsers and windows explorer constantly hanging and freezing.
These symptoms are related to Trojan.VawTrak which the computer was infected with.  Trojan.VawTrak copies it self into C:\ProgramData and spawns these processes with its malicious code.
Trojan.VawTrak can be cleaned up with Malware Bytes or manually.
Trojan.VawTrak is a virus you definitely want to get rid of as it is designed to steal online banking information.  Some of the common tasks it performs are:
  • Disables antivirus protection.
  • Communicates with remote C&C servers – executes commands from a remote server, sends stolen information, downloads new versions of itself and web-injection frameworks.
  • Hooks standard API functions, injects itself into new processes.
  • Steals passwords, digital certificates, browser history, and cookies.
  • Logs keystrokes.
  • Takes screenshots of desktop or particular windows with highlighted mouse clicks.
  • Captures user actions on desktop in an AVI video.
  • Opens a VNC11 (Virtual Network Computing) channel for a remote control of the infected machine.
  • Creates a SOCKS12 proxy server for communication through the victim's computer.
  • Changes or deletes browser settings (e.g. disable Firefox SPDY13) and history. Vawtrak supports three major browsers to operate in – Internet Explorer, Firefox, and Chrome. It also supports password stealing from the other browsers.
  • Modifies browser communication with a web server.
  • Stores internal settings into encrypted registry keys.
Due to the severity of this Trojan and the rate it is spreading, AVG has done a detailed writeup which is available here:


Thursday, September 24, 2015

Quick access in Windows 10 with Direct Access

Quick access is a feature in Windows 10 which lets you quickly view recently opened documents and folders.  This is handy for users to gain access to files they access on a regular basis.

On the Avantgarde Technologies network all our employees use Direct Access to provide seamless connectivity back to resources in the office.  We found on links with poor bandwidth and high latency, Quick access causes performance issues and causes Windows Explorer to hang up to 10 seconds every time a user tries to save a file or open a new Explorer window.
To ensure employees which are outside the office are not affected with performance issues, we disabled this technology on all our Windows 10 workstations.  The two registry keys you want to modify are under this location:
Both are REG_DWORD values.
Simply modify both these registry keys to 0.
 Deploy these registry keys to all your users using Group Policy Preferences.

After making this change, Windows Explorer will be a lot more snappy for remote users connecting via Direct Access or another VPN technology.

Monday, September 21, 2015

Remove all Printers Deployed from a specific Print Server with Powershell

I had a customer which had deployed printers from a legacy print server utilising scripts.  They have recently built a new 2012 R2 print server where they deployed the printers utilising Print Management Console and Group Policy.

All printers were redeployed from the new print server.

The customer however had a number of printers setup on workstations still pointing to the legacy print server.  As such they wanted to remove all printers deployed from the hostname of the legacy print server.

The following PowerShell script achieves this and can be easily deployed with Group Policy.  Simply replace "PRINTSERVER" with the name of your print server and then deploy the PowerShell script.

$PrintServer = "\\PRINTSERVER"
$Printers = Get-WmiObject -Class Win32_Printer
ForEach ($Printer in $Printers) {
If ($Printer.SystemName -like "$PrintServer") {
(New-Object -ComObject WScript.Network).RemovePrinterConnection($($Printer.Name))


Tuesday, September 15, 2015

Regaining Access to an SQL Instance

After a previous employee left an organisation, no one had access to an SQL Instance and the SA Password was unknown.  In this article I will show you how to regain access to an SQL Instance.

This process was performed in SQL Server 2012 Enterprise Edition by booting the SQL server into Single User Mode.

First stop the SQL service for which we need to recover the password.

Then start the service by entering a Start parameter of "-m" in the services window in Control Panel.

Next connect to the instance with SQLCMD.exe -S "SERVERNAME\Instance".

To grant sysadmin to a user or entire group such as Domain Admins, run the following command:

EXEC sp_addsrvrolemember 'DOMAIN\Domain Admins', 'sysadmin'; 

Next run "quit" to close SQLCMD.

After this, remove the -m from the SQL Instance and start the instance normally.  Now anyone in the Domain Admins group will have sysadmin rights to the instance.  Login with a Domain Admin account and reset the SA password (provided your Instance is setup for Mixed authentication).

Enter the new password and click OK.

Thursday, September 10, 2015

Howto Decrypting an Active Directory Password

In this post I will show a tool which makes decrypting Active Directory passwords easy.  It is important to note decrypting highly secure passwords takes a long time and is not always achievable within a reasonable period of time based on the complexity of the password however I have had success recently using this product.

Find a Windows Computer where the user has logged into recently and has their password cached.  Next obtain the Network Password Recovery Wizard (NPRW) tool from:


After the tool simply use the GUI for performing the password encryption.  I found this video very helpful.


Wednesday, September 9, 2015

The SMTP availability of the Receive connector Default was low

In Exchange 2013 CU9 you may start seeing the following error message more often.

Log Name:      Application
Source:        MSExchangeTransport
Date:          10/09/2015 12:15:21 PM
Event ID:      1040
Task Category: SmtpReceive
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CCEX2.toph.local
The SMTP availability of the Receive connector Default was low (98 percent) in the last 15 minutes.

This is due to a new Transport Agent introduced by CU9 called the System Probe Drop Smtp Agent - undocumented, but apparently responsible for dropping probe emails - which does not "drop the email" in a traditional sense, but removes all email recipients instead. Alas, this method of stopping an email from getting delivered has an adverse effect, namely the email still gets passed down the pipeline, but now with the recipient information destroyed.

This results in the email delivery failing only for the System Mailbox "probe drop" emails as shown in the following screenshot.

As a result, Event ID 1040 for the MSExchangeTransport service is logged saying that a small percentage of emails are failing.  These are generally the probe messages from Exchange 2013 CU9.

I do recommend you run the above Get-MessageTrackingLog command with an EventId of FAIL to determine that there is no other emails which have failed. 

Monday, August 17, 2015

Enable Split Tunneling on Windows 10 VPN Connections

In previous versions of Windows Server, Split Tunneling was enabled by removing the default gateway from the IPv4 settings under the properties of a Windows PPTP, L2TP or SSTP VPN connection.  This was done on the Networking tab and selecting Properties on the Internet Protocol 4 (TCP/IPv4) settings.

In Windows 10 if you click properties on the Internet Protocol Version 4 (TCP/IPv4) settings, nothing happens the button has no code behind it.
On Windows 10, to enable Split Tunneling this must now be done with PowerShell. 
Set-VpnConnection "VPN Connection Name" -SplitTunneling $true
 After enabling the VPNConnection for Split Tunneling this achieves the same affect as the "Use as Default Gateway" button from the IPv4 properties dialog box.

Friday, August 14, 2015

MAPI NSPI Issues with Exchange 2013

A customer running two Exchange 2013 CU9 multi-role servers in a cluster with an F5 load balancer contacted me this morning reporting issues where they could not create new Outlook Profiles for users.  Users with existing Outlook profiles were not affected by the issue.

Users primarily connect with MAPI over HTTP which is enabled on the Exchange Organization configuration.

When a user attempted to create a new profile, they would receive the following error:

The connection to Microsoft Exchange is unavailable.  Outlook must be online or connected to complete this action.

When running the Microsoft Exchange Remote Connectivity Analyzer against the Exchange organisation, the following errors were captured:

Testing MAPI over HTTP connectivity to server webmail.company.com
MAPI over HTTP connectivity failed.
  Additional Details
  HTTP Response Headers:
request-id: 1f79419f-0c42-4a38-bf07-99b1c6882928
Set-Cookie: ClientId=MUZOXYKGXN9GLA; expires=Sat, 13-Aug-2016 02:50:02 GMT; path=/; HttpOnly
Server: Microsoft-IIS/8.5
WWW-Authenticate: Negotiate,NTLM
X-Powered-By: ASP.NET
Date: Fri, 14 Aug 2015 02:50:02 GMT
Content-Length: 0
Elapsed Time: 7764 ms. 

Testing the MAPI Address Book endpoint on the Exchange server.
An error occurred while testing the address book endpoint.
  Additional Details
  Elapsed Time: 5784 ms. 

  Test Steps
  Testing the address book "Check Name" operation for user TestUser@company.com against server webmail.company.com.
  An error occurred while attempting to resolve the name.
  Additional Details
  A protocol layer error occured. HttpStatusCode: 500
FailureLID: 47372

###### REQUEST [2015-08-14T02:50:03.1064218Z] ######

POST /mapi/nspi/?mailboxId=e110343c-c351-4dac-b066-3b552417a51b@company.com HTTP/1.1
Content-Type: application/octet-stream 
User-Agent: MapiHttpClient 
X-RequestId: 57959cb1-fd5d-4ae1-ac55-4494f0a61748:1 
X-ClientInfo: 6016fb10-ca94-4a67-847c-549dd99bacb8:1 
X-ClientApplication: MapiHttpClient/ 
X-RequestType: Bind 
Authorization: Negotiate [truncated] 
Host: webmail.company.com 
Content-Length: 45 

--- REQUEST BODY [+0:05.159] ---
..[BODY SIZE: 45]

--- REQUEST SENT [+0:05.159] ---

###### RESPONSE [+0:05.781] ######

HTTP/1.1 500 Internal Server Error
request-id: 9653be79-d895-4cf3-8169-98ceeb218197 
X-CalculatedBETarget: Server1.company.local 
X-DiagInfo: Server1 
X-BEServer: Server1 
X-FailureContext: BackEnd;500;NTAw;U3lzdGVtLk5ldC5XZWJFeGNlcHRpb246IFRoZSByZW1vdGUgc2VydmVyIHJldHVybmVkIGFuIGVycm9yOiAoNTAwKSBJbnRlcm5hbCBTZXJ2ZXIgRXJyb3IuDQogICBhdCBTeXN0ZW0uTmV0Lkh0dHBXZWJSZXF1ZXN0LkVuZEdldFJlc3BvbnNlKElBc3luY1Jlc3VsdCBhc3luY1Jlc3VsdCkNCiAgIGF0IE1pY3Jvc29mdC5FeGNoYW5nZS5IdHRwUHJveHkuUHJveHlSZXF1ZXN0SGFuZGxlci48PmNfX0Rpc3BsYXlDbGFzczJjLjxPblJlc3BvbnNlUmVhZHk+Yl9fMmIoKQ==;;ProtocolError; 
Persistent-Auth: true 
X-FEServer: Server1 
Transfer-Encoding: chunked 
Cache-Control: private 
Content-Type: text/html; charset=utf-8 
Date: Fri, 14 Aug 2015 02:50:08 GMT 
Set-Cookie: X-BackEndCookie=e110343c-c351-4dac-b066-3b552417a51b=u56Lnp2ejJqBz5nPyM2cxpnSncebmdLLyZue0sfJnJ7SnM2Znc7Iys7LnJnMgYHNz87K0s/G0s7Mq8/NxcrPxc/H; expires=Sun, 13-Sep-2015 02:50:08 GMT; path=/mapi; secure; HttpOnly 
Server: Microsoft-IIS/8.5 
X-AspNet-Version: 4.0.30319 
X-Powered-By: ASP.NET 

Also on only on Server1, the following error was experienced in the event log.

Log Name:      Application
Source:        ASP.NET 4.0.30319.0
Date:          14/08/2015 5:54:03 PM
Event ID:      1309
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Server1
Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 14/08/2015 5:54:03 PM 
Event time (UTC): 14/08/2015 9:54:03 AM 
Event ID: 6e8a15487387414b9279078f9f6aba51 
Event sequence: 2 
Event occurrence: 1 
Event detail code: 0 

Application information: 
    Application domain: /LM/W3SVC/2/ROOT/mapi/nspi-3615-130840196323691514 
    Trust level: Full 
    Application Virtual Path: /mapi/nspi 
    Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\mapi\nspi\ 
    Machine name: Server1

Process information: 
    Process ID: 13528 
    Process name: w3wp.exe 
    Account name: NT AUTHORITY\SYSTEM 

Exception information: 
    Exception type: HttpException 
    Exception message: RpcObjectSetType
   at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
   at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
   at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

   at Microsoft.Exchange.Rpc.RpcServerBase.ThrowRpcException(String message, Int32 rpcStatus)
   at Microsoft.Exchange.Rpc.ProcessAccess.ProcessAccessRpcServer.RegisterInterface(Void* ifSpec, ValueType mgrTypeGuid, _GUID* pMgrTypeUuid, Void* pMgrEpv, UInt32 flags, UInt32 maxCalls)
   at Microsoft.Exchange.Rpc.RpcServerBase.RegisterServer(Type type, ObjectSecurity sd, UInt32 desiredAccess, ValueType mgrTypeGuid, Void* mgrEpv, String annotation, Boolean isLocalOnly, Boolean autoListen, UInt32 maxCalls)
   at Microsoft.Exchange.Data.ApplicationLogic.ProcessAccessManager.RegisterComponent(IDiagnosable diagnosable)

Request information: 
    Request URL: https://localhost:444/mapi/nspi 
    Request path: /mapi/nspi 
    User host address: ::1 
    Is authenticated: False 
    Authentication Type:  
    Thread account name: NT AUTHORITY\SYSTEM 

Thread information: 
    Thread ID: 81 
    Thread account name: NT AUTHORITY\SYSTEM 
    Is impersonating: False 
    Stack trace:    at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
   at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
   at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

After further investigation, we discovered that only mailboxes which resided on an Active database on Server1 experienced the issue creating Outlook profiles, the same server which experienced EventID 1309.  After failing mailbox databases over to Server2, users were able to login to Outlook without issues.

After this we identified the issue was only present with Server1.

On Server1 we tested the NSPI website by navigating to the Exchange backend in IIS Manager and selecting Browse:444 (https).

The browse failed on Server1 with an "unhandled exception error".  This was aligned with the HTTP 500 exception shown in the Event Logs on Server1 and the Exchange Remote Connectivity Analyzer.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: Microsoft.Exchange.Rpc.RpcException: RpcObjectSetType

Source Error: 
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. 

On Server2 when repeat the same process.

On Server2 it prompted as authentication as normal.

This shows the MAPI/NSPI results as expected.


We rebooted Exchange Server 1 and after a reboot, it started responding to MAPI/NSPI requests as expected.  We believe it was a side affect after the CU9 upgrade, however will monitor the situation and update this blog thread in the event the issue reoccurs.

Wednesday, July 29, 2015

DNS Host A Record Being Deleted on Server Reboot

Our company Avantgarde Technologies was performing a domain controller upgrade for a client from Server 2008 R2 domain controllers to Server 2012 R2.  After deploying the new domain controllers we went through the process of updating the DNS Client records on member servers which had these addresses statically assigned.

After updating a few DNS client settings on member servers, our customer complained that when they rebooted these servers, the member servers Host A record in DNS disappeared.  This issue only occurred with 2008 and 2008 R2 member servers, 2012 servers were not affected.

After looking into this issue, we found that there is a bug with the DNS client in Server 2008 and Server 2008 R2.  Windows Vista and Windows 7 clients are also effected.  This bug is documented under KB2520155 and Microsoft has released a hotfix.

The issue is simple, as per the Microsoft KB article:

"This issue occurs because of an issue in the DNS Client service. When the DNS server configuration information is changed on a client, the DNS Client service deletes the DNS host record of the client from the old DNS server and then adds it to the new DNS server. Because the DNS record is present on the new server that is a part of the same domain, the record is not updated. However, the old DNS server replicates the deletion operation to the new DNS server and to other DNS servers. Therefore, the new DNS server deletes the record, and the record is deleted across the domain."

To get the member server to re-register itself in DNS, you must perform one of the following actions:
  • Restart the computer.
  • Restart the DNS Client service.
  • Run the ipconfig /registerdns command.
After the dynamic registration is re-created the record will not delete unless the DNS client addresses are again changed on the server.

If you require the ability to change the DNS client addresses on member servers on a regular basis, I strongly recommend installing the patch which is available from KB2520155.  In this instance we did not install the patch as this customer does not change the DNS client addresses on member servers on a regular basis.

For this customer we did not install the patch, we simply performed an ipconfig /registerdns on any servers 2008 R2 servers after their DNS client was updated and server was rebooted.

Tuesday, July 7, 2015

The folders cannot be queried. Element not found.

In in the process of setting up a new DFS Namespace server at a remote site for a customer, I faced an issue where I was unable to utilize the DFS management console to add the remote server as a namespace server.  The error received in the DFS Management console was:

The folders cannot be queried. Element not found

After some investigation it turns out that the error was due to a setting the customer had configured:

"Exclude targets outside of the client's site".

After changing the Ordering method back to "Lowest Cost" and waiting 120 minutes for AD Replication and the DFS Namespace to refresh, we were able to configure the DFS Namespace server at the remote site.

Tuesday, June 30, 2015

Newly Built Hyper-V 2012 R2 Server Blue Screening

We experienced an issue with a newly built Hyper-V 2012 R2 server running on Windows Server 2012 R2 Datacentre edition with all the latest critical and security and critical as of 23/06/2015.  The server was built, setup with a number of virtual machines and ran without issues for a period of approximately 1 month.  After approximately 1 month, the server started blue screening on a regular basis, no changes had been made to the Hyper-V Host.

The Hyper-V Host was setup on a Dell PowerEdge R730XD with 192GB of memory.


The cause of the crash as per the original error was from the NT Kernel (ntoskrnl.exe).

We opened a support case with Microsoft to get the full memory dump analysed.  After the memory dump was analysed, we were advised it was a known bug in Windows Server 2012 R2 identified 14th of April 2015.  They pointed us at KB3055343 which resolves a issue with Windows that can cause inconsistent network interface data on the system.

After downloading and installing KB3055343 the issue was resolved.


Tuesday, June 23, 2015

Legacy File Servers with User Profile Disks

I have been involved in a number of Remote Desktop Services 2012 R2 deployments and from my experiences, I recommend to my clients to only utilise Server 2012 R2 file servers.  This is due to two issues which I have experienced with legacy file servers such as Windows Server 2008 R2:
  • Problems using DFS Namespaces
  • Temporary Profiles
DFS Namespaces
DFS Namespaces are supported with User Profile Disks however from my experience both the file server and DFS namespace servers must be Server 2012 R2.  If they are not Server 2012 R2 you will receive this error:
Unable to enable user disks on UserVHDShare. Could not create template VHD.  Error Message: The network location "\\domain.local\namespace\foldertarget" is not available.
 I have replicated this twice in my lab and at a production site when dealing with 2008 R2 file servers.  If you utilise 2012 R2 file servers with 2012 R2 DFS namespace servers you will not experience this error.

Temporary Profiles

I have seen at a customer sites using 2008 R2 file servers sometimes results in temporary profiles.  This happens when the user logs off and the 2008 R2 file server does not release the file handle to the server.  This results in the VHD being locked and as a result, when users login they receive a temporary profile.

In this scenario we confirmed there was no real time AV scanning on the server or third party products that could be locking the server.  It only happens rarely but enough to get the odd user complain to service desk they are getting a temporary profile.

When the VHD is locked on the file server, restarting the Server service or restarting the file server unlocks the file again (both which result in downtime).

I worked with Microsoft PSS support on this case under 115052712773011 and got no where.  After upgrading the file server to 2012 R2, we had no more issues with user profile disks being locked under a file handle on the file server.  Microsoft said that it could be an issue when dealing with the legacy SMB 2.1 protocol where the legacy protocol fails to remove the file handle on the User Profile Disk after the user logs off.

When a Windows machine connects to a legacy operating system for file shares, it will always use the legacy version of the SMB protocol as shown in the following table.  It is recommended to always utilise SMB 3.02 when using Remote Desktop Services 2012 R2 which means you need a 2012 R2 file server for storing the profile disks.