Avantgarde Technologies

<a href="http://www.avantgardetechnologies.com.au">Avantgarde Technologies</a>
Perth's IT Experts

Monday, August 17, 2015

Enable Split Tunneling on Windows 10 VPN Connections

In previous versions of Windows Server, Split Tunneling was enabled by removing the default gateway from the IPv4 settings under the properties of a Windows PPTP, L2TP or SSTP VPN connection.  This was done on the Networking tab and selecting Properties on the Internet Protocol 4 (TCP/IPv4) settings.

In Windows 10 if you click properties on the Internet Protocol Version 4 (TCP/IPv4) settings, nothing happens the button has no code behind it.
On Windows 10, to enable Split Tunneling this must now be done with PowerShell. 
Set-VpnConnection "VPN Connection Name" -SplitTunneling $true
 After enabling the VPNConnection for Split Tunneling this achieves the same affect as the "Use as Default Gateway" button from the IPv4 properties dialog box.

Friday, August 14, 2015

MAPI NSPI Issues with Exchange 2013

A customer running two Exchange 2013 CU9 multi-role servers in a cluster with an F5 load balancer contacted me this morning reporting issues where they could not create new Outlook Profiles for users.  Users with existing Outlook profiles were not affected by the issue.

Users primarily connect with MAPI over HTTP which is enabled on the Exchange Organization configuration.

When a user attempted to create a new profile, they would receive the following error:

The connection to Microsoft Exchange is unavailable.  Outlook must be online or connected to complete this action.

When running the Microsoft Exchange Remote Connectivity Analyzer against the Exchange organisation, the following errors were captured:

Testing MAPI over HTTP connectivity to server webmail.company.com
MAPI over HTTP connectivity failed.
  Additional Details
  HTTP Response Headers:
request-id: 1f79419f-0c42-4a38-bf07-99b1c6882928
Set-Cookie: ClientId=MUZOXYKGXN9GLA; expires=Sat, 13-Aug-2016 02:50:02 GMT; path=/; HttpOnly
Server: Microsoft-IIS/8.5
WWW-Authenticate: Negotiate,NTLM
X-Powered-By: ASP.NET
Date: Fri, 14 Aug 2015 02:50:02 GMT
Content-Length: 0
Elapsed Time: 7764 ms. 

Testing the MAPI Address Book endpoint on the Exchange server.
An error occurred while testing the address book endpoint.
  Additional Details
  Elapsed Time: 5784 ms. 

  Test Steps
  Testing the address book "Check Name" operation for user TestUser@company.com against server webmail.company.com.
  An error occurred while attempting to resolve the name.
  Additional Details
  A protocol layer error occured. HttpStatusCode: 500
FailureLID: 47372

###### REQUEST [2015-08-14T02:50:03.1064218Z] ######

POST /mapi/nspi/?mailboxId=e110343c-c351-4dac-b066-3b552417a51b@company.com HTTP/1.1
Content-Type: application/octet-stream 
User-Agent: MapiHttpClient 
X-RequestId: 57959cb1-fd5d-4ae1-ac55-4494f0a61748:1 
X-ClientInfo: 6016fb10-ca94-4a67-847c-549dd99bacb8:1 
X-ClientApplication: MapiHttpClient/ 
X-RequestType: Bind 
Authorization: Negotiate [truncated] 
Host: webmail.company.com 
Content-Length: 45 

--- REQUEST BODY [+0:05.159] ---
..[BODY SIZE: 45]

--- REQUEST SENT [+0:05.159] ---

###### RESPONSE [+0:05.781] ######

HTTP/1.1 500 Internal Server Error
request-id: 9653be79-d895-4cf3-8169-98ceeb218197 
X-CalculatedBETarget: Server1.company.local 
X-DiagInfo: Server1 
X-BEServer: Server1 
X-FailureContext: BackEnd;500;NTAw;U3lzdGVtLk5ldC5XZWJFeGNlcHRpb246IFRoZSByZW1vdGUgc2VydmVyIHJldHVybmVkIGFuIGVycm9yOiAoNTAwKSBJbnRlcm5hbCBTZXJ2ZXIgRXJyb3IuDQogICBhdCBTeXN0ZW0uTmV0Lkh0dHBXZWJSZXF1ZXN0LkVuZEdldFJlc3BvbnNlKElBc3luY1Jlc3VsdCBhc3luY1Jlc3VsdCkNCiAgIGF0IE1pY3Jvc29mdC5FeGNoYW5nZS5IdHRwUHJveHkuUHJveHlSZXF1ZXN0SGFuZGxlci48PmNfX0Rpc3BsYXlDbGFzczJjLjxPblJlc3BvbnNlUmVhZHk+Yl9fMmIoKQ==;;ProtocolError; 
Persistent-Auth: true 
X-FEServer: Server1 
Transfer-Encoding: chunked 
Cache-Control: private 
Content-Type: text/html; charset=utf-8 
Date: Fri, 14 Aug 2015 02:50:08 GMT 
Set-Cookie: X-BackEndCookie=e110343c-c351-4dac-b066-3b552417a51b=u56Lnp2ejJqBz5nPyM2cxpnSncebmdLLyZue0sfJnJ7SnM2Znc7Iys7LnJnMgYHNz87K0s/G0s7Mq8/NxcrPxc/H; expires=Sun, 13-Sep-2015 02:50:08 GMT; path=/mapi; secure; HttpOnly 
Server: Microsoft-IIS/8.5 
X-AspNet-Version: 4.0.30319 
X-Powered-By: ASP.NET 

Also on only on Server1, the following error was experienced in the event log.

Log Name:      Application
Source:        ASP.NET 4.0.30319.0
Date:          14/08/2015 5:54:03 PM
Event ID:      1309
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Server1
Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 14/08/2015 5:54:03 PM 
Event time (UTC): 14/08/2015 9:54:03 AM 
Event ID: 6e8a15487387414b9279078f9f6aba51 
Event sequence: 2 
Event occurrence: 1 
Event detail code: 0 

Application information: 
    Application domain: /LM/W3SVC/2/ROOT/mapi/nspi-3615-130840196323691514 
    Trust level: Full 
    Application Virtual Path: /mapi/nspi 
    Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\mapi\nspi\ 
    Machine name: Server1

Process information: 
    Process ID: 13528 
    Process name: w3wp.exe 
    Account name: NT AUTHORITY\SYSTEM 

Exception information: 
    Exception type: HttpException 
    Exception message: RpcObjectSetType
   at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
   at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
   at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

   at Microsoft.Exchange.Rpc.RpcServerBase.ThrowRpcException(String message, Int32 rpcStatus)
   at Microsoft.Exchange.Rpc.ProcessAccess.ProcessAccessRpcServer.RegisterInterface(Void* ifSpec, ValueType mgrTypeGuid, _GUID* pMgrTypeUuid, Void* pMgrEpv, UInt32 flags, UInt32 maxCalls)
   at Microsoft.Exchange.Rpc.RpcServerBase.RegisterServer(Type type, ObjectSecurity sd, UInt32 desiredAccess, ValueType mgrTypeGuid, Void* mgrEpv, String annotation, Boolean isLocalOnly, Boolean autoListen, UInt32 maxCalls)
   at Microsoft.Exchange.Data.ApplicationLogic.ProcessAccessManager.RegisterComponent(IDiagnosable diagnosable)

Request information: 
    Request URL: https://localhost:444/mapi/nspi 
    Request path: /mapi/nspi 
    User host address: ::1 
    Is authenticated: False 
    Authentication Type:  
    Thread account name: NT AUTHORITY\SYSTEM 

Thread information: 
    Thread ID: 81 
    Thread account name: NT AUTHORITY\SYSTEM 
    Is impersonating: False 
    Stack trace:    at System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(HttpContext context, HttpApplication app)
   at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
   at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
   at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)

After further investigation, we discovered that only mailboxes which resided on an Active database on Server1 experienced the issue creating Outlook profiles, the same server which experienced EventID 1309.  After failing mailbox databases over to Server2, users were able to login to Outlook without issues.

After this we identified the issue was only present with Server1.

On Server1 we tested the NSPI website by navigating to the Exchange backend in IIS Manager and selecting Browse:444 (https).

The browse failed on Server1 with an "unhandled exception error".  This was aligned with the HTTP 500 exception shown in the Event Logs on Server1 and the Exchange Remote Connectivity Analyzer.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: Microsoft.Exchange.Rpc.RpcException: RpcObjectSetType

Source Error: 
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. 

On Server2 when repeat the same process.

On Server2 it prompted as authentication as normal.

This shows the MAPI/NSPI results as expected.


We rebooted Exchange Server 1 and after a reboot, it started responding to MAPI/NSPI requests as expected.  We believe it was a side affect after the CU9 upgrade, however will monitor the situation and update this blog thread in the event the issue reoccurs.

Wednesday, July 29, 2015

DNS Host A Record Being Deleted on Server Reboot

Our company Avantgarde Technologies was performing a domain controller upgrade for a client from Server 2008 R2 domain controllers to Server 2012 R2.  After deploying the new domain controllers we went through the process of updating the DNS Client records on member servers which had these addresses statically assigned.

After updating a few DNS client settings on member servers, our customer complained that when they rebooted these servers, the member servers Host A record in DNS disappeared.  This issue only occurred with 2008 and 2008 R2 member servers, 2012 servers were not affected.

After looking into this issue, we found that there is a bug with the DNS client in Server 2008 and Server 2008 R2.  Windows Vista and Windows 7 clients are also effected.  This bug is documented under KB2520155 and Microsoft has released a hotfix.

The issue is simple, as per the Microsoft KB article:

"This issue occurs because of an issue in the DNS Client service. When the DNS server configuration information is changed on a client, the DNS Client service deletes the DNS host record of the client from the old DNS server and then adds it to the new DNS server. Because the DNS record is present on the new server that is a part of the same domain, the record is not updated. However, the old DNS server replicates the deletion operation to the new DNS server and to other DNS servers. Therefore, the new DNS server deletes the record, and the record is deleted across the domain."

To get the member server to re-register itself in DNS, you must perform one of the following actions:
  • Restart the computer.
  • Restart the DNS Client service.
  • Run the ipconfig /registerdns command.
After the dynamic registration is re-created the record will not delete unless the DNS client addresses are again changed on the server.

If you require the ability to change the DNS client addresses on member servers on a regular basis, I strongly recommend installing the patch which is available from KB2520155.  In this instance we did not install the patch as this customer does not change the DNS client addresses on member servers on a regular basis.

For this customer we did not install the patch, we simply performed an ipconfig /registerdns on any servers 2008 R2 servers after their DNS client was updated and server was rebooted.

Tuesday, July 7, 2015

The folders cannot be queried. Element not found.

In in the process of setting up a new DFS Namespace server at a remote site for a customer, I faced an issue where I was unable to utilize the DFS management console to add the remote server as a namespace server.  The error received in the DFS Management console was:

The folders cannot be queried. Element not found

After some investigation it turns out that the error was due to a setting the customer had configured:

"Exclude targets outside of the client's site".

After changing the Ordering method back to "Lowest Cost" and waiting 120 minutes for AD Replication and the DFS Namespace to refresh, we were able to configure the DFS Namespace server at the remote site.

Tuesday, June 30, 2015

Newly Built Hyper-V 2012 R2 Server Blue Screening

We experienced an issue with a newly built Hyper-V 2012 R2 server running on Windows Server 2012 R2 Datacentre edition with all the latest critical and security and critical as of 23/06/2015.  The server was built, setup with a number of virtual machines and ran without issues for a period of approximately 1 month.  After approximately 1 month, the server started blue screening on a regular basis, no changes had been made to the Hyper-V Host.

The Hyper-V Host was setup on a Dell PowerEdge R730XD with 192GB of memory.


The cause of the crash as per the original error was from the NT Kernel (ntoskrnl.exe).

We opened a support case with Microsoft to get the full memory dump analysed.  After the memory dump was analysed, we were advised it was a known bug in Windows Server 2012 R2 identified 14th of April 2015.  They pointed us at KB3055343 which resolves a issue with Windows that can cause inconsistent network interface data on the system.

After downloading and installing KB3055343 the issue was resolved.


Tuesday, June 23, 2015

Legacy File Servers with User Profile Disks

I have been involved in a number of Remote Desktop Services 2012 R2 deployments and from my experiences, I recommend to my clients to only utilise Server 2012 R2 file servers.  This is due to two issues which I have experienced with legacy file servers such as Windows Server 2008 R2:
  • Problems using DFS Namespaces
  • Temporary Profiles
DFS Namespaces
DFS Namespaces are supported with User Profile Disks however from my experience both the file server and DFS namespace servers must be Server 2012 R2.  If they are not Server 2012 R2 you will receive this error:
Unable to enable user disks on UserVHDShare. Could not create template VHD.  Error Message: The network location "\\domain.local\namespace\foldertarget" is not available.
 I have replicated this twice in my lab and at a production site when dealing with 2008 R2 file servers.  If you utilise 2012 R2 file servers with 2012 R2 DFS namespace servers you will not experience this error.

Temporary Profiles

I have seen at a customer sites using 2008 R2 file servers sometimes results in temporary profiles.  This happens when the user logs off and the 2008 R2 file server does not release the file handle to the server.  This results in the VHD being locked and as a result, when users login they receive a temporary profile.

In this scenario we confirmed there was no real time AV scanning on the server or third party products that could be locking the server.  It only happens rarely but enough to get the odd user complain to service desk they are getting a temporary profile.

When the VHD is locked on the file server, restarting the Server service or restarting the file server unlocks the file again (both which result in downtime).

I worked with Microsoft PSS support on this case under 115052712773011 and got no where.  After upgrading the file server to 2012 R2, we had no more issues with user profile disks being locked under a file handle on the file server.  Microsoft said that it could be an issue when dealing with the legacy SMB 2.1 protocol where the legacy protocol fails to remove the file handle on the User Profile Disk after the user logs off.

When a Windows machine connects to a legacy operating system for file shares, it will always use the legacy version of the SMB protocol as shown in the following table.  It is recommended to always utilise SMB 3.02 when using Remote Desktop Services 2012 R2 which means you need a 2012 R2 file server for storing the profile disks.


Thursday, May 21, 2015

SSTP VPN Bug with Server 2012 R2

Tonight I stumbled across what appears to be a bug/glitch with Server 2012 R2 when attempting to setup an SSTP VPN connection.  I completed the config as per MS best practice with a valid public digital certificate which I obtained from DigiCert however when I went to connect, I received the following error:

Error Code: 0x80072746
Error Description: 0x80072746: An existing connection was forcibly closed by the remote host.

My certificate was bound to the RRAS instance correctly and TCP443 was forwarded through.

When I ran a "netsh http show ssl" however it did not show the binding, despite me configuring it in the RRAS interface and the certificate existing in the local computer certificate store with private key.
I tried changing the certificate in RRAS to another, then changing it back, but still no binding.
As a result, I exported the certificate off the machine with Private Key, deleted the certificate, then imported it back with Private Key.  After performing this task, I then rebound it in Routing and Remote Access (RRAS) using the same method.
Running the command again, the binding worked.
 And that's why we get paid the big bucks!

Wednesday, May 20, 2015

Testing GRE Protocol with PPTP VPN connections

In this post I am going to show you how to test GRE Protocol is open and working when setting up PPTP VPN Servers.  GRE Protocol being blocked on network infrastructure is one of the most common issues faced when setting up a PPTP VPN Server.

The first step is to stop the VPN Server.  This is done by right clicking the VPN server in Routing and Remote Access and clicking "Stop" under All Tasks.  This is done because our testing tool also listens on TCP1723 so we need to ensure this port is no longer in use on the server in question.
Next download our testing tools called pptpsrv.exe and pptpclnt.exe.  These tools are available in the Windows XP SP2 support tools available from the following URL:
Copy the pptpsrv.exe file to C:\Windows\System32 on your PPTP VPN Server then run it from an Administrative Command Prompt.  Ensure Windows Firewall does not block it.  Run the tool on the server and it will begin listening on TCP1723.
Next run the pptpclnt.exe from another remote computer and target it against the PPTP VPN Server like this:
My local IP address of my PPTP VPN Server is  Then when prompted enter in some sample text to send through the TCP1723 port socket.  I wrote "Hello this is clint".
Next go back to the PPTP VPN Server and verify the text was received.  We can see that the VPN Server received the text "Hello this is clint" and then went onto the second test of closing down the port socket TCP1723 and beginning a socket for GRE Protocol test.

The client automatically goes on to test GRE protocol.  GRE Protocol on my local network is currently blocked by our Cisco Infrastructure (L3 Switch/Router) and as a result we get a Socket Failure error.
This means I need to go back to the Cisco Infrastructure, resolve this problem then test again.
Hope this post has been helpful for troubleshooting PPTP VPN servers.

Wednesday, May 6, 2015

Shared Mailboxes and Sent Items in Microsoft Exchange

In this post I'm going to focus on an Exchange feature introduced back Exchange 2010 SP2 Update Rollup 4 which has not had heavy exposure out there in the real world however is a feature in demand by many companies.  Ever since Exchange 5.5, companies have been able to create shared mailboxes which multiple users use to access email content.  Users with shared mailbox are able to view incoming email which is handy for many shared mailbox scenarios such an "Accounts mailbox", a "Service Desk mailbox", a "HR mailbox" and so on.  However, when a user with access to the shared mailbox "Sends As" the shared mailbox or "Sends on behalf" of the shared mailbox, the Sent Item goes to the users mailbox, not the shared mailbox.  This means users that share the shared mailbox cannot see the email sent on behalf of the mailbox.

See the problem?

Well this has been addressed in Exchange 2010 SP2 Update Rollup 4 and the latest update rollup in Exchange 2010 SP3.  If you are still on Exchange 2003 or 2007 you will need to upgrade to be able to use this feature.

What Microsoft has done is implement a new feature which "copies" the email "Sent As" or "Sent on behalf" of the shared mailbox TO the shared mailboxes sent items from the users primary mailbox.  This means the sent item will appear in both the shared mailbox and the users mailbox.

This needs to be manually configured on each shared mailbox with the following command:

Set-MailboxSentItemsConfiguration "shared mailbox name" -SendAsItemsCopiedTo SenderAndFrom -SendOnBehalfOfItemsCopiedTo SenderAndFrom

This command configures both SendOnBehalfOf and SendAs, if you only want to configure one of them, simply remove the one you want to not configure.

There is a problem however, this command the Set-MailboxSentItemsConfiguration and Get-MailboxSentItemsConfiguration commands were not included with Exchange 2013 or Office 365 and are still not available in the latest release Exchange 2013 CU8.

However, this feature will be coming out soon in Exchange 2013 CU9 but will have different commands and will no longer use the Set-MailboxSentItemConfiguration and Get-MailboxSentItemConfiguration cmdlets.  The commands instead will use the Set-Mailbox cmdlet with the following syntax:

set-mailbox -MessageCopyForSentAsEnabled $False
set-mailbox -MessageCopyForSendOnBehalfEnabled $False
set-mailbox -MessageCopyForSentAsEnabled $True
set-mailbox -MessageCopyForSendOnBehalfEnabled $True

Also, this feature will be enabled BY DEFAULT on all mailboxes in Exchange 2013 CU9.  Companies will no longer need to worry, shared mailboxes will automatically receive any emails "Sent As" or "Sent on behalf as".

More information on this new feature in Exchange 2013 CU9 can be found below:


Monday, April 6, 2015

Unable to Remove Mailbox Permission in Exchange 2013

After a cross-forest domain  migration, a customer complained that a number of their users have random SIDs of unknown accounts linked to their mailbox as shown in the following screenshot.

 This object is not inherited anywhere in the configuration partition in Active Directory, I checked the ACLs on the Databases Exchange Org and the Active Directory containers in between.

The following error shows the mailbox permission in question in more detail showing it is not Inherited.  It also shows the error message we receive when attempting to remove it:

WARNING: Can't remove the access control entry on the object "CN=ceo,OU=Civic Centre,OU=CEO,OU=Office of CEO,OU=Users,OU=COMPANY,DC=DOMAIN,DC=LOCAL" for account "S-1-5-21-3350901170-1262693169-1119774923-3651" because the ACE doesn't exist on the object.

The same error is also experienced if we include "-InheritanceType All" on the command.

After reading into this a bit more, some people have posted that using the Exchange 2003 management tools allows the object to be removed.



I do not believe building a WinXP machine and installing the legacy Adminpak.msi and Exchange 2003 system manager tools is the way to go!

After scratching my head for a while, it turns out that adding "-Deny:$True" to the end of the command allowed me to remove the ACL.

Hope this post saves you some time figuring this one out!

Monday, March 30, 2015

How to Remove SID History from Active Directory Object

In this blog post I'm going to show you how to remove the SIDHistory from an object in Active Directory after a domain migration.  If you attempt to use standard Microsoft tools such as ADSIEdit to remove the SIDHistory from an object regardless what access rights you have been assigned, the following error will be presented.

Operation failed.  Error code: 0x5
Access is denied

00000005: SecErr: DSID-031A1256, problem 4003

To remove SIDHistory from an object you need to use the following VBScript from Microsoft KB295758.


Simply copy and paste the script into a notepad document then run the script with the following arguments to remove the SIDHistory entries from the object in question.


Sunday, March 29, 2015

Windows 8 Unable to Connect through RD Gateway

A customer of mine today logged a support case stating users on Windows 8 or Windows 8.1 were unable to connect to remote computers by using a Remote Desktop Gateway (RD Gateway).

I tested this functionality and was able to reproduce the issue.  The error experienced was:

Remote Desktop can't connect to the remote computer for one of these reasons:
  1. Remote access to the server is not enabled
  2. The remote computer is turned off
  3. The remote computer is not available on the network
Make sure the remote computer is turned on and connected to the network, and that remote access is enabled.

Windows 7 clients did not receive any problems connecting through an RD Gateway.

After reviewing the group policy applied to the domain, I noticed a Group Policy object was setting the LAN Manager Authentication Level to "Send LM & NTLM - use NTLMv2 session security if negotiated".

The new RDP 8.0 client (built into Windows 8 and 8.1) requires this policy be set to "Send NLTMv2 response only" as it uses NTLMv2 and does not have the ability to negotiate authentication.  "Send NTLMv2 response only" is the default for Windows 8 and Windows 8.1.
As a test, I set a local policy on one of the Windows 8 computers using gpedit.msc and then did a gpupdate /force followed by a reboot.

After setting the LAN Manager authentication level to "Send NLTMv2 response only" I was able to connect to RD Gateways without issues.

I did not test this, but I assume if a Windows 7 client was updated to RDP v8 by installing Microsoft KB2592687, the same issue would be experienced if the LAN Manager authentication level is changed to anything other then the default.

Friday, March 27, 2015

PowerShell - Find All Files Beginning With

A customer of mine was hit with another one of those Viruses which encrypt all data on shared drives mapping back to the file server.  The entire shared drive was encrypted and users were no longer able to access documents on the volume.

I restored all encrypted files from backup however I still had these HELP_DECRYPT Ransome ware files in every directory on the file server.

As a result I needed an easy way to find and delete each of these files.


First set the path you want to search, mine was H:\Shared.

Next run the following command to search any files containing HELP_DECRYPT with the following command:

Get-ChildItem $Path -Recurse | Where{$_.Name -Match "HELP_DECRYPT"}

 This went through and listed all of these HELP_DECRYPT files in every directory of the file server recursively.

After you have carefully went through all the results and confirmed that no legitimate files were listed, you can pipe the output from the Get-ChildItem command into Remove-Item cmdlet.

After piping the Output into Remove-Item, run the command to list the items again to ensure they were all deleted correctly.  Getting no output as per above means the files were removed successfully.

Saturday, March 21, 2015

Repairing ContentIndexState on DAG Nodes in Exchange 2013

In Exchange Server 2013, sometimes the content index state can go corrupt on databases.  When this occurs, Exchange 2013 FAST search technology is no longer available for the database meaning people cannot search for content from OWA, Active Sync or Outlook in online mode.

In DAG environments, you can simply reseed the Content Index State from a healthy node in the cluster.  I will show you how to perform this in this blog post.

Here we have two databases in a DAG cluster which have the index status in a failed state:

To manually update the ContentIndexState from a healthy node simply run the following command:

Update-MailboxDatabaseCopy "database\server" -CatalogOnly

In my case the database I want to update was "CCEX2-DB-02" on server "CCEX1" so I ran:

Update-MailboxDatabaseCopy "CCEX2-DB-02\CCEX1" -CatalogOnly

After running the command you can see we brought the index for the database back to a healthy state.

Repeat this process for any other databases with a failed content index.

Friday, March 20, 2015

Public Folder Migration Error:Property expression isn't valid

When migrating public folders from legacy Exchange 2007 and Exchange 2010 environments to Exchange 2013, you may receive the following error message:

Error: Property expression "Anglicare RT" isn't valid. Valid values are: Strings formed with characters from A to Z (uppercase or lowercase), digits from 0 to 9, !, #, $, %, &, ', *, +, -, /, =, ?, ^, _, `, {, |, } or ~. One or more periods may be embedded in an alias, but each period should be preceded and followed by at least one of the other characters. Unicode characters from U+00A1 to U+00FF are also valid in an alias, but they will be mapped to a best-fit US-ASCII string in the e-mail address, which is generated from such an alias.

This is caused by an invalid alias format generally created from legacy versions of Exchange such as 2000 or 2003.  Exchange 2010/2013 does not support spaces in the Public Folder Alias.

On the legacy Exchange 2007/2010 server open up Public Folder Management console and navigate to the Public Folder in question.  On the Exchange General tab you will receive an error message saying the object contains invalid data.

Simply remove the space from the Alias, this is no longer supported.

Next go back to your Exchange 2013 server and resume your public folder migration request.
You may need to repeat this process a few times as it is likely multiple public folders have incorrect aliases.

Monday, February 23, 2015

VSS Snapshot error. The maximum number of snapshots for this volume has been reached.

Today on a customers SBS 2008 server still running Backup Exec 2010, backups started failing with the following error.

 - AOFO: Initialization failure on: "\\JCC-SBS\Microsoft Information Store\First Storage Group". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).

V-79-10000-11234 - VSS Snapshot error. The maximum number of snapshots for this volume has been reached.  Microsoft Volume Shadow Copy Service (VSS)  will not allow any more snapshots.

The Advanced Open File Option was set to Automatically select open file technology - I changed this to use "System - Use Microsoft Software Shadow Copy Provider".

After making this change, I tested another backup and it completed successfully.

Interestingly, if you read the error it was already using the Microsoft Volume Shadow Provider yet it failed.  Hardcoding it in the settings of the backup job did the trick.

Phantom Calls to Cisco Phone with Hosted VOIP Service - How to Lockdown

An issue was experienced where a number of VOIP phones on a customer network were experiencing phantom calls from the Internet.  A phantom call generally occurs when someone is scanning for SIP devices on a network using a port scanning program.  A port scan against SIP (usually UDP 5060 and 5061) will cause many SIP handsets to annoyingly ring and when the user answers, no one is there!

As you could gather, this is very annoying for users on the network.

The solution for this is locking down SIP to only the Public IP addresses of your SIP IP Gateway from your VOIP provider on the public interface of the router.  This customer was using a Cisco 887va router connected on a DSL connection so the public interface in this instance is dialer0.

First of all you need to create a access rule.  Here is the Access Rule I created (this matches the Faktortel VOIP providers public IP addresses here in Australia).  Faktortel also use 5062, 5063, 5064 and 5065 UDP ports for SIP communication.

ip access-list extended VOIPLockDown
remark allow SIP from certain addresses
permit udp host any eq 5060
permit udp host any eq 5061
permit udp host any eq 5062
permit udp host any eq 5063
permit udp host any eq 5064
permit udp host any eq 5065
permit udp host any eq 5060
permit udp host any eq 5061
permit udp host any eq 5062
permit udp host any eq 5063
permit udp host any eq 5064
permit udp host any eq 5065
permit udp host any eq 5060
permit udp host any eq 5061
permit udp host any eq 5062
permit udp host any eq 5063
permit udp host any eq 5064
permit udp host any eq 5065
permit udp host any eq 5060
permit udp host any eq 5061
permit udp host any eq 5062
permit udp host any eq 5063
permit udp host any eq 5064
permit udp host any eq 5065
permit udp host any eq 5060
permit udp host any eq 5061
permit udp host any eq 5062
permit udp host any eq 5063
permit udp host any eq 5064
permit udp host any eq 5065
permit udp host any eq 5060
permit udp host any eq 5061
permit udp host any eq 5062
permit udp host any eq 5063
permit udp host any eq 5064
permit udp host any eq 5065
remark block SIP from all other addresses
deny udp any any eq 5060
deny udp any any eq 5061
deny udp any any eq 5062
deny udp any any eq 5063
deny udp any any eq 5064
deny udp any any eq 5065
remark Allow all other IP traffic
permit ip any any

This rule permits all SIP traffic to all of their public IP addresses then blocks all other traffic entering on these ports and finally permits any/any.

Next you need to assign the access rule to the public interface on the router you wish to filter.  Because this is a Cisco 887va router on DSL, the public interface is Dialer0.  Simply apply the access rule to Dialer0 as follows:

interface dialer0
ip access-group VOIPLockDown in

After applying this config change, all phantom calls to phones on the network stopped.

Important: Some SIP providers use both TCP and UDP for SIP, so it can be useful to block both UDP and TCP on these port numbers.  For this customer, the SIP handsets only supported UDP so there was no point blocking the TCP ports.  If your not sure, block both by slightly modifying the config above.

Tuesday, February 3, 2015

Direct Access Clients Cannot Establish Network Connectivity after NLS Becomes Unavailable

In Microsoft Direct Access Deployments, you need to configure a server (or server cluster) on your network to act as a Network Location Server (NLS).  The NLS is simply a Web Server with a HTTPS server certificate... see the following TechNet article on how to configure an NLS:


If the DirectAccess client can establish a connection to the NLS, it will believe that it is on the internal network and the DirectAccess tunnels will not be established. If a connection to the NLS cannot be established, the DirectAccess client believes it is outside the corporate network and will attempt to establish DirectAccess connectivity. It is for this reason that the NLS should not be resolvable in public DNS and should not be reachable externally. If it is, DirectAccess clients will always think they are inside the corporate network, and DirectAccess will never be established. Also, it is extremely important that the NLS be highly available. If the NLS server is offline or unreachable for any reason, DirectAccess clients that are on the internal network will suddenly think they are outside the corporate network and attempt to establish DirectAccess connectivity. This will fail, leaving DirectAccess clients unable to connect to any internal resources until the NLS is once again available.  For this reason it is strongly recommended that you implement at least two NLS servers in a cluster, using either Network Load Balancing (NLB) or an external hardware load balancer.

However for smaller deployments of Direct Access, a NLS cluster for redundancy is not always practical and sometimes only a single NLS server is deployed.  This is often the case with all in one Direct Access Deployments running Windows Server 2012 R2 when a single Direct Access server is deployed to act as both the NLS and Direct Access endpoint.  If the all in one Direct Access server was to fail, all clients on the Internal network will no longer think they are on the internal network as they cannot contact the NLS and as a result attempt to establish a Direct Access connection which will also fail leaving them in a state of no connectivity to any domain names configured with a Name Resolution Policy Table (NRPT) in the Direct Access group policy.  Most Direct Access deployments have the Active Directory domain namespace configured as an NRPT policy and sometimes additional domain names.  To understand what NRPT is, please see the following blog post:


What Happens if Your In this Situation?

What happens if your in this situation when you have a single Direct Access server which has failed and as a result all Direct Access configured clients on the internal network now have no network connectivity?

The NRPT Policies which are pushed out via Group Policy are located under the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows NT\DnsClient\DnsPolicyConfig

You can manually connect to the direct access computers and remove these registry keys by IP address using the "Connect Network Registry" option in regedit.exe.  Once you are connected to the remote computer, you can delete the two NRPT policies named DA-{GUID].

After you have deleted the two policies remotely, simply reboot the workstation and it will regain connectivity.

I know this is a very manual process and can be very time consuming if there are hundreds of computers experiencing the issue - I would like to hear from Microsoft for a better solution in this scenario.  I sent of an email regarding this and hopefully will hear back soon!

It is also important to note, for my customer who's Direct Access server failed, rebuilding a new server with the same IP address, hostname and configuration did not resolve this issue.  I still needed to delete the NRPT policies from the registry from the effected computers and reboot.  After this, the clients had restored connectivity to the internal network and were able to refresh group policy again in which they downloaded the new Direct Access Client Settings containing the new NRPT policies.

Friday, January 23, 2015

Test Application for Group Policy Software Installation Troubleshooting

Sometimes when diagnosing problems with Group Policy software deployment, it is good to try a different application to rule out issues with the application package your trying to deploy.  One application package I have found which is great for testing Group Policy software deployment is "RealWorld Paint".  This is a lightweight MSI installer (8.7 MB) in size and deploys easy through Group Policy to Windows XP, Windows 7 and Windows 8.1 machines.

You can download this MSI installer from the following location:


If you have issues deploying this with Group Policy, then the issue does not lay with the MSI package your trying to deploy but something else!

Also check out this common issue with Group Policy application deployment, it may be of help in diagnosing your Group Policy deployment issue:


Tuesday, January 20, 2015

HyperV Network Types

Microsoft HyperV servers have three Network Types you can assign to Virtual Machines and Virtual Switches.
  • Private
  • Internal
  • External
This is shown below on both a virtual machine and a virtual switch configuration.

What is the difference between these?
An External Network Type provides communication between a virtual machine and a physical network by creating an association with a physical network adapter on the virtualization server.  This is the most common type used by organisations.
An Internal Network provides communication between the virtualization server and the virtual machines.
A Private Network only provides communication between the virtual machines and not the HyperV host server.
Internal and Private network types can be confusing - the only difference is Internal also allows the virtual machines to communicate with the HyperV host server.

Thursday, January 8, 2015

GFI MailEssentials 2014 SR2 Transport Issues during Exchange Migration

I had an issue with a third party email filtering product GFI MailEssentials 2014 SR2 during an Exchange 2010 to Exchange 2013 migration.  GFI MailEssentials 2014 SR2 is a spam filtering product which you install directly on an Exchange transport server.  It integrates with Exchange server through six transport agents which all perform various tasks as shown below:

When migrating to a new version of Exchange, as part of the process you are required to redirect mail flow to the new Exchange transport server so it can route the mail to the legacy transport environment until a point when the mailboxes can be moved.
As the new Exchange 2013 server will be the new external point of connectivity for SMTP, I installed GFI Mail Essentials on the new Exchange 2013 server and redirected mail flow as shown below.
After making this change, users were not able to receive email from external users.  I verified the following things:
  • Exchange 2013 was receiving emails from external users as validated in the SMTP Protocol logs.
  • Exchange 2013 was forwarding emails to the Exchange 2010 server as per standard functionality.
  • Exchange 2010 successfully received the email communication from Exchange 2013 at transport level and was verified in the protocol logs.
  • GFI MailEssentials Transport Agents on the Exchange 2010 server receive the email for processing.
  • GFI MailEssentials does not place the email back into the Exchange Pickup folder giving it back to Exchange for processing.
I was not able to locate where the emails were moved to within GFI primarily due to my limited knowledge of the product (to me it is just a custom Exchange transport agent).  I contacted GFI support here in Australia who were also unable to advise me where the emails went after being relayed to the Exchange 2010 server.  Fantastic, so we have emails going into a black hole disappearing forever.
One thing GFI support were able to advise me was their transport agents only filter email which was sent from a public IP address, all private IP addresses were excluded from filtering.  This was in line with my symptoms, all users internally were able to receive emails sent from internal devices such as Printers being relayed through the Exchange 2013 server.
In the following screenshot I have included the message tracking log from the Exchange 2010 server.  The first two entries are from when the Cisco Router was forwarding email directly at the Exchange 2010 server.  All other entries are from when mail was relaying through Exchange 2013.  As you see email is received via SMTP however not delivered to the information store via the Exchange Store Driver due to GFI not releasing the mail.
GFI Mail Essentials modifies the header of emails that are filtered and appears to not deal with emails correctly which already have their header modified by another instance of GFI Mail Essentials.
As a work around I simply disabled the GFI Transport Agents on the Exchange 2010 server to prevent it from interfering with mail processing to resolve the issue as shown below:
This resolved the issue and did not compromise the environment as security and spam filtering was now being performed by GFI on the Exchange 2013 server.