Avantgarde Technologies

<a href="http://www.avantgardetechnologies.com.au">Avantgarde Technologies</a>
Perth's IT Experts

Tuesday, November 18, 2014

Distributed File Server Replication on Windows Server 2012 R2 Bug 2951262

I came across a bug with Windows Server 2012 R2 where a spoke server randomly failed to replicate to the hub server.  In total there were three DFSR servers partaking in replication within a single replication group.

The following diagram shows an overview of the topology and the server experiencing issues:

Randomly SPOKE2 stopped replicating to the hub server HUBSERVER and is no longer responding to DFS Health Reports resulting in the health report generation process hanging forever.


When restarting the DFS-R service on SPOKE2, SPOKE2 gets reported to be in an Indeterminate State for approximately 2-3 hours.


 After being in an Indeterminate State for 2-3 hours, the status change to “Auto Recovery” for approximately 6 more hours.  During this time the DFS-R service generates a large amount of disk activity as it goes through and checks all the files.  This can be observed using Windows Resource Monitor.

The Auto Recovery process never completes successfully, nor are there any errors in the event log on SPOKE2.  Rebooting SPOKE2 or restarting the DFS-R service results in the server going back to an indeterminate state for another 2-3 hours then starting the Auto Recovery process again.

Resolution

This issue is caused by a bug with Windows Server 2012 R2 documented on Microsoft KB 2951262.

http://support.microsoft.com/kb/2951262

You must manually request the Hotfix which Microsoft will email to you and install it on the effected servers.  After installing the hotfix the server will revert to an Indeterminate state then start Auto Recovery again however this time after the Auto Recovery process, it will resume replication as normal.

Public Folder Migration Request with StatusDetail of FailedOther

I was in the process of migrating to Exchange 2013 "Modern Public Folders" for a customer of mine here in Western Australia from an Exchange 2010 SP3 server to Exchange 2013 SP1.  After commencing the migration with the New-PublicFolderMigrationRequest cmdlet the migration request shortly after failed.

Looking at the Migration Request Statistics with the following cmdlet it came up with StatusDetail as FailedOther:

Get-PublicFolderMigrationRequest | Get-PublicFolderMigrationRequestStatistics

 
 When looking into the error in more detail with

"Get-PublicFolderMigrationRequest | Get-PublicFolderMigrationRequestStatistics | fl"

 The following was logged regarding the error message:

FailureCode: -2146233088
FailureType: DataValidationException
Message: Error: Property expression "Outlook Security Settings" isn't valid.  Valid values are: Strings formed with characters from A to Z (uppercase and lowercase), digits from 0 to 9, !, #, $, &, ', *, +, -, /, =, ?, ^, _, `, {, |, } or ~. One or more periods may be embedded in the alias, but each period should be preceded and followed by at least one of the other characters.


In the public folder structure at my customer, there was folder named "Outlook Security Settings" as per the error message.

 
This folder name or any of the sub items did not appear to breach any of the invalid characters as per the stated error message.  Luckily this folder was not required by the business and I simply remove the problematic public folder from the Exchange 2010 server and repeated the move request.  After preforming this I was able to successfully complete the public folder migration request.

Monday, November 10, 2014

Problems When Upgrading your domain and forest functional level from 2003 to 2008 R2

A customer of mine upgraded their Domain Functional Level (DFL) and Forest Functional Level (FFL) to Windows Server 2008 R2 yesterday.  Today when employees started work, they experienced lengthy login times, did not receive their network drive mapping to the file server and were unable to connect to Exchange Server 2010 with Microsoft Outlook 2010.

The first thing I did was have a look at the Active Directory replication after the functional level upgrade using the following command "repadmin /showrepl" on one of the Active Directory domain controllers.  This showed the following error:

Last error: -2146892990 (0x80090342)
The encryption type requested is not supported by the KDC

 
In 2003 functional level the Kerberos Key Distribution Centre (KDC) used either RC4-HMAC 128-bit or DES-CBC-MD5 56-bit for Kerberos Encryption however when moving to 2008 Domain Functional Level (or higher) you upgrade the Key Distribution Centre (KDC) to use Advanced Kerberos Encryption which uses AES 128 and AES 256 encryption.
 
Generally this transition is smooth and does not cause problems however in this instance the KDC did not detect the functional level change and continued to operate using the legacy 2003 functional level encryption technology.  As a result the error "The encryption type required is not supported by the KDC".
 
To resolve this problem was very simple, we simply restarted the Kerberos Key Distribution Centre on all of the Active Directory domain controllers in the domain.
 
 
Within 5 minutes of the service restart, things were back to normal.

Tuesday, November 4, 2014

Failed to submit the request because public folder migration has already been successfully completed previously

When migrating to Exchange 2013 modern public folders, you may want to repeat the migration especially if you need to tweak the Public Folder-to-Mailbox Mapping File and perform the migration again.

To repeat the migration progress you need to perform the following steps:
  1. Removing all public folders from the newly created public folder mailbox with Get-PublicFolder –Recurse | Remove-PublicFolder (make sure you run this on Exchange 2013 Management Shell so you don't delete the public folders on your 2007/2010 environment).
  2. Removing the default public folder hierarchy mailbox with Get-Mailbox –PublicFolder | Remove-Mailbox -PublicFolder
  3. Clearing the MsExchDefaultPublicFolderMailbox attribute on the Exchange organisation with ADSIEdit as per Exchange MVP Satheshwaran Manoharan's article http://careexchange.in/how-to-recreate-public-folder-master-hierarchy-in-exchange-2013/
  4. Remove the Exchange 2013 public folder move request using Get-PublicFolderMigrationRequest | Remove-PublicFolderMoveRequest

At this point you would think you should be ready to migrate the public folders to Exchange 2013 again however there is a final step which needs to be performed.  This final step has caught out many Exchange Admins and is not documented at all online.  Without the final step, the following error is experienced:

"Failed to submit the request because public folder migration has already been successfully completed previously"


This error occurs if the PublicFolderMigrationComplete attribute is set to $true on the Exchange Organisation Configuration as shown in the following screenshot.

Simply use the following PowerShell command to set the PublicFolderMigrationComplete attribute to $false.

Set-OrganizationConfig -PublicFolderMigrationComplete $false


After this final step you can go ahead and remigrate your public folders to Exchange 2013 from scratch.

Tuesday, October 14, 2014

Exchange 2010 User Sends Two Out of Office Messages

One of my customers had a mailbox user which generated two Out of Office messages whenever her Out of Office automated replies were enabled.  The first Out of Office message is the message the user set on her mailbox as the automated reply for both internal and external Out of Office.  The second Out of Office message is a different message for a user which no longer exists on the network and the contents of the Out of Office message does not display in Outlook Web App or Outlook.

The legitimate Out of Office is provided with a message subject of:

Automatic reply:

The second unwanted Out of Office is provided with a message subject of:

Out of Office

I created a test user account and send the problematic user a test email when her Out of Office was enable.  The following screenshot shows the extra Out of Office message sent:



First thing I checked was the MailboxAutoReplyConfiguration set on the mailbox which shows the Out of Office message set on the Mailbox for both Internal and External recipients.  This did not reveal the second Out of Office message causing the problem for this user account.


This means the second problematic Out of Office response must be stored elsewhere in the users mailbox.  Before going gown the path of downloading MFCMAPI and manually browsing through the problematic mailbox for the second Out of Office rule, I decided to run the "outlook.exe /cleanrules" Outlook 2010 command.  This command cleans and removes both server side and client side rules including any Out of Office rules which may exist in the mailbox.

Simply close Outlook on the users PC and then execute the following command:

 
After cleaning the rules and retesting the users Out of Office from my test mailbox, I only received the single Out of Office message which is the one we configured on the users mailbox as shown in the following screenshot.
 
 
Summary

Whilst a resolution to the issue was found, I am not sure what caused this issue.  The problematic mailbox is a member of a Database Availability Group cluster and was migrated cross-forest from an Exchange 2007 environment.  This should not cause problems as Exchange 2007 utilises a similar architecture for Out of Office messages as Exchange 2010 supporting the concept of both Internal and External Out of Office responses.  I believe this mailbox has been around for a long time and this legacy rule may have been brought across from Exchange 2003 as this particular customer uses role based mailboxes and when new users come on-board they inherit the last users mailbox keeping the same email address.

Please leave your comments if you also experience this issue.

Monday, October 13, 2014

Error Moving Mailbox Exchange 2007 to Exchange 2010

Today I had an issue moving a mailbox from Exchange 2007 to Exchange 2010 where the following error was generated in Exchange Management Shell.

Active Directory operation failed on domaincontroller.domain.local. This error is not retriable. Additional information: Insufficient access rights to perform the operation.
Active directory response: 00002098: SecErr: DSID-03150BB9, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
    + CategoryInfo          : NotSpecified: (0:Int32) [New-MoveRequest], ADOperationException
    + FullyQualifiedErrorId : 7189915F,Microsoft.Exchange.Management.RecipientTasks.NewMoveRequest


 
The error occurs if the account being moved does not have correct Exchange 2010 permissions set on the user account in Active Directory.  On the user account in Active Directory Users and Computers on the Security Tab under Advanced, "Include inheritable permissions from this object's parent" was not selected as shown below:
 
Note: In order to see the Security Tab on the user account in Active Directory Users and Computers you must enable "Advanced Features" which can be found under the view menu.
 
Simply select Include inheritable permissions from this object's parent.
 
 
 After making this change the move worked as expected.

 

Thursday, October 9, 2014

Windows XP clients must match the Certificate Common Name in Outlook for RPC over HTTP

Back in 2009 I wrote an article called "Configuring Outlook Anywhere Settings via Autodiscover" which can be found at the following URL:

http://clintboessen.blogspot.com.au/2009/06/configuring-outlook-anywhere-settings.html

Also in 2009 I wrote an article called "Outlook Anywhere keeps prompting for Password" which can be found at the following URL:

http://clintboessen.blogspot.com.au/2009/06/outlook-anywhere-keeps-prompting-for.html

In this blog post I am going to touch on these two topics a bit further as this issue will impact many organisations moving to Exchange 2013 for those still running Windows XP (despite the fact it is no longer supported) seeming Exchange 2013 only supports RPC over HTTP or MAPI over HTTP (out of scope for this article).  To recap what I wrote about in the previous articles 5 years ago, for RPC over HTTP(s) aka Outlook Anywhere to work on any version of the Outlook Client on the XP, the MSSTD value must match the "Common Name" on the certificate.  What am I talking about?  Well let me show you...

The MSSTD is specified under "Only connect to proxy servers that have this principal name in their certificate" which can be found within Outlook under Account Settings, Open the Account, More Settings, Connection Tab and finally Exchange Proxy Settings.

 
This value must match the "Common Name" of the certificate which in this example is mail.example.com as shown below next to "Issued to:"
 
 
From Windows Vista onwards the MSSTD can match any name in the Certificate which includes the Certificate Common Name as shown above and any Subject Alternative Names (SAN) which may exist on the certificate.  These are often used and can be easily located on the details tab of a certificate in Microsoft Windows as shown below:
 
 
For Windows XP the "Only connect to proxy servers that have this principal name in their certificate" value MUST MATCH the common name on the Certificate.  The symptom for having this not matching is the user continuously being prompted for credentials in an infinite loop as I addressed under the article Outlook Anywhere keeps prompting for Password.
 
 Note: This also applies to Windows Server 2003 for people with legacy Terminal Server / Citrix environments.
 
What about for Wild Card certificates, how do I set the "Only connect to proxy servers that have this principal name in their certificate" MSSTD value for these for legacy Windows XP clients?
 
For Wild Card certificates the MSSTD value must be set to:
 
msstd:*.domain.com
 
You must actually put the astricts in the DNS name instead of the name the client is connecting to in order for Windows XP to be happy so it matches the exact name of the Wildcard certificate.
 
What about Autodiscover?
 
Now that we understand that the MSSTD must match the common name of the certificate for our legacy Windows XP clients, how do we automatically push this out to our clients via Autodiscover?
 
Exchange 2013 no longer directly uses EXPR/EXCH Outlook Providers which we configured in Exchange 2007/2010, it has two different dynamically generated EXHTTP providers. Users with mailboxes on 2013 will get one set of EXHTTP settings for internal usage and one set of EXHTTP settings for external usage.  You will see this returned to the Outlook client in the Autodiscover response via what looks like a new provider, ExHTTP.
 
However ExHTTP isn’t an actual provider, it is a calculated set of values from the EXCH (internal Outlook Anywhere) and EXPR (External Outlook Anywhere) settings.  The concept of internal Outlook Anywhere and External Outlook Anywhere is new in Exchange 2013 as we
got rid of direct RPC to the Exchange server.  We want the ability to set NTLM authentication for internal and Basic authentication for External as well as different connection URLs.
 
To change the internal MSSTD returned value from Autodiscover use the following command:
 
Internal Outlook Anywhere:
 
Get-OutlookProvider "EXCH" | Set-OutlookProvider -CertPrincipalName "msstd:mail.domain.com"
 
External Outlook Anywhere:
 
Get-OutlookProvider "EXPR" | Set-OutlookProvider -CertPrincipalName "msstd:mail.domain.com"
 
If you have a private internal namespace such as .local, .lan or .internal you may also need to setup split horizon DNS so that you can make the Certificate Common Name match the MSSTD!
 
Hope this article has been helpful.

Monday, October 6, 2014

Setup Wizard for Update Rollup 7 for Exchange 2010 Service Pack 3 (KB2961522) ended prematurely

When installing Update Rollup 7 on Exchange Server 2010 Service Pack 3 I received the following error message:

Setup Wizard for Update Rollup 7 for Exchange 2010 Service Pack 3 (KB2961522) ended prematurely.

Setup Wizard for Update Rollup 7 for Exchange 2010 Service Pack 3 (KB2961522) ended prematurely because of an error.  Your system has not been modified.  To install this program at a later time, please run the installation again.


This error was encountered after running the Exchange2010-KB2961522-x64-en.msp file downloaded from the Microsoft Website.

In the ServiceControl.txt file located under C:\ExchangeSetupLogs the following errors were present:

[12:58:25] Service 'IIS Admin Service (IISAdmin)' cannot be configured due to the following error: Access is denied
[12:58:25] Enabling service 'MSExchangeAB'.
[12:58:25] Service 'Microsoft Exchange Address Book (MSExchangeAB)' cannot be configured due to the following error: Access is denied
[12:58:25] Enabling service 'MSExchangeADTopology'.
[12:58:25] Service 'Microsoft Exchange Active Directory Topology (MSExchangeADTopology)' cannot be configured due to the following error: Access is denied
[12:58:25] Enabling service 'MSExchangeAntispamUpdate'.
[12:58:25] Service 'Microsoft Exchange Anti-spam Update (MSExchangeAntispamUpdate)' cannot be configured due to the following error: Access is denied
[12:58:25] Enabling service 'MSExchangeEdgeSync'.
[12:58:25] Service 'Microsoft Exchange EdgeSync (MSExchangeEdgeSync)' cannot be configured due to the following error: Access is denied
[12:58:25] Enabling service 'MSExchangeFBA'.
[12:58:25] Service 'Microsoft Exchange Forms-Based Authentication service (MSExchangeFBA)' cannot be configured due to the following error: Access is denied
[12:58:25] Enabling service 'MSExchangeFDS'.
[12:58:25] Service 'Microsoft Exchange File Distribution (MSExchangeFDS)' cannot be configured due to the following error: Access is denied
[12:58:25] Enabling service 'MSExchangeIMAP4'.
[12:58:25] Service 'Microsoft Exchange IMAP4 (MSExchangeIMAP4)' cannot be configured due to the following error: Access is denied
[12:58:25] Enabling service 'MSExchangeIS'.
[12:58:25] Service 'Microsoft Exchange Information Store (MSExchangeIS)' cannot be configured due to the following error: Access is denied
[12:58:25] Enabling service 'MSExchangeMailboxAssistants'.
[12:58:25] Service 'Microsoft Exchange Mailbox Assistants (MSExchangeMailboxAssistants)' cannot be configured due to the following error: Access is denied
[12:58:25] Enabling service 'MSExchangeMailboxReplication'.
[12:58:25] Service 'Microsoft Exchange Mailbox Replication (MSExchangeMailboxReplication)' cannot be configured due to the following error: Access is denied
[12:58:25] Enabling service 'MSExchangeMailSubmission'.
[12:58:25] Service 'Microsoft Exchange Mail Submission (MSExchangeMailSubmission)' cannot be configured due to the following error: Access is denied
[12:58:25] Enabling service 'MSExchangeMonitoring'.


and so on..

Generally when you run a .msp update file it automatically provides an User Account Control (UAC) exception however on this customers network it did not.  As a result UAC prevented the update from performing the required tasks.

To resolve the problem I simply needed to run the update from an elevated command prompt window as follows:

 

Cannot bind argument to parameter 'Identity' because it is null

When attempting to install Exchange 2010 SP3 Service Pack on an Exchange 2010 SP1 Server I received the following on the Mailbox Server Role.

Error:
The following error was generated when "$error.Clear();
          if ($RoleCreatePublicFolderDatabase)
          {
            $publicDB = get-PublicFolderDatabase -Server:$RoleFqdnOrName -ErrorAction SilentlyContinue;
            $DB = get-MailboxDatabase -Server:$RoleFqdnOrName -ErrorAction SilentlyContinue;
            if ($publicDB -and $DB)
            {
                set-mailboxdatabase `
                  -Identity:$DB.Identity `
                  -publicFolderDatabase:$publicDB.Identity `
                  -DomainController $RoleDomainController
            }                 
          }
        " was run: "Cannot bind argument to parameter 'Identity' because it is null.".

Cannot bind argument to parameter 'Identity' because it is null.


If you received this error you most likely also received:

http://clintboessen.blogspot.com.au/2014/10/this-server-role-cant-be-installed.html

This error occurs if the original Exchange 2010 SP1 or SP2 installation did not finish "Finalizing Setup" which is the final stage in the installation process.  Failure means important registry keys are not set correctly.  In my case the Finalizing Setup did not complete because of the following error:
 
 
This error is generated if the MailboxRole is missing a ConfiguredVersion string value under the following registry key:
 
HKLM\SOFTWARE\Microsoft\ExchangeServer\v14\MailboxRole
 
As you see in the following screenshot the ConfiguredVersion string value is missing:
 
 
Simply create the string value called ConfiguredVersion and give it the same value as UnpackedVersion as shown in the following screenshot:
 
 
After this I deleted the Exchange 2010 SP3 installation watermark (both the “Action” and “Watermark” entries in the registry) and re-ran setup.  Google for this, lots of information already available on the Internet

This server role can't be installed because the following roles aren't current: AdminToolsRole

When attempting to install Exchange 2010 SP3 service pack on an Exchange 2010 SP1 server I received the following error:

This server role can't be installed because the following roles aren't current: AdminToolsRole

 
This error occurs if the original Exchange 2010 SP1 or SP2 installation did not finish "Finalizing Setup" which is the final stage in the installation process.  Failure means important registry keys are not set correctly.  In my case the Finalizing Setup did not complete because of the following error:
 
 
The above error is triggered if the ConfiguredVersion does not match the UnpackedVersion registry key.  These registry keys can be found under the following location:
 
HKLM\SOFTWARE\Microsoft\ExchangeServer\v14\AdminTools
 
To fix this I made sure the ConfiguredVersion key matches the UnpackedVersion key as shown in the following screenshots:
 
Before:
 
 
After:
 
 
After this I deleted the Exchange 2010 SP3 installation watermark (both the “Action” and “Watermark” entries in the registry) and re-ran setup.  Google for this, lots of information already available on the Internet.
 
Note: If you experienced this error you may also experience the following:
 

Couldn't resolve the user or group "domain.local/Microsoft Exchange Security Groups/Discovery Management." If the user or group is a foreign forest principal, you must have either a two-way trust or an outgoing trust.

I ran into a problem when running the Exchange 2010 SP1 setup in an environment with a single Exchange 2007 SP3 server.  The setup failed on the Mailbox Server role with the following error:

Error:
The following error was generated when "$error.Clear();
          $name = [Microsoft.Exchange.Management.RecipientTasks.EnableMailbox]::DiscoveryMailboxUniqueName;
          $dispname = [Microsoft.Exchange.Management.RecipientTasks.EnableMailbox]::DiscoveryMailboxDisplayName;
          $dismbx = get-mailbox -Filter {name -eq $name} -IgnoreDefaultScope -resultSize 1;
          if( $dismbx -ne $null)
          {
            $srvname = $dismbx.ServerName;
            if( $dismbx.Database -ne $null -and $RoleFqdnOrName -like "$srvname.*" )
            {
              Write-ExchangeSetupLog -info "Setup DiscoverySearchMailbox Permission.";
              $mountedMdb = get-mailboxdatabase $dismbx.Database -status | where { $_.Mounted -eq $true };
              if( $mountedMdb -eq $null )
              {
                Write-ExchangeSetupLog -info "Mounting database before stamp DiscoverySearchMailbox Permission...";
                mount-database $dismbx.Database;
              }

              $mountedMdb = get-mailboxdatabase $dismbx.Database -status | where { $_.Mounted -eq $true };
              if( $mountedMdb -ne $null )
              {
                $dmRoleGroupGuid = [Microsoft.Exchange.Data.Directory.Management.RoleGroup]::DiscoveryManagementWkGuid;
                $dmRoleGroup = Get-RoleGroup -Identity $dmRoleGroupGuid -DomainController $RoleDomainController -ErrorAction:SilentlyContinue;
                if( $dmRoleGroup -ne $null )
                {
                  Add-MailboxPermission $dismbx -User $dmRoleGroup.Identity -AccessRights FullAccess -DomainController $RoleDomainController -WarningAction SilentlyContinue;
                }
              }
            }
          }
        " was run: "Couldn't resolve the user or group "domain.local/Microsoft Exchange Security Groups/Discovery Management." If the user or group is a foreign forest principal, you must have either a two-way trust or an outgoing trust.".

Couldn't resolve the user or group "domain.local/Microsoft Exchange Security Groups/Discovery Management." If the user or group is a foreign forest principal, you must have either a two-way trust or an outgoing trust.
The trust relationship between the primary domain and the trusted domain failed.


The mailbox role is the last role which is installed as part of the Exchange 2010 SP1 setup process.  Despite this error occurring, the Exchange 2010 SP1 installation did complete with all Exchange 2010 services present on the server, started and in a functional state.  The Exchange 2010 SP1 management tools were also installed and working.

As a result I pushed on and began installing the Exchange 2010 SP3 upgrade installation package.  The Exchange 2010 SP3 upgrade installation package also did not complete the installation correctly and failed with the same error:

Error:
The following error was generated when "$error.Clear();
          $name = [Microsoft.Exchange.Management.RecipientTasks.EnableMailbox]::DiscoveryMailboxUniqueName;
          $dispname = [Microsoft.Exchange.Management.RecipientTasks.EnableMailbox]::DiscoveryMailboxDisplayName;
          $dismbx = get-mailbox -Filter {name -eq $name} -IgnoreDefaultScope -resultSize 1;
          if( $dismbx -ne $null)
          {
            $srvname = $dismbx.ServerName;
            if( $dismbx.Database -ne $null -and $RoleFqdnOrName -like "$srvname.*" )
            {
              Write-ExchangeSetupLog -info "Setup DiscoverySearchMailbox Permission.";
              $mountedMdb = get-mailboxdatabase $dismbx.Database -status | where { $_.Mounted -eq $true };
              if( $mountedMdb -eq $null )
              {
                Write-ExchangeSetupLog -info "Mounting database before stamp DiscoverySearchMailbox Permission...";
                mount-database $dismbx.Database;
              }

              $mountedMdb = get-mailboxdatabase $dismbx.Database -status | where { $_.Mounted -eq $true };
              if( $mountedMdb -ne $null )
              {
                $dmRoleGroupGuid = [Microsoft.Exchange.Data.Directory.Management.RoleGroup]::DiscoveryManagementWkGuid;
                $dmRoleGroup = Get-RoleGroup -Identity $dmRoleGroupGuid -DomainController $RoleDomainController -ErrorAction:SilentlyContinue;
                if( $dmRoleGroup -ne $null )
                {
                  Add-MailboxPermission $dismbx -User $dmRoleGroup.Identity -AccessRights FullAccess -DomainController $RoleDomainController -WarningAction SilentlyContinue;
                }
              }
            }
          }
        " was run: "Couldn't resolve the user or group "domain.local/Microsoft Exchange Security Groups/Discovery Management." If the user or group is a foreign forest principal, you must have either a two-way trust or an outgoing trust.".

Couldn't resolve the user or group "domain.local/Microsoft Exchange Security Groups/Discovery Management." If the user or group is a foreign forest principal, you must have either a two-way trust or an outgoing trust.
The trust relationship between the primary domain and the trusted domain failed.


In the Exchange 2010 with SP1 installation package, the Management Tools were installed before the Hub Transport, Client Access and Mailbox Server roles.  When running Exchange 2010 SP3 upgrade package, the Management Tools are installed after the Hub Transport, Client Access and Mailbox Server roles and as a result were not upgraded as part of the Service Pack.  When the Service Pack failed on the Mailbox role it skipped updating the Management Tools - as a result this problem needed to be addressed.

The error is stating it could not resolve user or groups for Discovery Management - there is only one discovery user created by default with Exchange 2010 called:

DiscoverySearchMailbox {D919BA05-46A6-415f-80AD-7E09334BB852}

As a result I decided to recreate this mailbox by deleting it and recreating it.  This was done with the following commands:

Disable-Mailbox "DiscoverySearchMailbox {D919BA05-46A6-415f-80AD-7E09334BB852}"

Enable-Mailbox "DiscoverySearchMailbox {D919BA05-46A6-415f-80AD-7E09334BB852}" -Arbitration


Next I assigned the Discovery Management user permissions to the new mailbox with the following command:

Add-MailboxPermission -Identity:"cosp.internal/Users/DiscoverySearchMailbox {D919BA05-46A6-415f-80AD-7E09334BB852}" -User:"Discovery Management" -AccessRights:"FullAccess"


After this I deleted the Exchange 2010 SP3 installation watermark (both the “Action” and “Watermark” entries in the registry) and re-ran setup.  Google for this, lots of information already available on the Internet.

After making these changes I was able to successfully patch the Exchange 2010 SP1 server with Exchange 2010 SP3.

 

Thursday, October 2, 2014

What is the SCCM Management Point?

A management point is a site system role that provides policy and service location information for clients and it also receives configuration data from clients. When we deploy software to a client, the client sends a content request to a management point.  The management point sends a list of the preferred distribution points to the client, and the client uses one of the preferred distribution points as source location for content. If contents are not available on the preferred distribution point, the management point sends a list to the client with distribution points that have the content available.

Management Point can be defined on client computers when they are installed, or client can get the list of Management points through DNS or WINS.

Clients search for a Management Point by using the below options in the order specified:
  • Management point
  • Active Directory  Domain Services
  • DNS
  • WINS
 

Sunday, September 28, 2014

Direct Access - IPHTTPS interface creation failed 0x643

Today I had an issue with a Windows 7 Enterprise laptop on my domain failing to successfully create a Direct Access connection to my Windows Server 2012 R2 server.  The error raised on the HTTPSTunnel interface was 0x643 with a status of IPHTTPS interface creation failure.


Also in Device Manager, the httpstunnel interface had a yellow explanation mark.

This problem can be caused by a few things, one of the most common causes is the DisabledComponents DWORD not being set to 0 which in effect disables IPv6 which is required by Direct Access.  Check this under the following registry key:

HKLM\SYSTEM\CurrentControlSet\services\TCPIP6\Parameters

Note: If this registry key does not exist, it is the same as having it being set to 0.


Another issue which can cause this error is when the computer looses its trust with the computer account object in Active Directory.  When you bring the computer in and plug it into the internal network, you will see one of these two errors:

The trust relationship between this workstation and the primary domain failed.

 
The security database on the server does not have a computer account for this workstation trust relationship.
 
 
Simply re-join the computer to the Active Directory domain and this will resolve the Direct Access error IPHTTPS interface creation failed 0x643.

Monday, September 22, 2014

Exchange Database file: '-546 The log file sector size does not match the sector size of the current volumn

When attempting to backup an Exchange 2007 server running on Small Business Server 2008 with Backup Exec 2010 R2, the Exchange backup component of the selection failed with the following error in the job log.

Backup- \\SERVER\Microsoft Information Store\First Storage GroupV-79-57344-918 - Unable to complete the operation for the selected resource using the specified options.  The following error was returned when opening the Exchange Database file:  '-546 The log file sector size does not match the sector size of the current volumn. '
Backup- \\SERVER\Microsoft Information Store\Second Storage GroupV-79-57344-918 - Unable to complete the operation for the selected resource using the specified options.  The following error was returned when opening the Exchange Database file:  '-546 The log file sector size does not match the sector size of the current volumn. '


 This error occurred when backing up to a HP Tape Library attached to the server via SCSI.  When I attempted to do a disk based backup with Backup Exec 2010 R2 as a test, it worked successfully.

After leasing with Symantec, it turned out when backing up to tape, Backup Exec requires a temporary staging directory to be use during the backup job.  By default, Backup Exec selects the largest volume with the most available free space.  This cannot be a Removable USB Drive!

Backing up to disk does not require this staging area.

I had a 4TB USB Drive attached to the SBS 2008 server which was causing this issue.  To resolve the issue you need to manually specify what drive to use for staging by following these steps:

1. On the Exchange Server, click Start -> Run and type "regedit".  Click OK.
2. Within the Registry Editor, navigate to HKLM\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Engine\Exchange
3. Right click and add a new String Value, "OnHostTemp".
4. Right click and set the value of OnHostTemp to "C:\Temp"
5. Restart the Backup Exec Remote Agent for Windows Systems service on the Exchange Servers.
6. Run tape backup job of Exchange Resources from media server.

I selected H:\Temp as it had the most available free space on my SBS 2008 server.


After putting this RegKey in place it resolved the issue.

Outlook 2010 and Exchange 2013 Users Prompted for Authentication

Over the past months I have had customers complain about Outlook 2010 users getting prompted for username/password when moving to Exchange 2013.  In previous versions of Exchange server such as 2003, 2007 and 2010, users connected to the Exchange server using RPC or Outlook Anywhere.  Exchange connectivity for clients has changed significantly in Exchange 2013 and now only Outlook Anywhere is supported, with the Exception of MAPI over HTTP for Exchange 2013 SP1 only when using Outlook 2013 SP1 clients.

The following table summaries the connection methods available for the various versions of Outlook and Exchange Server.


The default method of Outlook connecting to the Exchange server has always been to use RPC for
internal connections and Outlook Anywhere for external connections.  As Outlook Anywhere was originally only designed to be used for external connections, the Autodiscover service in Exchange 2007 and 2010 only provided Outlook clients with one set of configuration parameters used for external connectivity.

The screenshot below displays the configuration output from Outlook Anywhere on a Exchange 2010 client access server.  Notice there is only one External Hostname for connectivity and one Client Authentication Method you can specify.


In Exchange 2013 we now have the ability to specify different hostnames and authentication methods based on if the client is internal or external.


The authentication type is very important:
  • NTLM Authentication will leverage the credentials you used when signing into Windows and result in the Outlook client automatically signing in without prompting for authentication.
  • Basic Authentication is clear text authentication which does not use your Windows credentials.  As long as the Basic Authentication is encapsulated within Secure Socket Layer will it be secure.
As a general rule of thumb you want to use Basic Authentication for external connections and NTLM Authentication for internal connections.  You can use NTLM externally as well however I have had issues with it passing through some firewalls and proxy servers on remote networks so I advise my customers to always use Basic Authentication for maximum supportability for remote connections.

Here is where things get a little tricky.  Outlook 2010 RTM only understands the External Autodiscover response for Outlook Anywhere, not the Internal response.  This is shown in the screenshot below, notice the Server address is my ExternalURL and the Authentication is Basic.

This means provided you have split DNS in place for the External FQDN used for connectivity "mail.company.com", your clients will connect but with Basic Authentication.  This will result in the Outlook clients being prompted for authentication.


As of Outlook 2010 SP1 and higher it supports the Internal and External Autodiscover response for Outlook Anywhere which I have displayed below in two screenshots as I needed to scroll down in the Test E-mail AutoConfiguration screen:

Note: I went straight to Outlook 2010 SP2 in the screenshot below.

 
 
To ensure Outlook clients are not prompted for Authentication, ensure they are set to use NTLM authentication.  If Outlook 2010 clients have not been service packed, they will always receive the External authentication method.

In summary make sure you have done the following:
  • Configured your External and Internal Authentication types correctly using the information provided above.
  • Upgraded your Outlook clients to the latest service pack
 I hope this post has been helpful.

Thursday, September 18, 2014

V-79-10000-11226 - VSS Snapshot error Microsoft Exchange 2007

An SBS Customer of mine running Exchange 2007 on Microsoft Windows Server SBS 2008 with Backup Exec 2010 R2 ran into a backup issue where their Exchange 2007 backups with GRT began failing.  The backups had been operational for over 3 years and suddenly started failing with the following errors.

Before I cover of the errors we were experiencing, it is important to note that during this time we also had disk problems with disks in a RAID5 array failing and requiring replacing.  The failing disks also resulted in some slight disk corruption and I needed to repair the Exchange database with eseutil /p.

The following errors were experienced:

- AOFO: Initialization failure on: "\\SERVER\Microsoft Information Store\First Storage Group". Advanced Open File Option used: Microsoft Volume Shadow Copy Service (VSS).
V-79-10000-11226 - VSS Snapshot error. The Microsoft Volume Shadow Copy Service (VSS) snapshot provider selected returned: "Unexpected provider error". Ensure that all provider services are enabled and can be started. Check the Windows Event Viewer for details.


In addition to the above Backup Exec error, the following Windows Application Event Logs were logged:

Log Name:      Application
Source:        VSS
Date:          9/18/2014 7:45:11 PM
Event ID:      12293
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Server.domain.local
Description:
Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {b5946137-7b9f-4925-af80-51abd60b20d5}. Routine details EndPrepareSnapshots({2c88e07d-a06c-4f6f-b826-b6e8bbfd3e10}) [hr = 0x8000ffff].
Operation:
   Executing Asynchronous Operation
Context:
   Current State: DoSnapshotSet


After researching VSS EventID 12293 it brought me to http://support.microsoft.com/kb/924262 which said the resolution was:

"To resolve this issue, you can delete a Snapshot copy that lets you continue with your present backup."

I went and deleted all Exchange VSS backups with diskshadow.exe by using the following commands:

unexposed e:

delete shadows all

Note: E:\ is the volume containing my Microsoft Exchange databases.


After cleaning up the existing shadow copies, Exchange backups resumed to normal.  I believe the problem was introduced due to the disk corruption problems we experienced.

Monday, September 15, 2014

Exchange 2007 Mailbox Import Export Issues with Outlook 2010

A customer of mine experienced two hard disks dying in a RAID5 array over a weekend which held their Exchange 2007 mailbox databases.  They also had no recent backup of the Exchange mailbox databases.

Luckily this customer only has approximately 20 users all who run cached Exchange mode within Outlook 2010.  As a result, their mail was stored locally on each workstation in their OST file.  I went around to each workstation and exported the users mailbox to a PST file.

Next I replaced the disks and rebuild the RAID5 array, created a new NTFS partition and started the Information Store.  This generated new black mailbox databases.

With Exchange 2007, you need to use the legacy Import-Mailbox cmdlet instead of the New-MailboxImportRequest cmdlet available in Exchange 2010 and Exchange 2013.  Import-Mailbox requires a 32bit computer running 32bit version of Outlook.  I downloaded the Exchange 2007 SP3 32bit installation which I installed on a 32bit Windows 7 workstation joined to the domain and installed only the Exchange 2007 Management Tools.

When attempting to import a mailbox it failed with the following error:

[PS] C:\Windows\system32>Import-Mailbox -Identity information -PSTFolderPath C:\pstfiles\info.pst -Verbose

VERBOSE: Import-Mailbox : Beginning processing.
VERBOSE: Import-Mailbox : Trying to open registry key 'Software\\Microsoft\\Windows\\CurrentVersion\\App Paths\\OUTLOOK.EXE'.
VERBOSE: Import-Mailbox : The default value of the registry key is 'C:\PROGRA~1\MICROS~1\Office14\OUTLOOK.EXE'.
VERBOSE: Import-Mailbox : The version of Outlook.exe is '14.0.6131.5000'.
VERBOSE: Import-Mailbox : Searching objects "information" of type "ADUser" under the root "$null".
VERBOSE: Import-Mailbox : Previous operation run on global catalog server 'JCC-SBS.domain.local'.
VERBOSE: Import-Mailbox : Processing object "domain.local/MyBusiness/Users/Exchange/Shared Mailboxes/Information".
VERBOSE: Import-Mailbox : Searching objects "jcc-sbs" of type "Server" under the root "$null".
VERBOSE: Import-Mailbox : Previous operation run on domain controller 'JCC-SBS.domain.local'.
VERBOSE: Import-Mailbox : Searching objects "JCC-SBS\First Storage Group\Mailbox Database" of type "MailboxDatabase" under the root "$null".
VERBOSE: Import-Mailbox : Previous operation run on domain controller 'JCC-SBS.domain.local'.

Confirm
Are you sure you want to perform this action?
Importing mailbox content from .pst file 'C:\pstfiles\info.pst' to mailbox 'Information'. This operation may take a long time to complete.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help
(default is "Y"):y

VERBOSE: Import-Mailbox : Ending processing.
VERBOSE: Import-Mailbox : [information] The operation has started.
VERBOSE: Import-Mailbox : [information] Initializing MAPI, loading library.
VERBOSE: Import-Mailbox : [information] Approving object.
VERBOSE: Import-Mailbox : [information] Logging on to the MAPI profile.
VERBOSE: Import-Mailbox : [information] Opening Exchange mailbox.

Import-Mailbox : Error was found for Information (info@domain.com) because: Error occurred in the step: Approving object. An unknown error has occurred., error code: -2147221233
At line:1 char:15
+ Import-Mailbox <<<<  -Identity information -PSTFolderPath C:\pstfiles\info.pst -Verbose
    + CategoryInfo          : InvalidOperation: (0:Int32) [Import-Mailbox], RecipientTaskException
    + FullyQualifiedErrorId : 39DE607E,Microsoft.Exchange.Management.RecipientTasks.ImportMailbox

VERBOSE: Import-Mailbox : [information] The operation has finished.


Most resolutions on the Internet for this problem is to simply run FIXMAPI.exe from a command prompt.  This however did not resolve my issue.  After further research, I found that two updates released by Microsoft for Outlook 2010 cause this problem:
  • KB2597090
  • KB2687623
Simply uninstall these updates from "Programs and Features" in Control Panel.  Make sure you enable "View Installed Updates" so that the updates come up in the list.

After uninstalling these patches, I rebooted the Windows 7 workstation and straight away I was able to import mailboxes and export.

 

Sunday, August 24, 2014

Unable to Delete Emails on Exchange 2010 Sent from Scanner

A customer of mine running Exchange 2010 SP3 with no Update Rollups had an issue where users were unable to delete emails sent from a scanner.  The issue was experienced in both Microsoft Outlook and Microsoft Outlook Web App.

The following screenshot shows Exchange 2010 Outlook Web App on Internet Explorer 11 where I have selected a bunch of emails all sent from a scanner at my customers office.


After selecting all emails and selecting the delete button, emails simply did not delete as shown in the following screenshot.


Note: The Outlook Web Access is running OWA Light as Exchange 2010 SP3 Update Rollup 3 is required for the full client to work correctly on Internet Explorer 11.

After investigating I discovered this was caused by a bug in Exchange Server which was first introduced in Exchange 2010 SP2 RU6 and was around until Exchange 2010 SP3.  The bug has been documented on the following Microsoft Knowledge Base article and matches the symptoms of my customer:

http://support.microsoft.com/kb/2822208

The resolution for this issue is to simply install the latest Update Rollup on the Exchange 2010 infrastructure.  As of this writing the latest Update Rollup is 6 for Exchange 2010 SP3 which is available from the following website:

http://www.microsoft.com/en-au/download/details.aspx?id=43101