Monday, October 31, 2016

Exchange Server Lost Trust to the Domain

A customer of mine running Exchange 2010 SP3 after a UPS had issues with Exchange loosing trust to the Active Directory domain.  This renders Microsoft Exchange unusable as all important Exchange configuration is stored within Active Directory.

Computer accounts like user accounts also have passwords.  These change every 30 days by default by Active Directory and member servers and workstations are automatically updated with the new password.  In the event the workstation or member server is not updated with the latest computer password; the trust fails and the machine displays the error “The trust relationship between the workstation and the primary domain failed” as shown in the screenshot below:


As a general fix for this issue, the PC is simply needs to be rejoined to the domain which works for most member servers and workstations.

Exchange however stores all its config in Active Directory and cannot be removed from a domain.

In the event you experience your Exchange Server loosing trust to Active Directory, you can re-establish trust using the following command on the Exchange Server after logging in with the local administrator account:

netdom resetpwd /server:AnyDomainController.yourdomain.local /userD:domain\administrator /PassworD:"youradminpassword"

Hope this post has been helpful.

Need IT Support with Microsoft Exchange in Perth?  Contact Avantgarde Technologies.

Direct Access Server not displaying Connection Statistics

A customer of mine had an issue with a Direct Access Server not displaying connection statistics.  My clients are connecting to the server without issues using IPHTTPS but we have no visibility to who is connected and for how long.

All connections and total bytes display 0 in both PowerShell and "Remote Access Management Console".

 
 
Also on the Remote Client Status page, no active clients are displayed.


This issue occurs when Windows Firewall is disabled on a Direct Access server.

Re-enable Windows Firewall and reboot the server.  After rebooting the server, wait 24 hours and you will notice statistics will start generating again.



Hope this post has been helpful.

Need IT Support or IT Services in Perth?  Contact Avantgarde Technologies.

Thursday, October 13, 2016

EventID 1006 MSExchangeFastSearch

A customer had an issue with Microsoft Exchange 2013 search not working.  Users received an error "Your search didn't return any results" in Outlook Web App.

 
This following error was generated in the Application Logs on the server.
 
Log Name:      Application
Source:        MSExchangeFastSearch
Date:          14/10/2016 1:10:14 PM
Event ID:      1006
Task Category: General
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Exchange.domain.local
Description:
The FastFeeder component received a connection exception from FAST. Error details: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://localhost:3847/. The connection attempt lasted for a time span of 00:00:02.0469288. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:3847.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:3847
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.ServiceModel.Channels.SocketConnectionInitiator.ConnectAsyncResult.OnConnect(IAsyncResult result)
   --- End of inner exception stack trace ---
Server stack trace:
   at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at System.ServiceModel.Channels.CommunicationObject.EndOpen(IAsyncResult result)
Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at System.ServiceModel.ICommunicationObject.EndOpen(IAsyncResult result)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.CreateProxy()
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.AcquireProxy()
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteServiceCall(IProcessingEngineChannel& serviceProxy, Action`1 call, Int32 retryCount)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.ExecuteAndReadPage(QueryParameters parameters, String outputName)
   at Microsoft.Exchange.Search.OperatorSchema.PagingImsFlowExecutor.GetHitCount(QueryParameters parameters)
   at Microsoft.Exchange.Search.Fast.ExchangeQueryExecutor.<>c__DisplayClass20.b__1f()
   at Microsoft.Exchange.Search.Fast.ExchangeQueryExecutor.RunUnderExceptionHandler[T](Func`1 call, IDiagnosticsSession session, String flowName)

 
This issue occurs when the "Microsoft Exchange Search Host Controller" service is in a stopped state.  My customer installed the latest Cumulative Update for Exchange 2013 and after the installation finished, the Search Host Controller was not set back to an Automatic.

Issues with Local Mailbox Moves on Exchange 2010 SP3

A customer running an Exchange 2010 SP3 UR15 multi-role server had issues moving mailboxes between databases.

 The error experienced was as follows:

Get-Mailbox "mailboxname" | New-MoveRequest -TargetDatabase "Mailbox Database Canada" -BadItemLimit 10

There are no available servers running the Microsoft Exchange Mailbox Replication service.
    + CategoryInfo          : NotSpecified: (0:Int32) [New-MoveRequest], NoMRSAvailableTransientException
    + FullyQualifiedErrorId : C7FE28BB,Microsoft.Exchange.Management.RecipientTasks.NewMoveRequest



Running Test-MRSHealth cmdlet to test the Mailbox Replication Service (responsible for processing mailbox moves) returns the following error on the environment.

RunspaceId : 4c95273c-3876-4369-86e3-1d9c90bc999e
Check      : ServiceCheck
Passed     : True
Message    : The Mailbox Replication Service is running.
Identity   : EXCHANGESERVER
IsValid    : True

 
RunspaceId : 4c95273c-3876-4369-86e3-1d9c90bc999e
Check      : RPCPingCheck
Passed     : False
Message    : The RPC endpoint for the Microsoft Exchange Mailbox Replication service couldn't respond: The call to 'net.tcp://exchangeserver/Microsoft.Exchange
             .MailboxReplicationService' failed. Error details: Access is denied.. --> Access is denied..
Identity   : EXCHANGESERVER
IsValid    : True


RunspaceId : 4c95273c-3876-4369-86e3-1d9c90bc999e
Check      : QueueScanCheck
Passed     : True
Message    : The Microsoft Exchange Mailbox Replication service is scanning mailbox database queues for jobs. Last scan age: 00:03:35.0822558.
Identity   : EXCHANGESERVER
IsValid    : True



Despite the error indicating Access is denied, all permissions were set correctly on the Mailbox Import Export role which was validated with:

Get-ManagementRoleAssignment -Role "Mailbox Import Export"

After further investigation, the following error was present:

 Log Name:      Application
Source:        MSExchangeIS Mailbox Store
Date:          13/10/2016 12:50:33 PM
Event ID:      7043
Task Category: IS/AD Interactions
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      Exchange.domain.local
Description:
The mailbox GUID of an external system mailbox ('Mailbox - SystemMailbox{1c3fa654-fe01-4041-a3a6-0444ca10f96c}') does not match the information in the Active Directory for the mailbox. The existing GUID ('8c0c971d-c5d7-495b-aff7-53356904789a: /o=Exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=SystemMailbox{1c3fa654-fe01-4041-a3a6-0444ca10f96c}') has been replaced with the expected GUID ('ed55827e-7cdd-466e-9421-a0dbce5f6486: /o=Exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=SystemMailbox{1c3fa654-fe01-4041-a3a6-0444ca10f96c}').

 

 The GUID of the SystemMailbox no longer matched the GUID of the Active Directory mailbox at the customer site.

To restore this, we ran "Setup /PrepareAD" from the Exchange 2010 SP3 setup media.  This restored the Exchange SystemMailbox association with Active Directory.

After running Setup /PrepareAD on the environment, we were able to move mailboxs again.

Monday, September 26, 2016

Cisco UCS Blades KVM Not Working

I needed to access a Cisco UCS Blade enclosure at one of my customers after a major ESX failure.  When attempting to access the KVM over the Java applicate for the Cisco UCSB-B200-M3 blade servers, I received the following error:

The viewer has terminated.
Reason: The network connection has been dropped.

 
After troubleshooting the issue for a while, I decided to downgrade my version of Java to an older build.  I tried the following build of Java in a Virtual Machine:
 
Java SE Runtime Environment 7u79
 
Success!
 
 There is an issue with the latest Java build and the Cisco UCS KVM application.
 

Sunday, September 18, 2016

0x80190194 OAB Error when Migrating to Exchange 2016

When migrating from Exchange 2013 to Exchange 2016, I encountered an error 0x80190194 when downloading the Offline Address Book on workstations.

0x80190194 The Operation Failed

 
0x80190194 is a very common error when downloading the OAB and there are many server side problems which can generate this error.

Exchange 2013 by default had the servers responsible for the Offline Address Book hard coded as a Virtual Directory as shown below.


In Exchange 2016, by default we no longer want to hard code the Virtual Directories and instead enable GlobalWebDistribution which allows the Autodiscover service to automatically select the best Virtual Directory for the distribution request.

To set this up, we want to ensure the VirtualDirectories attribute for each Offline Address Book is set to $null.  We also want to ensure GlobalWebDistribution is enabled so that Autodiscover can take care of it.

This is done with the following command.

Get-OfflineAddressBook | Where {$_.ExchangeVersion.ExchangeBuild.Major -Eq 15} | Set-OfflineAddressBook -GlobalWebDistributionEnabled $True -VirtualDirectories $Null

Following this, perform an iisreset.

The Offline Address Book now downloads correctly again on Exchange 2016.

Need IT Support in Perth?  Contact Avantgarde Technologies now.

Thursday, September 15, 2016

Bug Creating Frontend Transport Receive Connectors in Exchange 2016 CU2

There is a bug creating Frontend Transport Connectors from Exchange Administrative Center in Exchange 2016 CU2.

Currently it is not possible to select Hub Transport or Frontend Transport when creating a new connector as the option is greyed out.


In Exchange 2013, it is possible as shown below:


Microsoft are aware of this bug and will be fixing it in Exchange 2016 CU3.

Until this bug is fixed, you need to create Frontend Transport receive connectors from shell as follows:

New-ReceiveConnector -Name "Application Receive Connector" -Bindings ("0.0.0.0:25") -RemoteIPRanges ("10.1.10.10") -MaxMessageSize 50MB –TransportRole FrontendTransport -Usage Custom –Server Bentley-EXCH


Monday, September 12, 2016

Deploying Exchange 2016 into an Exchange 2013 Organisation with MAPI over HTTPS Enabled

This issue may be encountered when migrating to Exchange 2016 from Exchange 2013 when MAPI over HTTPS is enabled.  The default Exchange 2013 MAPI over HTTPS authentication settings set IIS and Internal Authentication methods as Negotiate and External as null.  This is shown below:


The Default Exchange 2016 MAPI over HTTPS authentication settings are configured as "Ntlm, OAuth and Negotiate"


Proxying MAPI over HTTPS connections between Exchange 2016 and Exchange 2013 requires NTLM be enabled.  The default Exchange 2013 MAPI over HTTPS authentication settings will cause Outlook connectivity issues when both Exchange 2016 and Exchange 2013 are in the same Active Directory site.

The error which is generated by the Exchange Remote Connectivity Analyzer in this configuration is as follows:
 
https://testconnectivity.microsoft.com/Images/Error.png
 
 
Testing the MAPI Mail Store endpoint on the Exchange server.
 
An error occurred while testing the Mail Store.
 
https://testconnectivity.microsoft.com/Images/Minus.gif
Additional Details
 
Elapsed Time: 1243 ms.
 
https://testconnectivity.microsoft.com/Images/Minus.gif
Test Steps
 
https://testconnectivity.microsoft.com/Images/Error.png
Attempting to log on to the Mailbox.
 
An error occurred while logging on to the Mailbox.
 
https://testconnectivity.microsoft.com/Images/Minus.gif
Additional Details
 
A protocol layer error occured. MapiHttpServiceCode: 1722
FailureLID: 56412
FailureInfo:

###### REQUEST [2016-08-28T13:10:48.4483314Z] ######

POST /mapi/emsmdb/?mailboxId=a9888e6b-81d6-4495-b4b0-bcda772e782f@avantgardetechnologies.com.au HTTP/1.1
Content-Type: application/octet-stream
User-Agent: MapiHttpClient
X-RequestId: 0d3ddde1-1147-4cbe-a50b-ee75d2d1319d:2
X-ClientInfo: dfba427f-ffa7-4003-981f-a676bced12eb:1
X-ClientApplication: MapiHttpClient/15.0.4420.1017
X-RequestType: Execute
Authorization: Negotiate [truncated]
Host: mail.avantgardetechnologies.com.au
Cookie: ClientId=PAVTTKRDEJLCYBAF9MA; MapiContext=MAPIAAAAAOms6aTto+TJjNSX3/zO/s/51OTc8cP72+rY4tPh2+ragKOSpJSilaWUoZWn7QEAAAAAAAA=; MapiSequence=0-WbZNDg==; X-BackEndCookie=a9888e6b-81d6-4495-b4b0-bcda772e782f=u56Lnp2ejJqBy5nGz8vMz8/SysyaxtLLysqd0p3Jz5vSyMzKzpqbzJ2ancicgYHNz87J0s/G0s3Iq87Mxc7Pxc7G
Content-Length: 172

--- REQUEST BODY [+0.128] ---
..[BODY SIZE: 172]

--- REQUEST SENT [+0.128] ---

###### RESPONSE [+0.416] ######

HTTP/1.1 200 OK
Transfer-Encoding: chunked
request-id: 7f93a99a-4a53-4866-a978-8de3671a1dd7
X-CalculatedBETarget: leeming-exch.at.local
X-ServerApplication: Exchange/15.00.1210.002
X-RequestId: 0d3ddde1-1147-4cbe-a50b-ee75d2d1319d:2
X-ClientInfo: dfba427f-ffa7-4003-981f-a676bced12eb:1
X-RequestType: Execute
X-PendingPeriod: 30000
X-ExpirationInfo: 900000
X-ResponseCode: 0
X-DiagInfo: LEEMING-EXCH
X-BEServer: LEEMING-EXCH
Cache-Control: private
Content-Type: application/octet-stream
Set-Cookie: MapiSequence=1-S1NbMA==; path=/mapi/emsmdb; secure; HttpOnly,MapiContext=MAPIAAAAAOms6aTto+TJjNSX3/zO/s/51OTc8cP72+rY4tPh2+ragKOSpJSilaWUoZWn7QEAAAAAAAA=; path=/mapi/emsmdb; secure; HttpOnly,X-BackEndCookie=a9888e6b-81d6-4495-b4b0-bcda772e782f=u56Lnp2ejJqBy5nGz8vMz8/SysyaxtLLysqd0p3Jz5vSyMzKzpqbzJ2ancicgYHNz87J0s/G0s3Iq87Mxc7Pxc7G; expires=Tue, 27-Sep-2016 13:10:19 GMT; path=/mapi; secure; HttpOnly
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
Persistent-Auth: true
X-Powered-By: ASP.NET
X-FEServer: LEEMING-EXCH
Date: Sun, 28 Aug 2016 13:10:19 GMT

--- RESPONSE BODY [+0.416] ---
..[BODY SIZE: 4195]
PROCESSING [@2016-08-28T13:10:48.8643314Z]
DONE [+00:00:00]
X-StartTime: Sun, 28 Aug 2016 13:10:19 GMT
X-ElapsedTime: 16

..[DATA SIZE: 4112]

--- RESPONSE DONE [+0.418] ---

###### REMOTE-EXCEPTION-INFO ######

Microsoft.Exchange.Rpc.RpcException: Connection must be re-established ---> Microsoft.Exchange.RpcClientAccess.ServerUnavailableException: Connection must be re-established ---> Microsoft.Exchange.RpcClientAccess.SessionDeadException: The primary owner logon has failed. Dropping a connection. ---> Microsoft.Exchange.Data.Storage.TooManyObjectsOpenedException: Cannot open mailbox /o=AT/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=Clint Boessenaa7. ---> Microsoft.Mapi.MapiExceptionSessionLimit: MapiExceptionSessionLimit: Unable to open message store. (hr=0x80040112, ec=1246) Diagnostic context: Lid: 55847 EMSMDBPOOL.EcPoolSessionDoRpc called [length=502] Lid: 43559 EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=256][latency=0] Lid: 52176 ClientVersion: 15.0.1210.3 Lid: 50032 ServerVersion: 15.0.1210.6003 Lid: 23226 --- ROP Parse Start --- Lid: 27962 ROP: ropLogon [254] Lid: 17082 ROP Error: 0x4DE Lid: 26937 Lid: 21921 StoreEc: 0x4DE Lid: 27962 ROP: ropExtendedError [250] Lid: 1494 ---- Remote Context Beg ---- Lid: 47536 Lid: 57936 dwParam: 0x20 Msg: MoMT Lid: 33360 dwParam: 0x21 Lid: 57384 StoreEc: 0x4DE Lid: 56872 dwParam: 0xFE Lid: 42712 StoreEc: 0x4DE Lid: 10786 dwParam: 0x0 Msg: 15.00.1210.000:Leeming-EXCH Lid: 1750 ---- Remote Context End ---- Lid: 26849 Lid: 21817 ROP Failure: 0x4DE Lid: 26297 Lid: 16585 StoreEc: 0x4DE Lid: 32441 Lid: 1706 StoreEc: 0x4DE Lid: 24761 Lid: 20665 StoreEc: 0x4DE Lid: 25785 Lid: 29881 StoreEc: 0x4DE
at Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)
at Microsoft.Mapi.ExRpcConnection.OpenMsgStore(OpenStoreFlag storeFlags, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, String& correctServerDn, ClientIdentityInfo clientIdentityAs, String userDnAs, Boolean unifiedLogon, String applicationId, Byte[] tenantHint, CultureInfo cultureInfo)
at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, Boolean unifiedLogon, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout, TimeSpan callTimeout, Byte[] tenantHint)
at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, ClientIdentityInfo clientIdentity, String applicationId, Byte[] tenantPartitionHint, Boolean unifiedLogon)
at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore, Boolean unifiedSession)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore, Boolean unifiedSession)
at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, IExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity, Boolean unifiedSession)
at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass1c.b__1a(MailboxSession mailboxSession)
at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, IExchangePrincipal owner, DelegateLogonUser delegatedUser, CultureInfo cultureInfo, String clientInfoString, IBudget budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessio
HTTP Response Headers:
Transfer-Encoding: chunked
request-id: 7f93a99a-4a53-4866-a978-8de3671a1dd7
X-CalculatedBETarget: leeming-exch.at.local
X-ServerApplication: Exchange/15.00.1210.002
X-RequestId: 0d3ddde1-1147-4cbe-a50b-ee75d2d1319d:2
X-ClientInfo: dfba427f-ffa7-4003-981f-a676bced12eb:1
X-RequestType: Execute
X-PendingPeriod: 30000
X-ExpirationInfo: 900000
X-ResponseCode: 0
X-DiagInfo: LEEMING-EXCH
X-BEServer: LEEMING-EXCH
Cache-Control: private
Content-Type: application/octet-stream
Set-Cookie: MapiSequence=1-S1NbMA==; path=/mapi/emsmdb; secure; HttpOnly,MapiContext=MAPIAAAAAOms6aTto+TJjNSX3/zO/s/51OTc8cP72+rY4tPh2+ragKOSpJSilaWUoZWn7QEAAAAAAAA=; path=/mapi/emsmdb; secure; HttpOnly,X-BackEndCookie=a9888e6b-81d6-4495-b4b0-bcda772e782f=u56Lnp2ejJqBy5nGz8vMz8/SysyaxtLLysqd0p3Jz5vSyMzKzpqbzJ2ancicgYHNz87J0s/G0s3Iq87Mxc7Pxc7G; expires=Tue, 27-Sep-2016 13:10:19 GMT; path=/mapi; secure; HttpOnly
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
Persistent-Auth: true
X-Powered-By: ASP.NET
X-FEServer: LEEMING-EXCH
Date: Sun, 28 Aug 2016 13:10:19 GMT
ServiceCode: 1722 Unavailable
Elapsed Time: 1243 ms.

To ensure all servers are configured correctly to proxy connections between Exchange 2013 and Exchange 2016, run the following PowerShell command:

Get-MapiVirtualDirectory | Set-MAPIVirtualDirectory -IISAuthenticationMethods Ntlm, OAuth, Negotiate


Hope this post has been helpful.

If you need IT Support in Perth, contact Avantgarde Technologies today.
 

Saturday, August 27, 2016

Certificate Warnings when upgrading to Exchange 2016

Because Exchange Server runs most of its configuration at an "Organisation Level" adding new Exchange Servers to an existing Exchange Environment can be a difficult challenge to ensure users get a seamless experience.  When adding new Exchange Servers to an organisation (such as Exchange 2016) in an existing Exchange 2013 organisation, the new Exchange 2016 server will immediately start advertising its SCP Autodiscover record and other internalURLs such as the MapiVirtualDirectory.

Whilst this does not cause direct issues to Exchange Resources, it will present certificate warnings on Outlook clients as the default Self Signed certificate will not be trusted on the Outlook clients.

Outlook Clients (if they are in the same Active Directory site) as the Autodiscover Site Scope will immediately start picking up the new Exchange server and communicating with it - hence generating certificate warnings such as the one below.

 
As an Exchange Administrator, your first task after building the new server is to immediately install a valid trusted certificate on your new Exchange server and update the Autodiscover SCP record on the new ClientAccessService with the Set-ClientAccessService cmdlet.  It is then very important to update all other URLs such as the MapiVirtualDirectory, Outlook Anywhere etc.

Changing the values for your new Exchange 2013/2016 servers however will not stop the certificate warnings from being displayed to users right away however.  Even though you update your Records, Outlook clients will continue receiving the old records for some time as shown in the screenshot below.


This occurs as when the Exchange 2016 server is first built, your Exchange 2013 servers will cache in the IIS AppPool these original records.  Your Exchange 2013 servers will continue to return via Autodiscover the record of the Exchange 2016 FQDN that does not match the name on the digital certificate.

To force your Exchange 2013 servers to start forcing the correct name immediately, an iisreset is required on all Exchange 2013 servers in the same Active Directory site as the new Exchange 2016 server.  This will cause a slight disruption for users.

See the issue?
  • As soon as your new Exchange 2016 server is installed, users will begin getting certificate warnings.
  • To quickly update the certificate and names of the Exchange Web Services, the iisreset on the Exchange 2013 servers will cause a slight outage.
Make sure you plan for this in your Exchange 2016 rollout.  Let users know in advance to ignore the certificate warning which will be displayed after the first Exchange 2016 server is built.  This will reduce the load on your companies service desk.

Wednesday, August 10, 2016

The Danger of the Local Administrator Account

The local administrator account resides on every Windows Server and is usually in an enabled state.  This account is a major security vulnerability and is commonly prone to hacking attempts.

Security flaws with this account include:
  • This account cannot be locked out and does not adhere to local or domain account lockout password policies.  This allows brute force attacks to be conducted against the account.
  • The local administrator account is a well known SID, it always begins with S-1-5- and end with -500.  There are also tools allowing you to login with a SID rather then an account name so an attacker could launch a brute force without knowing the account!

Quote from Microsoft https://technet.microsoft.com/en-us/library/jj852165.aspx"

"The built-in Administrator account cannot be locked out no matter how many failed logons it accrues, which makes it a prime target for brute-force attacks that attempt to guess passwords. Also, this account has a well-known security identifier (SID), and there are non-Microsoft tools that allow authentication by using the SID rather than the account name. Therefore, even if you rename the Administrator account, an attacker could launch a brute-force attack by using the SID to log on. All other accounts that are members of the Administrator's group have the safeguard of locking out the account if the number of failed logons exceeds its configured maximum."

If security if your top concern, my recommendation is to disable this account and always create a new Administrator account regardless if it is the default domain Administrator account or default local Administrator account.

Need IT Support in Perth, give me a call now on 08 9468 7575

Thursday, July 21, 2016

Error downloading unsupported File Types with IIS

A customer of mine was complained that when users attempted to download unknown file types, they received a "404 - File or directory not found" error message.

The file type my user was attempting to download was a ".indd" file.


This error is caused when the filetype is not a supported "MIME" an IIS Web Server.

In the event you want to add a new unknown file type under the IIS Website, click MIME Types.


Add the unknown file extension users are attempting to download.


After adding the mime, do an "iisreset".

Users will now be able to download the unknown file type.

Wednesday, July 20, 2016

DFSR Event ID 6404 - The Replicated Folder has an Invalid Local Path

I was attempting to add a new DFSR node to an existing DFSR Replication Group consisting of 17 file servers.  When adding the server the following error was experienced.

DFS Replication cannot replicate the replicated folder REPLICATEDFOLDER because the local path E:\replicationgroup is not the fully qualified path name of an existing, accessible local folder. This replicated folder is not replicating to or from this server. Event ID: 6404


This particular error is generally experienced when people attempt a non NTFS volume such as ReFS to a DFSR replication group as documented on the following forum.

https://social.technet.microsoft.com/Forums/windowsserver/en-US/8860b354-3bf2-43ed-9acf-07fa43931046/dfsr-replication-cant-be-used-on-a-refs-volume?forum=winserver8gen

 In my case, I experienced this error with DFSR setup on NTFS volume on a new branch server.

The customer moved the DFSR node from one office, to another office.  The C:\ "SYSTEM" volume was formatted and reloaded with a fresh copy of 2012 R2 with latest patches, however the E:\ "DATA" volume was not formatted.  As a result it has its legacy DFSR Database, Staging data etc.

To clean up the staging data I created a new directory in C:\ called "empty":

C:\Empty

I then ran the following commands after stopping the DFSR Replication service on the spoke server:

Robocopy "C:\Empty" "E:\System Volume Information\DFSR" /MIR

Followed by

rmdir "E:\System Volume Information\DFSR"

Starting the DFSR replication service resulted in the error above.

After playing around a bit more, I found that in addition to flushing all data in System Volume Information\DFSR you must also remove the DfsrPrivate link which points to the System Volume Information\DFSR sub directory for DfsrPrivate data.


After doing this, starting the DFS Replication service kicks of its normal DFSR Initial Sync shown by a state of 2:

Wmic /namespace:\\root\microsoftdfs path dfsrreplicatedfolderinfo get replicationgroupname,replicatedfoldername,state


State Values (Uninitialized, Initialized, Initial Sync, Auto Recovery, Normal, In Error), ValueMap (0, 1, 2, 3, 4, 5)}

Need help with DFSR In Perth?  Click for IT Support.

Tuesday, July 19, 2016

DFSR Backlog Count Not Reducing - Server 2012 R2

Microsoft recently released an update which causes DFSRS.exe CPU usage to hit 100%.

https://support.microsoft.com/en-us/kb/3156418

I wrote about this problem on the following blog post:

http://clintboessen.blogspot.com.au/2016/07/dfsr-service-100-cpu-usage.html

Since installing this update and removing this update, one of our spoke servers failed to replicate back to the primary hub server.  All data from the spoke server replicated correctly to the hub, however data from the hub was not propagating down to the spoke.

The backlog count continued to increase.
 


DFSR Hotfix https://support.microsoft.com/en-us/kb/2996883 was not installed on the affected hub server.

We knew there was no issue with our hub server as this was replicating successfully to 16 other DFSR spoke servers.

On the affected spoke server, I worked with the Directory Services team from Microsoft who made the following changes to the Network Interface.


C:\Windows\system32>netsh int tcp set global chimney=disabled
Ok.


C:\Windows\system32>netsh int tcp set global rss=disabled
Ok.


C:\Windows\system32>netsh int ip set global taskoffload=disabled
Ok.


C:\Windows\system32>netsh int tcp set global autotuninglevel=disabled
Ok.


C:\Windows\system32>netsh int tcp set global ecncapability=disabled
Ok.


C:\Windows\system32>netsh int tcp set global timestamps=disabled
Ok.


C:\Windows\system32>netsh advf set allp state off
Ok.


Microsoft then disabled the following Offload features on the HP Network Interface card on the spoke server:
  • IPv4 Large Send Offload
  • Large Send Offload V2 (IPv4)
  • Large Send Offload V2 (IPv6)
  • TCP Connection Offload (IPv4)
  • TCP Connection Offload (IPv6)
  • TCP/UDP Checksum Offload (IPv4)
  • TCP/UDP Checksum Offload (IPv6)
 
After disabling these components of the network interface card and restarting the DFS Replication service, the backlog count from the hub server to the spoke server slowly started decreasing.
 
TCP offload engine is a function used in network interface cards (NIC) to offload processing of the entire TCP/IP stack to the network controller. By moving some or all of the processing to dedicated hardware, a TCP offload engine frees the system's main CPU for other tasks

Sunday, July 10, 2016

DFSR Service 100% CPU Usage

I had an enterprise DFSR customer of mine with 17 File Server nodes complaining that CPU utilisation was at 100% on numerous DFSR nodes in the cluster.


In addition to the DFSR.exe process maxing out CPU on the file server, the following error was being generated on a regular basis:

Log Name:      Application
Source:        ESENT
Date:          7/07/2016 2:33:18 PM
Event ID:      623
Task Category: Transaction Manager
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SERVER.DOMAIN.LOCAL
Description:
DFSRs (1696)
\\.\E:\System Volume Information\DFSR\database_5AAC_EEEA_ACEE_C01D\dfsr.db: The version store for this instance (0) has reached its maximum size of 127Mb. It is likely that a long-running transaction is preventing cleanup of the version store and causing it to build up in size. Updates will be rejected until the long-running transaction has been completely committed or rolled back.
Possible long-running transaction:
 SessionId: 0x00000032FFE94EC0
 Session-context: 0x00000000
 Session-context ThreadId: 0x00000000000012BC
 Cleanup: 0
 Session-trace:
32997@2:32:18 PM


 
After speaking with the Directory Services team at Microsoft, this was due to a bug with KB3156418.  This was documented on the knowledge base website:
 
 
Symptoms
If you installed update rollup 3156418 on Windows Server 2012 R2, the DFSRS.exe process may consume a high percentage CPU processing power (up to 100 percent). This may cause the DFSR service to become unresponsive to the point at which the service cannot be stopped. You must hard-boot affected computers to restart them.

Workaround

To work around this problem, uninstall update rollup 3156418.

Status

Microsoft is aware of this problem and is working on a solution.

Make sure you do not install KB3156418 on any DFSR nodes until this issue has been fixed by Microsoft.
 

Wednesday, July 6, 2016

Bug with Windows 7 and Access Based Enumeration

I encountered an interesting bug with Windows 7 workstations with Access Based Enumeration enabled on a SMB Share and DFS Namespace running on Windows Server 2012 R2.

When a user tries to create a file or folder under a location which they have "Full Modify Rights" in Windows Explorer they receive the following error:

Drive Mapping refers to a location that is unavailable. It could be a hard drive on this computer, or on a network. Check to make sure that the disk is properly inserted, or that you are connected to the Internet on your network, and then try again. If it still cannot be located, the information might have been moved to a different location.


This issue occurs under the following circumstances:
  • Access Based Enumeration is enabled on a Network Share or DFS Namespace
  • If a Mapped Network Drive is created to the Share
  • The user is connecting from a Windows 7 workstation.

The Windows 7 client works under the following circumstances:
  • If the user creates a file from an application such as Microsoft Word (not Windows Explorer) using a mapped network drive to the folder share, it works corrctly.
  • If the user opens the UNC Path of the share \\server\share\folder, not via the mapped network drive it works correctly.
Note: If the user connects from 2008 R2, Windows 8.1 or Windows 10 it connects without issues.

When setting up Access Based Enumeration, the root folder should have:
  • List Folder / Read Data
  • Applies to "This Folder Only"
This ensures that users have the rights to list all folders at the base level folder for Access Based Enumeration, but requires additional rights to all sub folders hence the folders "hidden" as expected.

The root level permissions are shown below.  All sub folders are provided with full modify permissions for the respective security groups.

This works with 2008 R2, Windows 8.1 and Windows 10.

 
With Windows 7 clients they need additional permissions granted at the root level folder including:
  • Transverse folder / execute file
  • List folder / read data
  • Read Attributes
  • Read extended attributes
  • Read permissions
 
These extra permissions at the root level folder are only required if:
  • You run Access Based Enumeration
  • You have Windows 7 Clients on the network

Wednesday, June 29, 2016

IMAP4 Frontend with Exchange 2013

A customer of Avantgarde Technologies was having issues with IMAP4 on Exchange 2013.  In Exchange 2013 the IMAP Role has changed and how has a "front end" and "backend role".  This is shown by two services below.


The IMAP4 backend role listens on 127.0.0.1:143

If we telnet 127.0.0.1:143or "localhost" we hit the IMAP backend service.


If we telnet the server on any other IP address on TCP143 we hit the IMAP front end service.

This is what is confusing.  By default the IMAP4 frontend proxy server component is disabled even if the service is started.

If you telnet the server on its local IP address as shown below.


You will simply get a blank Telnet screen which will automatically close the session seconds after.


This happens as the ImapProxy frontend is disabled by default on the Server Component State of the server.


To enable it, use the following command.

Set-ServerComponentState -Identity Servername -State Active -Requester HealthAPI -Component ImapProxy

Need IT Support with Microsoft Exchange in Perth?  Click Here.

Tuesday, June 21, 2016

Modern Public Folders in Multi-Tenant Environments

Modern Public Folders is a new feature introduced in Exchange 2013 and also available in Exchange 2016 to allow companies to leverage Database Availability Groups and utilise log shipping as opposed to SMTP for replication providing numerous advantages which we will not focus on in this article.

If you new to Modern Public Folders, here is a good article to get you started:

http://www.msexchange.org/articles-tutorials/exchange-server-2013/planning-architecture/exchange-2013-preview-public-folders-part1.html

In this article, we will focus on Modern Public Folders in a multi-tenant environment.  We will be going through how to achieve multi-tenancy with Microsoft Exchange 2013 / 2016.

It is important to note, there are numerous methods for setting up Public Folder in a multi tenant environment.

In the example below we have:
  • One root Public Folder per tenant/company
  • One Public Folder Mailbox per tenant/company
  • One security group to represent all users from each tenant/company
  • Access based enumeration to each tenant/company can only see their root level Public Folder.
With Exchange 2013 and Exchange 2016, the first Public Folder mailbox created is always deployed as the Master Hierarchy Mailbox.  This mailbox is the only Public Folder mailbox with a writeable copy of the Public Folder hierarchy.  All additional Public Folder mailboxes created contain a read-only copy of the hierarchy.

When we are talking about Hierarchy, we are not talking about Public Folder content, only the folder structure which makes up the Public Folders.

First thing we need to create is a master Public Folder Hierarchy mailbox.  In a multi-tenant environment I generally recommend no content be placed in the master Public Folder Hierarchy mailbox and it only be used to maintain the writeable copy of the Public Folder Hierarchy.

As the first Public Folder mailbox created is always the Master Hierarchy, simply use the New-Mailbox command to create the mailbox.  I always recommend clearly naming this mailbox so it is easily identifiable as the Public Folder Mailbox Hierarchy mailbox.  This was done by giving it the name of "MasterHierarchy".


Next create a Public Folder mailbox for each tenant/company.  The intent here is all content for each company will be stored in their respective public folder mailbox.  In this example we will be using my company Avantgarde Technologies as an example tenant.  I'm using the naming convention CompanyPF for each respective Public Folder mailbox.


Next create a new Public Folder at the root "\" of the Hierarchy.  Make sure you specify the Public Folder mailbox you want to store the Public Folder in or by default Exchange will automatically pick any Public Folder mailbox which could be the Master Hierarchy mailbox or another tenants mailbox.

The name of the Public Folder specified below is the name the tenant will see in Outlook.


By default, all root public folders can be seen by all tenants.  To ensure no tenants can see the "Avantgarde Technologies" root level Public Folder, remove the Default user Access Rights as shown in the screenshot below.  This will ensure no one can see this Public Folder.


Lastly, ensure you have a Security Group containing all users from the tenant/company. Grant the group access to the root level Public Folder - I recommend Owner or PublishingEditor rights or refer to the following TechNet article about other Public Folder permissions you can grant here.


Only users of the "Avantgarde Users" security group will be able to see the root Public Folder Avantgarde Technologies and all other Tenants in the environment will be hidden to the Avantgarde Technologies employees.


To add additional tenants to the environment, repeat the process documented above.  Make sure you ensure that:
  • All root level public folders have the default user permissions removed straight away to protect privacy of each tenant on your Exchange environment.
  • When creating the root level public folders in PowerShell you manually specify the correct Public Folder mailbox or Exchange will pick one at random.
  • Consider using provisioning scripts to remove user error and protect yourself against privacy breaches.
All sub-folders created in Outlook will automatically append to the parents Public Folder mailbox.

One last thing I want to touch on is the "-DefaultPublicFolderMailbox" of the Set-Mailbox command.  Many people when they initially go about setting up Public Folders for multi-tenant Exchange they think about creating a public folder mailbox for each tenant then using "-DefaultPublicFolderMailbox" for all user mailboxes of each tenant.  Before writing this article I googled around to see if there was already an article similar, and saw people were attempting this incorrect method of deployment.  The reason this approach will not work is as mentioned earlier, all Public Folder Mailboxes have a "read only" copy of the entire Public Folder Hierarchy (meaning all Public Folders in the Exchange Organisation).  This means "yes they do have the Public Folder structure of other tenants/companies in the environment".  We want to ensure tenants only see public folders related to their company by locking down permissions to meet privacy reasons.

I hope this article has been informative for you and I would like to thankyou for reading.

Click here for IT Support in Perth with Microsoft Exchange