Avantgarde Technologies

<a href="http://www.avantgardetechnologies.com.au">Avantgarde Technologies</a>
Perth's IT Experts

Monday, May 2, 2016

Windows 7 Computers Rebooting During Day for Updates

A customer was having an issue where Windows 7 computers randomly rebooted during the day for Windows Updates without providing a prompt for users the option to postpone updates.  This was resulting in frustrated users with computers rebooting in the middle of sending important emails, word processing tasks etc.

We checked Group Policy Windows Update settings, all was configured correctly however computers still rebooted.

After troubleshooting further, we found that a deadline in WSUS was set to "Same day approval at 5:00AM".  This meant as a deadline was set at 5:00AM in the morning, as soon a computers received the update upon boot, they already missed the deadline and immediately installed without prompting users to postpone the reboot.

We removed the setting for same day approval and this resolved the problem.

Avantgarde Technologies, a leading IT Support Perth based company.

Wednesday, April 13, 2016

Windows 7 SP1 hanging on Checking for Updates

Trying to perform a simple task of installing Windows 7 x64 Enterprise with Service Pack 1 on some virtual machines in my lab to test a product.  Windows 7 SP1 comes with Internet Explorer 8 and is very out of date in most aspects for application testing.

After in building a few Windows 7 VM's from my ISO, all of them sat there hanging on "Checking for Updates" for hours.

Ugg... something I didn't have time for as I was trying to test something urgently for a customer.

After installing the latest Windows Update client from https://www.microsoft.com/en-us/download/details.aspx?id=49540 on each freshly built Windows 7 workstation, they then detected updates in 4 minutes and I was able to start patching.

Wednesday, March 16, 2016

Microsoft Word Performance Issues - KB3114717

A customer contacted me mid-February complaining of significant performance issues with Microsoft Word 2013 SP1 (32bit).  When users copied and pasted text, scrolled up and down a document or changed formatting Microsoft Word continuously hung and entered a not responding state.  In addition users sometimes experienced up a 60 second delay when typing from when characters appeared on the screen.

We did significant troubleshooting on the issue including
  • Disabled Anti-Virus products
  • Full malware scan using multiple AV engines
  • Disabled all non Microsoft services and applications from System Startup
  • Disabled all Microsoft Word Add-ins
  • Disabled graphics acceleration in Microsoft Word
None of these troubleshooting steps resolved the issue.

After talking to Microsoft, it turns out that Microsoft released a bad Windows Update KB3114717.  This update was released on the 9th of February and caused numerous performance issues with Microsoft Office.  After removing this update from all workstations, it resolved the issue.

Tuesday, March 15, 2016

Searching RBL Agent Logs on Microsoft Exchange

In this blog post I will show you a quick way to search through large amounts of Real Time Blacklists logs on an Exchange Server.  This article assumes you have RBL Providers in place on an Exchange server which can be enabled as per the following article:


Once RBL listing is turned on, you will have a bunch of log files under the following directory (provided Exchange was installed to the default C:\ directory):

C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\AgentLog

We want to track down which RBL provider blocked an email from a spammer with the from address of bqmppf@apremiertravel.com but we first need to identify what log file contains the entry.  To do this open a command prompt and navigate to the FrontEnd\AgentLogs directory.

Run the following command:

find /c "bqmppf@apremiertravel.com" *.log | find ":" | find /v ": 0"

After running this command we can see that it found one entry for bqmppf@apremiertravel.com in the file AGENTLOG20160313-1.LOG.

Open the log and search for the email in question.

We can see this email was blocked by zen.spamhaus.org RBL provider.

Hope this post was helpful. 

Wednesday, March 9, 2016

Remove Run menu from Start Menu - Policy Bug

Today I stumbled across a bug with Windows 8.1 / 2012 R2 with the Group Policy setting:

User Configuration ->Policies -> Administrative Templates -> Start Menu and Taskbar-> Remove Run menu from Start Menu

I had this policy applied to a Remote Desktop Session Host through Loopback in a user session lockdown policy.

When this policy was applied, I could not access any network share on my 2012 R2 file servers and received the following error message:

Accessing the resource \\fileserver\share has been disallowed.

I could received the same error when attempting to access the file server shares through a DFS namespace.

After removing this policy setting from my RD Session Hosts, the issue was resolved.

Monday, February 15, 2016

Windows cannot move object because Access is denied

I came across an interesting error today when attempting to move a security group to a new location in Active Directory.

Windows cannot move object because Access is denied

This issue was caused by "Protect object from accidental deletion"

By default this feature is only enabled on organisational units however in this environment, some groups and user objects had this feature enabled as well.  I believe this was manually enabled by a sysadmin in the past.

Tuesday, January 26, 2016

Why my Domain Password Policy Not Applying?

Back in 2009 I published a very popular article "The Low-Down on Password Policies" which has been viewed by thousands of IT Professionals and referenced by application vendors in online documentation such as SysOp Tools Software.


In this post we are going to talk about password policies further and cover off what appears to be a bug but is actually "by design".

My customer had a handful of domain controllers with a single 2008 R2 domain controller and three Server 2012 R2 domain controllers.  The PDC Emulator resides on Server 2008 R2.

The Server 2008 R2 domain controller was applying the password policy correctly however the 2012 R2 domain controllers were not (or so I thought).

Running an rsop.msc on the 2008 R2 domain controller (the PDC) shows the policy being applied from the Default Domain Policy.

 The 2012 R2 domain controllers the resultant set of policy displayed no policies being applied.

The same was experienced running an "gpresult /v" on the 2008 R2 or 2012 R2 domain controllers.

"gpresult /v" on 2008 R2:

"gpresult /v" on 2012 R2:
The account policies above are the domain Kerberos policy, not the password policy.
The password policy simply did not apply to the 2012 servers.  After further investigation in my test lab, I saw that only the domain controller running the PDC emulator displays the password policy when performing a Resultant Set of Policy.
This means every domain controller in a domain will not display the password policy from a resultant set of policy apart from the primary domain controller.
How do I check if the password policy is applying correctly on my domain controllers?
There are two commands which check the password policy:
  • net accounts (checks local password policies on a server)
  • net accounts /domain (checks the domain password policy on a server)
Domain Policy always wins over a local policy.
Computer Role: Backup means it is not a Primary Domain Controllers (PDC).
So in summary... if you see a password policy not applying to a domain controller when you check Group Policy, this is normal behaviour and is by design unless the server is the PDC emulator.

Thursday, January 14, 2016

Exchange 2013 - Could not find any available Global Catalog in forest

I was contracted to redesign a companies AD Sites and Services Topology - it was never setup correctly and despite being a 500 user organisation with 13 branch sites, they were still running of the "Default-First-Site-Name" which is generated automatically by Active Directory for a new domain.

As part of the new design, I updated the Default-First-Site-Name to a name which reflects their main datacentres then went through the process creating the additional site objects, site links and subnet objects.

After renaming Default-First-Site-Name I also updated the AutodiscoverSiteScope on the Client Access Servers in the Exchange 2013 cluster to reflect the new site name (as required for correct site SCP lookups).

After approximately 30 minutes, the IT Department complained they were no longer able to work on Exchange 2013 servers - all commands in the Exchange Management Shell failed with:

Could not find any available Global Catalog in forest

Oh dear!

After a quick investigation, the issue was only isolated to Exchange 2013 management tools and Outlook clients were not affected by the Site Object rename.

In order to force Exchange Server to redetect Active Directory Sites and Services topology, a restart of the "Microsoft Exchange Active Directory Topology" service is required on all Exchange servers.  Unfortunately almost every Exchange Service is dependent on this service!

As a result, we needed to wait until after business hours where we rebooted every Exchange 2013 server in the cluster.

This resolved the problem.

Tuesday, January 5, 2016

Windows DNS Forwarder Population

A customer contacted me today asking why when they promoted all these domain controllers, they had old DNS forwarders automatically configured on each server.

When DCPROMO installs the DNS Server service it also activates, by default, the auto-configuration of the DNS Server service. This auto-configuration process configures the forwarders list, the root-hints and the resolver, among other things, like creating the zones if required.

During the automatic configuration of the Forwarders, the following process occurs:
  1. Try to copy the forwarders list from a peer DNS server. A peer DNS server is any DNS server that has a copy of this DC domain’s zone. To get the peer server list the process queries for the NS list of the domain’s zone and then contacts each server returned on the list until it finds one from which it can copy the forwarders list. Once the process finds a peer from which it can copy the forwarders list it skips the next step. If no peer is found (because the NS query returned empty), none of them could be contacted, or none of them has forwarders configured, then move to step #2.
  2. If the previous step was not able to provide a forwarders list, then use as forwarders all the DNS Servers that are currently listed in the resolver for all the adapters, without any specific order.
  3. If none of the previous two steps can provide a forwarders list, then the new DNS server will not have forwarders configured.
If you have different DNS Forwarders configured for various sites on your network, the DNS server will automatically configure itself to one at random so make sure you check the forwarders after promoting a new server!

Wednesday, December 30, 2015

Data Deduplication Enhancement in Windows Server 2012 R2

Windows Server 2012 R2 has a new feature which I can see very handy in the real world especially with VDI environments which have lots of Virtual Hard Disk files (VHD's) of similar nature.

In Server 2012 R2, Data Deduplication is now supported on VHD data stores.  This was not supported with the initial release of Server 2012.

Data Deduplication is also supported on Cluster Shared Volumes (CSVs) with file servers configured in scaled-out for high availability.

For companies that run a Microsoft-based VDI pool with multiple hosts, Data Deduplication can reduce the storage requirements of the VDI environment up to 90%.

Sunday, December 27, 2015

The Dirty Little Secret about P2V Migration with System Center Virtual Machine Manager

Physical to Virtual Migration has been around for a long time ever since companies started making the transition to Virtualisation as a standard back in 2008 with the release of VMware ESX 3.x quickly followed by 4.x and vSphere.

There a many tools on the market for Physical to Virtual migration of machines with the most common being "VMware vCenter Converter: P2V Virtual Machine Converter", "Microsoft Virtual Machine Converter 3.0" and the handy little tool from sysinternals "Disk2vhd".

In the brand new shiny System Center Virtual Machine Manager (VMM) 2012 R2, this tool also supports Physical to Virtual migration of workstations as an easy transition to a virtual platform for physical servers.

However if you look at the fine print in the "prerequisites" you will see:

"Cannot have any volumes larger than 2040 GB"


What the @$%@!!!

Very disappointing seeming this is the latest release of VMM and this limitation is still around... this would trip up many companies who are still looking to virtualise that legacy file server or mail server sitting around on their network!

Saturday, December 26, 2015

Event Viewer Tasks

I just want to touch against a feature in Windows Server 2008 R2 - 2012 which I believe is very cool.  Windows Event Viewer has the ability to launch tasks automatically when a particular error occurs.  This is great for companies that do not have System Center (or similar) tools in the environment to perform remediation tasks when problems occur on server infrastructure.

The button in Event Viewer is called "Attach Task to This Event"

Clicking it we can see that it actually relies on the Task Scheduler service to monitor the event logs.

Select Start a Program then associate it with cscript.exe or powershell.exe to launch a script that performs remediation tasks whenever the event error reoccurs.  You can instruct your script to also notify administrators via email which is very easy with PowerShell using the Send-MailMessage cmdlet.

Monday, December 21, 2015

Common DCDIAG Error with NCSecDesc

When running a DCDiag on 2008 or 2008 R2 domain controllers, it is very common to see the following error when running a dcdiag.exe.

Starting test: NCSecDesc
   Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have Replicating Directory Changes In Filtered Set access rights for the naming context:
   Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn't have Replicating Directory Changes In Filtered Set access rights for the naming context:
   ......................... DC1 failed test NCSecDesc

This is caused on Active Directory domains which have not prepared Active Directory for read only domain controllers with "adprep /rodcprep".

Server 2012 / 2012 R2 domain controllers do not receive this error for NCSecDesc.

Also it is recommended you do not prepare you domain for RODC unless you intend to deploy Read Only Domain Controllers provided you have the requirement for specific branch locations from a physical security perspective.

Sunday, December 20, 2015

MSExchange ADAccess EventID 4027

A customer of mine contacted me today regarding an EventID 4027 from MSExchange ADAccess they wanted resolved.  This error was being generated on all Exchange 2013 servers in their cluster.

After looking into the issue, I found that legacy Cross-Forest configuration remained in configuration partition from a previous Cross-Forest Exchange Migration.  This is located under the Configuration --> Services --> Microsoft Exchange Autodiscover.

Simply remove the additional references to the legacy forest (the ones highlighted in yellow above).

Do not remove "Microsoft Exchange Online".  This is a default entry and is used when you create a Hybrid deployment with Office 365.

Monday, December 14, 2015

Outloook 2010 Starting in Safe Mode?

Microsoft recently released a bad Windows Update (KB3114409) which caused Outlook 2010 to start loading in safe mode for multiple clients of mine.  This update has recently been recalled by Microsoft due to the number of issues it caused.

In the event your company installed it across multiple workstations, you can quietly uninstall it across all computers.

One of the easiest ways to do this is creating a Startup Script with Group Policy and create a batch script with the following command:

C:\Windows\System32\wusa.exe /uninstall /kb:3114409 /quiet /norestart

Make sure the batch file is launched via a Startup Script and not a logon script.  Logon Scripts in Group Policy require users to have local administration rights to make system wide changes (something which is not best practice).  Startup Scripts will run under the SYSTEM account with administrative rights.

Sunday, November 29, 2015

Active Directory Topology Diagrammer Error

I'm running Windows 10 with Visio 2016 (both x64 builds).  I needed to draw out the topology from a customers Active Directory domain however when attempting to draw the topology using the free Microsoft tool I received the following error message:

Could not open the Visio Stencil !!!
ADTD can not continue with the drawing.
The current drawing will be canceled !

To resolve this issue, in Visio click File --> Options.
Open the "Trust Center" then click "Trust Center Settings".

Untick all Open and Save boxes as shown below:

After making this change the topology should generate without issues:


Thursday, October 29, 2015

Active Directory Issues - Network Drives Not Mapping

A customer of mine raised an issue in regards network drives not being mapped for users.  This includes drives mapped via Group Policy and Home Drives mapped via the NT4 Home Drive option of the Active Directory user account.

When users attempt to navigate to the UNC paths manually or map a drive manually it works as expected, however mapping network drives automatically upon logon simply did not work.

Also users were unable to navigate to the domain name "\\domain.local".  However users could navigate to "\\domain.local\netlogon" and "\\domain.local\sysvol".  Navigating to the domain root resulted in this error being generated:

\\domain.local is not accessible.  You might not have permissions to use this network resource.  Contact the administrator of this server to find out if you have access permissions.

Logon Failure: The target account name is incorrect.

This issue with not being able to navigate to the domain root UNC share occurred on all member workstations and servers throughout the organisation.  Domain Controllers were not effected by the issue.

The following three event logs were also found throughout the SYSTEM event log on all client workstations throughout the companies domain.

Log Name:      System
Source:        NETLOGON
Date:          26/10/2015 7:56:15 AM
Event ID:      5719
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      COMPUTER.domain.local
This computer was not able to set up a secure session with a domain controller in domain DOMAIN due to the following: 
There are currently no logon servers available to service the logon request. 
This may lead to authentication problems. Make sure that this computer is connected to the network. If the problem persists, please contact your domain administrator.  

Log Name:      System
Source:        Microsoft-Windows-Security-Kerberos
Date:          23/10/2015 4:02:46 PM
Event ID:      4
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      COMPUTER.domain.local
The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server candc1$. The target name used was cifs/domain.local. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (DOMAIN.LOCAL) is different from the client domain (DOMAIN.LOCAL), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

Log Name:      System
Source:        Microsoft-Windows-GroupPolicy
Date:          28/10/2015 6:18:58 PM
Event ID:      1006
Task Category: None
Level:         Error
User:          DOMAIN\UserAccount
Computer:      COMPUTER.domain.local
The processing of Group Policy failed. Windows could not authenticate to the Active Directory service on a domain controller. (LDAP Bind function call failed). Look in the details tab for error code and description.

Some computers were also receiving:

The processing of Group Policy failed. Windows attempted to read the file \\domain.local\sysvol\domain.local\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}\gpt.ini from a domain controller and was not successful. Group Policy settings may not be applied until this event is resolved. This issue may be transient and could be caused by one or more of the following: 
a) Name Resolution/Network Connectivity to the current domain controller. 
b) File Replication Service Latency (a file created on another domain controller has not replicated to the current domain controller). 
c) The Distributed File System (DFS) client has been disabled.

This GUID referenced the Default Domain Policy (the first policy in the domain).  There was nothing wrong with the Default Domain Policy on the customers network, policy was simply not applying to domain members due an issue in Active Directory.

One issue which I focused on was the Kerberos error "KRB_AP_ERR_MODIFIED".  I have seen this issue before when the same SPN was registered on at least two accounts. For example, a SPN was registered on two accounts: A and B. What happens is that KDC will generate a service ticket that may be encrypted with password of account A. Then, when the client sends that ticket to the service during authentication, the service may try to decrypt this using account B.

I searched the customers domain for duplicated SPN's using "setspn.exe -X" and found some however they were not related to the error received.  Also I have never seen the "KRB_AP_ERR_MODIFIED" error generated on EVERY domain member workstation/server in an Active Directory environment.

Looking further into the SPN's we decided to dump the entire domain with every object/attribute to a text file using:

ldifde -f out.txt -d dc=domain,dc=local

Generally SPN against a user account reference a member server on the network running a particular service account such as SQL  However as this issue was affecting the entire domain and KRB_AP_ERR_MODIFIED refers to duplicated SPN records, we looked to see if there were any SPNs set at domain level on an account by searching our output from ldifde for "host/domain.local".

We found an SPN set to the root of the domain from the search results.  Important content from output below is blurred to protect the privacy of the customer.

We went and removed the incorrectly setup SPN record from the problematic service account svc_adfs using Active Directory Users and Computers with Advanced Features turned on then forced replication with "repadmin /syncall /APeD".

After removing the incorrectly configured SPN, we purged the kerberos tickets off a workstation then attempted to start explorer at the root of the domain "\\domain.local".  We were able to navigate to this share successfully.

The issues with network drives not mapping on logon were also resolved.

The SVC_ADFS account was created as part of an AD FS deployment for federation with applications and Microsoft Cloud Services.  AD FS backend roll was installed on two corporate domain controllers and two proxy servers were deployed in a DMZ setup to process the authentication requests from external services.  This is the Microsoft Best Practice for corporate organisations under 1000 seats as it reduces the amount of servers required and provides high redundancy by leveraging NLB on both the backend and frontend AD FS servers as per:


I went over the engineers build documentation who was in charge of implementing AD FS and could not see how the SPN was set, he did not manually set it.

Hope this post helps someone who experiences this same issue.

Monday, October 26, 2015

Enable Firewall Logging on Windows

Are your packets being dropped by Windows Firewall?  Want an insight into what is going on?  Simply open local group policy on a workstation / server (gpedit.msc) or configure a GPO in Group Policy Management Console (GPMC).  Under Windows Firewall with Advanced Security, go to the general properties.  Select the profile --> Logging and enable Logging on the set profile.  The log file by default goes to:


Very handy for troubleshooting.

Exchange 2013 POP3 Proxy Inactive

A customer complained that POP3 was no longer working.  After looking into this, it turned out that PopProxy was Inactive on the Exchange 2013 server.  As to why it was inactive is unknown.

To start the PopProxy was challanging, generally you change ServerComponentState using the maintenance requester for most components.  However running the following command did nothing:

Set-ServerComponentState -State Active -Requester Maintenance -Component PopProxy -Identity AB-EXCH-01

To start the PopProxy component, I needed to use the Exchange 2013 Health API as a requester.

Set-ServerComponentState -State Active -Requester HealthAPI -Component PopProxy -Identity AB-EXCH-01

As shown below:

Exchange 2013 421 4.3.2 Service not active

A customer of mine upgraded an Exchange 2013 cluster node from Exchange 2013 CU7 to Exchange 2010 CU10.  After the upgrade, emails failed to come in on the cluster node with the following SMTP error being generated "421 4.3.2 Service not active".

This can be reproduced by simply telneting the faulty Exchange 2013 server.  After entering MAIL FROM: into the SMTP syntax, the error occurs and is shown numerous times throughout the receive connector protocol logs on the frontend transport stack.

After further investigation I found out that majority of the Server Components for the faulty Exchange Server were in an inactive state.

To bring the server back to an active state, ServerWideOffline was set to Active which resumes all services using a requester of Maintenance.  This was done with the following command:

Set-ServerComponentState -State Active -Requester Maintenance -Identity AB-EXCH-02 -Component ServerWideOffline

Note: ForwardSyncDaemon and ProvisioningRps is Inactive by default.

After running this command all Exchange 2013 components were back to an active state apart from components disabled by default.

Exchange 2013 Cluster Issues 0x80071736

A customer of mine went upgraded one node of a two node DAG from Exchange 2013 CU7 to Exchange 2013 CU10.  After installing the update on the first cluster node, they contacted us complaining complaining that the DAG was no longer available online in Failover Cluster Manager.

The following error was presented when attempting to bring the cluster online:

Failed to bring the resource 'Cluster Name' online.
Error code 0x80071736
The resource failed to come online due to the failure of one or more provider resources.

The resource it was complaining about was the Cluster IP address was unavailable (

In addition to this error, the following errors were logged in event viewer on a regular basis.

Log Name:      System
Source:        Microsoft-Windows-DistributedCOM
Date:          26/10/2015 8:45:01 AM
Event ID:      10028
Task Category: None
Level:         Error
Keywords:      Classic
User:          DOMAIN\administrator
Computer:      Exchange1.domain.local

DCOM was unable to communicate with the computer DAG1.domain.local using any of the configured protocols; requested by PID 6bc4 (C:\Windows\system32\ServerManager.exe).

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          26/10/2015 9:58:05 AM
Event ID:      1223
Task Category: IP Address Resource
Level:         Error
User:          SYSTEM
Computer:      AB-EXCH-01.domain.local
Cluster IP address resource 'Cluster IP Address' cannot be brought online because the cluster network 'Cluster Network 1' is not configured to allow client access. Please use the Failover Cluster Manager snap-in to check the configured properties of the cluster network.

These cluster nodes both had two interfaces:
  • Mapi Interface
  • Replication Interface

After patching one of the Exchange 2013 servers the MapiDagNetwork got disabled on the with IgnoreNetwork set to true on the DagGroupNetwork.  As this DAGNetwork contained the DAG cluster IP address (within the subnet), it forced the cluster offline.  To re-enable the cluster we simply needed to set IgnoreNetwork to false.

After setting IgnoreNetwork to false, we were able to manually start the DAG in FailoverManager by right clicking and clicking Bring Online.

Monday, October 12, 2015

Removing Large Amounts of Spam from Exchange Server Queue Database

A customer running Exchange 2010 experienced a large number of spam emails in the submission queue (over 80,000).

All the Spam Emails started with "ID#ALERT#", as a result we ran the following command to clean up any emails with "ID#ALERT#" in the subject.

Get-Message -ResultSize unlimited | Where-Object {$_.Subject -match "ID#ALERT#"} | Remove-Message

Due to the significant amount of spam, the command failed to remove the messages and just hung for hours.  However we could remove messages individually.

We could however list the emails with the following command:

Get-Message -ResultSize unlimited | Where-Object {$_.Subject -match "ID#ALERT#"}

This gave us an idea to write a foreach command to remove each email individually from a CSV file.  First we needed a detailed list with the identity of each spam email so we ran the following command:

Get-Message -ResultSize unlimited | Where-Object {$_.Subject -match "ID#ALERT#"} | select Identity > C:\output.txt

We then formatted the CSV file to ensure we specified a name for the Identity column.  As you see we added "Identity" to the top of the file.

We then ran the Remove-Message command multiple times for each Identity in the CSV file using the following command:

Import-Csv "C:\output.txt" | ForEach-Object {Remove-Message -Identity $_.Identity -confirm:$false}

This ran the Remove-Message command over 80,000 times for each message in the queue and was able to clean it up successfully.

Sunday, October 4, 2015

Websense Appliance Services Not Starting after Hard Shutdown

When a Websense v5000 or v10000 appliance is forcefully shutdown due to power loss or hard system failure, upon restart the Filtering service, Policy service, User service and Usage monitor can fail to start in the Websense Appliance Manager portal.  This is an issue I have seen more then once and as a result decided to do a write-up.

This issue occurs due to a number of temporary files which are not cleaned up (a process that occurs during a graceful shutdown).  To remove these files manually, we must connect to the appliance using an SSH shell session.

To connect to a shell session you need to login to Websense Appliance Manager first then under Administration --> Toolbox, click Technical Support Tools and find the passcode under "Websense Remote Access".  This is the password used for SSH which is randomly generated by the appliance.

Next login to the IP address of the Websense Appliance (the same IP you used for the Appliance Manager web interface).
The username is "websense-ts" and the password is the one obtained above.
Navigate to /opt/Websense/bin and remove all temporary p12 files which were not deleted due to an incorrect shutdown.
rm -f *.p12
 Also remove the journal.dat file under the /opt/Websense/bin using the following command:
rm -f 'journal.dat

After the temporary files have been removed, restart all websense services.  This can be done from the Websense Appliance Manager website or from the shell by running the following command:

./WebsenseAdmin restart

Now all the Websense services on the appliance have returned to a running healthy state.