Often, blog titles are sensationalised and designed to draw the readers attention. In September 2023, we did actually observe the behavior described in the blog title. Windows Defender Exploit Guard, specifically Attack Surface Reduction (ASR) rules, were responsible for causing an issue which prevents Google Chrome from being updated successfully by various Remote Monitoring and Management (RMM) solutions. Its not the first time we have seen ASR rules have low confidence on a specific processes behavior that it had previously been OK with.
Its quite apt that something scary was afoot given its October and rapidly approaching Halloween👻🎃 . You may have been affected by this issue, without realizing it, and now Google Chrome cannot be updated by your RMM tool. Read on!
What Happened?
We started to get multiple reports in September that Google Chrome was failing to update. Soon, momentum was building in various forums with other’s reporting similar issues. During an update, the Google Chrome installer failed but was able to deliver the latest Chrome binaries to disk. Unfortunately, the failure meant that the update was not reflected in Add/Remove programs. The had confidence the Google Chrome updates were fine, we tested them manually, something else was causing the update to fail.
Why was this bad?
Many RMM solutions rely on the Add/Remove program data that is stored in the system registry in order to determine applicable for a third-party software update. An example of that logic might be “If The DisplayName -eq ‘Google Chrome’ and the DisplayVersion -lt ‘118.0.5993.70’ then this update is applicable”. This data was not present any more in the registry. This meant those RMM solutions would never detect future Google Chrome updates as being needed, because there was no registry entry to evaluate applicability against.
Investigation Started
We started looking through log files and event logs and started to spot a pattern. The first clue was presented in the verbose MSI log for Google Chrome. Chrome_Installer.log can be found in C:\Windows\temp
[0912/180519.266:ERROR:move_tree_work_item.cc(90)] failed move C:\Program Files (x86)\Google\Chrome\Temp\source5400_1011282578\Chrome-bin\116.0.5845.188 to C:\Program Files (x86)\Google\Chrome\Application\116.0.5845.188: Access is denied. (0x5)
Our attention was drawn to the Access denied message. Sometimes this was also accompanied with an exit code of 1603 for the installer too – but not always.
Naturally, when you have been in this game long enough, the knee jerk reaction is to point to anti-virus products. So that is where we started looking next. Windows Event log was showing some pretty interesting activity, specifically for Defender ASR rules.
You can search these logs with a custom view applied to Microsoft-Windows-Windows Defender events. Paste the following snippet and save the XML to disk as ASR_Events.xml:-
Microsoft-Windows-Windows Defender/Operational,Microsoft-Windows-Windows Defender/WHC1121,1122,50070FalseASR_Rules[System[(EventID=1121 or EventID=1122 or EventID=5007)]][System[(EventID=1121 or EventID=1122 or EventID=5007)]]
Now import that custom view into the Windows event viewer:-
We were seeing events like:-
Microsoft Defender Exploit Guard audited an operation that is not allowed by your IT administrator. 2023-09-15T20:40:31.838Z User: NT AUTHORITY\SYSTEM Path: C:\Windows\SystemTemp\GUMEA0.tmp\GoogleUpdate.exe Process Name: C:\Windows\System32\lsass.exe Target Commandline: C:\Windows\SystemTemp\GUMEA0.tmp\GoogleUpdate.exe /update /sessionid
GoogleUpdate.exe, used to update Google Chrome, was flagged by Windows Defender Exploit Guard against the process lsass.exe
lsass.exe
We were getting somewhere Dr Watson. Why would GoogleUpdate.exe be trying to access lsass.exe? Before we jump down that hole, let’s understand why it was flagged, what is so special about lsass.exe?
Lsass stands for Local Security Authority Subsystem Service. Lsass is responsible for enforcing the security policy on the system, authenticating users, issuing security tokens, and handling passwords. If other processes are seen to be interfering with her, quite rightly that activity is interesting and possibly concerning for Windows Defender.
Let’s resume rabbit hole diving. Why was the Google Chrome updater accessing lsass?
If you have any software that enumerates the running processes list in windows, it will inevitably crawl across lsass.exe at some point. GoogleUpdate.exe, we presume, is looking for running apps to understand if they should be closed to be updated, removed, modified. As soon as GoogleUpdate.exe enumerated lsass.exe, Windows Defender Exploit Guard flagged it, and blocked it!
Specifically, the ASR rule responsible for the block was:-
“Block credential stealing from the Windows local security authority subsystem (lsass.exe)”
Why now?
GoogleUpdate.exe has been around for millennia, probably before life was breathed into Windows Visa. Why is Defender flagging it now? We don’t have insight into that, but we understand there could have been a confidence issue in a recent definition update. The security intelligence version definition that “seems” to have caused an issue is 1.397.972.0. The issue was quickly rectified in a follow-up definition update. Whatever happened, it looks like the issue is “not an issue” any more, normal service has resumed. But many people have Chrome left in a broken state.
Oh, and this isn’t the first time ASR rules have caused havoc. Remember the fiasco early in 2023 where program shortcuts were deleted?
Windows Defender – ASRFalsely blocking and removing applications : r/sysadmin (reddit.com)
Fixing Google Chrome
The simple approach here is to re-install Google Chrome. Many organisations deploy the Google Chrome MSI Enterprise Installer. The difficulty comes with targeting. Remember that the uninstall hive information, required to detect Google Chrome, is missing.
What are your options?
Blast it out to your devices
If you have the latest Google Chrome app ready to go with your RMM of choice, simply deploy it, as required, to your devices. Remember, unless you can dynamically build target groups based on file data, it is going to be difficult to target these broken installs accurately.
Dynamically target broken instances with Configmgr
Our mind springs to ConfigMgr, oh what a sweet thing. We can build query based collections! Building a Configuration Item/Baseline to target your devices seems logical. Using this method, you can create collections based on non compliant devices. Look for systems that contain the binary on disk but is missing the registry information. An example might look like this (Credit Andrew Jimenez):-
$Apps = (Get-ChildItem HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall) | Get-ItemProperty | Select-Object DisplayName $Apps += (Get-ChildItem HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall) | Get-ItemProperty | Select-Object DisplayName if ((Test-Path "$env:ProgramFiles\Google\Chrome\Application\chrome.exe") -or (Test-Path "${env:ProgramFiles(x86)}\Google\Chrome\Application\chrome.exe")) { if ($null -eq ($apps | Where-Object DisplayName -EQ "Google Chrome")) { Write-Output $false } Write-Output $true } else { Write-Output $true }
Summary
Deploying Google Chrome will fix the missing registry items. This is going to be a bit of a wrangle, depending on the RMM you have. There is no native way, in Intune for example, to target devices based on file/registry data. There are community solutions that can build custom inventory records from a device, invoked using a Proactive Remediation, and send it to log analytics. These are awesome, but not as intuitive as the Configuration baseline approach we see above.
Ultimately, we do not want this to happen again. So what choices do we have? The brave might choose to add and exception to their ASR rules for C:\Windows\SystemTemp\*\GoogleUpdate.exe – but this should be carefully considered, exclusions are not the safest approach.
The ASR rule “Block credential stealing from the Windows local security authority subsystem (lsass.exe)” was ultimately responsible. We have seen this ASR rule cause issues with other programs that enumerate the running processes list. Chances are, this may come back and bite us again. Regularly reviewing the audit events for Windows Defender Exploit Guard is a good idea. You may not even know these rules are enabled if a consultant enabled and tweaked some security baselines in Intune and left the party saying “Yup, you’re secure and good to go”.
The important thing “not to do” is move your energy of hatred of the situation towards a product and just “switch it off”. Windows Defender Exploit Guard is a fine defense product. Perhaps now is a good time to evaluate those policies and plan should this ASR rule bite again.
The very fact that you are reading this post indicates that you care about patching and you care about security. You are in good company. At Patch My PC we stand beside you and care equally. Keep an eye for more blogs and webinar deepdives from our team. If you want to know more about what we do, book some time with an engineer and we can show you how we integrate with ConfigMgr and Intune to make patching easy!