Friday, August 25, 2017

Windows PE and Cisco ISE authentication 802.1x

Windows PE  and Cisco ISE authentication

This blog entry is intended to assist you when implementing a Cisco ISE next generation network across the organisation.  Without ISE profiles the SCCM Task Sequence will fail to connect to Distribution Points and the MDT database.  UNC paths are blocked and network access is restricted.
Cisco ISE by design will restrict network access to prevent unauthorized clients from simply plugging their equipment into the network and being routed like a authorised client.
Computer and User Authentication (explain in detailed section)
Cisco ISE profiles should be implemented in two ways; Cisco ISE profiles via Group Policy for domain joined systems, and to bake ISE profiles into the SCCM Boot Image.  The guide below will explain how to implement both configuration setups.


Tutorial - WinPE

Microsoft has detailed the two XML files required to achieve User Authentication when in WinPE here

Create an XML called "EthernetLANProfile.xml" containing the following. The Thumb Print detailed within <TrustedRootCA> should reflect a trusted Third Party Cert; This ISE certification should also be deployed to all Domain Joined systems for GPO ISE Profiles (see below Tutorial - OS)

<?xml version="1.0"?>
<!-- Sample LAN profile: EthernetLANProfile.xml" -->
<LANProfile xmlns="http://www.microsoft.com/networking/LAN/profile/v1">
  <MSM>
    <security>
      <OneXEnforced>false</OneXEnforced>
      <OneXEnabled>true</OneXEnabled>
      <OneX xmlns="http://www.microsoft.com/networking/OneX/v1">
        <cacheUserData>true</cacheUserData>
        <authMode>user</authMode>
        <EAPConfig><EapHostConfig
          xmlns="http://www.microsoft.com/provisioning/EapHostConfig"><EapMethod><Type
          xmlns="http://www.microsoft.com/provisioning/EapCommon">25</Type><VendorId
          xmlns="http://www.microsoft.com/provisioning/EapCommon">0</VendorId><VendorType
          xmlns="http://www.microsoft.com/provisioning/EapCommon">0</VendorType><AuthorId
          xmlns="http://www.microsoft.com/provisioning/EapCommon">0</AuthorId></EapMethod><Config
          xmlns="http://www.microsoft.com/provisioning/EapHostConfig"><Eap
          xmlns="http://www.microsoft.com/provisioning/BaseEapConnectionPropertiesV1">
        <Type>25</Type><EapType
          xmlns="http://www.microsoft.com/provisioning/MsPeapConnectionPropertiesV1">
        <ServerValidation>
          <DisableUserPromptForServerValidation>false</DisableUserPromptForServerValidation>
          <ServerNames></ServerNames>
          <TrustedRootCA>1a 2b 3c 4d 56 78 90 aa bb cc dd ee ff 1a 2b 3c 4d 5e 6f</TrustedRootCA>
          </ServerValidation><FastReconnect>true</FastReconnect>
          <InnerEapOptional>false</InnerEapOptional><Eap
            xmlns="http://www.microsoft.com/provisioning/BaseEapConnectionPropertiesV1">
          <Type>26</Type><EapType
            xmlns="http://www.microsoft.com/provisioning/MsChapV2ConnectionPropertiesV1">
          <UseWinLogonCredentials>false</UseWinLogonCredentials></EapType></Eap>
          <EnableQuarantineChecks>false</EnableQuarantineChecks>
          <RequireCryptoBinding>false</RequireCryptoBinding><PeapExtensions>
          <PerformServerValidation
            xmlns="http://www.microsoft.com/provisioning/MsPeapConnectionPropertiesV2">false
          </PerformServerValidation><AcceptServerName
            xmlns="http://www.microsoft.com/provisioning/MsPeapConnectionPropertiesV2">false
            </AcceptServerName><PeapExtensionsV2
            xmlns="http://www.microsoft.com/provisioning/MsPeapConnectionPropertiesV2">
          <AllowPromptingWhenServerCANotFound
            xmlns="http://www.microsoft.com/provisioning/MsPeapConnectionPropertiesV3">true
          </AllowPromptingWhenServerCANotFound></PeapExtensionsV2></PeapExtensions></EapType>
        </Eap></Config></EapHostConfig></EAPConfig>
      </OneX>
    </security>
  </MSM>
</LANProfile>



Create another XML file called "EAP_UserData.xml" containing the Service Account User Credentials.

<?xml version="1.0"?> <!-- Sample EAP user data: EAP_UserData.xml" --> <EapHostUserCredentials xmlns="http://www.microsoft.com/provisioning/EapHostUserCredentials" xmlns:eapCommon="http://www.microsoft.com/provisioning/EapCommon" xmlns:baseEap="http://www.microsoft.com/provisioning/BaseEapMethodUserCredentials"> <EapMethod> <eapCommon:Type>25</eapCommon:Type> <eapCommon:AuthorId>0</eapCommon:AuthorId> </EapMethod> <Credentials xmlns:eapUser="http://www.microsoft.com/provisioning/EapUserPropertiesV1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:baseEap="http://www.microsoft.com/provisioning/BaseEapUserPropertiesV1" xmlns:MsPeap="http://www.microsoft.com/provisioning/MsPeapUserPropertiesV1" xmlns:MsChapV2="http://www.microsoft.com/provisioning/MsChapV2UserPropertiesV1"> <baseEap:Eap> <baseEap:Type>25</baseEap:Type> <MsPeap:EapType> <MsPeap:RoutingIdentity>onex\administrator</MsPeap:RoutingIdentity> <baseEap:Eap> <baseEap:Type>26</baseEap:Type> <MsChapV2:EapType> <MsChapV2:Username>SVC-Account-Name</MsChapV2:Username> <MsChapV2:Password>SVC-Account-password</MsChapV2:Password>
<MsChapV2:LogonDomain>SVC-Account-Domain</MsChapV2:LogonDomain>
</MsChapV2:EapType> </baseEap:Eap> </MsPeap:EapType> </baseEap:Eap> </Credentials> </EapHostUserCredentials>


The SCCM Boot image will need to contain the WinPE-Dot3Svc Optional Component.  However, from my experience this component doesnt work in Windows 10 version 1607 or 1703.
The component wizard completes without errors however, the Dot3Svc cannot start in WinPE. Micrsoft have detailed the issue here

The component will need to be manually installed; download the KB4025632 MSU file from Windows Update Catalog- ## https://www.catalog.update.microsoft.com/Search.aspx?q=KB4025632

The following two command will mount the Boot.wim Image and inject the Dot3Svc component.

Dism /Mount-Image /ImageFile:"C:\temp\WinPEx64\sources\boot.wim" /index:1 /MountDir:"C:\temp\WinPE_amd64-mount"

Dism /Add-Package /Image:"C:\temp\WinPE_amd64-mount" /PackagePath:"C:\temp\WinPEx64\windows10.0-kb4025632-x64_af86717e4eec306948b23cd1e82ff95640e51f5e.msu"

Before the Boot image is dismounted and copied to SCCM we also need to bake the ISE XML profiles into the Boot image.

Copy the EthernetLANProfile.xml & EAP_UserData.xml created earlier into the folder  "windows\system32"

C:\temp\WinPE_amd64-mount\windows\system32\

Dism /Unmount-Image /MountDir:"C:\temp\WinPE_amd64-mount" /commit

After installing the component copy the wim to your Boot image source location.
Add a custom prestart command.  Open the Properties of the Boot image and go to the customization tab, enable the prestart command and type the following three commands (enable Dot3svc, import Ethernet Profile , import User Auth Profile"

cmd /c powershell -noninteractive -command net start dot3svc & cmd /c netsh lan add profile filename=%SYSTEMDRIVE%\windows\system32\EthernetLANProfile.xml interface=*  & cmd /c netsh lan set eapuserdata filename="%SYSTEMDRIVE%\windows\system32\EAP_UserData.xml" alluser=yes Interface=*


 Within the SCCM console update the distribution point to inject the SCCM binaries and distribute the WIM to your PXE enabled distribution Points.

Once the Boot image is loaded and you have typed your WinPE password (if present) the Prestart command will launch (Custom Hook).  WinPE will run the commands in the TSConfig.ini file located on the root of the X drive.

Before the list of Task Sequences (if available) are presented you will see a command window appear starting the dot3svc service and configure the User Authentication ISE profiles created earlier.

If you wanted to check that ISE is running before kicking off the Task Sequence then you can:-

Hit F8 for a command prompt (if enabled in the boot image)
Type:
 PowerShell -command Get-Service dot3svc 

You should see the service status as running

Running       dot3svc            Wired AutoConfig





Thursday, August 3, 2017

SCCM SUP WSUS Pool keeps stopping or the server is unresponsive

SCCM SUP WSUS Pool keeps stopping or the server is unresponsive

Scenario: Our WSUS/SUP had become unresponsive and the decision to reinstall the server role was made. After the Site server had been reinstalled  I became aware that Windows Defender updates were failing to update (3 days old) and even though the updates were sync'd, downloaded, and deployed in SCCM the client was still unable to receive them.


Client Log analysis:

ScanAgent.log

ScanJob({999C9FFA-A463-4BE8-8771-67EE96D4140B}): CScanJob::OnScanComplete -Scan Failed with Error=0x80240440
ScanJob({999C9FFA-A463-4BE8-8771-67EE96D4140B}): CScanJobManager::OnScanComplete- failed at CScanJob::OnScanComplete with error=0x80240440



Update Deployment.log

Job error (0x80240440) received for assignment ({bf7a48e6-d220-4070-bb9b-ecc239107584}) action        UpdatesDeploymentAgent       
Updates will not be made available       

WUAHandler.log

Async searching of updates using WUAgent started.       
Async searching completed.       
OnSearchComplete - Failed to end search job. Error = 0x8024401c.        
Scan failed with error = 0x8024401c.        
Its a WSUS Update Source type ({3AAB6A76-CE2D-4E8A-9F11-741AE69677A2}), adding it.        
OS Version is 6.3.9600   
Existing WUA Managed server was already set (http://CMSUP.domain.co.uk:8530), skipping Group Policy registration.        

Added Update Source ({3AAB6A76-CE2D-4E8A-9F11-741AE69677A2}) of content type: 2        

WindowsUpdate.log

Report WARNING: CSerializationHelper:: InitSerialize failed : 0x80070002
AU        WARNING: Failed to get Wu Exemption info from NLM, assuming not exempt, error = 0x80070032
WS        WARNING: Nws Failure: errorCode=0x803d0006
WS        WARNING: Original error code: 0x80072ee2
WS        WARNING: There was an error communicating with the endpoint at 'http://CMSUP.domain.co.uk:8530/ClientWebService/client.asmx'.
WS        WARNING: There was an error receiving the HTTP reply.
WS        WARNING: The operation did not complete within the time allotted.
WS        WARNING: The operation timed out


Analysis Results:
Since I removed the role and put it back it was like the role had been installed for the first time.  Every client within SCCM (9k+) would need to do a Full Software Update & scan cycle. This would generate a heavy load on the ISS pool on the SUP server.  It is very important to configure the pool correctly otherwise the server will stop responding to clients as the client will receive "Service Unavailable" responses similar to DOS.

WSUS Pool Config
Queue Length: 25000
Limit (Percent) 70
Limit Action Throttle
Maximum workers (0 Numa) (1 Default) (2 if you have the server resource, not NUMA)
"Service Unavailable" Response TcpLevel
Failure Interval (Minutes) 30
Maximum Failures 60
Private Memory Limit (KB) 0


Conclusion
 Once we had the SUP role reinstalled synchronizing with SCCM (see WCM.log, Wsyncmgr.log) as well as the correct WSUS pool settings, we notice the server was still spiking CPU with ISS workers taking the full 70% limit without breaks.
Upon reviewing the Client Policy (within SCCM Client Settings) for the estate the "Software Update Scan schedule" was set to 1 day (not the recommended 7 days). This had the affect of overloading the WSUS server with 503 errors in the IIS log "service unavailable". As the schedule was every day the server could not get through the backlog before the whole process began again.
After setting the scan schedule to 7 days, and as clients were checking for policy every hour we notice a steady decline in CPU activity within the hour and all clients were able to complete there scan Software Update & scan cycle  and all clients were able to download Software updates.