What do you do when SCEP doesn’t work with Jamf Pro? First off, don’t panic. 99% of the time it’s a misconfiguration somewhere.
Always remember, the S in SCEP stands for Simple.
Prerequisites
- If you haven’t looked at any logs, at least look at Jamf Pro server logs and information provided by the Jamf Pro server:
- JAMFSoftwareserver.log (found in Jamf Pro by navigating to Settings -> Jamf Pro Information -> Jamf Pro Server Logs)
- Failed MDM commands for installing certificate/802.1x profiles
JAMFSoftwareserver.log
So you’ve got JAMFSoftwareserver.log and now you’re overwhelmed. Holy wall of text batman! What does this mean? What do I look for?
Open the log and search for “SCEP”. Ignore anything that doesn’t start with WARN, ERROR, or Caused By.
Examples:
- [WARN ] [Thread-157 ] [MFEventNotificationCenter] – com.jamfsoftware.scep.ms.MSSCEPChallenge@52a53e6 threw an exception while processing event “com.jamfsoftware.eventnotifications.events.SCEPChallengeRequested@43c2c171” with this object: com.jamfsoftware.eventnotifications.shellobjects.SCEPChallengeRequestShell@1e74ffb6
com.jamfsoftware.jss.core.exception.InvalidRequestDataException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target- Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
- Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
- [WARN ] [Thread-157 ] [MFEventNotificationCenter] – com.jamfsoftware.scep.ms.MSSCEPChallenge@52a53e6 threw an exception while processing event “com.jamfsoftware.eventnotifications.events.SCEPChallengeRequested@54697d94” with this object: com.jamfsoftware.eventnotifications.shellobjects.SCEPChallengeRequestShell@7e5e1e0c
com.jamfsoftware.jss.core.exception.InvalidRequestDataException: org.apache.http.conn.HttpHostConnectException: Connect to url.rubyraccoon.net:80 [url.rubyraccoon.net/10.10.10.69] failed: Connection timed out (Connection timed out)- Caused by: org.apache.http.conn.HttpHostConnectException: Connect to url.rubyraccoon.net:80 [url.rubyraccoon.net/10.10.10.69] failed: Connection timed out (Connection timed out)
- Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
- [WARN ] [Thread-157] [CEPChallengeConfiguration] – $MSSCEPCHALLENGE found for a challenge password but no event listeners returned a challenge. Setting to a blank string.
- [ERROR] [Thread-157] [InstallProfile] – Error getting SCEP challenge response from server
com.jamfsoftware.jss.exceptions.mdm.ScepChallengeBlankException: Setting a blank challenge, returning!
What do the logs mean?
- Invalid SSL or TLS Server certificate:
- SSLHandshakeException: PKIX path building failed
- unable to find valid certification path to requested target
- This could mean that IIS has the wrong server certificate installed for SCEP, or the server certificate is not trusted by Jamf Pro.
- Server certificates should either be added to the Tomcat keystore on the Jamf Pro server or should be publicly trusted
- This could also mean that a load balancer or reverse proxy is in front of the SCEP server and not configured to pass through.
- This means that Jamf Pro will see the reverse proxy certificate, not the SCEP server certificate.
- This can be fixed by enabling TLS Passthrough or otherwise allowing the connection to “pass through” the load balancer/proxy and use the SCEP server certificate.
- Jamf Pro cannot resolve the URL or reach the specified IP address:
- HttpHostConnectException: Connect to url.rubyraccoon.net:80 [url.rubyraccoon.net/10.10.10.69] failed: Connection timed out
- This could mean that a firewall, load balancer, or reverse proxy is not configured correctly and Jamf Pro can’t contact the SCEP server.
- This could also mean that the URL is not resolvable, the IP address can’t be reached, or that the SCEP server isn’t listening on the correct port.
- SCEP can listen on any port, but commonly uses port 80 for unencrypted communication and port 443 for encrypted communication.
- SCEP server will not or cannot process the SCEP challenge
- Error getting SCEP challenge response from server
- ScepChallengeBlankException: Setting a blank challenge, returning!
- This usually means one of two things:
- SCEP Password cache is full
- SCEP service account credentials are expired/wrong or the account is locked out
- Other errors that are harder to find in the Jamf Pro server logs:
- NDES has the wrong certificate template configured in the registry, or doesn’t have permission to use the specified certificate template.
- The certificate signing request is badly misconfigured and the certificate cannot be created or signed.
- If using Jamf Pro as a SCEP proxy, the SCEP signing certificate could be misconfigured and causing issues.
Sometimes the Jamf server logs just don’t show you anything useful. What do you do then?
IIS Server Logs and Event Viewer
So you’ve looked through the JAMFSoftwareserver logs and they just aren’t useful at all. What next? There’s two places to look now:
- IIS Server Logs – C:\inetpub\logs\LogFiles\. This will be covered below.
- Jamf Pro configuration profile logs OR Management History of a computer that can’t install a SCEP profile.
- These are simple and self-explanatory. They also often match up to an error in the JAMFSoftwareserver log that you ma have missed.
C:\inetpub\logs\LogFiles
These logs are… terrible, but they hold a ton of information.
2022-07-15 17:03:43 10.10.10.69 GET /certsrv/mscep_admin/ - 80 - 10.10.10.42 Apache-HttpClient/4.5.13+(Java/11.0.15) - 401 2 5 0
2022-07-15 17:03:43 10.10.10.69 GET /certsrv/mscep_admin/ - 80 RUBYRACCOON\scep_svc 10.10.10.42 Apache-HttpClient/4.5.13+(Java/11.0.15) - 200 0 0 284
2022-07-15 17:03:43 10.10.10.69 GET /certsrv/mscep/mscep.dll operation=GetCACaps&message= 80 - 10.10.10.169 CertificateService/1+CFNetwork/1325.0.1+Darwin/21.1.0 - 200 0 0 8
2022-07-15 17:03:43 10.10.10.69 GET /certsrv/mscep/mscep.dll operation=GetCACert&message=RUBYRACCOON-ISSUING-CA 80 - 10.10.10.169 CertificateService/1+CFNetwork/1325.0.1+Darwin/21.1.0 - 200 0 0 8
2022-07-15 17:03:46 10.10.10.69 POST /certsrv/mscep/mscep.dll operation=PKIOperation 80 - 10.10.10.169 CertificateService/1+CFNetwork/1325.0.1+Darwin/21.1.0 - 200 0 0 28Incredible right? So much text and hardly any sense of meaning. Let’s break it down. This is how IIS formats this log:
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
Not every field is used in every request. Unused fields are skipped, so you may see some fields mentioned below that are in some lines but not others.
2022-07-15 17:03:43– The first two blocks are just thedateandtimethat the request came in.10.10.10.69– This block is the IP address of the SCEP server (s-ip)GETorPOST– This is the HTTP method used in the request. (cs-method)/certsrv/mscep_admin/– This is the URL slug, http://url.rubyraccoon.net/certsrv/mscep_admin/ (cs-uri-stem)operation=GetCACaps&message=,operation=GetCACert&message=RUBYRACCOON-ISSUING-CA, oroperation=PKIOperation– This is HTTP query sent. (cs-uri-query)80– This is the port that the request is sent to. (s-port)RUBYRACCOON\scep_svc– This is the username of the account when authentication is required. (cs-username)10.10.10.42or10.10.10.169– This is the IP address of the client communicating with the server. (c-ip)Apache-HttpClient/4.5.13+(Java/11.0.15)orCertificateService/1+CFNetwork/1325.0.1+Darwin/21.1.0– This is the service sending the request. (cs(Referer))Apache-HttpClientis Jamf Pro authenticatingRUBYRACCOON\scep_svcvia the built-in Apache web client.CertificateServiceis the supplicant (the test Macbook Pro VM in this case) requesting and receiving the certificate via a built-in process.
401or200– This is the HTTP response code. (sc-status)- 200 = OK
- 401 = Unauthorized
- 403 = Forbidden
2 5 0or0 0 284– These are the substatus (sc-substatus), Win32 error codes (sc-win32-status), and then time taken (time-taken) respectively.- Substatus is often seen with 403 errors:
- 403.7 or 403.16 are the most common errors. See Microsoft documentation here: https://docs.microsoft.com/en-us/troubleshoot/developer/webapps/iis/www-authentication-authorization/errors-403-7-reject-client-certificate-rquest and here: https://docs.microsoft.com/en-US/troubleshoot/developer/webapps/iis/health-diagnostic-performance/http-403-forbidden-access-website .
- Win32 errors are a whole beast unto themselves. https://docs.microsoft.com/en-us/windows/win32/debug/system-error-codes?redirectedfrom=MSDN
- Time taken is exactly that, how much time is taken.
- Substatus is often seen with 403 errors:
Windows Event Viewer
I’ll stop you here. If you’re thinking of looking at Event Viewer, something has gone terribly wrong. There might be useful information here, but check elsewhere first. Occasionally, something important will actually be logged here though so we can’t overlook it.
- How do I use Windows Event Viewer to find relevant logs?
- Search “Event Viewer” in the Windows search box
- Expand “Custom Views”
- Expand “Server Roles”
- Click on either “Active Directory Certificate Services” or “Web Server (IIS)”
- Marvel at the lack of information.
- Occasionally though, something important will actually be logged here. Look for WARN and ERROR level logs primarily. The logs themselves are fairly self explanatory.
- Issues you might see in Event Viewer:
- SCEP does not have permission to use the specified certificate template.
- The specified certificate template does not exist.
- The certificate signing request is unable to be processed for some reason.
- IIS or NDES have catastrophically failed.
Conclusion
Hopefully this helps to demystify how to troubleshoot SCEP. If you have searched for everything everywhere and just can’t figure it out – it’s probably a group policy or security policy being applied to the Windows Server or IIS Server that SCEP/NDES lives on. Otherwise, the major takeaways should be:
- Check logs. Everything to do with SCEP is logged somewhere.
- If there’s nothing in JAMFSoftwareServer logs, check the IIS server logs.
- If there’s nothing in the IIS server logs, check Event Viewer.
- If there’s nothing in Event Viewer, it’s likely an issue with the Active Directory environment. Unfortunately you’ll have to figure that one out yourself.
- Almost everything that could go wrong with SCEP is a misconfiguration:
- Wrong permissions for the certificate template used by SCEP.
- SCEP server certificate not accepted by Jamf Pro or supplicants.
- NDES/SCEP server has a full password cache.
- Wrong URL or Port configured in Jamf Pro.
- Bad or uninformed networking decisions making the SCEP server unreachable or messing up it’s certificate.
- Misconfigured IIS/Active Directory environment causing undesired IIS behavior.
- Misconfigured configuration profile.
- Misconfigured Jamf Pro SCEP Proxy settings.
- Everything that can be reasonably fixed will have evidence found in logs.