Issue Description and Background:
We have an enterprise voice deployment of Skype for Business Server 2015 integrated with Microsoft (MS) Teams Phone System running in a hybrid configuration with direct routing using a Ribbon Communications SBC. Inbound calls to our main phone number are answered by a MS Teams auto attendant (AA). However, that suddenly stopped working after we had incidentally recently deployed Direct Routing for MS Teams and afterwards we noticed the AA's were no longer reachable via on-premises Skype for Business Server 2015 via federation.
The following errors were in the Event Viewer logs on the Edge Server under Lync Server.
EVENT ID: 14428
TLS outgoing connection failures.
Over the past 66 minutes, Skype for Business Server has experienced TLS outgoing connection failures 31 time(s). The error code of the last failure is 0x800B0109(CERT_E_UNTRUSTEDROOT) while trying to connect to the server "sipfed.online.lync.com" at address [52.112.65.203:5061], and the display name in the peer certificate is "sipfed.online.lync.com".
Cause: Most often a problem with the peer certificate or perhaps the host name (DNS) record used to reach the peer server. Target principal name is incorrect means that the peer certificate does not contain the name that the local server used to connect. Certificate root not trusted error means that the peer certificate was issued by a remote CA that is not trusted by the local machine.
Resolution:
Check that the address and port matches the FQDN used to connect, and that the peer certificate contains this FQDN somewhere in its subject or SAN fields. If the FQDN refers to a DNS load balanced pool then check that all addresses returned by DNS refer to a server in the same pool. For untrusted root errors, ensure that the remote CA certificate chain is installed locally. If you have already installed the remote CA certificate chain, then try rebooting the local machine.
Multiple invalid incoming certificates.
In the past 147 minutes the server received 1 invalid incoming certificates. The last one was from host 52.112.67.51.
Cause: This can happen if a remote server presents an invalid certificate due to an incorrect configuration or an attacker.
Resolution:
No action needed unless the number of failures is large. Contact the administrator of the host sending the invalid certificate and resolve this problem.
We also captured a SIP trace of a test inbound call that was failing using the ClsLogging tool and noticed the following error.
TL_INFO(TF_PROTOCOL) [LYNCPOOL\btfls13fe]1F64.347C::06/13/2022-23:20:45.600.000133D2 (SIPStack,SIPAdminLog::ProtocolRecord::Flush:ProtocolRecord.cpp(261)) [2430505428]
Trace-Correlation-Id: 2430505428
Instance-Id: 18C69
Direction: incoming
Peer: LSEDGEPOOL.hq.biztechfusion.com:5061
Message-Type: response
Start-Line: SIP/2.0 504 Server time-out
From: "AHMASI XADEAN"<sip:12024891495;phone-context=PstnGateway_10.1.20.3@biztechfusion.com;user=phone>;epid=8D00A9C65B;tag=937ee3bfb9
To: <sip:4433600959;phone-context=PstnGateway_10.1.20.3@biztechfusion.com;user=phone>;tag=11BF95CFA045B9B046D298304CD0978C
Call-ID: 292742da-6c2f-47f0-b881-f4f6379787e9
CSeq: 5797 INVITE
Via: SIP/2.0/TLS 10.1.20.33:49948;branch=z9hG4bKAEF14639.AB56DBC7A7D924DD;branched=FALSE;ms-received-port=49948;ms-received-cid=1E4C00
Via: SIP/2.0/TLS 10.1.20.33:53216;branch=z9hG4bKd3415c7;ms-received-port=53216;ms-received-cid=26A800
Content-Length: 0
ms-diagnostics: 1010;reason="Certificate trust with another server could not be established";ErrorType="The peer certificate is not chained off a trusted root";tls-target="sipfed.online.lync.com";PeerServer="sipfed.online.lync.com";HRESULT="0x800B0109(CERT_E_UNTRUSTEDROOT)";source="LYNCSIP.biztechfusion.com"
ms-edge-proxy-message-trust: ms-source-type=EdgeProxyGenerated;ms-ep-fqdn=LSEDGEPOOL.hq.biztechfusion.com;ms-source-verified-user=verified
Server: RTC/6.0
$$end_record
The test call to 4433600959 is translated to +7006 (Teams Auto Attendant), then the Skype FE/Med Srvr sends the call to the Edge Srvr to get to O365 Cloud Voice Mail / Teams Auto Attendant, and gets a Cert error. Looks like it doesn't trust the O365 side server's cert, for some reason.
RESOLUTION:
The error messages was clearly indicating that our Edge Server was not trusting the SSL certificate being presented by Microsoft's Edge Servers (sipfed.online.lync.com). Therefore, we had to do the following to update the trusted root CA store on the Edge Server.
Open an elevated command prompt and enter the following commands:
# The following command downloads the Trusted Root CA's to a serialized store file:
Certutil -generateSSTFromWU WURoots.sst
# The following command opens the serialized store file (you can also simply double-click on the file to open it):
start explorer.exe wuroots.sst
Once the serialized store file is open, expand to the lowest level "Certificates" folder in the left navigation pane, and then you will see certificates for all of the trusted root certification authorities in the right action pane.
Next, you will need to open the Microsoft Management Console (right-click on start menu and select run, then type mmc and click OK) and add the Certificates snap-in for your Local Machine. Navigate to the Trusted Root Certification Authorities and right-click the Certificates subfolder in the left navigation pane.
Select "All Tasks" from the context menu and then "Import". Click next to progress through the wizard to browse and select the Microsoft Certificate Serialized Store (.SST) file you created and finish to complete importing all the root CA certs.
After importing the certs, federation between our Edge and Microsoft (sipfed.online.lync.com) began working immediately.
You may also determine whether or not you would like to update your registry on the server to automatically update the trusted root certification authorities by performing the following action.
Save the following to a text file and save it as "AuthRoot.reg":
BEGIN FILE [Do Not Include in the Text File]
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\SystemCertificates\AuthRoot]
"DisableRootAutoUpdate"=dword:00000000
"EnableDisallowedCertAutoUpdate"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\SystemCertificates\AuthRoot\Certificates]
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\SystemCertificates\AuthRoot\CRLs]
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\SystemCertificates\AuthRoot\CTLs]
END FILE [Do Not Include in the Text File]
Now, double-click on the AuthRoot.reg file to configure the registry accordingly.