Issue
One of our customers encountered a strange issue when running Rocky Linux 9 and LetsEncrypt.
For some reason despite their certificate being authorised and built inside of the system correctly the domain which was meant to use the certificate seemed to ignore it completely and revert to default self-generated system / apache certificates.
Their certificate was issued with:
certbot --apache
This was their output:
[root@sub conf.d]# certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Enter email address (used for urgent renewal and security notices)
(Enter 'c' to cancel): [email protected]
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Account registered.
Which names would you like to activate HTTPS for?
We recommend selecting either all domains, or all domains in a VirtualHost/server block.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: sub.mydomain.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Requesting a certificate for sub.mydomain.com
Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/sub.mydomain.com/fullchain.pem
Key is saved at: /etc/letsencrypt/live/sub.mydomain.com/privkey.pem
This certificate expires on 2023-05-23.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.
Deploying certificate
Successfully deployed certificate for sub.mydomain.com to /etc/httpd/conf.d/zabbix-le-ssl.conf
Congratulations! You have successfully enabled HTTPS on https://sub.mydomain.com
We were unable to subscribe you the EFF mailing list because your e-mail address appears to be invalid. You can try again later by visiting https://act.eff.org.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
* Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
* Donating to EFF: https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Looking in their config file we could see that the certificate is explicitly being called:
[root@sub ~]# cat /etc/httpd/conf.d/zabbix-le-ssl.conf | grep SSL
SSLCertificateFile /etc/letsencrypt/live/sub.mydomain.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/sub.mydomain.com/privkey.pem
Trying to restart apache does nothing either, the certificate remains untrusted in the web browser:
systemctl restart httpd
Resolution
As subtle as it is, the actually cause of the issue was the fact that sub.mydomain.com was being referenced within their /etc/hosts file. This is how their hosts file looked:
[root@sub ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
127.0.0.1 sub.mydomain.com sub
We changed this to look like the following:
[root@sub ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
Restarted apache:
systemctl restart httpd
The certificate error disappeared when loading the website inside of the web browser. Defining the FQDN inside of the hosts file forces local certificates to be used rather than generated ones (even if these are referenced in the apache vhosts configuration file).
If you are running a singular domain on a server we advise not referencing it inside of /etc/hosts or alternatively symlinking the LetsEncrypt certificates to those contained within:
/etc/pki/tls/certs/
Reference to these can be found inside of Apaches default SSL file which is usually here:
/etc/httpd/conf.d/ssl.conf
