Why TLS/SSL certificates might break on AMI relaunch
With some network configurations, TLS/SSL might break when relaunching an EC2 instance from an AMI backup.
By. Jacob
Edited: 2021-11-30 03:46
I have been in the process of running some tests recently, and on this occasion I also wanted to try a stronger EC2 server for Beamtic. Unfortunately, this is not possible because of my reliance on a specific private IP.
The only way to do it, that I am aware of, is to completely terminate the old instance, and then launch a new instance from the AMI; but then I may as well just reconfigure my services, since that seems to be an easier and safer solution.
The primary symptom can be that your SSL/TLS certificate breaks when moving the server, and when you then try to visit your website, your browser will complain about an insecure connection on HTTPS; but, if you try HTTP, you just might get Apache's default page, because your IP-based configuration is no longer accurate.
I have had a few crashes doing the time I have been with AWS where I have been forced to relaunch my instances from an AMI; the breakage of my Let's Encrypt certificates used to confuse me a lot, because it just did not occur to me, at the time, what was causing it. I was just thinking that it did not make any sense that I had to renew my certificates doing relaunch, since, in theory, nothing should really be changed when you relaunch an instance from a fresh AMI backup.
Private IPs matter too
The reason why I depend on a specific private IP is because I have multiple services hosted, and some use a different network interface. The way to make these services only respond when someone connects on a specific interface, is to make them listen on the associated private IP.
When you connect an interface to an EC2 instance, it will both get a public IP, that is the WAN IP address, and a private IP that is only accessible on the "LAN". If you assign multiple interfaces (and thereby public IPs), each of them will also get its own private IP.
This is also possible with Apache Virtual Hosts (VHOSTs); of course, normally you you would just configure your VHOSTs to listen on all interfaces, that is *:80, because then you do not have to think about private interface IPs. But, if you wanted one site to be hosted on a different public IP, then it would be important to configure your Apache server using an IP-based v-host configuration.
Why?
Because otherwise some bumbling buffoon might run this from a terminal:
curl -H "host example.com" http://10.0.0.1/
And if "example.com" is then configured to listen on *:80 someone might also get a response, and thereby they would be able to deduce that the site is hosted on the same server even though the public DNS record points to a different IP.
I personally used this trick to track someone who was creating a lot of fake websites with no apparent purpose. Even though they deleted the domains in the DNS, the files were still on the server, and so I was able to deduce that they probably belonged to the same person.
Lesson learned
Even when you think you have EC2 figured out, some weird unforeseen problem suddenly occurs, and you have to think hard to solve it. Well, not so hard this time, but it felt awful anyway. I also got some actual paid work, plus a broken hard disk that I want to try data recovery on.
These things can be really time consuming, and it is not fun when time is the thing you feel that you do not have. No wonder many choose to simply host with a common shared hosting provider; not as fun and effective perhaps, but it works and it saves you time.
Enough little anecdotes. I got work to do :-p
Tell us what you think: