Out Of Memory and Network Problems
If you suddenly can not connect to your server in the cloud for no apparent reason, it may be because it is running out of physical memory. In this article, I will discuss a few solutions to this problem.
By. Jacob
Edited: 2020-05-27 14:55
Network connectivity issues can be caused by a variety of different issues, some of which are not immediately obvious. If you are sure that this is not a firewall issue, and the problem is not on your part, then it could also be caused by important processes getting terminated by the OOM killer if the server is running low on memory—I learned this the hard way while setting up a small AWS Lightsail instance.
In this tutorial, I will focus on AWS, but the information will also apply for Azure, Google, and others.
If an instance is running out of memory, processes may get terminated by the OOM killer; this may be indicated by finding Out of memory (OOM) messages in the /var/log/syslog file.
How to solve the problem will depend on the circumstances. If this is just a basic LAMP setup (Linux + Apache + MySQL + PHP), then you may be able to decrease memory consumption by adjusting various settings.
Enabling a swapfile can also help avoid processes getting shut down due to running out of memory, but it comes at a cost of speed.
Changing the MySQL settings
MySQL tends to use a significant amount of memory, and is therefor a obvious place to optimize. One thing we can try is disabling performance_schema in the configuration file.
Note. The performance_schema table contains information about server execution, and can be used by developers to optimize queries by identifying bottlenecks. Etc. We should be able to safely disable it.
See also: https://dev.mysql.com/doc/refman/8.0/en/performance-schema.html
The file to edit would be /etc/mysql/conf.d/mysqld.cnf (Ubuntu), if using another system, alternative locations to look for this file include:
- /etc/mysql/my.cnf
- /etc/my.cnf
The place to edit would be the mysqld section (note the d at the end):
[mysqld]
performance_schema = 0
If the section and/or option does not exist, you can add just it yourself.
Again, be sure that you are adding it in the mysqld section of the file, otherwise it is not going to work, and might even cause errors.
Debugging Lightsail network problems
Network problems are not limited to Lightsail instances, and might also happen with the more expensive EC2 instances. For a simple low-traffic web server, Lightsail should actually be a decent choice, even when handling thousands of visitors per day. But, it might require a custom configuration to best use available resources.
As with EC2, there is also ways to debug problems with Lightsail instances. For example, from the instance management page, we may check out the Matrics Graphs to see when exactly the problem occurred.
In my own case, I found a problem by observing the Instance status check failures; using this, I was able to see exactly when the problem started, and then I simply investigated my syslog around the time the problem occurred, to learn exactly what happened.
Sometime around the 20 of may, my server stopped responding to both HTTP and SSH traffic. Around this same time, I noticed in my /var/log/syslog, that I had out of memory errors:
May 20 13:13:13 ip-172-26-2-231 kernel: [117359.600102] Out of memory: Killed process 11140 (mysqld) total-vm...
You can read log files using zgrep for .gz files and grep for .txt files. The following would return entries for May 20:
zgrep 'May 20' /var/log/syslog.2.gz
Tell us what you think: