Self heal a server using Monit

  • Monit will be used to automtically restart a service if it stops
  • This is the kind of service that makes you a miracle worker and makes Windows Systems Administrators jealous.
  • Monit is easy to setup and provides BIG pay-off for creating servers that tend to themselves so you can focus on more important things - like sleeping at 3am.

Install monit

apt-get install monit

Configure Monit

FILE: /etc/monit/monitrc

# check services every 2 minutes
  set daemon 120
  
  # logging 
  set logfile /var/log/monit.log
  
  # eMail alerts
  set alert devon@hubner.org
  set mailserver localhost
  
  # web interface
  set httpd port 2812 and
      use address localhost # only accept connection from localhost
      allow localhost       # allow localhost to connect to the server    
      allow admin:monit     # require user admin with password monit

FILE: /etc/monit/conf.d/custom-server.conf

# Apache HTTP Server
  check process apache2 with pidfile /var/run/apache2.pid
    start program = "/etc/init.d/apache2 start"
    stop  program = "/etc/init.d/apache2 stop"
    if failed host 127.0.0.1 port 80 protocol http
       and request "/" with timeout 10 seconds then restart
    if 5 restarts within 5 cycles then timeout
  
  # MySQL Server
  check process mysqld with pidfile /var/run/mysqld/mysqld.pid
       start program = "/etc/init.d/mysql start"
       stop program  = "/etc/init.d/mysql stop"
       if failed host 127.0.0.1 port 3306 protocol mysql then restart
       if 5 restarts within 5 cycles then timeout
  
  # SSH Server
  check process sshd with pidfile /var/run/sshd.pid
      start program "/etc/init.d/ssh start
      stop program "/etc/init.d/ssh stop
      if failed port 22 protocol ssh then restart
      if 5 restarts within 5 cycles then timeout
  
  # MongoDB
  # tcp port: 27017
  check process mongod with pidfile /var/lib/mongodb/mongod.lock
    start program "/etc/init.d/mongodb start"
    stop  program "/etc/init.d/mongodb stop"
    if failed host 127.0.0.1 port 28017 protocol http
       and request "/" with timeout 10 seconds then restart
    if 5 restarts within 5 cycles then timeout

Check the monit configuration file syntax

monit -t

View the Monit status site at http://localhost:2812

  • username: admin
  • password: “monit”.

Test Monit’s ability to automatically restart services

Stop Apache2

service apache2 stop

Monit will restart the service and eMail you an alert

Troubleshooting

MySQLd Not Monitored

  • To have Monit monitor MySQLd using the PID file, make sure that the PID file is configured in your my.cnf
  • FILE: /etc/mysql/my.cnf
[mysqld]
  pid-file=/var/run/mysqld/mysqld.pid
categories: linux | monit |