尧图网页设计 
前言对于进程的监控最常见的需求就是进程挂了如何被自动拉起来现在可以由Kubernetes等先进的容器化技术来自动化管理那原来再物理服务器或者虚拟机中的进程有什么好的办法呢答案就是Monit/Supervisor等第三方应用来解决因为线上环境分别使用Monit来监控Core Logical ServiceSupervisor用在Codis Dashboard/FE/Proxy上使用下来的感受和网上的对比分析报告类似具体内容会在文章内引用推荐大家使用Monit替代Supervisor自动化管理和监控服务。使用Monit替代Supervisor自动化管理和监控服务小结更新历史2020年01月15日 - 初稿阅读原文 - https://wsgzao.github.io/post/monit/扩展阅读Monit SupervisorMonit介绍NAMEMonit - utility for monitoring services on a Unix systemSYNOPSISmonit[options]DESCRIPTIONMonitis a utility for managing and monitoring processes, programs, files, directories and filesystems on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. E.g. Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources. You can use Monit to monitor files, directories and filesystems for changes, such as timestamps changes, checksum changes or size changes.Monit is controlled via an easy to configure control file based on a free-format, token-oriented syntax. Monit logs to syslog or to its own log file and notifies you about error conditions via customisable alert messages. Monit can perform various TCP/IP network checks, protocol checks and can utilise SSL for such checks. Monit provides a HTTP(S) interface and you may use a browser to access the Monit program.WHAT TO MONITOR?You can use Monit to monitor daemonprocessesor similar programs running on localhost. Monit is particularly useful for monitoring daemon processes, such as those started at system boot time. For instance sendmail, sshd, apache and mysql. In contrast to many other monitoring systems, Monit can act if an error situation should occur, e.g.; if sendmail is not running, monit can start sendmail again automatically or if apache is using too many resources (e.g. if a DoS attack is in progress) Monit can stop or restart apache and send you an alert message. Monit can also monitor process characteristics, such as how much memory or cpu cycles a process is using.You can also use Monit to monitorfiles,directoriesandfilesystemson localhost. Monit can monitor these items for changes, such as timestamps changes, checksum changes or size changes. This is also useful for security reasons - you can monitor the md5 or sha1 checksum of files that should not change and get an alert or perform an action if they should change.Monit can monitornetwork connectionsto various servers, either on localhost or on remote hosts. TCP, UDP and Unix Domain Sockets are supported. Network test can be performed on a protocol level; Monit has built-in tests for the main Internet protocols, such as HTTP, SMTP etc. Even if a protocol is not supported you can still test the server because you can configure Monit to send any data and test the response from the server.Monit can be used to testprogramsor scripts at certain times, much like cron, but in addition, you can test the exit value of a program and perform an action or send an alert if the exit value indicates an error. This means that you can use Monit to perform any type of check you can write a script for.Finally, Monit can be used to monitor generalsystemresources on localhost such as overall CPU usage, Memory and System Load.https://mmonit.com/monit/documentation/monit.htmlSupervisor介绍Supervisor: A Process Control SystemSupervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.It shares some of the same goals of programs likelaunchd,daemontools, andrunit. Unlike some of these programs, it is not meant to be run as a substitute for init as “process id 1”. Instead it is meant to be used to control processes related to a project or a customer, and is meant to start like any other program at boot time.http://supervisord.org/Monit VS SupervisorMonit是什么Monit 是一个管理和监控 Unix 系统的小型开源组件.Monit 可以在出现错误的情况下, 自动维护, 修复和做一些有意义的行为为什么选择Monit除了 Monit 还有一些其他的第三方监控方案(eg. Supervisor), 我们考虑选择额 Monit 作为监控的原因有 - 超轻量, 稳定, 高可用 - 依赖少, 安装配置方便, 尽量减少运维及学习成本(即使没有任何 Monit 基础的人, 都能轻易的读懂大部分监控文件) - 非侵入式, 被监控的程序可以不用知道监控程序的存在(如果使用 Supervisor 监控, 则服务必须从 Supervisor 启动) - 基本功能完备(9 种类型监控, 邮件报警, 支持用户自定义 shell 扩展)Supervisor优缺点优点轻量、特性丰富、内存友好(好客套的优点。。。)对被监控进程的状态获取迅速且精确——通过子进程管理没办法不精确缺点被监控进程必须运行在前台(可以理解为有自己的控制终端)——这也是最为致命的一点无法管理依赖也就是说无法控制服务启动先后顺序无法管理被监控进程创建的子进程——重启服务时被监控进程的子进程无法正常退出隐患大无法控制进程失败重试的间隔时间——有些进程需要清理资源不过这点还好Monit优缺点优点安装配置简单同样轻量(似乎也是很客套了)可以监控前台进程和非前台进程——完美的弥补了supervisor的致命缺点除了监控进程还能监控文件、文件系统甚至系统资源CPU利用率内存使用也是可以的被监控的进程可以设置依赖控制启动顺序缺点无法监控没有pid文件的进程如shell脚本对进程监控的状态感知有延时即精度不够——采用轮训决定了它无法像supervisor一样实时感知被监控进程状态这样看起来还是monit更为普适一点。不过这催生了一个大胆的想法使用supervisor管理容器内多进程monit作为一个被监控进程挂在supervisor之下。这样对于无法前台运行的程序就可以通过monit监控而对服务中断感知强烈的则直接挂在supervisor之下。看起来似乎是个好办法有机会试试哈哈哈。从实际容器中运行的表现看monit经常出现各种未知异常而supervisor表现得十分稳定。Monit基本用法Monit常用命令# monit -h Usage: monit [options] [command] Options are as follows: -c file Use this control file -d n Run as a daemon once per n seconds -g name Set group name for monit commands -l logfile Print log information to this file -p pidfile Use this lock file in daemon mode -s statefile Set the file monit should write state information to -I Do not run in background (needed when run from init) --id Print Monits unique ID --resetid Reset Monits unique ID. Use with caution -B Batch command line mode (do not output tables or colors) -t Run syntax check for the control file -v Verbose mode, work noisy (diagnostic output) -vv Very verbose mode, same as -v plus log stacktrace on error -H [filename] Print SHA1 and MD5 hashes of the file or of stdin if the filename is omited; monit will exit afterwards -V Print version number and patchlevel -h Print this text Optional commands are as follows: start all - Start all services start name - Only start the named service stop all - Stop all services stop name - Stop the named service restart all - Stop and start all services restart name - Only restart the named service monitor all - Enable monitoring of all services monitor name - Only enable monitoring of the named service unmonitor all - Disable monitoring of all services unmonitor name - Only disable monitoring of the named service reload - Reinitialize monit status [name] - Print full status information for service(s) summary [name] - Print short status information for service(s) report [up|down|..] - Report state of services. See manual for options quit - Kill the monit daemon process validate - Check all services and start if not running procmatch pattern - Test process matching pattern想要让 Monit 可靠的为我们工作, 学习成本非常低, 只需要学习一些 Monit 命令行和配置文件写法# options - 选项 - monit - monit -t - monit -c /var/monit/monitrc # 指定配置文件 - monit -g groupname start/stop # Monit 可以对各个监控分组, 如果需要对某个分组统一操作, 可以用这个命令 # arguments - 参数 - monit reload - monit quit - monit start/stop/restart/monitor/unmonitor name/all # name: 每个监控都有一个独一无二的名字, 具体后面会提到; all: 所有监控服务Monit服务监控配置文件格式详细配置, 共计 9 种, 所有配置中, 都符合以下规则如果指定的 path 不存在, 而且配置块里包含 start 方法, 会调用这个 start 方法如果 path 指定的文件类型不对, Monit 不能监控这个项目ProcessCHECK PROCESS unique name PIDFILE path | MATCHING regex path pid-file 的绝对路径. 不存在 pid-file 文件或者 pid-file 文件没有对应的正在运行的程序, Monit 会执行 start 方法 regex 进程名称的正则表达来监控进程, 可以通过命令行测试正则是否写对了: monit procmatch regex-patternFileCHECK FILE unique name PATH path path file 的绝对路径.FifoCHECK FIFO unique name PATH path path fifo 的绝对路径.FilesystemCHECK FILESYSTEM unique name PATH path path 设备/磁盘, 挂载点的路径 或 NFS/CIFS/FUSE 链接字符串. 如果文件系统不可用, Monit 会执行 start 方法DirectoryCHECK DIRECTORY unique name PATH path path 目录问价的绝对路径Remote hostCHECK HOST unique name ADDRESS host host 可以是域名或者 IP 地址. eg: tildeslash.com or 64.87.72.95.SystemCHECK SYSTEM unique name unique name 通常来说是本机名称(可以用 $HOST), 也可以是其他名称. 用于邮件报警或者 M/Monit 的初始化名称 这类配置可以监控系统资源(CPU, memory, load average...)ProgramCHECK PROGRAM unique name PATH executable file [TIMEOUT number SECONDS] path 可执行程序或脚本的绝对路径. 允许检查程序退出状态.如果程序没能在 number 秒内执行完成, Monit 会终结这个程序, 默认是 300s 程序的输出会被记录, 用于用户界面或者报警, 默认 512 bytes(可以通过 set limits 修改)NetworkCHECK NETWORK unique name ADDRESS ipaddress | INTERFACE name # ipaddress 是被监控的 IPv4/IPv6 网卡地址. 用 eth0 也是可以的更多配置信息可以参考Monit官方文档和实例https://mmonit.com/documentation/https://mmonit.com/wiki/Monit/ConfigurationExamplesMonit配置实践创建templates模板利用python生成monit配置文件使用ansible推送到目标服务器中# 创建通用配置配置日志邮件告警 vim basic.j2 # log to monit.log set logfile /var/log/monit.log set daemon {{ monit_poll_interval }} set eventqueue basedir /var/lib/monit/events slots 5000 set mailserver smtp.xxx.com port 465 set alert xxxxxx.com { nonexist, timeout, resource } set mail-format { from: xxxxxx.com subject: monit alert -- $SERVICE $EVENT at $DATE message: $EVENT Service $SERVICE Date: $DATE Action: $ACTION Host: $HOST Description: $DESCRIPTION Your faithful employee, Monit } # 创建标准应用监控 vim daemon_set.j2 check process xxx with pidfile /run/xxx/daemon.pid start program /usr/bin/python2 /bin/xxx restart stop program /usr/bin/python2 /bin/xxx stop if 10 restarts within 10 cycles then unmonitor check process xxxx with matching xxxx start program /etc/init.d/xxxx start stop program /etc/init.d/xxxx stop if 10 restarts within 10 cycles then unmonitor # 创建非标准应用监控 vim logic_service.j2 check process {{ service_name }} with pidfile {{ root_dir }}/{{ service_name }}/deploy/{{ monit_name }}.pid start program /bin/bash -c cd {{ root_dir }}/{{ service_name }}/deploy ./start.sh start.log stop program /bin/bash -c cd {{ root_dir }}/{{ service_name }}/deploy ./stop.sh stop.log if 5 restarts within 15 cycles then unmonitor {% if memory_usage is defined %} if total memory usage {{ memory_usage }} for 10 cycles then restart {% endif %}参考文章Monit DocumentMonit 笔记Docker容器内多进程管理——supervisor VS monit 使用Monit替代Supervisor自动化管理和监控服务小结
前言对于进程的监控最常见的需求就是进程挂了如何被自动拉起来现在可以由Kubernetes等先进的容器化技术来自动化管理那原来再物理服务器或者虚拟机中的进程有什么好的办法呢答案就是Monit/Supervisor等第三方应用来解决因为线上环境分别使用Monit来监控Core Logical ServiceSupervisor用在Codis Dashboard/FE/Proxy上使用下来的感受和网上的对比分析报告类似具体内容会在文章内引用推荐大家使用Monit替代Supervisor自动化管理和监控服务。使用Monit替代Supervisor自动化管理和监控服务小结更新历史2020年01月15日 - 初稿阅读原文 - https://wsgzao.github.io/post/monit/扩展阅读Monit SupervisorMonit介绍NAMEMonit - utility for monitoring services on a Unix systemSYNOPSISmonit[options]DESCRIPTIONMonitis a utility for managing and monitoring processes, programs, files, directories and filesystems on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. E.g. Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources. You can use Monit to monitor files, directories and filesystems for changes, such as timestamps changes, checksum changes or size changes.Monit is controlled via an easy to configure control file based on a free-format, token-oriented syntax. Monit logs to syslog or to its own log file and notifies you about error conditions via customisable alert messages. Monit can perform various TCP/IP network checks, protocol checks and can utilise SSL for such checks. Monit provides a HTTP(S) interface and you may use a browser to access the Monit program.WHAT TO MONITOR?You can use Monit to monitor daemonprocessesor similar programs running on localhost. Monit is particularly useful for monitoring daemon processes, such as those started at system boot time. For instance sendmail, sshd, apache and mysql. In contrast to many other monitoring systems, Monit can act if an error situation should occur, e.g.; if sendmail is not running, monit can start sendmail again automatically or if apache is using too many resources (e.g. if a DoS attack is in progress) Monit can stop or restart apache and send you an alert message. Monit can also monitor process characteristics, such as how much memory or cpu cycles a process is using.You can also use Monit to monitorfiles,directoriesandfilesystemson localhost. Monit can monitor these items for changes, such as timestamps changes, checksum changes or size changes. This is also useful for security reasons - you can monitor the md5 or sha1 checksum of files that should not change and get an alert or perform an action if they should change.Monit can monitornetwork connectionsto various servers, either on localhost or on remote hosts. TCP, UDP and Unix Domain Sockets are supported. Network test can be performed on a protocol level; Monit has built-in tests for the main Internet protocols, such as HTTP, SMTP etc. Even if a protocol is not supported you can still test the server because you can configure Monit to send any data and test the response from the server.Monit can be used to testprogramsor scripts at certain times, much like cron, but in addition, you can test the exit value of a program and perform an action or send an alert if the exit value indicates an error. This means that you can use Monit to perform any type of check you can write a script for.Finally, Monit can be used to monitor generalsystemresources on localhost such as overall CPU usage, Memory and System Load.https://mmonit.com/monit/documentation/monit.htmlSupervisor介绍Supervisor: A Process Control SystemSupervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.It shares some of the same goals of programs likelaunchd,daemontools, andrunit. Unlike some of these programs, it is not meant to be run as a substitute for init as “process id 1”. Instead it is meant to be used to control processes related to a project or a customer, and is meant to start like any other program at boot time.http://supervisord.org/Monit VS SupervisorMonit是什么Monit 是一个管理和监控 Unix 系统的小型开源组件.Monit 可以在出现错误的情况下, 自动维护, 修复和做一些有意义的行为为什么选择Monit除了 Monit 还有一些其他的第三方监控方案(eg. Supervisor), 我们考虑选择额 Monit 作为监控的原因有 - 超轻量, 稳定, 高可用 - 依赖少, 安装配置方便, 尽量减少运维及学习成本(即使没有任何 Monit 基础的人, 都能轻易的读懂大部分监控文件) - 非侵入式, 被监控的程序可以不用知道监控程序的存在(如果使用 Supervisor 监控, 则服务必须从 Supervisor 启动) - 基本功能完备(9 种类型监控, 邮件报警, 支持用户自定义 shell 扩展)Supervisor优缺点优点轻量、特性丰富、内存友好(好客套的优点。。。)对被监控进程的状态获取迅速且精确——通过子进程管理没办法不精确缺点被监控进程必须运行在前台(可以理解为有自己的控制终端)——这也是最为致命的一点无法管理依赖也就是说无法控制服务启动先后顺序无法管理被监控进程创建的子进程——重启服务时被监控进程的子进程无法正常退出隐患大无法控制进程失败重试的间隔时间——有些进程需要清理资源不过这点还好Monit优缺点优点安装配置简单同样轻量(似乎也是很客套了)可以监控前台进程和非前台进程——完美的弥补了supervisor的致命缺点除了监控进程还能监控文件、文件系统甚至系统资源CPU利用率内存使用也是可以的被监控的进程可以设置依赖控制启动顺序缺点无法监控没有pid文件的进程如shell脚本对进程监控的状态感知有延时即精度不够——采用轮训决定了它无法像supervisor一样实时感知被监控进程状态这样看起来还是monit更为普适一点。不过这催生了一个大胆的想法使用supervisor管理容器内多进程monit作为一个被监控进程挂在supervisor之下。这样对于无法前台运行的程序就可以通过monit监控而对服务中断感知强烈的则直接挂在supervisor之下。看起来似乎是个好办法有机会试试哈哈哈。从实际容器中运行的表现看monit经常出现各种未知异常而supervisor表现得十分稳定。Monit基本用法Monit常用命令# monit -h Usage: monit [options] [command] Options are as follows: -c file Use this control file -d n Run as a daemon once per n seconds -g name Set group name for monit commands -l logfile Print log information to this file -p pidfile Use this lock file in daemon mode -s statefile Set the file monit should write state information to -I Do not run in background (needed when run from init) --id Print Monits unique ID --resetid Reset Monits unique ID. Use with caution -B Batch command line mode (do not output tables or colors) -t Run syntax check for the control file -v Verbose mode, work noisy (diagnostic output) -vv Very verbose mode, same as -v plus log stacktrace on error -H [filename] Print SHA1 and MD5 hashes of the file or of stdin if the filename is omited; monit will exit afterwards -V Print version number and patchlevel -h Print this text Optional commands are as follows: start all - Start all services start name - Only start the named service stop all - Stop all services stop name - Stop the named service restart all - Stop and start all services restart name - Only restart the named service monitor all - Enable monitoring of all services monitor name - Only enable monitoring of the named service unmonitor all - Disable monitoring of all services unmonitor name - Only disable monitoring of the named service reload - Reinitialize monit status [name] - Print full status information for service(s) summary [name] - Print short status information for service(s) report [up|down|..] - Report state of services. See manual for options quit - Kill the monit daemon process validate - Check all services and start if not running procmatch pattern - Test process matching pattern想要让 Monit 可靠的为我们工作, 学习成本非常低, 只需要学习一些 Monit 命令行和配置文件写法# options - 选项 - monit - monit -t - monit -c /var/monit/monitrc # 指定配置文件 - monit -g groupname start/stop # Monit 可以对各个监控分组, 如果需要对某个分组统一操作, 可以用这个命令 # arguments - 参数 - monit reload - monit quit - monit start/stop/restart/monitor/unmonitor name/all # name: 每个监控都有一个独一无二的名字, 具体后面会提到; all: 所有监控服务Monit服务监控配置文件格式详细配置, 共计 9 种, 所有配置中, 都符合以下规则如果指定的 path 不存在, 而且配置块里包含 start 方法, 会调用这个 start 方法如果 path 指定的文件类型不对, Monit 不能监控这个项目ProcessCHECK PROCESS unique name PIDFILE path | MATCHING regex path pid-file 的绝对路径. 不存在 pid-file 文件或者 pid-file 文件没有对应的正在运行的程序, Monit 会执行 start 方法 regex 进程名称的正则表达来监控进程, 可以通过命令行测试正则是否写对了: monit procmatch regex-patternFileCHECK FILE unique name PATH path path file 的绝对路径.FifoCHECK FIFO unique name PATH path path fifo 的绝对路径.FilesystemCHECK FILESYSTEM unique name PATH path path 设备/磁盘, 挂载点的路径 或 NFS/CIFS/FUSE 链接字符串. 如果文件系统不可用, Monit 会执行 start 方法DirectoryCHECK DIRECTORY unique name PATH path path 目录问价的绝对路径Remote hostCHECK HOST unique name ADDRESS host host 可以是域名或者 IP 地址. eg: tildeslash.com or 64.87.72.95.SystemCHECK SYSTEM unique name unique name 通常来说是本机名称(可以用 $HOST), 也可以是其他名称. 用于邮件报警或者 M/Monit 的初始化名称 这类配置可以监控系统资源(CPU, memory, load average...)ProgramCHECK PROGRAM unique name PATH executable file [TIMEOUT number SECONDS] path 可执行程序或脚本的绝对路径. 允许检查程序退出状态.如果程序没能在 number 秒内执行完成, Monit 会终结这个程序, 默认是 300s 程序的输出会被记录, 用于用户界面或者报警, 默认 512 bytes(可以通过 set limits 修改)NetworkCHECK NETWORK unique name ADDRESS ipaddress | INTERFACE name # ipaddress 是被监控的 IPv4/IPv6 网卡地址. 用 eth0 也是可以的更多配置信息可以参考Monit官方文档和实例https://mmonit.com/documentation/https://mmonit.com/wiki/Monit/ConfigurationExamplesMonit配置实践创建templates模板利用python生成monit配置文件使用ansible推送到目标服务器中# 创建通用配置配置日志邮件告警 vim basic.j2 # log to monit.log set logfile /var/log/monit.log set daemon {{ monit_poll_interval }} set eventqueue basedir /var/lib/monit/events slots 5000 set mailserver smtp.xxx.com port 465 set alert xxxxxx.com { nonexist, timeout, resource } set mail-format { from: xxxxxx.com subject: monit alert -- $SERVICE $EVENT at $DATE message: $EVENT Service $SERVICE Date: $DATE Action: $ACTION Host: $HOST Description: $DESCRIPTION Your faithful employee, Monit } # 创建标准应用监控 vim daemon_set.j2 check process xxx with pidfile /run/xxx/daemon.pid start program /usr/bin/python2 /bin/xxx restart stop program /usr/bin/python2 /bin/xxx stop if 10 restarts within 10 cycles then unmonitor check process xxxx with matching xxxx start program /etc/init.d/xxxx start stop program /etc/init.d/xxxx stop if 10 restarts within 10 cycles then unmonitor # 创建非标准应用监控 vim logic_service.j2 check process {{ service_name }} with pidfile {{ root_dir }}/{{ service_name }}/deploy/{{ monit_name }}.pid start program /bin/bash -c cd {{ root_dir }}/{{ service_name }}/deploy ./start.sh start.log stop program /bin/bash -c cd {{ root_dir }}/{{ service_name }}/deploy ./stop.sh stop.log if 5 restarts within 15 cycles then unmonitor {% if memory_usage is defined %} if total memory usage {{ memory_usage }} for 10 cycles then restart {% endif %}参考文章Monit DocumentMonit 笔记Docker容器内多进程管理——supervisor VS monit