Nginx配合keepalived实现双主负载均衡

king 运维技术 2022-11-13 468浏览 0

前言：由于之前一直使用的是主从高可用加后端负载，随着业务量的增大，前端一台服务在高峰时期有些吃力，所有对之前架构进行了一点改造，把主从升级成了双主，可以更充分的利用现有服务器资源，由于实验环境有限，后端的测试仅仅使用了一个静态页面，没有搭建动态环境，也没用使用数据库，如有需要可以参考其他文档。

一、架构规划

1、服务器IP地址规划

VIP1：192.168.1.149

VIP2：192.168.1.150

Keepalived1:192.168.1.151

Keepalived2:192.168.1.152

WebServer1:192.168.1.201

WebServer2:192.168.1.202

2、服务器操作系统

所使用的操作系统均为CentOS release 6.6 (Final) x86_64，最小化安装。

3、网络拓扑图

Nginx配合keepalived实现双主负载均衡 #p#

二、配置Nginx代理服务器

此部分Node1与Node2的配置完全相同。

1，准备编译环境

1	#yum–yinstallgccgcc-c++pcre-developensslopenssl-develwget

2，编译安装nginx

#./configure\ 
--prefix=/usr/local/nginx\ 
--sbin-path=/usr/local/nginx/sbin/nginx\ 
--conf-path=/etc/nginx/nginx.conf\ 
--error-log-path=/var/log/nginx/error.log\ 
--http-log-path=/var/log/nginx/access.log\ 
--pid-path=/var/run/nginx/nginx.pid\ 
--lock-path=/var/lock/nginx.lock\ 
--user=nginx\ 
--group=nginx\ 
--with-http_ssl_module\ 
--with-pcre 
#make&&makeinstall

3,为nginx提供SysV init脚本:

# vi /etc/rc.d/init.d/nginx

添加如下内容

#!/bin/sh 
# 
#nginx-thisscriptstartsandstopsthenginxdaemon 
# 
#chkconfig:-8515 
#description:NginxisanHTTP(S)server,HTTP(S)reverse\ 
#proxyandIMAP/POP3proxyserver 
#processname:nginx 
#config:/etc/nginx/nginx.conf 
#config:/etc/sysconfig/nginx 
#pidfile:/var/run/nginx.pid 
#Sourcefunctionlibrary. 
./etc/rc.d/init.d/functions 
#Sourcenetworkingconfiguration. 
./etc/sysconfig/network 
#Checkthatnetworkingisup. 
["$NETWORKING"="no"]&&exit0 
nginx="/usr/local/nginx/sbin/nginx"
prog=$(basename$nginx) 
NGINX_CONF_FILE="/etc/nginx/nginx.conf"
[-f/etc/sysconfig/nginx]&&./etc/sysconfig/nginx 
lockfile=/var/lock/subsys/nginx 
make_dirs(){ 
#makerequireddirectories 
user=`nginx-V2>&1|grep"configurearguments:"|sed's/[^*]*--user=\([^]*\).*/\1/g'-` 
options=`$nginx-V2>&1|grep'configurearguments:'` 
foroptin$options;do 
if[`echo$opt|grep'.*-temp-path'`];then 
value=`echo$opt|cut-d"="-f2` 
if[!-d"$value"];then 
#echo"creating"$value 
mkdir-p$value&&chown-R$user$value 
fi 
fi 
done 
} 
start(){ 
[-x$nginx]||exit5 
[-f$NGINX_CONF_FILE]||exit6 
make_dirs 
echo-n$"Starting$prog:" 
daemon$nginx-c$NGINX_CONF_FILE 
retval=$? 
echo 
[$retval-eq0]&&touch$lockfile 
return$retval 
} 
stop(){ 
echo-n$"Stopping$prog:" 
killproc$prog-QUIT 
retval=$? 
echo 
[$retval-eq0]&&rm-f$lockfile 
return$retval 
} 
restart(){ 
configtest||return$? 
stop 
sleep1 
start 
} 
reload(){ 
configtest||return$? 
echo-n$"Reloading$prog:" 
killproc$nginx-HUP 
RETVAL=$? 
echo 
} 
force_reload(){ 
restart 
} 
configtest(){ 
$nginx-t-c$NGINX_CONF_FILE 
} 
rh_status(){ 
status$prog 
} 
rh_status_q(){ 
rh_status>/dev/null2>&1 
} 
case"$1"in 
start) 
rh_status_q&&exit0 
$1 
;; 
stop) 
rh_status_q||exit0 
$1 
;; 
restart|configtest) 
$1 
;; 
reload) 
rh_status_q||exit7 
$1 
;; 
force-reload) 
force_reload 
;; 
status) 
rh_status 
;; 
condrestart|try-restart) 
rh_status_q||exit0 
;; 
*) 
echo$"Usage:$0{start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}" 
exit2 
esac

而后为此脚本赋予执行权限：

# chmod +x /etc/rc.d/init.d/nginx

添加至服务管理列表，并让其开机自动启动：

# chkconfig –add nginx

# chkconfig nginx on

4，配置Nginx代理

# cat nginx.conf

usernginxnginx; 
worker_processes1; 
pid/var/run/nginx/nginx.pid; 
worker_rlimit_nofile51200; 
events 
{ 
useepoll; 
worker_connections51200; 
} 
http{ 
includemime.types; 
default_typeapplication/octet-stream; 
server_names_hash_bucket_size128; 
client_header_buffer_size32k; 
large_client_header_buffers432k; 
client_max_body_size8m; 
sendfileon; 
tcp_nopushon; 
keepalive_timeout60; 
tcp_nodelayon; 
fastcgi_connect_timeout300; 
fastcgi_send_timeout300; 
fastcgi_read_timeout300; 
fastcgi_buffer_size64k; 
fastcgi_buffers464k; 
fastcgi_busy_buffers_size128k; 
fastcgi_temp_file_write_size128k; 
gzipon; 
gzip_min_length1k; 
gzip_buffers416k; 
gzip_http_version1.0; 
gzip_comp_level2; 
gzip_typestext/plainapplication/x-javascripttext/cssapplication/xml; 
gzip_varyon; 
upstreambackend 
{ 
ip_hash; 
server192.168.1.201:80; 
server192.168.1.202:80; 
} 
log_formataccess'$remote_addr-$remote_user[$time_local]"$request"' 
'$status$body_bytes_sent"$http_referer"' 
'"$http_user_agent"$http_x_forwarded_for'; 
access_log/var/log/nginx/access.logaccess; 
server{ 
listen80; 
server_namewww.test.com; 
location/{ 
root/var/www/html; 
indexindex.phpindex.htmindex.html; 
proxy_redirectoff; 
proxy_set_headerHost$host; 
proxy_set_headerX-Real-IP$remote_addr; 
proxy_set_headerX-Forwarded-For$proxy_add_x_forwarded_for; 
proxy_passhttp://backend; 
} 
location/nginx{ 
access_logoff; 
auth_basic"NginxStatus"; 
} 
} 
}

#p#

三、安装与配置keepalived

#wgethttp://www.keepalived.org/software/keepalived-1.2.16.tar.gz 
#yum-yinstalllibnl-devel 
#./configure--prefix=/usr/local/keepalived 
#make&&makeinstall 
#cp/usr/local/keepalived/sbin/keepalived/usr/sbin/ 
#cp/usr/local/keepalived/etc/sysconfig/keepalived/etc/sysconfig/ 
#cp/usr/local/keepalived/etc/rc.d/init.d/keepalived/etc/init.d/ 
#mkdir/etc/keepalived 
#cd/etc/keepalived/ 
#vimkeepalived.conf

以上步骤在两台keepalived机器上都需要进行

下面分别是两节点的配置文件

节点一

!ConfigurationFileforkeepalived 
 
global_defs{ 
notification_email{ 
ganen2008@126.com#接收警报的email地址,可以添加多个 
} 
notification_email_fromganen201405@126.com#发件人地址 
smtp_connect_timeout3#超时时间 
smtp_server127.0.0.1#发送邮件的服务器 
router_idLVS_DEVEL#loadbalancer的标识ID,用于email警报 
} 
vrrp_instanceVI_1{ 
stateMASTER 
interfaceeth0 
virtual_router_id51 
priority100#权值要比back高 
advert_int1 
authentication{ 
auth_typePASS 
auth_pass123456 
} 
virtual_ipaddress{ 
192.168.1.149#vip的地址 
} 
} 
vrrp_instanceVI_2{ 
stateBACKUP 
interfaceeth0 
virtual_router_id52 
priority90 
advert_int1 
authentication{ 
auth_typePASS 
auth_pass123456 
} 
virtual_ipaddress{ 
192.168.1.150 
} 
}

节点二

!ConfigurationFileforkeepalived 
global_defs{ 
notification_email{ 
ganen2008@126.com 
} 
notification_email_fromganen201405@126.com 
smtp_connect_timeout3 
smtp_server127.0.0.1 
router_idLVS_DEVEL 
} 
vrrp_instanceVI_1{ 
stateBACKUP 
interfaceeth1 
virtual_router_id51 
priority90 
advert_int1 
authentication{ 
auth_typePASS 
auth_pass123456 
} 
virtual_ipaddress{ 
192.168.1.149 
} 
} 
vrrp_instanceVI_2{ 
stateMASTER 
interfaceeth1 
virtual_router_id52 
priority100 
advert_int1 
authentication{ 
auth_typePASS 
auth_pass123456 
} 
virtual_ipaddress{ 
192.168.1.150 
} 
}

为两节点添加nginx状态监控脚本，由于keepalived本身不能检测到nginx的存活状态，需要借助于第三方脚本来实现，下面是出自余洪春前辈的一个检测脚本，在这里借用一下。

# vim /home/nginx_chk.sh

# chmod +x /home/nginx_chk.sh

#!/bin/bash 
while: 
do 
nginxpid=`ps-Cnginx--no-header|wc-l` 
if[$nginxpid-eq0];then 
/usr/local/nginx/sbin/nginx 
sleep5 
nginxpid=`ps-Cnginx--no-header|wc-l` 
echo$nginxpid 
if[$nginxpid-eq0];then 
/etc/init.d/keepalivedstop 
fi 
fi 
sleep5 
done

# chmod +x /home/nginx_chk.sh
后台执行该脚本
# nohup sh /home/nginx_chk.sh &

#p#

四、安装web server

这里为了测试我直接使用yum安装两台服务器上的nginx服务。

增加额外资源库

#yum-yinstallyum-priorities 
#rpm-Uvhhttp://mirrors.yun-idc.com/epel/6/x86_64/epel-release-6-8.noarch.rpm 
#rpm-Uvhhttp://rpms.famillecollet.com/enterprise/remi-release-6.rpm 
#yum-yinstallnginx 
#/etc/init.d/nginxstart

五、启动测试

首先测试两台WebServer的可用性

Nginx配合keepalived实现双主负载均衡