使用Nginx+Keepalived组建高可用负载平衡Web server集群

作者:W.S.T 发布时间:January 7, 2016 分类:Nginx No Comments

一,首先说明一下网络拓扑结构:
nginx-keepalived-webserver

1,Nginx 反向代理Server(HA):
①Nginx master:192.168.1.157
②Nginx backup:192.168.1.158
虚拟IP统一为:192.168.1.110
2,web服务器:
192.168.1.160 ,192.168.1.161,192.168.1.162   即web服务器,已配置好 Tomcat(Jboss等皆可)和java程序
3,mysql 数据库Server
mysql主从服务器

二,Nginx安装配置
1,安装
建议先用yum install yum-fastestmirror更新下源

下载并安装nginx1.0.9,下载文件均放到/usr/local/src目录下
cd /usr/local/src

①update yum
yum -y update

②利用CentOS Linux系统自带的yum命令安装、升级所需的程序库

安装依赖包
#yum install gcc pcre pcre-devel zlib zlib-devel openssl openssl-devel

下载nginx
#cd /usr/local/src
#wget http://www.nginx.org/download/nginx-1.0.9.tar.gz
#tar zxvf nginx-1.0.9.tar.gz
#cd nginx-1.0.9

配置安装:
#./configure --prefix=/usr --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx/nginx.pid --lock-path=/var/lock/nginx.lock --user=nginx --group=nginx --with-http_ssl_module --with-http_flv_module --with-http_gzip_static_module --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/tmp/nginx/client/ --http-proxy-temp-path=/var/tmp/nginx/proxy/ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/
#make
#make install

建立用户:
/usr/sbin/groupadd nginx
/usr/sbin/useradd -g nginx -M nginx
mkdir -p /var/tmp/nginx/client
启动nginx
#/usr/sbin/nginx

ps:如果中途提示缺少库,直接yum install xxx即可。
即:conf-path=/etc/nginx/nginx.conf,用户名为nginx,所属的组为nginx

2,配置
Nginx负载均衡设置:

①修改配置文件:
vi /etc/nginx/nginx.conf

步骤1,添加负载均衡的http upstream模块
upstream esbwebserver {
server 192.168.1.160:8888;
server 192.168.1.161:8888;
}

步骤2,server指令
server
{
listen 80;
server_name localhost;#注意此处为localhost

location / {
proxy_pass http://esbwebserver;# 添加的tomcat集群名称
......
}

②,重启Nginx,加载修改过的配置文件:
步骤1, 停止nginx引擎
killall -9 nginx

步骤2,启动nginx
/usr/sbin/nginx

到此为止,我们的负载均衡就实现了,下面实现Nginx的高可用,即双机热备。

三,Keepalived 安装配置
1,下载并安装keepalived-1.1.15.tar.gz,下载文件均放到/usr/local/src目录下
cd /usr/local/src
wget http://www.keepalived.org/software/keepalived-1.1.15.tar.gz

①解压缩
tar zxvf keepalived-1.1.15.tar.gz

②安装
cd keepalived-1.1.15
./configure --prefix=/usr/local/keepalived
make && make install

③安装成功后做成服务模式,方便启动和关闭
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/

④ 分别设置主和备Nginx上的 安装Keepalived配置文件。

配置文件位置:
/usr/local/keepalived/etc/keepalived/keepalived.conf

步骤一,先配置主Nginx server上的keepalived.conf文件,如下所示:

! Configuration File for keepalived

global_defs {
notification_email {
644856452@qq.com
}
notification_email_from 644856452@qq.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}

vrrp_script check_run {
script "/root/bin/nginx_check.sh"
interval 5
}
vrrp_sync_group VG1 {
group {
VI_1
}
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
mcast_src_ip 192.168.1.157
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}

track_script {
check_run
}

virtual_ipaddress {
192.168.1.110
}
}

步骤二,配置backup 服务器的keepalived.conf文件,如下所示:
! Configuration File for keepalived

global_defs {
notification_email {
644856452@qq.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}

vrrp_script check_run {
script "/root/bin/nginx_check.sh"
interval 5
}
vrrp_sync_group VG1 {
group {
VI_1
}
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}

track_script {
check_run
}
virtual_ipaddress {
192.168.1.110
}
}

参考下例配置:
-----------------------------------------------------带注释的示例----------------------------------------------------
在主服务器编写配置文件
vrrp_script check_run{
script "/opt/nginx_pid.sh"
###监控脚本
interval 2
###监控时间
weight 2 ###如果检测返回值不为真weight 2 表示减2,权重值降低,backup server权重值>现Master的,切换
}

vrrp_instance VI_1 {
state MASTER ### 设置为 主
interface eth0 ### 监控网卡
virtual_router_id 51 ### 这个两台服务器必须一样
priority 101 ### 权重值 MASTRE 一定要高于 BAUCKUP
authentication {
auth_type PASS ### 加密
auth_pass 1111 ###加密的密码,两台服务器一定要一样,不然会出错
}

track_script {
check_run ### 执行监控的服务
}

virtual_ipaddress {
192.168.1.110 ### VIP 地址
}
}

6.在backup server 服务器 keepalived 配置
vrrp_script check_run{
script "/opt/nginx_pid.sh"
interval 2
weight 2
}

vrrp_instance VI_1 {
state BACKUP ### 设置为 辅机
interface eth0
virtual_router_id 51 ### 与 MASTRE 设置 值一样
priority 100 ### 比 MASTRE权重值 低

authentication {
auth_type PASS
auth_pass eric ### 密码 与 MASTRE 一样
}

track_script {
check_run
}

virtual_ipaddress {
192.168.1.110
}
}
----------------------------------------------over-------------------------------------------------------------------

说明:
1,绑定虚拟IP:
ifconfig eth0:1 192.168.1.110 broadcast 192.168.1.255 netmask 255.255.255.0 up
route add -host 192.168.1.110 dev eth0:1

2, 启动,关闭keepalived :
service keepalived start
service keepalived stop

3,测试可用性:
①主Nginx停止Nginx或直接断网情况下(backup正常),访问虚拟IP:192.168.1.110的相关Web服务,正常,测试通过
②backup Nginx停止Nginx或直接断网情况下(Master正常),访问虚拟IP:192.168.1.110的相关Web服务,正常,测试通过

4,监控
可用安装Nagios监控服务,详细请网络搜寻,很多这种资料的。
也可以参考:
http://blog.csdn.net/luxiaoyu_sdc/article/details/7333416

nginx 安全优化配置https站点

作者:W.S.T 发布时间:December 14, 2015 分类:Nginx No Comments

Nginx安全优化HTTPS站点,通过SSL测试,验证结果A+,配置如下:


            ssl on;
            ssl_certificate ssl.crt;
            ssl_certificate_key ssl.key;
            add_header Strict-Transport-Security "max-age=31536000; includeSubdomains"; 
            ssl_protocols TLSv1 TLSv1.1 TLSv1.2 ;
            ssl_stapling on;
            resolver 114.114.114.114;
            ssl_prefer_server_ciphers on;
            ssl_stapling_verify on;
            ssl_dhparam wst2048.pem;
            ssl_session_cache shared:SSL:10m;
            ssl_session_timeout 10m;
            ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-RC4-SHA:!ECDHE-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDHE-RSA-AES256-SHA:!RC4-SHA:HIGH:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!CBC:!EDH:!kEDH:!PSK:!SRP:!kECDH;  

其中ssl.key ssl.cst为域名的SSL证书 wst2048.pem在linux通过命令
openssl gendh -out wst2048.pem 2048生成

这样通过https://www.ssllabs.com/ssltest/analyze.html 来测试分析https站点 就可以得分A+

nginx-https-ssl

CentOS安装shadowsocks服务器端

作者:W.S.T 发布时间:August 25, 2015 分类:Hack No Comments

Shadowsocks 主要的功能 Sockse服务代理 可以帮助我们防问被墙的网站(Google/Facebook/Twitter)等

CentOS6 服务器安装方法如下:

yum install build-essential autoconf libtool gcc git -y
git clone git://github.com/madeye/shadowsocks-libev.git
cd shadowsocks-libev
./configure
make
make install

启动服务器端指令格式如下
nohup /usr/local/bin/ss-server -s IP地址 -p 端口号 -m rc4 -k 密码(比如opvps.com) >/dev/null 2>/dev/null &

CentOS5 服务器安装方法如下:

第一步:安装openssl-devel

# yum install openssl-devel

第二步:安装make命令
make是gcc的编译器,VPS买来必定要安装。由于服务器是最小化安装centos系统,所以我们需要自己安装make命令
安装:
# yum -y install gcc automake autoconf libtool make
安装g++:
# yum -y install gcc gcc-c++

第三步:下载源码包,然后编译安装
# wget https://github.com/shadowsocks/shadowsocks-libev/archive/v1.4.5.zip
选中 v1.4.5下载并解压,注意如果系统是centos6以下,最高只能用此版本,否则编译出错。
# cd shadowsocks-libev
# ./configure
# make && make install

第四步:加入防火墙规则(可选,如果安装了防火墙。这一步我没有做)
# iptables -I INPUT -p tcp -m tcp --dport 8123 -j ACCEPT #加入规则,端口号就是
# service iptables save #保存
# iptables -L -n #查看的iptables规则

第五步:启动shadowsocks
nohup ss-server -s ipaddress -p port -k password &
说明:ipaddress是vps空间的ip地址,port是端口号,password是密码
如:nohup ss-server -s 162.x.x.80 -p 8123 -k 1234560 &

重要提示:如果运行不成功,切换到纯root的环境下执行,如:

sudo su
su - root
cd /usr/local/bin
nohup ss-server -s 100.42.xxx.xxx -p 8123 -k 1234560 &

Building openresty1.7.10.1 with luajit on windows using Cygwin, 在windows下用Cygwin下编译带Luajit的openresty

作者:W.S.T 发布时间:May 21, 2015 分类:Code No Comments

First install 32bit cygwin.

Install packet: openssl zlib-dev pcre gcc-core perl readline

openresty没有对Cygwin做兼容,我们需要自己来修改文件

Open bundle\lua-cjson and lua-rds-parser and lua-redis-parser three folder's Makefile for edit, Add those lines before “## ..... (Macports)”

bundle\lua-cjson的,lua-rds-parser的,lua-redis-parser的Makefile里,“## ***OSX (Macports)”上面加入:

[plain] view plaincopyprint?在CODE上查看代码片派生到我的代码片ifeq ($(OS),Windows_NT)
LDFLAGS2 += -L../luajit-root/usr/local/openresty/luajit -lcyglua51
endif

ifeq ($(OS),Windows_NT)
LDFLAGS2 += -L../luajit-root/usr/local/openresty/luajit -lcyglua51
endif

change

[plain] view plaincopyprint?在CODE上查看代码片派生到我的代码片$(CC) $(LDFLAGS) -o $@ $^

$(CC) $(LDFLAGS) -o $@ $^
(cjson may be "$(CC) $(LDFLAGS) $(CJSON_LDFLAGS) -o $@ $(OBJS)")

to

[plain] view plaincopyprint?在CODE上查看代码片派生到我的代码片$(CC) $(LDFLAGS) -o $@ $^ $(LDFLAGS2)

$(CC) $(LDFLAGS) -o $@ $^ $(LDFLAGS2)

(cjson may be "$(CC) $(LDFLAGS) $(CJSON_LDFLAGS) -o $@ $(OBJS) $(LDFLAGS2)")

Then run:

[plain] view plaincopyprint?在CODE上查看代码片派生到我的代码片./configure --without-select_module --with-luajit
let ./configure can found your luajit:
mv ./build/luajit-root/usr/local/openresty/luajit/include/luajit-2.1 /usr/local/include
cp ./build/LuaJIT-2.1-20150223/src/cyglua51.dll /lib/libluajit-5.1.a
./configure --without-select_module --with-luajit (must do this again)

./configure --without-select_module --with-luajit
let ./configure can found your luajit:
mv ./build/luajit-root/usr/local/openresty/luajit/include/luajit-2.1 /usr/local/include
cp ./build/LuaJIT-2.1-20150223/src/cyglua51.dll /lib/libluajit-5.1.a
./configure --without-select_module --with-luajit (must do this again)

LuaJIT-2.1-20150223 my be wrong dir name, you need press tab after LuaJIT-2.1-

LuaJIT-2.1-20150223目录名不一定一样,你需要在LuaJIT-2.1-文字后直接按Tab换成正确的目录。

[plain] view plaincopyprint?在CODE上查看代码片派生到我的代码片let make can link your luajit:
cp ./build/LuaJIT-2.1-20150223/src/cyglua51.dll ./build/luajit-root/usr/local/openresty/luajit
make -j8
make install DESTDIR=/usr2
cp ./build/LuaJIT-2.1-20150223/src/cyglua51.dll /usr2/usr/local/openresty/nginx/sbin/

let make can link your luajit:
cp ./build/LuaJIT-2.1-20150223/src/cyglua51.dll ./build/luajit-root/usr/local/openresty/luajit
make -j8
make install DESTDIR=/usr2
cp ./build/LuaJIT-2.1-20150223/src/cyglua51.dll /usr2/usr/local/openresty/nginx/sbin/

All will be ok. You can find openresty in /usr2/usr/local/ folder.

Also the ffi may be a problem, because under cygwin, -E can not be exported function symbols, i have discussed this issue with the author Agentzh, may be next version can solve this problem (Already fixed in 1.4.3.9).

另外Cygwin编译的使用jit的ffi可能有问题,因为cygwin下-E无法导出函数符号,春哥已生成补丁,预估以后版本可以解决(1.4.3.9已解决)。

在windows下编译OpenResty

作者:W.S.T 发布时间:May 21, 2015 分类:Code No Comments

折腾了一天,终于解决了

首先,准备cygwin环境
cygwin下载setup.exe,启动并开始安装,建议选163源或者日本的源,速度较快

需要的安装的包及其devel包: openssl zlib pcre

还有就是一些编译需要的工具: gcc4 make perl lua (不使用openresty内置的lua)

下载openresty, openresty官网

并解压到 C:\cygwin\tmp下

修正lua的C模块编译脚本
共需要修正3个文件(其实就是3个模块),而且都是一样的修改. 版本号日新月异,自己搞定啦

打开 C:\cygwin\tmp\ngx_openresty-1.2.8.6\bundle\lua-cjson-1.0.3\Makefile, 加入 -llua5.1

然后就是如法炮制,修正rds和redis处理模块

开始编译吧,童鞋们!
启动cgywin

开始执行配置,注意,这里使用系统的lua,而非openresty内置的lua,原因就是cjson等模块会找不到内置的lua(配一下也可以,但麻烦)

cd /tmp/ngx_openresty-1.2.8.6

./configure --without-select_module --prefix=/opt/openresty --with-lua51=/usr

开始编译(按你的实际情况设置并发数哦,不然很久很久的)

make -j8 #8就是内核数,并行编译,按你的实际情况而定

编译完成

安装那点小事
如果你直接执行make install, 你会看到这些错误(也许?)

这个我也纠结了一段时间,然后改成这样执行,注意是/opt2,而非原本的/opt

make install DSETDIR=/opt2

你以为完了?其实还没有,你需要把名字改回去

rm -fr /opt/openresty
mv /opt2/opt/openresty /opt/
先简单测试一下
测试最基本的配置文件检查
/opt/openresty/nginx/sbin/nginx.exe -t

然后就是测试核心的lua调用
打开nginx.conf文件,添加一个location

location /lua/hi {
content_by_lua 'ngx.say("LUA Here")' ;
}
保存,启动nginx, 然后curl一下看看

/opt/openresty/nginx/sbin/nginx.exe
curl -v http://127.0.0.1/lua/hi

测试数据库连接
数据库的resty.mysql库需要LuaBitOP库(汗,为啥openresty不包含?)

下载LuaBitOP库, 猛击下载地址,并解压到C:\cygwin\tmp下

惯例了,修正编译参数

编译然后拷贝到openresty的lualib

cd /tmp/LuaBitOp-1.0.2
make
cp bit.so /opt/openresty/lualib
接下来,就是修改nginx.conf,加上官方的测试例子了

location /lua/mysql {
content_by_lua '

local mysql = require "resty.mysql"
local db, err = mysql:new()
if not db then
ngx.say("failed to instantiate mysql: ", err)
return
end

-- 省略1000字,自行到官网拷贝吧
' ;
}
让nginx重新加载配置,然后访问之

/opt/openresty/nginx/sbin/nginx.exe -s reload
curl -v http://127.0.0.1/lua/mysql
哦也,上截图

打完收工!! 但有啥不足呢?
没有luajit,反正我没弄出来, configure阶段总是找不到库,不管了,windows就不那么追求性能了