百萬長連接並發的限制因素
(1) CPU：使用top,然后按1查看，如果有邏輯CPU跑到100%，那就是受限了。多線程或線程綁定CPU都可以;
(2) 內存：本文主要討論內存限制;
1. 一個TCP連接默認占內存大小
針對長連接來講，監聽和connect的過程是幾乎不消耗內存的。內存主要消耗在滑動窗口的讀寫緩存上。
使用附錄中的代碼，可以看到默認一個TCP連接占用的內存大小有多大：
[root@localhost test]# ./gtop
recv_buf = 85k
send_buf = 16k
可以看到一個TCP連接，默認占用的內存大小=85k+16k=101KB，那么我們可以計算了32GB內存，按照內存使用率70%來算，應該能支持的
穩定長連接數=32GB*70%/101KB=232,555≈23萬
PS:因為connect並不耗內存，而內存有可能是會被共享的，所以最初的讀寫並不是每個TCP連接一下就分配了101KB的內存，而是最終會分配這么多。socket讀寫越頻繁越大，內存降的越快。
2. 如何調整
如果根據自己的業務，我們可以調整這個緩沖區大小。
比如，我們的場景是簡歷長連接保持會話，每個報文1KB左右，1min發一次，顯然用不到這么大的發送和接收緩沖區。
可以從程序中設置：(建議從程序中設置)
//c++代碼
int nRecvBuf=16*1024;
setsockopt(s,SOL_SOCKET,SO_RCVBUF,(const char*)&nRecvBuf,sizeof(int));
int nSendBuf=32*1024;
setsockopt(s,SOL_SOCKET,SO_SNDBUF,(const char*)&nSendBuf,sizeof(int));
或者在系統參數中設置:/etc/sysctl.conf(比較不建議這樣設置)
#/etc/sysctl.conf
net.ipv4.tcp_rmem = 4096 87380 2063281
net.ipv4.tcp_wmem = 4096 16384 2063281
net.core.wmem_default = 388608
net.core.rmem_default = 388608
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
真正起作用的是87380、16384兩個參數所在的位置。設置之后用sysctl -p同步
[root@localhost test]# sysctl -p
同步之后，可以用剛才那個gtop再取一下參數看是否生效了。
附錄一：報錯信息
未調內存之前壓力測試30萬的時候，發現內存一點點減少，但是使用ps和top等工具並沒有查看到任何耗費內存比較多的進程或線程。
當系統內存低於600MB時，系統宕機(並非真的死機，ssh連不上，usb串口連不上)，報錯如下
Centos Linux ( )
Kernel 3 ,10.0-514.e17.64 an
loadtest3 login :[ 16221.310569]Out of memory :Kill process 1183 ( tuned)score or sacrif ice child
[ 16221.3406031Killed process 1183 ( tuned)total-vm :562728kB,anon-rss :0kB ,file-rss :0kB ,shmem-rss :0kB
[ 16221.357102]of memory :Kill process 926 ( )score or sacrif ice child
[ 16221.357130]Killed process 926 ( polkitd)total-vm :538128kB ,anon-rss :0kB ,file-rss :0kB ,shmem-rss :
[ 16221.3599841of memory :Kill process 932 ( gmain)score or sacrifice child
[ 16221.360826]Killed process 932 ( gmain)total-vm :538428kB ,anon-rss :0kB ,file-rss :0kB ,shmem-rss :
[ 16221.3627391of memory :Kill process 33234 ( dstat)score 0 or sacrifice child
[ 16221.362766]Killed process 33234 ( dstat)total-vm :150232kB ,anon-rss :0kB ,file-rss :0kB,shmem-rss
[ 16230,318655]Out of memory :process 937 ( NetworkMlanagerscore or sacrif ice child
6230.310689 ]Killed process 937 ( NetworkManager)total- :,anon-rss :0kB ,file-rss :,shm
6230.311222 ]Out of memory :Kill process 948 ( gdbus)score or sacrifice child
16230.311222
16238.3112501K
Killed process ( gdbus)total-um :450644kB ,anon-rss :,file-rss :4kB ,shmem-rss :0kB
16448.675291
75291 ]INFO :task /::31 blocked for more than 120 seconds
[16448.675316]
53161 echo > /proc/sys/kernel/hung_task_timeout_secs"disables this messag
16448.675401
[16448.675426
401 ]INFO :task fsnotify mark :189 blocked for more than seconds
than seconds
16448.675494
"echo> /proc/sys/kernel/hungtask_timeoutsecs "disables this message
[16448.675518
4 ]INFO :task kworker /:1 :209 blocked for more than 120 seconds
16448.675588
] "echo0 > /proc/sys/kernel/hung_task_timeout_secs"disables this message
16448.675611
INFO :task gdbus :948 blocked for more than seconds
[16448.675789]
echo 0 > /proc/sys/kernel/hung_task_timeout_secs"disables this message
[16448.675734]
INFO :task kworker /:2 :blocked for more than seconds
[16568.679395
echo > /proc/sys/kerne/hung task timeout secs "disables this messag
[16568.679424
INFO :task kworker /4:0 :blocked for more than 120 seconds
[16568.679522
echo > /proc/sys/kernel/hungtask_timeout_secs"disables this messa
16568.679553]
INFO :task fsnotify mark :189 blocked for more than 120 seconds
16568.679639
echo > /proc/sys/kernel/hungtasktimeout _secs"disables this message
[16568.679669
INFO :task kworker /u96:1 :209 blocked for more than seconds
[16568.679760
echo > /proc/sys//hungtask "disables this message
[16568.679788
INFO :task gdbus :948 blocked for more than 120 seconds
[16568.6798931
echo 0 > /proc/sys/kernel/hungtask_timeout_secs"disables this message
[16568.679923
INFO :task kworker /4:2 :16027 blocked for more than 120 seconds
] "echo0 > /proc/sys/kernel/hung_task_timeout_secs"disables this message
附錄二：查詢收發滑動窗口大小的代碼(前文用到的gtop)
編譯命令：g++ main.cpp -o gtop
#include <stdio.h>
#include <string.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <poll.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <fcntl.h>
#include <sys/epoll.h>
#include <sys/time.h>
#include <sys/resource.h>
#include <string>
#define MAXLINE 4096
#define OPEN_MAX 16
#define SERV_PORT 1555
const int EPOLL_MAX_FDSIZE = 0x4000;
int main()
{
int i , maxi ,listenfd , connfd , sockfd ,epfd, nfds;
int n;
char buf[MAXLINE];
struct epoll_event ev, events[EPOLL_MAX_FDSIZE];
socklen_t clilen;
struct pollfd client[OPEN_MAX];
struct sockaddr_in cliaddr , servaddr;
listenfd = socket(AF_INET , SOCK_STREAM , 0);
memset(&servaddr,0,sizeof(servaddr));
servaddr.sin_family = AF_INET;
servaddr.sin_port = htons(SERV_PORT);
servaddr.sin_addr.s_addr = htonl(INADDR_ANY);
int opt_rc_val;
socklen_t opt_rc_len = sizeof(opt_rc_val);
int opt_sd_val;
socklen_t opt_sd_len = sizeof(opt_sd_val);
getsockopt(listenfd,SOL_SOCKET,SO_RCVBUF,&opt_rc_val, &opt_rc_len);
getsockopt(listenfd,SOL_SOCKET,SO_SNDBUF,&opt_sd_val, &opt_sd_len);
printf("recv_buf = %dk\n", opt_rc_val / 1024);
printf("send_buf = %dk\n", opt_sd_val / 1024);
return 0;
}