淺析libcurl多線程安全問題
背景：使用多線程libcurl發送請求，在未設置超時或長超時的情況下程序運行良好。但只要設置了較短超時（小於180s），程序就會出現隨機的coredump。並且棧里面找不到任何有用的信息。
問題：1.為什么未設置超時，或者長超時時間（比如601s）的情況下多線程libcurl不會core？
問題：2.進程coredump並不是必現，是否在libcurl內多線程同時修改了全局變量導致？
先來看下官方libcurl的說明：
libcurl is free, thread-safe, IPv6 compatible, feature rich, well supported, fast, thoroughly documented and is already used by many known, big and successful companies and numerous applications.
可以看到官方自稱licurl是線程安全的，是否真的如此？再來看看代碼中用到的超時選項的說明：
CURLOPT_TIMEOUT
Pass a long as parameter containing the maximum time in seconds that you allow the libcurl transfer operation to take. Normally, name lookups can take a considerable time and limiting operations to less than a few minutes risk aborting perfectly normal operations. This option will cause curl to use the SIGALRM to enable time-outing system calls.
In unix-like systems, this might cause signals to be used unless CURLOPT_NOSIGNAL is set.
Default timeout is 0 (zero) which means it never times out.
選項提到了超時機制是使用SIGALRM信號量來實現的，並且在unix-like操作系統中又提到了另外一個選項CURLOPT_NOSIGNAL：
CURLOPT_NOSIGNAL
Pass a long. If it is 1, libcurl will not use any functions that install signal handlers or any functions that cause signals to be sent to the process. This option is mainly here to allow multi-threaded unix applications to still set/use all timeout options etc, without risking getting signals. The default value for this parameter is 0. (Added in 7.10)
If this option is set and libcurl has been built with the standard name resolver, timeouts will not occur while the name resolve takes place. Consider building libcurl with c-ares support to enable asynchronous DNS lookups, which enables nice timeouts for name resolves without signals.
Setting CURLOPT_NOSIGNAL to 1 makes libcurl NOT ask the system to ignore SIGPIPE signals, which otherwise are sent by the system when trying to send data to a socket which is closed in the other end. libcurl makes an effort to never cause such SIGPIPEs to trigger, but some operating systems have no way to avoid them and even on those that have there are some corner cases when they may still happen, contrary to our desire. In addition, usingCURLAUTH_NTLM_WB authentication could cause a SIGCHLD signal to be raised.
該選項說明提到，為了在多線程中允許程序去設置timeout選項，但不是使用signals，需要設置CURLOPT_NOSIGNAL為1 。
於是在代碼中加上了這句，測試再沒有發現有coredump的情況。
1 easy_setopt(curl, CURLOPT_NOSIGNAL, (long)1);
問題：3.timeout機制實現機制是什么，為什么設置了選項CURLOPT_NOSIGNAL線程就安全了？
為了解答上面的問題，需要查看libcurl的相關源代碼，以下是DNS解析的函數：
1 int Curl_resolv_timeout(struct connectdata *conn, 2 const char *hostname, 3 int port, 4 struct Curl_dns_entry **entry, 5 long timeoutms) 6 { 7 #ifdef USE_ALARM_TIMEOUT 8 #ifdef HAVE_SIGACTION 9 struct sigaction keep_sigact; /* store the old struct here */ 10 volatile bool keep_copysig = FALSE; /* wether old sigact has been saved */ 11 struct sigaction sigact; 12 #else 13 #ifdef HAVE_SIGNAL 14 void (*keep_sigact)(int); /* store the old handler here */ 15 #endif /* HAVE_SIGNAL */ 16 #endif /* HAVE_SIGACTION */ 17 volatile long timeout; 18 volatile unsigned int prev_alarm = 0; 19 struct SessionHandle *data = conn->data; 20 #endif /* USE_ALARM_TIMEOUT */ 21 int rc; 22 23 *entry = NULL; 24 25 if(timeoutms < 0) 26 /* got an already expired timeout */ 27 return CURLRESOLV_TIMEDOUT; 28 29 #ifdef USE_ALARM_TIMEOUT 30 if(data->set.no_signal) 31 /* Ignore the timeout when signals are disabled */ 32 timeout = 0; 33 else 34 timeout = timeoutms; 35 36 if(!timeout) 37 /* USE_ALARM_TIMEOUT defined, but no timeout actually requested */ 38 return Curl_resolv(conn, hostname, port, entry); 39 40 if(timeout < 1000) 41 /* The alarm() function only provides integer second resolution, so if 42 we want to wait less than one second we must bail out already now. */ 43 return CURLRESOLV_TIMEDOUT; 44 45 /************************************************************* 46 * Set signal handler to catch SIGALRM 47 * Store the old value to be able to set it back later! 48 *************************************************************/ 49 #ifdef HAVE_SIGACTION 50 sigaction(SIGALRM, NULL, &sigact); 51 keep_sigact = sigact; 52 keep_copysig = TRUE; /* yes, we have a copy */ 53 sigact.sa_handler = alarmfunc; 54 #ifdef SA_RESTART 55 /* HPUX doesn't have SA_RESTART but defaults to that behaviour! */ 56 sigact.sa_flags &= ~SA_RESTART; 57 #endif 58 /* now set the new struct */ 59 sigaction(SIGALRM, &sigact, NULL); 60 #else /* HAVE_SIGACTION */ 61 /* no sigaction(), revert to the much lamer signal() */ 62 #ifdef HAVE_SIGNAL 63 keep_sigact = signal(SIGALRM, alarmfunc); 64 #endif 65 #endif /* HAVE_SIGACTION */ 66 67 /* alarm() makes a signal get sent when the timeout fires off, and that 68 will abort system calls */ 69 prev_alarm = alarm(curlx_sltoui(timeout/1000L)); 70 71 /* This allows us to time-out from the name resolver, as the timeout 72 will generate a signal and we will siglongjmp() from that here. 73 This technique has problems (see alarmfunc). 74 This should be the last thing we do before calling Curl_resolv(), 75 as otherwise we'd have to worry about variables that get modified 76 before we invoke Curl_resolv() (and thus use "volatile"). */ 77 if(sigsetjmp(curl_jmpenv, 1)) { 78 /* this is coming from a siglongjmp() after an alarm signal */ 79 failf(data, "name lookup timed out"); 80 rc = CURLRESOLV_ERROR; 81 goto clean_up; 82 } 83 84 #else 85 #ifndef CURLRES_ASYNCH 86 if(timeoutms) 87 infof(conn->data, "timeout on name lookup is not supported\n"); 88 #else 89 (void)timeoutms; /* timeoutms not used with an async resolver */ 90 #endif 91 #endif /* USE_ALARM_TIMEOUT */ 92 93 /* Perform the actual name resolution. This might be interrupted by an 94 * alarm if it takes too long. 95 */ 96 rc = Curl_resolv(conn, hostname, port, entry); 97 98 #ifdef USE_ALARM_TIMEOUT 99 clean_up: 100 101 if(!prev_alarm) 102 /* deactivate a possibly active alarm before uninstalling the handler */ 103 alarm(0); 104 105 #ifdef HAVE_SIGACTION 106 if(keep_copysig) { 107 /* we got a struct as it looked before, now put that one back nice 108 and clean */ 109 sigaction(SIGALRM, &keep_sigact, NULL); /* put it back */ 110 } 111 #else 112 #ifdef HAVE_SIGNAL 113 /* restore the previous SIGALRM handler */ 114 signal(SIGALRM, keep_sigact); 115 #endif 116 #endif /* HAVE_SIGACTION */ 117 118 /* switch back the alarm() to either zero or to what it was before minus 119 the time we spent until now! */ 120 if(prev_alarm) { 121 /* there was an alarm() set before us, now put it back */ 122 unsigned long elapsed_ms = Curl_tvdiff(Curl_tvnow(), conn->created); 123 124 /* the alarm period is counted in even number of seconds */ 125 unsigned long alarm_set = prev_alarm - elapsed_ms/1000; 126 127 if(!alarm_set || 128 ((alarm_set >= 0x80000000) && (prev_alarm < 0x80000000)) ) { 129 /* if the alarm time-left reached zero or turned "negative" (counted 130 with unsigned values), we should fire off a SIGALRM here, but we 131 won't, and zero would be to switch it off so we never set it to 132 less than 1! */ 133 alarm(1); 134 rc = CURLRESOLV_TIMEDOUT; 135 failf(data, "Previous alarm fired off!"); 136 } 137 else 138 alarm((unsigned int)alarm_set); 139 } 140 #endif /* USE_ALARM_TIMEOUT */ 141 142 return rc; 143 }
由此可見，DNS解析階段timeout的實現機制是通過SIGALRM+sigsetjmp/siglongjmp來實現的。
解析前，通過alarm設定超時時間，並設置跳轉的標記：
1 /* alarm() makes a signal get sent when the timeout fires off, and that 2 will abort system calls */ 3 prev_alarm = alarm(curlx_sltoui(timeout/1000L)); 4 5 /* This allows us to time-out from the name resolver, as the timeout 6 will generate a signal and we will siglongjmp() from that here. 7 This technique has problems (see alarmfunc). 8 This should be the last thing we do before calling Curl_resolv(), 9 as otherwise we'd have to worry about variables that get modified 10 before we invoke Curl_resolv() (and thus use "volatile"). */ 11 if(sigsetjmp(curl_jmpenv, 1)) { 12 /* this is coming from a siglongjmp() after an alarm signal */ 13 failf(data, "name lookup timed out"); 14 rc = CURLRESOLV_ERROR; 15 goto clean_up; 16 }
在等到超時后，進入alarmfunc函數實現跳轉：
1 #ifdef USE_ALARM_TIMEOUT 2 /* 3 * This signal handler jumps back into the main libcurl code and continues 4 * execution. This effectively causes the remainder of the application to run 5 * within a signal handler which is nonportable and could lead to problems. 6 */ 7 static 8 RETSIGTYPE alarmfunc(int sig) 9 { 10 /* this is for "-ansi -Wall -pedantic" to stop complaining! (rabe) */ 11 (void)sig; 12 siglongjmp(curl_jmpenv, 1); 13 return; 14 } 15 #endif /* USE_ALARM_TIMEOUT */
而CURLOPT_NOSIGNAL選項的作用是什么呢？
1 case CURLOPT_NOSIGNAL: 2 /* 3 * The application asks not to set any signal() or alarm() handlers, 4 * even when using a timeout. 5 */ 6 data->set.no_signal = (0 != va_arg(param, long))?TRUE:FALSE; 7 break;
再回過頭看看DNS解析的那段代碼，你會發現在超時設定前有如下代碼：
1 #ifdef USE_ALARM_TIMEOUT 2 if(data->set.no_signal) 3 /* Ignore the timeout when signals are disabled */ 4 timeout = 0; 5 else 6 timeout = timeoutms; 7 8 if(!timeout) 9 /* USE_ALARM_TIMEOUT defined, but no timeout actually requested */ 10 return Curl_resolv(conn, hostname, port, entry);
沒錯！設置了CURLOPT_NOSIGNAL選項，會把超時時間設置為0，也就是DNS解析不設置超時時間！以此來繞過SIGALRM+sigsetjmp/siglongjmp的超時機制。以此引來新的問題，DNS解析沒有超時限制，不過這個官方有推薦的解決方法了。
了解了這些選項的原理之后，回到問題3，為什么使用了CURLOPT_NOSIGNAL選項后就保證了線程安全？繼續看sigsetjmp/siglongjmp實現就會發現：
1 #ifdef HAVE_SIGSETJMP 2 /* Beware this is a global and unique instance. This is used to store the 3 return address that we can jump back to from inside a signal handler. This 4 is not thread-safe stuff. */ 5 sigjmp_buf curl_jmpenv; 6 #endif
sigsetjmp/siglongjmp使用的curl_jmpenv是個全局唯一的變量！多個線程都會去修改該變量，破壞了棧的內容並導致coredump。看來這還是libcurl的實現問題，如果每個線程都有一個sigjmp_buf變量，是否就可以解決上面的問題呢？
看到這里，問題2也有了答案：當多個線程同時修改sigjmp_buf會出現問題，但線程間是串行的sigsetjmp/siglongjmp並不會出現問題，這有一定的隨機性。
問題1，未設置超時不會有問題這很好理解，但是為什么設置長超時也不會出現問題？
原因就是在libcurl超時前，apache服務器端先超時返回了。apache超時時間一般是180s。只要libcurl超時大於180s，libcurl客戶端永遠都不會觸發超時。而是直接返回504的錯誤。
是否設置了CURLOPT_NOSIGNAL就可以保證線程安全了呢？官方文檔還提到了另外兩個函數：
CURL *curl_easy_init( );
This function must be the first function to call, and it returns a CURL easy handle that you must use as input to other easy-functions. curl_easy_init initializes curl and this call MUST have a corresponding call to curl_easy_cleanup(3) when the operation is complete.
If you did not already call curl_global_init(3), curl_easy_init(3) does it automatically. This may be lethal in multi-threaded cases, since curl_global_init(3) is not thread-safe, and it may result in resource problems because there is no corresponding cleanup.
You are strongly advised to not allow this automatic behaviour, by calling curl_global_init(3) yourself properly. See the description in libcurl(3) of global environment requirements for details of how to use this function.
其中curl_easy_init函數體內會調用curl_global_init，而后者是非線程安全的。
在curl_easy_init函數體內，有且僅調用一次curl_global_init：
1 /* 2 * curl_easy_init() is the external interface to alloc, setup and init an 3 * easy handle that is returned. If anything goes wrong, NULL is returned. 4 */ 5 CURL *curl_easy_init(void) 6 { 7 CURLcode res; 8 struct SessionHandle *data; 9 10 /* Make sure we inited the global SSL stuff */ 11 if(!initialized) { 12 res = curl_global_init(CURL_GLOBAL_DEFAULT); 13 if(res) { 14 /* something in the global init failed, return nothing */ 15 DEBUGF(fprintf(stderr, "Error: curl_global_init failed\n")); 16 return NULL; 17 } 18 } 19 20 /* We use curl_open() with undefined URL so far */ 21 res = Curl_open(&data); 22 if(res != CURLE_OK) { 23 DEBUGF(fprintf(stderr, "Error: Curl_open failed\n")); 24 return NULL; 25 } 26 27 return data; 28 }
但是在curl_global_init函數體內，是非線程安全的。initialized++並非原子操作，有可能出現多個線程重復執行curl_global_init。
1 CURLcode curl_global_init(long flags) 2 { 3 if(initialized++) 4 return CURLE_OK; 5 6 /* Setup the default memory functions here (again) */ 7 Curl_cmalloc = (curl_malloc_callback)malloc; 8 Curl_cfree = (curl_free_callback)free; 9 Curl_crealloc = (curl_realloc_callback)realloc; 10 Curl_cstrdup = (curl_strdup_callback)system_strdup; 11 Curl_ccalloc = (curl_calloc_callback)calloc; 12 #if defined(WIN32) && defined(UNICODE) 13 Curl_cwcsdup = (curl_wcsdup_callback)_wcsdup; 14 #endif 15 16 if(flags & CURL_GLOBAL_SSL) 17 if(!Curl_ssl_init()) { 18 DEBUGF(fprintf(stderr, "Error: Curl_ssl_init failed\n")); 19 return CURLE_FAILED_INIT; 20 } 21 22 if(flags & CURL_GLOBAL_WIN32) 23 if(win32_init() != CURLE_OK) { 24 DEBUGF(fprintf(stderr, "Error: win32_init failed\n")); 25 return CURLE_FAILED_INIT; 26 } 27 28 #ifdef __AMIGA__ 29 if(!Curl_amiga_init()) { 30 DEBUGF(fprintf(stderr, "Error: Curl_amiga_init failed\n")); 31 return CURLE_FAILED_INIT; 32 } 33 #endif 34 35 #ifdef NETWARE 36 if(netware_init()) { 37 DEBUGF(fprintf(stderr, "Warning: LONG namespace not available\n")); 38 } 39 #endif 40 41 #ifdef USE_LIBIDN 42 idna_init(); 43 #endif 44 45 if(Curl_resolver_global_init() != CURLE_OK) { 46 DEBUGF(fprintf(stderr, "Error: resolver_global_init failed\n")); 47 return CURLE_FAILED_INIT; 48 } 49 50 #if defined(USE_LIBSSH2) && defined(HAVE_LIBSSH2_INIT) 51 if(libssh2_init(0)) { 52 DEBUGF(fprintf(stderr, "Error: libssh2_init failed\n")); 53 return CURLE_FAILED_INIT; 54 } 55 #endif 56 57 if(flags & CURL_GLOBAL_ACK_EINTR) 58 Curl_ack_eintr = 1; 59 60 init_flags = flags; 61 62 return CURLE_OK; 63 }
所以curl_global_init需要在單線程中執行，例如在程序的開頭。
最后，貼一個官方給出的多線程例子，稍作修改（docs/examples/multithreads.c）：
1 /*************************************************************************** 2 * _ _ ____ _ 3 * Project ___| | | | _ \| | 4 * / __| | | | |_) | | 5 * | (__| |_| | _ <| |___ 6 * \___|\___/|_| \_\_____| 7 * 8 * Copyright (C) 1998 - 2011, Daniel Stenberg, <daniel@haxx.se>, et al. 9 * 10 * This software is licensed as described in the file COPYING, which 11 * you should have received as part of this distribution. The terms 12 * are also available at http://curl.haxx.se/docs/copyright.html. 13 * 14 * You may opt to use, copy, modify, merge, publish, distribute and/or sell 15 * copies of the Software, and permit persons to whom the Software is 16 * furnished to do so, under the terms of the COPYING file. 17 * 18 * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY 19 * KIND, either express or implied. 20 * 21 ***************************************************************************/ 22 /* A multi-threaded example that uses pthreads extensively to fetch 23 * X remote files at once */ 24 25 #include <stdio.h> 26 #include <pthread.h> 27 #include <curl/curl.h> 28 29 #define NUMT 4 30 31 /* 32 List of URLs to fetch. 33 34 If you intend to use a SSL-based protocol here you MUST setup the OpenSSL 35 callback functions as described here: 36 37 http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION 38 39 */ 40 const char * const urls[NUMT]= { 41 "http://curl.haxx.se/", 42 "ftp://cool.haxx.se/", 43 "http://www.contactor.se/", 44 "www.haxx.se" 45 }; 46 47 static void *pull_one_url(void *url) 48 { 49 CURL *curl; 50 51 curl = curl_easy_init(); 52 curl_easy_setopt(curl, CURLOPT_URL, url); 53 curl_easy_setopt(curl, CURLOPT_TIMEOUT, 30L); /*timeout 30s,add by edgeyang*/ 54 curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1L); /*no signal,add by edgeyang*/ 55 curl_easy_perform(curl); /* ignores error */ 56 curl_easy_cleanup(curl); 57 58 return NULL; 59 } 60 61 62 /* 63 int pthread_create(pthread_t *new_thread_ID, 64 const pthread_attr_t *attr, 65 void * (*start_func)(void *), void *arg); 66 */ 67 68 int main(int argc, char **argv) 69 { 70 pthread_t tid[NUMT]; 71 int i; 72 int error; 73 74 /* Must initialize libcurl before any threads are started */ 75 curl_global_init(CURL_GLOBAL_ALL); 76 77 for(i=0; i< NUMT; i++) { 78 error = pthread_create(&tid[i], 79 NULL, /* default attributes please */ 80 pull_one_url, 81 (void *)urls[i]); 82 if(0 != error) 83 fprintf(stderr, "Couldn't run thread number %d, errno %d\n", i, error); 84 else 85 fprintf(stderr, "Thread %d, gets %s\n", i, urls[i]); 86 } 87 88 /* now wait for all threads to terminate */ 89 for(i=0; i< NUMT; i++) { 90 error = pthread_join(tid[i], NULL); 91 fprintf(stderr, "Thread %d terminated\n", i); 92 } 93 94 curl_global_cleanup(); /*add by edgeyang*/ 95 return 0; 96 }