文件同步傳輸工具比較多,傳輸的方式也比較多,比如:FTP、共享、HTTP等,我這里要講的就是基於HTTP協議的WEB API實現批量文件由一個服務器同步快速傳輸到其它多個服務器這樣的一個工具(簡稱:一端到多端的文件同步工具)
一、設計原理:
1.使用的技術:WinForm、WebApi
1.1 WinForm:為程序主界面,作為一端(一個源文件服務器)同步傳輸到多端(多個目的文件服務器)文件的業務處理中介;程序內部主要通過System.Timers.Timer+HttpClient來實現定時執行文件同步傳輸業務;
1.2 WebApi:實現通過HTTP協議批量下載或批量上傳多個文件(文件同步傳輸的核心業務邏輯);MultipartContent作為批量下載或批量上傳的唯一媒介。
2.實現思路:
2.1客戶端(WinForm程序主界面)通過HttpClient向源文件服務器目錄URL發送GET請求;
2.2源文件服務器服務端(WebApi)的GetFiles方法接收到GET請求后,按照web.config中配置的源文件路徑遞歸獲取所有文件的字節信息並轉換成MultipartFormDataContent對象后返回;(即:實現了批量下載)
2.3客戶端(WinForm程序主界面)將響應的結果顯式轉換並生成對應的目的文件服務器數量的多文件流內容(MultipartContent)對象列表,以便后續用於批量上傳;
2.4客戶端(WinForm程序主界面)啟用並行循環(Parallel.ForEach)來遍歷目的文件服務器目錄URL(采用並行循環是為了達到同時向多個目的文件服務器批量上傳文件的效果,從而提高運行效率),在循環遍歷中,每次將2.3中獲得的多文件流內容(MultipartContent)通過HttpClient向目的文件服務器目錄URL發送POST請求;
2.5目的文件服務器服務端(WebApi)的SaveFiles方法接收到POST請求后,解析2.4中POST過來的多文件流內容(MultipartContent),並循環遍歷文件流,在循環遍歷中,按照web.config中配置的上傳文件路徑,將文件流輸出保存到指定的文件路徑下,同時生成文件保存成功與失敗的日志信息字典,最后返回該字典。(即:實現了批量上傳保存)
2.6客戶端(WinForm程序主界面)將響應的結果顯式轉換成保存成功與失敗的日志信息字典,並添加到線程安全的無序集合對象中;(采用線程安全的無序集合對象是因為存在多線程並發更新的風險)
2.7客戶端(WinForm程序主界面)等待所有並行循環同步上傳執行完畢后,根據最后得到的保存成功與失敗的日志信息無序集合對象,獲得所有目的文件服務器全部保存成功文件名列表及保存成功與失敗的日志信息列表(判斷是否全部上傳成功:若某個文件應上傳到5個目的文件服務器,實際成功上傳5個,則視為成功,否則有一個未上傳成功則視為失敗),然后通過HttpClient向源文件服務器目錄URL發送PUT請求刪除源文件服務器中的同名文件,向源文件服務器LOG URL發送POST請求將此次文件同步傳輸的日志保存到源文件服務器目錄中
2.8源文件服務器服務端(WebApi)RemoveFiles方法接收到PUT請求后,循環遍歷PUT過來的保存成功文件名列表,依次刪除同名文件(含子目錄),WriteLog方法接收到POST請求后,直接將POST過來的日志信息列表輸出保存至源文件服務器web.config中配置的LOG文件路徑;為了避免日志無限增大及考慮日志的使用價值,日志文件每天重新覆蓋生成新的文件;
3.業務流程圖:
序列圖:
二、使用說明:
1.將KyFileSyncTransfer.Api項目分別發布到源文件服務器、目的文件服務器;
2.修改源文件服務器服務端(WebApi)web.config中的AppSettings節點,配置FileSyncDirectory(源文件存放目錄)、LogDirectory(日志文件保存路徑,僅限源文件服務器服務端配置),這兩個路徑支持當前項目的子目錄(即:配置時使用~/)或其它路徑(其它路徑則需直接配置完整的物理路徑);修改目的文件服務器服務端(WebApi)web.config中的AppSettings節點,配置FileSyncDirectory(上傳文件存放目錄),這個路徑支持當前項目的子目錄(即:配置時使用~/)或其它路徑(其它路徑則需直接配置完整的物理路徑)
注:為了能夠支持大文件批量上傳,同時需修改請求內容長度限制,如下設置成最大批量上傳1.9G:
3.將客戶端(WinForm程序主界面)部署到某台服務器上(只要能夠訪問源文件服務器、目的文件服務器即可,也可以他們服務器上的任意一台),然后開啟客戶端(WinForm程序主界面),將上述的源文件服務器服務端URL(http://源文件服務器Host/Api/Files)及多個目的文件服務器服務端URL(http://目的文件服務器Host/Api/Files)錄入到程序預設的地方,設置好時間間隔,最后點擊開啟即可(需保持該程序一直處於運行中,可以最小化到任務欄,雙擊圖標可以顯示界面);若需停止文件同步,點擊停止按鈕即可;若需查看運行日志,可以切換到運行日志頁簽瀏覽。
4.以上3步是完成了文件自動定時同步傳輸的所有工作,后續只需要將需要同步的文件放到源文件服務器服務端web.config中的AppSettings節點設置的FileSyncDirectory(源文件存放目錄)即可。
運行效果如下:
三、貼出主要的源代碼
服務端代碼(WEB API代碼,需要進行文件傳輸的每個服務器均需要部署該WEB API站點)
FilesController:(實現:批量下載文件、批量上傳文件、批量刪除文件、批量寫日志信息)
using System; using System.Collections.Generic; using System.IO; using System.Linq; using System.Net; using System.Net.Http; using System.Web; using System.Web.Http; using KyFileSyncTransfer.Api.Common; using System.Net.Http.Headers; using System.Net.Http.Formatting; using System.Threading.Tasks; namespace KyFileSyncTransfer.Api.Controllers { public class FilesController : ApiController { private static string fileSyncDirectory = null; private static string logDirectory = null; static FilesController() { fileSyncDirectory = BaseUtility.GetDirectoryFromConfig("FileSyncDirectory"); logDirectory = BaseUtility.GetDirectoryFromConfig("LogDirectory"); } /// <summary> /// 從源文件服務器獲取所有文件信息(采用JSON方式) /// </summary> /// <returns></returns> [HttpGet, Route("~/api/downfiles")] public IHttpActionResult GetFilesForJson() { if (!Directory.Exists(fileSyncDirectory)) { return BadRequest("同步文件目錄不存在或未配置。"); } Dictionary<string, byte[]> files = new Dictionary<string, byte[]>(); BaseUtility.LoadFileDatas(files, fileSyncDirectory); files = files.ToDictionary(kv => kv.Key.Replace(fileSyncDirectory, ""), kv => kv.Value); return Json(files); } /// <summary> /// 將所有文件同步保存到目的文件服務器(采用JSON方式) /// </summary> /// <param name="files"></param> /// <returns></returns> [HttpPost, Route("~/api/upfiles")] public IHttpActionResult SaveFilesForJson([FromBody]IDictionary<string, byte[]> files) { string requestUrl = HttpContext.Current.Request.Url.ToString(); var savedErrors = new Dictionary<string, string>(); if (files == null || !Directory.Exists(fileSyncDirectory)) { return BadRequest(); } foreach (var item in files) { string file = item.Key; string filePath = Path.GetDirectoryName(fileSyncDirectory + file); try { if (!Directory.Exists(filePath)) { Directory.CreateDirectory(filePath); } string saveFilePath = Path.Combine(filePath, Path.GetFileName(file)); File.WriteAllBytes(saveFilePath, item.Value); } catch (Exception ex) { savedErrors[file] = string.Format("[{0:yyyy-MM-dd HH:mm:ss}] -請求:{1}同步文件:{2}失敗,原因:{3}", DateTime.Now, requestUrl, file, ex.Message); } } return Json(savedErrors); } /// <summary> /// 從源文件服務器獲取所有文件信息 /// </summary> /// <returns></returns> [HttpGet] public HttpResponseMessage GetFiles() { if (!Directory.Exists(fileSyncDirectory)) { return Request.CreateErrorResponse(HttpStatusCode.BadRequest, "同步文件目錄不存在或未配置。"); } var response = new HttpResponseMessage(HttpStatusCode.OK); var content = new MultipartFormDataContent(); BaseUtility.CreateMultipartFormDataContent(content, fileSyncDirectory, fileSyncDirectory); response.Content = content; return response; } /// <summary> /// 將所有文件同步保存到目的文件服務器 /// </summary> /// <returns></returns> [HttpPost] public HttpResponseMessage SaveFiles() { if (!Request.Content.IsMimeMultipartContent()) { return Request.CreateErrorResponse(HttpStatusCode.UnsupportedMediaType, "未上傳任何文件"); } if (!Directory.Exists(fileSyncDirectory)) { return Request.CreateErrorResponse(HttpStatusCode.BadRequest, "未找到文件同步上傳目錄:" + fileSyncDirectory); } string requestUrl = HttpContext.Current.Request.Url.ToString(); Dictionary<string, Dictionary<string, string>> savedResult = new Dictionary<string, Dictionary<string, string>>(); var provider = new MultipartMemoryStreamProvider(); const string success = "success"; const string failure = "failure"; try { savedResult[success] = new Dictionary<string, string>(); savedResult[failure] = new Dictionary<string, string>(); //Request.Content.ReadAsMultipartAsync(provider).Wait(); Task.Run(async () => await Request.Content.ReadAsMultipartAsync(provider)).Wait(); foreach (var item in provider.Contents) { string fileName = item.Headers.ContentDisposition.FileName; if (string.IsNullOrEmpty(fileName)) { continue; } var fileData = item.ReadAsByteArrayAsync().Result; fileName = BaseUtility.ReviseFileName(fileName); string saveFilePath = fileSyncDirectory + fileName; string fileBasePath = Path.GetDirectoryName(saveFilePath); try { if (!Directory.Exists(fileBasePath)) { Directory.CreateDirectory(fileBasePath); } File.WriteAllBytes(saveFilePath, fileData); savedResult[success][fileName] = string.Format("[{0:yyyy-MM-dd HH:mm:ss}] - V 請求:{1}同步文件:<{2}>成功。", DateTime.Now, requestUrl, fileName); } catch (Exception ex) { while(ex.InnerException!=null) { ex = ex.InnerException; } savedResult[failure][fileName] = string.Format("[{0:yyyy-MM-dd HH:mm:ss}] - X 請求:{1}同步文件:<{2}>失敗,原因:{3}", DateTime.Now, requestUrl, fileName, ex.Message); } } } catch (Exception ex) { while (ex.InnerException != null) { ex = ex.InnerException; } return Request.CreateErrorResponse(HttpStatusCode.BadRequest, ex.Message); } return Request.CreateResponse(HttpStatusCode.OK, savedResult); } /// <summary> /// 移除源文件服務器指定的文件 /// </summary> /// <param name="files"></param> /// <returns></returns> [HttpPut] public IHttpActionResult RemoveFiles([FromBody]IEnumerable<string> files) { if (files == null || !Directory.Exists(fileSyncDirectory)) { return BadRequest(); } foreach (string file in files) { string filePath = Path.Combine(fileSyncDirectory, file); if (File.Exists(filePath)) { File.Delete(filePath); } } return Ok(); } /// <summary> /// 將同步的日志信息寫入到源文件服務器LOG文件中 /// </summary> /// <param name="savedErrors"></param> /// <returns></returns> [HttpPost] [Route("~/Api/Files/Log")] public IHttpActionResult WriteLog([FromBody] IEnumerable<string> savedResult) { if (!Directory.Exists(logDirectory)) { return BadRequest("同步日志目錄不存在或未配置。"); } BaseUtility.WriteLogToFile(logDirectory, savedResult.ToArray()); return Ok(); } } }
WebApiConfig:(增加全局token驗證及全部采用json數據返回)
using System; using System.Collections.Generic; using System.Linq; using System.Web.Http; using KyFileSyncTransfer.Api.Models; using System.Net.Http.Formatting; namespace FileSyncTransfer.Api { public static class WebApiConfig { public static void Register(HttpConfiguration config) { // Web API 配置和服務 // Web API 路由 config.MapHttpAttributeRoutes(); config.Routes.MapHttpRoute( name: "DefaultApi", routeTemplate: "api/{controller}/{id}", defaults: new { id = RouteParameter.Optional } ); config.Filters.Add(new TokenAuthentificationAttribute()); var jsonFormatter = new JsonMediaTypeFormatter(); config.Services.Replace(typeof(IContentNegotiator), new JsonContentNegotiator(jsonFormatter)); } } }
JsonContentNegotiator:
public class JsonContentNegotiator : IContentNegotiator { private readonly JsonMediaTypeFormatter _jsonFormatter; public JsonContentNegotiator(JsonMediaTypeFormatter formatter) { _jsonFormatter = formatter; } public ContentNegotiationResult Negotiate(Type type, HttpRequestMessage request, IEnumerable<MediaTypeFormatter> formatters) { var result = new ContentNegotiationResult(_jsonFormatter, new MediaTypeHeaderValue("application/json")); return result; } }
TokenAuthentificationAttribute:
public class TokenAuthentificationAttribute : AuthorizationFilterAttribute { public override void OnAuthorization(System.Web.Http.Controllers.HttpActionContext actionContext) { if (actionContext.ActionDescriptor.GetCustomAttributes<AllowAnonymousAttribute>().Count > 0) { base.OnAuthorization(actionContext); return; } //HttpContextBase context = (HttpContextBase)actionContext.Request.Properties["MS_HttpContext"];//獲取傳統context //HttpRequestBase request = context.Request;//定義傳統request對象 IEnumerable<string> requestToken = null; if (actionContext.Request.Headers.TryGetValues("token", out requestToken) && BaseUtility.ValidateToken(requestToken.ElementAt(0))) { base.OnAuthorization(actionContext); } else { actionContext.Response = actionContext.Request.CreateErrorResponse(HttpStatusCode.Unauthorized, "token驗證未通過。"); return; } } }
BaseUtility:(通用方法類)
using System; using System.Collections.Generic; using System.Linq; using System.Web; using System.Configuration; using System.Net.Http; using System.IO; using System.Net.Http.Headers; namespace KyFileSyncTransfer.Api.Common { public static class BaseUtility { public static string GetDirectoryFromConfig(string cfgName) { string dir = ConfigurationManager.AppSettings[cfgName]; if (string.IsNullOrEmpty(dir)) { return null; } if (dir.Contains('~')) { dir = HttpContext.Current.Server.MapPath(dir); } if (!Directory.Exists(dir)) { Directory.CreateDirectory(dir); } return dir; } public static MultipartFormDataContent CreateMultipartFormDataContent(MultipartFormDataContent content, string removeRootDir, string dir) { foreach (string file in Directory.GetFileSystemEntries(dir)) { if (File.Exists(file)) { byte[] fileBytes = File.ReadAllBytes(file); var fileContent = new ByteArrayContent(fileBytes); fileContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data") { Name = "files", FileName = file.Replace(removeRootDir, "") }; fileContent.Headers.ContentType = new MediaTypeHeaderValue(MimeMapping.GetMimeMapping(file)); fileContent.Headers.ContentLength = fileBytes.LongLength; content.Add(fileContent); } else { CreateMultipartFormDataContent(content, removeRootDir, file); } } return content; } public static void LoadFileDatas(Dictionary<string, byte[]> files, string path) { foreach (string file in Directory.GetFileSystemEntries(path)) { if (File.Exists(file)) { files[file] = File.ReadAllBytes(file); } else { LoadFileDatas(files, file); } } } public static void WriteLogToFile(string logDir, params string[] contents) { string logFilePath = Path.Combine(logDir, "KyFileSyncTransfer.log"); if (File.Exists(logFilePath) && !File.GetLastWriteTime(logFilePath).Date.Equals(DateTime.Today)) { File.Delete(logFilePath); } File.AppendAllLines(logFilePath, contents, System.Text.Encoding.UTF8); } public static bool ValidateToken(string token) { try { token = EncryptUtility.Decrypt(token); var tokenParts = token.Split(new[] { "-", string.Empty }, StringSplitOptions.RemoveEmptyEntries); if (tokenParts.Length != 2) { return false; } if (tokenParts[0] == string.Join(string.Empty, "KyFileSyncTransfer.Api".OrderBy(c => c))) //對固定KEY進行排序然后比對 { long tokenTstamp = -1; long svrTokenTimeStamp = GetTimeStamp(); if (long.TryParse(tokenParts[1], out tokenTstamp) && svrTokenTimeStamp - tokenTstamp <= 10) //時間戳<=10則視為有效 { return true; } } } catch { } return false; } public static string ReviseFileName(string fileName) { var regex = new System.Text.RegularExpressions.Regex("^\"+(?<name>.*)\"+$"); var matchResult = regex.Match(fileName); if (matchResult != null && matchResult.Length > 0) { return matchResult.Groups["name"].Value; } return fileName; } /// <summary> /// 獲取時間戳 /// </summary> /// <returns></returns> private static long GetTimeStamp() { TimeSpan ts = DateTime.UtcNow - new DateTime(1970, 1, 1, 0, 0, 0, 0); return Convert.ToInt64(ts.TotalSeconds); } } }
客戶端代碼:(這里面有一個需要注意的地方就是:GetNeedSyncTransferFilesDatas方法,這個是將從源文件服務器下載有流轉換成多個副本的多文件流對象,之前是用的GetNeedSyncTransferFilesData方法,但MultipartContent是一種字節流對象,一旦被用於請求后將會被關閉,再次使用時就會報錯)
using System; using System.Collections.Generic; using System.ComponentModel; using System.Data; using System.Drawing; using System.Linq; using System.Text; using System.Threading.Tasks; using System.Windows.Forms; using System.Threading; using KyFileSyncTransfer.Business; using System.Diagnostics; using System.Net.Http; using System.Collections.Concurrent; using System.IO; using System.Runtime.Serialization.Formatters.Binary; using System.Net.Http.Headers; namespace KyFileSyncTransfer { public partial class FrmMain : Form { private const string appVersion = "16.1215.1"; private FormWindowState thisFormWindowState; private System.Timers.Timer appTimer = null; private static int syncFlag = 0; private static object syncObj = new object(); private static DateTime lastRunTime = DateTime.MinValue; private int runInterval = 10; private string srcFileApiUrl = null; private List<WebApiUrlInfo> destFileApiUrlList = null; private const string success = "success"; private const string failure = "failure"; private const string RunInterval = "RunInterval"; private const string SrcFileApiUrl = "SrcFileApiUrl"; private const string DestFileApiUrls = "DestFileApiUrls"; public FrmMain() { InitializeComponent(); } #region 自定義方法區域 private void ExecuteFileTransfer() { List<string> srcFileNameList = new List<string>(); var needSyncTransferFilesDatas = GetNeedSyncTransferFilesDatas(srcFileNameList, destFileApiUrlList.Count).Result; WriteLog(string.Format("從源服務器目錄Http Url:{0},獲取到{1}個需要同步的文件.", srcFileApiUrl, srcFileNameList.Count)); if (needSyncTransferFilesDatas == null || srcFileNameList.Count <= 0) return; ShowFileInfoLogs("需要同步的文件列表如下:", srcFileNameList); var fileTransferResultBag = new ConcurrentBag<Dictionary<string, Dictionary<string, string>>>(); Parallel.ForEach(destFileApiUrlList, (destFileApiUrl) => { MultipartContent needSyncTransferFilesData = null; if (needSyncTransferFilesDatas.TryTake(out needSyncTransferFilesData)) { var savedResult = new Dictionary<string, Dictionary<string, string>>(); try { savedResult = SyncTransferFiles(destFileApiUrl.Url, needSyncTransferFilesData).Result; } catch (Exception ex) { while (ex.InnerException != null) { ex = ex.InnerException; } savedResult[success] = new Dictionary<string, string>(); savedResult[failure] = new Dictionary<string, string>(); savedResult[failure][destFileApiUrl.Url] = string.Format("[{0:yyyy-MM-dd HH:mm:ss}] - X 請求:{1} 響應失敗,原因:{3}", DateTime.Now, destFileApiUrl, ex.Message); } fileTransferResultBag.Add(savedResult); ShowSyncTransferFileLogs(savedResult); } }); #region 同步循環 //foreach (var destFileApiUrl in destFileApiUrlList) //{ // MultipartContent needSyncTransferFilesData = null; // if (needSyncTransferFilesDatas.TryTake(out needSyncTransferFilesData)) // { // var savedResult = new Dictionary<string, Dictionary<string, string>>(); // try // { // savedResult = SyncTransferFiles(destFileApiUrl.Url, needSyncTransferFilesData).Result; // } // catch (Exception ex) // { // while (ex.InnerException != null) // { // ex = ex.InnerException; // } // savedResult[success] = new Dictionary<string, string>(); // savedResult[failure] = new Dictionary<string, string>(); // savedResult[failure][destFileApiUrl.Url] = string.Format("[{0:yyyy-MM-dd HH:mm:ss}] - X 請求:{1} 響應失敗,原因:{2}", DateTime.Now, destFileApiUrl, ex.Message); // } // fileTransferResultBag.Add(savedResult); // ShowSyncTransferFileLogs(savedResult); // } //} #endregion List<string> needRemoveFileNameList = GetNeedRemoveFileNameList(srcFileNameList, fileTransferResultBag.Select(b => b[success])); RemoveSourceFiles(needRemoveFileNameList); WriteSyncTransferFileLog(GetSyncTransferFileLogList(fileTransferResultBag)); ShowFileInfoLogs("以下文件已成功同步保存到預設的所有目的服務器目錄Http Url中,且已移除在源服務器目錄Http Url中的相同文件:", needRemoveFileNameList); } private void ShowFileInfoLogs(string logTitle, List<string> fileList) { WriteLog(logTitle); foreach (string file in fileList) { WriteLog(file); } WriteLog("-".PadRight(30, '-')); } private void ShowSyncTransferFileLogs(Dictionary<string, Dictionary<string, string>> savedResult) { foreach (var kv in savedResult) { bool isError = (kv.Key == failure); foreach (var kv2 in kv.Value) { WriteLog(kv2.Value, isError); } } } private void ReviseFileNames(List<string> fileNameList) { var regex = new System.Text.RegularExpressions.Regex("^\"+(?<name>.*)\"+$"); for (int i = 0; i < fileNameList.Count; i++) { string fileName = fileNameList[i]; var matchResult = regex.Match(fileName); if (matchResult != null && matchResult.Length > 0) { fileNameList[i] = matchResult.Groups["name"].Value; } } } private string ReviseFileName(string fileName) { var regex = new System.Text.RegularExpressions.Regex("^\"+(?<name>.*)\"+$"); var matchResult = regex.Match(fileName); if (matchResult != null && matchResult.Length > 0) { return matchResult.Groups["name"].Value; } return fileName; } private List<string> GetSyncTransferFileLogList(IEnumerable<Dictionary<string, Dictionary<string, string>>> savedResult) { List<string> syncTransferFileLogList = new List<string>(); foreach (var dic in savedResult) { foreach (var kv in dic) { syncTransferFileLogList.AddRange(kv.Value.Values); } } return syncTransferFileLogList; } private List<string> GetNeedRemoveFileNameList(List<string> srcFileNameList, IEnumerable<Dictionary<string, string>> successResult) { List<string> successFileNameList = new List<string>(); int needTransferFileCount = destFileApiUrlList.Count; foreach (var dic in successResult) { successFileNameList.AddRange(dic.Keys); } var successFileNames = successFileNameList.GroupBy(f => f).Where(gp => gp.Count() >= needTransferFileCount).Select(gp => gp.Key); return srcFileNameList.Where(f => successFileNames.Contains(f, StringComparer.OrdinalIgnoreCase)).ToList(); } private void WriteSyncTransferFileLog(IEnumerable<string> savedResult) { try { string apiUrl = srcFileApiUrl + (srcFileApiUrl.EndsWith("/") ? "" : "/") + "/Log" + (srcFileApiUrl.Contains('?') ? "&rid=" : "?rid=") + Guid.NewGuid().ToString("N"); HttpClient client = new HttpClient(); client.DefaultRequestHeaders.Add("token", GetToken()); var result = client.PostAsJsonAsync(apiUrl, savedResult).Result; result.EnsureSuccessStatusCode(); } catch (Exception ex) { WriteLog("WriteSyncTransferFileLog錯誤:" + ex.Message, true); } } private void RemoveSourceFiles(IEnumerable<string> needRemoveFileNames) { try { string apiUrl = srcFileApiUrl + (srcFileApiUrl.Contains('?') ? "&rid=" : "?rid=") + Guid.NewGuid().ToString("N"); using (HttpClient client = new HttpClient()) { client.DefaultRequestHeaders.Add("token", GetToken()); var result = client.PutAsJsonAsync(apiUrl, needRemoveFileNames).Result; result.EnsureSuccessStatusCode(); } } catch (Exception ex) { WriteLog("RemoveSourceFiles錯誤:" + ex.Message, true); } } private async Task<Dictionary<string, Dictionary<string, string>>> SyncTransferFiles(string destFileApiUrl, HttpContent filesData) { string apiUrl = destFileApiUrl + (destFileApiUrl.Contains('?') ? "&rid=" : "?rid=") + Guid.NewGuid().ToString("N"); using (HttpClient client = new HttpClient()) { client.DefaultRequestHeaders.Add("token", GetToken()); var result = await client.PostAsync(apiUrl, filesData); result.EnsureSuccessStatusCode(); var savedResult = await result.Content.ReadAsAsync<Dictionary<string, Dictionary<string, string>>>(); return savedResult; } } private async Task<MultipartContent> GetNeedSyncTransferFilesData(List<string> srcFileNameList) { string apiUrl = srcFileApiUrl + (srcFileApiUrl.Contains('?') ? "&rid=" : "?rid=") + Guid.NewGuid().ToString("N"); var filesCont = new MultipartContent(); using (HttpClient client = new HttpClient()) { client.DefaultRequestHeaders.Add("token", GetToken()); var result = await client.GetAsync(apiUrl); result.EnsureSuccessStatusCode(); var provider = await result.Content.ReadAsMultipartAsync(); foreach (var item in provider.Contents) { string fileName = item.Headers.ContentDisposition.FileName; if (!string.IsNullOrEmpty(fileName)) { filesCont.Add(item); srcFileNameList.Add(ReviseFileName(fileName)); } } } return filesCont; } private async Task<ConcurrentBag<MultipartContent>> GetNeedSyncTransferFilesDatas(List<string> srcFileNameList, int destFileServerCount) { var multipartContents = new ConcurrentBag<MultipartContent>(); string apiUrl = srcFileApiUrl + (srcFileApiUrl.Contains('?') ? "&rid=" : "?rid=") + Guid.NewGuid().ToString("N"); using (HttpClient client = new HttpClient()) { client.DefaultRequestHeaders.Add("token", GetToken()); var result = await client.GetAsync(apiUrl); result.EnsureSuccessStatusCode(); for (int i = 0; i < destFileServerCount; i++) { multipartContents.Add(new MultipartContent()); } var provider = await result.Content.ReadAsMultipartAsync(); foreach (var item in provider.Contents) { string fileName = item.Headers.ContentDisposition.FileName; if (!string.IsNullOrEmpty(fileName)) { var bytes = await item.ReadAsByteArrayAsync(); foreach (var cont in multipartContents) { cont.Add(CreateByteArrayContent(bytes, fileName, item.Headers.ContentType.MediaType)); } srcFileNameList.Add(ReviseFileName(fileName)); } } } return multipartContents; } private HttpContent CreateByteArrayContent(byte[] fileBytes, string fileName, string mediaTypeValue) { var fileContent = new ByteArrayContent(fileBytes); fileContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data") { Name = "files", FileName = fileName }; fileContent.Headers.ContentType = new MediaTypeHeaderValue(mediaTypeValue); fileContent.Headers.ContentLength = fileBytes.LongLength; return fileContent; } private string GetToken() { string timeStamp = GetTimeStamp(); string key = string.Join(string.Empty, "KyFileSyncTransfer.Api".OrderBy(c => c)); return EncryptUtility.Encrypt(string.Format("{0}-{1}", key, timeStamp)); } /// <summary> /// 獲取時間戳 /// </summary> /// <returns></returns> private string GetTimeStamp() { TimeSpan ts = DateTime.UtcNow - new DateTime(1970, 1, 1, 0, 0, 0, 0); return Convert.ToInt64(ts.TotalSeconds).ToString(); } private void WriteLog(string msg, bool isError = false) { if (this.IsHandleCreated && this.InvokeRequired) { this.Invoke(new Action<string, bool>(WriteLog), msg, isError); } else { if (isError && !msg.Contains("Error:")) { msg = "Error:" + msg; } msg = string.Format("{0:yyyy-MM-dd HH:mm:ss} - {1}", DateTime.Now, msg); lstLog.Items.Add(msg); lstLog.SelectedIndex = lstLog.Items.Count - 1; } } private void SaveAppSettingsToConfig() { ConfigUtility.RemoveAppSettings(null); Dictionary<string, string> settings = new Dictionary<string, string>(); settings[RunInterval] = txtInterval.Text.Trim(); settings[SrcFileApiUrl] = txtSrcHttpUrl.Text.Trim(); settings[DestFileApiUrls] = string.Join(",", destFileApiUrlList.Select(u => u.ToString())); ConfigUtility.SetAppSettingValues(settings); } private void LoadAppSettingsFromConfig() { var settings = ConfigUtility.GetAppSettingValues(); foreach (var kv in settings) { if (kv.Key.Equals(RunInterval, StringComparison.OrdinalIgnoreCase)) { txtInterval.Text = settings[RunInterval]; } else if (kv.Key.Equals(SrcFileApiUrl, StringComparison.OrdinalIgnoreCase)) { txtSrcHttpUrl.Text = settings[SrcFileApiUrl]; } else if (kv.Key.Equals(DestFileApiUrls, StringComparison.OrdinalIgnoreCase) && !string.IsNullOrEmpty(kv.Value)) { var destFileApiUrls = settings[DestFileApiUrls].Split(new[] { ",", "" }, StringSplitOptions.RemoveEmptyEntries); destFileApiUrlList = destFileApiUrls.Select(u => new WebApiUrlInfo(u) { }).ToList(); } } } #endregion private void FrmMain_Load(object sender, EventArgs e) { this.Text = string.Format("{0} V{1}", this.Text, appVersion); this.notifyIconApp.Text = this.Text; destFileApiUrlList = new List<WebApiUrlInfo>(); LoadAppSettingsFromConfig(); dgvDestHttpUrls.AutoGenerateColumns = false; dgvDestHttpUrls.DataSource = new BindingList<WebApiUrlInfo>(destFileApiUrlList); appTimer = new System.Timers.Timer(); appTimer.Elapsed += appTimer_Elapsed; btnStop.Enabled = false; } private void appTimer_Elapsed(object sender, System.Timers.ElapsedEventArgs e) { if (Interlocked.Increment(ref syncFlag) == 1) //若不忙則執行(0+1=1表示不忙) { Stopwatch watch = new Stopwatch(); do { watch.Restart(); try { ExecuteFileTransfer(); } catch (Exception ex) { WriteLog("執行文件同步傳輸時發生錯誤:" + ex.Message, true); } watch.Stop(); } while (watch.ElapsedMilliseconds >= runInterval); //如果執行的時間超過同步頻率間隔,則直接再執行一次 Interlocked.Exchange(ref syncFlag, 0); //解除忙 } } private void lstLog_DrawItem(object sender, DrawItemEventArgs e) { if (e.Index < 0) return; e.DrawBackground(); string itemValue = lstLog.Items[e.Index].ToString(); if (itemValue.Contains("Error:"))//如果Error,則紅字顯示 { e.Graphics.DrawString(itemValue, e.Font, new SolidBrush(Color.Red), e.Bounds); } else { e.Graphics.DrawString(itemValue, e.Font, new SolidBrush(e.ForeColor), e.Bounds); } e.DrawFocusRectangle(); } private void notifyIconApp_MouseDoubleClick(object sender, MouseEventArgs e) { if (WindowState == FormWindowState.Minimized) { WindowState = thisFormWindowState; this.ShowInTaskbar = true; } } private void FrmMain_SizeChanged(object sender, EventArgs e) { //判斷是否選擇的是最小化按鈕 if (WindowState == FormWindowState.Minimized) { //隱藏任務欄區圖標 this.ShowInTaskbar = false; //圖標顯示在托盤區 this.notifyIconApp.Visible = true; } else { thisFormWindowState = WindowState; } } private void FrmMain_FormClosing(object sender, FormClosingEventArgs e) { if (MessageBox.Show("您確定要退出程序嗎?", "退出確認", MessageBoxButtons.OKCancel, MessageBoxIcon.Question) == DialogResult.Cancel) { e.Cancel = true; return; } this.notifyIconApp.Visible = false; SaveAppSettingsToConfig(); } private void btnStart_Click(object sender, EventArgs e) { try { srcFileApiUrl = txtSrcHttpUrl.Text.Trim(); if (string.IsNullOrEmpty(srcFileApiUrl)) { throw new Exception("源服務器目錄Http Url不能為空!"); } if (destFileApiUrlList.Count <= 0) { throw new Exception("目的服務器目錄Http Url列表不能為空!"); } if (!int.TryParse(txtInterval.Text, out runInterval) || runInterval < 10) { throw new Exception("時間間隔不正確,必需是整數且>=10!"); } runInterval = runInterval * 1000; appTimer.Interval = runInterval; appTimer.Start(); txtSrcHttpUrl.Enabled = false; dgvDestHttpUrls.Enabled = false; txtInterval.Enabled = false; btnStart.Enabled = false; btnStop.Enabled = true; } catch (Exception ex) { MessageBox.Show("發生錯誤:" + ex.Message, "錯誤提示"); } } private void btnStop_Click(object sender, EventArgs e) { appTimer.Stop(); Interlocked.Exchange(ref syncFlag, 0); txtSrcHttpUrl.Enabled = true; dgvDestHttpUrls.Enabled = true; txtInterval.Enabled = true; btnStart.Enabled = true; btnStop.Enabled = false; } } }
設計器生成的代碼:

namespace KyFileSyncTransfer { partial class FrmMain { /// <summary> /// 必需的設計器變量。 /// </summary> private System.ComponentModel.IContainer components = null; /// <summary> /// 清理所有正在使用的資源。 /// </summary> /// <param name="disposing">如果應釋放托管資源,為 true;否則為 false。</param> protected override void Dispose(bool disposing) { if (disposing && (components != null)) { components.Dispose(); } base.Dispose(disposing); } #region Windows 窗體設計器生成的代碼 /// <summary> /// 設計器支持所需的方法 - 不要 /// 使用代碼編輯器修改此方法的內容。 /// </summary> private void InitializeComponent() { this.components = new System.ComponentModel.Container(); System.ComponentModel.ComponentResourceManager resources = new System.ComponentModel.ComponentResourceManager(typeof(FrmMain)); this.groupBox1 = new System.Windows.Forms.GroupBox(); this.label1 = new System.Windows.Forms.Label(); this.txtSrcHttpUrl = new System.Windows.Forms.TextBox(); this.groupBox2 = new System.Windows.Forms.GroupBox(); this.dgvDestHttpUrls = new System.Windows.Forms.DataGridView(); this.DestHttpUrl = new System.Windows.Forms.DataGridViewTextBoxColumn(); this.notifyIconApp = new System.Windows.Forms.NotifyIcon(this.components); this.tabControl1 = new System.Windows.Forms.TabControl(); this.tabPage1 = new System.Windows.Forms.TabPage(); this.panel1 = new System.Windows.Forms.Panel(); this.btnStop = new System.Windows.Forms.Button(); this.btnStart = new System.Windows.Forms.Button(); this.groupBox3 = new System.Windows.Forms.GroupBox(); this.label4 = new System.Windows.Forms.Label(); this.label3 = new System.Windows.Forms.Label(); this.txtInterval = new System.Windows.Forms.TextBox(); this.tabPage2 = new System.Windows.Forms.TabPage(); this.groupBox4 = new System.Windows.Forms.GroupBox(); this.lstLog = new System.Windows.Forms.ListBox(); this.groupBox1.SuspendLayout(); this.groupBox2.SuspendLayout(); ((System.ComponentModel.ISupportInitialize)(this.dgvDestHttpUrls)).BeginInit(); this.tabControl1.SuspendLayout(); this.tabPage1.SuspendLayout(); this.panel1.SuspendLayout(); this.groupBox3.SuspendLayout(); this.tabPage2.SuspendLayout(); this.groupBox4.SuspendLayout(); this.SuspendLayout(); // // groupBox1 // this.groupBox1.Anchor = ((System.Windows.Forms.AnchorStyles)(((System.Windows.Forms.AnchorStyles.Top | System.Windows.Forms.AnchorStyles.Left) | System.Windows.Forms.AnchorStyles.Right))); this.groupBox1.Controls.Add(this.label1); this.groupBox1.Controls.Add(this.txtSrcHttpUrl); this.groupBox1.Location = new System.Drawing.Point(6, 19); this.groupBox1.Name = "groupBox1"; this.groupBox1.Size = new System.Drawing.Size(761, 82); this.groupBox1.TabIndex = 0; this.groupBox1.TabStop = false; this.groupBox1.Text = "源服務器配置:"; // // label1 // this.label1.AutoSize = true; this.label1.Location = new System.Drawing.Point(22, 38); this.label1.Name = "label1"; this.label1.Size = new System.Drawing.Size(131, 12); this.label1.TabIndex = 1; this.label1.Text = "源服務器目錄Http Url:"; // // txtSrcHttpUrl // this.txtSrcHttpUrl.Anchor = ((System.Windows.Forms.AnchorStyles)(((System.Windows.Forms.AnchorStyles.Top | System.Windows.Forms.AnchorStyles.Left) | System.Windows.Forms.AnchorStyles.Right))); this.txtSrcHttpUrl.Location = new System.Drawing.Point(159, 35); this.txtSrcHttpUrl.Name = "txtSrcHttpUrl"; this.txtSrcHttpUrl.Size = new System.Drawing.Size(582, 21); this.txtSrcHttpUrl.TabIndex = 0; // // groupBox2 // this.groupBox2.Anchor = ((System.Windows.Forms.AnchorStyles)((((System.Windows.Forms.AnchorStyles.Top | System.Windows.Forms.AnchorStyles.Bottom) | System.Windows.Forms.AnchorStyles.Left) | System.Windows.Forms.AnchorStyles.Right))); this.groupBox2.Controls.Add(this.dgvDestHttpUrls); this.groupBox2.Location = new System.Drawing.Point(6, 107); this.groupBox2.Name = "groupBox2"; this.groupBox2.Size = new System.Drawing.Size(764, 283); this.groupBox2.TabIndex = 1; this.groupBox2.TabStop = false; this.groupBox2.Text = "目的服務器配置:"; // // dgvDestHttpUrls // this.dgvDestHttpUrls.ColumnHeadersHeightSizeMode = System.Windows.Forms.DataGridViewColumnHeadersHeightSizeMode.AutoSize; this.dgvDestHttpUrls.Columns.AddRange(new System.Windows.Forms.DataGridViewColumn[] { this.DestHttpUrl}); this.dgvDestHttpUrls.Dock = System.Windows.Forms.DockStyle.Fill; this.dgvDestHttpUrls.Location = new System.Drawing.Point(3, 17); this.dgvDestHttpUrls.Name = "dgvDestHttpUrls"; this.dgvDestHttpUrls.RowTemplate.Height = 23; this.dgvDestHttpUrls.Size = new System.Drawing.Size(758, 263); this.dgvDestHttpUrls.TabIndex = 0; // // DestHttpUrl // this.DestHttpUrl.AutoSizeMode = System.Windows.Forms.DataGridViewAutoSizeColumnMode.Fill; this.DestHttpUrl.DataPropertyName = "Url"; this.DestHttpUrl.HeaderText = "目的服務器目錄Http Url"; this.DestHttpUrl.Name = "DestHttpUrl"; // // notifyIconApp // this.notifyIconApp.Icon = ((System.Drawing.Icon)(resources.GetObject("notifyIconApp.Icon"))); this.notifyIconApp.Text = "notifyIcon1"; this.notifyIconApp.Visible = true; this.notifyIconApp.MouseDoubleClick += new System.Windows.Forms.MouseEventHandler(this.notifyIconApp_MouseDoubleClick); // // tabControl1 // this.tabControl1.Controls.Add(this.tabPage1); this.tabControl1.Controls.Add(this.tabPage2); this.tabControl1.Dock = System.Windows.Forms.DockStyle.Fill; this.tabControl1.Location = new System.Drawing.Point(0, 0); this.tabControl1.Name = "tabControl1"; this.tabControl1.SelectedIndex = 0; this.tabControl1.Size = new System.Drawing.Size(778, 554); this.tabControl1.TabIndex = 2; // // tabPage1 // this.tabPage1.Controls.Add(this.groupBox1); this.tabPage1.Controls.Add(this.panel1); this.tabPage1.Controls.Add(this.groupBox2); this.tabPage1.Location = new System.Drawing.Point(4, 22); this.tabPage1.Name = "tabPage1"; this.tabPage1.Padding = new System.Windows.Forms.Padding(3); this.tabPage1.Size = new System.Drawing.Size(770, 528); this.tabPage1.TabIndex = 0; this.tabPage1.Text = "控制台"; this.tabPage1.UseVisualStyleBackColor = true; // // panel1 // this.panel1.Anchor = ((System.Windows.Forms.AnchorStyles)(((System.Windows.Forms.AnchorStyles.Bottom | System.Windows.Forms.AnchorStyles.Left) | System.Windows.Forms.AnchorStyles.Right))); this.panel1.Controls.Add(this.btnStop); this.panel1.Controls.Add(this.btnStart); this.panel1.Controls.Add(this.groupBox3); this.panel1.Location = new System.Drawing.Point(3, 396); this.panel1.Name = "panel1"; this.panel1.Size = new System.Drawing.Size(764, 129); this.panel1.TabIndex = 2; // // btnStop // this.btnStop.Anchor = System.Windows.Forms.AnchorStyles.Top; this.btnStop.BackColor = System.Drawing.Color.FromArgb(((int)(((byte)(255)))), ((int)(((byte)(192)))), ((int)(((byte)(192))))); this.btnStop.Location = new System.Drawing.Point(395, 73); this.btnStop.Name = "btnStop"; this.btnStop.Size = new System.Drawing.Size(90, 40); this.btnStop.TabIndex = 1; this.btnStop.Text = "停 止"; this.btnStop.UseVisualStyleBackColor = false; this.btnStop.Click += new System.EventHandler(this.btnStop_Click); // // btnStart // this.btnStart.Anchor = System.Windows.Forms.AnchorStyles.Top; this.btnStart.BackColor = System.Drawing.Color.FromArgb(((int)(((byte)(192)))), ((int)(((byte)(255)))), ((int)(((byte)(192))))); this.btnStart.Location = new System.Drawing.Point(280, 73); this.btnStart.Name = "btnStart"; this.btnStart.Size = new System.Drawing.Size(90, 40); this.btnStart.TabIndex = 1; this.btnStart.Text = "開 啟"; this.btnStart.UseVisualStyleBackColor = false; this.btnStart.Click += new System.EventHandler(this.btnStart_Click); // // groupBox3 // this.groupBox3.Controls.Add(this.label4); this.groupBox3.Controls.Add(this.label3); this.groupBox3.Controls.Add(this.txtInterval); this.groupBox3.Location = new System.Drawing.Point(3, 13); this.groupBox3.Name = "groupBox3"; this.groupBox3.Size = new System.Drawing.Size(680, 54); this.groupBox3.TabIndex = 0; this.groupBox3.TabStop = false; this.groupBox3.Text = "同步頻率:"; // // label4 // this.label4.AutoSize = true; this.label4.Location = new System.Drawing.Point(214, 23); this.label4.Name = "label4"; this.label4.Size = new System.Drawing.Size(17, 12); this.label4.TabIndex = 1; this.label4.Text = "秒"; // // label3 // this.label3.AutoSize = true; this.label3.Location = new System.Drawing.Point(4, 23); this.label3.Name = "label3"; this.label3.Size = new System.Drawing.Size(59, 12); this.label3.TabIndex = 1; this.label3.Text = "時間間隔:"; // // txtInterval // this.txtInterval.Location = new System.Drawing.Point(69, 20); this.txtInterval.Name = "txtInterval"; this.txtInterval.Size = new System.Drawing.Size(139, 21); this.txtInterval.TabIndex = 0; this.txtInterval.Text = "60"; // // tabPage2 // this.tabPage2.Controls.Add(this.groupBox4); this.tabPage2.Location = new System.Drawing.Point(4, 22); this.tabPage2.Name = "tabPage2"; this.tabPage2.Padding = new System.Windows.Forms.Padding(3); this.tabPage2.Size = new System.Drawing.Size(770, 528); this.tabPage2.TabIndex = 1; this.tabPage2.Text = "運行日志"; this.tabPage2.UseVisualStyleBackColor = true; // // groupBox4 // this.groupBox4.Controls.Add(this.lstLog); this.groupBox4.Dock = System.Windows.Forms.DockStyle.Fill; this.groupBox4.Location = new System.Drawing.Point(3, 3); this.groupBox4.Name = "groupBox4"; this.groupBox4.Size = new System.Drawing.Size(764, 522); this.groupBox4.TabIndex = 3; this.groupBox4.TabStop = false; // // lstLog // this.lstLog.Dock = System.Windows.Forms.DockStyle.Fill; this.lstLog.DrawMode = System.Windows.Forms.DrawMode.OwnerDrawFixed; this.lstLog.FormattingEnabled = true; this.lstLog.ItemHeight = 12; this.lstLog.Location = new System.Drawing.Point(3, 17); this.lstLog.Name = "lstLog"; this.lstLog.Size = new System.Drawing.Size(758, 502); this.lstLog.TabIndex = 0; this.lstLog.DrawItem += new System.Windows.Forms.DrawItemEventHandler(this.lstLog_DrawItem); // // FrmMain // this.AutoScaleDimensions = new System.Drawing.SizeF(6F, 12F); this.AutoScaleMode = System.Windows.Forms.AutoScaleMode.Font; this.ClientSize = new System.Drawing.Size(778, 554); this.Controls.Add(this.tabControl1); this.Icon = ((System.Drawing.Icon)(resources.GetObject("$this.Icon"))); this.MinimumSize = new System.Drawing.Size(600, 500); this.Name = "FrmMain"; this.StartPosition = System.Windows.Forms.FormStartPosition.CenterScreen; this.Text = "開源文件同步傳輸工具"; this.FormClosing += new System.Windows.Forms.FormClosingEventHandler(this.FrmMain_FormClosing); this.Load += new System.EventHandler(this.FrmMain_Load); this.SizeChanged += new System.EventHandler(this.FrmMain_SizeChanged); this.groupBox1.ResumeLayout(false); this.groupBox1.PerformLayout(); this.groupBox2.ResumeLayout(false); ((System.ComponentModel.ISupportInitialize)(this.dgvDestHttpUrls)).EndInit(); this.tabControl1.ResumeLayout(false); this.tabPage1.ResumeLayout(false); this.panel1.ResumeLayout(false); this.groupBox3.ResumeLayout(false); this.groupBox3.PerformLayout(); this.tabPage2.ResumeLayout(false); this.groupBox4.ResumeLayout(false); this.ResumeLayout(false); } #endregion private System.Windows.Forms.GroupBox groupBox1; private System.Windows.Forms.Label label1; private System.Windows.Forms.TextBox txtSrcHttpUrl; private System.Windows.Forms.GroupBox groupBox2; private System.Windows.Forms.DataGridView dgvDestHttpUrls; private System.Windows.Forms.NotifyIcon notifyIconApp; private System.Windows.Forms.TabControl tabControl1; private System.Windows.Forms.TabPage tabPage1; private System.Windows.Forms.Panel panel1; private System.Windows.Forms.Button btnStop; private System.Windows.Forms.Button btnStart; private System.Windows.Forms.GroupBox groupBox3; private System.Windows.Forms.Label label4; private System.Windows.Forms.Label label3; private System.Windows.Forms.TextBox txtInterval; private System.Windows.Forms.TabPage tabPage2; private System.Windows.Forms.GroupBox groupBox4; private System.Windows.Forms.ListBox lstLog; private System.Windows.Forms.DataGridViewTextBoxColumn DestHttpUrl; } }
特別說明:若客戶端需要直接使用ReadAsAsync<T>將響應的結果反序列化成對象,則Newtonsoft.Json應采用6.0.0.0版本,這個是WebApi框架默認使用的版本,若單獨通過NuGet引用可能是最新的9.0版本,會報錯,這點注意;
另外文中為何采用Winform作為客戶端,目的是可以實時查看運行結果,若沒有這個需求,那么就可以采用Windows服務程序即可,看個人意願了。
還有人可能認為,批量上傳與下載采取打包上傳與打包下載更快,這個我認為看各人的實際需求。
看到大家都想要源代碼,其實源代碼上面都有復制到項目中即可,但為了便於大家測試,故我也將GIT倉庫地址告訴給大家:git@git.oschina.net:zuowj/KyFileSyncTransfer.git,大家如有需要可以下載或查看。