.NET Core(C#) 使用IHttpClientFactory实现爬虫执行GET和POST请求

1､.NET Core项目中注册HttpClientFactory

1) ASP .NET Core项目

在Startup.cs中ConfigureService方法中注册HttpClient，代码如下，

services.AddHttpClient();

2) .NET Core项目

.NET Core项目中需要安装引用Microsoft.Extensions.Http和Microsoft.Extensions.Hosting，注册代码如下，

static void Main(string[] args) { var builder = Host.CreateDefaultBuilder(args); builder.ConfigureServices(x => { x.AddHttpClient(); }); Console.ReadKey(); }

2､配置HTTP请求头Header

1) 注册HttpClient时配置

services.AddHttpClient();
services.AddHttpClient("cjavapy", c =>
{
    c.BaseAddress = new Uri("https://api.cjavapy.com/"); c.DefaultRequestHeaders.Add("content-type","application/json"); c.DefaultRequestHeaders.Add("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat"); c.DefaultRequestHeaders.Add("Referer",""); });

或者

   static void Main(string[] args) { var builder = Host.CreateDefaultBuilder(args); builder.ConfigureServices(x => { x..AddHttpClient("cjavapy", c => { c.BaseAddress = new Uri("https://api.cjavapy.com/"); c.DefaultRequestHeaders.Add("content-type","application/json"); c.DefaultRequestHeaders.Add("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat"); c.DefaultRequestHeaders.Add("Referer",""); }); }); Console.ReadKey(); }

2) 执行请求之前配置

var httpClient = context.ServiceProvider.GetRequiredService<IHttpClientFactory>().CreateClient();
httpClient.BaseAddress = new Uri("https://api.cjavapy.com/"); httpClient.DefaultRequestHeaders.Add("content-type","application/json"); httpClient.DefaultRequestHeaders.Add("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat"); httpClient.DefaultRequestHeaders.Add("Referer","");

3､使用HttpClientFactory(IHttpClientFactory)执行GET和POST请求

命名空间：

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Reflection;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

1) GET请求获取网页代码

static void Main(string[] args) { var builder = Host.CreateDefaultBuilder(args); builder.ConfigureServices(x => { x.AddHttpClient(); }); using (var scope = builder.Build().Services.CreateScope()) { var httpClient = scope.ServiceProvider.GetRequiredService<IHttpClientFactory>().CreateClient(); httpClient.BaseAddress = new Uri("https://www.baidu.com"); httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat"); httpClient.DefaultRequestHeaders.Add("Referer", "https://www.baidu.com"); Console.WriteLine(httpClient.GetAsync("/").Result.Content.ReadAsStringAsync().Result); } Console.ReadKey(); }

2) POST请求提交数据

static void Main(string[] args) { var builder = Host.CreateDefaultBuilder(args); builder.ConfigureServices(x => { x.AddHttpClient(); }); using (var scope = builder.Build().Services.CreateScope()) { var httpClient = scope.ServiceProvider.GetRequiredService<IHttpClientFactory>().CreateClient(); httpClient.BaseAddress = new Uri("apiUrl"); httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36 MicroMessenger/7.0.9.501 NetType/WIFI MiniProgramEnv/Windows WindowsWechat"); //httpClient.DefaultRequestHeaders.Add("Referer", "https://www.baidu.com"); var result=""; try { //2 minute timeout on wait for response httpClient.Timeout = new TimeSpan(0, 2, 0); //Create an HttpRequestMessage object and pass it into SendAsync() HttpRequestMessage message = new HttpRequestMessage(); message.Headers.Add("Accept", "application/json"); message.Content = new StringContent("{\"user\":\"11\"}", System.Text.Encoding.UTF8, "application/json"); message.Method = HttpMethod.Post; message.RequestUri = new Uri(httpClient.BaseAddress.ToString() + "someapiendpoint"); HttpResponseMessage response = httpClient.SendAsync(message).Result; result = response.Content.ReadAsStringAsync().Result; //deserialize the result into proper object type } catch (Exception ex) { //Log exception } } Console.ReadKey(); }

.NET Core(C#) 使用IHttpClientFactory实现爬虫执行GET和POST请求

推荐文档

免责声明！