doubleclick cookie、動態腳本、用戶畫像、用戶行為分析和海量數據存取 推薦詞 京東 電商 信息上傳 黑洞 https://blackhole.m.jd.com/getinfo

doubleclick cookie 

https://mp.weixin.qq.com/s/vZUj-Z9FGSSWXOodGqbYkA

揭密Google的網絡廣告技術：基於互聯網大數據視角

原創： 曾劍平 互聯網大數據處理技術與應用 2018-04-11

 相信每個人在上網時都被各種網絡廣告所困擾，不斷地消耗着我們的流量。如果稍微細心觀察，或許會發現不同網站推送過來的廣告也比較適合自己的偏好，看來其中的技術手段並非簡單之事。涉及到互聯網大數據技術包括：cookie、動態腳本、用戶畫像、用戶行為分析和海量數據存取等。

 

      假如你在京東上點擊筆記本電腦，過幾天以后當你瀏覽一個從未訪問過的網站時，你很可能發現頁面上竟然有筆記本的廣告。

圖1 

 

       作為一個互聯網大數據技術研究者，本能反應當然是看看頁面的源代碼，確實可以找到相應的腳本，其中的“-ad-”大概表明了這里嵌入了廣告。

 

圖2

 

        但由於是動態腳本，無法看出廣告具體在哪個網站上。為此，可以通過瀏覽器的設置功能，進入開發者模式（Source），找到廣告條對應的腳本結構。

圖3 

 

        然后查看這段動態腳本執行完成后對應的URL，從下圖可以看出這個廣告URL指向了googleads.g.doubleclick.net，從域名看就是google的廣告。

圖4

 

        沒錯，doubleclick是一家互聯網廣告公司，在2008年被Google收購。它提供了多種廣告管理和廣告投放解決方案，幫助企業購買、制作或銷售在線廣告，允許用戶對網絡廣告活動進行集中策划、執行、監控和追蹤。由此我們可以畫出Google的網絡廣告技術平台架構圖。

 

圖5

 

整個流程按圖中標注的序號1-5。

1 需要做廣告的客戶到doubleclick上進行注冊、登記；

2 加入廣告聯盟的網站從doubleclick獲得嵌入廣告的動態腳本，即類似於圖2所示。並將這些代碼嵌入到頁面中；

3 互聯網用戶大眾通過瀏覽器訪問頁面，動態腳本在用戶瀏覽器上執行，獲得指向doubleclick的URL；

4 連接doubleclick時，doubleclick生成用戶的唯一標識，並寫入到本地cookie文件；

5 以后我們每次訪問含有廣告腳本的頁面時，自動讀取doubleclick的cookie，並由doubleclick抽取合適的廣告。這樣每個人的唯一身份就記錄到它的數據庫中了。而這個步驟，顯然是基於我們點擊廣告、瀏覽頁面的行為數據，是一個海量數據。精准的廣告推送需要進行大數據挖掘、用戶畫像。

 

       在這個流程中,cookie起到了很大作用，在每台電腦上幾乎都有doubleclick的cookie文件。對於win7下的IE，一般是在C:\Users\Administrator\AppData\Local\Microsoft\Windows\Temporary Internet Files中；Chrome瀏覽器可以Chrome設置->隱私設置->內容設置。找到后可以清除。

 

1. Request URL:
   https://blackhole.m.jd.com/getinfo
2. Request Method:
   POST
3. Status Code:
   200 OK
4. Remote Address:
   124.200.54.26:443
5. Referrer Policy:
   no-referrer-when-downgrade
1. Response Headers
1. Access-Control-Allow-Origin:
   *
2. Connection:
   keep-alive
3. Content-Length:
   95
4. Content-Type:
   text/plain
5. Date:
   Sun, 28 Apr 2019 02:17:39 GMT
6. Server:
   jfe
2. Request Headers
1. Provisional headers are shown
2. Content-Type:
   application/x-www-form-urlencoded
3. Origin:
   https://www.jd.com
4. Referer:
   https://www.jd.com/
5. User-Agent:
   Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36
3. Form Dataview sourceview URL encoded
1. body:
   {"appname":"jdwebm_hf","jdkey":"","whwswswws":"","businness":"pcHome","body":{"browser_info":"5a6de6eb4239d72a591cd732fcf557bc","client_time":1556417862694,"period":24,"shshshfpa":"93048091-c96c-4aff-40ec-0c8bb237d983-1556417862","whwswswws":"","cookie_pin":"","jdu":"1556417860906768591982","mba_muid":"","visitkey":"","msdk_version":"2.3.4","wid":"","language":"en-US","color_depth":24,"pixel_ratio":1,"resolution":"1280;800","available_resolution":"1227;800","session_storage":1,"local_storage":1,"indexed_db":1,"open_database":1,"cpu_class":"unknown","navigator_platform":"Win32","regular_plugins":"Chrome PDF Plugin::Portable Document Format::application/x-google-chrome-pdf~pdf;Chrome PDF Viewer::::application/pdf~pdf;Native Client::::application/x-nacl~,application/x-pnacl~","adblock":false,"touch_support":0,"app_code_name":"Mozilla","app_name":"Netscape","app_version":"5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36","cookie_enabled":true,"regular_mimetypes":"::::;::Portable Document Format::;::Native Client Executable::;::Portable Native Client Executable::","online":"unknown","hardwareConcurrency":2,"product":"Gecko","productSub":"20030107","vendor":"Google Inc.","vendorSub":"unknown","devicePixelRatio":1,"updateInterval":"unknown","orientationType":"landscape-primary","doNotTrack":0,"canvas":"canvas winding:yes~canvas fp:231bea7a22d38c7771b1fd991affdfc6","webgl":"fp:cba0abf4b20cd68cd9fb42b1524d0708~extensions:ANGLE_instanced_arrays;EXT_blend_minmax;EXT_color_buffer_half_float;EXT_frag_depth;EXT_shader_texture_lod;EXT_texture_filter_anisotropic;WEBKIT_EXT_texture_filter_anisotropic;OES_element_index_uint;OES_standard_derivatives;OES_texture_float;OES_texture_half_float;OES_texture_half_float_linear;OES_vertex_array_object;WEBGL_color_buffer_float;WEBGL_compressed_texture_s3tc;WEBKIT_WEBGL_compressed_texture_s3tc;WEBGL_debug_renderer_info;WEBGL_debug_shaders;WEBGL_depth_texture;WEBKIT_WEBGL_depth_texture;WEBGL_lose_context;WEBKIT_WEBGL_lose_context~aliased line width range:[1, 1]~aliased point size range:[1, 256]~alpha bits:8~antialiasing:yes~blue bits:8~depth bits:24~green bits:8~max anisotropy:16~max combined texture image units:20~max cube map texture size:4096~max fragment uniform vectors:221~max render buffer size:4096~max texture image units:16~max texture size:4096~max varying vectors:9~max vertex attribs:16~max vertex texture image units:4~max vertex uniform vectors:253~max viewport dims:[4096, 4096]~red bits:8~renderer:WebKit WebGL~shading language version:WebGL GLSL ES 1.0 (OpenGL ES GLSL ES 1.0 Chromium)~stencil bits:0~vendor:WebKit~version:WebGL 1.0 (OpenGL ES 2.0 Chromium)~unmasked vendor:Google Inc.~unmasked renderer:ANGLE (Mobile Intel(R) 4 Series Express Chipset Family Direct3D9Ex vs_3_0 ps_3_0)~vertex high float:23(127,127)~vertex medium float:23(127,127)~vertex low float:23(127,127)~fragment high float:23(127,127)~fragment medium float:23(127,127)~fragment low float:23(127,127)~vertex high int:0(24,24)~vertex medium int:0(24,24)~vertex low int:0(24,24)~fragment high int:0(24,24)~fragment medium int:0(24,24)~fragment low int:0(24,24)","device_memory":8,"is_headless_browser":0}}

 

 在保存有uuid情況下

Request URL:https://floor.jd.com/user/hotwords/get?pin=&uuid=1550390246822668439123&callback=jsonpHotWords
Request Method:GET
Status Code:200 OK
Remote Address:211.144.24.170:443
Referrer Policy:no-referrer-when-downgrade
Response Headers
view source
Connection:close
Content-Encoding:gzip
Content-Type:text/html; charset=utf-8
Date:Sun, 28 Apr 2019 02:31:52 GMT
Server:jfe
Transfer-Encoding:chunked
Vary:Accept-Encoding
Request Headers
view source
Accept:*/*
Accept-Encoding:gzip, deflate, sdch, br
Accept-Language:zh-CN,zh;q=0.8
Connection:keep-alive
Cookie:shshshfpa=e27a5d69-e1c0-282d-ea23-9695b1e69510-1550390256; TrackID=1SQI5uj2G1r220dR6ifodPmI8KRO5dZs3OmsiX1SfPCYPCDefRrnEfXWxtXJXAoVZxMHISD56FXkht7-BTmb0iK9S9AT1-UppuX4Q7Pf0u1M; pinId=aR72BDnyHaxs0LHNbV6fLg; __jdv=122270672|direct|-|none|-|1556382041032; areaId=11; ipLoc-djd=11-799-0; PCSYCityID=1137; __jda=122270672.1550390246822668439123.1550390247.1556382041.1556417842.9; __jdb=122270672.3.1550390246822668439123|9.1556417842; __jdc=122270672; shshshfp=8e7ec1c7e67d1ed4944e451a3574168a; shshshsID=6c6284d43d4431f049e3a6f3152e5d03_3_1556418662038; shshshfpb=z8d2uPw45jLHaI6jyaglJIw%3D%3D; __jdu=1550390246822668439123
Host:floor.jd.com
Referer:https://www.jd.com/
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0
Query String Parameters
view source
view URL encoded
pin:
uuid:1550390246822668439123
callback:jsonpHotWords