一、數倉搭建 - DWS 層
1.1 業務術語
1）用戶
用戶以設備為判斷標准，在移動統計中，每個獨立設備認為是一個獨立用戶。Android
系統根據 IMEI 號，IOS 系統根據 OpenUDID 來標識一個獨立用戶，每部手機一個用戶
2）新增用戶
首次聯網使用應用的用戶。如果一個用戶首次打開某 APP，那這個用戶定義為新增用
戶；卸載再安裝的設備，不會被算作一次新增。新增用戶包括日新增用戶、周新增用戶、月
新增用戶
3）活躍用戶
打開應用的用戶即為活躍用戶，不考慮用戶的使用情況。每天一台設備打開多次會被計
為一個活躍用戶
4）周（月）活躍用戶
某個自然周（月）內啟動過應用的用戶，該周（月）內的多次啟動只記一個活躍用戶
5）月活躍率
月活躍用戶與截止到該月累計的用戶總和之間的比例
6）沉默用戶
用戶僅在安裝當天（次日）啟動一次，后續時間無再啟動行為。該指標可以反映新增用
戶質量和用戶與 APP 的匹配程度
7）版本分布
不同版本的周內各天新增用戶數，活躍用戶數和啟動次數。利於判斷 APP 各個版本之
間的優劣和用戶行為習慣
8）本周回流用戶
上周未啟動過應用，本周啟動了應用的用戶
9）連續 n 周活躍用戶
連續 n 周，每周至少啟動一次
10）忠誠用戶
連續活躍 5 周以上的用戶
11）連續活躍用戶
連續 2 周及以上活躍的用戶
12）近期流失用戶
連續 n（2<= n <= 4）周沒有啟動應用的用戶。（第 n+1 周沒有啟動過）
13）留存用戶
某段時間內的新增用戶，經過一段時間后，仍然使用應用的被認作是留存用戶；這部分
用戶占當時新增用戶的比例即是留存率
例如，5 月份新增用戶 200，這 200 人在 6 月份啟動過應用的有 100 人，7 月份啟動過應用的有 80 人，8 月份啟動過應用的有 50 人；則 5 月份新增用戶一個月后的留存率是 50%，二個月后的留存率是 40%，三個月后的留存率是 25%
14）用戶新鮮度
每天啟動應用的新老用戶比例，即新增用戶數占活躍用戶數的比例
15）單次使用時長
每次啟動使用的時間長度
16）日使用時長
累計一天內的使用時間長度
17）啟動次數計算標准
IOS 平台應用退到后台就算一次獨立的啟動；Android 平台我們規定，兩次啟動之間的間隔小於 30 秒，被計算一次啟動。用戶在使用過程中，若因收發短信或接電話等退出應用30 秒又再次返回應用中，那這兩次行為應該是延續而非獨立的，所以可以被算作一次使用行為，即一次啟動。業內大多使用 30 秒這個標准，但用戶還是可以自定義此時間間隔
1.2 系統函數
1.2.1 collect_set 函數
1）創建原數據表
drop table if exists stud; create table stud (name string, area string, course string, score int);
2）向原數據表中插入數據
insert into table stud values('zhang3','bj','math',88); insert into table stud values('li4','bj','math',99); insert into table stud values('wang5','sh','chinese',92); insert into table stud values('zhao6','sh','chinese',54); insert into table stud values('tian7','bj','chinese',91);
3）查詢表中數據
select * from stud; stud.name stud.area stud.course stud.score zhang3 bj math 88 li4 bj math 99 wang5 sh chinese 92 zhao6 sh chinese 54 tian7 bj chinese 91
4）把同一分組的不同行的數據聚合成一個集合
select course, collect_set(area), avg(score) from stud group by course; chinese ["sh","bj"] 79.0 math ["bj"] 93.5
5） 用下標可以取某一個
select course, collect_set(area)[0], avg(score) from stud group by course; chinese sh 79.0 math bj 93.5
1.2.2 nvl 函數
1）基本語法
NVL（表達式 1，表達式 2）
如果表達式 1 為空值，NVL 返回值為表達式 2 的值，否則返回表達式 1 的值。 該函數的目的是把一個空值（null）轉換成一個實際的值。其表達式的值可以是數字型、字符型和日期型。但是表達式 1 和表達式 2 的數據類型必須為同一個類型
1.2.3 日期處理函數
1）date_format 函數（根據格式整理日期）
hive (gmall)> select date_format('2020-03-10','yyyy-MM'); 2020-03
2）date_add 函數（加減日期）
hive (gmall)> select date_add('2020-03-10',-1); 2020-03-09 hive (gmall)> select date_add('2020-03-10',1); 2020-03-11
3）next_day 函數
（1）取當前天的下一個周一
hive (gmall)> select next_day('2020-03-12','MO'); 2020-03-16 說明：星期一到星期日的英文（Monday，Tuesday、Wednesday、Thursday、Friday、Saturday、Sunday）
（2）取當前周的周一
hive (gmall)> select date_add(next_day('2020-03-12','MO'),-7); 2020-03-11
4）last_day 函數（求當月最后一天日期）
hive (gmall)> select last_day('2020-03-10'); 2020-03-31
1.3 DWS 層（用戶行為）
1.3.1 每日設備行為
每日設備行為，主要按照 設備 id 統計
1）建表語句
drop table if exists dws_uv_detail_daycount; create external table dws_uv_detail_daycount ( `mid_id` string COMMENT '設備唯一標識', `user_id` string COMMENT '用戶標識', `version_code` string COMMENT '程序版本號', `version_name` string COMMENT '程序版本名', `lang` string COMMENT '系統語言', `source` string COMMENT '渠道號', `os` string COMMENT '安卓系統版本', `area` string COMMENT '區域', `model` string COMMENT '手機型號', `brand` string COMMENT '手機品牌', `sdk_version` string COMMENT 'sdkVersion', `gmail` string COMMENT 'gmail', `height_width` string COMMENT '屏幕寬高', `app_time` string COMMENT '客戶端日志產生時的時間', `network` string COMMENT '網絡模式', `lng` string COMMENT '經度', `lat` string COMMENT '緯度', `login_count` bigint COMMENT '活躍次數' ) partitioned by(dt string) stored as parquet location '/warehouse/gmall/dws/dws_uv_detail_daycount';
2）數據裝載
insert overwrite table dws_uv_detail_daycount partition(dt='2020-03-10') select mid_id, concat_ws('|', collect_set(user_id)) user_id, concat_ws('|', collect_set(version_code)) version_code, concat_ws('|', collect_set(version_name)) version_name, concat_ws('|', collect_set(lang))lang, concat_ws('|', collect_set(source)) source, concat_ws('|', collect_set(os)) os, concat_ws('|', collect_set(area)) area, concat_ws('|', collect_set(model)) model, concat_ws('|', collect_set(brand)) brand, concat_ws('|', collect_set(sdk_version)) sdk_version, concat_ws('|', collect_set(gmail)) gmail, concat_ws('|', collect_set(height_width)) height_width, concat_ws('|', collect_set(app_time)) app_time, concat_ws('|', collect_set(network)) network, concat_ws('|', collect_set(lng)) lng, concat_ws('|', collect_set(lat)) lat, count(*) login_count from dwd_start_log where dt='2020-03-10' group by mid_id;
3）查詢加載結果
select * from dws_uv_detail_daycount where dt='2020-03-10';
1.4 DWS 層（業務）
DWS 層的寬表字段，是站在不同維度的視角去看事實表，重點關注事實表的度量值
1.4.1 每日會員行為
1）建表語句
drop table if exists dws_user_action_daycount; create external table dws_user_action_daycount ( user_id string comment '用戶 id', login_count bigint comment '登錄次數', cart_count bigint comment '加入購物車次數', cart_amount double comment '加入購物車金額', order_count bigint comment '下單次數', order_amount decimal(16,2) comment '下單金額', payment_count bigint comment '支付次數', payment_amount decimal(16,2) comment '支付金額' ) COMMENT '每日用戶行為' PARTITIONED BY (`dt` string) stored as parquet location '/warehouse/gmall/dws/dws_user_action_daycount/' tblproperties ("parquet.compression"="lzo");
2）數據裝載
with tmp_login as ( select user_id, count(*) login_count from dwd_start_log where dt='2020-03-10' and user_id is not null group by user_id ), tmp_cart as ( select user_id, count(*) cart_count, sum(cart_price*sku_num) cart_amount from dwd_fact_cart_info where dt='2020-03-10' and user_id is not null and date_format(create_time,'yyyy-MM-dd')='2020-03-10' group by user_id ), tmp_order as ( select user_id, count(*) order_count, sum(final_total_amount) order_amount from dwd_fact_order_info where dt='2020-03-10' group by user_id ) , tmp_payment as ( select user_id, count(*) payment_count, sum(payment_amount) payment_amount from dwd_fact_payment_info where dt='2020-03-10' group by user_id ) insert overwrite table dws_user_action_daycount partition(dt='2020-03-10') select user_actions.user_id, sum(user_actions.login_count), sum(user_actions.cart_count), sum(user_actions.cart_amount), sum(user_actions.order_count), sum(user_actions.order_amount), sum(user_actions.payment_count), sum(user_actions.payment_amount) from ( select user_id, login_count, 0 cart_count, 0 cart_amount, 0 order_count, 0 order_amount, 0 payment_count, 0 payment_amount from tmp_login union all select user_id, 0 login_count, cart_count, cart_amount, 0 order_count, 0 order_amount, 0 payment_count, 0 payment_amount from tmp_cart union all select user_id, 0 login_count, 0 cart_count, 0 cart_amount, order_count, order_amount, 0 payment_count, 0 payment_amount from tmp_order union all select user_id, 0 login_count, 0 cart_count, 0 cart_amount, 0 order_count, 0 order_amount, payment_count, payment_amount from tmp_payment ) user_actions group by user_id;
3）查詢加載結果
select * from dws_user_action_daycount where dt=‘2020-03-10’;
1.4.2 每日商品行為
1）建表語句
drop table if exists dws_sku_action_daycount; create external table dws_sku_action_daycount ( sku_id string comment 'sku_id', order_count bigint comment '被下單次數', order_num bigint comment '被下單件數', order_amount decimal(16,2) comment '被下單金額', payment_count bigint comment '被支付次數', payment_num bigint comment '被支付件數', payment_amount decimal(16,2) comment '被支付金額', refund_count bigint comment '被退款次數', refund_num bigint comment '被退款件數', refund_amount decimal(16,2) comment '被退款金額', cart_count bigint comment '被加入購物車次數', cart_num bigint comment '被加入購物車件數', favor_count bigint comment '被收藏次數', appraise_good_count bigint comment '好評數', appraise_mid_count bigint comment '中評數', appraise_bad_count bigint comment '差評數', appraise_default_count bigint comment '默認評價數' ) COMMENT '每日商品行為' PARTITIONED BY (`dt` string) stored as parquet location '/warehouse/gmall/dws/dws_sku_action_daycount/' tblproperties ("parquet.compression"="lzo");
2）數據裝載
注意：如果是 23 點 59 下單，支付日期跨天。需要從訂單詳情里面取出支付時間是今天，訂單時間是昨天或者今天的訂單
with tmp_order as ( select sku_id, count(*) order_count, sum(sku_num) order_num, sum(total_amount) order_amount from dwd_fact_order_detail where dt='2020-03-10' group by sku_id ), tmp_payment as ( select sku_id, count(*) payment_count, sum(sku_num) payment_num, sum(total_amount) payment_amount from dwd_fact_order_detail where dt='2020-03-10' and order_id in ( select id from dwd_fact_order_info where (dt='2020-03-10' or dt=date_add('2020-03-10',-1)) and date_format(payment_time,'yyyy-MM-dd')='2020-03-10' ) group by sku_id ), tmp_refund as ( select sku_id, count(*) refund_count, sum(refund_num) refund_num, sum(refund_amount) refund_amount from dwd_fact_order_refund_info where dt='2020-03-10' group by sku_id ), tmp_cart as ( select sku_id, count(*) cart_count, sum(sku_num) cart_num from dwd_fact_cart_info where dt='2020-03-10' and date_format(create_time,'yyyy-MM-dd')='2020-03-10' group by sku_id ), tmp_favor as ( select sku_id, count(*) favor_count from dwd_fact_favor_info where dt='2020-03-10' and date_format(create_time,'yyyy-MM-dd')='2020-03-10' group by sku_id ), tmp_appraise as ( select sku_id, sum(if(appraise='1201',1,0)) appraise_good_count, sum(if(appraise='1202',1,0)) appraise_mid_count, sum(if(appraise='1203',1,0)) appraise_bad_count, sum(if(appraise='1204',1,0)) appraise_default_count from dwd_fact_comment_info where dt='2020-03-10' group by sku_id ) insert overwrite table dws_sku_action_daycount partition(dt='2020-03-10') select sku_id, sum(order_count), sum(order_num), sum(order_amount), sum(payment_count), sum(payment_num), sum(payment_amount), sum(refund_count), sum(refund_num), sum(refund_amount), sum(cart_count), sum(cart_num), sum(favor_count), sum(appraise_good_count), sum(appraise_mid_count), sum(appraise_bad_count), sum(appraise_default_count) from ( select sku_id, order_count, order_num, order_amount, 0 payment_count, 0 payment_num, 0 payment_amount, 0 refund_count, 0 refund_num, 0 refund_amount, 0 cart_count, 0 cart_num, 0 favor_count, 0 appraise_good_count, 0 appraise_mid_count, 0 appraise_bad_count, 0 appraise_default_count from tmp_order union all select sku_id, 0 order_count, 0 order_num, 0 order_amount, payment_count, payment_num, payment_amount, 0 refund_count, 0 refund_num, 0 refund_amount, 0 cart_count, 0 cart_num, 0 favor_count, 0 appraise_good_count, 0 appraise_mid_count, 0 appraise_bad_count, 0 appraise_default_count from tmp_payment union all select sku_id, 0 order_count, 0 order_num, 0 order_amount, 0 payment_count, 0 payment_num, 0 payment_amount, refund_count, refund_num, refund_amount, 0 cart_count, 0 cart_num, 0 favor_count, 0 appraise_good_count, 0 appraise_mid_count, 0 appraise_bad_count, 0 appraise_default_count from tmp_refund union all select sku_id, 0 order_count, 0 order_num, 0 order_amount, 0 payment_count, 0 payment_num, 0 payment_amount, 0 refund_count, 0 refund_num, 0 refund_amount, cart_count, cart_num, 0 favor_count, 0 appraise_good_count, 0 appraise_mid_count, 0 appraise_bad_count, 0 appraise_default_count from tmp_cart union all select sku_id, 0 order_count, 0 order_num, 0 order_amount, 0 payment_count, 0 payment_num, 0 payment_amount, 0 refund_count, 0 refund_num, 0 refund_amount, 0 cart_count, 0 cart_num, favor_count, 0 appraise_good_count, 0 appraise_mid_count, 0 appraise_bad_count, 0 appraise_default_count from tmp_favor union all select sku_id, 0 order_count, 0 order_num, 0 order_amount, 0 payment_count, 0 payment_num, 0 payment_amount, 0 refund_count, 0 refund_num, 0 refund_amount, 0 cart_count, 0 cart_num, 0 favor_count, appraise_good_count, appraise_mid_count, appraise_bad_count, appraise_default_count from tmp_appraise )tmp group by sku_id;
3）查詢加載結果
select * from dws_sku_action_daycount where dt='2020-03-10';
1.4.5 每日購買行為
1）建表語句
drop table if exists dws_sale_detail_daycount; create external table dws_sale_detail_daycount ( user_id string comment '用戶 id', sku_id string comment '商品 id', user_gender string comment '用戶性別', user_age string comment '用戶年齡', user_level string comment '用戶等級', order_price decimal(10,2) comment '商品價格', sku_name string comment '商品名稱', sku_tm_id string comment '品牌 id', sku_category3_id string comment '商品三級品類 id', sku_category2_id string comment '商品二級品類 id', sku_category1_id string comment '商品一級品類 id', sku_category3_name string comment '商品三級品類名稱', sku_category2_name string comment '商品二級品類名稱', sku_category1_name string comment '商品一級品類名稱', spu_id string comment '商品 spu', sku_num int comment '購買個數', order_count bigint comment '當日下單單數', order_amount decimal(16,2) comment '當日下單金額' ) COMMENT '每日購買行為' PARTITIONED BY (`dt` string) stored as parquet location '/warehouse/gmall/dws/dws_sale_detail_daycount/' tblproperties ("parquet.compression"="lzo");
2）數據裝載
insert overwrite table dws_sale_detail_daycount partition(dt='2020-03-10') select op.user_id, op.sku_id, ui.gender, months_between('2020-03-10', ui.birthday)/12 age, ui.user_level, si.price, si.sku_name, si.tm_id, si.category3_id, si.category2_id, si.category1_id, si.category3_name, si.category2_name, si.category1_name, si.spu_id, op.sku_num, op.order_count, op.order_amount from ( select user_id, sku_id, sum(sku_num) sku_num, count(*) order_count, sum(total_amount) order_amount from dwd_fact_order_detail where dt='2020-03-10' group by user_id, sku_id )op join ( select * from dwd_dim_user_info_his where end_date='9999-99-99' )ui on op.user_id = ui.id join ( select * from dwd_dim_sku_info where dt='2020-03-10' )si on op.sku_id = si.id;
3）查詢加載結果
select * from dws_sale_detail_daycount where dt='2020-03-10';
1.5 DWS 層數據導入腳本
1）vim dwd_to_dws.sh
在腳本中填寫如下內容
#!/bin/bash APP=gmall hive=/opt/modules/hive/bin/hive # 如果是輸入的日期按照取輸入日期；如果沒輸入日期取當前時間的前一天 if [ -n "$1" ] ;then do_date=$1 else do_date=`date -d "-1 day" +%F` fi sql=" insert overwrite table ${APP}.dws_uv_detail_daycount partition(dt='$do_date') select mid_id, concat_ws('|', collect_set(user_id)) user_id, concat_ws('|', collect_set(version_code)) version_code, concat_ws('|', collect_set(version_name)) version_name, concat_ws('|', collect_set(lang))lang, concat_ws('|', collect_set(source)) source, concat_ws('|', collect_set(os)) os, concat_ws('|', collect_set(area)) area, concat_ws('|', collect_set(model)) model, concat_ws('|', collect_set(brand)) brand, concat_ws('|', collect_set(sdk_version)) sdk_version, concat_ws('|', collect_set(gmail)) gmail, concat_ws('|', collect_set(height_width)) height_width, concat_ws('|', collect_set(app_time)) app_time, concat_ws('|', collect_set(network)) network, concat_ws('|', collect_set(lng)) lng, concat_ws('|', collect_set(lat)) lat, count(*) login_count from ${APP}.dwd_start_log where dt='$do_date' group by mid_id; with tmp_login as ( select user_id, count(*) login_count from ${APP}.dwd_start_log where dt='$do_date' and user_id is not null group by user_id ), tmp_cart as ( select user_id, count(*) cart_count, sum(cart_price*sku_num) cart_amount from ${APP}.dwd_fact_cart_info where dt='$do_date' and user_id is not null and date_format(create_time,'yyyy-MM-dd')='$do_date' group by user_id ), tmp_order as ( select user_id, count(*) order_count, sum(final_total_amount) order_amount from ${APP}.dwd_fact_order_info where dt='$do_date' group by user_id ) , tmp_payment as ( select user_id, count(*) payment_count, sum(payment_amount) payment_amount from ${APP}.dwd_fact_payment_info where dt='$do_date' group by user_id ) insert overwrite table ${APP}.dws_user_action_daycount partition(dt='$do_date') select user_actions.user_id, sum(user_actions.login_count), sum(user_actions.cart_count), sum(user_actions.cart_amount), sum(user_actions.order_count), sum(user_actions.order_amount), sum(user_actions.payment_count), sum(user_actions.payment_amount) from ( select user_id, login_count, 0 cart_count, 0 cart_amount, 0 order_count, 0 order_amount, 0 payment_count, 0 payment_amount from tmp_login union all select user_id, 0 login_count, cart_count, cart_amount, 0 order_count, 0 order_amount, 0 payment_count, 0 payment_amount from tmp_cart union all select user_id, 0 login_count, 0 cart_count, 0 cart_amount, order_count, order_amount, 0 payment_count, 0 payment_amount from tmp_order union all select user_id, 0 login_count, 0 cart_count, 0 cart_amount, 0 order_count, 0 order_amount, payment_count, payment_amount from tmp_payment ) user_actions group by user_id; with tmp_order as ( select sku_id, count(*) order_count, sum(sku_num) order_num, sum(total_amount) order_amount from ${APP}.dwd_fact_order_detail where dt='$do_date' group by sku_id ), tmp_payment as ( select sku_id, count(*) payment_count, sum(sku_num) payment_num, sum(total_amount) payment_amount from ${APP}.dwd_fact_order_detail where dt='$do_date' and order_id in ( select id from ${APP}.dwd_fact_order_info where (dt='$do_date' or dt=date_add('$do_date',-1)) and date_format(payment_time,'yyyy-MM-dd')='$do_date' ) group by sku_id ), tmp_refund as ( select sku_id, count(*) refund_count, sum(refund_num) refund_num, sum(refund_amount) refund_amount from ${APP}.dwd_fact_order_refund_info where dt='$do_date' group by sku_id ), tmp_cart as ( select sku_id, count(*) cart_count, sum(sku_num) cart_num from ${APP}.dwd_fact_cart_info where dt='$do_date' and date_format(create_time,'yyyy-MM-dd')='$do_date' group by sku_id ), tmp_favor as ( select sku_id, count(*) favor_count from ${APP}.dwd_fact_favor_info where dt='$do_date' and date_format(create_time,'yyyy-MM-dd')='$do_date' group by sku_id ), tmp_appraise as ( select sku_id, sum(if(appraise='1201',1,0)) appraise_good_count, sum(if(appraise='1202',1,0)) appraise_mid_count, sum(if(appraise='1203',1,0)) appraise_bad_count, sum(if(appraise='1204',1,0)) appraise_default_count from ${APP}.dwd_fact_comment_info where dt='$do_date' group by sku_id ) insert overwrite table ${APP}.dws_sku_action_daycount partition(dt='$do_date') select sku_id, sum(order_count), sum(order_num), sum(order_amount), sum(payment_count), sum(payment_num), sum(payment_amount), sum(refund_count), sum(refund_num), sum(refund_amount), sum(cart_count), sum(cart_num), sum(favor_count), sum(appraise_good_count), sum(appraise_mid_count), sum(appraise_bad_count), sum(appraise_default_count) from ( select sku_id, order_count, order_num, order_amount, 0 payment_count, 0 payment_num, 0 payment_amount, 0 refund_count, 0 refund_num, 0 refund_amount, 0 cart_count, 0 cart_num, 0 favor_count, 0 appraise_good_count, 0 appraise_mid_count, 0 appraise_bad_count, 0 appraise_default_count from tmp_order union all select sku_id, 0 order_count, 0 order_num, 0 order_amount, payment_count, payment_num, payment_amount, 0 refund_count, 0 refund_num, 0 refund_amount, 0 cart_count, 0 cart_num, 0 favor_count, 0 appraise_good_count, 0 appraise_mid_count, 0 appraise_bad_count, 0 appraise_default_count from tmp_payment union all select sku_id, 0 order_count, 0 order_num, 0 order_amount, 0 payment_count, 0 payment_num, 0 payment_amount, refund_count, refund_num, refund_amount, 0 cart_count, 0 cart_num, 0 favor_count, 0 appraise_good_count, 0 appraise_mid_count, 0 appraise_bad_count, 0 appraise_default_count from tmp_refund union all select sku_id, 0 order_count, 0 order_num, 0 order_amount, 0 payment_count, 0 payment_num, 0 payment_amount, 0 refund_count, 0 refund_num, 0 refund_amount, cart_count, cart_num, 0 favor_count, 0 appraise_good_count, 0 appraise_mid_count, 0 appraise_bad_count, 0 appraise_default_count from tmp_cart union all select sku_id, 0 order_count, 0 order_num, 0 order_amount, 0 payment_count, 0 payment_num, 0 payment_amount, 0 refund_count, 0 refund_num, 0 refund_amount, 0 cart_count, 0 cart_num, favor_count, 0 appraise_good_count, 0 appraise_mid_count, 0 appraise_bad_count, 0 appraise_default_count from tmp_favor union all select sku_id, 0 order_count, 0 order_num, 0 order_amount, 0 payment_count, 0 payment_num, 0 payment_amount, 0 refund_count, 0 refund_num, 0 refund_amount, 0 cart_count, 0 cart_num, 0 favor_count, appraise_good_count, appraise_mid_count, appraise_bad_count, appraise_default_count from tmp_appraise )tmp group by sku_id; insert overwrite table ${APP}.dws_sale_detail_daycount partition(dt='$do_date') select op.user_id, op.sku_id, ui.gender, months_between('$do_date', ui.birthday)/12 age, ui.user_level, si.price, si.sku_name, si.tm_id, si.category3_id, si.category2_id, si.category1_id, si.category3_name, si.category2_name, si.category1_name, si.spu_id, op.sku_num, op.order_count, op.order_amount from ( select user_id, sku_id, sum(sku_num) sku_num, count(*) order_count, sum(total_amount) order_amount from ${APP}.dwd_fact_order_detail where dt='$do_date' group by user_id, sku_id )op join ( select * from ${APP}.dwd_dim_user_info_his where end_date='9999-99-99' )ui on op.user_id = ui.id join ( select * from ${APP}.dwd_dim_sku_info where dt='$do_date' )si on op.sku_id = si.id; " $hive -e "$sql"
2）增加腳本執行權限
chmod 770 dwd_to_dws.sh
3）執行腳本導入數據
dwd_to_dws.sh 2020-03-11
4）查看導入數據
select * from dws_uv_detail_daycount where dt='2020-03-11'; select * from dws_user_action_daycount where dt='2020-03-11'; select * from dws_sku_action_daycount where dt='2020-03-11'; select * from dws_sale_detail_daycount where dt='2020-03-11';