前言:最近的記憶力越來越差了,好多項目中剛剛做過的技術,過段時間就忘記了。所以不得不開始寫博客了,把平時遇到的一些bug排除過程和嘗試的新技術記錄下來。
項目中有一張表有上億的數據,根據業務的需求,需要對該表按照時間(每個月)進行表分區。PS:該項目的數據庫之前一直是sql server,現在由於客戶的要求,改為Mysql。
先說一下總結,再慢慢講一下填坑的過程:
對於mysql5.6.27版本里有兩種時間格式的數據類型:datetime和timestamp。RANGE分區,如果是datetime:應使用TO_DAYS()這個函數。我的具體代碼如下:
ALTER TABLE t_ec_energyitemhourresult
PARTITION BY RANGE(TO_DAYS(F_StartHour))
(
PARTITION p0 VALUES LESS THAN (TO_DAYS('2012-01-01')),
PARTITION p1 VALUES LESS THAN (TO_DAYS('2012-02-01')),
PARTITION p2 VALUES LESS THAN (TO_DAYS('2012-03-01')),
PARTITION p3 VALUES LESS THAN (TO_DAYS('2012-04-01')),
PARTITION p4 VALUES LESS THAN (TO_DAYS('2012-05-01')),
PARTITION p5 VALUES LESS THAN (TO_DAYS('2012-06-01')),
PARTITION p6 VALUES LESS THAN (TO_DAYS('2012-07-01')),
PARTITION p7 VALUES LESS THAN (TO_DAYS('2012-08-01')),
PARTITION p8 VALUES LESS THAN (TO_DAYS('2012-09-01')),
PARTITION p9 VALUES LESS THAN (TO_DAYS('2012-10-01')),
PARTITION p10 VALUES LESS THAN (TO_DAYS('2012-11-01')),
PARTITION p11 VALUES LESS THAN (TO_DAYS('2012-12-01')),
PARTITION p12 VALUES LESS THAN (TO_DAYS('2013-01-01')),
PARTITION p13 VALUES LESS THAN (TO_DAYS('2013-02-01')),
PARTITION p14 VALUES LESS THAN (TO_DAYS('2013-03-01')),
PARTITION p15 VALUES LESS THAN (TO_DAYS('2013-04-01')),
PARTITION p16 VALUES LESS THAN (TO_DAYS('2013-05-01')),
PARTITION p17 VALUES LESS THAN (TO_DAYS('2013-06-01')),
PARTITION p18 VALUES LESS THAN (TO_DAYS('2013-07-01')),
PARTITION p19 VALUES LESS THAN (TO_DAYS('2013-08-01')),
PARTITION p20 VALUES LESS THAN (TO_DAYS('2013-09-01')),
PARTITION p21 VALUES LESS THAN (TO_DAYS('2013-10-01')),
PARTITION p22 VALUES LESS THAN (TO_DAYS('2013-11-01')),
PARTITION p23 VALUES LESS THAN (TO_DAYS('2013-12-01')),
PARTITION p24 VALUES LESS THAN (TO_DAYS('2014-01-01')),
PARTITION p25 VALUES LESS THAN (TO_DAYS('2014-02-01')),
PARTITION p26 VALUES LESS THAN (TO_DAYS('2014-03-01')),
PARTITION p27 VALUES LESS THAN (TO_DAYS('2014-04-01')),
PARTITION p28 VALUES LESS THAN (TO_DAYS('2014-05-01')),
PARTITION p29 VALUES LESS THAN (TO_DAYS('2014-06-01')),
PARTITION p30 VALUES LESS THAN (TO_DAYS('2014-07-01')),
PARTITION p31 VALUES LESS THAN (TO_DAYS('2014-08-01')),
PARTITION p32 VALUES LESS THAN (TO_DAYS('2014-09-01')),
PARTITION p33 VALUES LESS THAN (TO_DAYS('2014-10-01')),
PARTITION p34 VALUES LESS THAN (TO_DAYS('2014-11-01')),
PARTITION p35 VALUES LESS THAN (TO_DAYS('2014-12-01')),
PARTITION p36 VALUES LESS THAN (TO_DAYS('2015-01-01')),
PARTITION p37 VALUES LESS THAN (TO_DAYS('2015-02-01')),
PARTITION p38 VALUES LESS THAN (TO_DAYS('2015-03-01')),
PARTITION p39 VALUES LESS THAN (TO_DAYS('2015-04-01')),
PARTITION p40 VALUES LESS THAN (TO_DAYS('2015-05-01')),
PARTITION p41 VALUES LESS THAN (TO_DAYS('2015-06-01')),
PARTITION p42 VALUES LESS THAN (TO_DAYS('2015-07-01')),
PARTITION p43 VALUES LESS THAN (TO_DAYS('2015-08-01')),
PARTITION p44 VALUES LESS THAN (TO_DAYS('2015-09-01')),
PARTITION p45 VALUES LESS THAN (TO_DAYS('2015-10-01')),
PARTITION p46 VALUES LESS THAN (TO_DAYS('2015-11-01')),
PARTITION p47 VALUES LESS THAN (TO_DAYS('2015-12-01')),
PARTITION p48 VALUES LESS THAN (TO_DAYS('2016-01-01')),
PARTITION p49 VALUES LESS THAN (TO_DAYS('2016-02-01')),
PARTITION p50 VALUES LESS THAN (TO_DAYS('2016-03-01')),
PARTITION p51 VALUES LESS THAN (TO_DAYS('2016-04-01')),
PARTITION p52 VALUES LESS THAN (TO_DAYS('2016-05-01')),
PARTITION p53 VALUES LESS THAN (TO_DAYS('2016-06-01')),
PARTITION p54 VALUES LESS THAN (TO_DAYS('2016-07-01')),
PARTITION p55 VALUES LESS THAN (TO_DAYS('2016-08-01')),
PARTITION p56 VALUES LESS THAN (TO_DAYS('2016-09-01')),
PARTITION p57 VALUES LESS THAN (TO_DAYS('2016-10-01')),
PARTITION p58 VALUES LESS THAN (TO_DAYS('2016-11-01')),
PARTITION p59 VALUES LESS THAN (TO_DAYS('2016-12-01')),
PARTITION p60 VALUES LESS THAN (TO_DAYS('2017-01-01')),
PARTITION p61 VALUES LESS THAN (TO_DAYS('2017-02-01')),
PARTITION p62 VALUES LESS THAN (TO_DAYS('2017-03-01')),
PARTITION p63 VALUES LESS THAN (TO_DAYS('2017-04-01')),
PARTITION p64 VALUES LESS THAN (TO_DAYS('2017-05-01')),
PARTITION p65 VALUES LESS THAN (TO_DAYS('2017-06-01')),
PARTITION p66 VALUES LESS THAN (TO_DAYS('2017-07-01')),
PARTITION p67 VALUES LESS THAN (TO_DAYS('2017-08-01')),
PARTITION p68 VALUES LESS THAN (TO_DAYS('2017-09-01')),
PARTITION p69 VALUES LESS THAN (TO_DAYS('2017-10-01')),
PARTITION p70 VALUES LESS THAN (TO_DAYS('2017-11-01')),
PARTITION p71 VALUES LESS THAN (TO_DAYS('2017-12-01')),
PARTITION p72 VALUES LESS THAN (TO_DAYS('2018-01-01')),
PARTITION p73 VALUES LESS THAN (TO_DAYS('2018-02-01')),
PARTITION p74 VALUES LESS THAN (TO_DAYS('2018-03-01')),
PARTITION p75 VALUES LESS THAN (TO_DAYS('2018-04-01')),
PARTITION p76 VALUES LESS THAN (TO_DAYS('2018-05-01')),
PARTITION p77 VALUES LESS THAN (TO_DAYS('2018-06-01')),
PARTITION p78 VALUES LESS THAN (TO_DAYS('2018-07-01')),
PARTITION p79 VALUES LESS THAN (TO_DAYS('2018-08-01')),
PARTITION p80 VALUES LESS THAN (TO_DAYS('2018-09-01')),
PARTITION p81 VALUES LESS THAN (TO_DAYS('2018-10-01')),
PARTITION p82 VALUES LESS THAN (TO_DAYS('2018-11-01')),
PARTITION p83 VALUES LESS THAN (TO_DAYS('2018-12-01')),
PARTITION p84 VALUES LESS THAN (TO_DAYS('2019-01-01')),
PARTITION p85 VALUES LESS THAN (TO_DAYS('2019-02-01')),
PARTITION p86 VALUES LESS THAN (TO_DAYS('2019-03-01')),
PARTITION p87 VALUES LESS THAN (TO_DAYS('2019-04-01')),
PARTITION p88 VALUES LESS THAN (TO_DAYS('2019-05-01')),
PARTITION p89 VALUES LESS THAN (TO_DAYS('2019-06-01')),
PARTITION p90 VALUES LESS THAN (TO_DAYS('2019-07-01')),
PARTITION p91 VALUES LESS THAN (TO_DAYS('2019-08-01')),
PARTITION p92 VALUES LESS THAN (TO_DAYS('2019-09-01')),
PARTITION p93 VALUES LESS THAN (TO_DAYS('2019-10-01')),
PARTITION p94 VALUES LESS THAN (TO_DAYS('2019-11-01')),
PARTITION p95 VALUES LESS THAN (TO_DAYS('2019-12-01')),
PARTITION p96 VALUES LESS THAN (TO_DAYS('2020-01-01')),
PARTITION p97 VALUES LESS THAN (TO_DAYS('2020-02-01')),
PARTITION p98 VALUES LESS THAN (TO_DAYS('2020-03-01')),
PARTITION p99 VALUES LESS THAN (TO_DAYS('2020-04-01')),
PARTITION p100 VALUES LESS THAN (TO_DAYS('2020-05-01')),
PARTITION p101 VALUES LESS THAN (TO_DAYS('2020-06-01')),
PARTITION p102 VALUES LESS THAN (TO_DAYS('2020-07-01')),
PARTITION p103 VALUES LESS THAN (TO_DAYS('2020-08-01')),
PARTITION p104 VALUES LESS THAN (TO_DAYS('2020-09-01')),
PARTITION p105 VALUES LESS THAN (TO_DAYS('2020-10-01')),
PARTITION p106 VALUES LESS THAN (TO_DAYS('2020-11-01')),
PARTITION p107 VALUES LESS THAN (TO_DAYS('2020-12-01')),
PARTITION p108 VALUES LESS THAN (TO_DAYS('2021-01-01')),
PARTITION p109 VALUES LESS THAN (MAXVALUE)
);
如果是timestamp類型,應使用UNIX_TIMESTAMP函數轉換一下。
ALTER TABLE t_ec_energyitemresult
PARTITION BY RANGE( UNIX_TIMESTAMP(F_StartTime))
(
PARTITION p0 VALUES LESS THAN (UNIX_TIMESTAMP('2012-01-01')),
PARTITION p1 VALUES LESS THAN (UNIX_TIMESTAMP('2012-02-01')),
PARTITION p2 VALUES LESS THAN (UNIX_TIMESTAMP('2012-03-01')),
PARTITION p3 VALUES LESS THAN (UNIX_TIMESTAMP('2012-04-01')),
PARTITION p4 VALUES LESS THAN (UNIX_TIMESTAMP('2012-05-01')),
PARTITION p5 VALUES LESS THAN (UNIX_TIMESTAMP('2012-06-01')),
PARTITION p6 VALUES LESS THAN (UNIX_TIMESTAMP('2012-07-01')),
PARTITION p7 VALUES LESS THAN (UNIX_TIMESTAMP('2012-08-01')),
PARTITION p8 VALUES LESS THAN (UNIX_TIMESTAMP('2012-09-01')),
PARTITION p9 VALUES LESS THAN (UNIX_TIMESTAMP('2012-10-01')),
PARTITION p10 VALUES LESS THAN (UNIX_TIMESTAMP('2012-11-01')),
PARTITION p11 VALUES LESS THAN (UNIX_TIMESTAMP('2012-12-01')),
PARTITION p12 VALUES LESS THAN (UNIX_TIMESTAMP('2013-01-01')),
PARTITION p13 VALUES LESS THAN (UNIX_TIMESTAMP('2013-02-01')),
PARTITION p14 VALUES LESS THAN (UNIX_TIMESTAMP('2013-03-01')),
PARTITION p15 VALUES LESS THAN (UNIX_TIMESTAMP('2013-04-01')),
PARTITION p16 VALUES LESS THAN (UNIX_TIMESTAMP('2013-05-01')),
PARTITION p17 VALUES LESS THAN (UNIX_TIMESTAMP('2013-06-01')),
PARTITION p18 VALUES LESS THAN (UNIX_TIMESTAMP('2013-07-01')),
PARTITION p19 VALUES LESS THAN (UNIX_TIMESTAMP('2013-08-01')),
PARTITION p20 VALUES LESS THAN (UNIX_TIMESTAMP('2013-09-01')),
PARTITION p21 VALUES LESS THAN (UNIX_TIMESTAMP('2013-10-01')),
PARTITION p22 VALUES LESS THAN (UNIX_TIMESTAMP('2013-11-01')),
PARTITION p23 VALUES LESS THAN (UNIX_TIMESTAMP('2013-12-01')),
PARTITION p24 VALUES LESS THAN (UNIX_TIMESTAMP('2014-01-01')),
PARTITION p25 VALUES LESS THAN (UNIX_TIMESTAMP('2014-02-01')),
PARTITION p26 VALUES LESS THAN (UNIX_TIMESTAMP('2014-03-01')),
PARTITION p27 VALUES LESS THAN (UNIX_TIMESTAMP('2014-04-01')),
PARTITION p28 VALUES LESS THAN (UNIX_TIMESTAMP('2014-05-01')),
PARTITION p29 VALUES LESS THAN (UNIX_TIMESTAMP('2014-06-01')),
PARTITION p30 VALUES LESS THAN (UNIX_TIMESTAMP('2014-07-01')),
PARTITION p31 VALUES LESS THAN (UNIX_TIMESTAMP('2014-08-01')),
PARTITION p32 VALUES LESS THAN (UNIX_TIMESTAMP('2014-09-01')),
PARTITION p33 VALUES LESS THAN (UNIX_TIMESTAMP('2014-10-01')),
PARTITION p34 VALUES LESS THAN (UNIX_TIMESTAMP('2014-11-01')),
PARTITION p35 VALUES LESS THAN (UNIX_TIMESTAMP('2014-12-01')),
PARTITION p36 VALUES LESS THAN (UNIX_TIMESTAMP('2015-01-01')),
PARTITION p37 VALUES LESS THAN (UNIX_TIMESTAMP('2015-02-01')),
PARTITION p38 VALUES LESS THAN (UNIX_TIMESTAMP('2015-03-01')),
PARTITION p39 VALUES LESS THAN (UNIX_TIMESTAMP('2015-04-01')),
PARTITION p40 VALUES LESS THAN (UNIX_TIMESTAMP('2015-05-01')),
PARTITION p41 VALUES LESS THAN (UNIX_TIMESTAMP('2015-06-01')),
PARTITION p42 VALUES LESS THAN (UNIX_TIMESTAMP('2015-07-01')),
PARTITION p43 VALUES LESS THAN (UNIX_TIMESTAMP('2015-08-01')),
PARTITION p44 VALUES LESS THAN (UNIX_TIMESTAMP('2015-09-01')),
PARTITION p45 VALUES LESS THAN (UNIX_TIMESTAMP('2015-10-01')),
PARTITION p46 VALUES LESS THAN (UNIX_TIMESTAMP('2015-11-01')),
PARTITION p47 VALUES LESS THAN (UNIX_TIMESTAMP('2015-12-01')),
PARTITION p48 VALUES LESS THAN (UNIX_TIMESTAMP('2016-01-01')),
PARTITION p49 VALUES LESS THAN (UNIX_TIMESTAMP('2016-02-01')),
PARTITION p50 VALUES LESS THAN (UNIX_TIMESTAMP('2016-03-01')),
PARTITION p51 VALUES LESS THAN (UNIX_TIMESTAMP('2016-04-01')),
PARTITION p52 VALUES LESS THAN (UNIX_TIMESTAMP('2016-05-01')),
PARTITION p53 VALUES LESS THAN (UNIX_TIMESTAMP('2016-06-01')),
PARTITION p54 VALUES LESS THAN (UNIX_TIMESTAMP('2016-07-01')),
PARTITION p55 VALUES LESS THAN (UNIX_TIMESTAMP('2016-08-01')),
PARTITION p56 VALUES LESS THAN (UNIX_TIMESTAMP('2016-09-01')),
PARTITION p57 VALUES LESS THAN (UNIX_TIMESTAMP('2016-10-01')),
PARTITION p58 VALUES LESS THAN (UNIX_TIMESTAMP('2016-11-01')),
PARTITION p59 VALUES LESS THAN (UNIX_TIMESTAMP('2016-12-01')),
PARTITION p60 VALUES LESS THAN (UNIX_TIMESTAMP('2017-01-01')),
PARTITION p61 VALUES LESS THAN (UNIX_TIMESTAMP('2017-02-01')),
PARTITION p62 VALUES LESS THAN (UNIX_TIMESTAMP('2017-03-01')),
PARTITION p63 VALUES LESS THAN (UNIX_TIMESTAMP('2017-04-01')),
PARTITION p64 VALUES LESS THAN (UNIX_TIMESTAMP('2017-05-01')),
PARTITION p65 VALUES LESS THAN (UNIX_TIMESTAMP('2017-06-01')),
PARTITION p66 VALUES LESS THAN (UNIX_TIMESTAMP('2017-07-01')),
PARTITION p67 VALUES LESS THAN (UNIX_TIMESTAMP('2017-08-01')),
PARTITION p68 VALUES LESS THAN (UNIX_TIMESTAMP('2017-09-01')),
PARTITION p69 VALUES LESS THAN (UNIX_TIMESTAMP('2017-10-01')),
PARTITION p70 VALUES LESS THAN (UNIX_TIMESTAMP('2017-11-01')),
PARTITION p71 VALUES LESS THAN (UNIX_TIMESTAMP('2017-12-01')),
PARTITION p72 VALUES LESS THAN (UNIX_TIMESTAMP('2018-01-01')),
PARTITION p73 VALUES LESS THAN (UNIX_TIMESTAMP('2018-02-01')),
PARTITION p74 VALUES LESS THAN (UNIX_TIMESTAMP('2018-03-01')),
PARTITION p75 VALUES LESS THAN (UNIX_TIMESTAMP('2018-04-01')),
PARTITION p76 VALUES LESS THAN (UNIX_TIMESTAMP('2018-05-01')),
PARTITION p77 VALUES LESS THAN (UNIX_TIMESTAMP('2018-06-01')),
PARTITION p78 VALUES LESS THAN (UNIX_TIMESTAMP('2018-07-01')),
PARTITION p79 VALUES LESS THAN (UNIX_TIMESTAMP('2018-08-01')),
PARTITION p80 VALUES LESS THAN (UNIX_TIMESTAMP('2018-09-01')),
PARTITION p81 VALUES LESS THAN (UNIX_TIMESTAMP('2018-10-01')),
PARTITION p82 VALUES LESS THAN (UNIX_TIMESTAMP('2018-11-01')),
PARTITION p83 VALUES LESS THAN (UNIX_TIMESTAMP('2018-12-01')),
PARTITION p84 VALUES LESS THAN (UNIX_TIMESTAMP('2019-01-01')),
PARTITION p85 VALUES LESS THAN (UNIX_TIMESTAMP('2019-02-01')),
PARTITION p86 VALUES LESS THAN (UNIX_TIMESTAMP('2019-03-01')),
PARTITION p87 VALUES LESS THAN (UNIX_TIMESTAMP('2019-04-01')),
PARTITION p88 VALUES LESS THAN (UNIX_TIMESTAMP('2019-05-01')),
PARTITION p89 VALUES LESS THAN (UNIX_TIMESTAMP('2019-06-01')),
PARTITION p90 VALUES LESS THAN (UNIX_TIMESTAMP('2019-07-01')),
PARTITION p91 VALUES LESS THAN (UNIX_TIMESTAMP('2019-08-01')),
PARTITION p92 VALUES LESS THAN (UNIX_TIMESTAMP('2019-09-01')),
PARTITION p93 VALUES LESS THAN (UNIX_TIMESTAMP('2019-10-01')),
PARTITION p94 VALUES LESS THAN (UNIX_TIMESTAMP('2019-11-01')),
PARTITION p95 VALUES LESS THAN (UNIX_TIMESTAMP('2019-12-01')),
PARTITION p96 VALUES LESS THAN (UNIX_TIMESTAMP('2020-01-01')),
PARTITION p97 VALUES LESS THAN (UNIX_TIMESTAMP('2020-02-01')),
PARTITION p98 VALUES LESS THAN (UNIX_TIMESTAMP('2020-03-01')),
PARTITION p99 VALUES LESS THAN (UNIX_TIMESTAMP('2020-04-01')),
PARTITION p100 VALUES LESS THAN (UNIX_TIMESTAMP('2020-05-01')),
PARTITION p101 VALUES LESS THAN (UNIX_TIMESTAMP('2020-06-01')),
PARTITION p102 VALUES LESS THAN (UNIX_TIMESTAMP('2020-07-01')),
PARTITION p103 VALUES LESS THAN (UNIX_TIMESTAMP('2020-08-01')),
PARTITION p104 VALUES LESS THAN (UNIX_TIMESTAMP('2020-09-01')),
PARTITION p105 VALUES LESS THAN (UNIX_TIMESTAMP('2020-10-01')),
PARTITION p106 VALUES LESS THAN (UNIX_TIMESTAMP('2020-11-01')),
PARTITION p107 VALUES LESS THAN (UNIX_TIMESTAMP('2020-12-01')),
PARTITION p108 VALUES LESS THAN (UNIX_TIMESTAMP('2021-01-01')),
PARTITION p109 VALUES LESS THAN (MAXVALUE)
);
下面是從mysql官網上copy下來的:
In MySQL 5.6, range optimization can be used for the TO_DAYS()
, TO_SECONDS()
, and YEAR()
functions. In addition, beginning with MySQL 5.6.3, UNIX_TIMESTAMP()
is treated as monotonic in partitioning expressions.
個人理解的就是:Mysql中Range分區不支持時間類型,需分別轉換到正整數。其中的函數就是上述英文中藍色的幾個函數。
說說這個坑:
由於同事的mysql里的數據是通過SQLlog這個軟件從sqlserver里導過來的。原來sqlserver分區列的datetime類型被轉換成了mysql中的timestamp(6),當他執行分區語句的時候,總是報錯:
The PARTITION function returns the wrong type,提示返回了錯誤類型。而我的mysql里的數據是通過navicat這個軟件從sqlserver里導過來的。一切正常。
后來經過排查:發現我的timestamp長度為0.
mysql> select UNIX_TIMESTAMP(CURRENT_TIMESTAMP);
+-----------------------------------+
| UNIX_TIMESTAMP(CURRENT_TIMESTAMP) |
+-----------------------------------+
| 1445306197 |
+-----------------------------------+
1 row in set
mysql> select UNIX_TIMESTAMP(CURRENT_TIMESTAMP(6));
+--------------------------------------+
| UNIX_TIMESTAMP(CURRENT_TIMESTAMP(6)) |
+--------------------------------------+
| 1445306210.812460 |
+--------------------------------------+
1 row in set
mysql> select CURRENT_TIMESTAMP;
+---------------------+
| CURRENT_TIMESTAMP |
+---------------------+
| 2015-10-20 09:57:42 |
+---------------------+
1 row in set
mysql> select CURRENT_TIMESTAMP(6);
+----------------------------+
| CURRENT_TIMESTAMP(6) |
+----------------------------+
| 2015-10-20 09:57:53.970072 |
+----------------------------+
1 row in set
因為timestamp(6)存儲了6位微秒級,所以無法轉換成整數了。