1. sharding sphere 4.0.0-RC1版本 按年分表實戰
1.1. 需求
需要對日志表進行按時間划分表，由於用於后台系統，日志量預估不會太大，因此按年划分表
經過我不斷的查閱sharding sphere資料和實踐，我最后還是決定先建表，再把actual-data-nodes表結點給定下來，為什么這么說？
我糾結的是到底要不要動態創建表，若想要不自己手動每隔幾年維護表，我們當然希望能自動創建。但經過我的實踐，sharding sphere本身沒有提供該功能，但可以通過分片算法實現類中自定義實現，但前提是我們要隨時知道要分片表有幾個分片，比如log_2019,log_2020,log_2021，只要我能初始化的時候知道分片有幾個表以及表名，那么我就不會查詢到不存在的表導致報錯，反之則容易報錯
我們知道mysql可以通過查詢information_schema.TABLES來查詢存在的表，但是不知道是不是sharding sphere的bug，我用庫名加表名查該庫它會強制給我改寫成我默認的連接庫，導致表不存在，根本查不到
所以我退而求其次，下面我列出我的方案，方案采用的版本是4.0.0-RC1
1.2. 引入pom
- 先把pom列出來，只給代碼不給pom都是耍流氓
<!-- 分庫分表 -->
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>sharding-jdbc-spring-boot-starter</artifactId>
<version>4.0.0-RC1</version>
</dependency>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>sharding-jdbc-spring-namespace</artifactId>
<version>4.0.0-RC1</version>
</dependency>
1.3. application.yml配置
- 如下配置，分表最重要的是
table-strategy分表策略，sharding-column表示分表字段，當插入查詢需要指定哪個分表時，必須帶上這個條件，否則可能出錯，actual-data-nodes表示你分了哪些表，它有一定語法，如下$->{0..1}表示system_log_2020,system_log_2021兩張表，我需要在mysql建好這兩張表
spring:
shardingsphere:
props:
sql:
show: true
datasource:
names: ds0
ds0:
type: com.alibaba.druid.pool.DruidDataSource
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://xxxxx:3306/test?useUnicode=true&characterEncoding=utf-8&serverTimezone=GMT%2B8&allowMultiQueries=true
username: xxx
password: xxx
sharding:
tables:
system_log:
actual-data-nodes: ds0.system_log_202$->{0..1}
table-strategy:
standard:
sharding-column: created
precise-algorithm-class-name: com.xxx.platform.system.log.LogShardingAlgorithm
range-algorithm-class-name: com.xxx.platform.system.log.LogShardingAlgorithm
1.4. 分表策略
- 最重要的就是
LogShardingAlgorithm這個類
import com.google.common.collect.Range;
import lombok.extern.slf4j.Slf4j;
import org.apache.shardingsphere.api.sharding.standard.PreciseShardingAlgorithm;
import org.apache.shardingsphere.api.sharding.standard.PreciseShardingValue;
import org.apache.shardingsphere.api.sharding.standard.RangeShardingAlgorithm;
import org.apache.shardingsphere.api.sharding.standard.RangeShardingValue;
import java.util.ArrayList;
import java.util.Collection;
/**
* @author: laoliangliang
* @description: 日志分片
* @create: 2020/1/2 10:19
**/
@Slf4j
public class LogShardingAlgorithm implements PreciseShardingAlgorithm, RangeShardingAlgorithm<Integer> {
@Override
public String doSharding(Collection availableTargetNames, PreciseShardingValue shardingValue) {
String target = shardingValue.getValue().toString();
return shardingValue.getLogicTableName() + "_" + target.substring(target.lastIndexOf("_") + 1, target.lastIndexOf("_") + 5);
}
@Override
public Collection<String> doSharding(Collection<String> availableTargetNames, RangeShardingValue<Integer> shardingValue) {
Collection<String> availables = new ArrayList<>();
Range valueRange = shardingValue.getValueRange();
for (String target : availableTargetNames) {
Integer shardValue = Integer.parseInt(target.substring(target.lastIndexOf("_") + 1, target.lastIndexOf("_") + 5));
if (valueRange.hasLowerBound()) {
String lowerStr = valueRange.lowerEndpoint().toString();
Integer start = Integer.parseInt(lowerStr.substring(0, 4));
if (start - shardValue > 0) {
continue;
}
}
if (valueRange.hasUpperBound()) {
String upperStr = valueRange.upperEndpoint().toString();
Integer end = Integer.parseInt(upperStr.substring(0, 4));
if (end - shardValue < 0) {
continue;
}
}
availables.add(target);
}
return availables;
}
}
- 我實現了
PreciseShardingAlgorithm, RangeShardingAlgorithm這兩個接口，分別表示當created條件為=和between時會分別進入這兩個方法，用來判斷sql語句命中哪個表 - 這里要注意，created的
><大於小於判斷是不起效果的，求范圍只能用between，如果我說錯了請提醒哦 - 接下來調用sql語句我是這樣寫的
SELECT created,user_name,`action`,id FROM system_log
<where>
<if test="id!=null and id!=''">
and pk_id=#{id}
</if>
<if test="startTime != null and endTime != null">
and created BETWEEN #{startTime} and #{endTime}
</if>
</where>
order by created desc
1.5. 結果
- mybatis插入后日志如下，可以看到mybatis打印的日志表名還是system_log，但實際對應數據庫有
system_log_2020,system_log_2021兩張表，我插入的時間是2020年，所以只插入2020的表
2020-01-07 16:40:28.165 DEBUG 7780 --- [pool-4-thread-1] c.o.p.p.m.S.insertSelective : ==> Preparing: INSERT INTO system_log ( type,pk_id,remark,user_name,created,action ) VALUES( ?,?,?,?,?,? )
2020-01-07 16:40:28.165 DEBUG 7780 --- [pool-4-thread-1] c.o.p.p.m.S.insertSelective : ==> Parameters: 1(Integer), 0(Integer), string(String), 15162191629(String), 2020-01-07 16:40:28.161(Timestamp), 內容(String)
2020-01-07 16:40:28.198 INFO 7780 --- [pool-4-thread-1] ShardingSphere-SQL : Rule Type: sharding
2020-01-07 16:40:28.198 INFO 7780 --- [pool-4-thread-1] ShardingSphere-SQL : Logic SQL: INSERT INTO system_log ( type,pk_id,remark,user_name,created,action ) VALUES( ?,?,?,?,?,? )
2020-01-07 16:40:28.198 INFO 7780 --- [pool-4-thread-1] ShardingSphere-SQL : SQLStatement: InsertStatement(super=DMLStatement(super=AbstractSQLStatement(type=DML, tables=Tables(tables=[Table(name=system_log, alias=Optional.absent())]), routeConditions=Conditions(orCondition=OrCondition(andConditions=[AndCondition(conditions=[Condition(column=Column(name=created, tableName=system_log), operator=EQUAL, compareOperator=null, positionValueMap={}, positionIndexMap={0=4})])])), encryptConditions=Conditions(orCondition=OrCondition(andConditions=[])), sqlTokens=[TableToken(tableName=system_log, quoteCharacter=NONE, schemaNameLength=0), SQLToken(startIndex=24)], parametersIndex=6, logicSQL=INSERT INTO system_log ( type,pk_id,remark,user_name,created,action ) VALUES( ?,?,?,?,?,? )), deleteStatement=false, updateTableAlias={}, updateColumnValues={}, whereStartIndex=0, whereStopIndex=0, whereParameterStartIndex=0, whereParameterEndIndex=0), columnNames=[type, pk_id, remark, user_name, created, action], values=[InsertValue(columnValues=[org.apache.shardingsphere.core.parse.old.parser.expression.SQLPlaceholderExpression@21625d01, org.apache.shardingsphere.core.parse.old.parser.expression.SQLPlaceholderExpression@34dda176, org.apache.shardingsphere.core.parse.old.parser.expression.SQLPlaceholderExpression@5d631384, org.apache.shardingsphere.core.parse.old.parser.expression.SQLPlaceholderExpression@13cfbf64, org.apache.shardingsphere.core.parse.old.parser.expression.SQLPlaceholderExpression@20f67249, org.apache.shardingsphere.core.parse.old.parser.expression.SQLPlaceholderExpression@79f9b130])])
2020-01-07 16:40:28.198 INFO 7780 --- [pool-4-thread-1] ShardingSphere-SQL : Actual SQL: ds0 ::: INSERT INTO system_log_2020 (type, pk_id, remark, user_name, created, action) VALUES (?, ?, ?, ?, ?, ?) ::: [1, 0, string, 15162191629, 2020-01-07 16:40:28.161, 內容]
2020-01-07 16:40:28.210 DEBUG 7780 --- [pool-4-thread-1] c.o.p.p.m.S.insertSelective : <== Updates: 1
- 如上的查詢語句結果也同理，只查2020年
查詢參數
{
"endTime": "2020-01-10 01:01:01",
"id": 435,
"page": 1,
"pageSize": 10,
"startTime": "2020-01-01 01:01:01"
}
查詢結果
2020-01-07 16:50:49.878 DEBUG 5408 --- [nio-9000-exec-2] c.o.p.p.m.S.getReportLogList : ==> Preparing: SELECT created,user_name,`action`,id,remark FROM system_log WHERE pk_id=? and created BETWEEN ? and ? order by created desc LIMIT ?
2020-01-07 16:50:49.879 DEBUG 5408 --- [nio-9000-exec-2] c.o.p.p.m.S.getReportLogList : ==> Parameters: 435(Integer), 2020-01-01 01:01:01.0(Timestamp), 2020-01-10 01:01:01.0(Timestamp), 10(Integer)
2020-01-07 16:50:49.891 INFO 5408 --- [nio-9000-exec-2] ShardingSphere-SQL : Rule Type: sharding
2020-01-07 16:50:49.891 INFO 5408 --- [nio-9000-exec-2] ShardingSphere-SQL : Logic SQL: SELECT created,user_name,`action`,id,remark FROM system_log
WHERE pk_id=?
and created BETWEEN ? and ?
order by created desc LIMIT ?
2020-01-07 16:50:49.891 INFO 5408 --- [nio-9000-exec-2] ShardingSphere-SQL : SQLStatement: SelectStatement(super=DQLStatement(super=AbstractSQLStatement(type=DQL, tables=Tables(tables=[Table(name=system_log, alias=Optional.absent())]), routeConditions=Conditions(orCondition=OrCondition(andConditions=[AndCondition(conditions=[Condition(column=Column(name=created, tableName=system_log), operator=BETWEEN, compareOperator=null, positionValueMap={}, positionIndexMap={0=1, 1=2})])])), encryptConditions=Conditions(orCondition=OrCondition(andConditions=[])), sqlTokens=[TableToken(tableName=system_log, quoteCharacter=NONE, schemaNameLength=0)], parametersIndex=4, logicSQL=SELECT created,user_name,`action`,id,remark FROM system_log
WHERE pk_id=?
and created BETWEEN ? and ?
order by created desc LIMIT ? )), containStar=false, firstSelectItemStartIndex=7, selectListStopIndex=42, groupByLastIndex=0, items=[CommonSelectItem(expression=created, alias=Optional.absent()), CommonSelectItem(expression=user_name, alias=Optional.absent()), CommonSelectItem(expression=action, alias=Optional.absent()), CommonSelectItem(expression=id, alias=Optional.absent()), CommonSelectItem(expression=remark, alias=Optional.absent())], groupByItems=[], orderByItems=[OrderItem(owner=Optional.absent(), name=Optional.of(created), orderDirection=DESC, nullOrderDirection=ASC, index=-1, expression=null, alias=Optional.absent())], limit=Limit(offset=null, rowCount=LimitValue(value=-1, index=3, boundOpened=false)), subqueryStatement=null, subqueryStatements=[], subqueryConditions=[])
2020-01-07 16:50:49.891 INFO 5408 --- [nio-9000-exec-2] ShardingSphere-SQL : Actual SQL: ds0 ::: SELECT created,user_name,`action`,id,remark FROM system_log_2020
WHERE pk_id=?
and created BETWEEN ? and ?
order by created desc LIMIT ? ::: [435, 2020-01-01 01:01:01.0, 2020-01-10 01:01:01.0, 10]
2020-01-07 16:50:49.898 DEBUG 5408 --- [nio-9000-exec-2] c.o.p.p.m.S.getReportLogList : <== Total: 2
1.6. 總結
這次主要的碰壁內容就是created的大於小於問題，大於小於觸發不了表分片行為，需要特別注意。希望對你有幫助
歡迎關注公眾號，一起學習進步