-
背景
采用了SharedingSphere进行分表处理、数据脱敏、读写分离。
-
核心功能
SharedingSphere-JDBC：
数据分片：分库分表
读写分离：读写操作分别路由到主库与从库
数据脱敏：对敏感字段进行加密存储，使用时解密
分布式事务：基于XA协议的两阶段提交
ShardingSphere-Proxy：数据库代理
...
-
优缺点（对比Mycat）
SharedingSphere-JDBC基于jdbc驱动，以jar包形式提供轻量级插件，运维成本较低；Mycat是基于Proxy,以中间件的形式提供服务，本身类似于一个Mysql数据库，所以需要关注Mycat服务的高可用，运维成本较高，需要有一定的运维实力去维护各种配置。
SharedingSphere-JDBC只提供java语言的jar包，不能够跨平台；
SharedingSphere-JDBC不支持部分sql语句
-
配置（基于版本5.0.x）
1. spring.shardingsphere.rules : 规则配置
2. spring.shardingsphere.rules.sharding.sharding-algorithms : 分库分表规则配置
3. spring.shardingsphere.rules.sharding.tables : 分库分表对象配置
4. spring.shardingsphere.rules.sharding.key-generators : 分布式序列配置
5. spring.shardingsphere.rules.replica-query : 读写分离配置
6. spring.shardingsphere.rules.encrypt : 数据脱敏配置
@SpringBootApplication(exclude = {DataSourceAutoConfiguration.
class})
@EnableConfigurationProperties
public class SharedingApplication {
-
分库分表配置
#数据分片 #YamlShardingRuleSpringBootConfiguration
#数据源
spring.shardingsphere.datasource.names=test1,test2
spring.shardingsphere.datasource.common.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.common.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.test1.url=jdbc:mysql://localhost:3306/testdb1?useUnicode=true&characterEncoding=utf-8&useSSL=false
spring.shardingsphere.datasource.test1.username=root
spring.shardingsphere.datasource.test1.password=
spring.shardingsphere.datasource.test2.url=jdbc:mysql://localhost:3306/testdb2?useUnicode=true&characterEncoding=utf-8&useSSL=false
spring.shardingsphere.datasource.test2.username=root
spring.shardingsphere.datasource.test2.password=
#分片配置
spring.shardingsphere.rules.sharding.tables.order.actual-data-nodes=testdb$->{1..2}.order_$->{0..1}
##分布式序列配置（该项为必需，不是很友好）
spring.shardingsphere.rules.sharding.key-generators.snowflake.type=SNOWFLAKE
spring.shardingsphere.rules.sharding.key-generators.snowflake.props.max-vibration-offset=2048
spring.shardingsphere.rules.sharding.key-generators.snowflake.props.worker-id=1
##分库规则
spring.shardingsphere.rules.sharding.sharding-algorithms.dbstrategy.type=INLINE
spring.shardingsphere.rules.sharding.sharding-algorithms.dbstrategy.props.algorithm-expression=testdb$->{ Long.parseLong(sku.replaceAll('[a-z]','').replaceAll('[A-Z]','')) % 2 + 1}
##分表规则
spring.shardingsphere.rules.sharding.sharding-algorithms.tbstrategy.type=INLINE
spring.shardingsphere.rules.sharding.sharding-algorithms.tbstrategy.props.algorithm-expression=order_$->{ Long.parseLong(sku.replaceAll('[a-z]','').replaceAll('[A-Z]','')) % 2}
##库配置
spring.shardingsphere.rules.sharding.tables.order.database-strategy.standard.sharding-column=sku
spring.shardingsphere.rules.sharding.tables.order.database-strategy.standard.sharding-algorithm-name=dbstrategy
##分表配置
spring.shardingsphere.rules.sharding.tables.order.table-strategy.standard.sharding-column=sku
spring.shardingsphere.rules.sharding.tables.order.table-strategy.standard.sharding-algorithm-name=tbstrategy
##分布式ID
spring.shardingsphere.rules.sharding.tables.order.key-generate-strategy.column=order_no
spring.shardingsphere.rules.sharding.tables.order.key-generate-strategy.key-generator-name=snowflake
-
读写分离配置
#读写分离 # YamlReplicaQueryRuleSpringBootConfiguration
#数据源
spring.shardingsphere.datasource.names=master,slave
spring.shardingsphere.datasource.common.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.common.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.master.url=jdbc:mysql://localhost:3306/testdb1?useUnicode=true&characterEncoding=utf-8&useSSL=false
spring.shardingsphere.datasource.master.username=root
spring.shardingsphere.datasource.master.password=
spring.shardingsphere.datasource.slave.url=jdbc:mysql://localhost:3306/testdb2?useUnicode=true&characterEncoding=utf-8&useSSL=false
spring.shardingsphere.datasource.slave.username=root
spring.shardingsphere.datasource.slave.password=
#读写分离
spring.shardingsphere.rules.replica-query.data-sources.readwrite.name=readwrite
spring.shardingsphere.rules.replica-query.data-sources.readwrite.primary-data-source-name=master
spring.shardingsphere.rules.replica-query.data-sources.readwrite.replica-data-source-names=slave
spring.shardingsphere.rules.replica-query.load-balancers.balancers.type=ROUND_ROBIN
spring.shardingsphere.rules.replica-query.load-balancers.balancers.props.value=value
-
数据脱敏
#数据加密 #YamlEncryptRuleSpringBootConfiguration
#数据源
spring.shardingsphere.datasource.names=testdb1
spring.shardingsphere.datasource.common.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.common.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.test1.url=jdbc:mysql://localhost:3306/testdb1?useUnicode=true&characterEncoding=utf-8&useSSL=false
spring.shardingsphere.datasource.test1.username=root
spring.shardingsphere.datasource.test1.password=
#数据加密
##加密算法
spring.shardingsphere.rules.encrypt.encryptors.aesencrypt.type=AES
spring.shardingsphere.rules.encrypt.encryptors.aesencrypt.props.aes-key-value=123456
spring.shardingsphere.rules.encrypt.tables.order.name=orderEncryptMobile
spring.shardingsphere.rules.encrypt.tables.order.columns.mobile.cipher-column=mobile
#spring.shardingsphere.rules.encrypt.tables.order.columns.mobile.logic-column=mobile
#spring.shardingsphere.rules.encrypt.tables.order.columns.mobile.plain-column=mobile
#spring.shardingsphere.rules.encrypt.tables.order.columns.mobile.assisted-query-column=mobile
spring.shardingsphere.rules.encrypt.tables.order.columns.mobile.encryptor-name=aesencrypt
-
综合配置
#数据源
spring.shardingsphere.datasource.names=test1,test2,slave1,slave2
spring.shardingsphere.datasource.common.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.common.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.test1.url=jdbc:mysql://localhost:3306/testdb1?useUnicode=true&characterEncoding=utf-8&useSSL=false
spring.shardingsphere.datasource.test1.username=root
spring.shardingsphere.datasource.test1.password=
spring.shardingsphere.datasource.test2.url=jdbc:mysql://localhost:3306/testdb2?useUnicode=true&characterEncoding=utf-8&useSSL=false
spring.shardingsphere.datasource.test2.username=root
spring.shardingsphere.datasource.test2.password=
spring.shardingsphere.datasource.slave1.url=jdbc:mysql://localhost:3306/testdb1?useUnicode=true&characterEncoding=utf-8&useSSL=false
spring.shardingsphere.datasource.slave1.username=root
spring.shardingsphere.datasource.slave1.password=
spring.shardingsphere.datasource.slave2.url=jdbc:mysql://localhost:3306/testdb2?useUnicode=true&characterEncoding=utf-8&useSSL=false
spring.shardingsphere.datasource.slave2.username=root
spring.shardingsphere.datasource.slave2.password=
#分片配置
spring.shardingsphere.rules.sharding.tables.order.actual-data-nodes=testdb$->{1..2}.order_$->{0..1}
##分布式序列配置
spring.shardingsphere.rules.sharding.key-generators.snowflake.type=SNOWFLAKE
spring.shardingsphere.rules.sharding.key-generators.snowflake.props.max-vibration-offset=2
spring.shardingsphere.rules.sharding.key-generators.snowflake.props.worker-id=2
##分库规则
spring.shardingsphere.rules.sharding.sharding-algorithms.dbstrategy.type=INLINE
spring.shardingsphere.rules.sharding.sharding-algorithms.dbstrategy.props.algorithm-expression=testdb$->{ Long.parseLong(sku.replaceAll('[a-z]','').replaceAll('[A-Z]','')) % 2 + 1}
##分表规则
spring.shardingsphere.rules.sharding.sharding-algorithms.tbstrategy.type=INLINE
spring.shardingsphere.rules.sharding.sharding-algorithms.tbstrategy.props.algorithm-expression=order_$->{ Long.parseLong(sku.replaceAll('[a-z]','').replaceAll('[A-Z]','')) % 2}
##库配置
spring.shardingsphere.rules.sharding.tables.order.database-strategy.standard.sharding-column=sku
spring.shardingsphere.rules.sharding.tables.order.database-strategy.standard.sharding-algorithm-name=dbstrategy
##分表配置
spring.shardingsphere.rules.sharding.tables.order.table-strategy.standard.sharding-column=sku
spring.shardingsphere.rules.sharding.tables.order.table-strategy.standard.sharding-algorithm-name=tbstrategy
##分布式ID
spring.shardingsphere.rules.sharding.tables.order.key-generate-strategy.column=order_no
spring.shardingsphere.rules.sharding.tables.order.key-generate-strategy.key-generator-name=snowflake
## 默认分片配置
spring.shardingsphere.rules.sharding.default-database-strategy.standard.sharding-column=sku
spring.shardingsphere.rules.sharding.default-database-strategy.standard.sharding-algorithm-name=dbstrategy
#读写分离
spring.shardingsphere.rules.replica-query.data-sources.readwrite.name=readwrite
spring.shardingsphere.rules.replica-query.data-sources.readwrite.primary-data-source-name=test1
spring.shardingsphere.rules.replica-query.data-sources.readwrite.replica-data-source-names=slave1
spring.shardingsphere.rules.replica-query.data-sources.readwrite1.name=readwrite1
spring.shardingsphere.rules.replica-query.data-sources.readwrite1.primary-data-source-name=test2
spring.shardingsphere.rules.replica-query.data-sources.readwrite1.replica-data-source-names=slave2
#负载均衡
spring.shardingsphere.rules.replica-query.load-balancers.balancers.type=ROUND_ROBIN
spring.shardingsphere.rules.replica-query.load-balancers.balancers.props.value=value
#数据加密
##加密算法
spring.shardingsphere.rules.encrypt.encryptors.aesencrypt.type=AES
spring.shardingsphere.rules.encrypt.encryptors.aesencrypt.props.haes-key-value=123456
spring.shardingsphere.rules.encrypt.tables.order.name=orderEncryptMobile
spring.shardingsphere.rules.encrypt.tables.order.columns.mobile.cipher-column=mobile
spring.shardingsphere.rules.encrypt.tables.order.columns.mobile.encryptor-name=aesencrypt
-
分布式序列
- SharedingSphere提供内置分布式序列算法：雪花算法、UUID
- 自定义分布式序列算法（SPI加载方式）
resources 文件夹下增加 META-INF.services org.apache.shardingsphere.sharding.spi.KeyGenerateAlgorithm
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
org.apache.shardingsphere.sharding.algorithm.keygen.SnowflakeKeyGenerateAlgorithm
org.apache.shardingsphere.sharding.algorithm.keygen.UUIDKeyGenerateAlgorithm
com.bruce.common.algorithm.MyKeyGenerateAlgorithm
import lombok.Getter;
import lombok.Setter;
import org.apache.shardingsphere.sharding.spi.KeyGenerateAlgorithm;
import java.util.Properties;
import java.util.UUID;
@Getter
@Setter
public final class MyKeyGenerateAlgorithm implements KeyGenerateAlgorithm {
private Properties props = new Properties();
@Override
public void init() {
}
@Override
public synchronized Comparable<?> generateKey() {
return 自定义分布式ID;
}
@Override
public String getType() {
return "自定义分布式ID算法名称";
}
}
数据脱敏方式
- SharedingSphere提供内置加密算法：MD5 AES RC4
- 自定义数据脱敏算法（SPI加载方式）：
resources 文件夹下增加 META-INF.services org.apache.shardingsphere.encrypt.spi.EncryptAlgorithm
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
org.apache.shardingsphere.encrypt.algorithm.MD5EncryptAlgorithm
org.apache.shardingsphere.encrypt.algorithm.AESEncryptAlgorithm
org.apache.shardingsphere.encrypt.algorithm.RC4EncryptAlgorithm
com.bruce.common.encrypt.MyEncryptAlgorithm
import lombok.Getter;
import lombok.Setter;
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.codec.binary.StringUtils;
import org.apache.shardingsphere.encrypt.spi.EncryptAlgorithm;
import java.nio.charset.StandardCharsets;
import java.util.Properties;
@Getter
@Setter
public final class MyEncryptAlgorithm implements EncryptAlgorithm {
private static final String HAES_KEY = "自定义属性";
private Properties props = new Properties();
private byte[] secretKey;
@Override
public void init() {
secretKey = createSecretKey();
}
private byte[] createSecretKey() {
return HAES_KEY.getBytes();
}
@Override
public String encrypt(final Object plaintext) {
if (null == plaintext) {
return null;
}
return 自定义加密算法;
}
@Override
public Object decrypt(final String ciphertext) {
if (null == ciphertext) {
return null;
}
return 自定义解密算法;
}
@Override
public String getType() {
return "自定义脱敏算法名称";
}
}
-
运行原理
SharedingSphere内核包括解析引擎、路由引擎、改写引擎、执行引擎、归并引擎
ShardingSphere -> ShardingSphereConnection -> ShardingSpherePreparedStatement -> ShardingSphereResultSet
private ShardingSpherePreparedStatement(
final ShardingSphereConnection connection,
final String sql,
final int resultSetType,
final int resultSetConcurrency,
final int resultSetHoldability,
final boolean returnGeneratedKeys)
throws SQLException {
if (Strings.
isNullOrEmpty(sql)) {
throw new SQLException(SQLExceptionConstant.
SQL_STRING_NULL_OR_EMPTY);
}
this.
connection = connection;
schemaContexts = connection.getSchemaContexts();
this.
sql = sql;
statements =
new ArrayList<>();
parameterSets =
new ArrayList<>();
//解析引擎
ShardingSphereSQLParserEngine sqlStatementParserEngine =
new ShardingSphereSQLParserEngine(DatabaseTypeRegistry.
getTrunkDatabaseTypeName(
schemaContexts.getDatabaseType()));
sqlStatement = sqlStatementParserEngine.parse(sql,
true);
parameterMetaData =
new ShardingSphereParameterMetaData(
sqlStatement);
statementOption = returnGeneratedKeys ?
new StatementOption(
true) :
new StatementOption(resultSetType, resultSetConcurrency, resultSetHoldability);
SQLExecutor sqlExecutor =
new SQLExecutor(
schemaContexts.getExecutorKernel(), connection.isHoldTransaction());
preparedStatementExecutor =
new PreparedStatementExecutor(connection.getDataSourceMap(),
schemaContexts, sqlExecutor);
rawExecutor =
new RawJDBCExecutor(
schemaContexts.getExecutorKernel(), connection.isHoldTransaction());
batchPreparedStatementExecutor =
new BatchPreparedStatementExecutor(
schemaContexts, sqlExecutor);
kernelProcessor =
new KernelProcessor();
}
public ResultSet executeQuery(
final String sql)
throws SQLException {
if (Strings.
isNullOrEmpty(sql)) {
throw new SQLException(SQLExceptionConstant.
SQL_STRING_NULL_OR_EMPTY);
}
ResultSet result;
try {
executionContext = createExecutionContext(sql);
List<QueryResult> queryResults;
//执行引擎
if (ExecutorConstant.
MANAGED_RESOURCE) {
Collection<InputGroup<StatementExecuteUnit>> inputGroups = getInputGroups();
cacheStatements(inputGroups);
queryResults =
statementExecutor.executeQuery(inputGroups);
}
else {
queryResults =
rawExecutor.executeQuery(getRawInputGroups(),
new RawSQLExecutorCallback());
}
//归并引擎
MergedResult mergedResult = mergeQuery(queryResults);
result =
new ShardingSphereResultSet(
statements.stream().map(
this::getResultSet).collect(Collectors.
toList()), mergedResult,
this,
executionContext);
}
finally {
currentResultSet =
null;
}
currentResultSet = result;
return result;
}
private ExecutionContext createExecutionContext(
final String sql)
throws SQLException {
clearStatements();
LogicSQL logicSQL = createLogicSQL(sql);
ExecutionContext result =
kernelProcessor.generateExecutionContext(logicSQL,
schemaContexts.getDefaultSchema(),
schemaContexts.getProps());
logSQL(logicSQL,
schemaContexts.getProps(), result);
return result;
}
public ExecutionContext generateExecutionContext(
final LogicSQL logicSQL,
final ShardingSphereSchema schema,
final ConfigurationProperties props) {
Collection<ShardingSphereRule> rules = schema.getRules();
//路由引擎
SQLRouteEngine sqlRouteEngine =
new SQLRouteEngine(rules, props);
SQLStatementContext<?> sqlStatementContext = logicSQL.getSqlStatementContext();
RouteContext routeContext = sqlRouteEngine.route(logicSQL, schema);
//改写引擎
SQLRewriteEntry rewriteEntry =
new SQLRewriteEntry(schema.getMetaData().getSchemaMetaData().getConfiguredSchemaMetaData(), props, rules);
SQLRewriteResult rewriteResult = rewriteEntry.rewrite(logicSQL.getSql(), logicSQL.getParameters(), sqlStatementContext, routeContext);
Collection<ExecutionUnit> executionUnits = ExecutionContextBuilder.
build(schema.getMetaData(), rewriteResult, sqlStatementContext);
return new ExecutionContext(sqlStatementContext, executionUnits, routeContext);
}
//执行引擎
public <I, O> List<O> execute(
final Collection<InputGroup<I>> inputGroups,
final ExecutorCallback<I, O> firstCallback,
final ExecutorCallback<I, O> callback,
final boolean serial)
throws SQLException {
if (inputGroups.isEmpty()) {
return Collections.
emptyList();
}
return serial ? serialExecute(inputGroups, firstCallback, callback) : parallelExecute(inputGroups, firstCallback, callback);
}
private <I, O> List<O> serialExecute(
final Collection<InputGroup<I>> inputGroups,
final ExecutorCallback<I, O> firstCallback,
final ExecutorCallback<I, O> callback)
throws SQLException {
Iterator<InputGroup<I>> inputGroupsIterator = inputGroups.iterator();
InputGroup<I> firstInputs = inputGroupsIterator.next();
List<O> result =
new LinkedList<>(syncExecute(firstInputs,
null == firstCallback ? callback : firstCallback));
for (InputGroup<I> each : Lists.
newArrayList(inputGroupsIterator)) {
result.addAll(syncExecute(each, callback));
}
return result;
}
private <I, O> List<O> parallelExecute(final Collection<InputGroup<I>> inputGroups, final ExecutorCallback<I, O> firstCallback, final ExecutorCallback<I, O> callback) throws SQLException {
Iterator<InputGroup<I>> inputGroupsIterator = inputGroups.iterator();
InputGroup<I> firstInputs = inputGroupsIterator.next();
Collection<ListenableFuture<Collection<O>>> restResultFutures = asyncExecute(Lists.
newArrayList
(inputGroupsIterator), callback);
return getGroupResults(syncExecute(firstInputs, null == firstCallback ? callback : firstCallback), restResultFutures);
}
private <I, O> ListenableFuture<Collection<O>> asyncExecute(final InputGroup<I> inputGroup, final ExecutorCallback<I, O> callback) {
Map<String, Object> dataMap = ExecutorDataMap.
getValue
();
return executorService.getExecutorService().submit(() -> callback.execute(inputGroup.getInputs(), false, dataMap));
}
ConnectionMode connectionMode =
maxConnectionsSizePerQuery < sqlUnits.size() ? ConnectionMode.
CONNECTION_STRICTLY : ConnectionMode.
MEMORY_STRICTLY;
-
不支持sql
- 不支持存储过程，函数，游标的操作
CASE WHEN中包含子查询不支持
CASE WHEN中使用逻辑表名不支持（请使用表别名） 不支持 HAVING、UNION (ALL)
- 不支持部分子查询
- 子查询中包含聚合函数目前无法支持
- ...