項目里面一直用Sharding-JDBC,今天整理一下,就當溫故而知新了,也是穩固而知新了。
一、整體介紹
- 項目采用的框架是SpringBoot+Mybatis+Sharding-JDBC,采用的是properties的形式;
- 分為兩個數據庫sharding_0,sharding_1。每個庫三個表,t_user_00,t_user_01,t_user_02;
- 分庫策略:age % 2 = 0的數據存儲到sharding_0 ,為1的數據存儲到sharding_1;
- 分表策略:user_id % 3 = 0的數據存儲到t_user_00,為1的數據存儲到t_user_01,為2的數據存儲到t_user_02;
- Sharding-JDBC官網
二、數據庫文件
分別在sharding_0,sharding_1兩個數據庫中執行下列腳本。
SET NAMES utf8mb4; SET FOREIGN_KEY_CHECKS = 0; -- ---------------------------- -- Table structure for t_user_00 -- ---------------------------- DROP TABLE IF EXISTS `t_user_00`; CREATE TABLE `t_user_00` ( `id` bigint(0) NOT NULL, `user_id` int(0) NOT NULL, `name` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL, `age` int(0) NOT NULL, PRIMARY KEY (`id`) USING BTREE ) ENGINE = InnoDB AUTO_INCREMENT = 10 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic; -- ---------------------------- -- Table structure for t_user_01 -- ---------------------------- DROP TABLE IF EXISTS `t_user_01`; CREATE TABLE `t_user_01` ( `id` bigint(0) NOT NULL, `user_id` int(0) NOT NULL, `name` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL, `age` int(0) NOT NULL, PRIMARY KEY (`id`) USING BTREE ) ENGINE = InnoDB AUTO_INCREMENT = 6 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic; -- ---------------------------- -- Table structure for t_user_02 -- ---------------------------- DROP TABLE IF EXISTS `t_user_02`; CREATE TABLE `t_user_02` ( `id` bigint(0) NOT NULL, `user_id` int(0) NOT NULL, `name` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL, `age` int(0) NOT NULL, PRIMARY KEY (`id`) USING BTREE ) ENGINE = InnoDB AUTO_INCREMENT = 3 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic; SET FOREIGN_KEY_CHECKS = 1;
三、項目結構
四、pom.xml
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>2.1.13.RELEASE</version> <relativePath/> <!-- lookup parent from repository --> </parent> <groupId>com.gougou</groupId> <artifactId>shardingjdbc-demo</artifactId> <version>0.0.1-SNAPSHOT</version> <name>shardingjdbc-demo</name> <description>shardingjdbc-demo</description> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <java.version>1.8</java.version> </properties> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <optional>true</optional> </dependency> <dependency> <groupId>org.mybatis.spring.boot</groupId> <artifactId>mybatis-spring-boot-starter</artifactId> <version>1.3.2</version> </dependency> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> </dependency> <dependency> <groupId>com.dangdang</groupId> <artifactId>sharding-jdbc-core</artifactId> <version>1.5.4</version> </dependency> <!--Snowflake算法中使用-->
<dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-lang3</artifactId> <version>3.8.1</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </project>
五、application.properties
spring.application.name=shardingjdbc-demo #mybatis配置 # mapper映射文件位置 #mybatis.mapper-locations=classpath:mapper/*.xml # 實體類所在的位置 mybatis.type-aliases-package=com.gouggou.shardingtable.entity #datasource spring.devtools.remote.restart.enabled=false #data source1 spring.datasource.db0.driverClassName=com.mysql.jdbc.Driver spring.datasource.db0.jdbcUrl=jdbc:mysql://127.0.0.1:3306/sharding_0?serverTimezone=UTC spring.datasource.db0.username=root spring.datasource.db0.password=root #data source2 spring.datasource.db1.driverClassName=com.mysql.jdbc.Driver spring.datasource.db1.jdbcUrl=jdbc:mysql://127.0.0.1:3306/sharding_1?serverTimezone=UTC spring.datasource.db1.username=root spring.datasource.db1.password=root
六、啟動類Application
@SpringBootApplication(exclude = {DataSourceAutoConfiguration.class}) @MapperScan(basePackages = "com.gouggou.mapper") @EnableTransactionManagement(proxyTargetClass = true) //開啟事物管理功能 public class Application { public static void main(String[] args) { SpringApplication.run(Application.class, args); } }
七、分庫分表策略配置
1、數據源配置
注意下面標紅的部分,這是分庫分表的部分邏輯體現。
package com.gouggou.config; import com.dangdang.ddframe.rdb.sharding.api.ShardingDataSourceFactory; import com.dangdang.ddframe.rdb.sharding.api.rule.BindingTableRule; import com.dangdang.ddframe.rdb.sharding.api.rule.DataSourceRule; import com.dangdang.ddframe.rdb.sharding.api.rule.ShardingRule; import com.dangdang.ddframe.rdb.sharding.api.rule.TableRule; import com.dangdang.ddframe.rdb.sharding.api.strategy.database.DatabaseShardingStrategy; import com.dangdang.ddframe.rdb.sharding.api.strategy.table.TableShardingStrategy; import org.apache.ibatis.session.SqlSessionFactory; import org.mybatis.spring.SqlSessionFactoryBean; import org.mybatis.spring.SqlSessionTemplate; import org.mybatis.spring.annotation.MapperScan; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.boot.jdbc.DataSourceBuilder; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Primary; import org.springframework.core.io.support.PathMatchingResourcePatternResolver; import org.springframework.jdbc.datasource.DataSourceTransactionManager; import javax.sql.DataSource; import java.sql.SQLException; import java.util.*; @Configuration @MapperScan(basePackages = "com.gouggou.mapper", sqlSessionTemplateRef = "test1SqlSessionTemplate") public class DataSourceConfig { /** * 主鍵生成器 * * @return */ @Bean public DefaultKeyGenerator defaultKeyGenerator() { return new DefaultKeyGenerator(); } /** * 配置數據源0 * * @return */ @Bean(name = "dataSource0") @ConfigurationProperties(prefix = "spring.datasource.db0") public DataSource dataSource0() { return DataSourceBuilder.create().build(); } /** * 配置數據源1 * * @return */ @Bean(name = "dataSource1") @ConfigurationProperties(prefix = "spring.datasource.db1") public DataSource dataSource1() { return DataSourceBuilder.create().build(); } /** * 配置數據源規則,即將多個數據源交給sharding-jdbc管理,並且可以設置默認的數據源, * 當表沒有配置分庫規則時會使用默認的數據源 * * @param dataSource0 * @param dataSource1 * @return */ @Bean public DataSourceRule dataSourceRule(@Qualifier("dataSource0") DataSource dataSource0, @Qualifier("dataSource1") DataSource dataSource1) { Map<String, DataSource> dataSourceMap = new HashMap<>(); dataSourceMap.put("dataSource0", dataSource0); dataSourceMap.put("dataSource1", dataSource1); //設置默認庫,兩個庫以上時必須設置默認庫。默認庫的數據源名稱必須是dataSourceMap的key之一 return new DataSourceRule(dataSourceMap, "dataSource0"); } /** * 配置數據源策略和表策略,具體策略需要自己實現 * * @param dataSourceRule * @return */ @Bean public ShardingRule shardingRule(DataSourceRule dataSourceRule) { //分表策略 TableRule userTableRule = TableRule.builder("t_user") .actualTables(Arrays.asList("t_user_00", "t_user_01", "t_user_02")) .tableShardingStrategy(new TableShardingStrategy("user_id", new ModuloTableShardingAlgorithm())) .dataSourceRule(dataSourceRule) .build(); //綁定表策略,在查詢時會使用主表策略計算路由的數據源,因此需要約定綁定表策略的表的規則需要一致,可以一定程度提高效率 List<BindingTableRule> bindingTableRules = new ArrayList<BindingTableRule>(); bindingTableRules.add(new BindingTableRule(Arrays.asList(userTableRule))); return ShardingRule.builder() .dataSourceRule(dataSourceRule) .tableRules(Arrays.asList(userTableRule)) .bindingTableRules(bindingTableRules) .databaseShardingStrategy(new DatabaseShardingStrategy("age", new ModuloDatabaseShardingAlgorithm())) .tableShardingStrategy(new TableShardingStrategy("user_id", new ModuloTableShardingAlgorithm())) .build(); } /** * 創建sharding-jdbc的數據源DataSource,MybatisAutoConfiguration會使用此數據源 * * @param shardingRule * @return * @throws SQLException */ @Bean(name = "dataSource") public DataSource shardingDataSource(ShardingRule shardingRule) throws SQLException { return ShardingDataSourceFactory.createDataSource(shardingRule); } /** * 手動配置事務管理器 * * @param dataSource * @return */ @Bean public DataSourceTransactionManager transactitonManager(@Qualifier("dataSource") DataSource dataSource) { return new DataSourceTransactionManager(dataSource); } @Bean(name = "test1SqlSessionFactory") @Primary public SqlSessionFactory testSqlSessionFactory(@Qualifier("dataSource") DataSource dataSource) throws Exception { SqlSessionFactoryBean bean = new SqlSessionFactoryBean(); bean.setDataSource(dataSource); bean.setMapperLocations(new PathMatchingResourcePatternResolver().getResources("classpath:mapper/*.xml")); return bean.getObject(); } @Bean(name = "test1SqlSessionTemplate") @Primary public SqlSessionTemplate testSqlSessionTemplate(@Qualifier("test1SqlSessionFactory") SqlSessionFactory sqlSessionFactory) throws Exception { return new SqlSessionTemplate(sqlSessionFactory); } }
2、分庫策略配置
package com.gouggou.config; import com.dangdang.ddframe.rdb.sharding.api.ShardingValue; import com.dangdang.ddframe.rdb.sharding.api.strategy.database.SingleKeyDatabaseShardingAlgorithm; import com.google.common.collect.Range; import java.util.Collection; import java.util.LinkedHashSet; /** * 分庫策略 */ public class ModuloDatabaseShardingAlgorithm implements SingleKeyDatabaseShardingAlgorithm<Integer> { /** * equals查詢 * * @param databaseNames * @param shardingValue * @return */ @Override public String doEqualSharding(Collection<String> databaseNames, ShardingValue<Integer> shardingValue) { for (String each : databaseNames) { if (each.endsWith(Integer.parseInt(shardingValue.getValue().toString()) % 2 + "")) { return each; } } throw new IllegalArgumentException(); } /** * in查詢 * * @param databaseNames * @param shardingValue * @return */ @Override public Collection<String> doInSharding(Collection<String> databaseNames, ShardingValue<Integer> shardingValue) { Collection<String> result = new LinkedHashSet<>(databaseNames.size()); for (Integer value : shardingValue.getValues()) { for (String tableName : databaseNames) { if (tableName.endsWith(value % 2 + "")) { result.add(tableName); } } } return result; } /** * between查詢 * * @param databaseNames * @param shardingValue * @return */ @Override public Collection<String> doBetweenSharding(Collection<String> databaseNames, ShardingValue<Integer> shardingValue) { Collection<String> result = new LinkedHashSet<>(databaseNames.size()); Range<Integer> range = (Range<Integer>) shardingValue.getValueRange(); for (Integer i = range.lowerEndpoint(); i <= range.upperEndpoint(); i++) { for (String each : databaseNames) { if (each.endsWith(i % 2 + "")) { result.add(each); } } } return result; } }
3、分表策略配置
package com.gouggou.config; import com.dangdang.ddframe.rdb.sharding.api.ShardingValue; import com.dangdang.ddframe.rdb.sharding.api.strategy.table.SingleKeyTableShardingAlgorithm; import com.google.common.collect.Range; import java.util.Collection; import java.util.LinkedHashSet; /** * 分表策略 */ public class ModuloTableShardingAlgorithm implements SingleKeyTableShardingAlgorithm<Integer> { /** * equals查詢 * * @param tableNames * @param shardingValue * @return */ @Override public String doEqualSharding(Collection<String> tableNames, ShardingValue<Integer> shardingValue) { for (String each : tableNames) { if (each.endsWith(shardingValue.getValue() % 3 + "")) { return each; } } throw new IllegalArgumentException(); } /** * in查詢 * * @param tableNames * @param shardingValue * @return */ @Override public Collection<String> doInSharding(Collection<String> tableNames, ShardingValue<Integer> shardingValue) { Collection<String> result = new LinkedHashSet<>(tableNames.size()); for (Integer value : shardingValue.getValues()) { for (String tableName : tableNames) { if (tableName.endsWith(value % 3 + "")) { result.add(tableName); } } } return result; } /** * between查詢 * * @param tableNames * @param shardingValue * @return */ @Override public Collection<String> doBetweenSharding(Collection<String> tableNames, ShardingValue<Integer> shardingValue) { Collection<String> result = new LinkedHashSet<>(tableNames.size()); Range<Integer> range = (Range<Integer>) shardingValue.getValueRange(); for (Integer i = range.lowerEndpoint(); i <= range.upperEndpoint(); i++) { for (String each : tableNames) { if (each.endsWith(i % 3 + "")) { result.add(each); } } } return result; } }
4、分布式主鍵生成策略
不一定非要使用Snowflake算法為主鍵,你也可以自己實現,然后實現KeyGenerator接口。
package com.gouggou.config; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang3.RandomUtils; import org.apache.commons.lang3.StringUtils; import org.apache.commons.lang3.SystemUtils; import java.net.Inet4Address; import java.net.UnknownHostException; /** * 分布式數據庫主鍵生成策略 */ @Slf4j public class SnowflakeIdWorker { /** * 開始時間截 (2015-01-01) */ private final long twepoch = 1489111610226L; /** * 機器id所占的位數 */ private final long workerIdBits = 5L; /** * 數據標識id所占的位數 */ private final long dataCenterIdBits = 5L; /** * 支持的最大機器id,結果是31 (這個移位算法可以很快的計算出幾位二進制數所能表示的最大十進制數) */ private final long maxWorkerId = -1L ^ (-1L << workerIdBits); /** * 支持的最大數據標識id,結果是31 */ private final long maxDataCenterId = -1L ^ (-1L << dataCenterIdBits); /** * 序列在id中占的位數 */ private final long sequenceBits = 12L; /** * 機器ID向左移12位 */ private final long workerIdShift = sequenceBits; /** * 數據標識id向左移17位(12+5) */ private final long dataCenterIdShift = sequenceBits + workerIdBits; /** * 時間截向左移22位(5+5+12) */ private final long timestampLeftShift = sequenceBits + workerIdBits + dataCenterIdBits; /** * 生成序列的掩碼,這里為4095 (0b111111111111=0xfff=4095) */ private final long sequenceMask = -1L ^ (-1L << sequenceBits); /** * 工作機器ID(0~31) */ private long workerId; /** * 數據中心ID(0~31) */ private long dataCenterId; /** * 毫秒內序列(0~4095) */ private long sequence = 0L; /** * 上次生成ID的時間截 */ private long lastTimestamp = -1L; private static SnowflakeIdWorker idWorker; static { idWorker = new SnowflakeIdWorker(getWorkId(), getDataCenterId()); } //==============================Constructors===================================== /** * 構造函數 * * @param workerId 工作ID (0~31) * @param dataCenterId 數據中心ID (0~31) */ public SnowflakeIdWorker(long workerId, long dataCenterId) { if (workerId > maxWorkerId || workerId < 0) { throw new IllegalArgumentException(String.format("workerId can't be greater than %d or less than 0", maxWorkerId)); } if (dataCenterId > maxDataCenterId || dataCenterId < 0) { throw new IllegalArgumentException(String.format("dataCenterId can't be greater than %d or less than 0", maxDataCenterId)); } this.workerId = workerId; this.dataCenterId = dataCenterId; } // ==============================Methods========================================== /** * 獲得下一個ID (該方法是線程安全的) * * @return SnowflakeId */ public synchronized long nextId() { long timestamp = timeGen(); //如果當前時間小於上一次ID生成的時間戳,說明系統時鍾回退過這個時候應當拋出異常 if (timestamp < lastTimestamp) { throw new RuntimeException( String.format("Clock moved backwards. Refusing to generate id for %d milliseconds", lastTimestamp - timestamp)); } //如果是同一時間生成的,則進行毫秒內序列 if (lastTimestamp == timestamp) { sequence = (sequence + 1) & sequenceMask; //毫秒內序列溢出 if (sequence == 0) { //阻塞到下一個毫秒,獲得新的時間戳 timestamp = tilNextMillis(lastTimestamp); } } //時間戳改變,毫秒內序列重置 else { sequence = 0L; } //上次生成ID的時間截 lastTimestamp = timestamp; //移位並通過或運算拼到一起組成64位的ID return ((timestamp - twepoch) << timestampLeftShift) | (dataCenterId << dataCenterIdShift) | (workerId << workerIdShift) | sequence; } /** * 阻塞到下一個毫秒,直到獲得新的時間戳 * * @param lastTimestamp 上次生成ID的時間截 * @return 當前時間戳 */ protected long tilNextMillis(long lastTimestamp) { long timestamp = timeGen(); while (timestamp <= lastTimestamp) { timestamp = timeGen(); } return timestamp; } /** * 返回以毫秒為單位的當前時間 * * @return 當前時間(毫秒) */ protected long timeGen() { return System.currentTimeMillis(); } private static Long getWorkId() { try { String hostAddress = Inet4Address.getLocalHost().getHostAddress(); int[] ints = StringUtils.toCodePoints(hostAddress); int sums = 0; for (int b : ints) { sums += b; } return (long) (sums % 32); } catch (UnknownHostException e) { // 如果獲取失敗,則使用隨機數備用 return RandomUtils.nextLong(0, 31); } } private static Long getDataCenterId() { int[] ints = StringUtils.toCodePoints(SystemUtils.getHostName()); int sums = 0; for (int i : ints) { sums += i; } return (long) (sums % 32); } /** * 靜態工具類 * * @return */ public static synchronized Long generateId() { return idWorker.nextId(); } }
package com.gouggou.config; import com.dangdang.ddframe.rdb.sharding.keygen.KeyGenerator; public class DefaultKeyGenerator implements KeyGenerator { @Override public Number generateKey() { return SnowflakeIdWorker.generateId(); } }
八、controller
package com.gouggou.controller; import com.gouggou.config.DefaultKeyGenerator; import com.gouggou.entity.User; import com.gouggou.service.UserService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RestController; import java.util.ArrayList; import java.util.Arrays; import java.util.List; import java.util.Random; @RequestMapping("user") @RestController public class UserController { @Autowired private UserService userService; @Autowired private DefaultKeyGenerator defaultKeyGenerator; @RequestMapping("save") public String save() { User user = null; for (int i = 0; i < 60; i++) { user = new User(); user.setId(defaultKeyGenerator.generateKey().longValue()); user.setUserId(new Random().nextInt(1000) + 1); user.setName("張三" + user.getUserId()); user.setAge(new Random().nextInt(80) + 1); userService.insert(user); } return "創建成功!"; } @RequestMapping("findById") public List<User> findById(Integer id){ List<Integer> list = new ArrayList<>(1); list.add(id); return userService.findByUserIds(list); } }
九、service
package com.gouggou.service; import com.gouggou.entity.User; import java.util.List; public interface UserService { Integer insert(User u); List<User> findByUserIds(List<Integer> userIds); }
package com.gouggou.service.impl; import com.gouggou.entity.User; import com.gouggou.mapper.UserMapper; import com.gouggou.service.UserService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; import java.util.List; @Service @Transactional public class UserServiceImpl implements UserService { @Autowired private UserMapper userMapper; @Override public Integer insert(User u) { return userMapper.insert(u); } @Override public List<User> findByUserIds(List<Integer> userIds) { return userMapper.findByUserIds(userIds); } }
十、entity
package com.gouggou.entity; import lombok.Data; import java.io.Serializable; @Data public class User implements Serializable { private static final long serialVersionUID = -5514139686858156155L; private Long id; private Integer userId; private String name; private Integer age; }
十一、mapper
package com.gouggou.mapper; import com.gouggou.entity.User; import org.springframework.stereotype.Repository; import java.util.List; @Repository public interface UserMapper { Integer insert(User u); List<User> findByUserIds(List<Integer> userIds); }
<?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd" > <mapper namespace="com.gouggou.mapper.UserMapper"> <resultMap id="resultMap" type="com.gouggou.entity.User"> <id column="id" property="id" jdbcType="INTEGER"/> <result column="user_id" property="userId" jdbcType="INTEGER"/> <result column="name" property="name" jdbcType="VARCHAR"/> <result column="age" property="age" jdbcType="INTEGER"/> </resultMap> <insert id="insert"> insert into t_user (id,user_id,name,age) values (#{id},#{userId},#{name},#{age}) </insert> <select id="findByUserIds" resultMap="resultMap"> select <include refid="columnsName"/> from t_user where user_id in ( <foreach collection="list" item="item" separator=","> #{item} </foreach> ) </select> <sql id="columnsName"> id,user_id,name,age </sql> </mapper>
十二、總結
1、我使用的分庫分表依賴是com.dangdang.sharding-jdbc-core。但是后來當當把它捐給Apache,依賴變為org.apache.shardingsphere.sharding-jdbc-core,代碼可能會有所不同,但是原理是相通的;
2、SpringBoot的配置文件有兩種格式,一個是properties,一個是yml。這里采用的properties,yml格式的配置在探索中;
3、這里我從網上找的SnowFlake算法生成的id都是偶數,本來我打算使用id作為分庫策略的,后來沒辦法就用了age做分庫策略;
4、以后再補充讀寫分離的功能;