JDBC連接clickhouse cluster

JDBC連接clickhouse cluster

兩種方式，一種是使用clickhouse-jdbc連接集群中的每一個節點，另外一種是使用SLB提供一個對外的統一地址

使用BalancedClickhouseDataSource

參考以下clickhouse-jdbc中的代碼中的注釋：jdbc:clickhouse://localhost:8123,localhost:8123/database?compress=1&decompress=2

• clickhouse-jdbc/BalancedClickhouseDataSource.java at master · yandex/clickhouse-jdbc · GitHub

/**
     * create Datasource for clickhouse JDBC connections
     *
     * @param url address for connection to the database
     *            must have the next format {@code jdbc:clickhouse://<first-host>:<port>,<second-host>:<port>/<database>?param1=value1&param2=value2 }
     *            for example, {@code jdbc:clickhouse://localhost:8123,localhost:8123/database?compress=1&decompress=2 }
     * @throws IllegalArgumentException if param have not correct format, or error happens when checking host availability
     */
    public BalancedClickhouseDataSource(final String url) { this(splitUrl(url), getFromUrl(url)); } /** * create Datasource for clickhouse JDBC connections * * @param url address for connection to the database * @param properties database properties * @see #BalancedClickhouseDataSource(String) */ public BalancedClickhouseDataSource(final String url, Properties properties) { this(splitUrl(url), new ClickHouseProperties(properties)); } /** * create Datasource for clickhouse JDBC connections * * @param url address for connection to the database * @param properties database properties * @see #BalancedClickhouseDataSource(String) */ public BalancedClickhouseDataSource(final String url, ClickHouseProperties properties) { this(splitUrl(url), properties.merge(getFromUrlWithoutDefault(url))); }

使用SLB

使用LB均衡到各個副本，保證應用方查詢單host，本次不使用BalancedClickhouseDataSource，從github issue上看BalancedClickhouseDataSource在之前版本出現副本故障時沒能故障轉移，不知道是否有修復。

配置LB：使用標准JDBC連接時需要映射http協議到clickhouse的8123端口(http監聽端口)

驗證LB配置是否生效

echo 'SELECT * from dm.delphi_membership_properties FORMAT Pretty' | curl 'internal-clickhouse-prod-621097858. cn-north-1.elb.amazonaws.com.cn:80/?' --data-binary @-

貼一下在Springboot中使用標准JDBC數據源HikariDataSource÷連接clickhouse的配置：

package com.kezaihui.delphi.core.config; import com.baomidou.mybatisplus.entity.GlobalConfiguration; import com.baomidou.mybatisplus.enums.DBType; import com.baomidou.mybatisplus.enums.IdType; import com.baomidou.mybatisplus.spring.MybatisSqlSessionFactoryBean; import com.zaxxer.hikari.HikariDataSource; import lombok.extern.slf4j.Slf4j; import org.apache.ibatis.session.SqlSessionFactory; import org.mybatis.spring.annotation.MapperScan; import org.mybatis.spring.boot.autoconfigure.MybatisProperties; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.io.support.PathMatchingResourcePatternResolver; import org.springframework.core.io.support.ResourcePatternResolver; import javax.sql.DataSource; /** * clickhouse 數據源配置 * * @author Suncle * @date 2019-07-05 */ @Slf4j @Configuration @MapperScan(basePackages = { "com.kezaihui.delphi.core.membership.**.mapper" }, sqlSessionFactoryRef = "ckSqlSessionFactory") public class CkDataSourceConfig { @Autowired private MybatisProperties mybatisProperties; /** * 讀取數據源 * * @return javax.sql.DataSource 數據源 */ @Bean(name = "ckDataSource") @ConfigurationProperties(prefix = "spring.clickhouse.datasource") public DataSource dataSource() { return new HikariDataSource(); } /** * sql 會話工廠配置 * * @param ckDataSource javax.sql.DataSource 數據源 * @return SqlSessionFactory */ @Bean(name = "ckSqlSessionFactory") public SqlSessionFactory sqlSessionFactory(@Qualifier("ckDataSource") DataSource ckDataSource) { MybatisSqlSessionFactoryBean bean = new MybatisSqlSessionFactoryBean(); bean.setDataSource(ckDataSource); try { GlobalConfiguration configuration = new GlobalConfiguration(); configuration.setDbType(DBType.OTHER.name()); configuration.setIdType(IdType.AUTO.getKey()); configuration.setDbColumnUnderline(true); bean.setGlobalConfig(configuration); ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(); bean.setMapperLocations(mybatisProperties.resolveMapperLocations()); bean.setConfigLocation(resolver.getResource(mybatisProperties.getConfigLocation())); return bean.getObject(); } catch (Exception e) { throw new RuntimeException(e); } } }

配置好之后可以使用mybatis查詢數據

原生TCP協議連接clickhouse cluster

同樣是采用SLB映射到不同的節點上，但是映射的時候需要注意不同之處：

• 映射TCP協議到clickhouse的9000端口

連接方式可以參考clickhouse-driver的連接，也可以直接使用python語言clickhouse-driver庫

clickhouse python client的選擇

官方沒有維護各語言的driver，全部由第三方維護，主要有以下兩個，對比參見后面。

結論：選擇clickhouse-driver，數倉項目使用orm的意義不大。因為應用層不是python項目，是java項目

clickhouse-driver

GitHub - mymarilyn/clickhouse-driver: ClickHouse Python Driver with native interface support 活躍度高，star數最高。語法主要是執行原生sql

infi.clickhouse_orm

GitHub - Infinidat/infi.clickhouse_orm: A Python library for working with the ClickHouse database (https://clickhouse.yandex/) 活躍度高，star數第二高。是一個為clickhouse封裝的orm框架，寫起來有django的感覺

參考：

• https://github.com/jneo8/clickhouse-setup
• https://clickhouse.yandex/docs/zh/operations/table_engines/replication/
• https://clickhouse.yandex/docs/zh/operations/table_engines/distributed/
• https://clickhouse.yandex/docs/en/operations/table_engines/replication/
• https://clickhouse.yandex/tutorial.html
• https://hzkeung.com/2018/06/30/clickhouse-cluster-test
• https://hzkeung.com/2018/06/21/clickhouse-cluster-install
• https://www.jianshu.com/p/383cae967a64