MySQL-8.0.19 优化日志及压测

概要

MySQL-8.0.19 发布的第一时间 dbm 就提供了支持，那 dbm 自动化安装的 MySQL 实例在性能上是一个怎样的表现呢？我的测试环境如下。

IP	CPU	Mem	Disk	系统版本	MySQL版本	角色
192.168.100.10	32(逻辑核心)	128G	4TSSD	centos-7.6	MySQL-8.0.19	Master
192.168.100.20	32(逻辑核心)	128G	4TSSD	centos-7.6	MySQL-8.0.19	Slave

---

dbm-agent 在安装 MySQL 时会根据主机配置自动的完成参数优化，默认情况下的性能表现如下图。

---

安装

安装 Master。

dbma-cli-single-instance --port=3306 --max-mem=131072 install

安装 Slave。

dbma-cli-build-slave --host=192.168.100.10 --port=3306 --max-mem=131072 build-slave

验证同步是否正常。

mysql> show slave status \G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event Master_Host: 192.168.100.10 Master_User: repluser Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000051 Read_Master_Log_Pos: 88792484 Relay_Log_File: relay.000078 Relay_Log_Pos: 433520819 Relay_Master_Log_File: mysql-bin.000027 Slave_IO_Running: Yes Slave_SQL_Running: Yes 

dbm-agent 自动生成的配置文件如下。

[mysqld] ####for basic user = mysql3306 basedir = /usr/local/mysql-8.0.19-linux-glibc2.12-x86_64 datadir = /database/mysql/data/3306 server_id = 1794 port = 3306 bind_address = * admin_address = 127.0.0.1 mysqlx_port = 33060 admin_port = 33062 socket = /tmp/mysql-3306.sock mysqlx_socket = /tmp/mysqlx-33060.sock pid_file = /tmp/mysql-3306.pid character_set_server = utf8mb4 open_files_limit = 102000 max_prepared_stmt_count = 1048576 skip_name_resolve = 1 super_read_only = OFF log_timestamps = system event_scheduler = OFF auto_generate_certs = ON activate_all_roles_on_login = ON end_markers_in_json = OFF tmpdir = /tmp/ max_connections = 1024 autocommit = ON sort_buffer_size = 256K join_buffer_size = 256K eq_range_index_dive_limit = 200 ####for table big_tables = OFF sql_require_primary_key = OFF lower_case_table_names = 1 auto_increment_increment = 1 auto_increment_offset = 1 table_open_cache = 4000 table_definition_cache = 2000 table_open_cache_instances = 32 ####for net max_allowed_packet = 1G connect_timeout = 10 interactive_timeout = 28800 net_read_timeout = 30 net_retry_count = 10 net_write_timeout = 60 net_buffer_length = 32K ####for logs log_output = FILE general_log = OFF general_log_file = general.log ##error log_error = err.log log_statements_unsafe_for_binlog = ON ##slow slow_query_log = ON slow_query_log_file = slow.log long_query_time = 2 log_queries_not_using_indexes = OFF log_slow_admin_statements = OFF log_slow_slave_statements = OFF ##binlog log_bin = /binlog/mysql/binlog/3306/mysql-bin binlog_checksum = none log_bin_trust_function_creators = ON binlog_direct_non_transactional_updates = OFF binlog_expire_logs_seconds = 604800 binlog_error_action = ABORT_SERVER binlog_format = ROW max_binlog_stmt_cache_size = 1G max_binlog_cache_size = 1G max_binlog_size = 1G binlog_order_commits = ON binlog_row_image = FULL binlog_row_metadata = MINIMAL binlog_rows_query_log_events = ON binlog_stmt_cache_size = 32K log_slave_updates = ON binlog_transaction_dependency_history_size =25000 binlog_transaction_dependency_tracking = WRITESET sync_binlog = 1 binlog_cache_size = 96K binlog_group_commit_sync_delay = 0 binlog_group_commit_sync_no_delay_count = 0 ####for replication rpl_semi_sync_master_enabled = 1 rpl_semi_sync_slave_enabled = 1 rpl_semi_sync_master_timeout = 1000 rpl_semi_sync_master_wait_point = AFTER_SYNC rpl_semi_sync_master_wait_no_slave = ON rpl_semi_sync_master_wait_for_slave_count = 1 master_info_repository = table sync_master_info = 10000 skip_slave_start = OFF slave_load_tmpdir = /tmp/ plugin_load_add = semisync_master.so plugin_load_add = semisync_slave.so relay_log = relay sync_relay_log = 10000 sync_relay_log_info = 10000 relay_log_info_repository = table slave_preserve_commit_order = ON slave_parallel_type = logical_clock slave_parallel_workers = 2 slave_max_allowed_packet = 1G ####for gtid gtid_mode = ON binlog_gtid_simple_recovery = ON enforce_gtid_consistency = ON gtid_executed_compression_period = 1000 ####for clone plugin-load-add = mysql_clone.so clone = FORCE_PLUS_PERMANEN ####for engines default_storage_engine = innodb default_tmp_storage_engine = innodb internal_tmp_mem_storage_engine = TempTable ####for innodb innodb_data_home_dir = ./ innodb_data_file_path = ibdata1:64M:autoextend innodb_page_size = 16K innodb_default_row_format = dynamic innodb_log_group_home_dir = ./ innodb_redo_log_encrypt = OFF innodb_online_alter_log_max_size = 256M innodb_undo_directory = ./ innodb_undo_log_encrypt = OFF innodb_undo_log_truncate = ON innodb_max_undo_log_size = 1G innodb_rollback_on_timeout = OFF innodb_rollback_segments = 128 innodb_log_checksums = ON innodb_checksum_algorithm = crc32 innodb_log_compressed_pages = ON innodb_doublewrite = ON innodb_commit_concurrency = 0 innodb_read_only = OFF innodb_dedicated_server = OFF innodb_old_blocks_pct = 37 innodb_old_blocks_time = 1000 innodb_random_read_ahead = OFF innodb_read_ahead_threshold = 56 innodb_max_dirty_pages_pct_lwm = 20 innodb_max_dirty_pages_pct = 90 innodb_lru_scan_depth = 1024 innodb_adaptive_flushing = ON innodb_adaptive_flushing_lwm = 10 innodb_flushing_avg_loops = 30 innodb_buffer_pool_dump_pct = 50 innodb_buffer_pool_dump_at_shutdown = ON innodb_buffer_pool_load_at_startup = ON innodb_buffer_pool_filename = ib_buffer_pool innodb_stats_persistent = ON innodb_stats_on_metadata = ON innodb_stats_method = nulls_equal innodb_stats_auto_recalc = ON innodb_stats_include_delete_marked = ON innodb_stats_persistent_sample_pages = 20 innodb_stats_transient_sample_pages = 8 innodb_status_output = OFF innodb_status_output_locks = OFF innodb_buffer_pool_dump_now = OFF innodb_buffer_pool_load_abort = OFF innodb_buffer_pool_load_now = OFF innodb_thread_concurrency = 0 innodb_concurrency_tickets = 5000 innodb_thread_sleep_delay = 15000 innodb_adaptive_max_sleep_delay = 150000 innodb_read_io_threads = 8 innodb_write_io_threads = 4 innodb_use_native_aio = ON innodb_flush_sync = OFF innodb_spin_wait_delay = 6 innodb_purge_threads = 6 innodb_purge_batch_size = 300 innodb_purge_rseg_truncate_frequency = 128 innodb_deadlock_detect = ON innodb_print_all_deadlocks = ON innodb_lock_wait_timeout = 50 innodb_table_locks = ON innodb_sync_array_size = 1 innodb_sync_spin_loops = 30 innodb_print_ddl_logs = OFF innodb_replication_delay = 0 innodb_cmp_per_index_enabled = OFF innodb_disable_sort_file_cache = OFF innodb_numa_interleave = OFF innodb_strict_mode = ON innodb_sort_buffer_size = 1M innodb_fast_shutdown = 1 innodb_force_load_corrupted = OFF innodb_force_recovery = 0 innodb_temp_tablespaces_dir = ./#innodb_temp/ innodb_tmpdir = ./ innodb_temp_data_file_path = ibtmp1:64M:autoextend innodb_page_cleaners = 4 innodb_adaptive_hash_index = ON innodb_adaptive_hash_index_parts = 8 innodb_flush_log_at_timeout = 1 innodb_fsync_threshold = 0 innodb_fill_factor = 90 innodb_file_per_table = ON innodb_autoextend_increment = 64 innodb_open_files = 100000 innodb_buffer_pool_chunk_size = 128M innodb_buffer_pool_instances = 32 innodb_log_files_in_group = 8 innodb_log_file_size = 256M innodb_flush_neighbors = 0 innodb_io_capacity = 4000 innodb_io_capacity_max = 20000 innodb_autoinc_lock_mode = 2 innodb_change_buffer_max_size = 25 innodb_flush_method = O_DIRECT innodb_log_buffer_size = 2G innodb_flush_log_at_trx_commit = 1 innodb_buffer_pool_size = 96G ####for performance_schema performance_schema =ON performance_schema_consumer_global_instrumentation =ON performance_schema_consumer_thread_instrumentation =ON performance_schema_consumer_events_stages_current =ON performance_schema_consumer_events_stages_history =ON performance_schema_consumer_events_stages_history_long =OFF performance_schema_consumer_statements_digest =ON performance_schema_consumer_events_statements_current =ON performance_schema_consumer_events_statements_history =ON performance_schema_consumer_events_statements_history_long =OFF performance_schema_consumer_events_waits_current =ON performance_schema_consumer_events_waits_history =ON performance_schema_consumer_events_waits_history_long =OFF performance-schema-instrument ='memory/%=COUNTED' # -- ~ _ ~ ~ _ ~ ~ _ ~ --{@class=h5 text-secondary mb-4} # base on mysql-8.0.19{@class=h5 text-secondary mb-4} # generated by https://www.sqlpy.com 2020年1月15日 10:32{@class=h5 text-secondary mb-4} # wechat: jianglegege{@class=h5 text-secondary mb-4} # email: [e-mail]{@class=h5 text-secondary mb-4} # -- ~ _ ~ --{@class=h5 text-secondary mb-4} 

---

压测

用 sysbench 对 MySQL 进行压力测试。

准备数据。

sysbench --mysql-host=192.168.100.10  --mysql-port=3306 \
--mysql-user=sysbench --mysql-password=sysbench --tables=120 --table_size=1000000 \
--mysql-db=tempdb --time=3600 --threads=120 \
oltp_read_write prepare

性能压测。

sysbench --mysql-host=192.168.100.10  --mysql-port=3306 \
--mysql-user=sysbench --mysql-password=sysbench --tables=120 --table_size=1000000 \
--mysql-db=tempdb --time=3600 --threads=120 \
oltp_read_write run

---

MySQL 的性能表现。

操作系统层面的资源消耗情况。

从资源使用情况来看 120 个并发并没有把主机的性能完全利用起来，接下来准备一步步加大并发。

---

150并发

把并发加大到 150。

sysbench --mysql-host=192.168.100.10  --mysql-port=3306 \
--mysql-user=sysbench --mysql-password=sysbench --tables=120 --table_size=1000000 \
--mysql-db=tempdb --time=3600 --threads=150 \
oltp_read_write run

提高并发后 select 由之前的 46k 提升到 57k，其它类型的处理能力也得到了提升。

资源使用率上升。

总体上看负载还行， iowait 也比较小，可以继续加大并发。

---

180并发

把并发加大到 180。

sysbench --mysql-host=192.168.100.10  --mysql-port=3306 \
--mysql-user=sysbench --mysql-password=sysbench --tables=120 --table_size=1000000 \
--mysql-db=tempdb --time=3600 --threads=180 \
oltp_read_write run

提高并发后 select 由之前的 57k 提升到 67k，其它类型的处理能力也得到了提升。

资源使用率上升到 70%。

---

210并发

把并发加大到 210。

sysbench --mysql-host=192.168.100.10  --mysql-port=3306 \
--mysql-user=sysbench --mysql-password=sysbench --tables=120 --table_size=1000000 \
--mysql-db=tempdb --time=3600 --threads=210 \
oltp_read_write run

提高并发后 select 由之前的 67k 提升到 74k，其它类型的处理能力也得到了提升。

资源使用率上升到 80%。

---

240并发

把并发加大到 240。

sysbench --mysql-host=192.168.100.10  --mysql-port=3306 \
--mysql-user=sysbench --mysql-password=sysbench --tables=120 --table_size=1000000 \
--mysql-db=tempdb --time=3600 --threads=240 \
oltp_read_write run

提高并发后 select 由之前的 74k 提升到 79k，其它类型的处理能力也得到了提升。

资源使用率上升到 85%。

---

270并发

把并发加大到 270。

sysbench --mysql-host=192.168.100.10  --mysql-port=3306 \
--mysql-user=sysbench --mysql-password=sysbench --tables=120 --table_size=1000000 \
--mysql-db=tempdb --time=3600 --threads=270 \
oltp_read_write run

当并发达到 270 的时候性能已经开始下降。

资源使用率还是 85%。

---

300并发

把并发加大到 300。

sysbench --mysql-host=192.168.100.10  --mysql-port=3306 \
--mysql-user=sysbench --mysql-password=sysbench --tables=120 --table_size=1000000 \
--mysql-db=tempdb --time=3600 --threads=300 \
oltp_read_write run

当并发达到 300 的时候性能还在下降。

同时资源使用率也稳定在 85% 左右不再下降。 

---

结论

dbm-agent 自动生成的 my.cnf 文件还是比较机智的，不怎么用 DBA 操心，整体上并发与性能的关系如下。