database management-phase 252 in-depth introduction of multi-master and Multi-Active database technology-Cantian storage engine (II)(20241017)

Ace 2024-10-25 16:37 published in China

Database management-issue 252 deep introduction of multi-master and Multi-Active database technology-Cantian storage engine (II)(20241017)

author: fat head Fish Tank (Yin Haiwen)
Oracle ACE Pro: Database(Oracle and MySQL)
PostgreSQL ACE Partner
10 years of experience in the database industry, now mainly engaged in database services
with OCM 11g/12c/19c, MySQL 8.0 OCP, Exadata, CDP and other certifications
MVP of motianlun, star of mojili of the year, ITPUB certified expert, member of expert group of 100, OCM lecturer, technical consultant of PolarDB open source community, external technical consultant of HaloDB, member of OceanBase observer mission, technical Adviser, MOP technical community of youth society (youth database learning Mutual Aid Association)
the circle has the titles of "director", "security guard", "the biggest enemy of domestic database" and so on. It is not a famous social terrorist (social terrorist)
public number: fish tank of fat head fish; CSDN: fish tank of fat head fish (Yin Haiwen); Mo Tianlun: fish tank of fat head fish; ITPUB:yhw1809.
Except for authorized reprinting and indicating the source, all are "illegal" plagiarism.

This topic describes how to deploy the Cantian storage engine.

1 deployment planning

this deployment uses the latest Cantian 24.06. The Deployment architecture is as follows:

the server configuration is as follows:

the configuration of the shared disk is as follows:
use nfs to share data.

Cantian Deployment flowchart:

2. Basic server configuration

the following configurations must be performed on all servers:

2.1 configure HOSTS

  cat >> /etc/hosts <<EOF 10.10.10.111 ct01 10.10.10.112 ct02 10.10.11.110 mgr 10.10.11.111 ct01-mgr 10.10.11.112 ct02-mgr 20.20.20.111 ct01-priv 20.20.20.112 ct02-priv 30.30.30.113 ctst 30.30.30.111 ct01-san 30.30.30.112 ct02-san EOF

2.2 close the firewall

  systemctl stop firewalld.service systemctl disable firewalld.service

2.3 disable SELinux

  sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config setenforce 0

2.4 configure the yum source

  rm -rf /etc/yum.repos.d/* wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-8.repo dnf clean all dnf repolist

3 compile server configuration

the server used for compilation can be mgr or ct01/ct02. The mgr server is not deployed due to resource problems. The compilation operation is performed on ct01. in actual production, we recommend that you use a separate server. Some of these operations must be performed on all servers.

3.1 install git

 # git version> = 2.18.0 dnf -- showduplicates list git# Check the git version in the repository, which is 2.27.0, to meet the requirements dnf -y install git

3.2 install docker

 # docker version> = 19.03 # 1 is required. Load the br_netfilter kernel modprobe br_netfilter cat <<EOF | sudo tee /etc/modules-load.d/netfilter.conf br_netfilter EOF lsmod | grep br_netfilter # 2. Configure sysctl cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward = 1 net.ipv4.conf.all.proxy_arp = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.netfilter. Nf_conntrack_max = 2310720 vm.swappiness = 0 net.ipv6.conf.all. Disable_ipv6 = 1 net.ipv6.conf.default. Disable_ipv6 = 1 net.ipv6.conf.lo. Disable_ipv6 = 1 EOF sysctl -p #3. Configure the docker yum source dnf install -y yum-utils dnf-utils dnf config-manager -- add-repo = https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo dnf -- showduplicates list docker-ce# Check the docker version in the repository. The minimum version is 19.03.13, which meets the requirements #4. install docker-ce dnf -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -- allowerasing # 5. Configure docker to start systemctl enable -- now docker # 6. Check docker version docker -- version [root @ ct01 ~]# docker -- version Docker version 26.1.3, build b72abbb

3.3 install container dependent software

 # cmake 3.14.1,automake 1.16.1,libtool 2.4.6,g 8.5.0,pkgconfig 0.29.1-3,rpm-build 4.14.3, the software versions in the repository all meet the conditions dnf -y install cmake automake libaio-devel rpm-build libtool pkgconf-pkg-config -- allowerasing

4. Configure shared storage servers

NFS is used to implement storage sharing.

4.1 install NFS

  dnf install nfs-utils

4.2 create a Mount Directory

  mkdir /ctst mkdir /ctst/dbstore mkdir /ctst/metadata mkdir /ctst/share mkdir /ctst/archive

4.3 process and mount disks

here, a 200g disk is used for demonstration:

  pvcreate /dev/nvme0n2 vgcreate ctst /dev/nvme0n2 lvcreate -L 150G -n dbstore ctst lvcreate -L 5G -n metadata ctst lvcreate -L 2G -n share ctst lvcreate -L 10G -n archive ctst mkfs.xfs /dev/mapper/ctst-archive mkfs.xfs /dev/mapper/ctst-dbstore mkfs.xfs /dev/mapper/ctst-metadata mkfs.xfs /dev/mapper/ctst-share cat >> /etc/fstab << EOF /dev/mapper/ctst-dbstore /ctst/dbstore xfs defaults 0 0 /dev/mapper/ctst-metadata /ctst/metadata xfs defaults 0 0 /dev/mapper/ctst-share /ctst/share xfs defaults 0 0 /dev/mapper/ctst-archive /ctst/archive xfs defaults 0 0 EOF mount /ctst/dbstore mount /ctst/metadata mount /ctst/share mount /ctst/archive

4.4 configure NFS

  cat >> /etc/exports << EOF /ctst 30.30.30.0/24(rw,sync,no_subtree_check,crossmnt,fsid=0,no_root_squash) /ctst/dbstore 30.30.30.0/24(rw,sync,no_subtree_check,no_root_squash) /ctst/metadata 30.30.30.0/24(rw,sync,no_subtree_check,no_root_squash) /ctst/share 30.30.30.0/24(rw,sync,no_subtree_check,no_root_squash) /ctst/archive 30.30.30.0/24(rw,sync,no_subtree_check,no_root_squash) EOF

configuration description: The fsid = 0 configuration in the first row can be mounted with/ctst as the root directory. Enter only the name of the secondary directory in this directory.

4.5 start NFS

  systemctl start nfs-server systemctl enable nfs-server

note: You do not need to mount disks on ct01 and ct02.

5 compile Cantian

5.1 download source code

  mkdir /ctdb/cantian_compile cd /ctdb/cantian_compile git clone -b cantian24.6 https://gitee.com/openeuler/cantian.git git clone -b cantian24.6 https://gitee.com/openeuler/cantian-connector-mysql.git

5.2 download MySQL source code package

  cd /ctdb/cantian_compile/cantian-connector-mysql# Copy the 8.0.26 source code package and mysql-server-mysql-8.0.26.tar.gz it to the directory tar -zxf mysql-server-mysql-8.0.26.tar.gz mv mysql-server-mysql-8.0.26 mysql-source

5.3 Add image repository acceleration

  cat > /etc/docker/daemon.json <<EOF { "registry-mirrors": [ "https://dockerpull.com", "https://hub.uuuadc.top", "https://docker.anyhub.us.kg", "https://dockerhub.jobcher.com", "https://dockerhub.icu", "https://docker.ckyl.me", "https://docker.awsl9527.cn", "https://y8y6vosv.mirror.aliyuncs.com", "https://registry.dockercn.com", "https://docker.mirrors.ustc.edu.cn", "https://dockerhub.azk8s.cn", "http://hubmirror.c.163.com" ] } EOF systemctl reload-daemon systelctl restart docker

5.4 build an image

  cd /ctdb/cantian_compile/cantian/docker# The Dockfile content vim needs to be adjusted first./Dockfile #RUN wget -- progress = bar:force -O /tmp/libasan5-8.2.1-3.el7.x86_64.rpm http://mirror.centos.org/centos/7/sclo/x86_64/rh/Packages/l/libasan5-8.2.1-3.el7.x86_64.rpm# The original content. The corresponding rpm Package address has expired. RUN wget -- progress = bar:force -O /tmp/libasan5-8.2.1-3.el7.x86_64.rpm https://vault.centos.org/7.9.2009/sclo/x86_64/rh/Packages/l/libasan5-8.2.1-3.el7.x86_64.rpm docker build -t cantian_dev:latest -f Dockerfile . docker images

5.5 compile source code

#1. Start the container cd /ctdb/cantian_compile/cantian/docker sh container.sh dev# enter the container
 #2. Go to the compilation script directory cd /home/regress/CantianKernel/build
 #3. Generate the Cantian engine software package sh build_cantian.sh release

 #4. Check the generated file ls /tmp/cantian_output

5.6 close the container

  docker ps docker stop [CONTAINER ID]

 # (Optional) clean up image and cache docker images docker rmi [IMAGE ID] docker system prune

6 deploy Cantian

6.1 check CPU support

  cat /proc/cpuinfo | grep -o constant_tsc | uniq

the content to be returned is constant_tsc to ensure the time accuracy of the database server.

6.2 root directory check

make sure that the root directory has at least 20GB of space left.

6.3 Python check

the Python version must be greater than or equal to 3.6.0.

Check whether pyopenssl and cryptography are installed

  pip3 display password pip3 display pyopenssl# if not installed, run the following command to install pip3 install pyopenssl -- trusted-hos t = mirrors.huaweicloud.com-I https://mirrors.huaweicloud.com/repository/pypi/simple/ pip3 install cryptography -- trusted-hos t = mirrors.huaweicloud.com-I https://mirrors.huaweicloud.com/repository/pypi/simple/

6.4检查/dev/shm权限

/dev/shm目录权限需要为drwxrwxrwt

 Ll-d/dev/shm # 如权限未正确，执行chmod 777 /dev/shm

6.5调整系统配置

 cat >> /etc/sysctl.conf <<EOF kernel.shmmax = 23192823399 EOFsysctl -p

6.创建用户

 用户add-m-u 5000 -d /home/ctdba ctdba

6.7创建安装目录

 mkdir /ctdb/cantian_install chmod 755 /ctdb/cantian_install chown ctdba:ctdba /ctdb/cantian_install光盘/ctdb/cantian_install mv /path/to/Cantian_24.06_x86_64_RELEASE.tgz ./ tar -zxvf Cantian_24.06_x86_64_RELEASE.tgz

Cantian6.8修改配置文件

修改配置文件/ctdb/cantian_install/cantian_connector/action/config_params_file.json

 {"deploy_mode":"nas","deploy_user":"ctdba:ctdba","node_id":"0","cms_ip":"20.20.20.111;20.20.20.112","storage_dbstore_fs":"dbstore"，"storage_share_fs":"共享"，"storage_archive_fs":"存档"，"storage_metadata_fs":"元数据"，"cantian_in_container":"0"，"mysql_in_container":"0","share_logic_ip":"30.30.30.113","archive_logic_ip":"30.30.30.113","metadata_logic_ip":"30.30.30.113","storage_logic_ip":"30.30.30.113","db_type":"1"，"mes_ssl_switch":false，"mysql_metadata_in_cantan":false，"MAX_ARCH_FILES_SIZE":"512"，"ca_path":""，"crt_path":"","key_path":"","redo_num":"3","redo_size":"1G"}

附:配置文件说明

6.9调整其他配置参数

在/ctdb/cantian_install/cantian_connector/action/cantian/ install_config.json配置中添加以下内容:

 "Z_KERNEL_PARAMETER3":"TEMP_BUFFER_SIZE = 1G","Z_KERNEL_PARAMETER4":"TEMP_POOL_NUM = 4","Z_KERNEL_PARAMETER5":"BUF_POOL_NUM = 16","Z_KERNEL_PARAMETER6":"LARGE_POOL_SIZE = 256M"，"Z_KERNEL_PARAMETER7":"DATA_BUFFER_SIZE = 8g"，"Z_KERNEL_PARAMETER8":"SHARED_POOL_SIZE = 512M"，"Z_KERNEL_PARAMETER9":"会话 = 512"

注:因为测试环境内存不足，需要调整MySQL相关内存参数。

6.10部署Cantian

 sh /ctdb/cantian_install/cantian_connector/action/appctl.sh install /ctdb/cantian_install/cantian_connector/action/config_params_file.json

这里在操作过程中出现了点小插曲、输出的检查日志的路径和实际的路径位置不一致:

设置cantiani管理员密码:

密码:Cantian_123

6.11节点2部署

在ct02节点重复6.5-6。10操作内容，注意配置文件 “node_id”: “1”。

注:如部署过程中(未完成)因为各种原因重启服务器后，需要执行以下命令启动自动挂载
systemctl启动悬臂
该服务自动挂载后续安装完成后将由服务cantian.timer接管

注:部署过程中会对操作系统相关配置进行调整，并创建用户cantian，用于运行Cantian存储引擎以及MySQL

6.12启动cantiani

在两个节点上分别执行以下命令:

 sh /opt/cantan/action/appctl.sh开始

Cantian6.13检查状态

 systemctl status cantian_logs_handler.timer su -s /bin/bash-cantiancms统计

注:如安装失败需要重新调整配置并部署，步骤如下:
1.关闭悬臂
sh /opt/cantian/action/appctl.sh stop
2.卸载悬臂
sh /opt/cantan/action/appctl.sh卸载覆盖
3.删除无效文件
# 所有计算节点
Rm-rf/mnt/dbdata/本地/*
# 存储节点
rm -rf /ctst/dbstore/*
rm -rf /ctst/metadata/*
rm-射频/ctst/共享/*
rm -rf /ctst/archive/*
4.重新部署
在所有调整完成后重新部署悬臂

7配置MySQL

7.1下载并上传MySQL源码包

这里需要注意，需要使用和前面编译过程中一样的8.0。26版本，需要使用编译安装，否则在后续加载坎蒂安相关插件过程中会出现异常。

 mysql-server-mysql-8.0.26.tar.gz

7.2安装依赖软件

 Dnf-y安装libncurses * ncurses ncurses-devel bison libarchive openssl-devel libtirpc-devel rpm -ivh rpcgen-1.3.1-4.el8.x86_64.rpm

7.3创建MySQL软件目录并解压软件

 mkdir /ctdb/mysql mkdir /ctdb/mysql/base mkdir /ctdb/mysql/boost chown cantian:cantian /ctdb/mysql -R cp /path/to/mysql-server-mysql-8.0.26.tar.gz /ctdb/mysql su-s/bin/bash - cantian光盘/ctdb/mysql tar -xvf mysql-server-mysql-8.0.26.tar.gz mv mysql-server-mysql-8.0.26/ mysql_source/

7.4编译MySQL

 光盘/ctdb/mysql/mysql_source/cmake-dcmake_install_prefix =/ctdb/mysql/base-dbuild_config = mysql_release -DWITH_UNIT_TESTS = 0 -DDOWNLOAD_BOOST = 1 -DCMAKE_C_COMPILER =/usr/bin/gcc -DCMAKE_CXX_COMPILER =/usr/bin/g -DFORCE_INSOURCE_BUILD = 1 -DWITH_BOOST =/ctdb/mysql/boost

 制造制造安装

7.5创建其他目录

 su -s /bin/bash-悬臂mkdir /ctdb/mysql/var mkdir /ctdb/mysql/数据mkdir /ctdb/mysql/tmp mkdir /ctdb/mysql/日志

7.6拷贝插件文件

 cp /opt/cantian/mysql/server/plugin/ha_ctc.so /ctdb/mysql/base/lib/plugin/ cp /opt/cantian/image/cantian_connector/for_mysql_official/mf_connector_mount_dir/cantian_lib/libctc_proxy.so /ctdb/mysql/base/lib/plugin/ cp /opt/cantianan/image/cantian_connector/for_mysql_official/mf_connector_mount_dir/cantian_lib/libsecurec。so /ctdb/mysql/base/lib/plugin/ cp /opt/cantian/image/cantian_connector/for_mysql_official/mf_connector_mount_dir/cantian_lib/libsecurec.a /ctdb/mysql/base/lib/plugin/ cp /opt/cantian/image/cantian_connector/for_mysql_official/mf_connector_mount_dir/cantian_lib/libctc_proxy.so /usr/lib64/ cp /opt/cantian/image/cantian_connector/for_mysql_official/mf_connector_mount_dir/cantian_lib/libsecurec.so /usr/lib64/ cp /opt/cantian/image/cantian_connector/for_mysql_official/mf_connector_mount_dir/cantian_lib/libsecurec.a /usr/lib64/ chmod 755 /usr/lib64/libctc_proxy.so chmod 755 /usr/lib64/libsecurec.所以chmod 755 /usr/lib64/libsecurec。a chmod 755 /ctdb/mysql/base/lib/plugin/ha_ctc.so chmod 755 /ctdb/mysql/base/lib/plugin/libctc_proxy.so chmod 755 /ctdb/mysql/base/lib/插件/libsecurec.所以chmod 755 /ctdb/mysql/base/lib/plugin/libsecurec。一个chown -R cantian: cantian/ctdb/mysql/base/lib/plugin/ha_ctc.so chown -R cantian: cantian/ctdb/mysql/base/lib/plugin/libctc_proxy.so chown -R cantian: cantian/ctdb/mysql/base/lib/plugin/libsecurec.所以cantian: cantian/ctdb/mysql/base/lib/plugin/libsecurec。a

7编写配置文件

编写配置文件/etc/my.Cnf

 [client] socket = /ctdb/mysql/var/mysql.sock port = 3306 [mysqld]############# 常规 ############### 事件调度器 = ON autocommit = OFF character_set_server = UTF8MB4 colation_server = UTF8MB4_UNICODE_CI explicit_defaults_for_timestamp = ON lower_case_table_table_names = 1 port = 3306 read_actived32 intercommitted = 隔离 = 600 M max_connections = 256 max_user_connections = 200 sort_buffer_size = 1m table_definition_cache = 1400 table_open_cache = 2000 table_open_cache_instances =4 thread_cache_size = 9 thread_stack = 256K tmp_table_size = 64m max_heap_table_size = 64m check_proxy_users = ON mysql_native_password_proxy_users = ON ##################### local_infile = OFF skip_name_resolve = ON sql_value_or_zero，auto_dation_dation_dation_value_or_zero，auto_ NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY ############# LOGGING ############# general_log = 0 log_queries_not_using_indexes = ON log_slow_admin_statements = ON long_query_time = 10 slow_query_log = ON log_raw = ON #############复制 ############# server_id = 101010111 ############### PATH ################## basedir = /ctdb/mysql/base datadir = /ctdb/mysql/data tmpdir = /ctdb/mysql/tmp socket = /ctdb/mysql/mysql。sock pid_file = /ctdb/mysql/data/mysql.pid innodb_data_home_dir = /ctdb/mysql/data log_error = /ctdb/mysql/log/error。日志general_log_file = /ctdb/mysql/log/general.日志slow_query_log_file = /ctdb/mysql/log/slow。日志 [mysql] ############### 客户端 ############## max_allowed_packet = 32m socket = /ctdb/mysql/var/mysql.袜子 [mysqldump] max_allowed_packet = 32m

注:以下参数必须配置
check_proxy_users = ON
mysql_native_password_proxy_users = 开启
log_raw = 开启

注:MySQL配置的最大_连接要低于Cantian部署时的 "Z_KERNEL_PARAMETER9": "SESSIONS = 512" 配置的值(详见本文6.9)，也可使用下面命令查询

 Su-s/bin/bash - cantian ctsql sys@127.0.0.1:1611 -q # 这里需要输入Cantian部署过程中配置的管理员密码

 显示参数会话

7.8加载插件

  chmod 755 /etc/my.cnf chown -R cantian:cantian /etc/my.cnf /ctdb/mysql/base/bin/mysqld --defaults-file=/etc/my.cnf --initialize-insecure --datadir=/ctdb/mysql/data --plugin-dir=/ctdb/mysql/base/lib/plugin --early-plugin-load="ctc_ddl_rewriter=ha_ctc.so;ctc=ha_ctc.so;" --initialize-insecure --user=cantian

7.9 启动MySQL

 /ctdb/mysql/base/bin/mysqld_safe --defaults-file=/etc/my.cnf --datadir=/ctdb/mysql/data --plugin-dir=/ctdb/mysql/base/lib/plugin --plugin-load="ctc_ddl_rewriter=ha_ctc.so;ctc=ha_ctc.so;" --check_proxy_users=ON --mysql_native_password_proxy_users=ON --default-storage-engine=CTC --user=cantian &

7.10 deploy and start MySQL on Node 2

you can directly copy the base directory of node 1 to the corresponding path of Node 2, and complete other content except MySQL source code compilation in this section. Note that you need to adjust server_id = 101010112 in the configuration file.

8 opinions and suggestions

in the process of compiling and deploying Cantian, I would like to complain about the following (opinions and suggestions) for Cantian 24.06:

only CentOS 8.2(x86_64) and OpenEuler-22.03-LTS(aarch64) operating system versions can be deployed. The database can only use MySQL 8.0.26. These two points cause poor adaptation of the operating system and database, and cannot obtain the latest updates, fixes, and security-related upgrades.
Compilation depends on docker and runs as a container, which is slightly more complex.
A large number of EL7 packages were used in the compilation process, but the version matching was not good, which is a bit confusing.
The compilation time is relatively long, and EMCC is deployed on a par with each other in hours (this may also be due to insufficient CPU resources I have given).
When git pulls the source code, enter version 24.6, which is inconsistent with version 24.06 corresponding to the document and installed after compilation.
Currently, only two nodes are supported. More nodes are supported.
Official documents and default deployment support for servers of different specifications is not very good, you need to lower some parameters to perform.
MySQL also needs to be compiled and installed. We recommend that you directly include the corresponding MySQL package generation during the Cantian compilation process (if any, please indicate it in a prominent position in the document).

Summary

this article is relatively complex from the perspective of directories. It can be said that it is the most complete documentation for the deployment of the Cantian storage engine and the docking of MySQL. It looks very complex and is not simple in practice, I also hope my opinions and suggestions can help the development of Cantian engine.
As usual, I know what I have written.

This article is authorized and reprinted by fat head Fish's Fish Tank (Yin Haiwen).

Replies（）

Sort By

Time

Ace

Posts

Followers

Following

 Follow

Recommended