Text/ yan hao, director of OceanData solutions for huawei's storage product line

in recent years, driven by independent innovation and cloudy transformation, the financial industry has started to transform traditional databases one after another. In the selection of target databases, distributed databases are very popular, but the actual implementation situation is not ideal. According to the statistics of financial database supply chain security development report, distributed database accounts for only 17.5% in banking industry, and even less than 4% in securities and insurance industry, the scale application in the core financial scenarios is progressing slowly.

What causes this phenomenon? It is true that distributed databases have drawn a beautiful blueprint in terms of architecture. However, three technical reasons have made the transformation of distributed databases in the financial industry remain at a low level:

first, the reliability does not meet the requirements of the financial industry. The importance of financial business is extraordinary. Business interruption and data loss will cause inestimable losses. Most distributed databases adopt an integrated storage and computing architecture. Servers provide computing resources and are responsible for data storage. Because the server is prone to failure, business interruption and data loss occur simultaneously after the failure, which can be said to be even worse. Currently, distributed databases generally improve reliability through multiple data replicas, but new problems follow one after another: multiple replicas must maintain strict consistency, so database performance is bound to be affected; If not, data loss may also occur.

Second, the performance is far from that of traditional databases. Currently, the performance of a single distributed database is poor. Therefore, you need to split the previous database into multiple databases to improve the performance by relying on the concurrency advantage. However, the traditional business data volume is large and the database and table structure is complex, which makes it difficult to perfect the transformation. In addition, during peak hours, excessive access to local databases can easily lead to congestion, resulting in the paralysis of the entire system.

Third, the transformation cost is too high and the operation and maintenance management is difficult. The transformation of database and table division is a systematic project, which involves synchronous modification of applications and is time-consuming and laborious. The number of servers and hard disks has increased greatly, resulting in high transformation costs. In addition, unlike professional storage, which can provide good hard disk health management capabilities, the risk of server failure often needs to be borne by operation and maintenance personnel. What is more terrible is that this kind of failure often occurs continuously, and it is never known which plate will become the "Black Swan" that causes the butterfly effect ".

For the above reasons, distributed databases are always difficult to make breakthroughs in the financial industry. So, where is the way to break the game? Throughout the development of IT industry, the industry often solves functional problems through software development; For stability and performance problems, IT often breaks through hardware technological innovation. The current problems faced by distributed databases are ultimately caused by the insufficient reliability of server hardware under the architecture of integrated storage and computing. Only relying on software-level solutions such as multiple replicas and Database Sharding and table sharding cannot completely solve the problem. Improving hardware capabilities is the key. Therefore, upgrading the distributed database to the storage-Computing separation architecture and storing data from more reliable professional storage can fundamentally solve the reliability problem of the distributed database.

First of all, in the memory-Computing separation architecture, data is stored in highly reliable professional storage, and data will not be lost even if the server is damaged. Therefore, it is not necessary to have multiple copies, naturally, data synchronization between replicas no longer exists, which can solve the performance problems of distributed databases. Secondly, the increase in the utilization rate of storage resources also greatly reduces the number of hard disks. Through professional storage for hard disk health management, system risks can be better eliminated. Thirdly, the computing and storage resources are unbound under the architecture of storage and computing separation, and the capacity and computing power are expanded on demand respectively, which can reduce the number of servers from the source, thus reducing procurement and management costs.

Database Sharding and table sharding have always been a major problem in using distributed databases. Essentially, Database Sharding and table sharding can compensate for the overall reliability and performance loss of the system through distributed deployment of data, but at the cost of high transformation costs. In fact, Database Sharding and table sharding can be avoided by improving the performance of a single database and reducing the system risk after a single machine failure. For example, Oracle databases ensure high availability of data layers through the Storage and computing separation architecture, allowing servers to access shared data, and using ASM(Automatic Storage Management) shared access to cache data allows multiple servers to access one database at the same time, which solves the performance and continuity of a single database. Coincidentally, huawei's OceanData distributed database storage solution adopts the self-developed apsara stack database acceleration engine, and also realizes cache sharing between database servers under the architecture of separation of storage and computing, in addition to OceanStor Dorado's high-performance all-flash storage, multiple distributed database instances can read and write a database at the same time and process transactions separately, greatly improving the overall performance of the database. With the separation of storage and computing architecture, distributed databases can achieve high performance and high reliability, and greatly reduce the use threshold and cost of enterprises.

Huawei's OceanData distributed database storage solution is actively adapting to a variety of application scenarios. In the core application scenario, huawei has built a joint solution of GaussDB and OceanStor Dorado to ensure that data is completely copied to disaster recovery sites by using dual-active storage to ensure that the performance of work sites is not affected, help GaussDB achieve dual-cluster disaster recovery for work sites and disaster recovery sites, meet the financial core-level business requirements, and implement commercial use in the core business of a large state-owned bank; In Internet application scenarios, through the storage and computing separation architecture and self-developed container storage solutions, Huawei can quickly recover container applications after server failures, storage failures, and site failures, and jointly create excellent practices with MySQL databases, A commercial firm in a city in Southwest China has been established.

Huawei's OceanData distributed database storage solution provides professional storage capabilities to help transform distributed databases more worry-free and assured! Facing the future, Huawei storage will continue to deepen innovation and cooperation with industry partners, promote innovation and upgrading of Data Inventory separation, and comprehensively improve performance, reliability and manageability, jointly promote the scale application of distributed databases in the financial industry.

Disclaimer: the content and opinions of the article only represent the author's own views, for readers' reference of ideological collision and technical exchange, and are not used as the official basis for Huawei's products and technologies. For more information about Huawei's products and technologies, visit the product and technology introduction page or consult Huawei's personnel.

In recent years, driven by independent innovation and cloudy transformation, the financial industry has started to transform traditional databases one after another. In the selection of target databases, distributed databases are very popular, but the actual implementation situation is not ideal. According to the statistics of financial database supply chain security development report, distributed database accounts for only 17.5% in banking industry, and even less than 4% in securities and insurance industry, the scale application in the core financial scenarios is progressing slowly.

What causes this phenomenon? It is true that distributed databases have drawn a beautiful blueprint in terms of architecture. However, three technical reasons have made the transformation of distributed databases in the financial industry remain at a low level:

first, the reliability does not meet the requirements of the financial industry. The importance of financial business is extraordinary. Business interruption and data loss will cause inestimable losses. Most distributed databases adopt an integrated storage and computing architecture. Servers provide computing resources and are responsible for data storage. Because the server is prone to failure, business interruption and data loss occur simultaneously after the failure, which can be said to be even worse. Currently, distributed databases generally improve reliability through multiple data replicas, but new problems follow one after another: multiple replicas must maintain strict consistency, so database performance is bound to be affected; If not, data loss may also occur.

Second, the performance is far from that of traditional databases. Currently, the performance of a single distributed database is poor. Therefore, you need to split the previous database into multiple databases to improve the performance by relying on the concurrency advantage. However, the traditional business data volume is large and the database and table structure is complex, which makes it difficult to perfect the transformation. In addition, during peak hours, excessive access to local databases can easily lead to congestion, resulting in the paralysis of the entire system.

Third, the transformation cost is too high and the operation and maintenance management is difficult. The transformation of database and table division is a systematic project, which involves synchronous modification of applications and is time-consuming and laborious. The number of servers and hard disks has increased greatly, resulting in high transformation costs. In addition, unlike professional storage, which can provide good hard disk health management capabilities, the risk of server failure often needs to be borne by operation and maintenance personnel. What is more terrible is that this kind of failure often occurs continuously, and it is never known which plate will become the "Black Swan" that causes the butterfly effect ".

For the above reasons, distributed databases are always difficult to make breakthroughs in the financial industry. So, where is the way to break the game? Throughout the development of IT industry, the industry often solves functional problems through software development; For stability and performance problems, IT often breaks through hardware technological innovation. The current problems faced by distributed databases are ultimately caused by the insufficient reliability of server hardware under the architecture of integrated storage and computing. Only relying on software-level solutions such as multiple replicas and Database Sharding and table sharding cannot completely solve the problem. Improving hardware capabilities is the key. Therefore, upgrading the distributed database to the storage-Computing separation architecture and storing data from more reliable professional storage can fundamentally solve the reliability problem of the distributed database.

First of all, in the memory-Computing separation architecture, data is stored in highly reliable professional storage, and data will not be lost even if the server is damaged. Therefore, it is not necessary to have multiple copies, naturally, data synchronization between replicas no longer exists, which can solve the performance problems of distributed databases. Secondly, the increase in the utilization rate of storage resources also greatly reduces the number of hard disks. Through professional storage for hard disk health management, system risks can be better eliminated. Thirdly, the computing and storage resources are unbound under the architecture of storage and computing separation, and the capacity and computing power are expanded on demand respectively, which can reduce the number of servers from the source, thus reducing procurement and management costs.

Database Sharding and table sharding have always been a major problem in using distributed databases. Essentially, Database Sharding and table sharding can compensate for the overall reliability and performance loss of the system through distributed deployment of data, but at the cost of high transformation costs. In fact, Database Sharding and table sharding can be avoided by improving the performance of a single database and reducing the system risk after a single machine failure. For example, Oracle databases ensure high availability of data layers through the Storage and computing separation architecture, allowing servers to access shared data, and using ASM(Automatic Storage Management) shared access to cache data allows multiple servers to access one database at the same time, which solves the performance and continuity of a single database. Coincidentally, Huawei's OceanData distributed database storage solution adopts the self-developed Apsara stack database acceleration engine, and also realizes cache sharing between database servers under the architecture of separation of storage and computing, in addition to OceanStor Dorado's high-performance all-flash storage, multiple distributed database instances can read and write a database at the same time and process transactions separately, greatly improving the overall performance of the database. With the separation of storage and computing architecture, distributed databases can achieve high performance and high reliability, and greatly reduce the use threshold and cost of enterprises.

Huawei's OceanData distributed database storage solution is actively adapting to a variety of application scenarios. In the core application scenario, Huawei has built a joint solution of GaussDB and OceanStor Dorado to ensure that data is completely copied to disaster recovery sites by using dual-active storage to ensure that the performance of work sites is not affected, help GaussDB achieve dual-cluster disaster recovery for work sites and disaster recovery sites, meet the financial core-level business requirements, and implement commercial use in the core business of a large state-owned bank; In Internet application scenarios, through the storage and computing separation architecture and self-developed container storage solutions, Huawei can quickly recover container applications after server failures, storage failures, and site failures, and jointly create excellent practices with MySQL databases, A commercial firm in a city in Southwest China has been established.

Huawei's OceanData distributed database storage solution provides professional storage capabilities to help transform distributed databases more worry-free and assured! Facing the future, Huawei storage will continue to deepen innovation and cooperation with industry partners, promote innovation and upgrading of Data Inventory separation, and comprehensively improve performance, reliability and manageability, jointly promote the scale application of distributed databases in the financial industry.

Disclaimer: the content and opinions of the article only represent the author's own views, for readers' reference of ideological collision and technical exchange, and are not used as the official basis for Huawei's products and technologies. For more information about Huawei's products and technologies, visit the product and technology introduction page or consult Huawei's personnel.