Adopting a Modern Big Data Platform to Address Data Challenges

Founded in 1991 and based in Beijing, China Galaxy Securities (CGS) is a leading Chinese securities institution that offers a wide range of investment and trading services in equities, bonds, funds, and derivatives. Over the recent years, CGS focuses on digital transformation and has been actively applying digital technology to its business and services.

Growing Demand for Efficient Data Processing

Digital economy has become an important driver for China's economic growth. As an asset-light and intelligence-intensive industry, securities will also be energized by the wave of digital transformation as China is entering a new stage of economic development. This will improve the adaptability and inclusiveness of the securities service system and contribute to the effectiveness and systematization of the securities risk control system.

The recent years have seen the great efforts of CGS to promote its digital transformation. However, as its business continues to grow, the demand for data processing sensitivity is increasing, so the traditional data processing platform can no longer meet these new developing needs. For the business related to real-time asset management CGS needs to transform and upgrade its entire counter system for accurate and low-latency tracking of customers' real-time assets. Meanwhile, as the data storage of CGS was too homogeneous, plus downstream data retrieval was only possible through data push, the singularity of data management cannot meet its demand for efficient data-based business.

Adopting a Modern Big Data Platform to Address Data Challenges

Since 2016, CGS has partnered with Cloudera, and has been expanding its use of Cloudera Big Data Platform features in recent years. After upgrading to Cloudera Data Platform (CDP), CGS has built a modern big data platform based on lakehouse technology that enables efficient processing, computing, and processing of data to bring new opportunities to business decisions and offer robust support and impetus.

CDP supports various data storage, collection, processing, and other service functions, while lakehouse technology enables real-time offline data processing and batch data push. Data management technology like this is flexible and complete, significantly enhancing CGS' data batch running capability per unit time and improving cost and efficiency in data management.

In January 2023, CGS upgraded the use of CDP with further functional expansions in data storage and cluster expansion.

Improving Cost and Efficiency in Data Management with Lakehouse

By migrating to CDP, CGS greatly enhances its data storage and batch running capabilities, gaining significant benefits, especially in real-time user asset tracking where real-time assets were addressed at great cost in the securities counter system. Given that a single customer may own multiple accounts, when all assets in the counter system are required to be displayed, real-time tracking can be challenging as it requires the T+1 offline calculation with high latency. The combination of the advanced lakehouse technology from CDP with Kafka for real-time stream collection and Spark for real-time business analysis and statistics enables timely display of real-time assets to customers, solving latency while saving costs.

In addition, the data storage has increased from 2T to 800T after the migration to CDP, along with significant improvement in computing performance that enables batch running of all data of CGS to be completed within 4 hours and the unit time of processing to increase from the previous 0.4T per hour to 200T per hour, making it five-hundreds times faster.

CGS will continue to expand its adoption of more CDP features to build a real-time data lake with flow-batch integration, contributing to the construction of a new data-driven "Intelligent Galaxy".