High concurrency and reliable access method for massive heterogeneous data based on dynamic balancing technology
Zhao Xun, Zhou Chengsheng, Jin Wenjing, Liu Xiaoman,Wang Guiwen
Institute of Security, China Academy of Information and Communications Technology, Beijing 100191, China
Abstract： With the arrival of the era of big data, the highly concurrent and reliable access of massive heterogeneous data has become an urgent problem. This paper proposes a high concurrent and reliable access method for massive heterogeneous data based on dynamic balance technology. The method adopts decentralized task allocation mechanism to access massive data sources. For various heterogeneous data sources, a variety of collection methods based on HTTPS, SFTP, Kafka, and corresponding node allocation and recovery mechanisms are designed. The dynamic load balancing strategy is used to adjust the collection resources in real time to adapt to the changing data load and achieve high concurrency processing. This research provides an effective solution for the efficient and reliable access of massive heterogeneous data.
Key words : massive heterogeneous data; high concurrency; dynamic load balancing strategy
