摘要
随着BI(Business Intelligence,即商业智能技术)的发展和工业界业务需求的转变,企业迫切需要将数据源的变更数据实时地更新到数据仓库中进行分析与处理,从而及时辅助业务人员做出商业决策,因此产生了对增量变更数据的实时监测并捕获更新的需求。传统的数据更新方法无法识别数据中的增量更新,且获得的数据已失去时效性,难以支撑实时的决策分析,因此提出了一种基于数据库日志的变更数据捕获更新方法。并以实际应用案例为背景,介绍了如何运用该方法低侵入地捕获多源异构数据库的变更日志,并且将变更数据实时同步到下游数据库或系统中。本文将该方法投入到工业大数据场景下的实际企业生产环境中进行应用,构建基于数据库日志的变更数据捕获原型系统,使用企业的异构原始生产数据和业务数据对系统进行测试,验证了系统在实际大数据场景下的变更数据捕获能力,目前该系统运行良好。本文提出的基于数据库日志的多源异构变更数据捕获更新方法可以帮助企业以低侵入、高实时、高可用的方式完成异构数据的采集、处理和同步,能够高效地帮助企业快速进行数据分析,支撑企业做出正确的商业决策。
With the development of BI(Business Intelligence)and the transformation of business needs in the industry,enterprises urgently need to update the change of data sources in real time to the data warehouse for analysis and processing,in order to timely assist business personnel in making business decisions.Therefore,there is a need for real-time monitoring and capture of incremental change data updates.The traditional data update method cannot identify incremental updates in the data,and the obtained data has lost timeliness,which is difficult to support real-time decision analysis.In response to the above problems,this article proposes a change in the method of data capture and update based on database logs.Taking practical application cases as background,it describes how to use this method to invasively capture change logs from heterogeneous multi-source databases and synchronize change data in real time to downstream databases or systems.The method is implemented in an actual enterprise production environment within the context of industrial big data,resulting in the development of a prototype system for change data capture based on database logs.This system undergoes testing using heterogeneous original production data and business data,confirming its capability for change data capture in real-world big data scenarios.The proposed multi-source heterogeneous change data capture and update method based on database logs offers enterprises a low-intrusion,high-real-time,and high-availability solution for collecting,processing,and synchronizing heterogeneous data.It efficiently supports rapid data analysis and enables informed business decision-making.
作者
王凯军
李飞
李素芳
鲁奕彤
Wang Kaijun;Li Fei;Li Sufang;Lu Yitong(Strategic Development Headquarters of HBIS Digital Technology CO.,Ltd.,Shijiazhuang 050000,Hebei;School of Computer Science and Technology,Xidian University,Xi'an 710000,Shaanxi)
出处
《河北冶金》
2024年第6期70-75,共6页
Hebei Metallurgy
关键词
多源异构
变更数据捕获
数据库日志
商业智能
实时更新
低侵入
multi-source heterogeneous
change data capture
database log
business intelligence
real-time update
low intrusion
