一种基于MapReduce的增量计算方法

Incremental computation method on basis of MapReduce

Abstract

本发明公开了一种基于MapReduce的增量计算方法,该方法包括:创建用于缓存不同的历史处理结果的增量处理模型,包括:缓存combiner结果的模型、缓存中间结果的模型与结果直接重用的模型;当获得输入数据时,根据所述输入数据的数据特性选择对应的增量处理模型进行数据处理,并在增量数据到来时,调用所述对应的增量处理模型进行数据处理时缓存的历史处理结果进行增量数据的计算。本发明公开的方法,通过选择适用于数据特性的模型进行计算,可以节省大量不必要的重复计算,从而提高了数据处理的效率。
The invention discloses an incremental computation method on the basis of MapReduce. The incremental computation method includes creating incremental processing models used for caching different historic processing results including one caching combiner results, one caching intermediate results and one directly reusing the results; selecting the corresponding incremental processing models for processing data according to the characteristics of the input data when the input data are acquired and computing the incremental data by calling the cached historic processing results during data processing of the corresponding incremental processing models when the incremental data are present. Via the method selecting the models applicable to the data characteristics for computation, many unnecessary repeated computations can be saved and data processing efficiency is improved.

Claims

Description

Topics

Download Full PDF Version (Non-Commercial Use)

Patent Citations (2)

    Publication numberPublication dateAssigneeTitle
    CN-103440244-ADecember 11, 2013广东电子工业研究院有限公司一种大数据存储优化方法
    US-2014067884-A1March 06, 2014International Business Machines CorporationAtomic incremental load for map-reduce systems on append-only file systems

NO-Patent Citations (0)

    Title

Cited By (1)

    Publication numberPublication dateAssigneeTitle
    WO-2017113865-A1July 06, 2017华为技术有限公司一种大数据增量计算方法和装置