Assessment of Various Chunking Techniques for De-duplication in Big Data

Rahul Rawat

doi:10.37591/rtpl.v3i2.793

Abstract

From previous fifteen years, the data is growing very rapidly; such a huge data is termed as big data. The size of big data can be in TB and PB. The main challenge in big data is to handle duplicate data and to obtain useful information. In this paper, an assessment of various chunking techniques has been presented. There are different chunking techniques available named; frequency based chunking, content based chunking, byte level chunking etc. Assessment is based on mapping of chunking techniques with big data applications and de-duplication performance metrics. Assessment shows that FBC is best suited for data de-duplication compared to other chunking techniques.Keywords: Big data, chunking, data de-duplication (De-dup), FBC (Frequency based Chunking), CDC (Content Defined Chunking)Cite this ArticleRahul Rawat. Assessment of Various Chunking Techniques for De-duplication in Big Data. Recent Trends in Programming Languages. 2016; 3(2): 7–12p.