`
djun100
  • 浏览: 165631 次
  • 性别: Icon_minigender_1
  • 来自: 大连
文章分类
社区版块
存档分类
最新评论

Snappy压缩库安装和使用之一

阅读更多

近日需要在毕业设计中引入一个压缩库，要求压缩与解压缩速度快，但是压缩率可以不那么苛刻。查找资料发现Google的snappy库比较合适，而且该库开源，由C++写成。所以就拿来使用一下，下面权作记录。下面引出的任何涉及Google公司的源代码，版权归Google公司所有，我权作学习交流。文章安排如下，首先简要介绍Snappy，之后安装之，然后以实际例子介绍如何使用，接着bzip2和gzip做了性能比较，最后提出一些使用上面的疑问。

（一）简要介绍

去官网下载之http://code.google.com/p/snappy/。在ProjectHome处有这么一段英文，我想许多地方都引用和翻译了这段。我也尝试翻译一下。

Snappyisacompression/decompressionlibrary.

Itdoesnotaimformaximumcompression,

orcompatibilitywithanyothercompressionlibrary;

instead,itaimsforveryhighspeedsandreasonablecompression.

Forinstance,comparedtothefastestmodeofzlib,

Snappyisanorderofmagnitudefasterformostinputs,

buttheresultingcompressedfilesareanywherefrom20%to100%bigger.

OnasinglecoreofaCorei7processorin64-bitmode,

Snappycompressesatabout250MB/secormoreand

decompressesatabout500MB/secormore.

SnappyiswidelyusedinsideGoogle,ineverythingfromBigTable

andMapReducetoourinternalRPCsystems.

译文：Snappy是一个压缩/解压缩库。它不是以最大压缩率，或者与其他压缩库兼容为目标；它旨在获得高速的压缩和合理的压缩率。例如，Snappy对大多数的输入比zlib的最快模式要快几个数量级，但是其压缩过后的文件通常会比zlib大20%到100%。在Corei7的单核64位模式下，Snappy压缩速度大概可以达到250MB/s或者更快，解压缩可以达到大约500MB/s或更快。

Snappy在Google内部广泛使用，从BigTable，MapReduce到公司内部的RPC系统。

（二）安装过程

下面描述安装过程：

下载snappy-1.0.5.tar.gz，snappy的安装过程与传统的安装过程一样。解压后的INSTALL文件有详细的安装说明。

gunzipsnappy-1.0.5.tar.gz

tarxfsnappy-1.0.5.tar

cdsnappy-1.0.5

./configure

make

makeinstall

安装完成后，生成的动态库和静态库位于/usr/local/lib处，编程需要用到的头文件位于/usr/local/include处。注意需要将这些库文件cp至/usr/lib处，不然就算在链接的时候加上-L/usr/local/lib，在运行时也会报错。./main:errorwhileloadingsharedlibraries:libsnappy.so.1:

cannotopensharedobjectfile:Nosuchfileordirectory

当然这是我的LD_LIBRARY_PATH环境变量的设置问题。

（三）使用snappy

解压出来的README文件介绍了一简单的使用方式。snappy是各种库标示符所在的命名空间。C++使用需要包含#include<snappy.h>头文件，C语言使用需要包含#include<snapyy-c.h>头文件。Snappy使用较为简单，我指的是跟bzip2的库比起来。所有的函数接口都暴露在上述两个头文件中，头文件中有详细的使用说明，并有简单的示例，而且英文通俗易懂。摘抄如下（Google公司版权所有）：

snappy.h

[cpp] view plaincopyprint?
  1. //Copyright2005andonwardsGoogleInc.
  2. //
  3. //Redistributionanduseinsourceandbinaryforms,withorwithout
  4. //modification,arepermittedprovidedthatthefollowingconditionsare
  5. //met:
  6. //
  7. //*Redistributionsofsourcecodemustretaintheabovecopyright
  8. //notice,thislistofconditionsandthefollowingdisclaimer.
  9. //*Redistributionsinbinaryformmustreproducetheabove
  10. //copyrightnotice,thislistofconditionsandthefollowingdisclaimer
  11. //inthedocumentationand/orothermaterialsprovidedwiththe
  12. //distribution.
  13. //*NeitherthenameofGoogleInc.northenamesofits
  14. //contributorsmaybeusedtoendorseorpromoteproductsderivedfrom
  15. //thissoftwarewithoutspecificpriorwrittenpermission.
  16. //
  17. //THISSOFTWAREISPROVIDEDBYTHECOPYRIGHTHOLDERSANDCONTRIBUTORS
  18. //"ASIS"ANDANYEXPRESSORIMPLIEDWARRANTIES,INCLUDING,BUTNOT
  19. //LIMITEDTO,THEIMPLIEDWARRANTIESOFMERCHANTABILITYANDFITNESSFOR
  20. //APARTICULARPURPOSEAREDISCLAIMED.INNOEVENTSHALLTHECOPYRIGHT
  21. //OWNERORCONTRIBUTORSBELIABLEFORANYDIRECT,INDIRECT,INCIDENTAL,
  22. //SPECIAL,EXEMPLARY,ORCONSEQUENTIALDAMAGES(INCLUDING,BUTNOT
  23. //LIMITEDTO,PROCUREMENTOFSUBSTITUTEGOODSORSERVICES;LOSSOFUSE,
  24. //DATA,ORPROFITS;ORBUSINESSINTERRUPTION)HOWEVERCAUSEDANDONANY
  25. //THEORYOFLIABILITY,WHETHERINCONTRACT,STRICTLIABILITY,ORTORT
  26. //(INCLUDINGNEGLIGENCEOROTHERWISE)ARISINGINANYWAYOUTOFTHEUSE
  27. //OFTHISSOFTWARE,EVENIFADVISEDOFTHEPOSSIBILITYOFSUCHDAMAGE.
  28. //
  29. //Alight-weightcompressionalgorithm.Itisdesignedforspeedof
  30. //compressionanddecompression,ratherthanfortheutmostinspace
  31. //savings.
  32. //
  33. //Forgettingbettercompressionratioswhenyouarecompressingdata
  34. //withlongrepeatedsequencesorcompressingdatathatissimilarto
  35. //otherdata,whilestillcompressingfast,youmightlookatfirst
  36. //usingBMDiffandthencompressingtheoutputofBMDiffwith
  37. //Snappy.
  39. #ifndefUTIL_SNAPPY_SNAPPY_H__
  40. #defineUTIL_SNAPPY_SNAPPY_H__
  42. #include<stddef.h>
  43. #include<string>
  45. #include"snappy-stubs-public.h"
  47. namespacesnappy{
  48. classSource;
  49. classSink;
  51. //------------------------------------------------------------------------
  52. //Genericcompression/decompressionroutines.
  53. //------------------------------------------------------------------------
  55. //Compressthebytesreadfrom"*source"andappendto"*sink".Returnthe
  56. //numberofbyteswritten.
  57. size_tCompress(Source*source,Sink*sink);
  59. boolGetUncompressedLength(Source*source,uint32*result);
  61. //------------------------------------------------------------------------
  62. //Higher-levelstringbasedroutines(shouldbesufficientformostusers)
  63. //------------------------------------------------------------------------
  65. //Sets"*output"tothecompressedversionof"input[0,input_length-1]".
  66. //Originalcontentsof*outputarelost.
  67. //
  68. //REQUIRES:"input[]"isnotanaliasof"*output".
  69. size_tCompress(constchar*input,size_tinput_length,string*output);
  71. //Decompresses"compressed[0,compressed_length-1]"to"*uncompressed".
  72. //Originalcontentsof"*uncompressed"arelost.
  73. //
  74. //REQUIRES:"compressed[]"isnotanaliasof"*uncompressed".
  75. //
  76. //returnsfalseifthemessageiscorruptedandcouldnotbedecompressed
  77. boolUncompress(constchar*compressed,size_tcompressed_length,
  78. string*uncompressed);
  81. //------------------------------------------------------------------------
  82. //Lower-levelcharacterarraybasedroutines.Maybeusefulfor
  83. //efficiencyreasonsincertaincircumstances.
  84. //------------------------------------------------------------------------
  86. //REQUIRES:"compressed"mustpointtoanareaofmemorythatisat
  87. //least"MaxCompressedLength(input_length)"bytesinlength.
  88. //
  89. //Takesthedatastoredin"input[0..input_length]"andstores
  90. //itinthearraypointedtoby"compressed".
  91. //
  92. //"*compressed_length"issettothelengthofthecompressedoutput.
  93. //
  94. //Example:
  95. //char*output=newchar[snappy::MaxCompressedLength(input_length)];
  96. //size_toutput_length;
  97. //RawCompress(input,input_length,output,&output_length);
  98. //...Process(output,output_length)...
  99. //delete[]output;
  100. voidRawCompress(constchar*input,
  101. size_tinput_length,
  102. char*compressed,
  103. size_t*compressed_length);
  105. //Givendatain"compressed[0..compressed_length-1]"generatedby
  106. //callingtheSnappy::Compressroutine,thisroutine
  107. //storestheuncompresseddatato
  108. //uncompressed[0..GetUncompressedLength(compressed)-1]
  109. //returnsfalseifthemessageiscorruptedandcouldnotbedecrypted
  110. boolRawUncompress(constchar*compressed,size_tcompressed_length,
  111. char*uncompressed);
  113. //Givendatafromthebytesource'compressed'generatedbycalling
  114. //theSnappy::Compressroutine,thisroutinestorestheuncompressed
  115. //datato
  116. //uncompressed[0..GetUncompressedLength(compressed,compressed_length)-1]
  117. //returnsfalseifthemessageiscorruptedandcouldnotbedecrypted
  118. boolRawUncompress(Source*compressed,char*uncompressed);
  120. //Returnsthemaximalsizeofthecompressedrepresentationof
  121. //inputdatathatis"source_bytes"bytesinlength;
  122. size_tMaxCompressedLength(size_tsource_bytes);
  124. //REQUIRES:"compressed[]"wasproducedbyRawCompress()orCompress()
  125. //Returnstrueandstoresthelengthoftheuncompresseddatain
  126. //*resultnormally.Returnsfalseonparsingerror.
  127. //ThisoperationtakesO(1)time.
  128. boolGetUncompressedLength(constchar*compressed,size_tcompressed_length,
  129. size_t*result);
  131. //Returnstrueiffthecontentsof"compressed[]"canbeuncompressed
  132. //successfully.Doesnotreturntheuncompresseddata.Takes
  133. //timeproportionaltocompressed_length,butisusuallyatleast
  134. //afactoroffourfasterthanactualdecompression.
  135. boolIsValidCompressedBuffer(constchar*compressed,
  136. size_tcompressed_length);
  138. //***DONOTCHANGETHEVALUEOFkBlockSize***
  139. //
  140. //NewCompressioncodechopsuptheinputintoblocksofatmost
  141. //thefollowingsize.Thisensuresthatback-referencesinthe
  142. //outputnevercrosskBlockSizeblockboundaries.Thiscanbe
  143. //helpfulinimplementingblockeddecompression.Howeverthe
  144. //decompressioncodeshouldnotrelyonthisguaranteesinceolder
  145. //compressioncodemaynotobeyit.
  146. staticconstintkBlockLog=15;
  147. staticconstsize_tkBlockSize=1<<kBlockLog;
  149. staticconstintkMaxHashTableBits=14;
  150. staticconstsize_tkMaxHashTableSize=1<<kMaxHashTableBits;
  152. }//endnamespacesnappy
  155. #endif//UTIL_SNAPPY_SNAPPY_H__

snapp-c.h

[cpp] view plaincopyprint?
  1. /*
  2. *Copyright2011MartinGieseking<martin.gieseking@uos.de>.
  3. *
  4. *Redistributionanduseinsourceandbinaryforms,withorwithout
  5. *modification,arepermittedprovidedthatthefollowingconditionsare
  6. *met:
  7. *
  8. **Redistributionsofsourcecodemustretaintheabovecopyright
  9. *notice,thislistofconditionsandthefollowingdisclaimer.
  10. **Redistributionsinbinaryformmustreproducetheabove
  11. *copyrightnotice,thislistofconditionsandthefollowingdisclaimer
  12. *inthedocumentationand/orothermaterialsprovidedwiththe
  13. *distribution.
  14. **NeitherthenameofGoogleInc.northenamesofits
  15. *contributorsmaybeusedtoendorseorpromoteproductsderivedfrom
  16. *thissoftwarewithoutspecificpriorwrittenpermission.
  17. *
  18. *THISSOFTWAREISPROVIDEDBYTHECOPYRIGHTHOLDERSANDCONTRIBUTORS
  19. *"ASIS"ANDANYEXPRESSORIMPLIEDWARRANTIES,INCLUDING,BUTNOT
  20. *LIMITEDTO,THEIMPLIEDWARRANTIESOFMERCHANTABILITYANDFITNESSFOR
  21. *APARTICULARPURPOSEAREDISCLAIMED.INNOEVENTSHALLTHECOPYRIGHT
  22. *OWNERORCONTRIBUTORSBELIABLEFORANYDIRECT,INDIRECT,INCIDENTAL,
  23. *SPECIAL,EXEMPLARY,ORCONSEQUENTIALDAMAGES(INCLUDING,BUTNOT
  24. *LIMITEDTO,PROCUREMENTOFSUBSTITUTEGOODSORSERVICES;LOSSOFUSE,
  25. *DATA,ORPROFITS;ORBUSINESSINTERRUPTION)HOWEVERCAUSEDANDONANY
  26. *THEORYOFLIABILITY,WHETHERINCONTRACT,STRICTLIABILITY,ORTORT
  27. *(INCLUDINGNEGLIGENCEOROTHERWISE)ARISINGINANYWAYOUTOFTHEUSE
  28. *OFTHISSOFTWARE,EVENIFADVISEDOFTHEPOSSIBILITYOFSUCHDAMAGE.
  29. *
  30. *PlainCinterface(awrapperaroundtheC++implementation).
  31. */
  33. #ifndefUTIL_SNAPPY_OPENSOURCE_SNAPPY_C_H_
  34. #defineUTIL_SNAPPY_OPENSOURCE_SNAPPY_C_H_
  36. #ifdef__cplusplus
  37. extern"C"{
  38. #endif
  40. #include<stddef.h>
  42. /*
  43. *Returnvalues;seethedocumentationforeachfunctiontoknow
  44. *whateachcanreturn.
  45. */
  46. typedefenum{
  47. SNAPPY_OK=0,
  48. SNAPPY_INVALID_INPUT=1,
  49. SNAPPY_BUFFER_TOO_SMALL=2,
  50. }snappy_status;
  52. /*
  53. *Takesthedatastoredin"input[0..input_length-1]"andstores
  54. *itinthearraypointedtoby"compressed".
  55. *
  56. *<compressed_length>signalsthespaceavailablein"compressed".
  57. *Ifitisnotatleastequalto"snappy_max_compressed_length(input_length)",
  58. *SNAPPY_BUFFER_TOO_SMALLisreturned.Aftersuccessfulcompression,
  59. *<compressed_length>containsthetruelengthofthecompressedoutput,
  60. *andSNAPPY_OKisreturned.
  61. *
  62. *Example:
  63. *size_toutput_length=snappy_max_compressed_length(input_length);
  64. *char*output=(char*)malloc(output_length);
  65. *if(snappy_compress(input,input_length,output,&output_length)
  66. *==SNAPPY_OK){
  67. *...Process(output,output_length)...
  68. *}
  69. *free(output);
  70. */
  71. snappy_statussnappy_compress(constchar*input,
  72. size_tinput_length,
  73. char*compressed,
  74. size_t*compressed_length);
  76. /*
  77. *Givendatain"compressed[0..compressed_length-1]"generatedby
  78. *callingthesnappy_compressroutine,thisroutinestores
  79. *theuncompresseddatato
  80. *uncompressed[0..uncompressed_length-1].
  81. *Returnsfailure(avaluenotequaltoSNAPPY_OK)ifthemessage
  82. *iscorruptedandcouldnotbedecrypted.
  83. *
  84. *<uncompressed_length>signalsthespaceavailablein"uncompressed".
  85. *Ifitisnotatleastequaltothevaluereturnedby
  86. *snappy_uncompressed_lengthforthisstream,SNAPPY_BUFFER_TOO_SMALL
  87. *isreturned.Aftersuccessfuldecompression,<uncompressed_length>
  88. *containsthetruelengthofthedecompressedoutput.
  89. *
  90. *Example:
  91. *size_toutput_length;
  92. *if(snappy_uncompressed_length(input,input_length,&output_length)
  93. *!=SNAPPY_OK){
  94. *...fail...
  95. *}
  96. *char*output=(char*)malloc(output_length);
  97. *if(snappy_uncompress(input,input_length,output,&output_length)
  98. *==SNAPPY_OK){
  99. *...Process(output,output_length)...
  100. *}
  101. *free(output);
  102. */
  103. snappy_statussnappy_uncompress(constchar*compressed,
  104. size_tcompressed_length,
  105. char*uncompressed,
  106. size_t*uncompressed_length);
  108. /*
  109. *Returnsthemaximalsizeofthecompressedrepresentationof
  110. *inputdatathatis"source_length"bytesinlength.
  111. */
  112. size_tsnappy_max_compressed_length(size_tsource_length);
  114. /*
  115. *REQUIRES:"compressed[]"wasproducedbysnappy_compress()
  116. *ReturnsSNAPPY_OKandstoresthelengthoftheuncompresseddatain
  117. **resultnormally.ReturnsSNAPPY_INVALID_INPUTonparsingerror.
  118. *ThisoperationtakesO(1)time.
  119. */
  120. snappy_statussnappy_uncompressed_length(constchar*compressed,
  121. size_tcompressed_length,
  122. size_t*result);
  124. /*
  125. *Checkifthecontentsof"compressed[]"canbeuncompressedsuccessfully.
  126. *Doesnotreturntheuncompresseddata;ifso,returnsSNAPPY_OK,
  127. *orifnot,returnsSNAPPY_INVALID_INPUT.
  128. *Takestimeproportionaltocompressed_length,butisusuallyatleasta
  129. *factoroffourfasterthanactualdecompression.
  130. */
  131. snappy_statussnappy_validate_compressed_buffer(constchar*compressed,
  132. size_tcompressed_length);
  134. #ifdef__cplusplus
  135. }//extern"C"
  136. #endif
  138. #endif/*UTIL_SNAPPY_OPENSOURCE_SNAPPY_C_H_*/


分享到：
| android中tesseract-ocr的介绍
评论
发表评论

您还没有登录,请您登录后再发表评论

相关推荐

Magicbox
Global site tag (gtag.js) - Google Analytics