Comments on: Hadoop at Twitter (part 1): Splittable LZO Compression http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/ Hadoop and Cloudera's Products and Services Fri, 10 Feb 2012 20:11:24 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: hadoop uncompress / process gzip | Active Intelligence http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/comment-page-1/#comment-16408 hadoop uncompress / process gzip | Active Intelligence Thu, 06 Oct 2011 06:46:09 +0000 http://www.cloudera.com/blog/?p=1576#comment-16408 [...] Hadoop at Twitter (part 1): Splittable… - cloudera.com [...] [...] Hadoop at Twitter (part 1): Splittable… - cloudera.com [...]

]]> By: Hadoop Compression – The Elephant That’s Not In The Room | CloudTimes http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/comment-page-1/#comment-16021 Hadoop Compression – The Elephant That’s Not In The Room | CloudTimes Tue, 24 May 2011 20:10:17 +0000 http://www.cloudera.com/blog/?p=1576#comment-16021 [...] Gzip (see Amazon AWS’ guide to Hadoop compression here) and LZO (see the blog post by Matt Massie back in 2009 about using LZO with Hadoop at Twitter) to reduce the amount of disk required. As we know binary compression has its limits, and comes [...] [...] Gzip (see Amazon AWS’ guide to Hadoop compression here) and LZO (see the blog post by Matt Massie back in 2009 about using LZO with Hadoop at Twitter) to reduce the amount of disk required. As we know binary compression has its limits, and comes [...]

]]> By: NerdyNick Blog | Some place on the Internet thats full of some random shit http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/comment-page-1/#comment-15992 NerdyNick Blog | Some place on the Internet thats full of some random shit Sun, 15 May 2011 02:52:28 +0000 http://www.cloudera.com/blog/?p=1576#comment-15992 [...] into chunks. Allowing for a much improved Map phase. You can read more on LZO with this great article from [...] [...] into chunks. Allowing for a much improved Map phase. You can read more on LZO with this great article from [...]

]]> By: Ravi http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/comment-page-1/#comment-14939 Ravi Mon, 03 Jan 2011 10:46:33 +0000 http://www.cloudera.com/blog/?p=1576#comment-14939 How do we test it whether LZO compression/decompression is correctly working or not. How do we test it whether LZO compression/decompression is correctly working or not.

]]> By: kailashbuki http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/comment-page-1/#comment-14852 kailashbuki Tue, 21 Dec 2010 05:28:48 +0000 http://www.cloudera.com/blog/?p=1576#comment-14852 Thanks for sharing your ideas!!! Thanks for sharing your ideas!!!

]]> By: CentOS?Hadoop-lzo????? http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/comment-page-1/#comment-12858 CentOS?Hadoop-lzo????? Fri, 09 Jul 2010 13:34:10 +0000 http://www.cloudera.com/blog/?p=1576#comment-12858 [...] cloudera??????????????????? code.google.com/p/hadoop-gpl-compression???????Twitter??????????????????? http://github.com/kevinweil/hadoop-lzo [...] [...] cloudera??????????????????? code.google.com/p/hadoop-gpl-compression???????Twitter??????????????????? http://github.com/kevinweil/hadoop-lzo [...]

]]> By: Lekhnath http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/comment-page-1/#comment-12843 Lekhnath Wed, 07 Jul 2010 12:34:58 +0000 http://www.cloudera.com/blog/?p=1576#comment-12843 Great work! Will love to test it Great work! Will love to test it

]]> By: Les contributions des entreprises et l'open source dans le contexte de NoSQL - Philippe Scoffoni http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/comment-page-1/#comment-11783 Les contributions des entreprises et l'open source dans le contexte de NoSQL - Philippe Scoffoni Tue, 27 Apr 2010 22:25:34 +0000 http://www.cloudera.com/blog/?p=1576#comment-11783 [...] Kevin a récemment « twitté » à propos de Hadoop-LZO, un projet sur lequel Twitter travaille avec Cloudera et [...] [...] Kevin a récemment « twitté » à propos de Hadoop-LZO, un projet sur lequel Twitter travaille avec Cloudera et [...]

]]> By: Hadoop Digest, March 2010 « Sematext Blog http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/comment-page-1/#comment-11608 Hadoop Digest, March 2010 « Sematext Blog Mon, 29 Mar 2010 18:06:38 +0000 http://www.cloudera.com/blog/?p=1576#comment-11608 [...] How to configure and use LZO compression? Take a look at http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/. [...] [...] How to configure and use LZO compression? Take a look at http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/. [...]

]]> By: schubertzhang http://www.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/comment-page-1/#comment-11316 schubertzhang Fri, 12 Mar 2010 03:08:20 +0000 http://www.cloudera.com/blog/?p=1576#comment-11316 When using such spliting method, how do you deal with the record/row boundary? A record/row may be truncated into two adjacent compression blocks. If the tow adjacent blocks are logically splited into different Mappers, it is difficult to determine the incomplete record/row and combine the to parts. When using such spliting method, how do you deal with the record/row boundary?

A record/row may be truncated into two adjacent compression blocks. If the tow adjacent blocks are logically splited into different Mappers, it is difficult to determine the incomplete record/row and combine the to parts.

]]>