Comments on: Grouping Related Trends with Hadoop and Hive

By: Project ideas for Hadoop « Ganbatte…!

Project ideas for Hadoop « Ganbatte…! — Thu, 15 Oct 2009 03:41:27 +0000

[...] http://www.cloudera.com/blog/2009/09/28/grouping-related-trends-with-had oop-and-hive/ [...]

By: andy.edmonds.be › links for 2009-09-29

andy.edmonds.be › links for 2009-09-29 — Wed, 30 Sep 2009 00:35:42 +0000

[...] Grouping Related Trends with Hadoop and Hive Â» Cloudera Hadoop & Big Data Blog (tags: hadoop hive datamining python trends graph) This was written by andy. Posted on Wednesday, September 30, 2009, at 1:35 am. Filed under Delicious. Bookmark the permalink. Follow comments here with the RSS feed. Post a comment or leave a trackback. [...]

By: Pete Skomoroch

Pete Skomoroch — Mon, 28 Sep 2009 21:58:08 +0000

Small correction found when looking at the query timings: the related links displayed are actually from another table that finds the “mutual links”, pages that are both outlinks and backlinks. That requirement gives a more narrow set of backlink pages that are also pointed to by the target page:

SELECT x,y,z
FROM backlinks JOIN LINKS ON (links.page_id = backlinks.page_id)
WHERE links.pl_title = backlinks.bl_title;

By: Pete Skomoroch

Pete Skomoroch — Mon, 28 Sep 2009 21:49:02 +0000

Otis,

That final SELECT into backlinks_reduced took 330 seconds, which ran the streaming code on 2,445,704 rows. The example Hive selects from backlinks_reduced take around 160 seconds. For use in a web application, I would export those final key-value pairs to MySQL and index by page_id or load them into a non-RDBMS datastore. Hive is best for offline batch operations where you need to run against all the data or a certain partition.

-Pete

By: Otis Gospodnetic

Otis Gospodnetic — Mon, 28 Sep 2009 21:24:50 +0000

Pete, can you share how fast those backlinks SELECTs on a dataset this large are?

By: Lalit Kapoor

Lalit Kapoor — Mon, 28 Sep 2009 17:12:33 +0000

Pete, this is a great post. I am glad that you organized it so well. Thanks for sharing your work. This is a really cool project to replicate if you want to learn a bit and get your hands dirty.