Comments on: Analyzing Apache logs with Pig

By: Dam

Dam — Fri, 16 Sep 2011 15:29:09 +0000

for my shiped script I build an auto extract bash script done this way (beside geostream.pl and its dependances)
wrapper.sh.template ( 755 mode ) :
#!/bin/bash
sed ’0,/^__ARCHIVE_BELOW__$/d’ $0 | tar xj
export PERL5LIB=$PERL5LIB:$(pwd)
./geostream.pl $1
rm -rf ./*
exit 0

__ARCHIVE_BELOW__

And then I build my script with this command :
cp wrapper.sh.template wrapper.sh; tar cjf – Geo geostream.pl >> wrapper.sh

then I can use it in my pig script :
DEFINE iplookup `wrapper.sh $GEO`
ship (‘wrapper.sh’)
cache(‘/GeoIP/$GEO#$GEO’);

The data file /GeoIP/GeoLiteCity.dat is stored on my hdfs storage. and copied with cache.

By: androm

androm — Mon, 11 Jul 2011 21:02:30 +0000

Hi I was able to run this in local mode; but when I tried to run this in hadoop mode, I received the error msg “ERROR 2055: Received Error while processing the map plan: ‘ipwrapper.sh GeoLiteCity.dat ‘ failed with exit status: 2″

Any ideas? Thanks!

By: Mike

Mike — Tue, 25 Jan 2011 22:58:47 +0000

What’s the status of “org.apache.pig.piggybank.filtering” ? I don’t see it yet in piggybank, as of release 0.8

By: Pig to analyze Apache Logs | user's Blog!

Pig to analyze Apache Logs | user's Blog! — Thu, 14 Oct 2010 12:39:35 +0000

[...] is an interesting blog post on Cloudera with title Analyzing Apache logs with Pig. The author starts with how to install pig on Ubuntu and goes on to create a world map to visualize [...]

By: tecosystems » A Non-Exhaustive Look at Some Free and Non-Free Web Analytics Packages

Wed, 02 Dec 2009 20:38:19 +0000

[...] Instead we have pageviews and all of the other metrics with which to draw imperfect conclusions. I expect this to be a temporary blindness, however: too many people want these kinds of answers for the questions to go unanswered for long. And if we can’t get the answers we need from someone else, we just might take matters into our own hands using tools like Hadoop and Pig. [...]

By: Joshua Barratt

Joshua Barratt — Sun, 28 Jun 2009 18:35:42 +0000

Great article, have been looking at needing to process web logs on a large scale already and this is a very useful example.

For the ‘packing and shipping a perl script’ part, check out Par::Packer.

It allows you to turn a perl script and all it’s dependencies into a single monolithic script file.

By: a little pig helping make me famous « spack++

a little pig helping make me famous « spack++ — Fri, 19 Jun 2009 03:19:48 +0000

[...] there you go. Â Recently, someone ported my stuff (which was awesome!), and folks at cloudera are blogging about [...]

By: David Gwartney

David Gwartney — Thu, 18 Jun 2009 19:30:42 +0000

This is great information. Its answered my question regarding streaming inside PIG that came up the other day.

Thanks for the info.

Dave

By: Analysing Apache logs « Dumbotics

Analysing Apache logs « Dumbotics — Thu, 18 Jun 2009 12:03:03 +0000

[...] Apache logs The Cloudera guys blogged about using Pig for examining Apache logs yesterday. Although it nicely illustrates several [...]