Disco Tip – Crunching web server logs

At my day job we use Disco, a Python + Erlang based Map-Reduce framework, to crunch our web servers and application logs to generate useful data. Each web server log file per day is a couple of GB of data which can amount to a lot of log data that needs to be processed on a daily. Since the files are big it was easier for us to perform all the necessary filtering of find the rows of interest in the “map” function. [Read More]

New programming languages forces you to re-think a problem in a fresh way (or why do we need new programming languages. always.)

Whenever a new programming language appears some claim its the best thing since sliced bread (tm – not mine ;-) ), other claim its the worst thing that can happen and you can implement everything that the language provides in programming language X (assign X to your favorite low level programming language and append a suitable library). After seeing Google’s new Go programming language I must say I’m excited. Not because its from Google and it got a huge buzz around the net. [Read More]