nsq-to-gs – Streaming NSQ messages directly to Google Cloud Storage


In addition to my previously published (very early) project to stream NSQ messages directly to BigQuery, I am happy to presents a modified version of nsq-to-s3 that supports streaming NSQ messages directly Google Cloud Storage.

Grab it while its hot from the nsq-to-gs repo.

I do see a future for a merged version of these two projects that supports both S3 and Google Cloud Storage but this would have to be enough for now.


The current version has the same functionality as the latest nsq-to-s3 version and was adapted to support Google Storage with minor modifications (such as the default path and filename formats).

nsq-to-bigquery – Stream messages from NSQ directly to Google BigQuery


In the spirit of nsq-to-XXX such as nsq-to-http and nsq-to-file – I bring you the very first version of nsq-to-bigquery.

nsq-to-bigquery, as the name suggest, streams data from an NSQ channel into Google’s BigQuery using the Streaming API and provide very effective means to stream data that should be then further analysed and aggregated by BigQuery’s excellent performance.

This is a (very) initial version so it has some limitations and assumptions.

Limitations / Assumptions

  • The BigQuery table MUST exist prior to streaming the data
  • The NSQ message being sent MUST be a valid JSON string
  • The JSON format MUST be a simple flat dictionary (key and simple value. Value can’t be another dictionary or list)
  • The JSON format MUST match the format of the BigQuery table
  • At the moment there is no support for batching so each message will issue an API call to BigQuery with a single line of data

Planed Features:

  • Support batching with flushing based on X number of rows or Y amount of time passed since last flush
  • Flushing will happen in parallel with receiving information so there is almost no delay

Stay tuned on the github repo for more news.

gonionoo – Go wrapper for the Tor Network Status Protocol – OnionOO

I’ve bene running a Tor exit node in the Netherlands since August 2013. I believe in the cause of Tor and it was only a matter of time before I started adding code in some for or another.

gonionoo is Go wrapper for OnionOO – the Tor Network Status protocol as is the first step in a slightly larger project I’m working on that I’ve been planning for a while ever since I’ve became a Tor exit node operator.

The OnionOO API has lots of interesting data on the Tor network. You can see it visualized as part of the Atlas project.

MongoDB Replica-Set Aware Backup Script

I’ve created a nice little bash script to take MongoDB backups that is replicaset aware.

It will only take a backup from a replica so if you have the classic master,replica,arbiter configuration you can setup the script via cron on both (current) master and replica and the backup will only run on the replica.

It will then tar.gz the backup and upload it to Google Storage. It can be easily adapted to upload the backup to S3 using s3cmd or the aws cli (aws-cli).

Cross posted at Forecast:Cloudy (my cloud blog).

Seedcamp Tel-Aviv 2012

It’s that time of the year and Seedcamp Tel-Aviv is back (for the forth year!). This time lool Ventures is part of the event!

In one of my hats I’m the CTO of lool Ventures and I’ll be there as a mentor to give advice and share from my experience in building a startup.

So if you have a great idea and started to work on it be sure to apply now.



Requiem for a modem

Two days ago I’ve shut down the longest running electronics device I ever owned.

Alcatel SpeedTouch Home - Image from isphelp.info

The device was my an Alcatel Speedtouch Home ADSL modem which I got circa 2001 when I was lucky enough to get an ADSL line at home.

It was only turned off when there was a power failure or when I moved an apartment.

It survived 6 PC, 5 Laptops, 4 routers, 6 apartments spanning 4 cities and about 10 different cell phones.

It was hacked to use PPPoE instead of its default PPTP. Was hacked again to function as a router, and back to being just a modem.

When I started using it I had a 1.5Mbit ADSL line. It grew to 2.5Mbit and finally 5Mbps – its maximum supported speed (taking into account the infrastructure state, my distance from the switchboard, etc).

When the New Generation Network (NGN) of my landline provider Bezeq was deployed, the modem couldn’t  keep up with its 5Mbit speed because the uplink speed changed and it couldn’t sync. I downgraded to 2.5Mbps until I could get a replacement modem.

Once I got the newer modem, I shutdown the old one for good. It was now obsolete, old and unable to support faster speeds. No one would want it. No one would need it. No one would use it.

I will always remember it as the device that saved me from my happy dial-up days and brought me into the broadband age. It never failed, never stopped working and handled whatever bits were thrown at it.

It is now time for you to rest in modems heaven, where the line is always synced and the bits flow freely.

May all my current and future modems will serve me as well as you did.

Goodbye old friend. We had good times.

Scott Berkun’s Mindfire: Big Ideas For Curious Minds – Book Review

I had the pleasure of reading Scott Berkun‘s newest book – Mindfire: Big Ideas for Curious Minds. I was also forunate to get it for free in the short period of time where Scott gave it for free on his site, but this is not a guilty book review of getting the book for free.

Mindfire is a collection of 30 essays which Scott wrote in various places, mostly on his blog. The essays got cleaned up and preped for the book which made the reading very clean and flowing. Scott’s writing style is very flowing and funny and while it may seem at times as a self emporment / self help book it really isn’t.

I look at it more as a collection of percise and clear set of obersvations on the human condition and behavior alone and in a group. Some of the essays specifically talk about work related situtations, but in most cases  you can apply some of the tips and wisdom of this book to almost any interaction with other people.

I really enjoyed reading it and in some situations fully sympathize with the eassy’s topic and resolution.

UIImage in iOS 5, Orientation and Resize

One of the things I found very strange is the fact that most operations that came with iOS prior iOS 5 which revolved around UIImage didn’t take into account the orientation of the image. This meant that if you want to read a picture from the camera roll and resize it, you’d have to roll your own code to correctly flip and/or rotate the image according to its orientation value.

Being my lazy self I used the fine code of Trevor Harmon in UIImage+Resize. Trevor added some categories to make handling UIImage a bit nicer. The code takes create of everything including orientation.

My app worked great on iOS 4 and early betas of iOS 5, however in the late beta of iOS 5 and in the release it wrongfully rotated the images.

After further investigation it seems iOS 5 already rotates the image correctly. UIImage+Resize rotated it again, causing the images to get skewed.
A quick fix would simply avoid the transposition code in UIImage+Resize.

Since the code ran perfectly fine in iOS 4, for backwards compatibility I added a check for OS version and for anything below 5.0 the old code would work.
Check out this gist:

For better performance I would store a boolean flag somewhere in the app saying you are running in iOS 5 and check that instead of keep on checking the OS version every run, but this is just to get you started.

Clone S3 Bucket Script

I had to backup an S3 bucket so I whiped out a small script to clone a bucket.

It’s written in Python and depends on the excellent Boto library. If you are running Python < 2.7 you’ll also need the argparse library (both available also via pip).

View the gist here: https://gist.github.com/1275085

Or here below:

“Those who don’t know history are destined to repeat it.”

I know the title is a bit alarming, but that was my first thought after reading @bryce‘s latest post “The Rising Generation“.

Briefly, he mentioned a 25 y/o asking a question on Quora about how was life before everyone had a cell phone and no one talked a lot or texted in public areas. Bryce also say that the new entrepreneurs, like the ones in yesterday’s Y Combinator Demo Day have different expectation, understanding and perceived value of technology than any other that has come before them.

I’m 30. Not too far from the 25 y/o who asked the question (although I was about 15 when cell phone started to penetrate Israel quickly and spread like wild fire). In most companies I worked at I was usually the youngest (or second youngest) in the company for quite some time. Most of my co-workers used to be (or are) between 9-15 years older than me and are still surprised of my knowledge of things from the past.

I’m a bit of a history buff when it comes to computers, science and technology (but also to general history). That’s why I do know what a ZX Spectrum is (and I don’t know how it reached my house when I was about 7 but I had the chance to play with it). I know how VMS systems work (long story form the Army :-) ). When I was six I did play with an Apple IIc my brother got for his Bar Mitzvah and really liked Captain Goodnight and Karateka. I do know how a modem sounds (and can even detect by sound what is the connection speed. Tiiii Taaa TiiTaTa TiiiTa – Yay 28KBPS! ). Heck, I even ran a BBS and was a node on FidoNet when I was 14.

That might be a bit unusual for most people my age that are into computers (and maybe even for older ones) but that’s not too different from people that hear Pink Floyd, Led Zeppelin, Bach and Mozart. While these composers and bands did not exist when I was born or when I was a child the music kept on going. I and a lot of other people both older and younger do know it, hear it and enjoy it. It even inspires some to go on and create new things.

Knowing a bit of the past and how it related to an idea or thought you have can give you a much better appreciation to the things others have done or to the things others advice you to do. After all, specifically in the computers and internet industry, we are all “standing on the shoulders of giants”. It can also give entrepreneurs some much needed perspective on how things were, how things are and how it should be.

So, in addition to Bryce’s hope that “they can rise high enough to meet the emerging opportunities generations before them have made possible”, my 2 cents are to also learn a bit from the past. Take a minute for some history and try to figure out how things were before you push on forward. If they won’t they are destined to repeat it.