nsq-to-gs – Streaming NSQ messages directly to Google Cloud Storage

nsq-to-googlestorage

In addition to my previously published (very early) project to stream NSQ messages directly to BigQuery, I am happy to presents a modified version of nsq-to-s3 that supports streaming NSQ messages directly Google Cloud Storage.

Grab it while its hot from the nsq-to-gs repo.

I do see a future for a merged version of these two projects that supports both S3 and Google Cloud Storage but this would have to be enough for now.

 

The current version has the same functionality as the latest nsq-to-s3 version and was adapted to support Google Storage with minor modifications (such as the default path and filename formats).

MongoDB Replica-Set Aware Backup Script

I’ve created a nice little bash script to take MongoDB backups that is replicaset aware.

It will only take a backup from a replica so if you have the classic master,replica,arbiter configuration you can setup the script via cron on both (current) master and replica and the backup will only run on the replica.

It will then tar.gz the backup and upload it to Google Storage. It can be easily adapted to upload the backup to S3 using s3cmd or the aws cli (aws-cli).

Cross posted at Forecast:Cloudy (my cloud blog).