---
title: "nsq-to-bigquery – Stream messages from NSQ directly to Google BigQuery"
description: "<p><a href="https://i1.wp.com/eran.sandler.co.il/wp-content/uploads/2015/11/nsq-to-bigquery.png"><img class="size-medium wp-image-414 aligncenter" src="https://i0.wp.com/eran.sandler.co.il/wp-content/uploads/2015/11/nsq-to-bigquery-300x117.png?fit=300%2C117" alt="nsq-to-bigquery" srcset="https://i1.wp.com/eran.sandler.co.il/wp-content/uploads/2015/11/nsq-to-bigquery.png?resize=300%2C117 300w, https://i1.wp.com/eran.sandler.co.il/wp-content/uploads/2015/11/nsq-to-bigquery.png?w=328 328w" sizes="(max-width: 300px) 100vw, 300px" data-recalc-dims="1" /></a></p>
<p>In the spirit of nsq-to-XXX such as nsq-to-http and nsq-to-file – I bring you the very first version of <a href="https://github.com/erans/nsq-to-bigquery">nsq-to-bigquery</a>.</p>
<p>nsq-to-bigquery, as the name suggest, streams data from an NSQ channel into Google’s BigQuery using the <a href="https://cloud.google.com/bigquery/streaming-data-into-bigquery">Streaming API</a> and provide very effective means to stream data that should be then further analysed and aggregated by BigQuery’s excellent performance.</p>
<p>This is a (very) initial version so it has some limitations and assumptions.</p>"
doc_version: "1"
last_updated: "2017-06-02"
date: 2015-11-16
tags: [bigquery, Go, golang, google bigquery, nsq]
canonical: "https://eran.sandler.co.il/2015/11/16/nsq-to-bigquery-stream-messages-from-nsq-directly-to-google-bigquery/"
---

## Sitemap

- [Home](https://eran.sandler.co.il/)

[<img class="size-medium wp-image-414 aligncenter" src="https://i0.wp.com/eran.sandler.co.il/wp-content/uploads/2015/11/nsq-to-bigquery-300x117.png?fit=300%2C117" alt="nsq-to-bigquery" srcset="https://i1.wp.com/eran.sandler.co.il/wp-content/uploads/2015/11/nsq-to-bigquery.png?resize=300%2C117 300w, https://i1.wp.com/eran.sandler.co.il/wp-content/uploads/2015/11/nsq-to-bigquery.png?w=328 328w" sizes="(max-width: 300px) 100vw, 300px" data-recalc-dims="1" />][1]

In the spirit of nsq-to-XXX such as nsq-to-http and nsq-to-file &#8211; I bring you the very first version of [nsq-to-bigquery][2].

nsq-to-bigquery, as the name suggest, streams data from an NSQ channel into Google&#8217;s BigQuery using the [Streaming API][3] and provide very effective means to stream data that should be then further analysed and aggregated by BigQuery&#8217;s excellent performance.

This is a (very) initial version so it has some limitations and assumptions.

#### Limitations / Assumptions

  * The BigQuery table MUST exist prior to streaming the data
  * The NSQ message being sent MUST be a valid JSON string
  * The JSON format MUST be a simple flat dictionary (key and simple value. Value can&#8217;t be another dictionary or list)
  * The JSON format MUST match the format of the BigQuery table
  * At the moment there is no support for batching so each message will issue an API call to BigQuery with a single line of data

#### Planed Features:

  * Support batching with flushing based on X number of rows or Y amount of time passed since last flush
  * Flushing will happen in parallel with receiving information so there is almost no delay

Stay tuned on the [github repo][2] for more news.

 [1]: https://i1.wp.com/eran.sandler.co.il/wp-content/uploads/2015/11/nsq-to-bigquery.png
 [2]: https://github.com/erans/nsq-to-bigquery
 [3]: https://cloud.google.com/bigquery/streaming-data-into-bigquery