Python Implementation of Twitter’s Snowflake Service

A while back Twitter announced the Snowflake service. Snowflake is a unique ID generator that is fast and generate 64bit integer unique ids that are “roughly sortable”. That is, newer ids are bigger than older ones, up to a certain point.
The service was originally written in Scala (which runs on the JVM) and has a Thrift interface, which means you can talk to it from almost any thinkable programming language.

The project was shared on GitHub.

Personally, I don’t really like the JVM. It’s rather bloated in memory terms and can make quite a mess when you need to fine tune it to low memory environments. Also, the Snowflake service code is rather simple and rarely allocate a lot of new objects, which means allocation wise, its rather fixed.

I’ve re-implemented the service in Python using the same Thrift interfaces for both testing as well as being able to run it on low memory environments without the need to fine tune the JVM.

This implementation is rather naive and doesn’t work too much around CPython’s Global Interpeter Lock (GIL) so it yields much less IDs per second than the Scala implementation, however you can compensate for it by running multiple processes.

You can grab the service code from here: https://github.com/erans/pysnowflake

I’ve also written a very simple Python client (it should support connecting to multiple Snowflake services, but the current version disregards this) which I only tested with PySnowflake (the Python server I created). I didn’t test it against the original Scala service.

You can grab the Python client code here: https://github.com/erans/pysnowflakeclient

While I do use some of this code in production, it is far from being fully tested and checked and I would use it as a reference or study it well and load test it before deploying it.

 

Yedda Twitter – Oh the joy!

I just wanted to let all of you know that we just released a new feature on Yedda which integrates nicely with your Twitter account.

You can read the official blog post here.

In a nut shell, upon giving Yedda your twitter username and password, you will be able to share your Yedda expecrience with your Twitter friends.

We will twitt on behalf of you about questions you ask, answers you give, questions you add to your watch list, thumbs up you give to other answers and questions you are being invited to answer by Yedda (all configurable through the settings screen).

Share the Yedda love through Twitter. It’s fun! (and I’m not just saying that because I am a part of Yedda ;-) really!)

Just beware: it’s addictive.

Yedda Twitter .NET / C# Library

This is a bit of shameless promotion but I think it’s worthwhile never the less :-)

One of the things I did lately on my day job (Yedda) was to integrate it with Twitter (check the integration here and add Yedda as your friend!).

Yedda is all about sharing and us sharing things like code with the rest of the world is no exception.

So, without further due, I’m proud to present the Yedda Twitter .NET / C# Library (you will see that it’s more of wrapper than a library… really ;-) ). The post about it in our Dev Blog is here and the details, source and binary are here.

The code is free as in beer and is provided on a “AS IS” basis.

If you have questions about the library, Twitter, C#, .NET, the API, the meaning of life etc, feel free to ask on Yedda.

Twitter and OpenID

Dave Winer says:

“[…] we could make Twitter the open identity system we’ve been looking for. Make your Twitter ID the one that you use to log on to other service […]”

I say let Twitter support OpenID with all of the good Relaying Party Best Practices including (but not limited to):

  • Ability to associate an existing account with an OpenID
  • Ability to switch to another OpenID (sort of a password recovery for OpenID)
  • Ability to create a new account directly with an external (non Twitter) OpenID (be a standard relaying party)

If they want to, they can also be an OpenID provider (which should be good for them, of course ;-) ).