I’ve recently had the chance to write a new project on AppEngine.
It’s been a long time since I tried I was too lazy (as always) to setup servers just for that.
I’ve decided to use Python but just to be sure I won’t be vendor locked into various AppEngine services I’ve decided to use:
- Flask (instead of webapp2)
- Cloud SQL (instead of DataStore)
This will ensure that I can break out of AppEngine easily with minimal code changes.
This was the first major Flask project I’ve written and I found its current cookie support a bit lacking compared to Tornado’s secure cookies (I won’t go into the debate of why it should be kept like that and why I’m not using a session cookie that points to the real session data somewhere else).
I’ve decided to create a small module to add Tornado’s secure cookie support into Flask.
It’s basically a modified version of the current Tornado Secure Cookie code and its quite easy to use in Flask as well.
Grab it and share your comments and opinions. It’s also available on PyPI under the name “flask-secure-cookie“.
In addition to my previously published (very early) project to stream NSQ messages directly to BigQuery, I am happy to presents a modified version of nsq-to-s3 that supports streaming NSQ messages directly Google Cloud Storage.
Grab it while its hot from the nsq-to-gs repo.
I do see a future for a merged version of these two projects that supports both S3 and Google Cloud Storage but this would have to be enough for now.
The current version has the same functionality as the latest nsq-to-s3 version and was adapted to support Google Storage with minor modifications (such as the default path and filename formats).
I’ve bene running a Tor exit node in the Netherlands since August 2013. I believe in the cause of Tor and it was only a matter of time before I started adding code in some for or another.
gonionoo is Go wrapper for OnionOO – the Tor Network Status protocol as is the first step in a slightly larger project I’m working on that I’ve been planning for a while ever since I’ve became a Tor exit node operator.
The OnionOO API has lots of interesting data on the Tor network. You can see it visualized as part of the Atlas project.
I’ve created a nice little bash script to take MongoDB backups that is replicaset aware.
It will only take a backup from a replica so if you have the classic master,replica,arbiter configuration you can setup the script via cron on both (current) master and replica and the backup will only run on the replica.
It will then tar.gz the backup and upload it to Google Storage. It can be easily adapted to upload the backup to S3 using s3cmd or the aws cli (aws-cli).
Cross posted at Forecast:Cloudy (my cloud blog).
One of the things I found very strange is the fact that most operations that came with iOS prior iOS 5 which revolved around UIImage didn’t take into account the orientation of the image. This meant that if you want to read a picture from the camera roll and resize it, you’d have to roll your own code to correctly flip and/or rotate the image according to its orientation value.
Being my lazy self I used the fine code of Trevor Harmon in UIImage+Resize. Trevor added some categories to make handling UIImage a bit nicer. The code takes create of everything including orientation.
My app worked great on iOS 4 and early betas of iOS 5, however in the late beta of iOS 5 and in the release it wrongfully rotated the images.
After further investigation it seems iOS 5 already rotates the image correctly. UIImage+Resize rotated it again, causing the images to get skewed.
A quick fix would simply avoid the transposition code in UIImage+Resize.
Since the code ran perfectly fine in iOS 4, for backwards compatibility I added a check for OS version and for anything below 5.0 the old code would work.
Check out this gist:
For better performance I would store a boolean flag somewhere in the app saying you are running in iOS 5 and check that instead of keep on checking the OS version every run, but this is just to get you started.
I had to backup an S3 bucket so I whiped out a small script to clone a bucket.
It’s written in Python and depends on the excellent Boto library. If you are running Python < 2.7 you’ll also need the argparse library (both available also via pip).
View the gist here: https://gist.github.com/1275085
Or here below:
A while back Twitter announced the Snowflake service. Snowflake is a unique ID generator that is fast and generate 64bit integer unique ids that are “roughly sortable”. That is, newer ids are bigger than older ones, up to a certain point.
The service was originally written in Scala (which runs on the JVM) and has a Thrift interface, which means you can talk to it from almost any thinkable programming language.
The project was shared on GitHub.
Personally, I don’t really like the JVM. It’s rather bloated in memory terms and can make quite a mess when you need to fine tune it to low memory environments. Also, the Snowflake service code is rather simple and rarely allocate a lot of new objects, which means allocation wise, its rather fixed.
I’ve re-implemented the service in Python using the same Thrift interfaces for both testing as well as being able to run it on low memory environments without the need to fine tune the JVM.
This implementation is rather naive and doesn’t work too much around CPython’s Global Interpeter Lock (GIL) so it yields much less IDs per second than the Scala implementation, however you can compensate for it by running multiple processes.
You can grab the service code from here: https://github.com/erans/pysnowflake
I’ve also written a very simple Python client (it should support connecting to multiple Snowflake services, but the current version disregards this) which I only tested with PySnowflake (the Python server I created). I didn’t test it against the original Scala service.
You can grab the Python client code here: https://github.com/erans/pysnowflakeclient
While I do use some of this code in production, it is far from being fully tested and checked and I would use it as a reference or study it well and load test it before deploying it.
For my latest venture, MyFamilio, I needed to know if a user’s Email address is a Gmail one so that I could show the user his/her contacts from Gmail.
Figuring out if the user is on Gmail is usually easy – the Email ends with @gmail.com. But what happens for all of those Google Apps for Your domain (like my own, which uses the @sandler.co.il domain) ?
Well, you can easily detect that by running a DNS query on the MX record.
I wrote a small function in Python which uses dnspyhon to do just that, determine if an Email address is hosted on Gmail or not.
Check the gist here.
Check the gist here.
I was searching an example of using Python Imaging Library (PIL) to extract the GPS data from EXIF data in images.
There were various half baked examples that didn’t handle things well, so I baked something of my own combining multiple examples.
You can get it here: https://gist.github.com/983821
Or see it embedded below:
I know it took me a while (sorry) but I had a couple things on my plate.
At first I wanted to release a more complete integration of OAuth within ASP.NET, but that will have to wait to the next time frame I can allocate to work on this.
In the meantime, there is some basic C# code in the OAuth code repository which generates the OAuth signature, which is the most complicated thing to implement in the spec (not that it’s that difficult to implement :-) It’s actually quite easy).
To use the C# code, simply do this (based on the samples in the spec):
OAuthBase oauth = new OAuthBase();
Uri url = new Uri(“http://photos.example.net/photos?file=vacation.jpg&size=original”);
string signature = oauth.GenerateSignature(url, “dpf43f3p2l4k3l03”, “kd94hf93k423kf44”, “nnch734d00sl2jdk”, “pfkkdhi9sl3r4s00”, “GET”, oauth.GenerateTimeStamp(), oauth.GenerateNonce(), OAuthBase.SignatureTypes.HMACSHA1);
After that you can concatenate the relevant query parameters as well as the signature value to the URL and use it.
If you have a different timestamp and/or nonce generation method, you can inherit and override these methods.
If you require a different hashing algorithm other than the default HMAC-SHA1 or the PLAINTEXT (which MUST be used with a secure communication channel such as HTTPS) you can use the “GenerateSignatureBase” method to generate the signature base string and then call “GenerateSignatureUsingHash” passing the signature base and the hash algorithm you are using.
That’s about it. I’ll update when I’ll have some more integrative code.