Scott Berkun’s Mindfire: Big Ideas For Curious Minds – Book Review

I had the pleasure of reading Scott Berkun‘s newest book – Mindfire: Big Ideas for Curious Minds. I was also forunate to get it for free in the short period of time where Scott gave it for free on his site, but this is not a guilty book review of getting the book for free.

Mindfire is a collection of 30 essays which Scott wrote in various places, mostly on his blog. The essays got cleaned up and preped for the book which made the reading very clean and flowing. Scott’s writing style is very flowing and funny and while it may seem at times as a self emporment / self help book it really isn’t.

I look at it more as a collection of percise and clear set of obersvations on the human condition and behavior alone and in a group. Some of the essays specifically talk about work related situtations, but in most cases  you can apply some of the tips and wisdom of this book to almost any interaction with other people.

I really enjoyed reading it and in some situations fully sympathize with the eassy’s topic and resolution.

UIImage in iOS 5, Orientation and Resize

One of the things I found very strange is the fact that most operations that came with iOS prior iOS 5 which revolved around UIImage didn’t take into account the orientation of the image. This meant that if you want to read a picture from the camera roll and resize it, you’d have to roll your own code to correctly flip and/or rotate the image according to its orientation value.

Being my lazy self I used the fine code of Trevor Harmon in UIImage+Resize. Trevor added some categories to make handling UIImage a bit nicer. The code takes create of everything including orientation.

My app worked great on iOS 4 and early betas of iOS 5, however in the late beta of iOS 5 and in the release it wrongfully rotated the images.

After further investigation it seems iOS 5 already rotates the image correctly. UIImage+Resize rotated it again, causing the images to get skewed.
A quick fix would simply avoid the transposition code in UIImage+Resize.

Since the code ran perfectly fine in iOS 4, for backwards compatibility I added a check for OS version and for anything below 5.0 the old code would work.
Check out this gist:

For better performance I would store a boolean flag somewhere in the app saying you are running in iOS 5 and check that instead of keep on checking the OS version every run, but this is just to get you started.

Clone S3 Bucket Script

I had to backup an S3 bucket so I whiped out a small script to clone a bucket.

It’s written in Python and depends on the excellent Boto library. If you are running Python < 2.7 you’ll also need the argparse library (both available also via pip).

View the gist here: https://gist.github.com/1275085

Or here below:

“Those who don’t know history are destined to repeat it.”

I know the title is a bit alarming, but that was my first thought after reading @bryce‘s latest post “The Rising Generation“.

Briefly, he mentioned a 25 y/o asking a question on Quora about how was life before everyone had a cell phone and no one talked a lot or texted in public areas. Bryce also say that the new entrepreneurs, like the ones in yesterday’s Y Combinator Demo Day have different expectation, understanding and perceived value of technology than any other that has come before them.

I’m 30. Not too far from the 25 y/o who asked the question (although I was about 15 when cell phone started to penetrate Israel quickly and spread like wild fire). In most companies I worked at I was usually the youngest (or second youngest) in the company for quite some time. Most of my co-workers used to be (or are) between 9-15 years older than me and are still surprised of my knowledge of things from the past.

I’m a bit of a history buff when it comes to computers, science and technology (but also to general history). That’s why I do know what a ZX Spectrum is (and I don’t know how it reached my house when I was about 7 but I had the chance to play with it). I know how VMS systems work (long story form the Army :-) ). When I was six I did play with an Apple IIc my brother got for his Bar Mitzvah and really liked Captain Goodnight and Karateka. I do know how a modem sounds (and can even detect by sound what is the connection speed. Tiiii Taaa TiiTaTa TiiiTa – Yay 28KBPS! ). Heck, I even ran a BBS and was a node on FidoNet when I was 14.

That might be a bit unusual for most people my age that are into computers (and maybe even for older ones) but that’s not too different from people that hear Pink Floyd, Led Zeppelin, Bach and Mozart. While these composers and bands did not exist when I was born or when I was a child the music kept on going. I and a lot of other people both older and younger do know it, hear it and enjoy it. It even inspires some to go on and create new things.

Knowing a bit of the past and how it related to an idea or thought you have can give you a much better appreciation to the things others have done or to the things others advice you to do. After all, specifically in the computers and internet industry, we are all “standing on the shoulders of giants”. It can also give entrepreneurs some much needed perspective on how things were, how things are and how it should be.

So, in addition to Bryce’s hope that “they can rise high enough to meet the emerging opportunities generations before them have made possible”, my 2 cents are to also learn a bit from the past. Take a minute for some history and try to figure out how things were before you push on forward. If they won’t they are destined to repeat it.

Python Implementation of Twitter’s Snowflake Service

A while back Twitter announced the Snowflake service. Snowflake is a unique ID generator that is fast and generate 64bit integer unique ids that are “roughly sortable”. That is, newer ids are bigger than older ones, up to a certain point.
The service was originally written in Scala (which runs on the JVM) and has a Thrift interface, which means you can talk to it from almost any thinkable programming language.

The project was shared on GitHub.

Personally, I don’t really like the JVM. It’s rather bloated in memory terms and can make quite a mess when you need to fine tune it to low memory environments. Also, the Snowflake service code is rather simple and rarely allocate a lot of new objects, which means allocation wise, its rather fixed.

I’ve re-implemented the service in Python using the same Thrift interfaces for both testing as well as being able to run it on low memory environments without the need to fine tune the JVM.

This implementation is rather naive and doesn’t work too much around CPython’s Global Interpeter Lock (GIL) so it yields much less IDs per second than the Scala implementation, however you can compensate for it by running multiple processes.

You can grab the service code from here: https://github.com/erans/pysnowflake

I’ve also written a very simple Python client (it should support connecting to multiple Snowflake services, but the current version disregards this) which I only tested with PySnowflake (the Python server I created). I didn’t test it against the original Scala service.

You can grab the Python client code here: https://github.com/erans/pysnowflakeclient

While I do use some of this code in production, it is far from being fully tested and checked and I would use it as a reference or study it well and load test it before deploying it.

 

Determine if an Email address is Gmail or Hosted Gmail (Google Apps for Your Domain)

For my latest venture, MyFamilio, I needed to know if a user’s Email address is a Gmail one so that I could show the user his/her contacts from Gmail.

Figuring out if the user is on Gmail is usually easy – the Email ends with @gmail.com. But what happens for all of those Google Apps for Your domain (like my own, which uses the @sandler.co.il domain) ?

Well, you can easily detect that by running a DNS query on the MX record.

I wrote a small function in Python which uses dnspyhon to do just that, determine if an Email address is hosted on Gmail or not.

Check the gist here.

Check the gist here.

Forecast: Cloudy – My New Cloud Related Technical Blog

I’ve started a new technical blog which talks about the cloud. It’s called Forecast: Cloudy and it will feature thoughts, ideas and some code driven mostly from my experience running services on cloud infrastructure.

The first post (after the traditional “Hello World“) already has some code :-)

Check it out and don’t forget to tell me what you think about it.

Extract GPS Latitude and Longitude Data from EXIF using Python Imaging Library (PIL)

I was searching an example of using Python Imaging Library (PIL) to extract the GPS data from EXIF data in images.

There were various half baked examples that didn’t handle things well, so I baked something of my own combining multiple examples.

You can get it here: https://gist.github.com/983821

Or see it embedded below:

Twitter’s Kestrel init script for Ubuntu 10.04 Lucid

Twitter’s Kestrel is a cool scala based queue server (based on Blaine Cook‘s (@blaine) Ruby based Sterling).

The two main features I like about Kestrel are:

  • Sort-of-transactional – If I take an item I can make sure others can’t get it. If the connect drops it will go back on the queue.
  • Read behind mode – If a certain queue reached a maximum pre-configured amount of RAM or items it will stop storing messages in RAM and will write it directly to the queue log file.

If you happen to be running it on Ubuntu 10.04 and want to use the provided init script (kestrel.sh) you’ll notice that it just won’t run.

Below is a link to an updated script that works on my Ubuntu 10.04 installations.

Assumptions:

  • You are using the paths as suggested in the init script (/usr/local/$APP-NAME, /var/log/$APP-NAME /var/run/$APP-NAME, etc)
  • You are using the openjdk-6-jre-headless package. The script defines the JAVA_HOME directly to where Ubuntu installs that package. You can remove it if needed or change it to suitable values
  • You’ll need to update the JAVA_OPTS parameters for RAM (xmx, etc) for your needs
  • You’ll need to download, compile and install “daemon” from here. It supports Ubuntu 10.04. I don’t know if it has a package, but it was too damn easy to compile it on my own and use it.

Download the script


PPTP VPN on Ubuntu 10.04 for your iPhone / iPad

Below are the steps necessary to connect your iPhone / iPad or any other computer via a PPTP VPN.

Why would I want to do this? For various reasons such as allow you to access information and servers that are behind a firewall, or maybe you just need to route traffic through different servers.

I’ve tested this on a 256mb Rackspace Cloud instance running Ubuntu 10.04 and with an iPhone and an iPad. Thanks to Yaniv for debugging the instructions.

Disclaimer: This is for educational uses only and I take no responsibility as to what you may do with it. The PPTP VPN setup via the instructions below has no encryption and uses the simplest and lowest form of password authentication. If you require stricter encryption and authentication methods you’ll need to read more about pptpd configuration.

Assumptions:

  • The instance you are using is blank, specifically from firewall rules in iptables, otherwise, you’ll need to patch things up.
  • All commands assume you are current a root user. If you logged in as root, that’s great. If not, run:
    sudo su
  • Instead of messing a lot with iptables commands, I’m using ufw (Uncomplicated FireWall). In general, to most people, it will be easier to manage and work with.

Setting up the PPTP Server

In general we are going to create a PPTP VPN that is very basic without encryption and with basic authentication security (not fancy authentication protocols). Since Rackspace Cloud instance has an external interface (eth0) that has the instance public IP and an internal interface (eth1) with an internal IP used to communicate with your other Rackspace Cloud server (if you have them), we’ll create an alias network interface card that will have some other set of internal ips, which will be given to the devices connected via the VPN.

  1. Install the necessary software (pptpd, pptp-linux, ppp and ufw – for firewall):
    apt-get install pptpd pptp-linux ppp ufw
  2. Enable port 22 (ssh) in the firewall, so we don’t get locked out of our instance:
    ufw allow 22
  3. Enable port 1723 (pptpd) in the firewall to enable access to the pptpd dameon:
    ufw allow 1723
  4. Enable ufw:
    ufw enable
  5. Add an aliased network interface card (eth0:0): (We use the address space of 192.168.88.0/24 since its usually free for most networks for most users. You can feel free to change this address if it is already taken)
    Edit /etc/network/interfaces:

    nano /etc/network/interfaces

    Enter the following text at the end of the file:

    auto eth0:0
    iface eth0:0 inet static
    address 192.168.88.1
    netmask 255.255.255.0
    gateway (same value as listed for eth0)
    dns-nameservers (same value as listed for eth0)

    Replace the value of “gateway” with the same value you will see in this file for “eth0”, the real public network interface.
    Replace the value of “dns-nameservers” with the same value you will see in this file for “eth0”

  6. Configure the pptpd daemon:
    Edit /etc/ppp/pptpd-options:

    nano /etc/ppp/pptpd-options

    Comment out (add a “#” char at the start of the line) the following lines:
    “refuse-pap”
    “refuse-chap”
    “refuse-mschap”
    “refuse-mschap-v2”
    “require-mppe-128″replace “#ms-dns 10.0.0.1” with “ms-dns 8.8.8.8”
    replace “#ms-dns 10.0.0.2” with “ms-dns 8.8.4.4”

    The last 2 lines above sets the DNS server the devices connecting to your PPTP VPN will use. The addresses above are for the Google Public DNS server, but can be any other DNS server (including the same DNS servers as Rackspace or your hosting provider use)

    Edit /etc/pptpd.conf :

    nano /etc/pptpd.conf

    Add at the bottom of the file:

    localip 192.168.88.1
    remoteip 192.168.88.2-20

    The value of “remoteip” will be the set of IP addresses the devices connecting to the VPN will get upon successful connection. Currently, we have here 18 addresses, which is enough for 18 concurrent devices. You can make this range bigger if needed.

  7. Configure the username and password that will be used to authenticate client accessing the VPN:
    Edit /etc/ppp/chap-secrets:

    nano /etc/ppp/chap-secrets
    # client server secret IP addresses
    [UserName] pptpd [Password] *

    Replace [UserName] with the username you wish to use.
    Replace [Password] with the password you wish to use (I suggest a long random password. Try this generator)

  8. Enable IP forwarding in the kernel:
    Edit /etc/sysctl.conf :

    nano /etc/sysctl.conf

    Uncomment the line “net.ipv4.ip_forward=1”
    For IPv6, uncomment “net.ipv6.conf.all.forwarding=1”

  9. Enable IP forwarding in ufw:
    Edit /etc/default/ufw:

    nano /etc/default/ufw

    Change the value of “DEFAULT_FORWARD_POLICY” from “DROP” to “ACCEPT”

  10. Add IP masquerading rule in ufw, so that NAT will work and devices connecting to the VPN will be seen as if the traffic goes out of the VPN server:
    Edit /etc/ufw/before.rules:

    nano /etc/ufw/before.rules

    Paste the text below after the header and before the “*filter” rules:

    # nat Table rules
    *nat
    :POSTROUTING ACCEPT [0:0]

    # Allow forward traffic from eth0:0 to eth0
    -A POSTROUTING -s 192.168.88.0/24 -o eth0 -j MASQUERADE

    # don’t delete the ‘COMMIT’ line or these nat table rules won’t be processed
    COMMIT

  11. Reboot the machine, cross your fingers and hope for the best :-)

Configuring your iPhone / iPad

  1. In your iPhone / iPad go to “Settings” -> “General” -> “Network” -> “VPN”
    PPTP VPN Configuration
  2. Select “Add VPN Configuration”
  3. Select “PPTP”
  4. In “Description” enter the name of the VPN connection
  5. In “Server” enter the IP address of the server (or a server name, if you mapped the server’s IP address to a domain name)
  6. In “Account” enter the username you have entered into the “/etc/ppp/chap-secrets” file
  7. In “Password” enter the password you entered for the above username in “/etc/ppp/chap-secrets”
  8. Make sure “Send All Traffic” is turned to “ON”
  9. Set “Encryption Level” to “None” (this is how we configured the PPTP server in this post, if you setup an encryption try to keep it in “Auto”
  10. Select save