Tag Archives: zmq

Using CruftFlake with ZeroMQ for ID generation instead of Twitter Snowflake

What is CruftFlake?

CruftFlake is essentially a PHP version of Twitter’s Snowflake. However, rather than using Apache Thrift, CruftFlake uses ZeroMQ.

Snowflake and CruftFlake are both used for generating unique ID numbers at high scale. In large, scalable systems – you tend to move away from the likes of MySQL (with its ever so lovely auto-increment), and move to NoSQL solutions such as MongoDB, CouchDB, Redis, Cassandra and Hadoop/Hbase.

There are many database solutions to address the problem of scalability, and one thing that you’ll find yourself needing more often than not: the ability to generate a unique ID – and that’s what CruftFlake is for.

Why?

I quote from my PHP UK Conference review post:

“If you use technology that was designed to be resilient, and then build your application atop of that with resilience in mind, then there is a very good chance that you app will also be resilient.”

To be fair, this is more about scalability than resilience, but one could argue they go hand in hand. Point being that you can’t rely on a single auto increment value if you have a 100 database servers; it would be… disgusting.

Installing

ZeroMQ

I’m still running things locally, so the installation below will assuming you’re running Mountain Lion.

First things first: Sadly, Mountain Lion and the new version of Xcode doesn’t appear to ship with Autoconf; so you’ll have to install it:

$ cd ~
$ mkdir src
$ cd src
$ curl -OL http://ftpmirror.gnu.org/autoconf/autoconf-latest.tar.gz
$ tar xzf autoconf-latest.tar.gz
$ cd autoconf-*
$ ./configure --prefix=/usr/local
$ make
$ sudo make install

Now that should have sorted that issue out!

Next, you’ll want to install ZeroMQ:

$ cd ~/src
$ curl -v http://download.zeromq.org/zeromq-3.2.2.tar.gz > zeromq.tar.gz
$ tar xzf zeromq.tar.gz
$ cd zeromq-*
$ configure
$ make
$ make install

After that you’ll need to install the PHP bindings:

$ sudo pear channel-discover pear.zero.mq
$ sudo pecl install pear.zero.mq/zmq-beta
$ sudo echo 'extension=zmq.so' >> /etc/php.ini

Verify the install:

$ php -i | grep libzmq
libzmq version => 3.2.2

If you’ve managed that with minimal effort then give yourself a huge pat on the back! 🙂

CruftFlake

Now here comes the easy part. I forked davegardnerisme/cruftflake over to my posmena/cruftflake repo and added in some composer love. I should really do a pull request and maybe @davegardnerisme will permit 🙂

Anyway, create a new folder called cruftflake and create a file called composer.json with the following:

{
    "minimum-stability": "dev",
    "require": {
        "posmena/cruftflake": "*"
    }
}

Then:

$ composer install

Generating an ID

Ready for the clever bit? Open two terminals, both in the cruftflake directory. In one of them, do:

$ php vendor/posmena/cruftflake/scripts/cruftflake.php
Claimed machine ID 512 via fixed configuration.
Binding to tcp://*:5599

That will set the service running. So, now if you want an id – you just go to the other window and type:

$ php vendor/posmena/cruftflake/scripts/client.php
153291408887775232

How’s that for a nice unique number? That’s using the system time, the configured machine id, in addition to a sequence number.

To show how fast it is, you can generate 10,000 in less than 2 seconds:

$ php vendor/posmena/cruftflake/scripts/client.php -n 10000

The duration it takes will of course depend on your server, but don’t forget that this is per process. You can have as many of these running as you wish!

Summary

Easy, right?!

I’m sure that as you’ve been following the tutorial, you’ve been looking at the code and seeing what it’s doing. You will see how ZeroMQ is set up as well as how the generator works.

You may notice that I’ve elected to skip using the ZooKeeper configuration. The reason for this is that ZooKeeper is for running multiple nodes; you don’t need multiple nodes for a quick demo!

I’ve found CruftFlake to be a really neat tool. It’s very much overkill for small projects, but the whole point is the play around with this stuff so you are aware of the scalable solutions out there.

Thanks to @davegardnerisme for letting me fork – if I do issue a pull request, I will be sure to update this post accordingly.

I shall definitely be blogging soon when implementing this into a real-world scenario. Stay tuned!

Share