Tag Archives: twitter

Using CruftFlake with ZeroMQ for ID generation instead of Twitter Snowflake

What is CruftFlake?

CruftFlake is essentially a PHP version of Twitter’s Snowflake. However, rather than using Apache Thrift, CruftFlake uses ZeroMQ.

Snowflake and CruftFlake are both used for generating unique ID numbers at high scale. In large, scalable systems – you tend to move away from the likes of MySQL (with its ever so lovely auto-increment), and move to NoSQL solutions such as MongoDB, CouchDB, Redis, Cassandra and Hadoop/Hbase.

There are many database solutions to address the problem of scalability, and one thing that you’ll find yourself needing more often than not: the ability to generate a unique ID – and that’s what CruftFlake is for.

Why?

I quote from my PHP UK Conference review post:

“If you use technology that was designed to be resilient, and then build your application atop of that with resilience in mind, then there is a very good chance that you app will also be resilient.”

To be fair, this is more about scalability than resilience, but one could argue they go hand in hand. Point being that you can’t rely on a single auto increment value if you have a 100 database servers; it would be… disgusting.

Installing

ZeroMQ

I’m still running things locally, so the installation below will assuming you’re running Mountain Lion.

First things first: Sadly, Mountain Lion and the new version of Xcode doesn’t appear to ship with Autoconf; so you’ll have to install it:

$ cd ~
$ mkdir src
$ cd src
$ curl -OL http://ftpmirror.gnu.org/autoconf/autoconf-latest.tar.gz
$ tar xzf autoconf-latest.tar.gz
$ cd autoconf-*
$ ./configure --prefix=/usr/local
$ make
$ sudo make install

Now that should have sorted that issue out!

Next, you’ll want to install ZeroMQ:

$ cd ~/src
$ curl -v http://download.zeromq.org/zeromq-3.2.2.tar.gz > zeromq.tar.gz
$ tar xzf zeromq.tar.gz
$ cd zeromq-*
$ configure
$ make
$ make install

After that you’ll need to install the PHP bindings:

$ sudo pear channel-discover pear.zero.mq
$ sudo pecl install pear.zero.mq/zmq-beta
$ sudo echo 'extension=zmq.so' >> /etc/php.ini

Verify the install:

$ php -i | grep libzmq
libzmq version => 3.2.2

If you’ve managed that with minimal effort then give yourself a huge pat on the back! 🙂

CruftFlake

Now here comes the easy part. I forked davegardnerisme/cruftflake over to my posmena/cruftflake repo and added in some composer love. I should really do a pull request and maybe @davegardnerisme will permit 🙂

Anyway, create a new folder called cruftflake and create a file called composer.json with the following:

{
    "minimum-stability": "dev",
    "require": {
        "posmena/cruftflake": "*"
    }
}

Then:

$ composer install

Generating an ID

Ready for the clever bit? Open two terminals, both in the cruftflake directory. In one of them, do:

$ php vendor/posmena/cruftflake/scripts/cruftflake.php
Claimed machine ID 512 via fixed configuration.
Binding to tcp://*:5599

That will set the service running. So, now if you want an id – you just go to the other window and type:

$ php vendor/posmena/cruftflake/scripts/client.php
153291408887775232

How’s that for a nice unique number? That’s using the system time, the configured machine id, in addition to a sequence number.

To show how fast it is, you can generate 10,000 in less than 2 seconds:

$ php vendor/posmena/cruftflake/scripts/client.php -n 10000

The duration it takes will of course depend on your server, but don’t forget that this is per process. You can have as many of these running as you wish!

Summary

Easy, right?!

I’m sure that as you’ve been following the tutorial, you’ve been looking at the code and seeing what it’s doing. You will see how ZeroMQ is set up as well as how the generator works.

You may notice that I’ve elected to skip using the ZooKeeper configuration. The reason for this is that ZooKeeper is for running multiple nodes; you don’t need multiple nodes for a quick demo!

I’ve found CruftFlake to be a really neat tool. It’s very much overkill for small projects, but the whole point is the play around with this stuff so you are aware of the scalable solutions out there.

Thanks to @davegardnerisme for letting me fork – if I do issue a pull request, I will be sure to update this post accordingly.

I shall definitely be blogging soon when implementing this into a real-world scenario. Stay tuned!

Share

Hacking TweetMyBlog (TwitterFeed Sucks)

I used to use TwitterFeed; but recently it’s been a real pain in the ass and not updating so I looked for an alternative.

Roll on TweetMyBlog.

TweetMyBlog is great for the simple fact that it is a WordPress plugin. It’s also a small download, and you can install it very easily. However, TweetMyBlog is hard-coded to use TinyURL and the URL it “makes tiny” is an affiliate link taking the user to a framed page. This is a bit of a bummer for me because I like to use bit.ly and I don’t really want my users going to some 3rd party site.

Roll on TweetMyBlog Hack

I took at a look @ the files and realise we can very easily change the Plugin to use bit.ly instead of TinyURL using the bit.ly API.

If you have installed TweetMyBlog you would have uploaded the “tweetmyblog” directory which had 3 files in it. Go and find, “twitter_api.php”:

and at the very bottom of the file you will see:

function get_tiny_url($url)
{
$this->snoopy->fetch('http://tinyurl.com/api-create.php?url='.$url);
return $this->snoopy->results;
}


As you can see, it uses TinyURL.

I’ve only hacked it to use bit.ly, but this is what I changed it to first (don’t copy it yet, there’s a better version further down):

   function get_tiny_url($url)
	{
    $this->snoopy->fetch('http://api.bit.ly/shorten?version=2.0.1&login=bobbyjason&apiKey=MYLONG_BIT_LY_API_KEY&format=text&longUrl='.$url);
	 return $this->snoopy->results;
	}


I gave it a shot, and sure enough it worked, I was using bit.ly instead of tinyurl, result! However, it was still using the silly Affiliate link with a frame & 3rd party site.

Okay, open “tweetmyblog.php” at around line 36 you will see:

$tweet = trim($tweetPrefix).' '.$blogPost->post_title.' '.$t->get_tiny_url('http://www.tweetmyblog.com/bpredir.php?r='.$tmb_affid.'&bp='. urlencode(get_permalink($postID)));

The problem is the tweetmyblog.com redirection, so I replaced it with this:

$tweet = trim($tweetPrefix).' '.$blogPost->post_title.' '.$t->get_tiny_url(urlencode(get_permalink($postID)));

Easy, right?

Well, this works great.

I do not need to use TwitterFeed ever again because WordPress automatically Tweets my blog post! I don’t have to wait 30 minutes, 1 hour, 5 hours for TwitterFeed to notice my blog post.

There was one last change that I made; after using the code above (using the bit.ly api) I realised that the URL wasn’t showing up in my bit.ly history. I checked this out and realised I needed to use the history=1 parameter. After I added this I’ve not had a single problem at all.
For the final code of “twitter_api.php”, see below:

function get_tiny_url($url)
	{
    $this->snoopy->fetch('http://api.bit.ly/shorten?version=2.0.1&history=1&login=bobbyjason&apiKey=MYLONG_BIT_LY_API_KEY&format=text&longUrl='.$url);
	 return $this->snoopy->results;
	}

I must say though; TweetMyBlog is a great plugin by John Merrick and Sean Jordansen and you should install it.

If you wish to monetise your tweets from your blog, then the above edits are silly. It’s just personally I would rather not have the user taken to a 3rd party site.

TweetMyBlog also provides a feed of your Tweets as a Widget, so it really is great!

Download TweetMyBlog now!

Share

My 7 Must-Have Plugins for WordPress

I make sure I use these on ALL my WordPress blogs. I use many more, but this is a list of plugins that I think anyone who is serious about maximising the potential of their blog should be using.

All in One SEO Pack
Google Analytics for WordPress
Google XML Sitemaps
TweetMyBlog!
wp-cache
Add to Any: Share/Bookmark/Email Button
YARPP

If you have others that you believe are essnetial and should be mentioned then leave a comment and I shall consider updating this list.

Share

8 Simple Reasons You Should Signup to Twitter

This is Step 2 of the “How to get your site into Google” post

The other day I was using Twitter and caught a glimpse that Katie Price & Peter Andre had split up. I Googled for it and couldn’t find any news about it. What was splashed on newspapers the next day? Their divorce.

Twitter BEAT Google. Now that’s serious because many folk use Google is their primary source of finding information and suddenly you have this website [twitter] with instant news!

If that doesn’t convince you, let me list some reasons why you should sign up:

1. You want to keep “connected” with like-minded individuals. Twitter is great for this.
2. It quick – to post an update it takes less than 30 seconds. A few updates a day with latest news and you’re sorted.
3. It’s easy. Really, it is.
4. It’s viral. Do it right and your news can be splashed around the net in a matter of hours, if not minutes.
5. It integrates with everyday tools like WordPress. You can even post from messenger & your mobile.
6. It’s live. The information on here is real-time news.
7. Social – it’s the easiest way to tell folks about your latest blog post and get those important comments.
8. And… well: it’s free; in this current climate you should grab everything that’s free!

Take all the points above, and you have a serious tool: its the mobile web; in real-time.

Sign Up Now!

Share

How to start with nothing and be in Google in less than 72 hours

My website is listed in Google 6 hours after domain name setup. Impressed?

That’s my whole aim. I want to show you how to push boundaries. I want to show you how to speed up the things that normally take a longer that they should. If you don’t know how – then it’s bound to take a longer than it should.

To show you my feat; I’m going to break down the 2 posts that I’ve made so far (this being the third). I’ll write an individual blog post for each step, but before I do that these are the steps you should be aiming for:

Step 1:

First off, I created a free gmail account. Gmail, or GoogleMail as it’s officially branded here in the UK is a fantastic web client for managing your email. The pioneer of “unlimited” email storage and of course owned by Google.

Step 2:

Signed up to Twitter. Shout at me for jumping on the band-wagon if you wish; but if you want to have a successful “social” site then you need to keep up with the times. In other words; it doesn’t matter if you think it’s a lame idea – the simple fact is you can, and you WILL gain a large audience from using Twitter. Add to that the fact that its free, and the fact its got a great API linking it to every day tools such as WordPress and Facebook; you’d be foolish to ignore it.

Step 3:

Internet Marketing Website. Yes, that’s where you are now. As an internet marketer in the 21st century, having your own website is as important as having an internet connection. If you don’t have your own website then you may as well work away now, or get your own website now! If you want to do what I did, just register the domain name and come back to this later.

Step 4:

Signing up to Squidoo. Not only does Squidoo integrate into Twitter but it’s also a great way to build a strong article in your niche area. As you create a “squidoo lens” it will allow you pull in other bits of data from other websites such as YouTube.

Step 5:

Joining LinkedIn. LinkedIn has been around for some time, it’s got a nice pagerank and its another great way to link your sites together.

Just from the first five steps; you should now have built a strong profile about yourself. LinkedIn with a link to your Internet Marketing Website; Twitter linking to your website (even if it is the registrar holding page). A Squidoo Lens linking to your homepage. Even without even setting up your website, you have built a strong portfolio about yourself and they are all linking back to your homepage.

Step 6:

Using Social Media Tools (SMT) is fun, but it can take longer than necessary without the right apps. An easy way to make your life a lot easier is to find the right apps to help you. Find out how to use TwiveAway, Twhirl, bit.ly, TwitPic and Create an OpenID.

Step 7:

So, you’ve got an arsenal of tools to help you along your way; just on their own these will work wonders for your Internet Marketing. The next step is to create a wordpress blog. This can be seen as the single biggest step. It’s the last step simply because if you make it your first step you’ll most probably be too lazy to go back to the other steps. Why is that? Well it’s because you’ll then spend hours scouring the Internet for worpdress themes. Be honest with yourself!

Step 8:

Another great step to make is to sign up to TwitterFeed using your OpenID. This means you don’t need to spend countless hours getting lost in the world of Twitter. If folk like your posts, they’ll read them, and Re-Tweet your link. As they do that you will gain more followers without even needed to login to Twitter. I’m not suggesting you avoid twitter.com but my point is you can you spend your time focussing on more important things.

That’s where I am now. Okay, I was lazy – I’ve not exactly gone to town on the “filling in my profile information” aspect of signing up to all these Social Media Tools but I’ve put in the basic info.

When did I start? 23:10 on Monday 11th May 2009. That was the exact time I received my welcome email from Google Mail and Did Steps 1-6. I went to bed at 01:30 meaning I spent a grand total of 2hr20 minutes.

I didn’t get chance to get on my laptop on Tuesday; but on Wednesday I had a 3 hour blast from 21:30 to 00:30.

I installed WordPress @ 22:30 last night, and 6 hours later my blog posts had been indexed by Google.

This is my no means a comprehensive list but it should be enough to give you a kick start. I shall start to write a blog post for each step; and edit the post as necessary.

Gracias

Share