Category Archives: PHP

PHP Tutorials

PHP UK Conference 2013

PHP UK Conference 2013

Sat in a Starbucks in St Pancras station waiting for my train back to Leeds after a great time at the PHP UK Conference – I figured I would use this opportunity to blog about my thoughts of the talks instead of sulking with my man flu. I’m not going in to great detail – just a quick overview!

NOTE: I don’t have the links to the talks yet but I’ve linked to the slides where possible.

The Talks

Fridays Keynote – You Are A Designer

Friday’s opening keynote from Aral Balkan great – I don’t want to play down the rest of the talks but in my opinion this was one of, if not the best talks of the conference. Aral reminded us that user experience should not be an afterthought, but something that is essential at the beginning of the project, right through to completion. There were some references to the genius of the late Steve Jobs, which were a great reminder of his excellent design philosophy – a great example being iDVD which Steve Jobs completely ditched the previous version (from an acquisition) and requested a very simple ‘burn DVD’ button.

It was a really entertaining presentation and I massively recommend you watch the recording – I shall link to it here once the folks at PHPUK have figured out how to upload YouTube videos 🙂 You can checkout his website

Event Stream Processing In PHP

I was sure Ian Barber would do a great job telling us about React PHP, especially after seeing his talk on ZMQ at PHPUK2011. I was right.

For those who don’t know, ReactPHP could be summarised as ‘node for PHP’. Ian had some great examples showing what sort of real world applications you could apply this to – by coding as he spoke. This made the talk much more effective as you could see exactly what he was trying to show. We learned about various ways in which you could set up an event stream for varying purposes and I would very much recommend that you go and watch his video when you get chance.(link here).

One thing that I think was very important to take away from this was that it wasn’t a ‘nodejs vs react’ – it was more of a case of: if you have a requirement for a real time system for a low volume of traffic then React will be perfect for you. If you start to need requests upwards of the hundreds of thousands and into millions – then you should use the correct technology for the job and learn how to use nodejs. Well, at least that’s what I took away from it anyway – and it’s definitely a great way to get into setting up real time systems just using PHP.

Cranking Nginx up to 11

A very informative talk from @h – however I have to admit to never getting round to installing or configuring Nginx before… With this being an advanced talk about squeezing every last bit of performance out of Nginx it was great but I didn’t recognise any of the basic config to start with! I believe Helgi did a talk later in the day at the ‘unconference’ for those who wanted some basic Nginx knowledge but sadly I wasn’t able to make it.

Nonetheless, I took some great tips such as knowing you can connect directly to MySQL and get Nginx to load balance for you. I went away from this talk with a firm assertion that I want to ditch Apache and figure out how to set up Nginx with php-fpm, so thanks Helgi! 🙂 Link to talk here or the slides are here

API Design: It’s Not Rocket Surgery

I consider myself rather knowledgable when it comes to APIs – and this talk from Dave Ingram (@dmi) pretty much confirmed my thoughts. There wasn’t anything in particular that I noted down for further reading – I was very happy with that as it means my API knowledge must be pretty good!

In summary, it covered that you should be creating documented, RESTful (CRUD) APIs, and using a sensible URL for each endpoint. It was a very informative talk and he covered many points such documentation, authentication, headers, formats, caching, documentation and versioning of your documented API.

A good point he made was CORS (Cross-Origin Resource Sharing) which is is a way to allow in-browser cross-origin XHR requests.

For those who would like further information – I’d strongly recommend David’s talk: link here or you can view his slides

Bottleneck Analysis

Another informative talk – this time from Ilia Alshanetsky. He went into the stuff that you would have expected such as using inspect element and/or firebug to see the load times of the resources. Making sure you load things as asynchronously as possible as well as parallelising over multiple DNSs was mentioned. One very useful addition for me was the note about, ‘Boomerang’; a tool that will send your true page load time to your server. It’s all very well knowing that your PHP page loads in 250ms but what use is that if the user can’t physically interact with your page until 2s have passed? Using Boomerang will give you a great insight to how long your users are having to wait. There is even an awesome addon called navtiming.js which pretty much sends back the individual timings of each resource (just like inspect element).

A useful tip I picked up was that if you’re running a redirect on your website to always send your user to a the ‘www’ version of your website in addition to running SSL then you could be incurring a massive delay in the redirect. At the time of writing, only Chrome will be able to show you this delay as firebug doesn’t cover Ssl.

One point that @mattoddie re-iterated to me, was that if your php error_log has any notices and warnings then your app will be slowed down massively. Production code should never have notices and warnings – make sure you check!

We also saw how you can throw these stats into graphite to keep an eye on your users load times, in addition to performing apache bench load testing. A very informative talk and you can view the recording here: link – Slides can be found on Ilias Website

Saturday – Keynote (the diabolical developer)

I’ll be honest; I didn’t fully understand this. Some said it was very clever sarcasm, some said it will change their approach and others didn’t get it either.

All I could take away from it was that you should always think about what you are doing, and why you are doing it. Don’t just do things because the person next to you is telling you that it’s the cool thing to do. You should be driving what is best for you. Be Better.

The link to the talk is here: link There’s already a YouTube video from 2 years ago if you want a very shortened version

The Hypermedia API

I found this to be a very useful talk. The best way I could describe it is an advanced API talk. Ben Longden briefly covered some of the areas Dave Ingram covered in his talk, but quickly progressed to talking about the ‘Richardson Maturity Level/Model’. Having discovered my APIs are typically level 2 (the RESTful ‘CRUD’ type), I discovered that I can basically enhance them by giving them some steroids! The end result is that the response not only contains the data, but also some information about that entity such as what the URL would be to edit it, or delete it.

It’s all centred around the Hypertext Application Language (HAL) which was new to me. Another useful point was reducing requests by using a zoom parameter (the ‘hypertext cache pattern’). So if you would like to get all the information for a user in addition to their messages for a user you could do: website.com/api/v1/user/123?zoom=messages

I shall definitely be doing more research into this, but to see the talk, here it is: link . You can already view his slideshow.

Scaling with HipHop

I think this is one that everyone was looking forward to. Sara Golemon (from Facebook) gave us a very informative talk on the the history of HipHop before progressing onto how it’s evolved.

In brief, HipHop used to be something that you had to use to compile your PHP code before running it. At Facebook this meant waiting 20mins for hphpc to build the application on over 100 build servers before being able to test your change; something that is clearly not ideal! On the road to finding the best solution, they created HPHPi for use in dev environments. This was so that developers didn’t need to wait for the code to build.

However, after much hard work – and lots of very clever people, they came up with HHVM. HHVM uses JIT (just in time) compilation to analyse the data types as the code is executed and generate the necessary machine code for optimal speed.

Sara was very keen to point out that HipHop has obviously been designed for the Facebook codebase; therefore the codebase that will benefit most from it is Facebook. Whilst speed improvements of up to 600% have been noted for FB, you can expect to see only 150-200% speed increase if you’re running WordPress. If you’re a company that has 50 servers to cope with load due to the PHP execution time, I’m pretty sure you’d love to only need 25 servers?

Sara also covered the use of XHP, a PHP extension which makes your frontend code a hell of a lot easier read as well as being a massive help with regard to prevent XSS attacks.

I definitely reccomend you watch the talk – I shall definitely be doing some blogging on the subject! Talk link here

Planning to Fail

I loved this talk from David Gardner. The point was the it’s easy enough to make a reliable system, but is it resilient?

David used Hailo (the taxi app) as his example of a portfolio of technology that was designed to be resilient. If you use technology that was designed to be resilient, and then build your application atop of that with resilience in mind, then there is a very good chance that you app will also be resilient.

Netflix famously announced their lessons learned from the AWS Cloud problems with the Chaos Monkey. David was showing us a multitude of ways that you can acheieve the end goal of having a Chaos Monkey.

I think David’s talk will the source of many blog posts for me, largely due to the great technologies they use at Hailo such as: Cassandra, ZooKeeper, ElasticSearch, NSQ & Cruftflake.

Keep an eye out for those blog posts, I shall probably been looking at Cruftflake soon as it’s a great way to generate unique ID’s far nicer tha UUIDs and the infamous MySQL auto-increment! It’s essentially a PHP version of Twitter Snowflake, but removes the dependency of Thrift.

Definitely a talk to watch: link here but he’s already uploaded his slideshow

You Can’t Optimise What You Cant Measure

So, Juozas “Joe” Kaziukėnas did a great talk – it expanded massively on what Ilia had covered in the bottleneck analysis talk; and that’s using statsd and graphite.

If you write to logs in your application, you’re slowing down your application. End of. With StatsD being a simple NodeJS daemon that utlises the UDP, you can be sure that it’s non-blocking in nature; therefore your app will not suffer. It also means you can literally measure anything and you don’t need to worry about switching on ‘debug mode’ – you can run it all in your live environment without worrying about performance (in fact, you can measure performance).

Joe went on to mention the great tool that is Graphite, which hooks up to StatsD perfectly. We actually use this at Sky, but it was great to have an explanation about how it works etc. Logster (or ‘Lobster’ as Joe likes to cal it) is a tool which allows you to throw all of the log files into Graphite if you happen to not be using StatsD. There was also a mention of DataDog – a website service you can pay if you want to offload your graphing to a 3rd party.

I loved this talk, and I shall definitely be doing my own research into StatsD. You can view his talk here: (link) – but he’s already put his slide up

Monitoring At Scale: Intuitive Dashboard Design

Lorenzo Alberton didn’t leave out any details when it came to effective monitoring. It’s difficult for me to summarise as there was simply a lot of information to take in – but I’ll try in the form of bullets (most using his slide headings!)

  • Create surprise with alerts
  • Show, don’t tell
  • Communicate with clarity
  • Too much data and too little information = problem
  • Heuristics
  • Organise information to support meaning
  • Correlate events to add context
  • Shapes, Sounds & Colours do help
  • Realtime – StadsD & Graphite
  • Averages SUCK – use percentiles
  • Patterns our brains should recognise
  • Heatmaps / Cacti
  • Make the subtle obvious
  • Make the complex/busy simple/clean

You should really view his excellent set of slides

Summary

The PHP UK Conference 2013 was great. I loved it. Apologies if I irritated anyone with my coughing/sneezing/nose blowing. Extra apologies if you also now have man flu!

I’ve learned a great deal, and I have plenty of things to blog about. So stay tuned!

Many thanks to everyone who made PHP UK what it was and of course, Sky for paying for my ticket. Definitely looking forward to next year!

Share

Creating a RESTful API with Silex using Composer and PHPUnit

Recently I’ve been playing around with Silex largely due to wanting to experiment with the various frameworks out there but also because I wanted to have a play around with composer. Previously I’ve used either Zend Framework or Kohana for my projects and whilst they don’t prevent you from using composer, they weren’t built with it in mind. For dependancy injection I’ve been using Sensio Labs’ Pimple – which I find to be awesome. Soon, I want to look at Symfony2 so I figured playing around with Silex (also by Sensio Labs) would be a great per-cursor; it uses all the same mod-cons and introduces the Symfony framework gradually.

So, where do we begin? Well I want to code from my MacBook to begin with – write some code and some tests. If I get that far, we can look at using My dev server to run Apache!

I’m assuming you’ve got a fresh install of Mountain Lion – so open up the terminal and smash in the following command:

sudo php /usr/lib/php/install-pear-nozlib.phar

Now, edit your php.ini and add the following:

zend_extension=/usr/lib/php/extensions/no-debug-non-zts-20090626/xdebug.so
include_path=.:/usr/lib/php/pear
memory_limit=512M
date.timezone=UTC
detect_unicode = Off

Hey presto, you now have PHP setup on your Mac. Does it work?

$ php -v

All ok? Good. Now lets install composer; the following is a good guide:

http://getcomposer.org/download/

…or just run this in your terminal to get the latest Composer version:

$ curl -s https://getcomposer.org/installer | php
$ sudo mv composer.phar /usr/local/bin/composer

Go to your working directory (/code/projectname), create composer.json and add the following:

{
    "minimum-stability": "dev",
    "require": {
        "silex/silex": "1.0.*@dev"
    },
    "autoload": {
        "psr-0": {"DVO": "src/"}
    },
    "require-dev": {
        "phpunit/phpunit": "3.7.*",
        "squizlabs/php_codesniffer": "1.*"
    }
}

Here we are specifying that phpunit and Codesniffer should only be installed hen the –dev flag is specified. So let’s go ahead and do just that:

$ composer install --dev

First things first you will notice we now have a vendor folder – this is where composer puts all our external libraries. If you want to add this project to a Git repository, make sure you add the vendor folder to the .gitignore file. You can see below I’ve added a few other items too.

.gitignore:

*.lock
vendor/
build/
.DS_Store

Okay, let’s create our directory structure:

/code
    /projectname
        /app
        /src
        /vendor
        /web

Open up web/index.php with the following:

<?php

use Symfony\Component\HttpFoundation\Request;

$app = require_once __DIR__.'/../app/app.php';

if (count($argv) > 0) {
    list($_, $method, $path) = $argv;
    $request = Request::create($path, $method);
    $app->run($request);    
} else {
    $app->run();    
}

This allows us to run our code via the CLI – we don’t want to have to get Apache involved just yet.

Now create app.php and bootstrap.php and put these in the app directory.

app/app.php:

<?php

require_once __DIR__.'/bootstrap.php';

use Silex\Application;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;

$app = new Application();
$app['debug'] = true;

// example route - kinda obvious what this does :)
$app->get('/hello/{name}', function ($name) use ($app) {
    return 'Hello '.$app->escape($name);
});

return $app;

bootstrap.php:

<?php

require_once __DIR__.'/../vendor/autoload.php';

Done! You can now go to your console and type the following command:

$ php web/index.php GET /hello/yourname

That’s all we need to call Silex, but as you can see it doesn’t do a great deal. To sort out our API and pull back some data we will now call some of our own library code. In here we will have the controllers and entities. The controllers are to handle our CRUD requests and the Entity stuff is where I’ve used a Factory and Gateway to access the data.

src/DVO/Controller/VoucherController.php:

<?php

namespace DVO\Controller;

use DVO\Entity\Voucher;
use DVO\Entity\Voucher\VoucherFactory;
use Symfony\Component\HttpFoundation\JsonResponse;

class VoucherController
{
    protected $_factory;

    /**
     * VoucherController constructor
     *
     * @param VoucherFactory $factory The voucher factory
     * 
     * @return void
     * @author
     **/
    public function __construct(VoucherFactory $factory)
    {
        $this->_factory = $factory;
    }

    public function indexJsonAction()
    {
        $vouchers = $this->_factory->getVouchers();
        $vouchers = array_map(function($voucher) {
            $vc = array();
            $vc['code'] = $voucher->getCode();

            return $vc;
        }, $vouchers);
        return new JsonResponse($vouchers);
    }

    public function createJsonAction()
    {
    }

    public function updateJsonAction()
    {
    }

    public function deleteJsonAction()
    {
    }
}

src/DVO/Entity/Voucher/VoucherFactory.php:

<?php

namespace DVO\Entity\Voucher;

use DVO\Cache;


class VoucherFactory
{
    protected $_gateway;
    protected $_cache;

    /**
     * VoucherFactory constructor
     *
     * @param VoucherGateway $gateway The voucher gateway
     * @param Cache          $cache   The cache
     * 
     * @return void
     * @author
     **/
    public function __construct(VoucherGateway $gateway, Cache $cache)
    {   
        $this->_gateway = $gateway;
        $this->_cache   = $cache;
    }

    /**
     * Creates the Voucher
     *
     * @return void
     * @author 
     **/
    public static function create()
    {
        return new \DVO\Entity\Voucher;
    }

    /**
     * Gets the vouchers
     *
     * @return void
     * @author 
     **/
    public function getVouchers()
    {
        $vouchers = array_map(function($voucher) {
            $vc = VoucherFactory::create();
            foreach ($voucher as $key => $value) {
                $vc->$key = $value;
            }

            return $vc;
        }, $this->_gateway->getAllVouchers());

        return $vouchers;
    }
}

src/DVO/Entity/Voucher/VoucherGateway.php:

<?php

namespace DVO\Entity\Voucher;

class VoucherGateway
{   
    /**
     * Get vouchers
     *
     * @return void
     * @author 
     **/
    public function getAllVouchers()
    {
        return array(array('code' => 'OFFER999'));
    }
}

src/DVO/Entity/Voucher.php:

<?php

namespace DVO\Entity;

/**
 * Voucher
 *
 * @package default
 * @author 
 **/
class Voucher extends EntityAbstract
{
    public function __construct()
    {
        $this->_data = array(
            'id'   => '',
            'code' => ''
            );
    }
}

src/DVO/Entity/EntityAbstract.php:

<?php

namespace DVO\Entity;

/**
 * Abstract Entity
 *
 * @package default
 * @author 
 **/
abstract class EntityAbstract
{
    protected $_data;
    /**
     * Magic function to capture getters & setters
     *
     * @param string $name      the name of the function
     * @param array  $arguments an array of arguments
     *
     * @return void
     */
    public function __call($name, array $arguments)
    {
        $type     = substr($name, 0, 3);
        $variable = strtolower(substr($name, 3));
        switch ($type) {
            case 'get':
                return $this->$variable;
            break;
            case 'set':
                $this->$variable = $arguments[0];
            break;
            default:
                return $this->invalid($type);
            break;
        }
    }

    public function getData()
    {
        return $this->_data;
    }

    /**
     * Magic function to capture getters
     *
     * @param string $name name of the variable
     *
     * @return mixed
     */
    public function __get($name)
    {   
        if (true === array_key_exists($name, $this->_data)) {
            return $this->_data[$name];
        } else {
            throw new Exception('Param ' . $name . ' not found in ' . get_called_class());
        }
    }

    /**
     * Magic function to capture setters
     *
     * @param string $name  the name of the var
     * @param string $value the value for the var
     *
     * @return void
     */
    public function __set($name, $value)
    {
        if (true === array_key_exists($name, $this->_data)) {
            $this->_data[$name] = $value;
        } else {
            throw new Exception('Param ' . $name . ' not found in ' . get_called_class());
        }
    }

    /**
     * called when invalid function is called
     *
     * @return boolean
     **/
    public function invalid($type)
    {
        throw new Exception('Error: Invalid handler in ' . get_called_class());
    }
}

src/DVO/Entity/Exception.php:

<?php

namespace DVO\Entity;

class Exception extends \Exception
{

}

src/DVO/Cache.php:

<?php

namespace DVO;

/**
 * Cache Class
 *
 * @package Cache Class
 * @author 
 **/
class Cache
{
}

Now we’ve added those lovely additional files, you can modify app/app.php:

<?php

require_once __DIR__.'/bootstrap.php';

use Silex\Application;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;

$app = new Application();
$app['debug'] = true;

// example route - kinda obvious what this does :)
$app->get('/hello/{name}', function ($name) use ($app) {
    return 'Hello '.$app->escape($name);
});

// setup the app cache
$app['cache'] = $app->share(function(){
    return new DVO\Cache;
});

// setup the voucher gateway
$app['vouchers.gateway'] = $app->share(function() {
    return new DVO\Entity\Voucher\VoucherGateway;
});

// setup the voucher factory
$app['vouchers.factory'] = $app->share(function() use($app) {
    return new DVO\Entity\Voucher\VoucherFactory($app['vouchers.gateway'], $app['cache']);
});

// setup the voucher controller
$app['vouchers.controller'] = $app->share(function() use ($app) {
    return new DVO\Controller\VoucherController($app['vouchers.factory']);
});

$app->register(new Silex\Provider\ServiceControllerServiceProvider());

$app->get('/vouchers', "vouchers.controller:indexJsonAction");
$app->post('/vouchers', "vouchers.controller:createJsonAction");
$app->put('/vouchers', "vouchers.controller:updateJsonAction");
$app->delete('/vouchers', "vouchers.controller:deleteJsonAction");

return $app;

Give it a whirl:

$ php web/index.php GET /vouchers

I’m not going to go into explaining about Gateway and Factory design patterns but I’m sure the code above is in a very simple enough form for you to understand. Some folks absoutely hate magic methods – but don’t forget that the EntityAbstract class is just what is says it is, an Abstract class. If a particular entity requires it’s own getter then go ahead and create it.

If you’ve gotten this far and it’s working then you’ve done well – good work! The next step is to use PHPUnit to test all of our beautiful code. For this we can setup PHPUnit.

Copy the below and put it in app/phpunit.xml:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit backupGlobals="false"
         backupStaticAttributes="false"
         colors="true"
         convertErrorsToExceptions="true"
         convertNoticesToExceptions="true"
         convertWarningsToExceptions="true"
         processIsolation="false"
         stopOnFailure="false"
         syntaxCheck="false"
         bootstrap="bootstrap.php">
    <testsuites>
        <testsuite name="YourApp Test Suite">
            <directory>../tests/</directory>
        </testsuite>
    </testsuites>

    <logging>
        <log type="coverage-html" target="../build/coverage" title="PHP_CodeCoverage"
        charset="UTF-8" yui="true" highlight="true"
        lowUpperBound="35" highLowerBound="70"/>
        <log type="coverage-clover" target="../build/logs/clover.xml"/>
        <log type="junit" target="../build/logs/junit.xml" logIncompleteSkipped="false"/>
    </logging>

    <filter>
        <whitelist addUncoveredFilesFromWhitelist="true">
        <directory suffix=".php">../src</directory>
        <exclude>
            <file>../src/DVO/Entity/Voucher/VoucherGateway.php</file>
        </exclude>
        </whitelist>
    </filter>

    <php>
        <server name="APP_DIR" value="/code/projectname/app" />
        <env name="env" value="test" />
    </php>
</phpunit>

I’ve set the phpunit.xml file to dump the code coverage logs to the build directory. It’s good to set up a vhost to point to this folder so you can view your code coverage.

Here, I’m just doing a very basic test just to show you how you can get one running.

tests/DVO/CacheTest.php:

<?php

/**
 * Cache Test
 *
 * @package DVO
 * @author 
 **/
class CacheTest extends \PHPUnit_Framework_TestCase
{
    /**
     * Cache is just an empty class atm!
     *
     * @return void
     * @author 
     **/
    public function testCache() {
        $cache = new \DVO\Cache;
        $this->assertInstanceOf('\DVO\Cache', $cache);
    }
}

Run the tests:

$ ./vendor/bin/phpunit -c app/phpunit.xml

I’ve not put all the tests in the tutorial – but you can find them in the repo on GitHub (https://github.com/posmena/voucherapi) – feel free to follow Posmena on GitHub!

So, you’ve used the above code and your tests pass? Maybe now you want to make sure you code is PSR2 compliant – a very popular coding standard. We already installed Codesniffer via composer earlier, so all you need to do to check your code against code sniffer:

$ ./vendor/bin/phpcs --config-set default_standard PSR2
$ ./vendor/bin/phpcs src

There’s a very good chance (certainty) that the above code will not pass PSR2 standards. I figured I’d leave a few little things for you to research. I won’t be too cruel though; PHP Closures don’t seem to be PSR2 compliant so you need to wrap them with the @codingStandardsIgnoreStart and @codingStandardsIgnoreEnd tags. Once your code is passing Codesniffer, you will simply not get any errors. Checkout the CodeSniffer options page for config options.

I hope you’ve enjoyed this article. I’m open to any comments/suggestions you may have!

Bobby (@bobbyjason)

Thanks to:

Share

Building a Data Feed Driven Website (with @WebgainsUK Feeds)

Part 1: Webgains Data feed Introduction

Where I work (@WebgainsUK), we often have a lot of affiliates contact us that want help with regard to data feeds. Whilst a lot of these queries tend to be technical questions relating to an issue with the feed, or reporting a problem with a merchant’s feed; some of these requests are as technical as asking how to build a site using the data feeds, how to import feeds into the database, or asking how to build a price comparison site. Sadly, as much as Webgains try to help with these sorts of questions wherever possible, it’s simply not possible to offer the in-depth time-consuming answer required due to the nature of the task in hand.

Thankfully, there are tools out there such as Easy Content Units, which allows affiliates to insert content units into their site without requiring any knowledge of data feeds. A plus point of being a Webgains affiliate is that Webgains is partnered with Easy Content Units and as such get the service for free. However, having 3×3, 4×4 or other size grids of products powered by someone else’s website isn’t always what an affiliate wants – some folks want to do their own thing, and know how it works. Also, having products powered by Easy Content Units doesn’t do much for your SEO; you can’t index each individual product, you can’t have an in-built search, you can’t categorise your products, and you’re also relying on their server be up 100% of the time. Throw in a few other excuses for not wanting to use it, and you have a compelling reason why affiliates are asking how to build their own data driven website.

So with that in mind, I figured I’d have a go at providing a solution in the format of a guide, “Building a Data Feed Driven Website”. This guide will come in several parts, I have no idea how many parts it will be, and I have no idea if I will ever finish it. First things first though, let’s set out the objective.

Objective

To build a website that:
> is able to download data feeds from Webgains on a regular basis, so that the prices are always correct.
> can display 1 product per page for SEO reasons.
> has SEO friendly URLs
> can categorise products.
> has a neat search facility.
> has a voucher facility to accompany the products.
> can list the latest offers and promotions from merchants.

For now, that can be the ‘simplistic’ objective – we can always add to it later if we desire. Notice how the requirement mentions ‘data feeds from Webgains’; the reason for this is that as a Webgains employee, I feel I have a duty to not help you promote other networks. If by reading my guide, you feel technically inclined to mash the code to allow you to download feeds from other networks then feel free, but don’t ask me how 🙂

Tools/Resources/Knowledge Required

Basic understanding of affiliate marketing is a given; you should know what products you want to promote, you should know what a data feed is, you should know why you are reading this, and you should know what it is you want to achieve. It’s probably worth noting that you won’t get very far without a Webgains affiliate account. Signup to Webgains here.

Ultimately, you are going to need access to a server capable of running PHP5 & MySQL5. In terms of getting a server up and running with PHP5 and MySQL5, I’m afraid it is beyond the realm of this guide. However, if you have a spare computer laying around; may I suggest that you install Ubuntu. If you need help with that, Google this: “ubuntu LAMP server”. Failing that, you could always try a webhost like Fasthosts.

In addtion, you will need a decent text editor (Notepad++, or EditPlus[I use this]) and a willingness to learn. As this guide is meant to be a resource to those who already have a basic understanding of PHP & MySQL, you will have to ‘catch up’ where necessary. If possible, I’ll try to give, ‘beginners guide tips’ – but the ultimate goal is for you to reach the objective, so maybe a twitter account will be required so you can ask me questions: @bobbyjason.

Notes

Due to the nature of complexities of this project (it’s not such a small project after all), I’m going to code the site in sections of how to do each bit. I can’t promise that the final project will involve all the final bits being put together, as each section is meant to be a mini-guide in it’s own right. Again, throw any questions my way: @bobbyjason.

Getting Started

The most obvious place to start, is at the heart of the problem; data feeds. If you knew how to work with data feeds, and you knew how to load them into a database – you probably wouldn’t be reading this guide, so it is for that reason I have decided to start here. For the sake of brevity, I’m not going to explain in detail the difference between downloading a datafeed directly, and downloading a feed via a URL. In short, for our objective we want to download the feed via a URL as this will allow our website to be automated.

So, head over to the “Data feed url generator” and select the “xml” option, rather than “csv”. XML is a much more structured approach, and it makes life much easier where trying to find a problem when things go wrong – plus, I prefer working with XML and its MY guide! Select the “.gz” option – this is the only option that allows you to download a compressed feed immediatly. “.zip”, and “.tar.gz” required you to hang around a little – not so good. Select ONE of the programs, not all – but ONE. You can then select ‘All’ at the categories option, and select the ‘exended’ fields option. Finally, enter your username and password, and grab the URL – you should something that looks similar to this:

http://content.webgains.com/affiliates/datafeed.html?action=download&campaign=12345&username=username@website.com&password=password&format=xml&zipformat=gzip_notar&fields=extended&programs=1234&categories=all

If you have a URL like the one above (with the correct username and password), you should be able paste it into your browser and download a .gz compressed feed of the program you selected. Notice how you can replace the “programs” parameter with any other program ID to download a feed for a different program. Straight away you should be able to see the logic required here; we need to create a PHP script that can loop through each of the programs’ feeds that we wish to download, and enter the correct program ID. In psuedo form, here it is:

foreach program
….download feed for this program
end foreach

Of course, in reality we don’t want to just download the feed, we also want to take the data out of it and populate our database, but I figured baby steps would be best as not every one will understand that just yet. So let’s get sizzling. Glancing at the clock I can see that it is 18/10/2010 22:17 – this means that I am missing the last episode of The Inbetweeners which is rather upsetting. Thankfully we have 4OD in this modern world, so I’ll create this script and then head off to bed!

<?php /* lesson1.php */

error_reporting(E_ALL|E_STRICT);
ini_set('display_errors', true);

// Create an array of the programs that we want to download
$programs = array(4084, 116);

print "Starting...\n";

// PHP foreach loop.
foreach ($programs as $program_id)
{
 // Use the "sprintf" function to pass the correct values to the string.
 // Also, splitting up the string to avoid long lines of code.
 $feed_url = sprintf('http://content.webgains.com/affiliates/datafeed.html?'.
 'action=download&campaign=%s&username=%s&password=%s'.
 '&format=xml&zipformat=gzip_notar&fields=extended'.
 '&programs=%d&categories=all',
12345, 'USERNAME', 'PASSWORD',$program_id);

 // Set the path of where you would like to save the file.
 $file = '/home/chops/xml/guide/feeds/'.$program_id.'.xml';
 $compressed = $file.'.gz';

 // Download the file (requires function, 'curl_get_file_contents'
 $data = curl_get_file_contents($feed_url);

 // Open the file for writing, write, and close the file.
 $fp = fopen($compressed, 'w+');
 fwrite($fp, $data);
 fclose($fp);

 // Call the a UNIX command to unzip the file, and move it to our desired location.
 shell_exec("gunzip -c $compressed > $file");

}

print "Done.\n";

// Function to download a file.
function curl_get_file_contents($url)
{
 // Output something so we know it's working.
 print "Downloading '".$url."'\n";
 flush();

 $c = curl_init();
 curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
 curl_setopt($c, CURLOPT_URL, $url);
 curl_setopt($c, CURLOPT_CONNECTTIMEOUT, 5000);
 curl_setopt($c, CURLOPT_TIMEOUT, 10000);
 $contents = curl_exec($c);
 curl_close($c);

 return $contents;
}
?>

You can copy that code into a text document, save it as ‘lesson1.php’ (be sure to replace username, password and campaignid with your values!). You can call the script directly in a browser if you wish, personally I shall use the terminal. The files will be downloaded to the specified location. Note: If there are any PHP Gurus out there, you will notice that this code may not be the most complex, but I’m trying to keep it simple for those that may not be so familiar with PHP.

All the script does is:
> loop through our selected programs.
> downloads the compressed feed.
> extracts the compressed file with the name of {program_id}.xml

For now that will have to do, because I want to publish this blog post and go to bed. I hope it’s useful to somebody!

Coming up: How to parse the XML files and insert the products into a database.

Share

Hacking TweetMyBlog (TwitterFeed Sucks)

I used to use TwitterFeed; but recently it’s been a real pain in the ass and not updating so I looked for an alternative.

Roll on TweetMyBlog.

TweetMyBlog is great for the simple fact that it is a WordPress plugin. It’s also a small download, and you can install it very easily. However, TweetMyBlog is hard-coded to use TinyURL and the URL it “makes tiny” is an affiliate link taking the user to a framed page. This is a bit of a bummer for me because I like to use bit.ly and I don’t really want my users going to some 3rd party site.

Roll on TweetMyBlog Hack

I took at a look @ the files and realise we can very easily change the Plugin to use bit.ly instead of TinyURL using the bit.ly API.

If you have installed TweetMyBlog you would have uploaded the “tweetmyblog” directory which had 3 files in it. Go and find, “twitter_api.php”:

and at the very bottom of the file you will see:

function get_tiny_url($url)
{
$this->snoopy->fetch('http://tinyurl.com/api-create.php?url='.$url);
return $this->snoopy->results;
}


As you can see, it uses TinyURL.

I’ve only hacked it to use bit.ly, but this is what I changed it to first (don’t copy it yet, there’s a better version further down):

   function get_tiny_url($url)
	{
    $this->snoopy->fetch('http://api.bit.ly/shorten?version=2.0.1&login=bobbyjason&apiKey=MYLONG_BIT_LY_API_KEY&format=text&longUrl='.$url);
	 return $this->snoopy->results;
	}


I gave it a shot, and sure enough it worked, I was using bit.ly instead of tinyurl, result! However, it was still using the silly Affiliate link with a frame & 3rd party site.

Okay, open “tweetmyblog.php” at around line 36 you will see:

$tweet = trim($tweetPrefix).' '.$blogPost->post_title.' '.$t->get_tiny_url('http://www.tweetmyblog.com/bpredir.php?r='.$tmb_affid.'&bp='. urlencode(get_permalink($postID)));

The problem is the tweetmyblog.com redirection, so I replaced it with this:

$tweet = trim($tweetPrefix).' '.$blogPost->post_title.' '.$t->get_tiny_url(urlencode(get_permalink($postID)));

Easy, right?

Well, this works great.

I do not need to use TwitterFeed ever again because WordPress automatically Tweets my blog post! I don’t have to wait 30 minutes, 1 hour, 5 hours for TwitterFeed to notice my blog post.

There was one last change that I made; after using the code above (using the bit.ly api) I realised that the URL wasn’t showing up in my bit.ly history. I checked this out and realised I needed to use the history=1 parameter. After I added this I’ve not had a single problem at all.
For the final code of “twitter_api.php”, see below:

function get_tiny_url($url)
	{
    $this->snoopy->fetch('http://api.bit.ly/shorten?version=2.0.1&history=1&login=bobbyjason&apiKey=MYLONG_BIT_LY_API_KEY&format=text&longUrl='.$url);
	 return $this->snoopy->results;
	}

I must say though; TweetMyBlog is a great plugin by John Merrick and Sean Jordansen and you should install it.

If you wish to monetise your tweets from your blog, then the above edits are silly. It’s just personally I would rather not have the user taken to a 3rd party site.

TweetMyBlog also provides a feed of your Tweets as a Widget, so it really is great!

Download TweetMyBlog now!

Share