I’m putting some domain names up for sale, including twitapps.com/net/org, and several others that are ideal for Twitter-related services, or whatever.

For more info please see the blog post over on 3ft9.com.

I know it’s been a while since posted anything on this blog, but I’ve had a lot going on and it hasn’t really been at the forefront of my mind. So, what’s been going on?

Firstly I’ve shut TwitApps.com down. I won’t go into the reasons because they’re well documented in the blog post announcing it. I’ve also released the code so anyone can make a clone if they feel so inspired. One big side effect of this is that I’m open to offers for that domain name and a bunch of others I don’t see myself using anymore. Please only contact me with serious offers.

Secondly I’ve been doing a lot of soul searching lately, and one of the results of that has been my new blog over at tastingtheacorn.com where I’m writing for my benefit rather than as a service to others, which is how I view this site. Check it out and let me know what you think.

Finally I have been all over the place in terms of my career lately, which I’ve written about on my new blog so I won’t repeat it here. Ultimately it means I’m on the lookout for interesting projects to get involved with. If you think I may be able to help you or your company please let me know.

Hopefully that explains where I’ve been. I plan to continue publishing tech and development posts on this site for the foreseeable future, but how often depends on how much my focus gets split in my new reality. I also plan to change the layout of this site – it’s starting to depress me because I never really finished it.

One of the pillars of a scalable website is ensuring that only activity which is required to build a page should be performed during the processing of a page request. Activities that fall under this category commonly include sending emails, recording statistics and general housekeeping such as removing temporary files.

Back when I started working on sites big enough for these activities to cause a problem I went down the obvious route of making a PHP CLI script for each job that needed doing and getting it to run using cron. This worked for a while but as the sites I was working on got bigger and more complex it quickly became clear that this was becoming difficult to manage, so I started to consider alternatives.

At the time all this happened I was mainly working with a site that ran on Zend Platform. One of the recent additions at the time was a module called Job Queue which appeared to do exactly what I needed. Unfortunately after spending a fair amount of time developing the infrastructure required to make it run the jobs I discovered that it really wasn’t very well tested; it was far from production quality [1] and nowhere near reliable enough so I went back to the drawing board.

Time passes…

After thinking about what I needed from such a system and what I had available I came up with an arrangement I’ve been using ever since with great success.

The core of the system is a DB table and a PHP script. The table contains the definition of jobs that need running and the script, erm, runs them.

I won’t go into the details of the table because the only important parts as far as this system go are the run_at, processor_pid and schedule_* fields.

The run_at field is a timestamp that simply indicates the time when that job should be executed.

The processor_pid field is an unsigned integer that defaults to 0 and will indicate the process that’s running a job if any.

The schedule_* fields specify how often the job should be executed. There are a number of ways of organising these depending on what your requirements are, but they fall into two general categories.

Before I move on to the script I apparently need to cover something that I missed when I first wrote this. You need to specify what to run in this table! I though this was pretty obvious but based on initial feedback I was wrong.

This can take any number of forms and will depend greatly on your specific application. Over the years I’ve used a number of different methods including a PHP script name, a CLI command and a method in a static class. Whatever you use you’ll want to make sure you do sufficient checks to ensure it’s secure.

Ok, on to the script which is designed to be run by cron according to a schedule that allows it to keep up with the size of the job queue you anticipate.

The following is a list of the basic steps the script performs. For this example I’m continuing with the assumption that the job queue is a table in a database.

1. Get the pid and put it in $pid.
2. Execute this SQL query…
   update `job_queue`
   set `processor_pid` = “$pid”
   where `run_at` <= unix_timestamp()
     and `processor_pid` = 0
   order by run_at asc
   limit 1
3. Check mysql_affected_rows() and exit if 0.
4. Execute this SQL query…
   select * from `job_queue`
   where `processor_pid` = “$pid”

   Note that these SQL statements (step 2 and this one) atomically grab a job and lock it. If you’re using a different storage system for your queue you’ll need to lock it while you select a job to run and then mark it as in progress.

5. Run the job. As mentioned already this can mean any number of things and will depend on your particular application. One of the useful things you can do here is to set up a clean, safe environment for the job to run in, along with ways to capture errors and other outputs so you can do something useful with them.
6. As soon as the job has finished executing we mark it as completed and record the success or failure status.
7. If the job has a schedule (i.e. it’s a recurring job) we create a new job by effectively cloning the job we’ve just run. We now calculate the time it should be run according to the schedule definition and save that in the run_at field of the new job. Finally we set the processor_pid of the new job to 0 so it’s then available to step 2 of this process.
8. Depending on the job configuration and status we now either remove the completed job from the queue or archive it complete with errors and output for later inspection.
9. If this processor has been running for > 60 minutes it exits, otherwise it goes back to step 2 and looks for another job to run.

To get this to do something useful we configure it to run via cron every n minutes where n depends upon the anticipated size of your job queue. For example running it every minute will automatically scale it up to 60 concurrent jobs at any one time. Running it every 5 minutes will reduce this to 12, and so on. There’s also nothing stopping you putting more than one line into the crontab so it runs two processes every minute which increases concurrent processors to 120.

Assuming your job queue is network accessible this system also scales across multiple machines with minimal changes. In fact the only change that’s required it to incorporate a machine identified into the processor_pid field. This could be as simple as <machine>_<pid>; the key thing is that it’s guaranteed to be unique to a given process across your entire infrastructure.

Crashed jobs

One problem you may need to deal with is how to handle crashed jobs. This will happen, you can’t get away from it and you’ll need a way to detect and deal with it when it does. Luckily the job queue makes detection fairly straightforward.

On a single machine you can implement a script (either run via cron separately or indeed run by the job queue) that will check that for each job that has a processor_pid > 0 there is a PHP process running with that PID.

If and when you’ve scaled across multiple machines this script essentially remains the same except that you need to run it on every machine that runs the job queue and filter the PID’s you check.

As far as what to do when you find a crashed job that’s really something you need to consider on a case-by-case basis. At the very basic level the script could simply reset the processor_pid field to 0 so it gets run again. At the other end of the spectrum in a very flexible system you could have a way to run a job with a flag to indicate that it had previously crashed; each job can then deal with crashes in their own custom way.

Potential additions

The components described above is just the core of a job queue system; you can add a lot of useful stuff above and around it to make it easier to manage and provide better feedback from your periodic tasks.

• Performance metrics
  Since you’re running all your jobs from a central script adding code around the actual execution to record execution time, load and possibly memory usage too is pretty simple. You then have the ability to compare the time a job took to previous executions of that job to detect potential errors.
• Management interface
  Since you have all the information regarding the jobs in the queue, what’s running right now and the status of jobs that have previously been executed it’s a pretty small leap to build a UI that will let you view and manage the whole thing. This can be especially useful for presenting error messages and performance metrics.

Final thoughts

I hope that’s useful to someone, I’ve certainly found it applicable to most web applications I now deal with. As I mentioned a few times you can implement the various parts in a number of ways, in particular how jobs are specified and how the scheduling works.

As an example of this flexibility I’ll make a passing mention to one implementation I’ve done that only needed to be able to execute scripts daily, hourly or every 15 minutes. To accomplish this I simply created a folder for the scripts, and three folders named 15min, hourly and daily within that. The processor script uses a custom locking system and CLI arguments to run each set of scripts according the the directories they’re in.

This system has proven to work very well and will continue to do everything that site needs until we need to execute something at a specific time; a requirement that has not yet surfaced.

One of the many side projects I’m working on is a reusable version of this system. At the moment it’s a fairly messy combination of scripts that doesn’t work very well so far, but as soon as I have something worth sharing I’ll definitely do so on this blog. Stay tuned.

If you have an questions or suggestions for improvement please don’t hesitate to leave a comment or contact me privately.

[1] This was a few years back and I’ve heard that the Job Queue module has received some attention since then so is now a lot better, but I no longer use Zend Platform so I’m not in a position to comment. If budget is not an issue for you I’d recommend checking it out.

Not sure if I believe this is real or not, but either way it’s a pretty cool idea.

A couple of days ago I released a new TwitApps tool called Follows. I know I haven’t talked about TwitApps on this blog yet – that post is coming – but if you follow me on Twitter you should be aware of it.

The Follows service monitors your followers on Twitter and sends you a daily, weekly or monthly email telling you who’s followed and who’s unfollowed you since your last email. Twitter itself has the facility to send you an email every time someone starts following you but does not offer any sort of notification when they stop.

I can see why Twitter have done it this way. When someone starts following you it’s likely you’ll want to check out their tweets and you might decide to follow them back. The same logic doesn’t really apply to when people stop following you, or does it?

I built the Follows service primarily as a technical exercise, but also because I was curious to get some visibility on people who stop following me.

A while back a service called Qwitter appeared on the scene that offered an unfollow notification service so I signed up. Unfortunately it stopped working shortly after that for unknown reasons, but in the few days while it did work it was quite enlightening. By paying attention to both follow and unfollow notifications it was possible to spot some interesting trends.

By a strange coincidence it would appear that shortly after I launched my Follows service Qwitter started working again, but so far it doesn’t appear to be particularly reliable. It attempts to match an unfollow event to one of your tweets, effectively trying to find the cause of the action. This is daft primarily because there’s not always a single reason why people stop following you but also because they can’t possibly monitor your followers continuously – the Twitter API has limits that would prevent that – so it’s likely the tweet they attribute the action to was not your latest tweet at the time.

Since Follows went live I’ve been keeping track of what people tweet about it and Qwitter and I’ve noticed some interesting behaviour in response to notifications.

Quite a few users react to people unfollowing them by calling them out with a public tweet asking them why. I can understand the desire to get an explanation but doing it publicly seems a bit … well I dunno really, but it certainly isn’t the best way to go about asking the question.

The other common reaction is the realisation that their self-esteem can’t take being informed when people stop finding them interesting. Again I can understand this to a certain extent but it’s not something you can take personally and remain well-balanced. It’s the constant battle between the need for validation and the bliss of ignorance – part of the “human condition”.

Another interesting side-effect has been some comments from other Twitterers to the effect that such a service is pointless and/or purely egotistical, but I’m convinced there’s great value in getting visibility of the ongoing changes to your follower list.

I use the daily emails I get from Follows to do two things. First of all I check every new follower to see if it’s worth following them back, just like I did when I was getting the individual notifications from Twitter but now I only do it once a day which is far more efficient.

The second thing I do is have a look at every ex follower and if I’m following them I consider whether it’s worth continuing to follow them despite their decision to stop following me. I would never ask them about it because if someone has a problem with what I’m saying, whether it’s frequency or content, I would hope they’d tell me about it. It’s difficult to learn without feedback.

For me Twitter is not a popularity contest and it’s not an ego broadcasting outlet. It’s a conversation. If someone decides to stop following me I take that to mean they no longer want to participate in conversations with me. That’s fine, but it doesn’t necessarily mean I no longer want to participate in conversations with them, or that I’m no longer interested in what they have to say, but knowing they have made that decision is empowering.

What do you think? Do you see value in knowing if and when people stop following you or is it too much information?

Absolute genius!

Several templating systems exist for PHP, but since PHP was originally created as a templating language I’ve never understaood why more developers don’t use pure PHP in their view layer. The following snippet is a small function that facilitates precisely that.

The only setup required is to simply define TPL_DIR to point to the root directory of your templates. You can then call the function, passing it the relative path to a template file in that directory, an array of variables to be used by the template and you can optionally have it return the results rather than sending them to the browser.

function & TPL($____filename, &$____data = array(), $____return = false)
{
  $____retval = '';

  $____tplfilename = TPL_DIR.'/'.$____filename;

  if (file_exists($____tplfilename))
  {
    if ($____return) ob_start();

    extract($____data);

    require $____tplfilename;

    if ($____return) $____retval = ob_get_clean();
  }
  else
  {
    trigger_error('Template not found: '.$____filename, E_USER_ERROR);
  }

  return $____retval;
}

As an example let’s take a simple hello world (you saw that coming right?!). First the template…

<html>
  <head>
    <meta http-equiv="Content-type" content="text/html; charset=utf-8">
    <title><?php echo htmlentities($title, ENT_QUOTES, 'UTF-8'); ?></title>
  </head>
  <body>
    <h1>Hello <?php echo htmlentities($name['first'], ENT_QUOTES, 'UTF-8'); ?> <?php echo htmlentities($name['last'], ENT_QUOTES, 'UTF-8'); ?></h1>
  </body>
</html>

Now let’s use it…

define('TPL_DIR', '/var/www/public_html/templates');

$data = array();
$data['title'] = 'Hello World Example';
$data['name'] = array('first' => 'Joe', 'last' => 'Bloggs');

TPL('homepage/hello.tpl.php', $data);

That’s all there is to it. From this simple yet powerful base I’ve built fully-featured classes that handle all aspects of template management and processing, it’s an extremely stable foundation.

Comments, suggestions, requests and abuse are all welcomed.

Really short snippet today – a password generator. Not much to say about this one since it’s pretty simple. Pass it the length of password you want and optionally a string of valid characters it can use and it’ll give you a random string back.

function GeneratePassword($len, $allowedchars = false)
{
  if ($allowedchars === false)
    $allowedchars = 'abcdefghijklmnopqrstuvwxyz01234567890';
  $retval = '';
  $maxidx = strlen($allowedchars)-1;
  for ($i = 0; $i < $len; $i++)
  {
    $retval .= $allowedchars[rand(0,$maxidx)];
  }
  return $retval;
}

I've got something a bit meatier lined up for the next snippet so I recommend subscribing to the RSS feed to make sure you don't miss it. As always comments, questions, suggestions and requests are welcomed.

Today’s snippet describes the singleton pattern. The singleton pattern is a method of creating a class that statically maintains an instance of itself and ensures that no other instances can be created. This is useful for classes that implement app-wide services such as logging, DB access and other shared resources.

The basic requirements for a singleton class are as follows…

• The constructor is private
  This ensures that nothing outside the class can create instances.
• A static method exists that returns the instance
  When first called the method creates a statically stored instance of the class. Subsequent calls return the existing instance.
• Magic methods to prevent the instance from being cloned or serialized (PHP-specific)
  PHP provides functionality to clone and serialize objects. By adding simple implementations of the __clone and __wakeup magic methods we can prevent this from happening.

That’s basically it. Here’s a simple example…

class Singleton
{
  static private $_instance = null;

  public static function & Instance()
  {
    if (is_null(self::$_instance))
    {
      self::$_instance = new self();
    }
    return self::$_instance;
  }

  private function __construct()
  {
    // Do normal instance initialisation here
    // Nothing singleton-related should be present
  }

  public function __destruct()
  {
    // This is just here to remind you that the
    // destructor must be public even in the case
    // of a singleton.
  }

  public function __clone()
  {
    trigger_error('Cloning instances of this class is forbidden.', E_USER_ERROR);
  }

  public function __wakeup()
  {
    trigger_error('Unserializing instances of this class is forbidden.', E_USER_ERROR);
  }

  private $var = '';

  public function SetVar($val)
  {
    $this->var = $val;
  }

  public function GetVar()
  {
    return $this->var;
  }
}

This is an extremely simple example but it demonstrates all of the core concepts. You use this class as follows…

$obj1 = Singleton::Instance();
$obj1->SetVar('some value');

$obj2 = Singleton::Instance();
echo $obj2->GetVar(); // This will echo 'some value'

If you attempt to clone or unserialize an instance your script will fail with an error.

I use this pattern all over the place and definitely recommend using it where it makes sense in your applications. An alternative would be to implement the entire class statically and to be honest I’m not sure whether there are advantages to either implementation. If someone knows please share in the comments.

That’s it for this snippet, stay tuned for more. As always comments, questions, suggestions and requests are welcomed.

Pretty boring snippet today, but I find it immensely useful. My time class provides representations of periods from OneMinute to OneYear. It only has one method which will calculate an absolute time by adding a given time period to the current time. You can optionally provide a format intended for use by the date function to return a formatted string rather than a timestamp.

Hope you find it useful too.

	class Time
	{
		const OneMinute = 60;
		const FiveMinutes = 300;
		const TenMinutes = 600;
		const FifteenMinutes = 900;
		const HalfHour = 1800;
		const OneHour = 3600;
		const SixHours = 21600;
		const HalfDay = 43200;
		const OneDay = 86400;
		const SevenDays = 604800;
		const ThirtyDays = 2592000;
		const OneYear = 31536000;

		public static function GetAbsolute($time, $format = false)
		{
			if (is_numeric($time) and $time < (time()-1))
			{
				$time = time() + $time;
			}
			else
			{
				$time = strtotime($time);
			}
			return (false === $format ? $time : date($format, $time));
		}
	}

Comments, questions, suggestions and requests are welcomed as always.