Bear in mind that these are unique URLs, not simply repeats of URLs already seen. Between seeing a URL in a tweet and it appearing on this page the system has followed any redirects, grabbed and parsed the contents of the page and stuffed it into a database, and it does it all in near-realtime. Pretty impressive stuff if you ask me.

The live functionality on TweetMeme.com has now been removed, but it was great fun to develop and demonstrated the rate at which the site discovered and processed new URLs back in April 2009; you can bet the rate is far higher now given the explosion in the popularity of Twitter since then.

The implementation used long-polling requests connected to an nginx module which internally checked a queue for new items. While it didn’t get a huge number of concurrent viewers during the time it was up on the site it scaled incredibly well for such a simple solution.

It’s a shame that it’s not available on the site anymore; it would be interesting to see how much the rate has increased in the past ten months.

The core of the goal is to create more freedom. Freedom to do what I want to do. Freedom to work when and how I want to. Freedom to experiment with new ways of earning a living.

Teaching

When I wonder what I’m passionate about I see a common thread… I love to teach people. Whether it’s teaching a class of kids, training a group of adults, mentoring an employee or answering questions in mailing lists, I love to share my knowledge and experience. I love seeing the “I get it!” moments and generally watching people develop new skills.

I have a couple of ideas when it comes to exploring this passion. I’m going to create a set of screencasts on at least one topic, possibly two. I haven’t decided what model to use for revenue yet, if any, but the first few episodes will definitely be free forever.

I want to look at what else I can do in this area because it’s a very important part of my personality and I know I have a lot to give.

Photography

I love photography, and this year I want to develop this passion and my existing skills. I have recently had my first exposure of on-location portrait photography and I found it extremely fulfilling. I want to do more of it.

I want to start building a stock portfolio. Over the years I’ve taken some excellent stock shots, but I’m yet to have one accepted by iStockphoto. The reasons given have been varied, but tend to be around the areas of resolution (old camera), sharpness (interpolation didn’t fool them) and subjects. I think I have a great eye for this type of shot but I’ve never taken photos specifically for that purpose. That will change this year.

Travel

Generally speaking I hate taking holidays, but I’ve realised that this is due to my attitude to the word. Travel is something else entirely. I really enjoy exploring new places. When combined with photography it’s completely different, so I want to do a lot more of it this year.

I haven’t worked out any details yet but I’m planning a number of trips both within the UK and beyond. Some will be day trips, others will be weekends and a couple will be longer, but the idea is to get out there and see the world, little by little .

As soon as I post the form to get my passport renewed……….

Consulting

I really love internet-based technologies. Design, development, languages, architecture, tools, strategies, SEO… everything about the internet excites me… except coding. These days I’m finding that spending my hours coding away, writing the same stuff I’ve always written is leaving me excessively deflated. Except when I have an interesting problem to solve.

There’s nothing I like more than when a new and interesting problem comes along that needs solving. I will happily spend 48 hours straight designing and coding a solution, because it has grabbed my attention. If coding’s what I’m going to be doing then that’s the sort of coding I need to find. If not then I want to be architecting solutions.

In an ideal world I would be consulting for a number of startup companies, bringing my broad range of expertise and experience to the table without the burden of a full-time employee. If you’d like to talk about how I can help you check out my LinkedIn profile and/or contact me.

And more…

That doesn’t cover everything I have planned but it’s the key points and all I’m willing to reveal publicly . I’m very excited about the next twelve months and the opportunities and experiences they’ll bring. If you want to be a part of it, let me know.

I’m looking to develop my portrait photography skills so I’m throwing this out there to see if anyone’s interested in a free photo session. I take a lot of photos but aside from weddings I generally haven’t had a lot of experience taking pictures of people. I want to change that.

So here’s the deal. I’m offering to do on-location photo shoots with you and anyone else you want to involve; just you, you and your partner, you and your friends, whatever. This will give me further experience of taking portraits and add to my portfolio. As such I will require that everyone involved signs a model release.

In return you’ll get a CD containing full size image files of the best shots from the shoot and you’re welcome to do anything non-commercial with them. If you want to use them commercially you’ll need to get my permission, but I don’t see that ever being a problem. I’m also more than happy to get them professionally printed and optionally mounted.

I’m based in Surrey but would be willing to travel a reasonable distance – at the moment I don’t really know how far that is so let me know where you are and I’ll decide whether it’s too far or not

The location is completely up to you, or if I know the area I’ll happily recommend places. It’s up to you to let me know what you want to get from it, in terms of the experience and the resulting photos. I don’t have a studio or any studio equipment but I’ve found the best shots come from a less formal environment.

If you’re interested or have any questions please don’t hesitate to contact me. If you don’t ask I can’t say yes!

Firstly I’ve shut TwitApps.com down. I won’t go into the reasons because they’re well documented in the blog post announcing it. I’ve also released the code so anyone can make a clone if they feel so inspired. One big side effect of this is that I’m open to offers for that domain name and a bunch of others I don’t see myself using anymore. Please only contact me with serious offers.

Secondly I’ve been doing a lot of soul searching lately, and one of the results of that has been my new blog over at stuartdallas.com where I’m writing for my benefit rather than as a service to others, which is how I view this site. Check it out and let me know what you think.

Finally I have been all over the place in terms of my career lately, which I’ve written about on my new blog so I won’t repeat it here. Ultimately it means I’m on the lookout for interesting projects to get involved with. If you think I may be able to help you or your company please let me know.

Hopefully that explains where I’ve been. I plan to continue publishing tech and development posts on this site for the foreseeable future, but how often depends on how much my focus gets split in my new reality. I also plan to change the layout of this site – it’s starting to depress me because I never really finished it.

Back when I started working on sites big enough for these activities to cause a problem I went down the obvious route of making a PHP CLI script for each job that needed doing and getting it to run using cron. This worked for a while but as the sites I was working on got bigger and more complex it quickly became clear that this was becoming difficult to manage, so I started to consider alternatives.

At the time all this happened I was mainly working with a site that ran on Zend Platform. One of the recent additions at the time was a module called Job Queue which appeared to do exactly what I needed. Unfortunately after spending a fair amount of time developing the infrastructure required to make it run the jobs I discovered that it really wasn’t very well tested; it was far from production quality [1] and nowhere near reliable enough so I went back to the drawing board.

Time passes…

After thinking about what I needed from such a system and what I had available I came up with an arrangement I’ve been using ever since with great success.

The core of the system is a DB table and a PHP script. The table contains the definition of jobs that need running and the script, erm, runs them.

I won’t go into the details of the table because the only important parts as far as this system go are the run_at, processor_pid and schedule_* fields.

The run_at field is a timestamp that simply indicates the time when that job should be executed.

The processor_pid field is an unsigned integer that defaults to 0 and will indicate the process that’s running a job if any.

The schedule_* fields specify how often the job should be executed. There are a number of ways of organising these depending on what your requirements are, but they fall into two general categories.

Before I move on to the script I apparently need to cover something that I missed when I first wrote this. You need to specify what to run in this table! I though this was pretty obvious but based on initial feedback I was wrong.

This can take any number of forms and will depend greatly on your specific application. Over the years I’ve used a number of different methods including a PHP script name, a CLI command and a method in a static class. Whatever you use you’ll want to make sure you do sufficient checks to ensure it’s secure.

Ok, on to the script which is designed to be run by cron according to a schedule that allows it to keep up with the size of the job queue you anticipate.

The following is a list of the basic steps the script performs. For this example I’m continuing with the assumption that the job queue is a table in a database.

1. Get the pid and put it in $pid.
2. Execute this SQL query…
   update `job_queue`
   set `processor_pid` = “$pid”
   where `run_at` <= unix_timestamp()
     and `processor_pid` = 0
   order by run_at asc
   limit 1
3. Check mysql_affected_rows() and exit if 0.
4. Execute this SQL query…
   select * from `job_queue`
   where `processor_pid` = “$pid”

   Note that these SQL statements (step 2 and this one) atomically grab a job and lock it. If you’re using a different storage system for your queue you’ll need to lock it while you select a job to run and then mark it as in progress.

5. Run the job. As mentioned already this can mean any number of things and will depend on your particular application. One of the useful things you can do here is to set up a clean, safe environment for the job to run in, along with ways to capture errors and other outputs so you can do something useful with them.
6. As soon as the job has finished executing we mark it as completed and record the success or failure status.
7. If the job has a schedule (i.e. it’s a recurring job) we create a new job by effectively cloning the job we’ve just run. We now calculate the time it should be run according to the schedule definition and save that in the run_at field of the new job. Finally we set the processor_pid of the new job to 0 so it’s then available to step 2 of this process.
8. Depending on the job configuration and status we now either remove the completed job from the queue or archive it complete with errors and output for later inspection.
9. If this processor has been running for > 60 minutes it exits, otherwise it goes back to step 2 and looks for another job to run.

To get this to do something useful we configure it to run via cron every n minutes where n depends upon the anticipated size of your job queue. For example running it every minute will automatically scale it up to 60 concurrent jobs at any one time. Running it every 5 minutes will reduce this to 12, and so on. There’s also nothing stopping you putting more than one line into the crontab so it runs two processes every minute which increases concurrent processors to 120.

Assuming your job queue is network accessible this system also scales across multiple machines with minimal changes. In fact the only change that’s required it to incorporate a machine identified into the processor_pid field. This could be as simple as <machine>_<pid>; the key thing is that it’s guaranteed to be unique to a given process across your entire infrastructure.

Crashed jobs

One problem you may need to deal with is how to handle crashed jobs. This will happen, you can’t get away from it and you’ll need a way to detect and deal with it when it does. Luckily the job queue makes detection fairly straightforward.

On a single machine you can implement a script (either run via cron separately or indeed run by the job queue) that will check that for each job that has a processor_pid > 0 there is a PHP process running with that PID.

If and when you’ve scaled across multiple machines this script essentially remains the same except that you need to run it on every machine that runs the job queue and filter the PID’s you check.

As far as what to do when you find a crashed job that’s really something you need to consider on a case-by-case basis. At the very basic level the script could simply reset the processor_pid field to 0 so it gets run again. At the other end of the spectrum in a very flexible system you could have a way to run a job with a flag to indicate that it had previously crashed; each job can then deal with crashes in their own custom way.

Potential additions

The components described above is just the core of a job queue system; you can add a lot of useful stuff above and around it to make it easier to manage and provide better feedback from your periodic tasks.

• Performance metrics
  Since you’re running all your jobs from a central script adding code around the actual execution to record execution time, load and possibly memory usage too is pretty simple. You then have the ability to compare the time a job took to previous executions of that job to detect potential errors.
• Management interface
  Since you have all the information regarding the jobs in the queue, what’s running right now and the status of jobs that have previously been executed it’s a pretty small leap to build a UI that will let you view and manage the whole thing. This can be especially useful for presenting error messages and performance metrics.

Final thoughts

I hope that’s useful to someone, I’ve certainly found it applicable to most web applications I now deal with. As I mentioned a few times you can implement the various parts in a number of ways, in particular how jobs are specified and how the scheduling works.

As an example of this flexibility I’ll make a passing mention to one implementation I’ve done that only needed to be able to execute scripts daily, hourly or every 15 minutes. To accomplish this I simply created a folder for the scripts, and three folders named 15min, hourly and daily within that. The processor script uses a custom locking system and CLI arguments to run each set of scripts according the the directories they’re in.

This system has proven to work very well and will continue to do everything that site needs until we need to execute something at a specific time; a requirement that has not yet surfaced.

One of the many side projects I’m working on is a reusable version of this system. At the moment it’s a fairly messy combination of scripts that doesn’t work very well so far, but as soon as I have something worth sharing I’ll definitely do so on this blog. Stay tuned.

If you have an questions or suggestions for improvement please don’t hesitate to leave a comment or contact me privately.

[1] This was a few years back and I’ve heard that the Job Queue module has received some attention since then so is now a lot better, but I no longer use Zend Platform so I’m not in a position to comment. If budget is not an issue for you I’d recommend checking it out.

This is fantastic. Every site should do this. It’s a really simple check to do and immediately informs the user that they have probably been compromised complete with advice as to what they should do next.

Top marks to Facebook!

I understand regional restrictions. I don’t necessarily agree with the reason they’re there, but I understand why.

What I can’t understand is why they’re not shown up-front when browsing available content. On sites like Joost, Hulu and others you have to actually start streaming content before you get told you can’t view it. They clearly know where you’re coming from but they still insist on not telling you what you can and can’t watch until you actually try.

Maybe there’s a technical reason why it works this way (I can’t think of one that couldn’t be easily worked around), or maybe there’s a licensing reason for it. Whatever the reason it makes attempting to use any of these sites more and more frustrating the more region-limited content they contain.

So here’s my plea… please will these sites make it clear from the highest level possible in the browsable structure whether content is viewable from my current location. That’s all I ask.

Boxee themselves don’t help out with this problem. Take theWB for example. You can’t access the website at all if you’re outside the US (see below), but you can if you use Boxee. Right down to the content level, but when you start streaming you then get a message telling you it’s restricted.

So, Boxee and co… sort it out!!

Twitter-related stuff » Handling email notifications

On the assumption that I’ll eventually get around to sharing some more Twitter-related stuff I’ve created a new section in the projects area of this site. If there’s anything related to Twitter integration that you’d like to see me cover there please let me know and I’ll be happy to help.