Stut.net

Template-based DNS Zone Management

2012-12-03T00:00:00+00:00

I own nearly 100 domain names, which is a pretty modest number against some people and organisations I know. Most of them are currently registered with Gandi.net and I've always found their service to be good value for money and that's still true today.

Most of the DNS for those domain names is also hosted with Gandi. Why? Because it's included in the registration cost. However, their zone management tool, while it had a major overhaul in the last year which has made it a lot better, it's still a major pain to use. Unfortunately this seems to be a common theme with DNS management interfaces.

I've recently been setting up a number of development systems for a couple of clients, and these have needed DNS changes so I've been using that tool a lot more lately than I would normally, and it's frustrating.

When I look at my zones I see a lot of duplication. Nearly all my email is with Google Apps, so most zones have the same MX records. I don't have a huge number of servers so most of the rest breaks down into a few small groups where the details are the same.

The duplication means that if I wanted to change my email provider it would be a major pain to change all of the MX records. I decided that this can't just be me and that there must be a zone management solution out there that manages domains based on a set of templates rather than individually. Either my Googling skills are waning or this doesn't exist outside an ISP control panel, and even then I'm not sure they're actually providing what I want.

So, what do I want? This calls for some bullet points:

To define a set of zone templates, with descriptive names like "Google Apps MX records".
With simple variable substitution. Some variables would be pre-defined based on the zone or other expected data, others would be completely custom.
To define a set of zones, then build the zone information from a combination of other zones, templates and individual entries.
It would also be nice to be able to organise zones into categories, with the ability to set variables on the category for use in templates.

That way I could define a zone as:

Standard site on b.3ft9.com with a static server at xyz on Rackspace CloudFiles with CDN enabled.
Google Apps MX records.

This would set up @ and www to point at the IP for b.3ft9.com, and static as a CNAME to xyz.rackcdn.com, and MX records for Google Apps complete with SPF TXT entry.

Now, let's say 40 of my domains are on b.3ft9.com and all follow this same pattern so they all have similar zones. Then I decide to switch from Rackspace CloudFiles to Amazon S3 it's simply a case of changing the "Standard site" template. The same applies if I decide to change my mail provider Simples.

The management interface would let me see both the templates that a particular zone is using and the zones that are using a particular template. That way I can be sure that I know what I'm affecting when I change a template.

If such a thing exists please tell me before I decide to go ahead and implement it myself. I find it hard to believe that this doesn't already exist in some form or another!

If I were to build it I'd be inclined to do it as a set of CLI tools that work on files, that way you could store your templates and zones in a version control system and generate standard zone files.

Very interested to hear opinions on this. I recently tried Amazon's Route 53 interface having already decided that they would have built something that could cope with users managing hundreds of domains, but apparently I was wrong. I guess my use case is specific to the independent developer who has a lot of ideas on the go at once, not the enterprise which probably has very few domains and rarely changes their hosting arrangements.

I still think there's a need for something similar to what I describe. Another one for the to do list.

PHP Snippet: HTTP Authentication

2012-11-11T00:00:00+00:00

HTTP authentication is the easiest way to make a page or area of a website secure. It's very easy to accomplish with pure PHP, so no web server configuration is required making it a lot more portable.

This function implements a very simple HTTP Basic Auth authentication system. Simply call it before you do anything else in your script, pass it an array of valid users (username => password), an optional description of what's being secured and it will do the rest.

Example usage:

Note that the function doesn't return anything or throw any exceptions. If it can't validate the user it will end the script after sending back an appropriate response.

Handling email notifications

2011-11-12T00:00:00+00:00

This article has previously been published on this site, but it got lost somewhere in the middle of several site rewrites.

Twitter sends notifications of new followers and direct messages by email. While you can use the API to keep track of those things it wastes hits and it's incredibly inefficient especially for accounts with minimal activity.

I run several Twitter-based services but they all share the same email address. This allowed me to use the same email handling setup to process all email coming in from Twitter. So far it's working really well but it means this script is a bit more involved than it needs to be.

I've littered the code with comments - far more than should be necessary, but it saves me from going through it step by step here. Read, learn and enjoy. If you have any questions or comments feel free to contact me on Twitter.

The script uses the mailparse extension so that must be available.

To use the script you just need to stick it somewhere accessible by the mail server, check the #! line will work, make it executable and add a piped alias to your aliases file - see your MTA documentation for details. My server runs Postfix and I have the following line in /etc/aliases...

twitmail: |/var/www/twitapps.com/common/incoming_mail.php

In my case I also have a line in the transport map to point the full email address including domain at that local alias. Your MTA requirements will vary.

!! Be sure to keep this email address secret !!

Due to the way email works there is no real way to validate that the email came from Twitter, and as a result nor is it reasonable to assert that the user mentioned in the email actually did what it says they did. The only real way to protect against fake emails at the moment is to ensure you use an email address that nobody else knows. Apply the same rules to this email address as you would to a password and you'll be fine. It's not great security but it's what we've got and I've not heard of anyone having a problem so far but you don't want to be the first!!

Writing handlers

The last piece of this solution is the actual handler scripts. There's nothing special about them - they can do whatever you need them to do. Here's an example is_following handler.

From within a handler script you have a few useful variables available from the main script...

$headers

This is an array of all the headers from the email that started with X-Twitter. At the time of writing this give you access to...

Key	Contains
createdat	When the email was created
emailtype	What type of notification it is, e.g. is_following or direct_message
senderid senderscreenname sendername	The ID, screen name and name of the sender
recipientid recipientscreenname recipientname	The ID, screen name and name of the receipient

$body

The unmodified body of the message. Note that for direct messages this contains more than just the message itself.

$info

This variable contains the return value from mailparse_msg_get_part_data. The most useful part of this is the raw headers from the incoming email ($info['headers']).

Errors and other output

If anything goes wrong during the processing of a message an error message along with the raw incoming email will be sent to the address in $____email_address (specified near the top of the file). This includes any output from the handler scripts themselves. The script will never bounce an email due because you don't want Twitter to decide the address is not valid.

Caveat

There is one important point regarding email notifications from Twitter that you need to be aware of, and that's the fact that you may not always get a notification for every event that happens.

The reason for this is spam. Twitter is still battling against a deluge of users and bots whose only purpose is to try an spam the legitimate users of their service. One of their defenses against this is to recognise spamtastic patterns and take action to minimise the impact on other users. One of the actions they take is to not send email notifications from users flagged as possible spammers.

One important side-effect of this is that a user unfollowing you and then following again will not necessarily trigger an is_following notification.

For TwitApps I have a cron job that runs once a day and checks the followers using the API and "catches up" with any that have been missed. This means most users get an instant response but there's a backup process that makes minimal use of the API to ensure that if anyone does get missed they only wait up to 24 hours for a response.

Education as we know it doesn't work

2011-10-19T00:00:00+00:00

I feel like I wasted large parts of my childhood by going through the "standard" British education system. It wasn't really until I went to university, and even then not until my second year, that I felt like I was doing something productive. I should sprinkle those statements with a note pointing out that my memories of my childhood are few so I don't entirely trust the impression I have of that period, but the overwhelming recollection I have is of being bored and uninterested in most of what I was being asked to do.

Now that you know that you'll probably be more able to appreciate my appreciation of the following video...

I can't adequately express how much I think every single person on the planet needs to watch that video. Aside from the awesome animation style (see more of this on the Cognitive Media site), the way Ken Robinson describes the assembly line style of education that we consider to be normal is dead on.

We've applied the "advances" of the industrial revolution to education, and it doesn't work. It's like building a machine to make bowling pins out of Bonsai trees. Sure, you can make it work, but in doing so you remove all the parts of the tree that make it so interesting. You take something full of variation and character, and strip it of everything that's different by pointing it out and labelling it as unwanted. That's what we're doing to our kids.

A little while ago I posted the following video of Sir Ken's talk from a TED conference in 2006. In this talk he puts forward the idea that schools kill creativity.

I agree completely. I've seen it. I've experienced it. I've fought against it.

Now, a lot of you reading this are probably thinking that I blame the teachers. After all, they're the key figure in kids' school experiences. But no, I don't. My family has a strong educational theme, both in the present and in the past, and I see a lot of teachers struggling against the constraints the system puts on them. The result-focused measurement of educational performance is imposed by the system along with a one-size-fits-all curriculum (albeit with minor allowable variations), which leaves little room for mainstream schools and teachers to cater for the wide spectrum of skill levels and learning styles that you find in any typical school.

One fairly recent phenomenon that he mentions in both talks is the emergence and explosion of ADHD. I believe ADHD exists, but it's not what they say it is. It's not a mental deficiency. It's not a disease. It's boredom.

A good friend once introduced me to a simple but mind-blowing idea. Most of the time, telling kids off when they "misbehave" is like being cross with a fly for landing on your food. They're not doing it with malicious intent, they're just behaving out of instinct. You're basically telling them off for not behaving like an adult. The ironic thing is that by telling them off you're teaching them how to get your attention, so they repeat it, often finding it funny the more irate you appear to be. This leads you to frustration which will commonly result in the cycle repeating. Thus begins the "normal" parent-child relationship.

ADHD is no different. ADHD is what "professionals" have taken to labeling kids who don't do well in the one-size-fits-all school system. Just because a kid can't sit still for hours on end listening to someone telling them the answers so their school can get good results, doesn't mean they're sick.

(Yes, I said "so their school can get good results" and I meant it!)

Take, for example, Clifford Stoll. I love this guy. He epitomises the way kids want to be; the way kids are. The way we spend years conditioning our kids to regard as "bad behaviour".

Make sure you watch it to the end and pay attention to how animated he is when he tells the story at the end. Note how that expression moves from his body to his voice when he gets to the point of the story.

Isn't he great? The world excites him, and he's not afraid to show it. He clearly has difficulty focusing on anything that doesn't interest him. Throughout that 18 minute talk he jumps from subject to subject and emotion to emotion, barely stopping to take a breath. If I ever have kids I hope they're just as fantastic as he is.

I titled this post "Education as we know it doesn't work" and it's missing a key part of the point. It does work if you want the society we have, but if you want a society where everyone is considered valuable regardless of their contribution, and creativity in all its forms is celebrated not stifled, education as we know it does not work.

One size does not fit all, it never has, and I hope to [insert deity here] it never will!

Sonalight Voice to Text

2011-10-13T00:00:00+00:00

So, the iPhone 4S has been announced, and the big story is Siri. I'm absolutely loving the promises and have dreamed of something similar for a while, so much so that I'm considering getting one despite being perfectly happy with my iPhone 4. Then I see this post on Techcrunch talking about a free voice to text app. Wow.

Here's the demo video...

The genius of this is not that it lets you send a text by talking to the phone. Siri can do that. The genius of this is two-fold. First, and most importantly for the driving example they focus on, is that you don't need to touch or even look at the phone to trigger the thing to respond to you. It's Star Trekky - always listening for you to speak the magic phrase. And second, it's not a Google app. A third-party app can hook into the voice recognition and text-to-speech engines. Just think of the possibilities!

Is this enough to convince me to get an Android phone and skip the iPhone 4S? I think it might be, but it's a decision I need to think about a bit more. The possibility of writing a mobile app with speech recognition and text-to-speech capabilities has my creative itches becoming very active.

OTOH I have a lot of apps on my iPhone that I'd find it hard to live without. Research time!

Rackspace CloudFiles plugin for Jekyll

2011-09-27T00:00:00+00:00

I posted a few months ago about the conversion of Stut.net to Jekyll. In that post I mentioned that I was planning to use it for a few other sites. One of those is my Photography site, and that's where the requirement for this plugin came from.

If you're a regular reader you've probably noticed that most of the sites I design have very few images which only leaves Javascript and CSS assets. These tend to be limited in number and size, so the use of a CDN doesn't make much sense. The same can't be said for a photography site. Rather than manage this manually (ouch!) I decided to write a plugin to automate as much as possible.

The plugin is up on GitHub. It's a single file and is pretty simple, but I thought I'd lay out my thinking behind what it does and how it works.

These were my requirements...

Implement a way of decorating the URLs so I can control which assets get pushed to the CDN rather than automatic processing of all assets
As far as possible remove all caching issues
Minimize operations and traffic to the CloudFiles service (both are charged by usage)
Allow the use of a CNAME, otherwise use the CDN URL provided by CloudFiles
Allow me to specify a prefix so I can organise the container
Clean up any unused objects in the container that match the prefix
Enable use of both the US and UK datacentres
Allow the plugin to be disabled, and when disabled ensure that the generated site still works

The plugin implements a tag for the Liquid templating system called cloud_files which tells the plugin to process a filename and replace it with the CDN URL. That takes care of the second requirement.

It uses the SHA1 hash of the file contents as the filename while retaining the source file's extension. Since that means that changes to the file will change the filename, there won't be any caching issues.

The plugin keeps track of the files and SHA1 hashes that it has already uploaded which ensures it limits it's interactions with the server to those that are actually required. I've just realised there's an additional optimisation I can make here by fetching the list of existing objects on initialisation and consulting that instead of asking the server whether each individual object is there. That could potentially lead to unnecessary uploads, but is more likely to simply reduce the number of operations it performs.

You can configure a CNAME, and it will fall back to the CDN URL provided by CloudFiles. You can specify a prefix that will get added to the name of the object. You can specify the datacentre your account is on, and it defaults to the US.

You can disable the plugin which will simply return the text (the URL) that was passed in. To make this work I had to require that all URLs are absolute, meaning they start with a / and exist at that location relative to the Jekyll project directory. I don't see this as a great issue, but it's not ideal.

The only requirement I haven't yet met is that of cleaning up unused objects. There doesn't seem to be a way to have something run when Jekyll has finished generating the site. I've implemented the code that will do the cleanup, but it's useless and untested until I can find a way to call it at the right time.

Other than that it does everything I wanted. Take a look and let me know what you think.

mysql_real_escape_string is not enough

2011-09-15T00:00:00+00:00

Most developers worth their income know this already, but I've never seen it explained as well as it was by Alex Nikitin on the PHP-General mailing list yesterday.

This was fine in the days of ASCII, but the tubes are hardly ASCII anymore, with Unicode, UTF-16, i have 1,112,064 code points, they are not even called characters anymore, because they really aren't. And if you are familiar with best-fit mapping, you would know that there are now dozens of characters that can represent any single symbol in ASCII, meaning that using the above type of blocking mechanisms is silly and technically insecure.

Alex goes on to suggest a couple of ways around this problem, so the full email is well worth reading: http://marc.info/?l=php-general&m=131603743606025&w=2

Fortune's Tower in PHP

2011-08-16T00:00:00+00:00

When I get bored while working I choose something from a list of things I want to have a go at someday. Some things are "read this blog post", or "learn about this technology", but most are along the lines of "make this". Today I picked a "make this" item... to implement the game Fortune's Tower.

I came across this game in the Fable® II Pub Games on my Xbox. I have never actually played Fable® II, and I've barely touched the other pub games in the Xbox Live Arcade Pub Games, err, game, but I keep returning to Fortune's Tower. While playing it I've tried various strategies but always wanted an automated way to test whether my strategies are actually any good. The best way to do that is to write some code that can play the game, so that's what I did.

Since I was doing Ruby at the time I decided to switch to PHP for this quick project. I had previously sketched out the rules of the game (I do that a lot - yup, I've killed a lot of trees in my time), so I got straight into the coding and had a working solution about half an hour later. After a few refinements, a couple of nasty bugs quashed and some display tweaking I had something that played the game and called out to a function after each row to get a decision as to whether to take the row or chance the next one.

Here's a video of the code in action...

I've put the quick 'n' dirty code up on GitHub. Run it by executing play.php. It uses very simple logic to make decisions...

If we've used the gate card, accept the offer if it's greater than 15.
Accept the offer if it's greater than 25.
Reject the offer.

I've spent some time playing around with the logic in the getDecision function, but I'm sure it could be improved. I've also played with the cost per game to see what effect that has and 15, the value used in the Xbox game, seems to be the Goldilocks value.

Anyway, feel free to play and let me know how many rounds you can reach. My maximum so far with the code as it is on GitHub is 254, but it tends to be closer to 20 most of the time.

Introducing FURL

2011-08-06T00:00:00+00:00

A couple of hours ago I needed a break from the project I'm currently working on, so decided to flex my Node.js muscles again. This time I've developed a really simple but useful server that presents a simple API for resolving short URLs to their final destination.

You can find it on GitHub: https://github.com/3ft9/furl

Photographers: Stand Your Ground

2011-08-05T00:00:00+00:00

Based on the following video it would appear that the police have finally gotten around to understanding the law. The same cannot be said for private security personnel!

On Tuesday 21 June 2011 six photographers were assigned different areas of the City to photograph. Some used tripods, some went hand held, one set up a 5 x 4.

All were instructed to keep to public land and photograph the area as they would on a normal day. The event aimed to test the policing of public and private space by private security firms and their reaction to photographers.

All six photographers were stopped on at least one occasion. Three encounters led to police action.

This is what happened.