Comments on: PHP Job Queue http://stut.net/2009/05/29/php-job-queue/ Ramblings of a random software engineer Fri, 12 Mar 2010 17:55:38 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Kevin Schroeder http://stut.net/2009/05/29/php-job-queue/comment-page-1/#comment-13679 Kevin Schroeder Fri, 12 Mar 2010 17:55:38 +0000 http://stut.net/blog/?p=380#comment-13679 I noticed that you had at the bottom stated that the Job Queue had been worked on. Actually, with Zend Server 5, it's been completely re-architected. It is faster and more stable in Zend Server than in Zend Platform. I have a blog post that I wrote about it at <a href="http://bit.ly/biyW9M" rel="nofollow">Do you queue? Introduction to the Zend Server Job Queue</a> I noticed that you had at the bottom stated that the Job Queue had been worked on. Actually, with Zend Server 5, it’s been completely re-architected. It is faster and more stable in Zend Server than in Zend Platform.

I have a blog post that I wrote about it at Do you queue? Introduction to the Zend Server Job Queue

]]> By: Job Queue in Symfony | SymfonyLab http://stut.net/2009/05/29/php-job-queue/comment-page-1/#comment-13649 Job Queue in Symfony | SymfonyLab Sun, 07 Mar 2010 15:42:36 +0000 http://stut.net/blog/?p=380#comment-13649 [...] 4. And eventually if you think that none of provided solutions is suitable you can always write own one using this article which provides general ideas on job queue implementation: http://stut.net/2009/05/29/php-job-queue/ [...] [...] 4. And eventually if you think that none of provided solutions is suitable you can always write own one using this article which provides general ideas on job queue implementation: http://stut.net/2009/05/29/php-job-queue/ [...]

]]> By: Stut http://stut.net/2009/05/29/php-job-queue/comment-page-1/#comment-12338 Stut Fri, 29 May 2009 21:36:44 +0000 http://stut.net/blog/?p=380#comment-12338 Aha, indeed, it's important to limit the number of processes that run concurrently. Aha, indeed, it’s important to limit the number of processes that run concurrently.

]]> By: Stut http://stut.net/2009/05/29/php-job-queue/comment-page-1/#comment-12337 Stut Fri, 29 May 2009 21:34:53 +0000 http://stut.net/blog/?p=380#comment-12337 Thanks Nathan. I avoid the forking model because it has the potential to run away with itself, whereas using the system I've described above it's limited to a certain number of concurrent job processors. This can mean that jobs don't run at an accurate time if things get backed up, but in return you're guaranteed that it won't start hundreds or thousands of processes when it tries to catch up. Thanks Nathan. I avoid the forking model because it has the potential to run away with itself, whereas using the system I’ve described above it’s limited to a certain number of concurrent job processors. This can mean that jobs don’t run at an accurate time if things get backed up, but in return you’re guaranteed that it won’t start hundreds or thousands of processes when it tries to catch up.

]]> By: Nathan Rixham http://stut.net/2009/05/29/php-job-queue/comment-page-1/#comment-12336 Nathan Rixham Fri, 29 May 2009 21:32:24 +0000 http://stut.net/blog/?p=380#comment-12336 actually I confused things there.. i don't for for each job; I grab 200 then array_chunk( $jobs, 20); then do a process for each block of 20, so at max 10 concurrent process running, and only ever as many as you need. apologies, details like that make or break these things. actually I confused things there..

i don’t for for each job; I grab 200 then array_chunk( $jobs, 20); then do a process for each block of 20, so at max 10 concurrent process running, and only ever as many as you need.

apologies, details like that make or break these things.

]]> By: Nathan Rixham http://stut.net/2009/05/29/php-job-queue/comment-page-1/#comment-12335 Nathan Rixham Fri, 29 May 2009 21:30:13 +0000 http://stut.net/blog/?p=380#comment-12335 Nice, and, snap! I've been using the same method almost identically for a few years now as well, honestly can't say I've found anything that runs better. general principle is: database of jobs get batch of all jobs due to be run (max 200) foreach jobs fork for each job process job do next batch also I run a second deamon doing the same thing purely picking up errors and on demand job sets; (for when a big report or a custom scrape / import needs done) works v well especially when grabbing remote files (due to multi-process). regards & nice post Nice, and, snap!

I’ve been using the same method almost identically for a few years now as well, honestly can’t say I’ve found anything that runs better.

general principle is:
database of jobs
get batch of all jobs due to be run (max 200)
foreach jobs
fork for each job
process job
do next batch

also I run a second deamon doing the same thing purely picking up errors and on demand job sets; (for when a big report or a custom scrape / import needs done)

works v well especially when grabbing remote files (due to multi-process).

regards & nice post

]]>