FcgidMaxProcessesPerClass Broken on Debian Squeeze

The server went down today with an out of memory kernel panic when Natural Jenius was under heavy load.  I started watching the system, and we were getting a separate php5-cgi process for each simultaneous request.  According to the Apache documentation, FcgidMaxProcessesPerClass is supposed to limit the number processes spawned per virtual host.  the default value is 100, but for a server our size it should be at about 2. So I fired up vi to edit the configuration.

$ cat /etc/apache2/mods-enabled/fcgid.conf
AddHandler fcgid-script .fcgi
 FcgidConnectTimeout 20
 FcgidMaxProcessesPerClass 2
$sudo service apache2 restart
 Restarting web server: apache2 ... waiting ...

I used the Apache benchmark tool from another machine to test the configuration. I started with twenty requests, five simultaneously.

$ ab -n 20 -c 5 naturaljenius.com/category/programming
This is ApacheBench, Version 2.3
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking naturaljenius.com (be patient)...apr_poll: \
    The timeout specified has expired (70007)

What the…? When I checked the server, there were a boatload php5-cgi processes, one for each request. The server was extremely slow and running out of RAM quickly.

$ ps aux | grep php
amcoder   5977  2.4  0.0      0     0 ?        Z    06:27   0:00 [php5-cgi]
amcoder   5978  2.3  3.6 224476  9056 ?        D    06:27   0:00 /usr/bin/php5-cgi
amcoder   5979  2.4  3.4 224476  8716 ?        D    06:27   0:00 /usr/bin/php5-cgi
amcoder   5980  2.6 12.9 222412 32508 ?        D    06:27   0:00 /usr/bin/php5-cgi
amcoder   5981  2.3  3.3 224476  8524 ?        D    06:27   0:00 /usr/bin/php5-cgi
amcoder   5995  3.1  5.2 221788 13136 ?        D    06:27   0:01 /usr/bin/php5-cgi
amcoder   5996  2.8  5.2 221788 13076 ?        D    06:27   0:00 /usr/bin/php5-cgi
amcoder   5997  3.0  5.4 221788 13624 ?        D    06:27   0:00 /usr/bin/php5-cgi
amcoder   5998  3.2  5.3 221788 13316 ?        D    06:27   0:01 /usr/bin/php5-cgi
amcoder   5999  2.9  4.9 221788 12556 ?        D    06:27   0:00 /usr/bin/php5-cgi
amcoder   6025  3.4 10.3 229632 26084 ?        D    06:27   0:00 /usr/bin/php5-cgi
amcoder   6027  3.0  0.7  62692  1832 ?        D    06:27   0:00 /usr/bin/php5-cgi
amcoder   6029  0.0  0.1   6072   472 pts/0    S+   06:27   0:00 grep php

$ free -m
             total       used       free     shared    buffers     cached
Mem:           245        241          3          0          0         43
-/+ buffers/cache:        197         48
Swap:          511        430         81

After searching around, I finally found the answer. There is a bug that prevents the module from respecting FcgidMaxProcessesPerClass in the version fcgid in Debian Squeeze.  Luckily, there is a patch in that thread.

So, I pulled the source, applied the patch, built the package, and installed it.

$ sudo apt-get install devscripts build-essential fakeroot
$ apt-get source libapache2-mod-fcgid
$ sudo apt-get build-dep libapache2-mod-fcgid
$ cd libapache2-mod-fcgid-2.3.6/
$ vi modules/fcgid/fcgid_spawn_ctl.c
$ debuild -us -uc
$ cd ..
$ sudo dpkg --install libapache2-mod-fcgid_2.3.6-1_amd64.deb
$ sudo service apache2 restart

I crossed my fingers and started Apache benchmark again – this time with 200 requests and 20 simultaneous requests. (I like to live on the edge.)

$ ab -n 200 -c 20 naturaljenius.com/categories/programming
This is ApacheBench, Version 2.3
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking naturaljenius.com (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests

Server Software:        Apache
Server Hostname:        naturaljenius.com
Server Port:            80

Document Path:          /categories/programming
Document Length:        0 bytes

Concurrency Level:      20
Time taken for tests:   14.088 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Non-2xx responses:      200
Total transferred:      67000 bytes
HTML transferred:       0 bytes
Requests per second:    14.20 [#/sec] (mean)
Time per request:       1408.761 [ms] (mean)
Time per request:       70.438 [ms] (mean, across all concurrent requests)
Transfer rate:          4.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:    49  964 2909.4     54   14086
Waiting:       49  964 2909.4     54   14086
Total:         49  964 2909.6     54   14087

Percentage of the requests served within a certain time (ms)
  50%     54
  66%     54
  75%     55
  80%     56
  90%   1069
  95%   9079
  98%  12081
  99%  14086
 100%  14087 (longest request)


Sweet. A quick look at the server confirmed that the php5-cgi processes were under control and we had plenty of memory.

$ ps aux | grep php
amcoder  12142 12.2 28.3 226316 71204 ?        S    06:52   0:00 /usr/bin/php5-cgi
amcoder  12144 21.0 24.1 218700 60668 ?        S    06:52   0:00 /usr/bin/php5-cgi
amcoder  12147  0.0  0.2   6072   568 pts/0    S+   06:52   0:00 grep php

$ free -m
             total       used       free     shared    buffers     cached
Mem:           245        224         20          0          2         94
-/+ buffers/cache:        127        117
Swap:          511         52        459


  1. [...] thing came to mind! A possible bug with debian squeeze and apache? But unfortunately the next night saw even more [...]

  2. Brian says:

    This looks like can resolve my problem :) but :

    vi modules/fcgid/fcgid_spawn_ctl.c

    How to apply this patch ??

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>