Beware of the chmod
One facet of my paid job is to host thousands of dynamic websites on a Caudium server. We use a bunch of custom 2nd level virtual hosting related modules as well as many custom RXML tags and containers to help our webmasters to builds websites and do complex applications, like online shops. And for the curious, we host some PHP apps too. We noticed the performance of our webserver was decreasing for the last weeks. The sites were served more slowly and the load average of the Linux server went higher. The 'ps' and 'top' command regularly showed the processus 'pike' on state 'D' (= uninterruptible sleep), which means we were waiting for some I/O. We worked to find the bottleneck: all of our modules were audited and optimized, with little success: they should have been done the right way from the beginning :) We checked the Caudium configuration and some parts of the Caudium code, but nothing seemed to be wrong here either. Stracing Caudium didn't showed something usefull. Then, we went down the filesystem. From what i could read on the net, ReiserFS could handle easily theses thousands of directories and files. So, nothing should be wrong here. Thus, i was stuck in front of my top, showing pike regularly being in state 'D'. By decreasing the delay interval to 1 second (top -d1), i saw that when pike was in status 'D', a chmod process sometimes briefly showed up in the process list, on 'D' state itself. After investigating, it appeared a shell script invoqued by the crontab was changing the permissions on some hundreds megabytes of data every minute on this server. These data were never accessed by Caudium, it wasn't website data. But, they were on the same partition, and it was sufficient to dramatically slow down the webserver and some other services. Since these data were constantly growing, 'chmod' took more and more time to complete, more and more slowing down other processes that accessed this filesystem. The script being now fixed, my Caudium server happily serves its websites under a low load.
Permalink No comments / Post a Comment