Linux2.2 LinuxThreadsスレッド制限の変更
11761 ワード
This document describes how to modify the per-process file descriptor limit and task limit of the Linux version 2.2 kernel to increase the number of sockets and threads that a Java server application can create. In particular, it describes the socket and thread requirements of the VOLANO® chat server, but similar requirements are likely to exist in any server application written for the Java platform. Note that the current Linux kernel version is 2.6, so this information is unlikely to apply to any recent Linux distribution.
File descriptors and tasks
When running on the Linux 2.2 kernel, each process is limited by default to 1,024 file descriptors. Since the VolanoChat server requires one file descriptor for each person chatting through a VolanoChat applet, you are limited by default to a maximum of 1,024 simultaneous users when using the Linux 2.2 kernel.
Some Java virtual machines, such as the IBM Developer Kit for Linux, Java Technology Edition, use native threads on Linux which require one task for each Java thread. Since VolanoChat requires two Java threads for each connection, you may need to increase the maximum number of tasks per user for such virtual machines. The default maximum number of tasks in a Linux 2.2 system is 512, with a maximum of 256 tasks per user. So by default, a VolanoChat server running under IBM JDK 1.1.6 for Linux can support a maximum of 128 people chatting at the same time.
You can increase the limits testing the the number of file descriptors and the number of tasks in the Linux 2.2 kernel by following the steps below. These steps were performed on a Red Hat Linux 6.0 system using Linux kernel 2.2.5-15. The instructions may be different for your Linux system.
To increase the maximum file descriptors per process to 4,096 (from 1,024) and the maximum tasks per user to 4,090 (from 256) on Linux 2.2, do the following steps. Change the following C header files. In To allow users to increase their file descriptor limits, change the following configuration files: In To increase the system-wide file descriptor limit, add the following three lines to the Change to the directory Make any other kernel configuration changes, such as setting the processor family type, by entering the command below. Save your changes and exit the configuration panel when you are done. Prepare for the kernel build by entering: Rebuild your kernel with the commands: Backup your old modules directory by entering (for example): Create a new set of kernel modules with the command: Backup your old kernel by entering (for example): Copy over your new kernel image with: Edit the Linux loader configuration in Generate the new Linux loader configuration by running: Shutdown and reboot with your new kernel by entering: After rebooting, verify that the changes were effective as shown in the sections below.
LinuxThreads 0.7 limits
Even after you rebuild your Linux 2.2 kernel to increase the maximum number of threads per Linux user from 256 to 4,090, there are still definitions in the thread library itself which limit the number of threads per process to less than 1,000. The VolanoChat server requires two threads for each person chatting through a VolanoChat applet, so the LinuxThreads implementation creates a limit of 500 people to a single VolanoChat server.
You can increase this limit to 8,192 threads with the following steps. Even though the thread library will allow 8,192 threads per process after these changes, you'll still be limited by the Linux 2.2 kernel to a maximum of 4,090 threads for the Linux user account running the VolanoChat server after the changes in the previous section. The steps below are shown for a Red Hat Linux 6.0 system, so they may be different for your Linux distribution. Install the Glibc 2.1.1 source package from Unpack the source archive Change the following C header files. In Rebuild the source archive, Change to the RPM specification directory, After a few hours, the new LinuxThreads library will be found as Copy the new thread library into your own Define the environment variable Verify the new thread limits as instructed below.
Verifying the new file descriptor limit
Increase your new limits to their maximum by entering
Both numbers printed should give your new file descriptor limit.
Verifying the new thread limit
You can use the following program to verify your new thread limits in the modified Linux 2.2 kernel and in the modified LinuxThreads library.
Compile the program with:
Running with the original LinuxThreads library will print:
while running with the modified LinuxThreads library will print:
You may need to enter
Notice that the thread stack addresses are spaced apart by 256 kilobytes instead of the earlier reserve of 2 megabytes per stack. Also notice that the program can create threads right up to the Linux 2.2 kernel limit of 4,090 for the system rather than the earlier LinuxThreads limit of 1,024 per process.
http://www.volano.org/articles/linux-2.2-sockets-threads/
File descriptors and tasks
When running on the Linux 2.2 kernel, each process is limited by default to 1,024 file descriptors. Since the VolanoChat server requires one file descriptor for each person chatting through a VolanoChat applet, you are limited by default to a maximum of 1,024 simultaneous users when using the Linux 2.2 kernel.
Some Java virtual machines, such as the IBM Developer Kit for Linux, Java Technology Edition, use native threads on Linux which require one task for each Java thread. Since VolanoChat requires two Java threads for each connection, you may need to increase the maximum number of tasks per user for such virtual machines. The default maximum number of tasks in a Linux 2.2 system is 512, with a maximum of 256 tasks per user. So by default, a VolanoChat server running under IBM JDK 1.1.6 for Linux can support a maximum of 128 people chatting at the same time.
You can increase the limits testing the the number of file descriptors and the number of tasks in the Linux 2.2 kernel by following the steps below. These steps were performed on a Red Hat Linux 6.0 system using Linux kernel 2.2.5-15. The instructions may be different for your Linux system.
To increase the maximum file descriptors per process to 4,096 (from 1,024) and the maximum tasks per user to 4,090 (from 256) on Linux 2.2, do the following steps.
/usr/include/linux/tasks.h
, change: NR_TASKS 512 to 4090
MAX_TASKS_PER_USER (NR_TASKS/2) to NR_TASKS
In /usr/include/linux/limits.h
, change: NR_OPEN 1024 to 4096
OPEN_MAX 256 to 4096
In /usr/include/linux/posix_types.h
, change: __FD_SETSIZE 1024 to 4096
In /usr/include/bits/types.h
, change: __FD_SETSIZE 1024 to 4096
/etc/security/limits.conf
, add the lines: * soft nofile 1024
* hard nofile 4096
In /etc/pam.d/login
, add: session required /lib/security/pam_limits.so
/etc/rc.d/rc.local
startup script: # Increase system-wide file descriptor limit.
echo 8192 > /proc/sys/fs/file-max
echo 24576 > /proc/sys/fs/inode-max
/usr/src/linux
and set up the kernel sources with: make mrproper
make xconfig
make dep
make clean
make bzImage
make modules
mv /lib/modules/2.2.5-15 /lib/modules/2.2.5-15.old
make modules_install
mv /boot/vmlinuz /boot/vmlinuz.old
cp /usr/src/linux/arch/i386/boot/bzImage /boot/vmlinuz
/etc/lilo.conf
so that your new kernel is the default and your old kernel is still available from the boot prompt, as in the example shown below: boot=/dev/hda
map=/boot/map
install=/boot/boot.b
prompt
timeout=50
image=/boot/vmlinuz
label=linux
root=/dev/hda1
read-only
image=/boot/vmlinuz.old
label=linux.old
root=/dev/hda1
read-only
/sbin/lilo
shutdown -r now
LinuxThreads 0.7 limits
Even after you rebuild your Linux 2.2 kernel to increase the maximum number of threads per Linux user from 256 to 4,090, there are still definitions in the thread library itself which limit the number of threads per process to less than 1,000. The VolanoChat server requires two threads for each person chatting through a VolanoChat applet, so the LinuxThreads implementation creates a limit of 500 people to a single VolanoChat server.
You can increase this limit to 8,192 threads with the following steps. Even though the thread library will allow 8,192 threads per process after these changes, you'll still be limited by the Linux 2.2 kernel to a maximum of 4,090 threads for the Linux user account running the VolanoChat server after the changes in the previous section. The steps below are shown for a Red Hat Linux 6.0 system, so they may be different for your Linux distribution.
/mnt/cdrom/SRPMS/glibc-2.1.1-6.src.rpm
on the Red Hat Linux CD-ROM #2. /usr/src/redhat/SOURCES/glibc-990416.tar.gz
so that you can modify two of the files. /usr/src/redhat/SOURCES/glibc/linuxthreads/internals.h
, change the size of the thread stack reserve from 2 megabytes down to 256 kilobytes (with a page size of 4,096 bytes): STACK_SIZE (2 * 1024 * 1024) -> (64 * PAGE_SIZE)
In sysdeps/unix/sysv/linux/bits/local_lim.h
under /usr/src/redhat/SOURCES/glibc/linuxthreads
, change the Posix thread implementation limit from 1,024 per process to 8,192 per process: PTHREAD_THREADS_MAX 1024 -> 8192
/usr/src/redhat/SOURCES/glibc-990416.tar.gz
, to include your changes, replacing the original copy. /usr/src/redhat/SPECS
, and enter the command: rpm -ba glibc-2.1.spec
libpthread.so
under: /usr/src/redhat/BUILD/glibc/build-i386-linux/linuxthread
lib
subdirectory: cd /usr/src/redhat/BUILD/glibc/build-i386-linux
cp linuxthread/libpthread.so ~/lib
cd ~/lib
ln -s libpthread.so libpthread.so.0
LD_LIBRARY_PATH
to include the subdirectory containing the new thread library libpthread.so
. Verifying the new file descriptor limit
Increase your new limits to their maximum by entering
"ulimit -n 4096"
for the Bash shell or "unlimit"
for the C shell. You can then check whether your changes were effective by running the following C program: #include <sys/types.h>
#include <fcntl.h>
#include <stdio.h>
void main(void) {
int k = 3;
while (dup(0) != -1)
k++;
printf("Opens = %d Fdsize = %d
", k, sizeof(fd_set) * 8);
}
Both numbers printed should give your new file descriptor limit.
Verifying the new thread limit
You can use the following program to verify your new thread limits in the modified Linux 2.2 kernel and in the modified LinuxThreads library.
#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
#define MAX_THREADS 10000
int i;
void run(void) {
char c;
if (i < 10)
printf("Address of c = %u KB
", (unsigned int) &c / 1024);
sleep(60 * 60);
}
int main(int argc, char *argv[]) {
int rc = 0;
pthread_t thread[MAX_THREADS];
printf("Creating threads ...
");
for (i = 0; i < MAX_THREADS && rc == 0; i++) {
rc = pthread_create(&(thread[i]), NULL, (void *) &run, NULL);
if (rc == 0) {
pthread_detach(thread[i]);
if ((i + 1) % 1000 == 0)
printf("%i threads so far ...
", i + 1);
}
else
printf("Failed with return code %i creating thread %i.
",
rc, i + 1);
}
exit(0);
}
Compile the program with:
gcc -lpthread -o count count.c
Running with the original LinuxThreads library will print:
blue:/home/john> count
Creating threads ...
Address of c = 3137535 KB
Address of c = 3135487 KB
Address of c = 3133439 KB
Address of c = 3131391 KB
Address of c = 3129343 KB
Address of c = 3127295 KB
Address of c = 3125247 KB
Address of c = 3123199 KB
Address of c = 3121151 KB
1000 threads so far ...
Failed with return code 11 creating thread 1023.
while running with the modified LinuxThreads library will print:
blue:/home/john> count
Creating threads ...
Address of c = 3144703 KB
Address of c = 3144447 KB
Address of c = 3144191 KB
Address of c = 3143935 KB
Address of c = 3143679 KB
Address of c = 3143423 KB
Address of c = 3143167 KB
Address of c = 3142911 KB
Address of c = 3142655 KB
1000 threads so far ...
2000 threads so far ...
3000 threads so far ...
4000 threads so far ...
You may need to enter
Ctrl-C
to exit the program. Notice that the thread stack addresses are spaced apart by 256 kilobytes instead of the earlier reserve of 2 megabytes per stack. Also notice that the program can create threads right up to the Linux 2.2 kernel limit of 4,090 for the system rather than the earlier LinuxThreads limit of 1,024 per process.
http://www.volano.org/articles/linux-2.2-sockets-threads/