Overcoming the Limit of Thread Creation

We want to hear from you! Please send us your FEEDBACK .

The following Technical Article may contain actual software programs in source code form. This source code is made available for developers to use as needed, pursuant to the terms and conditions of this license.

Introduction

This Technical Article discusses the limit of thread creation and the ways to overcome this limit. It will also provide an in-depth explanation of the internals of thread creation.

Problem Description

When developing an application that creates a large number of threads, after approximately the 3399th call to pthread_create(...) and subsequent calls thereafter the developer gets following error message:
errno 11 - EAGAIN - Resource temporarily unavailable .

The man page for pthread_create()in the Solaris[tm] 8 Operating Environment describes this error as the "ENOMEM". This is a bug; the man page has been corrected for a later version of Solaris, since POSIX standard expects EAGAIN for lack of memory.

In the following example, in each case, the thread performs some processing and then exits by running off the end of the thread function. Using the pthread_exit() function at the end of the threadFunc, creating and deleting pthread_t objects or inserting the sleep() function in the pthread creation loop have no effect on creating more than 3399 threads. The following sample code illustrates this point.
 

/* Compile with:
* gcc pt.c -lpthread -lposix4 -o pt
* cc -mt pt.c -lpthread -lposix4 -o pt
* Crashes after 3399 threads with 'Resource temporarily unavailable'
*/
 

#ifndef _REENTRANT
#define _REENTRANT
#endif

#include  <pthread.h >
#include  <stdio.h>
#include  <string.h>

#ifndef MAX_THREADS
#define MAX_THREADS 4000
#endif

void* threadFunc(void* pParam) {
       pthread_exit((void *)pthread_self());
}

int main(int argc, char** argv) {
     pthread_t threads[MAX_THREADS];
     int res;
    uint_t threadCount;

    printf("Testing for thread resource problem\n");
    threadCount = 0;
    while (( threadCount  <  MAX_THREADS ) &&
          ((res = pthread_create(&threads[threadCount], NULL, &threadFunc, (void*)threadCount)) == 0)) {
                     threadCount++;
    }
    if (res != 0) {
        printf("Stopped with error %d: %s trying to create thread #%d\n", res, strerror(res), threadCount);
    }
   else {
        printf("Test completed successfully\n");
   }
}

Note: Compile the program with Forte[tm] Workshop 6 Update 2 on a machine running Solaris 8.

testmachine% cc -mt -o pt pt.c -lpthread -lposix4

testmachine% ./pt

"Testing for thread resource problem Stopped with error 11: Resource temporarily unavailable trying to create thread #3399"
 

Solution

The above situation may appear to occur because of the way the threads are created using the default attributes. The value of the default attributes are as follows:
 
 

Attribute	Value	Result
scope	PTHREAD_SCOPE_PROCESS	New thread is unbound - not permanently attached to LWP.
detachstat	PTHREAD_CREATE_JOINABLE	Exit status and thread are preserved after the thread terminates.
stackaddr	NULL	New thread has system-allocated stack address.
stacksize	1 megabyte	New thread has system-defined stack size.
priority		New thread inherits parent thread priority.
inheritsched	PTHREAD_INHERIT_SCHED	New thread inherits parent thread scheduling priority.
schedpolicy	SCHED_OTHER	New thread uses Solaris-defined fixed priority scheduling; thread run until preempted by a higher-priority thread or until they block yield.

The default attributes imply that they are "joinable" (that is, the application can at some later time extract information from a deceased thread as to its return code). To facilitate joining, the threads libraries must maintain return code information for the deceased thread. The user application may try to achieve this by simply keeping the user level thread structure around, but freeing data such as the stack, which may be attached to it until the pthread_join() is done or the application exits. It is the accumulation of these structures which depletes the system's memory until you are unable to allocate a thread stack (which is 1MB by default for a 32 bit application) and see the EAGAIN error. If you are not interested in doing a join and getting a return code, you can create the threads in a detached mode as follows:

(The example in our problem description is modified to set the attribute with detached mode by setting the attribute "detachstate" to PTHREAD_CREATE_DETACHED.)

/* Compile with:
* gcc pt_detached.c -lpthread -lposix4 -o pt_detached
* cc -mt pt_detached.c -lpthread -lposix4 -o pt_detached
* Crashes after 3399 threads with 'Resource temporarily unavailable'
*/

#ifndef _REENTRANT
#define _REENTRANT
#endif

#include  <thread.h >
#include  <stdio.h>
#include  <string.h>

#ifndef MAX_THREADS
#define MAX_THREADS 4000
#endif

static volatile int threadId = 0;

void* threadFunc(void* pParam) {
       pthread_exit((void *)pthread_self());
}

int main(int argc, char** argv) {
     pthread_t threads[MAX_THREADS];
     pthread_attr_t thread_attr;
     int res;
     uint_t threadCount;
     void *status;
     uint_t ctr;

/* code shows how to set and initialize the thread attribute */

     res = pthread_attr_init(&thread_attr);
     if (res != 0)
         perror ("pthread_attr_init");
     res = pthread_attr_setdetachstate(&thread_attr, PTHREAD_CREATE_DETACHED);
     if (res != 0)
         perror ("pthread_attr_setdetachstate");

     printf("Testing for thread resource problem\n");
     threadCount = 0;
     while ((threadCount < MAX_THREADS) &&
           ((res = pthread_create(&threads[threadCount], &thread_attr,
           &threadFunc, (void*)threadCount)) == 0)) {
                  threadCount++;
      }
      if (res != 0) {
          printf("Stopped with error %d: %s trying to create thread #%d\n", res, strerror(res), threadCount);
      } else {
          printf("Test completed successfully\n");
      }
}

testmachine% cc -mt -o pt_detached pt_detached.c -lpthread -lposix4

testmachine% ./pt.detached

"Testing for thread resource problem Stopped with error 11: Resource temporarily unavailable trying to create thread #3399"

Unfortunately, changing the attribute to detached state does not allow you to have more than 3399 threads either.

The way to increase the number of threads beyond 3399 is to link the application with an alternate One-level libthread library, which is provided by the Solaris 8 Operating Environment. This One-level model provides the mapping of user level threads to one-to-one LWPs.

To link with the alternate implementation, use the following run-path -R option when linking the application.

For POSIX threads use:

cc -mt . . . -lpthread . . . -R/usr/lib/lwp    (32 bit  Application)
 

For Solaris threads use:

cc -mt . . . -lthread . . . -R/usr/lib/lwp      (32 bit  Application)

Notice the difference between -lpthread and -lthread

For multithreaded applications that have been previously linked with the standard threads library, the environment variable LD_LIBRARY_PATH can be set as follows to bind the application at run time to the alternate threads library:

LD_LIBRARY_PATH=/usr/lib/lwp  (32 bit application with Bourne Shell)

Now, let's run our test application program, setting the environment variable LD_LIBRARY_PATH.

testmachine% setenv LD_LIBRARY_PATH /usr/lib/lwp (with C-Shell environment)
testmachine% ./pt   (run the application with default attributes set)

Testing for thread resource problem
Test completed successfully

testmachine% ./pt_detached (run the application with detached state attribute set)

Testing for thread resource problem
Test completed successfully

NOTE: This allows you to create more than 100,000 threads with both default attributes set and the detached state attribute set.

How This Works:

In the Two-level threading model the user links their application with -lpthread (or -lthread). In the One-level threading model the user links their application with -R/usr/lib/lwp (32 bit mode).

The Two-level model, which is standard for Solaris 8, has much different semantics when a thread exits than the One-level of an alternate libthread implementation.

When a thread exits it is examined to see if it is joinable or detached. If it is joinable, it is kept around until a pthread_join() is done. The amount of data freed immediately upon exit varies drastically between the Two-level model and the One-level model. Since our above example never does a pthread_join(), this causes data structures and stacks to be accumulated until the thread exits. The number of threads one can create is dependent on how much swap space is available, how much of its stack each thread touches and how much address space is available.

Two-level model

Here, the stacks are mmapped in slightly more than 8MB chunks but with MAP_NORESERVE. Therefore, the actual amount of swap consumed is dependent on the application's code. Using a test system, our basic program does 465 mmaps of 8421376 bytes. This works out to approximately 3.8 GB of potential swap reservations. The real problem here is that we're running out of address space in the application, since it's a 32 bit application and can have only 4GB of space.  On a system with a smaller amount of swap, you might run out of swap first. The net result is that you can only create approximately 3700 threads inside a simple 32-bit application using default attributes, without using pthread_join() to recover resources used by a thread which has exited .

Why doesn't the detached model work? It suffers from the same problems but in a different way. If a detached thread exits it is placed on a reaper queue. There is a reaper thread whose job it is take things off the queue and do the final resource cleanup. The problem is that the reaping never seems to occur. When we ran the aforementioned basic program, upon adding a pause a blizzard of munmaps occurred after the failed pthread_create(). A supposition is that we get into a mode where we are creating new threads so quickly that they never get to run until the primary creation thread pauses. Alternatively, there might be a bug in the threads library itself. We stop when we run out of address space.

One-level model

The One-level model takes a much more simplistic approach. When a detached thread exits, its resources are immediately given back to the system. There is no reaper thread and the stacks are immediately available for reuse.

The "default" case works because the number and size of mmapped anonymous memory is drastically reduced. Each mmap consumes 1040384 bytes and there are fewer mmaps. This is driven more by the fact that threads in the One-level model are able to run faster than in the Two-level model. Since our threads just exit, the amount of work done is very trivial. Immediately having your own LWP to run on simplifies things greatly compared to the Two-level model. We had no more than 130 or so threads active at any one time in our One-level model test. The net result is that the same program uses approximately 130MB of Virtual Address to create and run through 4000 threads, compared to almost 4GB of Virtual Address in the standard Two-level threads model. Additionally, the One-level model recycles stacks upon exit while the Two-level model seems to recycle only when reclaimed via a pthread_join() or when a detached thread is reaped.

Given the structure of the testcase in the preceding description, it seems likely that you could drive the number of threads astronomically high with the One-level model (we have done 100,000) without failure. This might lead you to a totally unreasonable conclusion about the number of threads you can have active at any one time. You might conclude that the number of threads created with default attributes and stack sizes that you can have active any one time inside a 32 bit application will be around 3500 (depending on how much Virtual Address the rest of the application uses) irrespective of the threading model. This is driven solely by address space considerations. In order to have more threads, you need to create the threads with smaller stacks. For the greatest number of threads, you need a system with a lots of swap and need to compile your code as a 64-bit application.

References: Solaris 8 Software Developer Collection - Multithreaded Programming Guide.