Hướng dẫn increase stack size python

Question

I have a python program that uses a custom-built DLL. This DLL crashes due to a stack overflow. This overflow is not due to a recursive function gone bad, but to large allocations on the stack using alloca().

Nội dung chính Show

What is a Thread Stack Size
Why Change Thread Stack Size
How To Get the Thread Stack Size
How To Set Thread Stack Size
Examples of Setting the Thread Stack Size
Set the Minimum Thread Stack Size
Increase the Thread Stack Size
Set Invalid Thread Stack Size
Further Reading

I want to increase stack size to get rid of this error. Is there any way to do this?

asked Jan 14, 2010 at 21:17

1

The python thread module allows you to specify a new stack size for new threads. Try setting that to a value you feel is large enough, and then doing the work of this DLL in a new thread.

answered Jan 14, 2010 at 21:36

As noted in some related questions like here, it's generally not a good idea to play with the stack size to extend the recursion depth, but here's code that shows how to grow the stack to that effect. With python 3.5, on a Windows 10 x64 system, it demonstrates a very deep recursion that's normally impossible (the normally allowed recursion limit in my situation appears to be 993). I don't know how big the stack actually has to be for this example, but, on my machine, with half of the size specified below, python crashes.

import sys
import threading

class SomeCallable:
    def __call__(self):
        try:
            self.recurse(99900)
        except RecursionError:
            print("Booh!")
        else:
            print("Hurray!")
    def recurse(self, n):
        if n > 0:
            self.recurse(n-1)

SomeCallable()() # recurse in current thread

# recurse in greedy thread
sys.setrecursionlimit(100000)
threading.stack_size(0x2000000)
t = threading.Thread(target=SomeCallable())
t.start()
t.join()

answered Nov 12, 2015 at 0:39

SteinStein

1,38720 silver badges27 bronze badges

The functions in a dll can have no control over the stack size available when they are executed (unless you spawn new threads under the control of your library).

If the dll is custom, then can't you allocate on the heap rather than stack (or statically allocate, if appropriate), and stop the problem that way?

answered Jan 14, 2010 at 21:28

JamesJames

24k13 gold badges81 silver badges128 bronze badges

AFAIK a program can only change the stack size of new threads or processes (Windows' CreateThread function, for example). As Python (and the Win32 API for Python) does not expose such functionality, you should rather replace the stack allocation with heap memory. Or is there a specific reason for using the stack?? If you really have to use alloca you might want to create a separate thread for execution of DLL code (which is overkill I think).

EDIT: Correction - Python does allow for setting the stack size when creating new threads (see thread.stack_size)

answered Jan 14, 2010 at 21:32

AndiDogAndiDog

66.7k20 gold badges157 silver badges201 bronze badges

1

Last Updated on September 12, 2022

You can retrieve and change the thread stack size via the threading.stack_size() function.

In this tutorial you will discover how to configure the thread stack size in Python.

Let’s get started.

What is a Thread Stack Size
Why Change Thread Stack Size
How To Get the Thread Stack Size
How To Set Thread Stack Size
Examples of Setting the Thread Stack Size
- Set the Minimum Thread Stack Size
- Increase the Thread Stack Size
- Set Invalid Thread Stack Size
Further Reading
Takeaways

What is a Thread Stack Size

Python code is compiled at runtime into Python bytecode and executed in the Python virtual machine.

Python source code is compiled into bytecode, the internal representation of a Python program in the CPython interpreter. […] This “intermediate language” is said to run on a virtual machine that executes the machine code corresponding to each bytecode.
— bytecode, Python Glossary.

The Python bytecodes are executed using a stack data structure that adds instructions as functions are called and pops instructions as they are executed, e.g. a so-called stack machine.

Each Python thread will have its own stack of instructions that are maintained and executed.

The size of the stack is fixed and is pre-allocated by the Python interpreter for each thread. This is so that the management and execution of instructions is fast.

The size of the stack may limit the number of instructions that can be maintained by a thread, such as the depth of a tree of function calls and the local variables maintained in those function calls. This is particularly an issue when a thread executes a single function recursively.

It is hard to nail down exact details of the Python thread stack and stack size details in the CPython interpreter, but a good place to start is here:
— Python Data model

Now that we know what the Python thread stack size is, let’s look at why we may change it.

Got slow loops? Run your loops in parallel (using all CPUs)
Learn how by downloading my FREE ebook: Parallel Loops in Python

Why Change Thread Stack Size

Python will use a default thread stack size that is specific to your underlying operating system.

Note that some platforms may have particular restrictions on values for the stack size, such as requiring a minimum stack size > 32 KiB …
— threading — Thread-based parallelism

We may want to change the Python thread stack size in our program.

For example, we may have a large number of very simple worker threads that execute a single function with few local variables. In that case, we may want to set the thread stack size to the minimum value.

More likely, we may have one or more worker threads executing complex function call graphs. This might involve many small functions with many local variables and perhaps some functions called recursively.

In that case, we may want to increase the thread stack size. This may result in better performance for the threads given the stack size is pre-allocated, and may avoid stability problems if the interpreter imposes a hard limit on the size of a thread stack.

A limitation of a small thread stack size is that a program that requires a large thread stack size may result in a memory error resulting running out of space on the thread stack, called a stack overflow.

Now that we have some ideas on why we might adjust the stack size, let’s look at how we might do this.

Confused by the threading module API?
Download my FREE PDF cheat sheet

How To Get the Thread Stack Size

The thread stack size can be retrieved using the threading.stack_size() function.

For example:

...

# get the stack size

size=threading.sstack_size()

Note, the threading.stack_size() function is a synonym or alias for the _thread.stack_size() function.

Calling the threading.stack_size() function will return the stack size used by the Python interpreter when creating new threads.

Return the thread stack size used when creating new threads.
— threading — Thread-based parallelism

The function takes an argument named “size” which is optional. If not specified, it will be set to 0, which will cause the function to return the default thread stack size.

If size is not specified, 0 is used.
— threading — Thread-based parallelism

The example below shows how to retrieve the default thread stack size.

# SuperFastPython.com

# example of getting the thread stack size

from threading import stack_size

# get the default stack size

size=stack_size()

print(size)

Running the example retrieves and reports the default thread stack size.

In this case, we can see that the value of zero is returned and reported, which indicates that the default for the operating system is being used.

Need help with Python Threading?

Sign-up to my FREE 7-day email course and discover how to use the Python threading module, including how to create and start new threads, how to use a mutex and semaphore, and much more!

Click the button below and enter your email address to sign-up and get the first lesson right now.

Start Your FREE Email Course Now!

How To Set Thread Stack Size

The thread stack size can be set via the threading.stack_size() function, the same function for getting the thread stack size.

The optional size argument specifies the stack size to be used for subsequently created threads, and must be 0 (use platform or configured default) or a positive integer value of at least 32,768 (32 KiB).
— threading — Thread-based parallelism

The function takes a single argument named “size” that specifies the new thread stack size in bytes.

For example:

...

# set the stack size

old_size=threading.stack_size(32768)

Note, the threading.stack_size() function is a synonym or alias for the _thread.stack_size() function.

When setting a new thread stack size, the function will return the old (current) thread stack size.

A subsequent call to threading.stack_size() with no argument will return the current value of the thread stack, reporting the new value.

The thread stack size must be specified before new threads are created for those threads to take on the new value.

The minimum value for the “size” argument is 32 kilobytes (kb) which is 32,768 bytes.

32 KiB is currently the minimum supported stack size value to guarantee sufficient stack space for the interpreter itself.
— threading — Thread-based parallelism

Most operating systems impose a constraint on the value of the “size” argument that it must be a multiple of 4,096 bytes.

If an unsupported size value is specified, a ValueError may be raised.

If the specified stack size is invalid, a ValueError is raised and the stack size is unmodified. […] using multiples of 4096 for the stack size is the suggested approach in the absence of more specific information
— threading — Thread-based parallelism

Some operating systems do not permit the thread stack size to be modified, in which case a RuntimeError may be raised.

If changing the thread stack size is unsupported, a RuntimeError is raised.
— threading — Thread-based parallelism

Now that we know how to set the thread stack size, let’s look at some worked examples.

Examples of Setting the Thread Stack Size

In this section we will look at some examples of setting the thread stack size.

Set the Minimum Thread Stack Size

The minimum stack size is 32 kilobytes (kb) which is 32,768 bytes.

We can set the minimum thread stack size as follows:

...

# set the minimum thread stack size

stack_size(32768)

The example below demonstrates how to set the minimum thread stack size.

Once set, the current thread stack size is retrieved and reported, confirming that the change took effect.

# SuperFastPython.com

# example of setting the minimum thread stack size

from threading import stack_size

# set the minimum thread stack size

stack_size(32768)

# report the changed stack size

size= stack_size()

print(size)

Running the example first sets the stack size to 32 kilobytes.

The stack size is then retrieved, confirming the new value took effect.

Increase the Thread Stack Size

The thread stack size can be increased by specifying a value that is a multiple of 4,096 bytes and larger than 32 kilobytes (kb) which is 32,768 bytes.

Common values might include

4,096 * 8 = 32,768 bytes
4,096 * 16 = 65,536 bytes
4,096 * 32 = 131,072 bytes
4,096 * 64 = 262,144 bytes
4,096 * 128 = 524,288 bytes

For example:

...

# increase the stack size

stack_size(4096*16)

Choosing a suitable value for your specific application might require some trial and error with careful benchmarking to confirm the change had the desired improvement in performance and/or stability.

The example below shows how to increase the thread stack size to 64 kilobytes, or 65,536 bytes.

# SuperFastPython.com

# example of increasing the thread stack size

from threading import stack_size

# increase the stack size

stack_size(4096*16)

# report the changed stack size

size =stack_size()

print(size)

Running the example first sets the increased thread size to 64 kilobytes.

The stack size is then retrieved, confirming the new value took effect.

Set Invalid Thread Stack Size

Changing the thread stack size to an unsupported value will result in a ValueError.

Valid thread stack size values are multiples of 4,096 bytes and above 32,768 bytes.

For example, changing the thread stack size to 100,000 bytes will result in a ValueError on most platforms.

The example below demonstrates the effect of setting an invalid thread stack size.

# SuperFastPython.com

# example of setting an invalid thread stack size

from threading import stack_size

# set an invalid stack size

stack_size(100000)# will raise ValueError

Running the example results in a ValueError.

This is expected, given that the specified thread stack size of 100,000 is not a multiple of 4,096 bytes.

Traceback (most recent call last):

...

stack_size(100000) # will raise ValueError

ValueError: size not valid: 100000 bytes

Takeaways

You now know how to retrieve and configure the thread stack size in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Loïc Lassence on Unsplash

programming python Python stack size Python with resource Python resource management

Hướng dẫn increase stack size python

What is a Thread Stack Size

Why Change Thread Stack Size

How To Get the Thread Stack Size

How To Set Thread Stack Size

Examples of Setting the Thread Stack Size

Set the Minimum Thread Stack Size

Increase the Thread Stack Size

Set Invalid Thread Stack Size

Further Reading

Takeaways

Bài Viết Liên Quan

Quảng Cáo

Có thể bạn quan tâm

Toplist được quan tâm

Quảng cáo

Xem Nhiều

Quảng cáo

Chúng tôi

Điều khoản

Trợ giúp

Mạng xã hội