Thiết bị xuất chuẩn chuyển hướng python và thiết bị xuất chuẩn

Được tạo vào 2019-03-05 11. 17 bởi andriusl, thay đổi lần cuối 2022-04-11 14. 59 bởi quản trị viên. vấn đề giờ đã kết thúc

msg337178 - (view)Tác giả. Andrius Laukavičius (andriusl)Ngày. 2019-03-05 11. 17
It looks like logging library uses always first assigned stderr object and won't change it even if it was reset.

capture_output function redirects stdout and stderr to io.StringIO object, saves what was captured in string and returns it. And then context manager resets back stdout and stderr where it should be originally. Though it seems logging library ignores that.

Here is the code I encountered issue with:

import io
import sys
import contextlib
from typing import Optional
import logging

def capture_output(
    target: object,
    args: Optional[tuple] = None,
        kwargs: Optional[dict] = None) -> str:
    """Redirect stdout and stderr into string buffer and capture it.

    target object is executed with optional args, kwargs and all stdout/
    stderr that is captured, returned in string form.

        target: object to execute, usually a function.
        args: target positional arguments (default: {None})
        kwargs: target keyword arguments (default: {None})
    if not args:
        args = ()
    if not kwargs:
        kwargs = {}
    with io.StringIO() as sio:
        with contextlib.redirect_stdout(sio), contextlib.redirect_stderr(sio):
            target(*args, **kwargs)
            output = sio.getvalue()


def dummy():
    print('dummy test')
    logging.warning('dummy test')

def dummy2():
    print('dummy2 test')

capture_output(dummy)  # works
capture_output(dummy)  # won't work anymore.

capture_output(dummy2)  # works
capture_output(dummy2)  # works

Here is what I get running this code:

dummy test
WARNING:root:dummy test

dummy test
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.5/logging/", line 982, in emit
ValueError: I/O operation on closed file
Call stack:
  File "", line 43, in 
    capture_output(dummy)  # won't work anymore.
  File "", line 28, in capture_output
    target(*args, **kwargs)
  File "", line 36, in dummy
    logging.warning('dummy test')
Message: 'dummy test'
Arguments: ()

dummy2 test

dummy2 test

P.S. here is original question I asked on stack overflow: I got suggestion to report it as a bug.
msg337183 - (view)Tác giả. Karthikeyan Singaravelan (xtreak) *
Thiết bị xuất chuẩn chuyển hướng python và thiết bị xuất chuẩn
Ngày. 2019-03-05 11. 57
A little simplified reproducer :  

import io, contextlib, logging

message = 'dummy test'

with io.StringIO() as sio:
    with contextlib.redirect_stdout(sio), contextlib.redirect_stderr(sio):
        output = sio.getvalue()


$ python3.7 /tmp/
WARNING:root:dummy test

--- Logging error ---
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/logging/", line 1036, in emit
ValueError: I/O operation on closed file
Call stack:
  File "/tmp/", line 11, in 
Message: 'dummy test'
Arguments: ()
msg337193 - (view)Tác giả. Peter Otten (Peter. otten) *Ngày. 2019-03-05 14. 37
I see various options to address this.

(1) Change basicConfig() to use __stderr__ instead of stderr to avoid redirections from within the script.

(2) Change your script to call basicConfig() explicitly, before the temporary redirection takes place. 

In both cases logging output will always go to the "real" stderr.

(3) Write a special handler that fetches sys.stderr lazily. Here's a sketch:

class StderrHandler(logging.StreamHandler):
    def __init__(self):

    def get_stream(self):
        return sys.stderr

    def set_stream(self, _value):

    stream = property(get_stream, set_stream)


As written this is likely to fail with multiple threads...

My personal favourite is option (2).
msg337210 - (view)Tác giả. Andrius Laukavičius (andriusl)Ngày. 2019-03-05 15. 36
Peter Otten thanks for examples,
though with capture_output function, I had a little bit different intentions.

At first I actually did similarly as you described in case 2, where I called BasicConfig explicitly. Then I could specify where to redirect stream and what to do after I was done with capturing stderr (just reset back where it would be on default).

Though I actually wanted to just capture stdout and stderr indirectly, meaning I don't want to specifically initiate logging object or modify it.

I mean, I want to just capture any stdout/stderr that would be outputed while running some function that was passed via `target` argument (in my capture_output function). In this case, some function that I capture output from, might have some different way to handle sys.stderr (e.g. different basicConfig or some other implementation where sys.stderr is handled?).

So it is not possible to consistently manage stderr when it involves logging library without explicitly "manage" it?

P.S. don't know if I'm clear enough, so please ask me to clarify anything if its not clear..:)
msg337213 - (view)Tác giả. Karthikeyan Singaravelan (xtreak) *
Thiết bị xuất chuẩn chuyển hướng python và thiết bị xuất chuẩn
Ngày. 2019-03-05 15. 50
I find issue6333 slightly relatable to this issue where it talks about external modules possibly closing sys.stdout/stderr.
msg337284 - (view)Tác giả. Peter Otten (Peter. otten) *Ngày. 2019-03-06 09. 18
> So it is not possible to consistently manage stderr when it involves > logging library without explicitly "manage" it?

That's what I think, at least from within a script. To me redirect_xxx() always looked like a workaround to have old code write to a destination that was unforeseen in the initial design. I prefer



with redirect_stdout(dest):

That said -- do you have a suggestion how the logging library could be adapted to work smoothly with redirect_xxx()?

(A compelling use case would also be nice, but note that I'm not the one who makes the decisions.)
msg337316 - (view)Tác giả. Andrius Laukavičius (andriusl)Ngày. 2019-03-06 15. 34
well I don't have deep experience with logging lib to know all ins and outs. So I might not be correct person to define what should be changed specifically. But I think it should be consistent on all standard libraries how it handles stdout/stderr, unless there is a reason to do it differently?

For example `print` to me seems to behave how I expect, because if stdout is redirected elsewhere, print uses redirected stdout without a problem. 

Now with logging, it does feel strange that on first call of `logging`, it sets internally stderr where it is pointing at that moment and forgets about it. You call logging again, and it naively expects stderr to be where it was on initial call, when in reality it might be redirected elsewhere (as in my case).

Maybe you know the reasoning why logging even needs to behave this way? Though if logging behaves this way, why print behaves differently?

So to summarize this, I think all libraries should respect stdout/stderr where it is and have some unified way/interface (like I expected redirect_stdout/redirect_stderr to be) to specify that stdout/stderr has changed and related libraries must update their state regarding that (logging library case) or simply rely on where stdout/stderr is pointed, so nothing needs to be done on that library once stdout/stderr is redirected.

To comment on your view about `dump_stuff(dest)`. Well I think this suits well, when you have something to dump already, but what if something to dump is what is produced via stdout/stderr and you need to capture it first? How can I do that without first redirecting stdout where I need to?
msg337361 - (view)Tác giả. Vinay Sajip (vinay. sajip) *
Thiết bị xuất chuẩn chuyển hướng python và thiết bị xuất chuẩn
Ngày. 2019-03-07 06. 13
The StreamHandler allows you to specify *exactly* which stream you want to use to log to; sys.stderr is provided as a convenient default argument because that's what a lot of people want a lot of the time.

This is typically done at logging configuration time, or whenever a StreamHandler is created. This is done implicitly by your logging.warning() call (as documented, this calls logging.basicConfig(), which adds a StreamHandler using whatever sys.stderr is set to at the time the StreamHandler is instantiated). Also documented is that basicConfig() is only effective once (it will not do anything if a handler has already been added to the root logger - it is only meant to be use for simple one-off scripts). The documentation for basicConfig() is clear:

"Does basic configuration for the logging system by creating a StreamHandler with a default Formatter and adding it to the root logger. The functions debug(), info(), warning(), error() and critical() will call basicConfig() automatically if no handlers are defined for the root logger.

This function does nothing if the root logger already has handlers configured for it."

If you want to use the real console streams, don't use logging.warning(), but instead explicitly call basicConfig() using __stderr__, as Peter says. Alternatively, use the approach suggested in the cookbook for context-sensitive logging:

Closing, as this is not a bug in logging.
msg337369 - (view)Tác giả. Andrius Laukavičius (andriusl)Ngày. 2019-03-07 08. 04______8

Bạn có thể chuyển hướng thiết bị xuất chuẩn và thiết bị xuất chuẩn sang cùng một tệp không?

Chúng tôi có thể kết hợp thực thi cho thiết bị xuất chuẩn và thiết bị xuất chuẩn trong một lệnh chuyển hướng duy nhất . Trong ví dụ dưới đây, các thông báo lỗi sẽ được lưu trong “error. txt,” trong đó “đầu ra. txt” sẽ có đầu ra tiêu chuẩn là “lệnh ls. ”

stdin stdout và stderr trong python là gì?

stdin , stdout và stderr là các đối tượng tệp được xác định trước tương ứng với các luồng đầu vào, đầu ra và lỗi tiêu chuẩn của Python . Bạn có thể liên kết lại thiết bị xuất chuẩn và thiết bị xuất chuẩn với các đối tượng giống như tệp (các đối tượng cung cấp phương thức ghi chấp nhận đối số chuỗi) để chuyển hướng đích của thông báo lỗi và đầu ra.

Bản in python có chuyển sang thiết bị xuất chuẩn không?

Hàm print() của Python nhận dữ liệu python chẳng hạn như số nguyên và chuỗi, đồng thời in các giá trị đó ra tiêu chuẩn . Để nói rằng tiêu chuẩn ra là "văn bản" ở đây có nghĩa là một loạt các dòng, trong đó mỗi dòng là một chuỗi ký tự với ký tự dòng mới '\n' đánh dấu cuối mỗi dòng.