bash - Why does the output of some Linux programs go to neither STDOUT nor STDERR?

23
2014-04
  • Will Sheppard

    Why does the output of some Linux programs go to neither STDOUT nor STDERR?

    Actually, I want to know how to reliably capture all program output, no matter what 'stream' it uses. The problem I have is that some programs do not seem to let their output be captured.

    An example is the 'time' command:

    time sleep 1 2>&1 > /dev/null
    
    real        0m1.003s
    user        0m0.000s
    sys         0m0.000s
    

    or

    time sleep 1 &> /dev/null
    
    real        0m1.003s
    user        0m0.000s
    sys         0m0.000s
    

    Why do I see output both times? I expected it all to be piped into /dev/null.

    What output stream is time using, and how can I pipe it into a file?

    One way to work around the problem is to create a Bash script, for example, combine.sh containing this command:

    $@ 2>&1
    

    Then the output of 'time' can be captured in the correct way:

    combine.sh time sleep 1 &> /dev/null
    

    (no output is seen - correct)

    Is there a way to achieve what I want without using a separate combine script?

  • Answers
  • terdon

    This question is addressed in BashFAQ/032. In your example, you would:

    { time sleep 1; } 2> /dev/null
    

    The reason why

    time sleep 1 2>/dev/null
    

    doesn't behave how you're expecting is because with that syntax, you'll want to time the command sleep 1 2>/dev/null (yes, the command sleep 1 with stderr redirected to /dev/null). The builtin time works that way so as to make this actually possible.

    The bash builtin can actually do this because... well, it's a builtin. Such a behavior would be impossible with the external command time usually located in /usr/bin. Indeed:

    $ /usr/bin/time sleep 1 2>/dev/null
    $
    

    Now, the answer to your question

    Why does the output of some linux programs go to neither STDOUT nor STDERR?

    is: it does, the output goes to stdout or stderr.

    Hope this helps!

  • Alok

    Your particular question about time builtin has been answered, but there are some commands that don't write either to stdout or to stderr. A classic example is the Unix command crypt. crypt with no arguments encrypts standard input stdin and writes it to standard output stdout. It prompts the user for a password using getpass(), which by defaults outputs a prompt to /dev/tty. /dev/tty is the current terminal device. Writing to /dev/tty has the effect of writing to the current terminal (if there is one, see isatty()).

    The reason crypt can't write to stdout is because it writes encrypted output to stdout. Also, it's better to prompt to /dev/tty instead of writing to stderr so that if a user redirects stdout and stderr, the prompt is still seen. (For the same reason, crypt can't read the password from stdin, since it's being used to read the data to encrypt.)

  • Uwe Plonus

    The problem in your case is that the redirection works in another way. You wrote

    time sleep 1 2>&1 > /dev/null
    

    This redirects the standard output to /dev/null and then redirects the standard error to standard output.

    To redirect all output you have to write

    time sleep 1 > /dev/null 2>&1 
    

    Then the standard error will be redirected to the standard output and after that all the standard output (containing the standard error) will be redirected to /dev/null.


  • Related Question

    bash - Immediately tell which output was sent to stderr
  • Clinton Blackmore

    When automating a task, it is sensible to test it first manually. It would be helpful, though, if any data going to stderr was immediately recognizeable as such, and distinguishable from the data going to stdout, and to have all the output together so it is obvious what the sequence of events is.

    One last touch that would be nice is if, at program exit, it printed its return code.

    All of these things would aid in automating. Yes, I can echo the return code when a program finishes, and yes, I can redirect stdout and stderr; what I'd really like it some shell, script, or easy-to-use redirector that shows stdout in black, shows stderr interleaved with it in red, and prints the exit code at the end.

    Is there such a beast? [If it matters, I'm using Bash 3.2 on Mac OS X].


    Update: Sorry it has been months since I've looked at this. I've come up with a simple test script:

    #!/usr/bin/env python
    import sys
    
    print "this is stdout"
    print >> sys.stderr, "this is stderr"
    print "this is stdout again"
    

    In my testing (and probably due to the way things are buffered), rse and hilite display everything from stdout and then everything from stderr. The fifo method gets the order right but appears to colourize everything following the stderr line. ind complained about my stdin and stderr lines, and then put the output from stderr last.

    Most of these solutions are workable, as it is not atypical for only the last output to go to stderr, but still, it'd be nice to have something that worked slightly better.


  • Related Answers
  • sickill

    You can also check out stderred: https://github.com/sickill/stderred

  • jtbandes

    I just devised a crazy method involving FIFOs.

    $ mkfifo foo
    $ grep --color . foo &
    $ your_command 2>foo
    

    If you'd like the stderr output separate, you can open up two separate shells and run "grep --color . foo" in one without the &, then run the command in the other (still with the 2>foo). You'll get the stderr in the grep one and the stdout in the main one.

    This works because the stderr output is routed through the FIFO into grep --color, whose default color is red (at least it is for me). When you're done, just rm the FIFO (rm foo).

    Caveat: I'm really not sure how this will handle output order, you'll have to test it out.

  • nagul

    Yes, this is possible. Look at the section "Make STDERR red" on this site for a working example.

    The basic code is this

    # Red STDERR
    # rse <command string>
    function rse()
    {
        # We need to wrap each phrase of the command in quotes to preserve arguments that contain whitespace
        # Execute the command, swap STDOUT and STDERR, colour STDOUT, swap back
        ((eval $(for phrase in "$@"; do echo -n "'$phrase' "; done)) 3>&1 1>&2 2>&3 | sed -e "s/^\(.*\)$/$(echo -en \\033)[31;1m\1$(echo -en \\033)[0m/") 3>&1 1>&2 2>&3
    }
    

    A brief explanation is given in the function itself. What is does is move STDOUT and STDERR about, so sed gets STDERR in 1, colors it, and then swaps it back. Think of file stream 3 as a temp variable here.

    The call is pretty simple

    rse commands
    

    However, certain invocations will not work as expected; the caveats are all provided on the linked page.

    Btw, I think it is also possible to come with a solution of the form

    commands | rse
    

    where rse will colorize the output.

    I also found this hilite project that seems to do this. I haven't tried it out, but it might be what you're looking for.

    hilite is a tiny utility which executes the command you specify, highlighting anything printed to stderr. It is designed mainly for use with builds, to make warnings and errors stick out like a sore cliche.

    Other related projects:

  • Thomas

    Another program is ind:

    http :// www.habets.pp.se/synscan/programs.php?prog=ind (you have to assemble the hyperlink yourself, I don't have enough points for more than one per answer). There is a screenshot and even a screencast there.

    It runs the subprocess in a pty, which the others probably don't. This can matter where order matters (it often does), since stderr is immediately flushed in terminals and stdout is fully buffered when it's not a tty.

    For a full explanation, see this: http://blog.habets.pp.se/2008/06/Buffering-in-pipes

    Also, ind works with interactive programs and control characters. Bash works just like normal when started with "ind bash -i".

    This could work to give you colors while preserving Ctrl-P et.al.

    ind -P $(echo -ne '\033[31;1m') -p $(echo -ne '\033[0m') bash -i
    
  • srnka

    Here is a lot of responses for highlighting the stderr output yet. I can only add one pretty easy - in one row, which you attach to the command:

    command 2> >(while read line; do echo -e "\e[01;31m$line\e[0m"; done)
    

    But you have to add this after command each time.

    I, personally, like the possibility mentioned by @nagul , the rse function added to bashrc.

    But I want to add the solution for printing the exit code. You can add this value to the beginning of your bash prompt row:

    hostname$ command
      Some error occurred. Returned exit code.
    EC hostname$
    

    Where EC is exit code of the command.

    I have it set in the way, that when the exit code is 0, it will be not printed, but any other value is printed before next prompt in red color.

    The whole trick is done in ~/.bashrc:

    my_prompt() {
     EXITSTATUS="$?"
     RED="\[\033[1;31m\]"
     OFF="\[\033[m\]"
    
    PROMPT="${debian_chroot:+($debian_chroot)}\h \$ "
    
    if [ "${EXITSTATUS}" -eq 0 ]; then
       PS1="${PROMPT}"
    else
       PS1="${RED}$EXITSTATUS${OFF} ${PROMPT}"
    fi
    }
    
    PROMPT_COMMAND=my_prompt
    

    The prompt line is by default defined by PS1 variable. Copy whatever you have here into the variable PROMPT and then create the PS1 variable with or without exit code.

    Bash will show the information in PS1 variable in the prompt.

  • mykhal

    there's annotate-output utility (devscripts Debian package), if you would like to do it without colors

    $ annotate-output /tmp/test.py    
    14:24:57 I: Started /tmp/test.py
    14:24:57 E: this is stderr
    14:24:57 O: this is stdout
    14:24:57 O: this is stdout again
    14:24:57 I: Finished with exitcode 0