Bash redirections cheat sheet

dllthomas · on Sept 10, 2012

One thing worth noting is that "redirection of streams" is a poor metaphor.

If I have two streams,

  1 ---------- 1'

  2 ---------- 2'

and I redirect the 1>2

  1 ----
        \
  2 ---------- 2'

it seems to work okay.

But if you have three,

  1 ---------- 1'

  2 ---------- 2'

  3 ---------- 3'

and try "redirecting 1 to 2 then 2 to 3", the natural reading would be

  1 -
     \
  2 -----
         \
  3 ---------- 3'

But that's not what you get, because you're not "redirecting streams" - you're copying file descriptors. This is partially fixed by informing people to do the operations backwards, but there are corner cases where that breaks down, and "copying file descriptors in the order they appear on the line" never does because that's genuinely what's going on.

  1 -> 1'
  2 -> 2'
  3 -> 3'

foo 1>2 2>3

This is "run foo copying fd 2 to 1 and 3 to 2"

which gives

  1 -> 2'
  2 -> 3'
  3 -> 3'

wethesheeple · on Sept 11, 2012

Is there a typo: s/genuinely/not geniuinely

I guess what you're explaining is the process that happens as the kernel reserves fd's? And when you start your program there isn't necessarily any guarantee that a particular fd will be available, except for 0,1,2, right?

dllthomas · on Sept 11, 2012

Copying file descriptors is genuinely what's going on, so understanding it that way always works.

Edited for clarity:

The way file descriptors work is that the kernel maintains, for each process, a table mapping from numerical file descriptors to structures describing the open file/socket/device associated with that descriptor.

When the shell is setting up to spawn a new process (or subshell, for that matter), it processes the redirections in order, left to right. For redirections of the form "A>&B" this is a call to dup2(B, A), which has the kernel copy the entry in the table at B over the entry at A.

Incidentally, you will note the direction of the symbol doesn't matter when cloning file descriptors: 5>&7 and 5<&7 mean the same thing - you're just copying a file descriptor. It does matter when opening a file, as the file will be opened for reading or writing before before being dup2'ed over the requested descriptor. Having said that, of course use the correct symbol - it's good documentation.

wethesheeple · on Sept 11, 2012

Thanks for the this. In my opinion, there's too little written about passing fd's (use of dup2). It seems like a technique that is more useful than just as part of shell redirection. Safer than popen()?

dllthomas · on Sept 11, 2012

I'm pretty sure popen is just a wrapper around this stuff. It's basically,

pipe: get two fds connected by a pipe

fork: create a child process

dup2: move the read fd of the pipe to 0 (stdin) in the new process

exec: run the program in the child process

There's some cleanup but that's the gist - popen isn't a syscall (or reasonably close to one), so must necessarily rely on other stuff to get its work done where it actually interfaces with the kernel.

wethesheeple · on Sept 11, 2012

That makes perfect sense. It's the fork+exec that makes it dangerous (e.g. in the CGI context). This is something I've been wondering about and you have just provided a jolt of clarity. Many thanks, again.

dllthomas · on Sept 11, 2012

Happy to help :)

gwillen · on Sept 10, 2012

I think I previously had all the pieces required to do this, but only just figured out that it worked:

  Glenn-Willens-MacBook-Pro:~ gwillen$ exec 3<> /dev/tcp/<website>/80
  Glenn-Willens-MacBook-Pro:~ gwillen$ echo "GET / HTTP/1.0" >&3
  Glenn-Willens-MacBook-Pro:~ gwillen$ echo "" >&3
  Glenn-Willens-MacBook-Pro:~ gwillen$ cat <& 3
  HTTP/1.1 200 OK
  Date: Mon, 10 Sep 2012 21:43:33 GMT
  Server: Apache
  Last-Modified: Sun, 06 Sep 2009 18:44:42 GMT
  ETag: "30525-0-472ed20537a80"
  Accept-Ranges: bytes
  Content-Length: 0
  Connection: close
  Content-Type: text/html

  Glenn-Willens-MacBook-Pro:~ gwillen$

dllthomas · on Sept 11, 2012

One thing missing is <<- heredocs, which strip (some) leading whitespace. It lets you keep up your indentation in a shell script without passing a bunch of whitespace to the commands.

shabble · on Sept 10, 2012

I've found

    cmd <(another_cmd)

to be particularly useful in the past for pushing generated data into something that doesn't accept - as an alias for STDIN. I'm not sure if it's a different way of expressing one of the listed items though; iirc it creates a temporary fifo

daragh · on Sept 10, 2012

That FIFO temporary file is the file descriptor of the subprocess' stdout. You're accessing it through a pipe.

It's most useful when you want to diff the result of multiple commands, e.g.: diffing the contents of two directories:

  $ diff <(find /first/path | sort) <(find /second/path | sort)

akldfgj · on Sept 11, 2012

Aha!

In zsh "=(cmd)" does that, but "=" doesn't work in bash. "<(cmd)" works everywhere... why did zsh trick me with "="?

ralph · on Sept 11, 2012

zsh had the idea first IIRC.

bajsejohannes · on Sept 10, 2012

I've always done

   (cmd 2>&1) > file

to redirect stderr and stdout to a file. I always thought the parenthesis were necessary, but according to this I can just do

   cmd > file 2>&1

It seems very unintuitive, though. Does anyone know the rationale behind ordering of redirects?

(That said, I'm going to use 'cmd &> file' that I just learned from now on)

rcsorensen · on Sept 10, 2012

cmd is going to output to STDOUT and STDERR, which are file descriptors 1 and 2. In a basic shell command execution, those route out to the console.

Reading from left to right:

cmd

Run command cmd. File descriptors: {1: /dev/console, 2:/dev/console}

  > file

Open a file descriptor for "file" and copy it into our first file descriptor (stdout). File descriptors: {1: fopen('file'), 2: /dev/console }

  2>&1

Copy the file descriptor stored in 1 (stdout) into 2 (stderr). File descriptors: {1: fopen('file'), 2: fopen('file')}

dredmorbius · on Sept 10, 2012

In the first case, you're actually executing two shells.

    (cmd 2>&1)

runs in its own subshell, and copies stderr to stdout.

    >file

is interpreted in the parent shell, and copies stdout to 'file'.

In the second instance, you run a single shell, first copying stdout to 'file', then copying stderr to the (redirected) stdout, that is, 'file'.

anon987 · on Sept 10, 2012

> Does anyone know the rationale behind ordering of redirects?

I can't give you an exact rationale, but my old shell scripting books from the 90's use the cmd > file 2>&1 notation. My personal belief is that it was for situations when you would do something like this:

cmd > output.txt 2> errors.txt

ralph · on Sept 11, 2012

It's read as "cmd, stdout to file, stderr to the same place as stdout".

dredmorbius · on Sept 10, 2012

The network pseudo filehandles (/dev/tcp/<host>/<port> and /dev/udp/<host>/<port>) are explicitly disabled in certain distros (notably Debian). Ubuntu appears to support it.

Though not limited to just shell redirections, one of the true masters of bash (and zsh) is Larry Peek whose written/given multiple books, articles, and presentations on the topic.

dfc · on Sept 10, 2012

Do you know why debian did that? Any links to debian-devel threads?

dredmorbius · on Sept 11, 2012

http://lists.debian.org/debian-user/2006/07/msg00204.html

The feature is confusing (it uses virtual files visibly only to bash, and not extant on the filesystem) and overlaps existing functionality (netcat).

Also:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464

http://lists.debian.org/debian-user/2003/04/msg01591.html

http://lists.debian.org/debian-devel/2006/11/msg00794.htmlu

http://lists.debian.org/debian-user/2006/07/msg00204.html

luriel · on Sept 11, 2012

I can never remember the (ba)sh redirection syntax, and I used it for decades.

Rc on the other hand is much more clean: http://rc.cat-v.org

AceJohnny2 · on Sept 10, 2012

the "cmd <<EOL ..\n..\n EOL" part has been mystical to me for ages, especially as I didn't know what to google to find out. I recently learned it was called a "Here Document". http://en.wikipedia.org/wiki/Here_document

For extra credit, try using a "Here Document" as input for 'echo' and 'cat'. Explain the results.

akldfgj · on Sept 12, 2012

`man bash` , search for "<<" see the "Here Strings" section.

You can also search https://www.google.com/search?q=bash and follow links to "Documentation"

    xargs echo <<EOL  
    > line 1
    > line 2  
    > EOL
    line 1 line 2

    $ cat <<EOL
    > line 1
    > line 2 
    > EOL
    line 1
    line 2

line-terminators get eaten by xargs

allstruck · on Sept 10, 2012

What is so mystifying about this?

Doing this:

"cat << EOL

line one

line two

EOL"

Would be the same as this:

"cat < file"

That is if 'file' contained this:

"line one

line two"

This doesn't work with the echo command because it does not use stdin like cat does.

darkstalker · on Sept 10, 2012

The PNG version is rendered with subpixel antialiasing, which is kinda annoying to see. Besides that, awesome job.

jdpage · on Sept 11, 2012

Is there a particular reason you'd go for the PNG version over the PDF version? It's a more suitable format for these "cheat sheet" type pages, since it'll use the platform text rendering, and also keep the vector information (meaning it'll print better if you want to hang it up next to your desk).

zwilliamson · on Sept 11, 2012

Where is EOF? I use that a lot in my Bash scripts. I think it should be on this cheat sheet.

dllthomas · on Sept 11, 2012

I like the mention of PIPESTATUS - $PIPESTATUS[*] is at the start of my PS1

michaelhoffman · on Sept 10, 2012

Thanks! Much more useful than the other recent alphabetic cheat sheets.

trusko · on Sept 10, 2012

Great! Thanks for sharing, very helpful.

methoddk · on Sept 10, 2012

Very useful! Thank you for this.