Hardware/kernel : pipe implementation?

Contrary to some other operating systems, a pipe is not a disk file but an in-memory fifo.

So, the command:

program1 | program2

Is NOT internally handled as:

program1 >tempfile program2

There is a comp.unix.internals group, but I can't remember the last time I saw a post there.

Since you asked about "*nix", not just Linux, the answer is that there is a lot of variation in how pipes are implemented. On some systems they are bidirectional, on some unidirectional. Some have a fixed capacity, others have "high water" and "low water" levels (where the pipe stops accepting data when high water is reached but won't start accepting more data until enough has been read to reach low water). Some are simple, some are hugely complex - notably SVR4 and its descendents (UnixWare, Solaris, IRIX, etc.) implement pipes on top of STREAMS, which means you can use them for file descriptor passing (using ioctl with I_SENDFD and I_RECVFD).

--
Geoff Clare

$>cat test_pipe.c #include int main (int argc , char **argv) { printf("PIPE_BUF = %ld\n", pathconf("/", _PC_PIPE_BUF)); }

$>gcc test_pipe.c -o pipe

$>uname -a AIX abc 3 5 00014FDAD400 $>./pipe PIPE_BUF = 32768

$> uname -a Linux def 2.6.32-26-generic #48-Ubuntu SMP Wed Nov 24 09:00:03 UTC 2010 i686 GNU/Linux $> ./pipe PIPE_BUF = 4096

So, yes, 4 k for linux (at least on x86), but not for all other unices

Ok, but 32K is "small" as well. What I mean is that there is no buffer that will hold all the output of program1 until program2 wants to read it. The buffer will be filled and then the first program will be blocked until the second one reads some input.

In fact, in a Unix/Linux system the first program can produce infinite output and the second one can consume what it needs and then close its input (e.g. because it exits). At that time, the first program will receive a SIGPIPE signal, which will usually not be caught so it will terminate the program.

On "some other" systems (not to be named), that scenario will fail because the first program will be run until it fills the disk entirely.

The code shows the size of a named pipe (aka fifo) but I think the size of the buffer is the same as for anonymous pipes.

Yes

No. As you said above, only until the buffer gets full.

Yes, unless it has opened the pipe as read/write. For pipelines - not likely - but for named pipes - very likely

Well, that was on win95-ish systems, I think. The NT based ones has more of a unix approach to pipes. (they all live in the \\.\pipe\ directory, but I think they are memory mapped as well.

What I mean is that the first program can be an infinite loop that keeps producing output, and the second program can read as much of it as it needs and then exit, which will terminate the first program.

For example:

tr -dc 'A-Za-z0-9' > On "some other" systems (not to be named), that scenario will fail

There are other "wannabe Unix" systems (compatability shells that live on top of another OS) that do the same thing.

Ok.

Ok.

PIPE_BUF is not the capacity of the pipe, it is the largest amount that can be written atomically. You can find the "capacity" (or high water level for some systems) by writing to a non-blocking pipe until the write fails. Running the program below on a Linux system I get:

PIPE_BUF = 4096 capacity = 65536

On Solaris I get:

PIPE_BUF = 5120 capacity = 16384

#include #include #include #include

int main(void) { int flags, pfd[2]; ssize_t capacity;

if (pipe(pfd) == -1) { perror("pipe"); exit(1); }

flags = fcntl(pfd[1], F_GETFL); if (fcntl(pfd[1], F_SETFL, flags|O_NONBLOCK) == -1) { perror("fcntl"); exit(1); }

capacity = 0; while (write(pfd[1], "x", 1) == 1) ++capacity;

printf("PIPE_BUF = %ld\n", fpathconf(pfd[1], _PC_PIPE_BUF)); printf("capacity = %ld\n", (long)capacity);

exit(0); }

--
Geoff Clare

Oh, yes, you are right. I wrote that program 2 years ago, to see if named pipes could replace our IPC solution/kludge of semaphores/shared mem on Aix, and at the same times use the same mechanism on Linux and Windows.

But I forgot that I needed the write to be atomic. Hence my misinformation here... Sorry about that.

--
--

I remember the days when there was no IPC at all on many versions of Unix, except for pipes. And pipes where only possible between processes of the same ancestor. We used a shared diskfile and ordinary signals to communicate between applications and a database server...

perhaps comp.os.linux.misc, comp.os.linux.development.system or alt.os.linux

not really.

pipes have a small capacity (eg: 4Kib) if the reader stops reading the pipe the writer will be blocked from proceeding once the pipe is full and be forced to wait until room is made in the pipe.

if the writer tries to write more than the pipe's capacity in one go the system will split it up so that it fits

unlike files a pipe(7)'s file-handle can't be rewind(2)ed or seek(2) to different offset.

Due to the small size swapping a a pipe buffer out would have little effect.

further reading: run this:

man 7 pipe

--
umop apisdn

it's not _that_ rare. Windows NT does this too.

--
umop apisdn

did you have mmap() on thst disk file?

so this was pre-arpanet?

--
umop apisdn

It was in the early eighties. Those were different dialects of Unix (version 7, system III, system V) and later Xenix and they did not have networking code. The systems had multiport serial boards for async terminals, no ethernet. Datacomm was done using UUCP or proprietary protocols.

Not even mmap was available then, I think. We had a file somewhere and it had a slot per user where the request was written, then a signal was sent to the server process. The server process would read the file slots round-robin to find requests, handle them, write the reply and send a signal to the client process.

As features started to appear in the system, we gradually modified the library to take advantage of them.

Probably because the Unix-compatability shell in VMS does it too.

I found this (an MS article from 1999 - or at least it states so)

and under paragraph 'Named Pipes' it says:

" Though not compatible with UNIX Named Pipes, Windows NT Named Pipes are conceptually similar. Named pipes provide a high-level interface for passing data between two processes, regardless of network location. Named pipes, like files, are implemented as file objects in Windows NT and operate under the same constraints and security mechanisms as other Windows NT Executive objects. The named pipe file system driver is a pseudo-file system that stores pipe data in memory and retrieves it on-demand. When processing local or remote named pipe requests, it functions like an ordinary file system. "

I interpret that as not using temporary files. Or does the last sentence change the meaning?

Apparently there was a named pipe feature available at the time, but it was not possible to connect a process standard output or input to it without cooperation from the process.

That is what Unix does in the shell. It reconnects the filedescriptors

0 and 1 to a pipe before it execs the program, so the running program does not have to have any "pipe support" built in. It just writes to fd 1 or reads from fd 0 as it always does, and can detect that it is a pipe and not a tty when it wants to.

VMS and NT did not offer features for that, so they had to limb by using the intermediate files.

Ok. I remember trying to redirect stderr on a win95, which was impossible. It worked on NT though. So I figured it had something to do with pipeing, but perhaps not.

on the other hand, Wikipedia states

or not) that

" Anonymous pipes used in pipelining are actually named pipes with a random name. "

Which would have given support for not using temp files. But I'm just speculating and guessing here.

I know that the VMS designer (Ken something?) designed a great part of NT and if VMS needed temp files, NT may have inherited that. Although I do not think it is so anymore.

On 22 Aug 2014 18:50:59 GMT, Rob declaimed the following:

Boy, the 80s really spoiled me... Between VMS "mailboxes" and Amiga message ports (for all practical purposes, the Amiga lived on IPC), I find the various *n*x and Windows facilities quite limiting.

--
	Wulfraed                 Dennis Lee Bieber         AF6VN 
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/