Wrong forum, possibly.
Can two separate programs (not threads) share a file handle? I'd like to have one program open a file, read a little from it, then turn it over to another program that reads the rest of the file, all the way to EOF.
Sharing a memory region between the programs is apparently no problem.
Or, where should I ask this question?
-- John Larkin Highland Technology, Inc picosecond timing precision measurement jlarkin att highlandtechnology dott com http://www.highlandtechnology.com
Can you have the first program output the second part of the file and pipe it to the second program?
-- ----Android NewsGroup Reader---- http://usenet.sinaapp.com/
Den onsdag den 21. januar 2015 kl. 01.42.47 UTC+1 skrev John Larkin:
There is a way to pass file descriptors between processes with pipe/sockets I've never done it and afaict it is really easy to mess up
you can't just pass an offset to the second program?
-Lasse
Google something like "linux process share file descriptor" First couple of hits give a couple of ways.
-- Chisolm Republic of Texas
Two files can perform a simultaneous read on a file, but telling anapplication to "begin to read here" is not likely 'doable'.
A file "open" for editing, however, is usually in a "locked" state for others, though they may get "look", or "red only" access with a warning about immanent changes to be considered. Or the second user would be completely locked out until after the edit session is over.
Imagine two user simultaneously making an incremental edit to a database file.
One term invoked is "cache coherency", but the actual thing I suppose which takes place is that one gets the hot edit session, and the other gets a read only "look" session, until the edit session closes, and then that user gains edit access.
For read only shouldn't it be OK to just pass the int fd to the second process, and have the second process open /proc/self/fd/
-- ----Android NewsGroup Reader---- http://usenet.sinaapp.com/
Yeah, it looks really messy, sockets and complicated stuff. As far as I can tell, the file handle is unique to the process so we can't just hand it over to another process.
The first program "gen" is going to get user command-line input to open the file, and it terminates after executing the command line. The second program is a background demon that spools file data into a FIFO and thence a DAC. So the demon can't be a child of process gen. A child or thread does share file descriptors, but we can't do that here, I guess.
So gen will have to pass the filename to the demon, shared memory maybe, and ask it to open the file, and check/block until it's done. The demon will open the file and begin streaming into the fifo. Not the way I wanted to do it, but it should work.
Some people spell it 'daemon'.
-- John Larkin Highland Technology, Inc picosecond timing laser drivers and controllers jlarkin att highlandtechnology dott com http://www.highlandtechnology.com
After reading from the first program, call ftell(), send offset to second program, which opens the same file an calls fseek() with that offset ?
absolutely:
( head -5 > top.txt ; cat > rest.txt ) < 100lines.txt
there's several other ways to do this sort of thing.
-- umop apisdn
If one is a child of the other, this is easy to do. But in your case (from your later message) it can't work that way.
If all you're doing is reading the file, you can have as many programs as you want reading it at once, but they will all have separate file handles. (If you don't explicitly use flock() or equal, you can probably have multiple programs *writing* it at once, but that way lies madness.)
I'm guessing that gen needs to read something like a header block to maybe validate the file, set up GPIO pins, etc, and then the daemon needs to read "everything else" to get the waveform data.
IMHO the cleanest way is to put the file parsing into a function (or library), that gets compiled (or linked) into both gen and the daemon. Then gen doesn't have to tell the daemon where the data starts; the daemon can find the start of the data itself.
If the use case is that the user should be able to say "play back from
1:23 to 4:56 of this ten-minute file", then I still think having both the daemon and gen know how to parse the file is a good idea. gen can do whatever setup it normally does, maybe check that there actually is data all the way out at 4:56, pass just the 1:23 and 4:56 to the daemon, and the daemon can then compute that it needs to play from bytes 6789 to 101112 of the file.Random advice, for free: include a version number (at least a byte) somewhere in the header of the file. The second you think you'll never change the file format, "never" gets redefined to "next Thursday". :) Having a version number to check helps you a lot when v2 comes out.
If you're feeling fancy, and the version number string is unique enough, you can stick it in /etc/magic, so "file waveform.bin" will say something like "Highland Technologies model 123 waveform file, v1". This doesn't really help the code, but it helps the humans who are dealing with the files to identify what they have.
Depending on how you think the files will be generated, used, and possibly modified, it might be a good idea to put a checksum of some sort on the file. You pretty much don't have to worry about a bad read from the filesystem, unless you're a satellite or something. You might be interested in making the files a bit harder to edit, in case there are non-obvious consequences - say somebody clever decides to double the frequency of a given waveform by editing the binary waveform data, but that puts it over the top frequency of the D/A. Or they bump up the amplitude of the binary waveform, not knowing that you also need to set the attenuator or amplifier control bits in the header to make that work right. Basically, it nudges them towards going through whatever record tool you provide, so the record tool can say "sorry, Dave" if they ask for something silly.
Maybe one of the comp.os.linux groups, but I don't know which one to recommend or if they are any good.
Matt Roberds
Is the "little from it" known a priori? Or, is it determined by examining the context *in* the file?
In the former case, you can just strip what you need from the front of the file in the first program. Then, *reopen* the file in the second program (by passing the same "filename" argument to that) and skip past the "front portion".
A better choice would be to execve(2) both "programs" SEQUENTIALLY from a wrapper and let the second program inherit the file descriptors IN THEIR MODIFIED STATE after the first has finished with them. I.e., just let the first program *stop* reading the file (perhaps pushing back anything that it "accidentally" read beyond where it SHOULD have stopped) and then let the second program pick up from that file position.
This avoids the possibility of the file being accessed by "something else" in the time between Program1 accessing the "front" and Program2 accessing the "back". (i.e., atomic access -- use flock if necessary)
You probably (?) could manage this by building a filter as a shell script and letting first Program1 process stdin while leaving the balance to Program2.
Traditional Unix API allows FDs to be inherited by a child process. Have to be careful though because by default there's only one file pointer, so you need to hand-off cleanly and not have one process seeking while the other is trying to read a different bit.
With an inherited file handle, you can use fdup to copy it so you get your own pointer. A parent process can fork multiple children who can all fdup the same inherited file handle.
IIRC some unix variants may have implemented ways to pass FDs. You can open up permissions on /dev/fd/* (which points to /proc/self/fd/*). Windows has ways for a process to open a file and pass the open handle to another process, we had to use that once.
The command line program "gen" would open the user-specified file
gen -c h -file xxxx.dat # open xxxx and start loading generator h
(there are 8 wave generators, a...h, each with its own FIFO)
and mostly fill the FIFO, in preparation for later starting actual waveform output simultaneously on all channels. The FIFO can hold 32K samples, so we'd always pre-load it with, say, 28K. gen would, ideally, turn over to the file demon the responsibility to keep reading the file and topping-off the FIFO as needed, then terminate.
That would work. The demon would have to do the time-expensive operation of opening the file (while servicing as many as 7 other file/FIFO channels) but, I guess, somabody has to do it, and whoever does it punishes the file system anyhow. So policy should discourage opening waveform files when other channels are pumping out high-rate waveforms.
"gen" is a command-line program that will execute and terminate. It would block on the initial file open and data read operations, then exit. I have 8 channels to manage, and one file demon could do all 8, structured properly.
I'm not a Linux programmer, or an FPGA designer (or, in this case even the circuit designer) but I am doing the product definition, customer negotiation, and the general architecture. There are a zillion possible states: startup, shutdown, EOF, fifo-dry errors, multi-channel time and phase syncs, errors, visibility issues, potential races and hangups. I tend to think in simple, synchronous state machines, so I want to avoid sockets and stuff that I can't totally understand (ie, avoid asynchronous state machines). Seems like Linux isn't perfectly suited to this application, but it's pretty much mandatory. The compute platform is a microZED with the Xilinx 7020 SOC, dual-core ARM and FPGA on one chip. Waveform files will be on a
128 Gbyte SD card, which is a speed bottleneck.I can at least have shared memory between processes, so I guess I'll use state flags and have "gen" handshake with the file demon, and let the demon do all the file operations.
-- John Larkin Highland Technology, Inc picosecond timing laser drivers and controllers jlarkin att highlandtechnology dott com http://www.highlandtechnology.com
just have the demon running in the background all the time with a socket as a "command interface"
I'm working on a zynq and all the stuff of telling programs what to do, storing settings, error messages etc. etc. is done with a mysql database
-Lasse
As I said, I prefer using shared flag words, where I can make sure things are atomic. The async pipeline nature of sockets makes things a lot harder to think about in real time. I need a flag/state word per channel anyhow, for overall visibility.
I'm realtime critical here, trying to get the max throughput on all 8 analog outputs. Just opening a waveform file is scary enough. I'd go bare metal if we didn't have to swap files and stuff with the customer's Linux system.
There is an appnote for running Linux on one ARM core and bare metal on the other, but we need to muck with files, so we can't do that.
-- John Larkin Highland Technology, Inc picosecond timing precision measurement jlarkin att highlandtechnology dott com http://www.highlandtechnology.com
The waveform files are pure 16-bit samples, no headers. The hardware, FIFOs and such, is ready and waiting. gen will zero the FIFO and set up some other stuff (DDS frequency, filtering, whatever) and open the file and pre-load the FIFO, then exit. The file demon would take over when the waveforms actually start. If I can't pass a file descriptor, that won't work.
The head of the FIFO looks like a big dummy 64Kbyte address space; any write goes into the FIFO. I figure we can do file reads directly into that space, and avoid touching the data at all.
I want to keep the demon fast; it has to feed 8 channels concurrently. The less it does, the better.
I'll make the user convert whatever he has into pure 16-bit binary samples and deliver us those files. "Make it their problem."
-- John Larkin Highland Technology, Inc picosecond timing precision measurement jlarkin att highlandtechnology dott com http://www.highlandtechnology.com
Den onsdag den 21. januar 2015 kl. 19.19.34 UTC+1 skrev John Larkin:
it wouldn't be any more or less async than your commandline on your "gen" program.
imagine you "gen" program always running in the background and also containing your fifo filling demon
I doubt linux will limit throughput much the SDIO will be the limiting factor Linux will add a bit of latency, but with big buffers that shouldn't be an issue
-Lasse
I do not know if this is relevant,or if it has any elements which may be usable, but your remarks here sparked a memory of when I was trying to get my G15 keyboard LCD display to show a logo in Linux. I found a guy patching live video to it, and there were a lot of live file queues being read and pasted.
I think I even found and looked at the code at some point.
If the daemon had two threads, one could be doing the file handling and the other doing the hard RT stuff.
Cheers
Phil Hobbs
This. There are two cores, so use two processes. Also, if all the program states are as complicated as he says, I think it would be pathological to write the whole program in a language like C. Write the user interface/housekeeping code in a higher level interpreted language like Python, and then farm out the real time critical portions to compiled C library functions. It's easy to combine the two, and using a higher level language to organize things will make implementing customer changes much easier.
This is essentially how video games are programmed nowadays. The graphics engine needs to be high performance and is written in C/C++, but the game logic is often written in a scripting language for convenience.
-- ----Android NewsGroup Reader---- http://usenet.sinaapp.com/
ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.