I would like to know your opinion about the following requirements:
- data archival: 327.68 MB to process in 2.56 s, this task should be done in real time for very long periods of time - image generation: 5.12 MB to process in 2.56 s, also in real time (in each interval of 2.56s data should be processed)
Such high speeds are achievable in Linux ? Are there any tools to evaluate how fast data is written to harddisk ? Is there a way to evaluate image generation from raw data ?
Pretty much any speed is possible if you are happy to spend enough on hardware. The challenges here are for continuous real-time processing without pauses.
Do you need to save or buffer the data? Your description suggests you are reading in 327.68 MB data, processing it, then writing out 5.12 MB every 2.56 s. If you don't have to store anything, then you are just talking about processing speed.
Can you make use of multiple cores or threads?
If you have to store data, will it be a lot in total? If you don't need fast reading, storage at this speed (average 128 MB per second) is easy enough if you have a raid array and store to 3 disks in parallel (excluding any parity disks). Have plenty of ram for buffering.
When you want to have reliable real-time, or almost real time, performance, make sure your system has as few other tasks as possible. Keep your Linux installation small - no X, and no services that you don't specifically need. The last thing you want is an updatedb cron job reading through your entire file system in the middle of the night. Also keep the network isolated, so that the system is not having to deal with unnecessary packets.
Actually I need to store 327.68 MB to hard disk and in parallel to process 5.12 MB in order to generate an image. All this should be done continuously in a time interval of 2.56 s. I guess I have to either sent through network the processed image and don't use locally an X window system or to process and display everything locally. My question is how the system should be configured: process everything locally or sent the data for image processing to another PC ?
Back in the 80s, Real Time Unix systems such as Concurrent Computer or Masscomp did such things all under one hood, albeit with multiple processors. With Real Time Linux extensions and multicore CPUs, I'd suspect it's possible with proper deisgn such as
- pre-allocating the disk file so there's no delay for block allocation
- giving highest real time process priority to data acquisition and storage, low priority to display (which can catch-up if required).
- bonding high proprity tasks to separate CPU cores
ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.