In comp.os.linux.embedded Rainer Weikusat wrote: | snipped-for-privacy@ipal.net writes: |> In comp.os.linux.embedded Rainer Weikusat wrote: |>
|> |> but more importantly ensuring that |> |> another app doesn't eat the free disk space we saw when the file |> |> started coming in (some of our target platforms are tight on disk, and |> |> we're not the only swimmer in the pool). Example: |> |> - 8GB file starts coming in to our app, and there is 10 GB free on |> |> disk |> |> - while the 8 GB arrives, another app creates and fills a 3 GB file |> |> (legal in a sparse FS). |> |> - by the time we get to the 7 GB mark of OUR file, we hit disk |> |> full. |> | |> | If separate applications try to store more data on a disk than free |> | space is available, some or all of them will necessarily fail to store |> | what they wanted to store. |>
|> I think he wants to be able to force the dense allocation as a way to |> verify that the space is indeed available and cannot become unavailable. |> The 2nd process he mentioned that would come along and grab 3GB would be |> the one to fail, and it should fail before the 3GB it gets from somewhere |> is actually sent. The sender needs to know if the space is _committed_ |> before sending, is my guess. | | I understand what he would like to do, but that doesn't change the | fact that this is basically a policy-descision made by someone using | the respective applications. That someone needs to have enough disk | space available for all concurrently running tasks and if there isn't | enough disk space available, some or all of them will fail and the | solution is to either not have different applications compete for | resources that are scarce enough that there will be a 'loser' and a | 'winner' or to accept that this will happen.
Competing for resource is not an uncommon issue. But such competition does operate more gracefully if the latter tasks can at least know BEFORE they take certain steps, that the space is committed to them. Perhaps those steps are irreversable once started. In such a case a way to deal with the resource competition is to try to get it and at least know if it cannot get it right now, perhaps to try again later.
There's a deadlock possibility if this isn't done. Suppose some other process will take the arriving files, process them in some way, and remove them when done. But that process cannot operate unless and until the file is completely available (for example listing the contents of a PKZIP file, archiving the list, and removing the zip file). Suppose there is 10 GB of space to work with to stage the files as they arrive over the network. If the logic of the receiving daemon were to just pause when the space is full, waiting for it to empty out, this will produce a deadlock when 2 very large files together exceed the space (say 2 files of 6GB that have transmitted 5GB so far). There is no way out but to abort at least one of these. And if the nature of the sender is that it is committed once sending starts, that can mean loss of data. The way around that is to get a space commitment and if the commitment fails, try later or be put in a queue.