how to pre-grow file to certain size ?

Do you have a question? Post it now! No Registration Necessary

Translate This Thread From English to

Threaded View
Hello,

I would appreciate any advice on the following problem.

I would like to pre-grow file to certain size (few MBytes). I'll
explain what I mean by "pre-grow" by concete example. Let's say
that I have a file that is currently 1 MByte in size, and occupies
250 disk blocks (assuming disk block is 4 KBytes). I would like
to increase the size of this file to 2 MBytes, and extend it by
allocating additional 250 disk blocks to it. I do NOT want to
actually fill these additional 250 disk blocks with any contents;
I simply want them to be allocated, and all file system metadata
updated accordingly.

If case if file system is a factor here, I would be using FAT
file system.

Would that actually be possible ?

Thanks in advance and regards,
Andray


Re: how to pre-grow file to certain size ?
Quoted text here. Click to load it

To allocate the storage, you would actually need to write to the file,
and fill it with 'something'.


--
:wq
^X^Cy^K^X^C^C^C^C

Re: how to pre-grow file to certain size ?
Hello,

Quoted text here. Click to load it

You need to write something into the file. Only then the file system really
allocates that space.

Quoted text here. Click to load it

Yes, you need to make the file longer, append a lot of information. At least
a series of zeroes.

Quoted text here. Click to load it

I think NTFS can allocate without making the file longer using streams, like
in "myfile.txt:mystream". But, of course, this is not applicable in
comp.os.linux.* :-)

Quoted text here. Click to load it

dd does a great job at this. :-)
dd if=/dev/zero of=myfile bs10%24k count=1

Or if you want to preserve the data:
cat myfile /dev/zero | dd of=myfile bs10%24k count=2

Regards,
Sebastian



Re: how to pre-grow file to certain size ?

Quoted text here. Click to load it

I doubt there is any UNIX system on which the command above reliably
preserves the data:

$  echo "hello" > myfile; cat myfile /dev/zero | dd of=myfile bs=1k count=2 &&
od -c myfile
2+0 records in
2+0 records out
0000000   h   e   l   l   o  \n                    
0000020                                
*
0004000

$  echo "hello" > myfile; cat myfile /dev/zero | dd of=myfile bs=1k count=2 &&
od -c myfile
2+0 records in
2+0 records out
0000000                                
*
0004000

Where did "hello" disappear in the second try?
The "race condition" ate it :-(

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Re: how to pre-grow file to certain size ?
Hello,

Quoted text here. Click to load it

Strage. I thought "cat" can be used to copy files together and "dd" in this
case only restricts the length...

Quoted text here. Click to load it

I didn't think about something like that, sorry.

But how to do it otherwise?

Regards,
Sebastian



Re: how to pre-grow file to certain size ?

Quoted text here. Click to load it

The problem is that you specified the same name for the output file as
the input file.  The race is in whether cat manages to read your file
before dd clobbers it.  Try this instead:

blksize40%96
fsize=$(stat -c%s myfile)
fblks=$((($fsize + $blksize - 1) / $blksize))
addblks=$((500 - $fblks))
cat /dev/zero | dd of=myfile bs=$blksize seek=$fblks count=$addblks

GH

Re: how to pre-grow file to certain size ?
On Wed, 02 May 2007 14:46:34 -0400, Andray Kaganovsky

Quoted text here. Click to load it

In many file systems just seek to the desired position, write
something to that position, which moves the end-of-file mark to that
position and allocates all the space in between.

Using a large preallocation often helps avoiding file fragmentation,
since large fragments are allocated in this case, however, extending a
file with a few blocks at the time may fragment the file very badly,
since each extension goes into a separate segment of blocks.

Since this is a security risk, some operating systems designed with
higher security standards may limit this feature or actually clear the
content between the old EOF position and the extended seek position.

Quoted text here. Click to load it

At least the Microsoft documentation indicates that using fseek() to
the desired size will actually move the end-of-file mark, even without
writing something at the new position.

Use ftell() and fseek() to move around inside the file for reading and
writing.

Paul


Re: how to pre-grow file to certain size ?
Quoted text here. Click to load it

This only allocates space if the file system does not support sparse
files. I suppose FAT in fact does not support sparse files. NTFS and
EXT3 does.

-Michael

Re: how to pre-grow file to certain size ?
On Thu, 03 May 2007 10:24:06 +0200, Michael Schnell

Quoted text here. Click to load it

Sorry for the confusion,

I assumed that I was still reading comp.os.embedded (which is before
comp.os.linux.emebedded in my news reader group list :-) and some
historical file systems, such as Files-11 and RMS-11 (PDP-11/VAX) did
indeed support such features.

Unfortunately, file systems with some more complex features (such as
ISAM) were not popular among the Unix family.

Paul


Re: how to pre-grow file to certain size ?
Quoted text here. Click to load it

Some real-time operating systems offered a system call
to guarantee it was accomplished as desired:
allocating contiguous disk space, when possible.
That made it easier for DMA controllers to access
the disk directly, totally bypassing the file system
and operating system for raw access,
yet giving the blocks a filename for access via the OS environment.

Quoted text here. Click to load it

As another reply noted, some file systems support "sparse" files.
Most Unix/Linux systems allow files to have "holes":
unallocated blocks withing the file.
That is why 'du' (disk usage) does not always jive
with the file size reported by "ls".

A program that just lseeks to 1,000,000 and writes ONE byte
ought to show a file SIZE of 1,000,000
but 'du' will report only one BLOCK allocated to the file,
which may be 4k (8: 512 byte sectors) or other sizes as configured.

--

-- mejeep deMeep ferret!

Re: how to pre-grow file to certain size ?

Quoted text here. Click to load it

This was a more or less standard feature on mainframe OS's or
minicomputer OSs, such as RSX-11 and VMS.

Quoted text here. Click to load it

The real reason for using contiguous files is that you could
read/write the whole cylinder at the time and after that, only do a
track-to-track seek, which was faster than a end-to-end seek. In old
days, with rotational latencies in the 15-40 ms range and with
comparable seek times, it made a lot sense to put data to be read
sequentially into a contiguous disk region.

With a file fragmented all over the disk, it could take up to 50 ms to
read the next sector due to rotational and seek latencies.

However, with more intelligent disk drives accepting logical block
addresses and doing bad block replacement on the drive itself, the OS
is not necessary aware of the physical layout of blocks on the disk,
so the advantage of contiguous preallocated files is not so obvious.

Paul
  

Site Timeline