Special FAT filesystem questions [Was: Multiple files with same filename on FAT]

The discrepency between the file system identified by Ghost and the one identified by XP may be significant. It may even be a clue that the real file system is the same as the one used on the DVD, or at least similar to it (Joliet, perhaps)- that *would* simplify data transfers. If that just happened to be the case, you could be looking at an ISO 9660, Level 3, or UDF (ISO/IEC 13346) file system (with or without a bridge to ISO). The Wikipedia entries for those file systems are informative: and . Follow the link to ECMA-167 () for even more depth.

This isn't very relevant to the FAT on the partition, especially if that's the only partition out there. Don't take this discrepancy too close to heart. :)

I would strongly recommend getting a copy of the MS FAT documentation from their web site and read through it:

If you read the doc, you'll know it's effectively FAT28, look for 28 in the doc. :) Seems like I have a minor bug in my code: I don't zero the 4 top bits in GetFATEntry() (or its callers) and I zero them out in SetFATEntry(). As per the doc both things aren't quite right. So, on your HDD my code will have problems because of this (and the wrong partition type, to begin with).

Could be a copy. Check the bootsector for the number of FAT copies.

HTH, Alex

Thanks for your reply. Nevertheless, preliminary analysis make me think this disk has a "99%" compatible FAT32 format. First mount attempts show that XP can mount it, navigate directories and read most of the files OK. The "weird" symptoms are random-looking or absent file timestamps, multiple entries with the same name and disk-ruining chkdsk in case you let it run :-)

Thanks for the suggestion anyway.

Vicne

Noted :-)

it:

Oh yes indeed. That's much more complete than the few pages I worked with. Thanks for the pointer.

I quickly browsed the doc and I swear I'll read it thoroughly :-), but reading your answer, it strikes me that the spec speaks about 28 "address bits" indeed, while I observe 24 a full FF byte after 3 variable ones...

Oh, so "I" found a bug without even compiling the code ? :-)

It says 2.

But what I find strange is that I have somthing like : "FAT24" (1st copy) "FAT24" (2nd copy) "FAT32" (1st copy) "FAT32" (2nd copy)

The "FAT24" is only about 300 sectors long while the "FAT32" is about

You see, I'm wondering if there isn't a kind of proprietary structure that would replace or complement the FAT in case of files > 4Gb, that would create dummy directory entries (with duplicate filenames) in the directory and would consist of a "chain of files" or something, stored before the FAT.

On the other hand, I'm just a newbie regarding this as you know, so maybe I just can't see the obvious... I'll definitely read the doc and the relevant code you pointed me to, to see where exactly the boot sector says the FAT should begin. Hopefully, I'll start to write some code next week-end to test things out.

Thanks a lot for your time and help. It's much appreciated.

Vicne

PS: BTW, Alex, do you have a recommendation regarding the OS for compiling and running your code ? I first planned to use FreeDOS but it seems it messed with my PVR clone disk (FreeDOS now appears in the first copy of the boot sector !). Moreover, I tried to copy a (non-duplicate) Mpeg file with it and the copy was truncated, while both XP and Linux performed the full copy without any problem. Consequently, I now plan to use the MS-DOS bundled with Win98 (boot on floppy, "fdisk/format/sys c:" on the destination HD and copy over utils from the oldmsdos directory of Win98 CD. What do you think ? V.

it:

I don't know. Poke around. See if those things make sense, describe some data structure.

Could be, although if you're certain with the previous discoveries that big files are just equally named 4G pieces, then it would be odd to have a different accounting for those 2nd and further parts. Then again, remembering the ext2fs-like FS, maybe not. It was something like:

- file data

- ptr to block of N ptrs to next N data blocks

- ptr to block of N ptrs to N blocks of N ptrs to next N*N data blocks

- ptr to ... next N*N*N data blocks

Compiling: DOS or 32-bit Windows (9x, 2k, xp). Running: DOS or bare machine with BIOS only.

Read the documentation to see what tools you need and how things are compiled.

That environment should be safe so long as you don't write anything to the camera's disk with DOS. :) Actually, you could greatly simplify your life by simply taking the raw image of that drive and tweaking the test code in image.c to the drive's geometry. That way you can use the test code to "mount" the image and work with it. The test code is portable C code (no ASM, no BIOS calls to do disk I/O) which you can easily tweak and debug under any OS you want. See imgcpy.c. It was written to copy files to/from a disk image file, almost what you need (you only need to figure out the details of the FS and tweak FAT.c to work with it and duplicated names). Just get fattest and imgcpy compiled and working as-is and then use your image instead of the test one and start digging, tweaking, etc.

Alex

The ReservedSectors BPB field is the number of sectors from volume start to the first FAT. It is usually 1 or sometimes 2 for FAT32 (FAT32 has an FsInfo sector which is not used by Windows NT's FASTFAT).

All of which are indications that it is *not* FAT32.

Can you provide us with the exact Extended BPB values?

it:

My guess would be that there might be a short FAT24 partition for the DVR to use for code updates, local storage etc. whilst the FAT32 is used for the actual finished MPEGS. The FAT24 might be hidden from general view in the same way that many computers have a hidden partition for diagnostics code.

Matt

Matt

it:

Just a thought, but could the screwed up dates/times etc. be because the directory entry has used these fields to store the addresses or offsets to the next file chain, or next directory entry?

Matt

That would be strange to say the least, but well, many things seem strange to me for the moment. I guess the more I'll work on it, the clearer it will get.

It could be... Although, most of the timestamps carried the same weird value at first, but well, that's an interesting track to follow. I'll pay more attention to these fields.

Thanks for sharing your thoughts,

Vicne

Sure. I was not claiming any credit :-)

It looks like a linked list, but I'm going to leave it alone for now and concentrate on what is standard (or seems to be so). If I get stuck, I'll come back to it.

Yes, indeed...

Wow, some kind of tree, then ? Well, when I'll be a FAT-expert, I'll investigate those. Seems to be cool :-)

OK.

I have DJGPP installed and by screening your doc, it seems sufficient to build FAT module and test suite, so I'll begin with that.

No, of course. Well, unless by mistake, I mean :-)

You're right. I'll try to do that. I guess only NTFS is suited to both support a 160 Gb file and be recognized under XP, right ? So I need an imaging tool that runs in plain DOS and can a bit-by-bit image write to an NTFS drive. I'm only familiar with Ghost however, but it writes images in its own format if I'm not mistaken... What tool would you advise ?

I see. Great !

Yes. Sounds the best plan indeed.

Many thanks for your help

Vicne

it,

symptoms

the

This guy has blog on developing a modified FAT for DVR's. Most interesting is summary 2:

"However, at the same time, the Filesystem team developed an improved algorithm which should greatly help this situation: The original FAT code we were working with included a single per-open-file 32-bit cached entry of the last cluster accessed within the file, so sequential reads/writes always hit this cache and didn't need to retraverse the FAT chain"

Anyway, MS has produced a number of custom FAT FS, such as ExFAT (WinCE), FATX (Xbox), TFAT. exFAT includes TFAT. TFAT uses multiple copies of the FAT table which I think you mentioned somewhere...

FATX here:

TFAT here:

exFAT here:

Rod Pemberton

Hi,

Thanks for these pointers. Very interesting reading indeed. I'm still hoping too this "FAT24" is only there for performance reasons and not absolutely needed to read and extract files.

Kind regards,

Vicne

I made progress in that direction yesterday :

- built imginit and imgcpy,

- played with them to create a dummy image and simulate the directory structure on my PVR (\VIDEO folder with a few files in it)

- created a new 'imgdir' utility based on 'imgcpy', listing entries in a directory in both hex and "interpreted" form

- replaced the dummy image by an image of the first sectors of the actual drive and dumped the entries in the \VIDEO directory

All of this is working but I'm having difficulties determining what data was on the original copy and what was changed by mounting and accessing files under various OSes... Moreover, I only exported the first 35 Mb of the HD (MBR, BS/BPB, FAT, root dir, VIDEO subdir and the few first sectors of actual files) so no actual file extraction test can be performed...

So working with a fresh image is the next step but I'm still stuck with a detail: how can I build a bit-by-bit image of a 160 Gb hard- drive without mounting it first under Windows...

Anyway, thanks for providing the FAT toolkit, Alex, it's really great.

Vicne

Use one of the many live Linux CDs, to boot into linux and use the tools under linux to do this chore.

Regards Anton Erasmus

Sorry, linux newbie question : Can I seamlessly mount an NTFS drive under Linux and write a 160 Gb image file there so that the rest of the process can be performed under Windows ? While googling around, I found

Finally, what linux command can I use to be sure that not a single bit will be modified on the source drive and that the image will be perfect (including seemingly "corrupt" entries) ?

Thanks for the advice.

Vicne

AFAICR Knoppix can mount NTFS volumes without problems. All volumes are mounted as read only initially. From the Knoppix desktop it is easy to re-mount a volume in read/write mode.

To copy the whole contents of a harddrive one can do the following:

dd if=/dev/hda of=/mount/ntfsvolume/image.bin

This of course assumes that the drive you want to make an image of is an ATA drive connected to the master slot of your primary controller. SCSI and SATA drives are mounted as /dev/sda /dev/sdb etc. /dev/sda1 would be a partition on /dev/sda It also assumes that you have mounted the NTFS volume where you want to write the image at /mount/ntfsvolume.

A comercial product which provides a much simpler user interface, and which is very usefull for recovering data etc. is Paragon Partition Manager. There is a version which installs as an app in windows, as well as a live CD version which boots.

Regards Anton Erasmus

Paragon is junk.

Good image backup products - Norton Ghost, Acronis TrueImage and StorageCraft ShadowProtect.

Paragon is, for instance, _4 times slower_ then these three, which have very similar performance.