HTTP transfer errors

Are you using Internet Exploder? I don't know about current versions, since I never use it myself, but previously it was famous for claiming downloads are finished and reporting everything okay, even though it was missing the end of the files.

No, Firefox. File *sizes* are correct -- so nothing is "missing" (though that doesn't mean that parts of the file haven't been filled from /dev/null...).

I will wait until I get another "bad" transfer -- followed by a

*good* transfer -- and compare the files. This just doesn't make sense...

s

Hi Don, http does indeed run atop tcp. A possible scenario is that the file gets allocated with the correct size at the very transfer start, then the transfer never finishes (there can be a header line specifying size in http).

Dimiter

I don't think Firefox pre-allocates files even if it gets such a header line. It will also give a "size mismatch" error if the size in the header doesn't match that of received data.

I would be inclined to suspect the server - do you know what software it is running?

One vaguely possible situation is that Firefox is using multiple connections to download the file in parts, and the server is claiming to support http resume, but failing to implement it properly. (Firefox does not use multi-part downloads by default, but lots of "download accelerator" add-ons do.)

Even though I'd suspect server problems more than client problems, try with a different client (Chrome, wget, wxDownloadFast, etc.). Maybe it will help.

Interesting idea. I've never watched how Firefox actually build the file -- other than to note that it creates an empty file having the "eventual" file name and pairs this with a file named ".part" (or somesuch). I assume it moves the "part" file into the name reserved at the start of the download when the transfer completes (because the .part file disappears at that time).

I will try to be more observant -- now that I'm (partially) rested! :>

No idea. I've not taken an interest, before. "Unimportant details" to risk cluttering my mind with :> (a la Kelly Bundy)

I looked at *one* file that I THINK was symptomatic of this problem. I had two copies (one with an intentionally mangled filename) that differed by a few hours in timestamp (i.e., as if I had downloaded one, found it "defective", then downloaded it *again* some time later).

I did a bytewise compare of the two files and they coincide (!). Of course, I may be mistaken in my belief that one *was* "bad" previously. I will try to catch the sucker misbehaving, later today.

On the assumption that the one *was* bad, I figured the only way this could *have* been the case (despite it "looking good" now) would be if I am having system problems, here.

I see no messages in the event logs regarding disk errors (which is where I assume they would appear) so I have to guess that the problem is happening at execution time... maybe a bad bit in RAM or "whatever" (not sure if this machine uses ECC RAM as it is an old 1.5G P4).

Or, maybe flakey bulk decoupling caps around the processor. A few more issues to explore... If nothing turns up, I guess I'll just recycle the machine (I don't like "unexplained errors").

Thanks!

--don

This does not answer your questions but, for what it's worth, I had similar problems some times with both Internet Explorer and Firefox. Whenever possible (have a full path to the file(s), no need to solve "captcha"s, etc.) I use wget instead. It is much more reliable, and gives more control on how to handle the transfer.

-- Roberto Waltman

[ Please reply to the group. Return address is invalid ]

Yes, the "real filename" file is a placeholder which Firefox makes and locks, so that other processes won't access it or make another file of the same name. In the meantime, it downloads the file to a ".part" file, and only when it is complete does it do an atomic rename to the "real" filename (at least, it is atomic on *nix - I don't know if these operations are guaranteed atomic on Windows. It's close enough anyway - we are not talking about RTOS's here).

Of course, another thing to suspect is your hash checks. Maybe you are doing something wrong on one side or the other (perhaps using a cygwin version of md5sum with some sort of "automagical" DOS/Unix line ending convertion)?

Note "I did a bytewise compare of the two files and they coincide". Assuming the first *failed* the check (which resulted in my downloading the *second* copy of the same file) yet the second

*passed*, how could this be the case given they compare as identical? [actually, a software *bug* could cause this -- as can hardware problems, etc.]

Since I couldn't *swear* to the circumstances surrounding each file, I set out, today, to just "catch" something misbehaving. I now have three (maybe four?) copies of a file all but the last of which failed it's consistency check (this is a RAR archive so it puts the onus on WinRAR to Do The Right Thing... one less variable to deal with!)

All have the exact same size. I'll do bytewise compares and see if they are, in fact, "identical" -- or, if there is some

*pattern* of discrepancies among them. (i.e., the most bizarre would be for each file to differ from all of the others! :< )

Then, verify that these "differences" are static (and not something related to a glitch in the processor, etc.).

(sigh) What a waste of a day... :<

I only expose this one Windows machine to the outside world (wget(1) being available "internally"). Perhaps there is a windows port of wget... ?

Hmmm... that begs the question: how can *any* such file transfer ever terminate prematurely *if* the server makes the client aware of the actual file size? (or, is this an optional part of the protocol)

Oh that can happen easier than you think. I know I have done it, others may have misbehaved similarly. Generally, file size is used to allocate disk space at the beginning/ensure there is enough of it/ ensure the file is not scattered much over the disk (appending for a long time while other tasks allocate disk space means scattering). So once allocated the right size, it is conceivable that if the server closes the connection for some reason the client will forget this file had a defined size and happily truncate it, or worse, close it untruncated. Clearly an error but conceivable. Then size is not mandatory, signaling EOF by closing the connection is OK in http and in ftp, for that, which makes the coding error easier to get encoded. But this is not your case, if you have had many downloads and all files do compare either something other than tcp/http is not working or, most likely, you just need more rest :D (so do I, although at the moment I am more generally exhausted than tired and could use a week or two _real_ rest, alas, can't afford stopping now).

Dimiter

------------------------------------------------------ Dimiter Popoff Transgalactic Instruments

------------------------------------------------------

At the moment, it sounds like you are doubting everything - a good idea when you don't know what's going wrong and all the parts /should/ be reliable.

If you doubt WinRAR, try 7zip - it will happily check and open a rar file. And unlike 99% of WinRAR installations, it is legal to use.

There is indeed wget for windows - the easiest way is to install MSYS (it goes along with MinGW, if you want a gcc port). I couldn't imagine having to use a windows machine for work without MSYS.

HTTP doesn't actually inform accurately about file systems. You can optionally include a size line in the header, but there is no protocol-level guarantees. It's main purpose is to let browsers display progress bars and percentages while downloading. With HTTP, the server sends the entire file and then closes the connection - it is the connection close that indicates the end of the file. The client can tell if the connection was closed properly, or prematurely due to an error.

-- Roberto Waltman

[ Please reply to the group. Return address is invalid ]

I now have been able to observe different behavior on "identical" downloads (i.e., HTTP is out of the picture). I thought "perhaps a bad spot on the disk" (affecting one copy of the file and not another -- though why doesn't XP tell me about bad reads??). But, I have since been able to get different behaviors from the *same* file -- run the hash check twice (or three times, or four, etc. -- it should *always* give the same results, yet doesn't!)

The checks yield correct/consistent results when the files are moved to *another* machine.

So, I *probably* have a hardware problem. Simplest fixes are:

- check disk cable

- swap out RAM

- look for toasted 'lytics

- shitcan computer! :>

Of course, you can betcherass I won't be "moving" the files to the other machine but, instead, will *copy* them so the originals remain intact until I have verified the "copy" was successful (i.e., didn't copy "bad bits")

"Interesting" when you have time to "be interested" ;-) This week I had planned on installing the three picture windows in the kitchen/family room. Since that's a fair bit of demo and reframing work, I don't need any "other interests" currently! :-/

[the one consolation in all this is the problem has been "persistent"... didn't creep away for hours/days at a time leading to the frustration of chasing down an intermittent!]

Understood.

Well, the "rest" definitely improved my *attitude* :> I've learned not to keep bricks, hammers or other "implements of destruction" near machines when working -- lest I be too tempted to put an end to the problem! ;-)

Hopefully a sign that orders are coming in and *not* that things have gone down the crapper. Tell L you want/need a good back rub. Maybe she'll have "more interesting" suggestions! ;-)

[OTOH, she might come up with some other *chores* for you to do so you might want to keep a low profile, there! :-/ ]

Thanks, Roberto! I'll bookmark that and try to download it later. Though I am pretty convinced my particular problem is confined to this particular machine.

Annoying thing about flakey hardware is you can never

*confidently* salvage parts from the machine -- since you never know what the heck was flakey! :-/

Perhaps save directly to a USB stick or external drive?

Run memtest from any Linux live CD or installation CD.

Sometimes you can smell them....

Also test it from a live Linux CD to avoid any OS or software problems, and to avoid any disk issues. If you've still got problems, the memory is the likely cause, followed by motherboard or cpu issues.

So saying "it is the connection close that indicates the end of the file" isn't strictly correct. When EOF doesn't occur at exactly the right point, it's an error; nothing actually "indicates" the end of the file, and there's no specification as to what the browser should do about the data (whether to discard it, use the Content-Length, or use the EOF).

If the file isn't discarded, the EOF is more likely to be used, as a Content-Length header isn't allowed when a Transfer-Encoding header is present (e.g. if compression is used), but there will always be an EOF.

I thought of using an external disk. But, that moves data differently (i.e., slower moving data in/out USB than directly to/from a disk). E.g., if the problem is related to the amount of activity (in the processor, on the bus, in the disk, etc.) then this would change -- leading to misguided diagnosis of the "real" problem ("Ah! Disk must be bad since using USB drive 'fixes' the problem!")

The first "bad cap" motherboard I ever encountered exhibited symptoms like that: system would run *fine* -- except under heavy disk load.

Yeah. Or, just swap out the memory. If problem persists, chances are it's something *else* (or bad sockets)

IME, they're pretty easy to spot. Look for the warm spots on the board, around the processor, etc. If I was a *technician* and wanted to spend my days replacing caps, I could probably live quite comfortably!

I've got a dual 3.4G box half assembled. Maybe I'll just finish putting that together and retire this machine (I had originally planned on using it for multimedia presentations -- but that can wait).