Windows tcp Rx hanging

Considering that ftp doesn't utilize tcp, I think you left something out of your explanation, such as what program is servicing the windows end of the tcp connection. I would suspect that it isn't keeping up with the data flow.

...open foot, insert mouth

I did play with them - and they are nothing special anyway (e.g. 1460 bytes tcp segment size etc.), things are pretty much "normal".

And I must say it only happens with the ftp server, so it may be its fault after all (get clogged at some point and stays stuck with the socket buffer being full, hence the constant 0 window - while the rest of the network is OK and other tcp connections keep on working).

I'll try to locate and install some other ftp server and see what happens.

Dimiter

------------------------------------------------------ Dimiter Popoff Transgalactic Instruments

------------------------------------------------------

Original message:

You might have to adjust your system's MTU, MSS, or other networking parameters. I'm not certain how to do this in Windows, or why you would be having problems on a 100 Mbps Ethernet link.

Often this is the solution to such problems on dial up (PPP) or other oddball kinds of links.

--
Paul Hovnanian     mailto:Paul@Hovnanian.com
------------------------------------------------------------------
Applying information technology is simply finding the right wrench
to pound in the correct screw.

Actually are you sure your TCP is doing the right thing? This sounds a classic case of misimplementing TCP retransmission. Is this your own TCP implementation or an unusual one?

You must retransmit *DATA*, not packets and not segments. Otherwise, TCP can deadlock (for example, if two ACKs drop at the same time, one in each direction).

DS

Well, clearly I did not play enough. Just tried to *upload* from a wintel system to the same filezilla server - and guess what, it got stuck in the very same manner..... Apparently all tests I did for wintel - wintel communication have involved only download from that server so far. I guess I should thank that mosquito which bit bit me to wake me up and I did that test.

Dimiter

------------------------------------------------------ Dimiter Popoff Transgalactic Instruments

------------------------------------------------------

It is mine.

I was thinking about that myself. There is nothing which says I need to change the size of the last segment I have retransmitted; and there is no sane reason to do that. In this case, it was a

1460 bytes long segment; the ack was coming for about 1200 bytes (the latter figure is imprecise, don't remember the exact one). But what was making it clear it was the peers fault was the fact that it kept repeating that same ack with a window size of 0 for 30 seconds - and the windows system was OK, the ftp server running there would even accept "ABOR" via the control connection, close the hanging data connection and recover.

Now that I saw this gets locked up the same way when uploading to that server from another tcp (windows'), things are clear. Filezilla (the server software) gets stuck with its buffer (local connection buffer, I guess) full because of some bug - mind you, this does not occur at 10 MbpS (or did I not hold it long enough... I think I did, though) - and poor thing keeps on acking what it can take (< 1 segment) and reporting a 0 window.... Anyway, none of my problem. I was eager to make sure I am going out of tcp porting mode without leaving any (known) issues behind, well, that's done.

Well, segments with a 0 size do not get retransmitted, this is how tcp works, of course. Zero size segments are sent as acks only in reply to non-0 sized segments (apart from syn and fin segments, that is, syn is 0 and fin can be 0 sized).

Thanks for the insight,

Dimiter

------------------------------------------------------ Dimiter Popoff Transgalactic Instruments

------------------------------------------------------

Original message:

Then I'll almost bet your retransmit algorithm is busted. Post a dump of the last few packets before the hang and the hang.

You *cannot* retransmit segments. That will cause TCP to deadlock. You

*must* retransmit data.

Let me guess -- you kept sending it the same data it had already ACKed and it kept telling you it had already ACKed it. See the problem? You are refusing to honor the peer's ACK until it ACKs more data. The peer is refusing to ACK more data until you accept what it has already ACKed. You are in the wrong.

Again, TCP *does* *not* have any segment retransmission or packet retransmission and that WILL NOT work. Really.

Why do you keep refusing to accept the peer's ACK? Why do you keep retransmitting data it has already ACKed, forcing it to ACK it again?

Nonsense, unless I'm misunderstanding you.

DS

There are also some settings on the card itself usually (the device).

In the device settings dialogs, there is a page where hard 100Mb/s is selected or "auto", which is what the whole world typically uses. Hardly anyone hard sets a link to 100Mb/s. Auto negotiation has usually been the rule. Anyway, there are quite a few other settings there as well.

This is not registry settings, it is for the card itself in device manager (there are also other ways to get to this device dialog).

Try examining that. Perhaps you hard set it to 100Mb/s, which was "auto". If so, that is one of your mistakes.

Popeye would say "Ack, ack, ack, ack, ack..."

What a bad re-ACK-tion. :-] every ACK-tion has an equal and opposite re-ACK-tion.

Ah, you want to see some authority stamp. But I am not interested in demonstrating you my credentials nor will I enter any dialogue of that kind.

Anyway, feel free to bet when the "almost" is gone :-).

Uhm, you seem to have your own interpretation of the word "segment". In tcp, it means data of a certain size, encapsulated in a standardized way. The maximum segment size is agreed upon during syn/syn-ack exchange (if not, it is close to 64k, minus the headers etc; in our case, it had been agreed upon 1460 for both sides). There is no other known way of tcp data transmission except in a segment by segment manner, perhaps you will be willing to share your invention :-).

It is not telling it has acked it. It is telling it has a window size of 0 for half a minute.

So who (except you) says that. If the peer reports a window size of 0 forever it is just dead, period.

So why does the same peer fail exactly the same way with a different host with a different tcp implementation (I already posted the details about that in previous messages, please consult them before betting).

Again, it has not acked that data. Only part of them. Retransmission is an integral part of tcp and the receiver *must* be ready to rewind back and accept retransmitted data it has already acked, at any offset. Further, the latest data it has received has precedence over older data at that offset. But all that has been standardised since decades, consult rfc793 for the details.

Anyway, let me see what you think the transmitter must do in that situation: it sends a 1460 bytes segment, receives an ack for 1200 of them with a window size of 0 (repeat that forever).

The obvious meaning is "I am done with that file, cannot take any more but the last 1200 bytes you sent me, please go away". Which I do, after a 30 seconds timeout. An ftp client under windows I tried does the same so I am not the only one who thinks this is sane.

So tell that the implementor of the windows' tcp stack. It behaves exactly as mine in the same situation. I am fine with that - and my receiving side does not get stuck when its buffer size is not a multiple of the segment size, unlike windows. It does get slower when fed data at 100 MbpS and has to "disk allocate and write" (e.g. the file is of unknown size and the preallocated part has been filled up), but it just works. Not so with that ftp server under windows; it appears to have an issue exactly with that. Now whether it is the servers or tcp implementation issue I have no idea, perhaps both. None of my problem, as I said.

Dimiter

------------------------------------------------------ Dimiter Popoff Transgalactic Instruments

------------------------------------------------------

Original message:

How is a tcp implementation supposed to rewind into data it may have already delivered to the client application?

Obviously it is unable to do that. I can say what mine does in that case: it treats any data up to what the application has eaten as "water under the bridge", i.e. newer data cannot take precedece over that; my tcp will ignore(part of)the segment which it receives in duplication of the alredy passed data and will keep on ack-ing the highest offset it has continuosly buffered (it will never rewind its ack position, obviously). Crucially in the context of the failure I saw, it will report the valid window. There is no way it will selectively ack a part of a segment it has received, this would be illegal. So in our case

- it would open the window, take the 1460 byte segment and provide the difference to the 1200 it has already acked and devivered to the application for application consumption; it would ack the 1460 and ignore the first 1200.

- it would not put itself in that position in the first place because it would never partially ack a segment; the whole 1460 byte segment will remain unacked until it can be successfully buffered (but it will handle any retransmissions it gets, it has no way of knowing which of its acks made it to the peer if it is not sending data in this direction, which is the case here).

Dimiter

------------------------------------------------------ Dimiter Popoff Transgalactic Instruments

------------------------------------------------------

Original message:

All else notwithstanding, asking for a packet trace is a reasonable request when trying to diagnost TCP issues.

rick jones

-- The glass is neither half-empty nor half-full. The glass has a leak. The real question is "Can it be patched?" these opinions are mine, all mine; HP might not want them anyway... :) feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

A segment is a chunk of data encapsulated into a packet.

Right. Again, you *CANNOT* retransmit segments. That will cause TCP to deadlock.

Yes, you retransmit *data* (in a new segment).

It is telling you it has a window size of 0 because you ignored its ACK.

TCP implementations do not have some special way to handle an ignored ACK. They treat it like a dropped ACK.

And if a peer keep ignoring an ACK and retransmitting already-ACKed data, it's jsut dead, period. The difference is, the peer's behavior is permitted by TCP but your behavior is not.

Hard to say without seeing packet traces.

I AM TALKING ABOUT THE DATA IT HAS ACKED! That data, the data the peer has ACKed, why do you keep retransmitting it? That forces the peer to:

1) Conclude that its ACK has dropped. (TCP implementations do not distinguish between an "ignored ACK" and a dropped ACK.) 2) Send its ACK again.

The peer is concluding that its packets are not getting through because you are ignoring them. That is why no further progress is being made.

All of that is not the problem. The problem is, and only is, that you are ignoring the ACK.

It's hard to say without seeing exactly how you got there. But the one thing you must not do is refuse to honor the ACK and re-transmit data you know the peer has already ACKed. Of course, the peer will not barf on the data, but it will conclude the ACK was dropped, preventing it from opening up the window.

DS

st

Sure, here it is:

(full version, 20+M), (brief version, 800+k; just beg. and end), - binary log, one needs DPS to be able to process it into text (pppll hanglog.lg >textlog.txt ).

The packets are clipped to 48 bytes in the log, it takes a while until the failure occurs (this time I was lucky and got it almost immediately, hence the log of clipped packets is only 6M or so).

Dimiter

------------------------------------------------------ Dimiter Popoff Transgalactic Instruments

------------------------------------------------------

Now I remember why I like tcpdump so much :)

rick jones

--
the road to hell is paved with business decisions...
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com  but NOT BOTH...

Looks like they include the Ethernet header, so that means you are missing some of the TCP header no? 14 bytes Ethernet header, + 20 bytes IPv4 header plus 20 or more bytes of TCP header (depending on options present) would be 54+ bytes, or 6 bytes more than where you say the packets are clipped. If I am recalling my headers correctly that would suggest it would be missing the window, checksum and urgent pointer fields of the TCP headers and all the options (if present).

rick jones

--
a wide gulf separates "what if" from "if only"
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

Hey, I may be insane but I am not impractical, you know :-) . But I clip at 64, not 48, I am tired enough not to be able to count to 4, it seems, LOL. 4 lines, 16 bytes each....

All is there, of course, the tcp header starts at offset $22 (the flags being the last byte at this line, first two bytes at the lowest line are the tcp window; somewhere at the start you can see the scaling option being set to 3 shifts). Or look at the first few pages, you'll see the beginning of the text exchanges over the control connection (even part of the password - which is old enough not to worry about,

Cheers,

Dimiter

------------------------------------------------------ Dimiter Popoff Transgalactic Instruments

------------------------------------------------------

Original message: