Reminded me of a problem I had been debugging ages ago. The initial complaint was that the particular file transfer through a firewall was hanging. Further tests: it hangs at the same place.
More testing: not just this file, some other file transfers from unrelated places as well - and always at the same places within the file (but different across different files).
Take the tcpdump on both sides - always the same segment of data within each of the file transfers does not make it through the PIX. Take the offending segment and convert it into a small file of its own - this file is impossible to download through, gets dropped.
Needless to say, this all was observable only on that particular setup - not in the lab.
Finally I noticed that the CRC error counter on the inbound interface increments by one every time I try to push through the offending small file.
Replacing the Ethernet cable connecting that interface had solved the problem with all of the "hanging" transfers.
We did not do any further research into the root cause (the user did not want to put back the previous cable), but the working theory was that the initial cable was made bad, but not bad enough to not work at all - and the fault only showed up on particular sequences of data.
Given that neither 10BaseT nor 100BaseT used scrambling, this seemed plausible enough of a theory, but was quite fascinating nonetheless.
1) Problem report was: "I can login to remote host fine, but my interactive session hangs when I type 'ls -l'." Some diagnosis indicated the hanging only occurred when packets were above a certain size. This only happened when a command was run that had a large amount of streaming output. Turned out our provider's frame relay network had become misconfigured. Had to reduce the MTU at both ends as a workaround until I was able to convince our provider that something was wrong with their frame relay.
2) My home Linux box unable to talk to a Verizon pager gateway, but my Mac was fine. Turned out Verizon's firewall was dropping packets that had the ECN bit set. Had to disable ECN on the Linux box.
3) A Solaris host that was sporadically reachable over the LAN. It could always connect to other hosts, but orher hosts could only sometimes reach it. Turns out you can disable ARP replies on Solaris by deleting the "publish" entry for the box's own MAC addr from its ARP table. And someone or something had done that by accident. So the box was only reachable on the LAN till its MAC timed out of the other hosts' ARP caches. However, if the host in question first initiated a connnection to another host, the destination host would then learn its MAC again.
Troubleshooting the issue similar to (1) was one of my interview questions for a while - though with HTTP. It's a treasure trove of discussion at very different levels.
Another "fun" (and hard to debug for the first time) is the problems triggered by duplex mismatch - though the asymmetric ~1/10 difference in performance quickly becomes a signature once you met it once or twice. Also good anchor where you can go into many types of discussions.
The very similar to (3) I've seen manifested in IPv6 - due to Neighbor Discovery working over multicast, and potentially asymmetric ways of multicast propagating within the L2 segment (different L2 switches on two sides, etc.)
But very interesting to learn this asymmetricity in Solaris stack, thanks!
I had the same issue, file transfer from Windows client to a file server. It always stopped if the file was larger than a few MBs. It turned out the ethernet patch cable was damaged (CRC error counting), replacing it solved the issue.
Another spooky issue was Skype on my iPad stopped working every 10 days. It turned out that I accidently set the WiFi switch to a static IPv4 address on a LAN with DHCP server (and I powered off the WiFi switch every night). So every few days another device on the network got the dynamic IP address (from DHCP) that I statically set also for the WiFi switch, ups.
Sadly no. I told my co-worker about the cable and he told me that he had the same issue with a patch-cable too. He had damaged the cable with his office chair rolls and replaced the cable. So probably he stored it away, and I used his old damaged cable for months until I figured out my upload issue.
Seeing as the error happened on the same place each time, it might sound like you were getting errors that didn't _always_ get caught by the checksum algorithm (i.e the error "happened" to match the checksum) (maybe).
Sounds like another good reason to do separate checksumming of files after download (read: sha1/md5).
Oh yes, crypto checksum is always useful. Another bug (also eons ago) was where something was trying to patch the TCP options, with a very lax assumption about the packet - as a result with some way fragmentation it would think it was patching the option, whereas it was smashing the payload... Of course, after patching the "TCP option" it was adjusting the TCP checksum, or, rather, the place where it thought the TCP checksum was - which in that case was again within the payload.
Needless to say the whole operation made the corruption checksum-neutral from the viewpoint of the "real" checksum - so it was not caught by the TCP checksum, as a result the files were sometimes silently corrupted.
Took a lot of work to catch and debug, the symptom was that Gentoo's packages appeared to be corrupt and failed the SHA1 checksum check.
So, indeed, cryptographic checksumming of files is a good thing.
Good conclusion in that article! Sometimes (actually quite often in practice) neither of the parties does anything blatantly wrong, but just bends the assumptions in slightly different directions, together becoming a trigger for a failure.
More testing: not just this file, some other file transfers from unrelated places as well - and always at the same places within the file (but different across different files).
Take the tcpdump on both sides - always the same segment of data within each of the file transfers does not make it through the PIX. Take the offending segment and convert it into a small file of its own - this file is impossible to download through, gets dropped.
Needless to say, this all was observable only on that particular setup - not in the lab.
Finally I noticed that the CRC error counter on the inbound interface increments by one every time I try to push through the offending small file.
Replacing the Ethernet cable connecting that interface had solved the problem with all of the "hanging" transfers.
We did not do any further research into the root cause (the user did not want to put back the previous cable), but the working theory was that the initial cable was made bad, but not bad enough to not work at all - and the fault only showed up on particular sequences of data.
Given that neither 10BaseT nor 100BaseT used scrambling, this seemed plausible enough of a theory, but was quite fascinating nonetheless.