Help Interpreting traceroute output

[trlists at clayst.com (trlists at clayst.com)]

On 18 Oct 2004 dsr at tao.merseine.nu wrote:

> Remember that traceroute is not a performance measurement tool.
> It's a connectivity checker. 

Fair enough.  But it is sometimes a simple way to get a rough idea of 
where a performance problem lies, no?

> - ICMP is not handled the same way as UDP which is not the same
>   as TCP 
> 
> - your packet size will be different
> 
> - your routing is not guaranteed

OK.  But we are looking for a serious bottleneck here that is breaking 
an application.  I'm guessing that the differences in performance 
within a normal range of packet sizes will not be so great that the 
traceroute output is for that reason alone useless in finding an 
obvious bottleneck.  And the routing is usually the same from try to 
try.

I hadn't thought about the differences in packet types -- and I'm not a 
router expert so please bear with me.  Are the routing algorithms 
typically different for the different packet types, such that a notable 
bottleneck found via traceroute for UDP would routinely not apply to 
ICMP or TCP packets?

> routers may handle traceroutes and/or pings preferentially, may
> artificially restrict them, may route them differently, store
> them in different buffers, all sorts of things.

Again, fair enough, but will this affect the gross results or just less 
important details?

> Performance testing is best done between the points in question
> with the protocol in question using a payload as similar to
> "typical" as possible.

In this case we are trying to get a rough feel for why an installer 
can't retrieve data from a mirror at any reasonable rate.  The data is 
sent via HTTP POST and hence is coming as TCP packets.  I don't know 
the packet size (I didn't write the installer, I'm just looking for 
what's slowing it down) but I can probably find out.  Beyond that, what 
is a good method for doing this kind of basic performance testing 
without overdoing the level of precision?

For example, I can write some quick PHP code that opens a connection to 
port 80 on the relevant host and downloads whatever web page is sitting 
there enough times to be able to measure the throughput -- though in 
doing that we are measuring startup time for each connection as well.  
I can do all kinds of other things similar to that, but I probably 
can't get something to run at the other end that is going to provide 
low-level responses on a port of my choosing, adjust packet size, etc.  
It would be overkill for the problem I'm trying to solve, my client 
would balk at the time involved, and the admins at the other end aren't 
going to let that happen on their server anyway.

More to the point, we are pretty sure we already know what end-to-end 
testing will tell us -- the link is slow.  The question is why and 
where.  Is there something other than tranceroute that can be helpful 
in answering questions like that?

Any suggestions welcome ... thanks.

--
Tom