local farm makes tester slower


I'm just using a 2 computer LAN.  I have a xeon 2x 2697 machine with 96GB ram and then i have  xeon 2x 5650 with 12GB memory.

When I use the local agents it makes the tester around 3x slower.  My main computer is the faster one.  Is there any way to speed it up?