selection delay

I am having an odd problem.

Working with a drawing from an architect that is using ACAD 2007, but has saved
the file for me in 2004 format.

I am using ACAD 2005.

The drawing that is causing the problem opens fine. It has been purged, all of
the colors and linetypes have been set to By Layer.
All of the layer Linetypes have been set to Continuous.
If I open the drawing directly, I can edit it without delay.
Select Move, draw a window or pick objects.

The problem comes when I Xref this drawing in my drawing.
When I choose move (or any other command) if I start a selection window, it
takes 3 to 5 seconds for the first click to be registered.
Then when I make the second click, it takes another 3-5 seconds for the objects
to appear selected.

If I unload the xref, the drawing works as expected without delay, so I know it
is related to something within this drawing, but I cannot figure out what.
Any ideas?

I have also done a full repair/reinstall of acad.
I have also checked my video drivers, but this doesn't make sense because the
problem drawing can be opened and edited directly without problems.

SEPP!

OK - I have done a bit more troubleshooting:
If I freeze all the layers in the xref drawing, the delay is shorter, but still
there, which is still noticeable compared to unloading the entire xref.
It also seems to matter weather or not part of the xref is showing in the
window.

I am totally at a loss here - any pointers...
Anything I should do to the xref drawing before using it as an xref? If so, what
should I look at?