FAQ on Usability Testing in Progress

I am writing an FAQ on usability testing for my new team. I want to do a real one, not a marketing-style intro level one that establishes that testing your software is like eating Mom's apple pie. (Wait, that's a weird metaphor.) What I am after is the common objections, the issues people really have about topics like potentially poor recruiting, validity of testing with small numbers, the value of qualitative data at all, the variability and difficulty of analysis of the results, the biases that may go into defining tasks for the test, etc.

If anyone reading has any concerns, however un-PC(!), about doing usability testing on software (or has heard anyone else express doubts about the value), I will try to put them in. Whether I have a great answer or not (some concerns are entirely valid). If I get anything from you, I'll post the results here as a web essay too.

Go on, be tough.