Many anti-virus vendors take a black list approach: a huge list of 'naughty' files is drawn up and those naughty files are prevented from running on your machine.

The problem with this is to make the list the naughty files need to be found and analysed, then the list needs to be updated and sent all over the world. So, the easy attack is to make it difficult to find and analyse the files. For example the hacker defender project used to sell a customised version of their trojan. That way it would be difficult to find (if used in a targeted attack) and the analysis would only affect that version keeping the other versions undetected. Another example are the thousands of variants some malware has e.g. NetSky, MyDoom and Mytob.

A much better way of doing this would be to use a whitelist. Now, making a whitelist of allowed programs requires too much customisation, so a white list of behaviours is used instead. Many AV products are pretty bad at this. To test your products functionality, some ISC handlers have created Spycar. This provides some binaries which perform benign versions of naughty behaviours to see if your AV can pick them up. The name is a reference to the naughty file test suite EICAR.

While you're at it, check out the free Windows behaviour-based protection tool, WinPooch.