When frameserving from VirtualDUB, if you get an error message saying that VDR files are not supported then try this...

Go to your VirtualDUB folder and open AuxSetup.exe. Click Install Handler and you will have to click OK twice. You can then try to frameserve again (your machine may need to be rebooted). Then for the best chance of getting it to work, go to Option - Environmental Setting - VFAPI Plug-in and increase the VFW readers priority (Right click and click Higher Priority).

This should fix your problem. If it doesn't, you can visit out discussion forums and ask there.