Yes you can. Though not through the Broadband modem.
You have two options.
1. Purchase a hardware based recoder that records on normal audio tapes it will be much cheaper and simpler.
2. By a REAL modem (not the one that is emulated on top of a DSP chip, ost internal modems are like that). If the modem can detect reversal then there are chances then it can detect if the phone has been picked up. Such modems usually come with softwares that you can script for this purpose.