Hi Amit,
A) try to maintain EXT_ITS (Vendor portal URL) attribute within organizational structure
B) I do not thing you need to run the report RSALERTPROC - deadlines are being watched by
standard workflow reports that is why the notification gets issued immediately in case the deadline is reached in moment of posting the document. Alerting in case of past submission deadline is a task for OSS or try to find a SAP NOTE first.
Are you sure that BAdI gets not called? I think it is called in background using WF-BATCH user. Try an endless loop within implementation. Same with the SMARTFORM BBP_OUT_EXCEPTION - it should be called.
In case you select "notification" an e-mail should be send. Please check trx. SOST.
Please check wiki - it would be helpful.
Best regards,
Tomas