Before I was able to figure out this issue, I have been receiving more than 100 of these alerts in Exchange.
The delivery of a message sent by public folder <publicfolder> has failed.
The non-delivery report has been deleted.
The event ID does not give you a lot of information and it is hard to figure out where this is coming from.
By default, mail-enabled Public Folders are not able to receive NDRs that is why you are seeing these alerts in the logs.
Here are the steps I took to figure out where the emails are being generated:
1. Login to Exchange Management Console, open the Public Folder Management Console and drill down to the public folder mentioned in the event ID.
2. Right-click on the public folder, properties, Mail Flow Settings, Delivery Options and then add an email address where you want all NDRs to be sent (All other emails are also forwarded just an FYI). And DO NOT forget to check the box ‘Deliver message to both forwarding address and mailbox’ so that the regular emails are still being received on the mail-enabled public folder.
Once all settings above are setup, the NDRs will be forwarded to the email specified in the forwarding address field. And the NDRs will give you all info you need as with a regular NDR.
In my case, the emails are being generated externally by our vendor and I had to quarantine the emails on our SPAM firewall so they no longer reach the Exchange Server.
I hope this helps.
I also added this same comment to the forum in http://social.technet.microsoft.com/Forums/exchange/en-US/91d6b279-4090-467d-9880-d4894632e2f1/event-id-2028-the-delivery-of-a-message-sent-by-public-folder-has-failed?forum=exchange2010&prof=required.