It's a known fact that many email servers are stuffed with powerful fraud filters, which often block automated emails, sent by a PHP script. We have such emails installed to send on miniBB forums. They are profile registration emails, new message notifications, password reminders; as well as on miniBB.com such emails are sent to premium customers, confirming their purchases or restoring an access to their Customers Area.

I wasn't quite sure what happens with forums mailing functions, 'cause claims on them were rather rare, if none; but I was noticing for a long time, that customers often do not get their emails even in bulk or spam folders — like emails weren't sent at all (but they were). I've thought that this is a regular anti-spam filters behaviour, but I wasn't plugged in to investigate *why* does it happen.

Bits of theory

Yes, It would be great, if it would be possible to code a PHP script, which determines, if a supplied email address is valid, and/or if a sent message actually reached its recipient... But this is impossible, and it's just about how email protocols are designed. We couldn't do a lot about it. Only analyzing how a sender server and a recipient server interact with each other, we could do some conclusions.

And basically, they are just two:

A. if we send an email, and it doesn't get back, it means the recipient exists, and the email is delivered;

B. If some error message is returned from a server (amongst the original email), we should analyze it and indicate the problem.

Regarding case "B", it's important to supply "Errors-To" in message headers, then the recipient's server would know, where to return undelivered messages and related error reports.

If you send an email with your regular email client, using a dedicated SMTP server, in most cases the client makes your "Errors-to" address equal to "From" address ("Return-Path" could be used as well instead of "Errors-To" — but it's written by the receiving server and it couldn't be re-written in PHP).

For PHP script, "Errors-to" should be set separately, and in miniBB, it has been introduced since the very first versions. However it is important not just to supply such header, but also put in the proper value for it.

If an email server considers "Errors-to" as a wrong value, then the server will bounce a failed message to an email address supplied by the original sender email address, which belongs to a root configuration of the server (like www@system.domain or similar). In many cases it coulnd't be even associated with the domain, where your forum resides — specially if there are placed many domains on the same server. So in most cases, if it's not set up properly, you have no way to know what happens with emails your miniBB script is sending.

But if "Errors-to" value is set properly, then having an access to such "Errors-To" email address inbox, it's possible to investigate error messages and apply certain actions to forum profiles having wrong email addresses (for example, making them Inactive).

How did it come possible to understand what's wrong

For some time earlier, miniBB project changed a few shared hosting plans, where having a dedicated email address brought some troubles. To share less time for solving problems, my decision was to completely avoid setting up domain-related email addresses for communications with customers. It was too much pain to moderate.

But right now we run miniBB on a dedicated server, and despite it's not dedicated just to miniBB, recently I've took my time to analyze the emails pool, and discovered there are a lot of emails collected under the system email called 'www', which is registered to the original domain of the server (which is not minibb.com) and which actually configured as "Return-Path" by default for all scripts and domains on this server.

I didn't have direct access to this email folder, but I found a way to transfer all emails bounced back for about 10 years, to my email client, then sorted them out by certain phrases from error logs, and got the following view:



Below conclusions are based on analyzing of about 2,500 error messages.

1. CREATE AND MAINTAIN THE "FROM" EMAIL ADDRESS, HAVING AN ORIGINAL DOMAIN WHERE IT'S BEING SENT FROM.

It would be ideal, if we could supply the "From" email as the address which is easy to access and check. I did so for some time, setting up addresses on Gmail or GMX, until I've noticed, that Gmail gives an alarm on such emails, supposing they weren't sent by Gmail — and truly, they were not.

But while Gmail could still pass such messages in — if they were manually white-listed — they couldn't be rather accepted by other servers.

If you host miniBB forums on some domain, create and set up an email address on the same domain, which would be possible for you to check. We use now a virtual address webmaster (at) minibb.com, which emails are just redirected to a regular email address I have possibility to check. And it's now set as "Reply-To" and "Errors-To" email for all messages, so it accepts both manual replies and failure reports.

2. SET UP THE PTR RECORD PROPERLY.

Analyzing error messages, I've concluded that there's quite a simple approach, which a recipient server could use to avoid spam — checking the PTR record for reverse DNS lookup.

This means, a recipient email server checks for all domains associated with the sender's IP address, and if the domain supplied in "From" value is not in this list, an email becomes black-listed and is being returned back to the originated server, with one of the following error messages (in parentheses, I will mention also the amount of emails with this message our Return-Path inbox received):

"554 Bad DNS PTR resource record" (1539)
"550 Reverse DNS verification failed" (174)
"Client host rejected: cannot find your reverse hostname" (28)
"Host (...) refused to talk to me: ... (DNS:NR) or `no rDNS`" (301)

In miniBB case, approximately 81% of all emails bounced because of a failed PTR record verification.

Mostly it was exactly because emails were sent "like FROM" an email address, which domain was not associated with our IP.

So, you should take care of PTR record. You could perform a reverse IP lookup using many tools available online; I've used MXToolbox.

PTR record is set up on the Datacenter end. If you couldn't find your domain by reversing DNS, that's a bad sign, and you should contact your ISP or web host to add appropriate PTR records for your server's IP.

There's no way to setup it on your own, and that's why this thing works as a spam signal: spammers love to substitute fake "From" addresses, but they couldn't register PTR records easily.

Note that this wouldn't work if you move your website too often. Your website should reside on a "static" IP address, and stay there for a longer time. That's also the good sign of a valid domain.

3. SET THE MINIBB OPTION $reply_to_email EQUAL TO THE EMAIL ADDRESS YOU'VE SET UP ABOVE in p.1.

For regular automated forum messages, miniBB will use $reply_to_email as a "From" and "Reply-To" value; and if "Errors-To" email address is not defined, then $reply_to_email would be also used as "Errors-To". The sending script also specifies "Return-Path" equal to "Errors-To", but as said above, this would be in most cases rewritten on the recipient server.

Currently an exception is the "Contact Us" add-on, where it's possible to define different values for the "From", "Reply-to" and "Errors-To" headers (and respectively, there are different settings for this).

If you don't have a chance to install an email address for automated forums messages on your original domain, it's anyway better to opt-in a non-existing email address like 'noreply@minibb.com' rather than setting up a 3rd party address from another domain.

4. VALIDATE EMAILS sent from your forum

If you have set up everything as described, it's time to validate your forum email's headers. Set this option in your setup_options.php:

— it will allow newly registered accounts to get a confirmation email from your forum.

Then try to register a new account on you miniBB forum with an email address you could access from a kind of advanced email client like Mozilla Thunderbird. Note that regular web-based email providers like Gmail often hide all headers and they are not accessible to view.

When viewing a message in Thunderbird, you could select: View -> Headers -> All. In the expanded area, check the following important headers (you may skip a lot of technical "garbage" assigned to each letter):

— "From" value should be equal to the $reply_to_email you specified, and it should be based on your original forum domain;
— "Errors to" value should be equal to the same value as above;
— "Reply to" should be equal to the same value as above (except it could be configured in the "Contact Us" add-on as an email address provided by the letter author);
— "Return-Path", ideally, should come from the same forum domain; but it could be also equal to the original domain of your server, which should be specified in the PTR record.

The similar approach could be applied to any kind of email clients, which support the viewing of all message headers (usually all major desktop apps allow this).

More reasons of undelivered messages

Pulling out more from my error messages, I was able to collect some stats based on the following error reports:

"The email account that you tried to reach does not exist" (67)
"Delivery to the following recipient failed permanently" (62)
"This user doesn't have a yahoo.com account" (67)
"550 No Such User Here", "Sorry, no mailbox here by that name", "User unknown" — and similar messages coming "in reply to RCPT TO command" (118)

— these messages indicate that the recipient was not found, or not available. ~12 % of the total undelivered amount. Unfortunately, there's no way to avoid users putting wrong email addresses, so if you enable sending registration messages, you will always deal with this percent of failures. BUT you could disable such accounts, if you see one of such error message coming in.

***

"Host or domain name not found" (71) — it says all for itself, and it indicates just a fake email address registered. ~2 %.

***

There is also about ~2% of messages (62) with the other nature of errors, like "Try again later", "mail for ... loops back to myself", "Delivery to the following recipient has been delayed" etc.

***

"550 spam message rejected"
"550 High probability of spam"
"Our system has detected that this message is 550-5.7.1 likely unsolicited mail"

etc. — all such error messages contain the 'spam' keyword (28), and they take approx. of ~1% of the total undelivered amount.

***

"Operation timed out" (26) — belongs to a routine recipient error, when there's something wrong with the connection to the server. ~1% of the total undelivered amount.

***

"550 Mail content denied" (15) — I could only expect, that sometimes forum messages like registration emails could be blocked because of the unnatural account registrations, which were applied massively on other forums as well. That's why the spam filter could block such messages with the error provided. These messages take about less than 1% of the total undelivered amount.

***

Looks too difficult for you to get in?..

Just to remind: miniBB has an option to completely disable all script mailings in one touch (except the "Contact Us" add-on where mailing is a required part of the process). For this, set $genEmailDisable=1; in setup_options.php. Avoiding email notification, you'll keep your email pool clean from any problems.

However don't forget that email notifications, even nowadays, is the top approach for keeping your visitors, informing them and getting back to website. The IT world still didn't invent anything more stable and accessible than email addresses.

So, roll them nicely on a proper wheel :)

This topic has an update regarding the recent Google support's notice:

I've experienced the same with my Gmail account: messages sent via PHP from miniBB.com (such as those from the Contact-Us form, or admin notifications I've set up on the forum) were marked as spam and bounced back to my server, not even reaching the Gmail's spam folder. The error message contained the following:

Here, virtualis.lv stands for the name of the email server where miniBB resides, and 92.240.80.80 is both domains IP.

To work it out, the most simple solution is to set up SPF (standing for Sender Policy Framework) by publishing an SPF record at the domain, and include the IP address from which emails are being sent, to this record. I've contacted my DNS provide for this, and most likely, you could do the same.

To check if a sending IP is allowed in the Sender Policy Framework, use one of widely available SPF record checking tools. I've used this one on MXToolBox. In result when checking out your domain, you should see something similar to what's shown on the screenshot below (click it to enlarge).



As soon as the similar SPF record is added to your domain among the IP address which sends emails, they will pass on Gmail side. At least it absolutely works for me now.