The Avalon Contact Form is not working

[ThePythonicCow]

Posted by Miller (here)
In fact it might be worth the mods having a skim through this as well, in case similar starts happening here.

To minimize the risk of a website takedown, whether by script kiddies or satanists:

• One has to register the URL for one's website (e.g. "projectavalon.net") with a separate, reliable, DNS provider.
• One has to keep offsite backups and be prepared to move one's entire site to a different hosting service w/o warning.
• One has to have administrative staff who hangs around when the excrement impacts the oscillating blades.
• One has to keep a trusted https certificate.
• One has to have a long term supportive membership

So far Avalon is batting 1000%.

[ThePythonicCow]

.
Well, well ... thanks to my log file examination skills, and the talents of one other Avalon staff member (not sure if he wants to be named ... he or Bill are Welcome to let him take a bow) ... this problem has changed.

It turns out that another problem is that we (the Avalon server software) were trying to send too much, too fast, to the Gmail servers, at two times each day.

(All following times are GMT)

At 1 minute past midnight, we start sending out Happy Birthday wishes. On days with lots of Birthdays, this can take 2 or 3 minutes, and apparently sets off some alarm in the Gmail server, which rejects the unlucky members who ended up at the end of the line. Over the last two months, we tried to send out 1749 wishes, of which Gmail refused 136. We send out about 1 per second, and all wishes we try to send out in the 00:01 minute make it, but some where in the 00:02 minute, any remaining Birthday wishes we try to send are rejected by the Gmail server.

The same thing happens at 30 minutes past midnight, when we send out updates to members who want to be notified of posts to their subscribed threads or replies to their posts. Just as with Birthdays ... on light days, all the updates make it, but on busy days, the unfortunate members who ended up at the end of the line don't get their update. About 336 out of 2772 update messages failed this way in the last two months

The only New Member email messages (just 3 of them) that we lost just happen to come from prospective members trying to sign up right at the same time as one of these traffic jams.

Meanwhile, another Avalon staff member has tweaked the Avalon account page on Gmail, and re-enabled the infamous "noreply" user that our PHP code credits as sending these automated messages, so, for now anyway, that's once again not a problem.

=== end of facts === beginning of speculation ===

Apparently our "noreply" email bot sender got the credit blame, by Gmail for these frequent (once every 4 or 5 days on average) traffic jams, and this is why Gmail banned "noreply" for a while from sending email from the Avalon server to the Gmail server.

So now I will be looking at ways to spread out our Birthday and update email message sending more evenly over the day, so as not to annoy Gmail.

[John Hilton]

As I'm no expert, I asked my ISP to comment:

"Let's start with the bit that's not true.... The "Inter"net is made up of millions of servers transacting data between them day in and day out, any individual server can connect to another server to send email, it's how the internet works (I personally believe nobody would want to be responsible for all that email and the bandwidth it would require!!). So there is no "It has to go through one of the big boys".

What is true is that PHP mail is notorious for being spammy, I can write a script that will send a thousand emails a minute, upload it to 200 web servers and spam someone completely (I hasten to point out I'd get banned pretty quick and it's not the type of person I am!!), so if you can you should always SMTP when sending mail through the web server (so it looks like it's come through a mail client). To send SMTP through our servers you have to authorise with your email account username and password (so we know who you are and can rate limit spammers - we do rate limit PHP through other means!), when your authenticated mail hits our SMTP server (Dovecot) we open an outbound connection to the MX record of that mail server to deliver it (a digital knock on the door so to speak).

What has changed in the last few weeks is the the big boys have asked that you have the relevant records on your domain SPF/DKIM/DMARC that they'll take more notice of if you're sending lots of email, these records both show who's authorised to send as your domain, which servers (which could be any IP address of any server anywhere) and who to send spam reports to.

So yes, the SMTP servers of the big boys only accept first point of contact SMTP traffic from their paying customers, exactly the same as we do, but the connection after that can deliver to any email server anywhere."

[ThePythonicCow]

Posted by John Hilton (here)
As I'm no expert, I asked my ISP to comment:

"Let's start with the bit that's not true.... The "Inter"net is made up of millions of servers transacting data between them day in and day out, any individual server can connect to another server to send email, it's how the internet works (I personally believe nobody would want to be responsible for all that email and the bandwidth it would require!!). So there is no "It has to go through one of the big boys".

Absolutely - if any of our current, or potential new, members choose to setup their own SMTP email server, our Avalon server could send them email directly, without going through any of the big boys.

Indeed, that's how I did things for a while, back in the 1990's, when I ran my own email server and email to me went directly to my server, without any reliance on any intermediate "big boy" email service provider.

However ... how many of our present or potential Avalon members run their own email server these days?

Off hand, I doubt there are any that do. Nowadays, we all get our email feeds from or via one of the big boys. So we have to play by their rules, and pay their admission fee. Fortunately, there are "big boy" email players who are more suitable for Avalon's needs than Gmail.

[ThePythonicCow]

Posted by John Hilton (here)
To send SMTP through our servers you have to authorise with your email account username and password (so we know who you are and can rate limit spammers - we do rate limit PHP through other means!), when your authenticated mail hits our SMTP server (Dovecot) we open an outbound connection to the MX record of that mail server to deliver it (a digital knock on the door so to speak).

Ah - yes - we could send email more directly to which ever big boy server each email address referred to.

Here are the top 31 email domains of our Avalon members, from some few year old data I happened to have right at hand. This shows how many email addresses were in each domain, by number and cumulative percentage. So doing this would have reduced the number of messages we sent directly to gmail down to 39% of the total, while introducing more email administrative work on Avalon. At the time of that data, there some 1329 domains represented in our members email addresses.

Yes, I would expect that an ISP would do exactly that. I would not expect most small websites to do that. I did indeed over simplify my "big boy" comment a bit ... though that doesn't change how I would expect most websites the size of Avalon or smaller, to operate.

4699 39.0% gmail.com
6263 51.9% yahoo.com
7705 63.9% hotmail.com
7986 66.2% aol.com
8169 67.8% hotmail.co.uk
8339 69.2% yahoo.co.uk
8467 70.2% comcast.net
8590 71.2% live.com
8684 72.0% msn.com
8763 72.7% outlook.com
8841 73.3% googlemail.com
8911 73.9% yahoo.ca
8970 74.4% btinternet.com
9029 74.9% ymail.com
9083 75.3% live.co.uk
9135 75.8% sbcglobal.net
9184 76.2% me.com
9228 76.5% icloud.com
9271 76.9% yahoo.com.au
9312 77.2% protonmail.com
9351 77.6% bigpond.com
9389 77.9% hushmail.com
9425 78.2% mac.com
9461 78.5% verizon.net
9496 78.8% mail.com
9530 79.0% mail.ru
9562 79.3% cox.net
9590 79.5% earthlink.net
9618 79.8% gmx.com
9644 80.0% ntlworld.com

[John Hilton]

As I wrote, I'm no expert, which is why I asked my ISP how my email works. AFAIK I can send messages to anyone.

[ThePythonicCow]

Posted by John Hilton (here)
why I asked my ISP how my email works.

Yup - ISP's are part of the internet infrastructure. Part of their job is moving packets, whether email or otherwise, toward wherever they are destined. For email, that's going to be wherever the MX record in the destination IPs DNS record points. They will expect to connect to an SMTP server there, so they can pass the email along.

For the rest of us, who's main job is not operating as part of the internet infrastructure, but rather some other reason to be on the Internet, we typically send and receive our Internet traffic through our ISP, and we access our email traffic either through typical email client software on our PC, mobile or browser, or for websites such as Avalon, through an SMTP client that is connected to either to our ISP's SMTP server or, via our ISP, to some other email service provider on the Internet.

(Sufficiently large companies, such as Amazon, Google and Cloudflare, usually do all of the above internally, and only connect to the rest of the Internet at whichever one of their sites is closest to the end point for that connection.)

So .. the answer one gets from an ISP to such email questions is not the same as the answer one gets from a backroom geek (e.g. myself) at some discussion forum (e.g. Avalon).

Posted by John Hilton (here)
AFAIK I can send messages to anyone.

Yup - if your email can't get to anyone via a valid MX record in the global DNS records, then your email is broken somehow.

[ThePythonicCow]

Posted by ThePythonicCow (here)

Posted by John Hilton (here)
I create the form in html and that calls up a PHP routine, which I wrote, using method "Post".

We do similar - using Dovecot (an SMTP server) rather than PHP's builtin email handler, to send outgoing email on its way.

Belated technical correction: We use Sendmail, not Dovecot, to send email out to Gmail, and on to its final destiny.

[ThePythonicCow]

Posted by ThePythonicCow (here)
Apparently our "noreply" email bot sender got the credit blame, by Gmail for these frequent (once every 4 or 5 days on average) traffic jams, and this is why Gmail banned "noreply" for a while from sending email from the Avalon server to the Gmail server.

It might well be that the reason that the Honorable Mr "noreply" got the blame for all that Gmail traffic from our Avalon server is that our server's secure SMTP login to the Gmail server (done each time our server wants to send out an email message) happens to login to that Gmail server using the user name noreply@projectavalon.net.

We use noreply@projectavalon.net in several places, but this particular use (for the Avalon server to login into the Gmail server to send email) is configured on vBulletin's AdminCP page:

• https://projectavalon.net/forum4/admincp

in the settings option:

• Settings -> Options -> Email Options

(The above observation means little now, as we've resolved this problem with Gmail, and are soon moving to Fastmail anyway.)

[ThePythonicCow]

Posted by ThePythonicCow (here)
Belated technical correction: We use Sendmail, not Dovecot, to send email out to Gmail, and on to its final destiny.

Correction of correction:

While the venerable sendmail program is installed on the Avalon server, as are also email programs Postfix and Dovecot, those are not what we use to send mail. Rather we use one or another of a couple of vBulletin and/or PHP routines. Just which one(s) is a matter of current investigation by myself, in preparation for the Gmail -> Fastmail move.

[mountain_jim]

Thank you so much for your Avalon service, Paul.

[ThePythonicCow]

Posted by ThePythonicCow (here)

While the venerable sendmail program is installed on the Avalon server, as are also email programs Postfix and Dovecot, those are not what we use to send mail.

Yesterday, I disabled all three of sendmail, postfix, and dovecot, which were the three major email handling programs installed on our Avalon server.

We were not using any of them, and even if we were, the postfix+dovecot combination breaks sendmail, so it made no sense to have all three.

Rather builtin PHP code handles all our forum server's outgoing email, using email configuration settings (pointing to Gmail's SMTP server, soon to be pointing to Fastmail's SMTP server) that are set in the vBulletin Admin Control Panel.

When I get bored, and am finished trying to break the forum's email in this big Gmail->Fastmail shift (for incoming email), I will likely deinstall sendmail, postfix and dovecot entirely. They are just wasting space on our server's disk now.

===

Meanwhile, I expect that tomorrow (Sunday), Gmail will end the 2+ day delay they imposed on me, and then I will be able to brew a stiff cup of coffee and do the actual Big Flip, redirecting our incoming email to our Fastmail accounts.