How Gmail's blocking of email is changing Avalon

[ThePythonicCow]

.
Over one third of Avalon members use Gmail, a major, free (for individual use) email service.

Perhaps the biggest problem I've had to deal with, since returning to active duty as a backroom Avalon administrator these last six months, is Gmail blocking Avalon's email.

My initial suspicions, as a long standing conspiracy theorist, were that "Avalonians were thinking too much, so getting censored."

While that may well be (and I'm a big fan of "thinking too much"; that's why I'm here), there's something else going on, that has broader, albeit more routine, impact.

As we well know, Gmail, and it's associate corporations such as Google, Youtube and Alphabet, are primarily funded by advertising (publicly at least ... their dark budget is ... well ... dark.)

So what I'm seeing is that Gmail distinguishes between email on a couple of metrics:

• Is someone paying for it, such as for ads or marketing? If so, that's good email.
  
• Does it seem to be the individually composed email from a human that Gmail can track (for its dark intelligence buddies) and that attracts a massive audience (for its marketing buddies)? If so, that's good email.

On the other hand, unpaid for bot generated email, once identified, is routinely dropped by gmail without further notice.

I understand that way of thinking well. It's the same sort of approach (with different goals) that I took with the firewall I developed and ran here on Avalon for years. A majority of the incoming traffic to projectavalon.net was dropped; sometimes over a 1000 IP addresses were on the "drop" list, an automatically adapting list of "bad" IP's. Packets from that day's "bad" IP's were dropped so fast that the main Avalon web server never even saw them. Whenever I noticed a new kind of Internet attack on, or abuse of, the Avalon web server, I'd train my firewall to identify that pattern and start blocking IP's exhibiting that pattern as well.

Gmail and other big websites do the same thing ... on a very different scale, and with quite different goals. It's one old fart computer geek whose hobby is writing strange software, versus billions of dollars of computer infrastructure world wide running the latest AI, both tracking and identifying patterns in the traffic worth acting on.

Here's how Avalon has been and will be adapting to this environment.

Avalon's response to Gmail's AI bots has and will effect Avalon members in three ways.

1. We've moved our forum email that forum members, applicants, and staff use in various ways, from Gmail, to Fastmail. The main effect of this email migration has been some email outages, when this migration didn't go as smoothly as we liked.
   
2. The various email messages that our Avalon server generates are being refined (under the covers) to look more individual, and to avoid the patterns that Gmail's bot detectors can easily detect. Unless I make a mistake, this won't effect Avalon's members.
   
3. The most controversial adaption: I intend (if Bill consents) to restrict heavy use of email notifications (of posts to subscribed threads and similar) by members who are using gmail accounts. This will effect less than a handful of users, who can either switch their Avalon email address to some provider other than gmail, or not get their daily bucket full of notifications.

Bill requested, encouraged and fully participated in item (1) above. Item (2) is just "invisible goodness" at his level of oversight.

Bill has not heard of item (3) above, as I just formulated it as I awoke this morning. I've yet to have my first cup of coffee. (Old geeks prefer to work the night shift ... fewer interruptions.)

===

P.S. -- The move from Gmail to Fastmail came just in time. Two weeks ago, Gmail apparently started blocking most traffic coming from Avalon, even if it came via another email service such as Fastmail. I'm still digging my way out of that hole.

[arwen]

Thanks Paul. For me, I have a very low use gmail account which I happen to have subscribed to Avalon with years ago when it was still fine, and I have no problem switching my Avalon subscription to another e-mail account I have. I have already dropped gmail for other things since they are requiring login authentication (provide phone number etc) almost every second time I log in, so I am already phasing them out. (Also I personally do not do subscription notifications, cant stand all the e-mail).

Sound like feasible solutions since gmail is definitely not an option anymore.

[ExomatrixTV]

• 13 Best Gmail Alternatives: Their Pros, Cons, and Use Cases
• 10 Best Alternatives to Gmail in 2024 (Private & Secure)
• Free Gmail Alternatives of 2024 - Our Top 10 Picks

I know it is super lame of me still using gmail ... I will soon register & activate a new free email account

Gmail has one of the best "automated" spam filter, but it may have also way too many "censor" filters too.

cheers,
John 🦜🦋🌳

[Kryztian]

Posted by ExomatrixTV (here)

• 13 Best Gmail Alternatives: Their Pros, Cons, and Use Cases
• 10 BEST Gmail Alternatives (Free Email Better Than Gmail)
• Free Gmail Alternatives of 2024 - Our Top 10 Picks

I know it is super lame of me still using gmail ... I will soon register & activate a new free email account

Gmail has one of the best "automated" spam filter, but it may have also to many "censor" filters too.

cheers,
John 🦜🦋🌳

I dropped gmail in 2018. It was such a great service, but the company is creepy.

At first I used Yandex (the Russian equivalent of Yahoo) and it was great except America Online would not accept email from them. I imagine that this problem might get worse.

Now a Protonmail User - very happy with them, however, because of all the security issues, it is slow to upload email from them, especially from my phone.

[shaberon]

Posted by arwen (here)
I have already dropped gmail for other things since they are requiring login authentication (provide phone number etc) almost every second time I log in, so I am already phasing them out.

I tripped into this by doing something that seemed normal when the service started--i. e., log in from another machine, somewhere else--and Gmail was just like "No sir, go away".

Defeated its own purpose.

I am pretty sick of authentication, and, well, passwords actually.

On the other hand I am pretty sick of spam, too, so I was kind of grateful for Gmail to start diverting a lot of things I did not ask for. I don't ask for Avalon notifications, or anything additional from what might be sent out once in a while to everyone, which seems to be fine.

I can understand it's not top-notch for the server to generate loads of messages that Gmail is going to be picky about, and yes, I find a lot of their ways to be irritating.

[ThePythonicCow]

Posted by ThePythonicCow (here)
On the other hand, unpaid for bot generated email, once identified, is routinely dropped by gmail without further notice

Wrong - correction - update ...

Gmail started enforcing an SMTP RFC rule on our email messages sent to @gmail accounts.

To quote their error message:

This message is not RFC 5322 compliant. There are multiple Date 550-5.7.1 headers.

Over the last week or two, I've sent out our Avalon server's email three different ways. Only the third way, that I started using a half a day ago, actually returned that error message, which they got on each of dozens of messages when they tried to pass the message further, onto Gmail's server.

So Gmail did give "notice" on the failure ... but getting that notice all the way to an Avalon administrator is a hit or miss process.

In any case, now that I know that Gmail is using that error to discard all email we've been sending the last week or two (at least), I could fix that error, which I have done. Now the email our Avalon server sends out only has one Date line in the header. Time now for me to run my @gmail test, to see if there's some other impediment.

Like much networking stuff, the RFC's for SMTP transactions and formats are sufficiently obtuse that Gmail can play this game forever ... a kind of "law fair" between one party with a legal budget orders of magnitude larger than the other party.

[grapevine]

So as a gmail user I wouldnt know whether avalon emails are getting through or not (altho I'm not a subscriber to notifications). The only concern I have would be if it interfered with any donations I make through Paypal. Can you confirm this is a separate matter and wont happen Paul?

[ThePythonicCow]

Posted by Miller (here)
So as a gmail user I wouldnt know whether avalon emails are getting through or not (altho I'm not a subscriber to notifications).

I'm not able to read your mind ...what do you have in mind here: email messages to or from whom, using avalon in what sort of intermediate capacity by what mechanism ...?

Posted by Miller (here)
The only concern I have would be if it interfered with any donations I make through Paypal. Can you confirm this is a separate matter and wont happen Paul?

I am 100% confident that if I broke the ability for Bill to receive Paypal donations to the Avalon forum, then either I'd be tasked to fix it, immediately, or I'd be fired and Bill would find some else to fix it. That's an essential function for maintaining this forum.

[ThePythonicCow]

... after sleeping on this a bit ... there's a change I should make to what capabilities this Avalon forum offers its members.

Disabling the "Send Email" link for forum members (other than "staff'):

Any member can, if they wish, enable a "Send Email" option on the forum that lets other members send them email.

I'm thinking I should disable that feature, for forum members other than "staff".

If member "Alice" enables this option, then an additional "Send Email" link lights up in her user profile, that's visible to other members. Some other member "Bob" can click on Alice's "Send Email" link in her profile, and compose and submit an email message, that the Avalon forum software will then package up and send out into the world's email network, addressed to the email address that the forum has on file for Alice. The email that Alice receives has additional text automatically added by the forum software, surrounding what Bob actually wrote, explaining that that email is coming from the Avalon forum, and that Bob asked the forum software to send the included message to her, Alice.

This facility opens up the possibility for abuse of the Avalon forum's reputation on the world's email network, and also opens up the possibility for confusions as to who is who and who has access to who else's email. I don't see any way for this facility to avoid those risks and confusions.

Rather it makes more sense to me that the forum and its members trust the "staff" at Avalon, its moderators, administrators and founder Bill. This "staff" has access to the email address being used by every member of this forum, and can send email to any member using that address.

So the "Send Email" link would remain visible, but only to "staff". If enabled by any member "Alice", it would provide a convenient way for any forum "staff" member to send email to "Alice", from the Avalon forum server, while that "staff" member was interacting with the forum web software (just as "staff" can do now). But other members (such as "Bob" in my above example) would lose that "Send Mail" link. Perhaps as an option "Bob" might choose to ask some "staff" member to send "Alice" an email message, and perhaps the "staff" member might agree to do so. Even if Alice doesn't enable this feature, any forum administrator (moderator too ... I don't know) can look up Alice's email address in the forum database and send Alice an email if they want to.

Reputation and identify ... who's who, who can read who's email, who can send email from which server on the Web, and who can be trusted to not issue spam ... these increasingly matter, and should be transparent and trusted, in the world of email.

If you're a forum member who uses this "Send Email" feature to send email to other members (those members anyway who have enabled their "Send Email" link for others to use), and if you'd miss this feature if I took it away ... let us know ... perhaps I'm missing a valid use case that I can or should continue to keep working ... or perhaps I should have had my morning cup of coffee before posting this

[grapevine]

Posted by ThePythonicCow (here)

Posted by Miller (here)
So as a gmail user I wouldnt know whether avalon emails are getting through or not (altho I'm not a subscriber to notifications).

I'm not able to read your mind ...what do you have in mind here: email messages to or from whom, using avalon in what sort of intermediate capacity by what mechanism ...?

Posted by Miller (here)
The only concern I have would be if it interfered with any donations I make through Paypal. Can you confirm this is a separate matter and wont happen Paul?

I am 100% confident that if I broke the ability for Bill to receive Paypal donations to the Avalon forum, then either I'd be tasked to fix it, immediately, or I'd be fired and Bill would find some else to fix it. That's an essential function for maintaining this forum.

I'm not a mind reader either Paul. To my discredit, perhaps, I don't know anything about the workings of the forum and am happy unless I can't log in. What I meant was that I do use gmail and if anyone sent me an email via PA and I didn't get it, how would I know? .

I gather from other posts that I don't need to change my gmail account as I don't use the email facility, which is a relief.


Thanks for the confirmation about donations.

[Bill Ryan]

Posted by ThePythonicCow (here)

Posted by Miller (here)
The only concern I have would be if it interfered with any donations I make through Paypal. Can you confirm this is a separate matter and wont happen Paul?

I am 100% confident that if I broke the ability for Bill to receive Paypal donations to the Avalon forum, then either I'd be tasked to fix it, immediately, or I'd be fired and Bill would find some else to fix it. That's an essential function for maintaining this forum.

Wow. I've just myself fallen victim to being unaware of a possible new error simply because of the absence of notifications.

I've not been receiving any notifications of PayPal donations for a while now. I never even realized, because they weren't there to be noticed.

The notifications are sent to donation@projectavalon.net (which is the Avalon PayPal address), and either

1. The PayPal notifications not being received (or sent!) at all, or
2. They're not being relayed to my bill@projectavalon.net address where everything used to be consolidated.

Looking at the PayPal account itself, it seems that the relatively small number of recurring monthly donations are continuing to come in (thank you...!!! ), but there are no one-off donations listed there since 22 April.

Maybe there haven't been any! But usually (based on previous stats) we'd have likely received something, even a small kindness of $10, in a 20 day period like that (22 April—today). So (@Paul — sorry!! ), I have to find myself suspecting another new problem.

Of course, this can be easily tested if someone might be kind enough to send a one-off donation of (e.g.) $0.50, or less, using the orange donation button at the top right of any Avalon page. Paul, maybe you could try this when you see this post??

[grapevine]

[QUOTE=Bill Ryan;1613151]
Of course, this can be easily tested if someone might be kind enough to send a one-off donation of (e.g.) $0.50, or less, using the orange donation button at the top right of any Avalon page. Paul, maybe you could try this when you see this post??[/QUOTE

Done. Seemed to go OK my end and have received a Paypal notification, but I didn't return to the forum automatically afterwards; I don't know whether that's a glitch

(soz mucked up the formatting here . . .)

[Michi]

Posted by Bill Ryan (here)
Of course, this can be easily tested if someone might be kind enough to send a one-off donation of (e.g.) $0.50, or less, using the orange donation button at the top right of any Avalon page. Paul, maybe you could try this when you see this post??

Just did one (but increased to $50 ), using the orange button, labelled "How to Donate"

[ThePythonicCow]

As of a couple of minutes ago, I see, in the last 4 hours:

• 4 hours ago - a recurring donation of $20
• 40 minutes ago - a one-time donation of $10
• 12 minutes ago - a one-time donation of $50
• 12 minutes ago - a one-time donation of $10 annotated "test test"

If you see them too, Bill, then good.

If not, that's because I don't have the proper filing of incoming Avalon email on our new Fastmail account properly sorted yet.

[Bill Ryan]

Posted by Michi (here)
Just did one (but increased to $50 ), using the orange button, labelled "How to Donate"

Posted by Miller (here)
Done. Seemed to go OK my end and have received a Paypal notification, but I didn't return to the forum automatically afterwards; I don't know whether that's a glitch

Thank you both SO SO much! I logged into the PayPal a/c, and immediately saw the new donations right there. But I never received any notification.

It's VERY useful to know both of these things — (a) that donations are working just fine, but (b) the notifications seem to be missing.

[ThePythonicCow]

Posted by Miller (here)
. What I meant was that I do use gmail and if anyone sent me an email via PA and I didn't get it, how would I know? .

You would not (yet!) know if an Avalon member or some automatic software on Avalon forum had sent your gmail account email in the last 2 or 3 weeks. Since you apparently have not enabled the "Send Mail" link for other members to send you email, you presumably didn't miss anything there, but you may have missed other stuff (unlikely in your situation, but possible).

I have log files of email sent (or attempted to be sent), and can see from those log files what the forum software tried to send that definitely or likely didn't get received. This especially effects new members trying to join the forum, and a few efforts to reset login passwords by existing members.

On my task list is going through those log files, identifying the likely failures, and (perhaps with some help from other forum moderators) following up on them, to regain contact with new applicants and members having password problems.

As of about a half day ago, I am pretty sure all server outbound email is working again. I have one more test of it, to start today, that will take a day to run. Server inbound email delivery never stopped, but some inbound email over the last few weeks might well not yet be in folders, or forwarded to individual email accounts of staff, where all the intended moderator and administrator (and perhaps even Bill) recipients can find it. Sorting that out is also on my todo list.

Posted by Miller (here)
I gather from other posts that I don't need to change my gmail account as I don't use the email facility, which is a relief.

Good - yes - since you didn't enable the "Send Email" link for other Avalon members, and since you're not a new applicant trying to join the forum, and since you apparently haven't lost your Avalon password, and since you apparently are not a heavy user of email notifications of new posts on subscribed threads ... you've likely not missed anything. But if I see you did miss something from the logs, we'll likely have you on the list of intended recipients of such email as I will find in the server's "lost + found" email logs, to be resent or followed up on, once we're confident the email system is fully working again.

[ThePythonicCow]

Posted by Bill Ryan (here)
It's VERY useful to know both of these things — (a) that donations are working just fine, but (b) the notifications seem to be missing.

Notifications aren't missing ... I've been seeing them all along, as for example the latest four such notification emails from Paypal that I listed above.

But I've definitely got to sort out the incoming email so that you, Bill, and the Avalon staff, can see it too, properly sorted by intended recipient(s), whether sent to just you Bill, or just an individual moderator or administrator, or the group "staff" account of all of us forum staff.

I had "scheduled" that sorting to happen already now, but that got pushed aside when I realized that I (with a little help from some avalon, gmail, fastmail, and mailtrap server software) had broken our server's outbound email in at least three different ways.

[ThePythonicCow]

Posted by Miller (here)
Done. Seemed to go OK my end and have received a Paypal notification, but I didn't return to the forum automatically afterwards; I don't know whether that's a glitch

Note to self: look into this (not returning to forum from Paypal link).

[ThePythonicCow]

Posted by ThePythonicCow (here)
... properly sorted by intended recipient(s), whether sent to just you Bill, or just an individual moderator or administrator, or the group "staff" account of all of us forum staff.

... or to the spam bucket.

As I've heard from several Avalon staff members, and as I've been seeing myself for the last couple of weeks (while apparently I was about the only one reading incoming Avalon email) ... there's lots of spam in there.

Our new Fastmail account provides a rich set of tools for managing spam, and I have a long history of using those tools on my own email accounts. My skills will have a fine opportunity to be put to good use here.

Gmail of course also has rich spam management tools. However the missions of Gmail and Fastmail are, as I noted in the first post of this present thread, fundamentally different. Gmail lives primarily to serve the email needs of various marketing and intelligence organizations, whereas Fastmail lives to serve the email needs of technically savvy users and small to medium groups. These missions are fundamentally different, which is reflected in their spam management.

[Vangelo]

Posted by ThePythonicCow (here)

Posted by Miller (here)
Done. Seemed to go OK my end and have received a Paypal notification, but I didn't return to the forum automatically afterwards; I don't know whether that's a glitch

Note to self: look into this (not returning to forum from Paypal link).

FYI: There is no "Return to Forum" button on the "How to Donate" page https://projectavalon.net/donation.html