The issue reported here is linked to App Engine and Gmail tightening up their spam filters. The root cause was an increase in organizations sharding out their spam systems to utilize App Engine’s free tier in such a way that is (a) in direct violation of our ToS and (b) making all of our lives suck a bit more (raise your hand if you want spam). It’s unfortunate that while App Engine is trying to provide a free tier that enables developers to easily use our platform, others see it as an opportunity for exploitation. Even more unfortunate is that it has a negative effect on legitimate users. It’s a fine balance that has been highlighted by several users within this thread.
Spam filtering is not a perfect science, and we’re constantly tweaking things -- with our customers in mind. This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues (which was also highlighted by a few within this thread). If this isn’t the case email me: cramsdale@google.com. For those asking, “hey, why am I being penalized for being a new customer?” See my previous comment about spam filtering not being a perfect science. Then email me.
Just a quick thank you for the free tier that App Engine offers. I use it to provide an educational app for my wife, who's a teacher, that she uses as part of her classes. It's nice to be able to provide that to her and her class, without hitting the bank (the school does that enough...). So, thank you.
Long time App Engine user here. If you're sending email as part of your app, I'd recommend using Mailgun (or similar services). You can set up DKIM, inbound routes, etc, and generally have more control over how your mail is processed. I never got the impression App Engine really wanted to be in the email delivery business, so if that's important to you, there are tools that are focused on that (and free up to a monthly quota). My $0.02.
Love Mailgun. You can configure it to just be a very easy mail provider or use the APIs to send mail and turn on all kinds of things like campaign monitoring, tags, click opens, etc.
The best part for me when using it for customers has been the bounce/click through rate tracking. When someone asks me, "how do I know they got it?" It's incredibly nice to point them to
the dashboard and show a less than 1% bounce rate (people putting in bad email is almost always the reason) with a log of every single email sent.
Most my clients get this service for free because their volume us low enough. They have quite a generous free tier.
+1 as well, after the changes to Mandrill recently, was looking for a new provider and can also recommend Mailgun (at least, for ease and configurability in set up and testing so far, not yet in production).
Yes, Mandrill should have kept the free quota after DKIM and SPF verification. Now, I will be trying out the competitors (Sendgrid and Mailgun) and if I find them at par with Mandrill then for paid projects, I might use them rather than Mandrill.
For our clients, the appeal of Mandrill is the Mailchimp Template Editor. I think this is where their focus is going and so if you don't need that, then definitely check out competitors.
Out of around 18 projects, in which we have used Mandrill, only in 1 we have used the template editor. (I think only about half went for paid accounts when the project went live.) Most of our need was simple and quick transactional messages (Registration, Forgot Password, etc.). The main reason we chose Mandrill was ease of set up over Amazon SES. (Even though we had AWS accounts for all projects.)
Anyways, it's up to Mandrill to choose who they want to serve. I really liked the service though. Best of Luck.
Thanks for the suggestion! I was having issues with SendGrid and had never heard of Mailgun. Took literally 2 minutes to get set up on Mailgun, add the credentials to my app, and send my first email! It's a super small app so don't need anything fancy, Mailgun is perfect.
edit: Not to say Mailgun isn't fancy, I have no idea if it is or not, I can say that it works in minutes though.
> This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues.
1. My app is not new. It's been running without issue for 2 or 3 years.
2. My app is not on the free tier.
3. On average, my app sends out under 8 emails a day, and it's done the same for over 2 years.
How your algorithm considers my app a spam risk sure beats me.
I'm missing 11 days worth of quote requests from customers. Are these recoverable? (is there a hidden outgoing-spam bin?)
Possible reason that email service providers and other email sending services might see an increase in new signups and free tier use recently: http://blog.mandrill.com/important-changes-to-mandrill.html Mandrill in essence raised their price, eliminated free tier, and required prepurchased blocks of sending capacity IIRC
The numbers Mandrill released about that business suggest that they had a large number of low volume senders, who may now be looking for a new home.
As an email service provider it's like that and more, since (1) once an abuser uses your service, they've gotten the benefit immediately and keep it even if their account is discovered as fraudulent later, e.g. stolen CC number & chargeback. (2) Abusive users can directly harm good users such as by harming the deliverability of the overall platform. It's not just bad debt, it's bad experience too. (3) Unlike Candy Japan where fraudsters mostly just wanted to check CC numbers and not actually buy product, email abusers really want to send emails (4) It can be hard to tell good and bad senders apart because some companies with an internet presence aren't email savvy and might make mistakes or might get hacked.
Spam filters are always tough because if you give someone transparency into which actions of theirs that you consider abuse, then they will quickly detect and route around your attempt to block them. (See Candy Japan article) It's pretty easy for a human to guess what might be the sign of their fraud and run a few experiments to see what gets flagged e.g. By comparison a machine learning system might be hard to outsmart, but then it's also challenging to explain and troubleshoot false positives. Hence what's effective is often a combination of machine-learned filters and heuristics along with manual overrides by human judgment.
All other things equal, new users are a lot more likely to engage in fraud than existing ones, and so tend to be under more suspicion. Aside from B2B fraud where companies take out lines of credit and then go bankrupt intentionally, it's uncommon for existing established customers to turn fraudulent - they're already vetted. (Consider: who is more likely to be fraudulent. The first time subscriber to Candy Japan, or a subscriber who has been using it for 12 months and is about to buy their 13th month?) It's not a great experience as a new user to be under suspicion, but if it's temporary and easily overridden by a human it can be a decent trade-off - the need to reach out acts a deterrent to spammers but does not deter legitimate users as much (speaking generally).
Chris, you say Gmail tightened up their spam filters... Did this take place in the last ~6 weeks or so?
I've noticed the spam filter on Gmail for Google Apps has gotten significantly less accurate recently, resulting in far more false-positives than usual. Any ideas if this is a known issue? I can only presume it's more accurate overall, but it was definitely a noticeable change for our organisation.
Google Cloud Support here. To correct the misleading title, this is not a generalized outage of any kind, but an issue with some applications sending mail that trigger spam/malware/etc filters. Since spammers generally do not comply with RFC 3514, filtering is unfortunately not an exact science, so false positives happen and we're working with the customers in question on a case-by-case basis.
Also, App Engine is integrated with both Sendgrid and Mailgun, and we strongly recommend using these if you're planning on sending larger quantities of mail:
5 weeks ago your infrastructure stopped sending out mail. Our apps didn't change, your infrastructure did.
Whether it's your outgoing spam filter that's overzealous, or if a datacenter is on fire is irrelevant. Your infrastructure stopped doing what your docs said it would.
Paying customers like myself have apps running on your infrastructure. Your infrastructure stopped delivering the exact same emails that it had been delivering for years. It did this without notification and without triggering any errors.
Approximately 35 quote requests that should have been delivered have disappeared into thin air. You've cost me money, and I'm disappointed to find out that you'd been notified of this issue weeks before it began affecting me.
I really have to highlight how dangerous an approach this is. You have no idea if the previous user of that IP was spamming. You should seriously consider mailgun or sendgrid. They also bring a lot of features that enable you to track when an email was read etc.
You seem to assume a lot of people know about the ins and outs of many different online mail delivery services. I would trust Google to do a little better than drop emails on the floor for weeks and not say something.
Infrastructure always changes. You need to have something in place to detect failure of important features of your apps, that get checked every day, every hour or even every minute, depending on how costly an outage is for you. There are always doublechecks possible.
Why do you think services like sendgrid, mailgun etc exist? It's pretty common knowledge that if you need to send mission critical email, you should use a service specifically designed for that. Sending email programmatically directly from a web server has ALWAYS been a gray area.
At the VERY least you could have easily set up a gmail account and sent them from smtp. Choose the right tool for the job.
> It's pretty common knowledge that if you need to send mission critical email, you should use a service specifically designed for that. Sending email programmatically directly from a web server has ALWAYS been a gray area.
I partially agree with this - dedicated email services have a higher deliverability rate than a random web server, especially a cloud-based server that might be using an external IP that was previously used to send spam. However, I can understand the parent poster being annoyed that their emails were being delivered one day, and not the next, without any changes on their end.
> At the VERY least you could have easily set up a gmail account and sent them from smtp. Choose the right tool for the job.
This is absolutely the wrong tool for the job. You should not be using a Gmail account to send out automated emails that you care about.
I wonder if people at Google know how awful it is to report an Appengine bug. You have to battle for days to get your bug acknowledged.
I've noticed that they've added more support people since the early days but it's still a pain as they seem to have an incentive to answer the tickets as fast as possible without doing any real investigation.
The linked issue is a good example of this behavior.
If you're running production services, you should probably sign up for a paid support package, which have guaranteed response times down to 15 minutes: https://cloud.google.com/support/
I think this is a terrible attitude. I will never implement it in my own support model. People shouldn't have to pay me in order to report a bug in my system. Arguably I should pay them, because one report is the tip of an iceberg of silent affected customers.
More often than not, what most people want is simply an acknowledgement. They're not seeking a bug bounty. And definitely not the brush-off you just gave the GP: when someone's house is on fire, you don't demand prepayment for the water.
You absolutely do not have to pay us to report a bug, period. The same day this bug was filed, one of my colleagues (pay...@google.com) jumped on the report and started asking questions to determine what was happening.
It seems to me that the GP's complaint is that this process was too slow: too much back-and-forth, too little dedicated attention to get the problem figured out immediately. That's a frustration which is easy to understand.
The point about support contracts was most likely intended to emphasize that if your livelihood depends on a service, you should have an agreement in place that guarantees you can wake up an engineer on the weekend.
I can't comment on the paid support. I've just noticed that the non-paid support engineers seems to be under pressure.
I feel little bit bad for them sometimes. They don't seem to have the time to look into problems. It must be quite a boring job for those assigned on the public issues.
Hi, I manage the team. Do you have some recent examples I can show to the higher-ups? Our incentives are built around quality of work, not speed, so what you're describing is definitely not working as intended.
Form my perspective: sometimes when I report a bug it's more so that other users don't waste their time troubleshooting it than to have it fixed immediately.
Finally there are issues like these ones that are not clear cut but that I can't investigate myself because it requires time and resources (they eventually cost me a few bucks running instances for testing purposes).
https://code.google.com/p/googleappengine/issues/detail?id=1...
You guys have a good product. I think that the promise of not having to manage infrastructure hit a cord with many people. However you do have many bugs to fix to make the platform more stable and (non intentionally) discouraging people from reporting issues will make things improve at a slower rate.
Google specifically seems to target their services much more at the large enterprise, rather than small companies and individuals. Enterprise will always pay for support, so if they are your target market, it's a given you can get them to pay. It's less of a necessity for providing lower costs, and more of a business decision to not expend resources to acquire customers that aren't bringing in tens or hundreds of thousands of dollars every month.
$10-20/month VPS providers often respond to, and actually take action, on tickets within 5-15 minutes at no extra cost. Of course there is no guarantee, and a ticket can certainly wind up unresolved for 24+ hours. In my experience, if the ticket you file clearly explains your issue and represents something they can realistically help you with, the level of service provided for such a low server cost can be exceptional.
For a comparision, AWS also makes you pay for prompt support, and all of their services cost money (unlike AppEngine which has a lot of free services).
I'm not sure what you mean by this. Every AWS service I can think of has "free tier" for lower use and then charges for higher use, like AppEngine. For example, AWS's email service (SES) is free up to 60,000 emails per month.
To me, the whole point of Appengine is to abstract out system administration. GCE (like AWS) abstracts out the systems, but not the administration - you suddenly become responsible for all the scaling, fault-tolerance, and all the subsystems that Appengine manages for you.
Confusingly enough Google Container Engine != GCE. GCE is Google Comput Engine, which doesn't abstract out much of the administration. Google Container Engine (or GKE) on the other hand does abstract out most of the administration via Kubernetes. We were recently attempting to use appengine for something but gave up due to how bad the support for Go is and migrated to GKE. It definitely isn't quite as abstracted as Appengine but it's darn close.
I believe you're talking about GCE (Compute Engine), while @kordless is talking about Container Engine.
I've tried Containers with Google App Engine Flexible Environment (in beta now) and liked that it's much more customizable than standard GAE. Basically, you just need any Docker container listening on port 8080.
But deploy time took 10-20 minutes, so I'm still preferring Standard. Hopefully they'll fix that before GA release.
Microservice based architectures, or SOAs, are ideal for segmenting operational and development tasks. Using Container Engine (which != GCE) will give users a BETTER way to abstract out system administration. You imply contrary.
My request is for a tool which would provide this functionality in the form of code which exposes an App Engine like interface to the developer, except it's running in Container Engine. Given Container Engine has the required code to restart processes that fail and oodles of other automated goodies, this give the fault tolerance desired.
> We wouldn't use it.
Given the "tool" doesn't appear to exist IRL, that statement is illogical. You can't not use something that doesn't exist. And, if there really did exist a migration tool that provided additional operational abstraction inside the framework you were migrating to, you'd run the risk of being in conflict if you didn't consider using it.
Reading through the thread, it looks like in the last few days it's been narrowed down to "messages which contain the URL of the App Engine instance silently disappear with no bounces or errors."
I wrote about this last year, but I feel it's still relevant. Spam filters, especially Google and Microsoft's (whose e-mail servers don't play by any of the standard rules nor do they send DMARC reports), are horrible over aggressive.
I have seen more phishing/ransomware e-mails come through mine (lots of malicious Javascript made to look like Excel files), so I can understand how great the risk is. The fact still remains though. E-mail is broken and is less reliable than an envelop dropped in a letterbox.
>especially Google and Microsoft's (whose e-mail servers don't play by any of the standard rules nor do they send DMARC reports)
I don't know about Microsoft, but I get a couple of DMARC reports per day from Google's email infrastructure. I also get reports from aol.com, yahoo.com, linkedin.com, comcast.com, fastmail.com, etc.
I think there is not a spam filter but this is a rule to prevent people from setting up phishing scams or other deceptive emails and they are using an appengine instance to host their scam.
Maybe I'm old-fashioned but if I was affected by this issue (and it had persisted for so long) I would set up an SMTP server and an appropriate MX record, then demonstrate using mail logs or port snooping that this was definitely not a receive-side issue.
Of course the support organisation here (Google) should be well equipped to set up this kind of testing themselves and work with a customer to root cause the issue. In my experience though these issues are rarely taken seriously until there is no-one else left for support to blame.
I had a Google engineer tell me once that the App Engine email api was never meant to send a lot of mail. That is why the quota is so low. I would recommend using a third party service to do e-mail.
I'm pretty surprised by the long wait times between "updates" from support... that's pretty rough. I understand it can be hard to get a good signal to noise ratio with support, but when so many other people are reporting issues, perhaps it's time to take it a bit more seriously?
This thread is a great example of why support is so hard.
not really. I wish I could get away with providing such crap support as the big players do.
If my customers don't get a reply within a few hours, then all hell breaks loose. Even if, there request comes in at 3am / 0300 on a Sunday morning my time.
To be fair, when I point this out to them they do apologise, didn't realise time difference, are in a different part of the world where Sunday is not a day off etc. I then ask what they expect from some billion dollar companies support wise and they say they are happy with a reply within a week. Asking them why a one man shop doing quite a bit less than billions of dollars is expected to provide so much more.
Anyway, rant over. Support isn't hard. You hire enough competent people as your customers require. The end. Happy customers
Must be an isolated issue. I run two App Engine apps and send 30K emails a day and I haven't noticed any change in my email open/click stats, which implies that emails are being delivered as usual.
Of course not. However, in the past ~7 years, while running my own mail stack, I never had a delivery issue, though obviously this is small traffic.
At work, we handle pretty large traffic for mail, with self-hosted mail servers as well - those are exim. No delivery issues either, although these are only the real, important, not even remotely spam-like, core business messages. Spam-like newsletters are done via sendgrid there.
The problem is that pesky humans can't decide what they want. Not sending mail is a bug but not receiving mail (i.e. spam filtering) is a feature? I can appreciate how this is confusing.
Not sending is a bug when it's undocumented, when it happens without notification, when there's no error message, and when there's no outgoing-spam bin.
The GAE docs say "do x and we'll send your mail". If we do x and mail isn't sent, then that is a problem.
Except as soon as you tell the spammer what is blocked and what isn't, they just add that to the playbook.
I agree with you that some measure of notification that there's spam filtering would be good, but it's Google we're talking about. There's spam filtering.
> I agree with you that some measure of notification that there's spam filtering would be good, but it's Google we're talking about. There's spam filtering.
Nonsense! I had no idea this was a feature of GAE. I pay for the use of GAE, and so long as I send less than my quota of emails, it's not Google's business what the content of those emails are.
There's no reason that there should be silent spam filtering on outgoing mail sent by my app. If they suspected that my app was sending spam, they should have notified me and disabled my app. That would have alerted me to the problem.
I find it absolutely abhorrent that Goole would just send private communications sent to me by prospective clients into the trash where they can never be retrieved or restored, and to do so without telling me is beyond belief.
Nope, I don't think there's an outgoing spam filter. My guess is that individual messages are only stopped when they enter a gmail account; if there's an abuse filter for email senders, it's probably on the whole account after a manual complaint process.
The point: G is accepting cash from one group of people to send a ton of messages (mostly unwanted by addressees) and earning ad dollars from another group to stop unwanted messages.
30 years ago this would be textbook conflict of interest but the 'veil of automation' means we're blind to a lot of unsavory business practices.
> My guess is that individual messages are only stopped when they enter a gmail account
That's been tested for and eliminated. My app logs that an email has created and is going to be sent. The send_email function is called to send the email. Then my app logs that the email has been sent.
The emails are sent to addresses hosted at both gmail and outlook. They are not received. They do not appear in spam bins.
GAE has a bounce api that will log if an email is bounced back. In my case, no reports are logged by my bounce handler.
The content of my emails has not changed for 2 years. For 2 years both gmail and outlook received my emails in their inboxes (not spam bins).
Given all this, I'm pretty sure there is an outgoing spam filter.
Yeah, but pricing is ten times Appengine's (actually more - recently we don't seem to be being charged for email at all). Amazon SES is the only service with a comparable cost that I've found. I'd love to hear of other alternatives at the 1c/100 level.
At least a few reported that it was bouncing at the destination once they set up an appropriate handler. They are only "not being sent" when there is a url in the message and that is a big red flag on the receiving side (not much text but a url). It doesn't look like an App Engine problem. It looks like messages being identified as spam problem.
No, gmail constantly drops messages without putting them in spam. That's actually a huge problem. Microsoft does the same. That's for e-mails that don't even contain any links!
Interesting. I have been somewhat weary of the "machine learning for everything!" approach that google has taken recently -- i just don't think it's there yet. I wonder if a learning algorithm on the google side just started deciding outgoing spam and not giving correct feedback. It is very curious that no url works and with url doesn't. That should give straightforward repeatability (if they can simulate as your account). Have you tried sending to different target e-mail addresses during the test? I saw once it just worked and a bunch of stuff flew through. Also saw there was some question as to quotas. Did you get it clarified? Are you maybe above usage quotas with the regular e-mails so test e-mails are being stopped/delayed? Are test and production separate? It seems like if it were a google side issue there would be an uproar from the users.
I ran into a similar problem a while back (using Heroku) --- after a brief foray into the world of spam filtering, it turned out to be caused by our domain being identified by SpamAssassin (IIRC) as a spam signal. We changed the wording of our emails and moved to a different subdomain and suddenly all the email got through. It was.. interesting to debug.
Greetings folks! One of the App Engine PMs here. That couldn't be further from the truth - we absolutely want people to use App Engine. If you want to manage your own infrastructure, Google Compute Engine is great. If you want something that scales automatically, and requires less management - App Engine is a great option. We've actually made a ton of changes recently, specifically on our support for docker based runtimes, new docs, and new runtime support (like nodejs, ruby, python3, etc).
Appengine's whack pricing says: we'd like to kick you off appengine onto GCE but that would scare away all the enterprise users from GCE. If we announce regular Moore's Law following price cuts for GCE and nothing for appengine, well hey, we can't stop you leaving. We're totally not twisting you're arm! (aside: we are)
The new Tesla Model 3 is cheaper than the Model S. Is that Tesla encouraging people to migrate to the
Model 3?
They're not the same product, and in this case they serve different points in the space with different underlying requirements. App Engine Standard's cost structure on the backend just hasn't changed nearly as much as compute engine's lately where it's a much more direct: oh look Haswell released, more cores per host at similar dollars yields less $$/core.
Disclosure: I work on Cloud (Compute Engine mostly) and care a lot about our prices.
Jeez, if the Snapchat backend looks anything like the ux works this isn't much of an endorsement. Just because some company moves a lot of bits doesn't mean they know what they are doing. The easiest way to move a lot of bits is to be bad at infrastructure and have investors with deep pockets. Fuck just run a rack of open relays and bad NTP/DNS servers.
>> if the Snapchat backend looks anything like the ux works
>What does this even mean.
I find their app to be very poorly designed, I wouldn't be surprised if their back end is also poorly designed.
>You have to have some measure of competency to be able to operate at Snapchat scale. You really do.
Meh, they are big but they aren't that big. There are about 3000 bigger sites out there and not that many are built on GAE so that's a strange way to measure the competency of GAE or snap chat for that matter.
You can run at scale and do it poorly, Enterprises do it all the time. Size and scale aren't really a good indication of quality.
If you want to make a case for appengine, do that, but don't tell me that 'these guys vetted it, and they are kinda big, so it must be good' There's a lot more big sites that don't use app engine. Size doesn't equal competency.
This is what happens when you buy a service from an organization oriented toward ad-supported services. The mindset is that it doesn't matter if the user isn't being served.
The issue reported here is linked to App Engine and Gmail tightening up their spam filters. The root cause was an increase in organizations sharding out their spam systems to utilize App Engine’s free tier in such a way that is (a) in direct violation of our ToS and (b) making all of our lives suck a bit more (raise your hand if you want spam). It’s unfortunate that while App Engine is trying to provide a free tier that enables developers to easily use our platform, others see it as an opportunity for exploitation. Even more unfortunate is that it has a negative effect on legitimate users. It’s a fine balance that has been highlighted by several users within this thread.
Spam filtering is not a perfect science, and we’re constantly tweaking things -- with our customers in mind. This issue should be limited to new applications where the trust signal might be a bit lower. Thus existing apps / customers shouldn’t be experiencing issues (which was also highlighted by a few within this thread). If this isn’t the case email me: cramsdale@google.com. For those asking, “hey, why am I being penalized for being a new customer?” See my previous comment about spam filtering not being a perfect science. Then email me.
We’re here and we want to help.
-- Chris (Lead PM for App Engine)