Updated Considerations When Validating Email with Mailgun
Well, they say that all good things must come to an end, which turns out to be the case for Mailgun's free email address validation API, a service about which I've spoken highly in the past. In a June 27th blog post, along with new features, they announced a usage based pricing model. So, what's changed? Should you still probably be validating your email with Mailgun? The answer, as usual, is that it depends.
Features: Obviously this is what they lead with in the blog post, before they discuss pricing. The first three features have to do with deeper analysis of the mailbox being verified itself.
Mailbox Verification: By default, this option is off (it adds noticeable overhead to the request time). As the API docs explain it, "If set to true, a mailbox verification check will be performed against the address." The API response will then include a
mailbox_verificationkey, with a value of true, false, or unknown.
Unknown is returned if the SMTP request could not be completed or mailbox verification is not supported on the target mailbox provider.[^1] Which providers support mailbox verification? They don't say. I ran some checks, and this is what I found:
Provider Support gmail.com Yes Consistently returned true/false . Emailfirstname.lastname@example.org format was also fully supported. hotmail.com Yes comcast.net Yes att.net Yes msn.com Yes icloud.com Yes aol.com Yes/Partial Prone to timeouts, returning unknown outlook.com Yes/Partial Prone to timeouts, returning unknown Office 365 Partial Valid email addresses returned true. Invalid returned unknown yahoo.com No Consistently returned unknown
Conclusion: It's an impressive feature, with fairly wide support, and you get an answer, most of the time.
Role-based Address Check: Effectively, this indicates whether a mailbox is a distribution list. It's returned with every validation request. I didn't have a large sample size, but I found it to be accurate. However, I don't know what specific actions I would take, based on knowing this.
Disposable Mailbox Detection: On the other hand, the value of knowing if an email address is from a known disposable mailbox provider is readily apparent. As with the role-based address check, this is returned with every verification request. Does it work? I did a quick check, after Googling "disposable email addresses" and using the first few results.
Provider Address Correct? guerrillamail.com email@example.com Yes mailinator.com ZippityTinkerton@mailinator.com Yes getnada.com firstname.lastname@example.org Yes emailondeck.com email@example.com NO throwawaymail.com firstname.lastname@example.org NO
Conclusion: It's another helpful feature, and you get the right answer more often than you would without it, but it's not foolproof.
Rate Limiting of Validation API Key: This final feature is not related to email verification, but rather account management. There seem to be some issues and bugs with the documentation and implementation of this feature, which I'll discuss after explaining it.
By default, this feature is not enabled, so there is no limit on email validation API calls. Once enabled, it is supposed to give you the ability to set an upper limit for public API calls, after which additional requests are denied. This is important for controlling costs, but also because every existing implementation of the Mailgun email validation API is set up with a public API key and many of them are web accessible. There's obviously a heightened potential for abuse in these cases, so being able to stop excessive API calls is essential.
So, my first question was, is the limit daily or monthly? The documentation is contradictory. The introduction to the email validation portion of the API documentation indicates that the limit is monthly, as do the email alerts.
However, the portions of the API documentation discussing access via the private API key suggest that it is a daily limit.
I reached out to the Mailgun team on Twitter for clarification, but they haven't responded. Of more concern is that in my testing, I've somehow ended up above the limit that I set, and requests are not being denied.
I plan on reaching out to Mailgun again about this. I'll update this post if this situation is clarified or corrected.
SLAs: Honestly, a service level agreement for email validation isn't of much interest or use to me, though apparently there are clients who want it. More power to them for providing it. Of significantly more interest, as Disqus user @chris points out, would be an SLA for the base sending API; those outages are frustrating.
Pricing: This is the kicker when it comes to reevaluating if Mailgun is the right solution for your email validations. The first 100 validations are free each month, but after that, they start at $.01 per validation. Here's a link to the pricing calculator.
I understand the need for a company to make money to pay the bills and keep offering their services. However, I tend to side with the majority of comments on the article, the new pricing model seems a bit off. Or, perhaps better put, it's not very 'develop-friendly'.
For smaller users, the 10,000 free messages included in the sending API is generous and affordable; it makes testing possible and strikes a good balance where the costs largely fall on users who have grown to the point where they can/should be able to handle it.
On the other hand, the 100 validations disappear very quickly, especially when testing. While writing this post I've used 79 of mine. As a developer, it's much harder to choose Mailgun for projects I'm starting when the free cap is low and the costs jump rapidly.
Should I Still Be Using Mailgun to Validate Emails?
The Mailgun email validation API is fantastic, and when it was free, using it was an easy decision. The new features are promising, and can provide a degree of certainty when dealing with user email addresses that I don't think is possible elsewhere. If this is compelling proposition for you, then you have your answer.
That being said, I no longer have a strong argument here; you need to decide for yourself, based on your use cases and budget. For a lot of developers, especially with smaller projects, a free service or method that's slightly inferior likely trumps a fantastic but expensive one.
[^1]: It doesn't appear in the documentation, but it also returns `unknown` if the email address is *de facto* invalid. I would have expected false in those cases.