4 ways to automate email verification and MFA in end-to-end testing

Most software products today use email verification and multi-factor authentication (MFA) to secure user accounts. This often ends up being a specific pain point for test automation.

Signup and login are critical failure points for any web application. Because they rely on third parties or shared states, authentication flows are hard to test with automated scripts. Some third parties also block browser automation. We've heard many stories of logins and signup flow breaking silently, and nobody noticing until analytics show a drop in user engagement.

At Heal, we believe that end-to-end tests should be as close as possible to the real user experience. In other words, they should replicate what a manual QA tester would do, including verifying emails and entering OTPs.

In this article we'll look at 4 strategies to test applications that use email verification and MFA. We'll give code samples in Playwright, but the same principles apply to other testing frameworks like Cypress or Selenium.

There's actually two distinct scenarios here: do you want to find a way to test features behind email verification or MFA? Or do you want to specfically test email verification and MFA? In the former, having some kind of bypass is advisable (option 1 and 2). In the latter, finding ways to actually automate the email verification and MFA is needed (option 3 and 4).

Option 1: avoid the problem

The simplest way to avoid the problem is to bypass email verification and MFA in your test environment. The most common implementation is to have a flag that turns off email verification and MFA in dev and staging.

We've also seen customers disable auth completely in dev/staging, and either use basic HTTP auth to protect the environment, or not expose it to the internet at all.

Using basic HTTP authentication is actually very straightforward if you use Playwright:

playwright.config.ts

import { defineConfig } from '@playwright/test';

export default defineConfig({
  use: {
    httpCredentials: {
      username: 'user',
      password: 'pass',
    },
  },
});

Option 2: have dedicated testing endpoints in the backend

Another common approach is to have dedicated endpoints in the backend to mock verification emails, SMS or TOTPs. This is a great approach because it allows you to test the entire flow. Typically, you'd set up a global setup test that runs before all the others, and save the browser state for reuse in all the other tests.

For instance, on the staging server, one can expose a endpoint in the form GET /verification-sms that would return the code to use. Such endpoint must not be deployed to production as it would open the door to a security issue nullifying the SMS validation altogether. This is one of the reason such approaches are to be considered very carefully!

There are a few drawbacks to this approach:

• it requires some backend work to implement these endpoints.
• testing will use a different code path than production, so we're again introducing a diff between the system users will see and the testing system

Option 3: Use actual mailbox/phone numbers/OTP generators

In the 2 previous options,we looked at ways to bypass the email verification and MFA. The downside shared by all of those was to require to change the system under test to accommodate the tests. Now we'll switch gear and see if we can make the tests work without changing the actual system.

Email verification

The most common way to verify emails is to send an email with a token. To automate that flow we'll use a mailbox service that provides an API. There are many services that provide this, including hacking on top of the gmail API, or using a dedicated service like mailslurp or mailinator.

To allow for running tests in parallell, you'll need to create a new email address for each test run, for example,test+<testId>@provider.com.

Let's take a look at what it would look like with Mailinator. We'll be using the mailinator ts client.

1. Create a mailinator account and get verified
2. Create a private inbox, and get your API key
3. npm i mailinator-client and npm install typed-rest-client --save

import { GetInboxRequest, GetMessageRequest, Inbox, MailinatorClient, Message, Sort } from "mailinator-client";
import { IRestResponse } from "typed-rest-client";

const MAILINATOR_API_KEY = ...
const MAILINATOR_DOMAIN = ...
const MAILINATOR_INBOX = ...

const mailinatorClient: MailinatorClient = new MailinatorClient(MAILINATOR_API_KEY);

// Fetch the latest email
const inbox: IRestResponse<Inbox> = await mailinatorClient.request(
    new GetInboxRequest(MAILINATOR_DOMAIN, MAILINATOR_INBOX, undefined, 1, Sort.DESC),
);

if(inbox.statusCode !== 200) {
    throw new Error("Failed to fetch inbox")
} else if (!inbox?.result?.msgs.length) {
    throw new Error("No messages found")
} 

const msg :IRestResponse<Message> = await mailinatorClient.request(
    new GetMessageRequest(MAILINATOR_DOMAIN, inbox.result.msgs[0].id),
);

if(msg.statusCode !== 200) {
    throw new Error("Could not fetch message")
}

const codeRegex = /[\s.\r\n]([0-9]{6})[\s.\r\n]/
let code:string|undefined = undefined

for(const part of msg?.result?.parts ?? []){
    const match = codeRegex.exec(part.body)
    if(match && match.length>0){
        code = match[0]
        break
    }
}

if(code){
    console.log(code)
    await page.getByLabel('Last name').fill(code);
} else {
    throw new Error("Could not find code in email")
}

This implementation is extremely naive:

• it expects the verification email to be in the inbox, but it might take some time to arrive.
• it always reads the latest message, but what if that message is a message from an old run?

We'd have to make it way more robust and handle wait time in order to avoid flakes and have some logic to generate a new inbox for each test if running tests in parallell. In order to make this robust, we'd need to:

• parallellism - If several tests using email run in parallell, make sure they don't interfere with each other by using one unique inbox per test.
• Delivery time - Handle cases where the email takes a while to arrive, for example,by adding retries.
• Improve parsing - Make sure that parsing the code from the email is robust and works even if the email format changes
• support code and email linkg - Support both email OTP and following a verification link
• Old emails - Make sure we never read old emails.

Time-based One-Time Passwords (TOTPs)

Time-based One-Time Passwords (TOTPs) are a common way to implement MFA. The TOTP algorithm generates a unique, time-sensitive code by combining a shared secret key with the current timestamp, using a cryptographic hash function. The generated code is valid only for a short time window, enhancing security for authentication systems.

The most common implementation is to send a 6-digit code to the user's phone number or an authenticator app. To automate this flow we could use a service that provides an API to generate TOTPs, but it's quite easy to generate TOTPs ourselves.

After getting the OTP secret, we can generate the TOTP using react-otp, and add some logic to make sure we generate a new token if the current one is about to expire to avoid flakiness.

const secret = ... // parse the TOTP secret from the page

const totp = new OTPAuth.TOTP({ secret: secret});
console.log(`Generating OTP for secret: ${secret}`)
const sleep = ms => new Promise(res => setTimeout(res, ms));
let token = totp.generate();
const seconds = () => totp.period - (Math.floor(Date.now() / 1000) % totp.period);
console.log("Token expires in", seconds(), "seconds");
if (seconds() < 25){
    await sleep(seconds() * 1000)
    token = totp.generate();
    console.log("New token expires in", seconds(), "seconds");
}
console.log("New long expiry token is", token)

await page.getByLabel('otp').fill(token);

SMS

Like email, SMS can be automated using any SMS API provider like Twilio or Mailosaur.

This is typically a little bit more involved than email because it's harder and more expensive to create phone numbers on the fly than to create inboxes.

1. Create an SMS and assign it to a Test
2. (Optional) if several tests run in parallel, make sure tests don't interfere with each other. For instance, use a pool of phone numbers with one phone number by test (simple but expensive) or have a lock on each phone number so that a test cannot use a phone number that is already in use until the SMS is received.
3. Create a service with a webbhook that twilio (or another service) can post to when receiving an SMS. See for example this post.
4. Set up the test to fill in the phone number, then wait for the SMS to be received and type it in.

In order to make this robust, we'd need to:

• parallellism - If several tests using SMS run in parallell, make sure they don't interfere with each other. For example, we could have a pool of phone numbers and assign one to each test.
• Delivery time - Handle the case where the SMS takes a while to arrive. We could add a retry mechanism to wait for the SMS to arrive.
• Missing message - Handle the case where the SMS is not received. We could add a timeout mechanism to fail the test if the SMS is not received within a certain time.

Option 4: Use Heal.dev for real testing without the hassle

If all of this sounds like too much hassle, read on! With heal, we can set up an end to end test with email verification and MFA in 5 minutes. In the following video, I've picked a website with a relatively long signup flow. Feel free to watch the video at 2x speed, I've purposefully kept it as is to show the real time it takes to set up a test.

Conclusion

Testing email verification and MFA ensures that you're actually testing what the user sees. While it's recommended to bypass auth for most tests, there should be a few end-to-end tests that specifically test auth and onboarding. Similarly, testing privilege escalation and other security features can be critical for some apps.

While it's possible to build all the infrastructure to run those tests from scratch, it takes a bit of work:

• A scalable email testing service that handles parallellism and retries to avoid flakes.
• A scalable SMS testing service that manages numbers for you.
• A TOTP generator.
• All the tooling you need to define signup, email verificaton, and MFA flows in 5 minutes, and never worry about them again.

Heal.dev lets you do that in less than 5 minutes. Book a demo if you'd like to get access!