Hướng dẫn dùng assert exception trong PHP

TLDR; scroll to: Use PHPUnit's Data Provider

Nội dung chính

• How to test multiple Exceptions?
• Split multiple exceptions into separate tests:
• Catch an exception and check it with an assertion
• Use PHPUnit's Data Provider
• Testing Exceptions Gotchas
• exception of type "TypeError"
• exception of type "TypeError" again

Nội dung chính

• How to test multiple Exceptions?
• Split multiple exceptions into separate tests:
• Catch an exception and check it with an assertion
• Use PHPUnit's Data Provider
• Testing Exceptions Gotchas
• exception of type "TypeError"
• exception of type "TypeError" again

Nội dung chính

• How to test multiple Exceptions?
• Split multiple exceptions into separate tests:
• Catch an exception and check it with an assertion
• Use PHPUnit's Data Provider
• Testing Exceptions Gotchas
• exception of type "TypeError"
• exception of type "TypeError" again

Nội dung chính

• How to test multiple Exceptions?
• Split multiple exceptions into separate tests:
• Catch an exception and check it with an assertion
• Use PHPUnit's Data Provider
• Testing Exceptions Gotchas
• exception of type "TypeError"
• exception of type "TypeError" again

Nội dung chính

• How to test multiple Exceptions?
• Split multiple exceptions into separate tests:
• Catch an exception and check it with an assertion
• Use PHPUnit's Data Provider
• Testing Exceptions Gotchas
• exception of type "TypeError"
• exception of type "TypeError" again

PHPUnit 9.5 offers following methods to test exceptions:

$this->expectException(string $exceptionClassName);
$this->expectExceptionCode(int|string $code);
$this->expectExceptionMessage(string $message);
$this->expectExceptionMessageMatches(string $regularExpression);
$this->expectExceptionObject(\Exception $exceptionObject);

However the Documentation is vague about the order of any of the above methods in the test code.

If you get used to using assertions for example:

assertSame($expected, $actual);
    }
}

output:

 ✔ Simple assertion
OK (1 test, 1 assertion)

you may be surprised by failing the exception test:

expectException(\InvalidArgumentException::class);
    }
}

output:

 ✘ Exception
   ├ InvalidArgumentException:

ERRORS!
Tests: 1, Assertions: 0, Errors: 1.

The error is because:

Once an exception is thrown the PHP can not return to the line of code that comes after the line that thrown the exception. Catching an exception changes nothing in this regard. Throwing an exception is a one way ticket.

Unlike errors, exceptions don't have a capability to recover from them and make PHP continue code execution as if there was no exception at all.

Therefore PHPUnit does not even reach the place:

$this->expectException(\InvalidArgumentException::class);

if it was preceded by:

throw new \InvalidArgumentException();

Moreover, PHPUnit will never be able to reach that place, no matter its exception catching capabilities.

Therefore using any of the PHPUnit's exception testing methods:

$this->expectException(string $exceptionClassName);
$this->expectExceptionCode(int|string $code);
$this->expectExceptionMessage(string $message);
$this->expectExceptionMessageMatches(string $regularExpression);
$this->expectExceptionObject(\Exception $exceptionObject);

must be before a code where an exception is expected to be thrown in contrary to an assertion that is placed after an actual value is set.

A proper order of using exception test:

expectException(\InvalidArgumentException::class);
        throw new \InvalidArgumentException();
    }
}

Because call to PHPUnit internal methods for testing exceptions must be before an exception is thrown it makes sense that PHPUnit methods related to test exceptions start from $this->excpect instead of $this->assert.

Knowing already that:

Once an exception is thrown the PHP can not return to the line of code that comes after the line that thrown the exception.

You should be able to easily spot a bug in this test:

expectException(\RuntimeException::class);
        throw new \RuntimeException();

        # Should Fail
        $this->expectException(\RuntimeException::class);
        throw new \InvalidArgumentException();
    }
}

The first $this->expectException() should be OK, it expects an exception class before an exact exception class as expected is thrown so nothing wrong here.

The second that should fail expects RuntimeException class before a completely different exception is thrown so it should fail but will PHPUnit execution reach that place at all?

The test's output is:

 ✔ Throw exception

OK (1 test, 1 assertion)

OK?

No it is far from OK if the test passes and it should Fail on the second exception. Why is that?

Note that the output has:

OK (1 test, 1 assertion)

where the count of tests is right but there is only 1 assertion.

There should be 2 assertions = OK and Fail that makes test not passing.

That's simply because PHPUnit is done with executing testThrowException after the line:

throw new \RuntimeException();

that is a one way ticket outside the scope of the testThrowException to somewhere where PHPUnit catches the \RuntimeException and does what it needs to do, but whatever it could do we know it will not be able to jump back into testThrowException hence the code:

# Should Fail
$this->expectException(\RuntimeException::class);
throw new \InvalidArgumentException();

will never be executed and that's why from PHPUnit point of view the test result is OK instead of Fail.

That's not a good news if you would like to use multiple $this->expectException() or a mix of $this->expectException() and $this->expectExceptionMessage() calls in the same test method:

expectException(\RuntimeException::class);
        throw new \RuntimeException('Something went wrong');

        # Fail
        $this->expectExceptionMessage('This code will never be executed');
        throw new \RuntimeException('Something went wrong');
    }
}

gives wrong:

OK (1 test, 1 assertion)

because once an exception is thrown all other $this->expect... calls related to testing exceptions will not be executed and the PHPUnit test case result will only contain the result of the first expected exception.

How to test multiple Exceptions?

Split multiple exceptions into separate tests:

expectException(\RuntimeException::class);
        throw new \RuntimeException();
    }

    public function testThrowExceptionFoo(): void
    {
        # Fail
        $this->expectException(\RuntimeException::class);
        throw new \InvalidArgumentException();
    }
}

gives:

 ✔ Throw exception bar
 ✘ Throw exception foo
   ┐
   ├ Failed asserting that exception of type "InvalidArgumentException" matches expected exception "RuntimeException". Message was: "" at

FAILURES!
Tests: 2, Assertions: 2, Failures: 1.

FAILURES as it should.

This method however has a downside at its fundamental approach - for every exception thrown you need a separate test. That will make a flood of tests just to check exceptions.

Catch an exception and check it with an assertion

If you can't continue script execution after an exception was thrown you may simply catch an expected exception and later get all data about it with the methods an exception provides and use that with combination of expected values and assertions:

getMessage();
            $code = $e->getCode();
        }

        $expectedClass = \RuntimeException::class;
        $expectedMsg = 'Something went wrong';
        $expectedCode = 0;

        if (empty($className)) {
            $failMsg = 'Exception: ' . $expectedClass;
            $failMsg .= ' with msg: ' . $expectedMsg;
            $failMsg .= ' and code: ' . $expectedCode;
            $failMsg .= ' at: ' . $location;
            $failMsg .= ' Not Thrown!';
            $this->fail($failMsg);
        }

        $this->assertSame($expectedClass, $className);
        $this->assertSame($expectedMsg, $msg);
        $this->assertSame($expectedCode, $code);

        # ------------------------------------------

        # Fail
        unset($className);
        try {
            $location = __FILE__ . ':' . (string) (__LINE__ + 1);
            throw new \InvalidArgumentException('I MUST FAIL !'); 

        } catch (\Exception $e) {
            $className = get_class($e);
            $msg = $e->getMessage();
            $code = $e->getCode();
        }

        $expectedClass = \InvalidArgumentException::class;
        $expectedMsg = 'Something went wrong';
        $expectedCode = 0;

        if (empty($className)) {
            $failMsg = 'Exception: ' . $expectedClass;
            $failMsg .= ' with msg: ' . $expectedMsg;
            $failMsg .= ' and code: ' . $expectedCode;
            $failMsg .= ' at: ' . $location;
            $failMsg .= ' Not Thrown!';
            $this->fail($failMsg);
        }

        $this->assertSame($expectedClass, $className);
        $this->assertSame($expectedMsg, $msg);
        $this->assertSame($expectedCode, $code);
    }
}

gives:

 ✘ Throw exception
   ┐
   ├ Failed asserting that two strings are identical.
   ┊ ---·Expected
   ┊ +++·Actual
   ┊ @@ @@
   ┊ -'Something·went·wrong'
   ┊ +'I·MUST·FAIL·!'

FAILURES!
Tests: 1, Assertions: 5, Failures: 1.

FAILURES as it should but oh my lord, did you read all that above? You need to take care for clearing variables unset($className); to detect if an exception was thrown, then this creature $location = __FILE__ ... to have a precise location of the exception in case it was not thrown, then checking if the exception was thrown if (empty($className)) { ... } and using $this->fail($failMsg); to signal if the exception was not thrown.

Use PHPUnit's Data Provider

PHPUnit has a helpful mechanism called a Data Provider. A data provider is a method that returns the data (array) with data sets. A single data set is used as the argument(s) when a test method - testThrowException is called by PHPUnit.

If the data provider returns more than one data set then the test method will be run multiple times, each time with another data set. That is helpful when testing multiple exceptions or/and multiple exception's properties like class name, message, code because even though:

Once an exception is thrown the PHP can not return to the line of code that comes after the line that thrown the exception.

PHPUnit will run the test method multiple times, each time with different data set so instead of testing for example multiple exceptions in a single test method run (that will fail).

That's why we may make a test method responsible for testing just one exception at the time but run that test method multiple times with different input data and expected exception by using the PHPUnit's data provider.

Definition of the data provider method can be done by doing @dataProvider annotation to the test method that should be supplied by the data provider with a data set.

            [
                [
                    'input' => 1,
                    'className' => \RuntimeException::class
                ]
            ],

            \InvalidArgumentException::class =>
            [
                [
                    'input' => 2,
                    'className' => \InvalidArgumentException::class
                ]
            ]
        ];
        return $data;
    }

    /**
     * @dataProvider ExceptionTestProvider
     */
    public function testThrowException($data): void
    {
        $this->expectException($data['className']);
        $exceptionCheck = new ExceptionCheck;

        $exceptionCheck->throwE($data['input']);
    }
}

gives result:

 ✔ Throw exception with RuntimeException
 ✔ Throw exception with InvalidArgumentException

OK (2 tests, 2 assertions)

Note that even there is just a one test method in the entire ExceptionTest the output of the PHPUnit is:

OK (2 tests, 2 assertions)

So even the line:

$exceptionCheck->throwE($data['input']);

threw the exception at the first time that was no problem for testing another exception with the same test method because PHPUnit ran it again with different data set thanks to the data provider.

Each data set returned by the data provider can be named, you just need to use a string as a key under which a data set is stored. Therefore the expected exception class name is used twice. As a key of data set array and as a value (under 'className' key) that is later used as an argument for $this->expectException().

Using strings as key names for data sets makes that pretty and self explanatory summary:

✔ Throw exception with RuntimeException

✔ Throw exception with InvalidArgumentException

and if you change the line:

if ($data === 1) {

to:

if ($data !== 1) {

of the public function throwE($data)

to get wrong exceptions thrown and run the PHPUnit again you'll see:

 ✘ Throw exception with RuntimeException
   ├ Failed asserting that exception of type "InvalidArgumentException" matches expected exception "RuntimeException". Message was: "" at (...)

 ✘ Throw exception with InvalidArgumentException
   ├ Failed asserting that exception of type "RuntimeException" matches expected exception "InvalidArgumentException". Message was: "" at (...)

FAILURES!
Tests: 2, Assertions: 2, Failures: 2.

as expected:

FAILURES! Tests: 2, Assertions: 2, Failures: 2.

with exactly pointed out the data sets' names that caused some problems:

✘ Throw exception with RuntimeException

✘ Throw exception with InvalidArgumentException

Making public function throwE($data) not throwing any exceptions:

public function throwE($data)
{
}

and running PHPUnit again gives:

 ✘ Throw exception with RuntimeException
   ├ Failed asserting that exception of type "RuntimeException" is thrown.

 ✘ Throw exception with InvalidArgumentException
   ├ Failed asserting that exception of type "InvalidArgumentException" is thrown.

FAILURES!
Tests: 2, Assertions: 2, Failures: 2.

It looks that using a data provider has several advantages:

1. The Input data and/or expected data is separated from the actual test method.
2. Every data set can have a descriptive name that clearly points out what data set caused test to pass or fail.
3. In case of a test fail you get a proper failure message mentioning that an exception was not thrown or a wrong exception was thrown instead of an assertion that x is not y.
4. There is only a single test method needed for testing a single method that may throw multiple exceptions.
5. It is possible to test multiple exceptions and/or multiple exception's properties like class name, message, code.
6. No need for any non-essential code like try catch block, instead just using the built in PHPUnit's feature.

Testing Exceptions Gotchas

exception of type "TypeError"

With PHP7 datatype support this test:

expectException(\InvalidArgumentException::class);
        $chat = new DatatypeChat;
        $chat->say(array());
    }
}

fails with the output:

 ✘ Say
   ├ Failed asserting that exception of type "TypeError" matches expected exception "InvalidArgumentException". Message was: "Argument 1 passed to DatatypeChat::say() must be of the type string, array given (..)

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

even though there is in the method say:

if (!is_string($msg)) {
   throw new \InvalidArgumentException('Message must be a string');
}

and the test passes an array instead of a string:

$chat->say(array());

PHP does not reach the code:

throw new \InvalidArgumentException('Message must be a string');

because the exception is thrown earlier due to the type typing string:

public function say(string $msg)

therefore the TypeError is thrown instead of InvalidArgumentException

exception of type "TypeError" again

Knowing that we don't need if (!is_string($msg)) for checking data type because PHP already takes care about that if we specify the data type in the method declaration say(string $msg) we may want to throw InvalidArgumentException if the message is too long if (strlen($msg) > 3).

 3) {
            throw new \InvalidArgumentException('Message is too long');
        }
        return "Hello $msg";
    }
}

class ExceptionTest extends \PHPUnit\Framework\TestCase
{
    public function testSayTooLong(): void
    {
        $this->expectException(\Exception::class);
        $chat = new DatatypeChat;
        $chat->say('I have more than 3 chars');
    }

    public function testSayDataType(): void
    {
        $this->expectException(\Exception::class);
        $chat = new DatatypeChat;
        $chat->say(array());
    }
}

Modifying also ExceptionTest so we have two cases (test methods) where an Exception should be thrown - first testSayTooLong when the message is too long and second testSayDataType when the message is a wrong type.

In both tests we expect instead of a specific exception class like InvalidArgumentException or TypeError just a generic Exception class by using

$this->expectException(\Exception::class);

the test result is:

 ✔ Say too long
 ✘ Say data type
   ├ Failed asserting that exception of type "TypeError" matches expected exception "Exception". Message was: "Argument 1 passed to DatatypeChat::say() must be of the type string, array given (..)

FAILURES!
Tests: 2, Assertions: 2, Failures: 1.

testSayTooLong() expecting a generic Exception and using

$this->expectException(\Exception::class);

passes with OK when the InvalidArgumentException is thrown

but

testSayDataType() using the same $this->expectException(\Exception::class); Fails with the description:

Failed asserting that exception of type "TypeError" matches expected exception "Exception".

It looks confusing that PHPUnit complained that exception TypeError was not an Exception, otherwise it would not have had any problem with $this->expectException(\Exception::class); inside the testSayDataType() as it did not have any problem with the testSayTooLong() throwing InvalidArgumentException and expecting: $this->expectException(\Exception::class);

The problem is that PHPUnit misleads you with the description as above because TypeError is not an exception. TypeError does not extends from the Exception class nor from any other its children.

TypeError implements Throwable interface see documentation

whereas

InvalidArgumentException extends LogicException documentation

and LogicException extends Exception documentation

thus InvalidArgumentException extends Exception as well.

That's why throwing the InvalidArgumentException passes test with OK and $this->expectException(\Exception::class); but throwing TypeError will not (it does not extend Exception)

However both Exception and TypeError implement Throwable interface.

Therefore changing in both tests

$this->expectException(\Exception::class);

to

$this->expectException(\Throwable::class);

makes test green:

 ✔ Say too long
 ✔ Say data type

OK (2 tests, 2 assertions)

See the list of Errors and Exception classes and how they are related to each other.

Just to be clear: it is a good practice to use a specific exception or error for unit test instead of a generic Exception or Throwable but if you ever encounter that misleading comment about exception now you will know why PHPUnit's exception TypeError or other exception Errors are not in fact Exceptions but Throwable