How Should You Test Your Machine Learning Project? A Beginner’s Guide | by François Porcher | Jul, 2024

A pleasant introduction to testing machine studying tasks, through the use of normal libraries corresponding to Pytest and Pytest-cov

Testing is a vital element of software program growth, however in my expertise, it’s broadly uncared for in machine studying tasks. Plenty of folks know they need to check their code, however not many individuals know easy methods to do and truly do it.

This information goals to introduce you to the necessities of testing varied elements of a machine studying pipeline. We’ll concentrate on fine-tuning BERT for textual content classification on the IMDb dataset and utilizing the trade normal libraries like pytest and pytest-cov for testing.

I strongly advise you to observe the code on this Github repository:

Here’s a temporary overview of the venture.

bert-text-classification/
├── src/
│   ├── data_loader.py
│   ├── analysis.py
│   ├── fundamental.py
│   ├── coach.py
│   └── utils.py
├── checks/
│   ├── conftest.py
│   ├── test_data_loader.py
│   ├── test_evaluation.py
│   ├── test_main.py
│   ├── test_trainer.py
│   └── test_utils.py
├── fashions/
│   └── imdb_bert_finetuned.pth
├── surroundings.yml
├── necessities.txt
├── README.md
└── setup.py

A standard apply is to separate the code into a number of elements:

• src: incorporates the principle recordsdata we use to load the datasets, practice and consider fashions.
• checks: It incorporates completely different python scripts. More often than not, there’s one check file for every script. I personally use the next conference: if the script you wish to check known as XXX.py then the corresponding check script known as test_XXX.py and situated within the checks folder.

For instance if you wish to check the analysis.py file, I exploit the test_evaluation.py file.

NB: Within the checks folder, you may discover a conftest.py file. This file just isn’t testing operate per correct say, but it surely incorporates some configuration informations in regards to the check, particularly fixtures, that we’ll clarify a bit later.

You’ll be able to solely learn this text, however I strongly advise you to clone the repository and begin taking part in with the code, as we all the time study higher by being energetic. To take action, it’s good to clone the github repository, create an surroundings, and get a mannequin.

# clone github repo
git clone https://github.com/FrancoisPorcher/awesome-ai-tutorials/tree/fundamental
# enter corresponding folder
cd MLOps/how_to_test/
# create surroundings
conda env create -f surroundings.yml
conda activate how_to_test

Additionally, you will want a mannequin to run the evaluations. To breed my outcomes, you may run the principle file. The coaching ought to take between 2 and 20 min (relying you probably have CUDA, MPS, or a CPU).

python src/fundamental.py

If you do not need to fine-tune BERT (however I strongly advise you to high quality tune BERT your self), you may take a inventory model of BERT, and add a linear layer to get 2 lessons with the next command:

from transformers import BertForSequenceClassification
mannequin = BertForSequenceClassification.from_pretrained(
"bert-base-uncased", num_labels=2
)

Now you’re all set!

Let’s write some checks:

However first, a fast introduction to Pytest.

pytest is a typical and mature testing framework within the trade that makes it straightforward to put in writing checks.

One thing that’s superior with pytest is that you may check at completely different ranges of granularity: a single operate, a script, or your complete venture. Let’s learn to do the three choices.

What does a check appear like?

A check is a operate that checks the behaviour of an different operate. The conference is that if you wish to check the operate referred to as foo , you name your check operate test_foo .

We then outline a number of checks, to test whether or not the operate we’re testing is behaving as we would like.

Let’s use an instance to make clear concepts:

Within the data_loader.py script we’re utilizing a really normal operate referred to as clean_text , which removes capital letters and white areas, outlined as follows:

def clean_text(textual content: str) -> str:
"""
Clear the enter textual content by changing it to lowercase and stripping whitespace.
Args:
textual content (str): The textual content to wash.
Returns:
str: The cleaned textual content.
"""
return textual content.decrease().strip()

We wish to make it possible for this operate behaves properly, so within the test_data_loader.py file we are able to write a operate referred to as test_clean_text

from src.data_loader import clean_text
def test_clean_text():
# check capital letters
assert clean_text("HeLlo, WoRlD!") == "hey, world!" 
# check areas eliminated
assert clean_text("  Areas  ") == "areas"
# check empty string
assert clean_text("") == ""

Observe that we use the operate assert right here. If the assertion is True, nothing occurs, if it’s False, AssertionError is raised.

Now let’s name the check. Run the next command in your terminal.

pytest checks/test_data_loader.py::test_clean_text

This terminal command means that you’re utilizing pytest to run the check, most particularly the test_data_loader.py script situated within the checks folder, and also you solely wish to run one check which is test_clean_text .

If the check passes, that is what you need to get:

What occurs when a check doesn’t go?

For the sake of this instance let’s think about I modify the test_clean_text operate to this:

def clean_text(textual content: str) -> str:
# return textual content.decrease().strip()
return textual content.decrease()

Now the operate doesn’t take away areas anymore and goes to fail the checks. That is what we get when operating the check once more:

This time we all know why the check failed. Nice!

Why would we even wish to check a single operate?

Effectively, testing can take a whole lot of time. For a small venture like this one, evaluating on the entire IMDb dataset can already take a number of minutes. Generally we simply wish to check a single behaviour with out having to retest the entire codebase every time.

Now let’s transfer to the subsequent degree of granularity: testing a script.

The way to check an entire script?

Now let’s complexify our data_loader.py script and add a tokenize_text operate, which takes as enter a string, or a record of string, and outputs the tokenized model of the enter.

# src/data_loader.py
import torch
from transformers import BertTokenizer
def clean_text(textual content: str) -> str:
"""
Clear the enter textual content by changing it to lowercase and stripping whitespace.
Args:
textual content (str): The textual content to wash.
Returns:
str: The cleaned textual content.
"""
return textual content.decrease().strip()
def tokenize_text(
textual content: str, tokenizer: BertTokenizer, max_length: int
) -> Dict[str, torch.Tensor]:
"""
Tokenize a single textual content utilizing the BERT tokenizer.
Args:
textual content (str): The textual content to tokenize.
tokenizer (BertTokenizer): The tokenizer to make use of.
max_length (int): The utmost size of the tokenized sequence.
Returns:
Dict[str, torch.Tensor]: A dictionary containing the tokenized information.
"""
return tokenizer(
textual content,
padding="max_length",
truncation=True,
max_length=max_length,
return_tensors="pt",
)

Simply so you may perceive a bit extra what this operate does, let’s strive with an instance:

from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
txt = ["Hello, @! World! qwefqwef"]
tokenize_text(txt, tokenizer=tokenizer, max_length=16)

This can output the next end result:

{'input_ids': tensor([[ 101, 7592, 1010, 1030,  999, 2088,  999, 1053, 8545, 2546, 4160, 8545,2546,  102,    0,    0]]), 
'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 
'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]])}

• max_length: is the utmost size a sequence can have. On this case we selected 16, however we are able to see that the sequence is of size 14, so we are able to see that the two final tokens are padded.
• input_ids: Every token is transformed into its related id, that are the worlds which are a part of the vocabulary. NB: token 101 is the token CLS, and token_id 102 is the token SEP. These 2 tokens mark the start and the tip of a sentence. Learn the Consideration is all of your want paper for extra particulars.
• token_type_ids: It’s not essential. Should you feed 2 sequences as enter, you should have 1 values for the second sentence.
• attention_mask: This tells the mannequin which tokens it must attend within the self consideration mechanism. As a result of the sentence is padded, the eye mechanism doesn’t have to attend the two final tokens, so there are 0 there.

Now let’s write our test_tokenize_text operate that may test that the tokenize_text operate behaves correctly:

def test_tokenize_text():
"""
Check the tokenize_text operate to make sure it appropriately tokenizes textual content utilizing BERT tokenizer.
"""
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
# Instance enter texts
txt = ["Hello, @! World!", 
"Spaces    "]
# Tokenize the textual content
max_length = 128
res = tokenize_text(textual content=txt, tokenizer=tokenizer, max_length=max_length)
# let's check that the output is a dictionary and that the keys are appropriate
assert all(key in res for key in ["input_ids", "token_type_ids", "attention_mask"]), "Lacking keys within the output dictionary."
# let's test the scale of the output tensors
assert res["input_ids"].form[0] == len(txt), "Incorrect variety of input_ids."
assert res['input_ids'].form[1] == max_length, "Incorrect variety of tokens."
# let's test that every one the related tensors are pytorch tensors
assert all(isinstance(res[key], torch.Tensor) for key in res), "Not all values are PyTorch tensors."

Now let’s run the total check for the test_data_loader.py file, that now has 2 features:

• test_tokenize_text
• test_clean_text

You’ll be able to run the total check utilizing this command from terminal

pytest checks/test_data_loader.py

And you need to get this end result:

Congrats! You now know easy methods to check an entire script. Let’s transfer on to ultimate leve, testing the total codebase.

The way to check an entire codebase?

Persevering with the identical reasoning, we are able to write different checks for every script, and you need to have the same construction:

├── checks/
│   ├── conftest.py
│   ├── test_data_loader.py
│   ├── test_evaluation.py
│   ├── test_main.py
│   ├── test_trainer.py
│   └── test_utils.py

Now discover that in all these check features, some variables are fixed. For instance the tokenizer we use is similar throughout all scripts. Pytest has a pleasant strategy to deal with this with Fixtures.

Fixtures are a strategy to arrange some context or state earlier than operating checks and to wash up afterward. They supply a mechanism to handle check dependencies and inject reusable code into checks.

Fixtures are outlined utilizing the @pytest.fixture decorator.

The tokenizer is an efficient instance of fixture we are able to use. For that, let’s add it to theconftest.py file situated within the checks folder:

import pytest
from transformers import BertTokenizer
@pytest.fixture()
def bert_tokenizer():
"""Fixture to initialize the BERT tokenizer."""
return BertTokenizer.from_pretrained("bert-base-uncased")

And now within the test_data_loader.py file, we are able to name the fixture bert_tokenizer within the argument of test_tokenize_text.

def test_tokenize_text(bert_tokenizer):
"""
Check the tokenize_text operate to make sure it appropriately tokenizes textual content utilizing BERT tokenizer.
"""
tokenizer = bert_tokenizer
# Instance enter texts
txt = ["Hello, @! World!", 
"Spaces    "]
# Tokenize the textual content
max_length = 128
res = tokenize_text(textual content=txt, tokenizer=tokenizer, max_length=max_length)
# let's check that the output is a dictionary and that the keys are appropriate
assert all(key in res for key in ["input_ids", "token_type_ids", "attention_mask"]), "Lacking keys within the output dictionary."
# let's test the scale of the output tensors
assert res["input_ids"].form[0] == len(txt), "Incorrect variety of input_ids."
assert res['input_ids'].form[1] == max_length, "Incorrect variety of tokens."
# let's test that every one the related tensors are pytorch tensors
assert all(isinstance(res[key], torch.Tensor) for key in res), "Not all values are PyTorch tensors."

Fixtures are a really highly effective and versatile instrument. If you wish to study extra about them, the official doc is your go-to useful resource. However no less than now, you’ve the instruments at your disposal to cowl most ML testing.

Let’s run the entire codebase with the next command from the terminal:

pytest checks

And you need to get the next message:

Congratulations!

Within the earlier sections we’ve got discovered easy methods to check code. In massive tasks, you will need to measure the protection of your checks. In different phrases, how a lot of your code is examined.

pytest-cov is a plugin for pytest that generates check protection studies.

That being stated, don’t get fooled by the protection share. It isn’t as a result of you’ve 100% protection that your code is bug-free. It’s only a instrument so that you can determine which elements of your code want extra testing.

You’ll be able to run the next command to generate a protection report from terminal:

pytest --cov=src --cov-report=html checks/

And you need to get this:

Let’s have a look at easy methods to learn it:

1. Statements: complete variety of executable statements within the code. It counts all of the strains of code that may be executed, together with conditionals, loops, and performance calls.
2. Lacking: This means the variety of statements that weren’t executed through the check run. These are the strains of code that weren’t coated by any check.
3. Protection: share of the whole statements that have been executed through the checks. It’s calculated by dividing the variety of executed statements by the whole variety of statements.
4. Excluded: This refers back to the strains of code which were explicitly excluded from protection measurement. That is helpful for ignoring code that’s not related for check protection, corresponding to debugging statements.

We are able to see that the protection for the fundamental.py file is 0%, it’s regular, we didn’t write a test_main.py file.

We are able to additionally see that there’s solely 19% of the analysis code being examined, and it provides us an thought on the place we must always focus first.

Congratulations, you’ve made it!

Thanks for studying! Earlier than you go:

For extra superior tutorials, test my compilation of AI tutorials on Github

You ought to get my articles in your inbox. Subscribe here.

If you wish to have entry to premium articles on Medium, you solely want a membership for $5 a month. Should you enroll with my link, you help me with part of your charge with out extra prices.