Exercise:AllergenicityPrediction ans

Answers

PART A: Investigating Allergenicity

• Q1: What would be your first approach to determine whether brown algae contain potential allergens?

Answer: A first approach would be to search a curated allergen database, such as the WHO/IUIS Allergen Nomenclature database, for experimentally characterized allergens from brown algae (Phaeophyceae) or closely related organisms.

If no experimentally characterized allergen is found, this does not demonstrate that brown algae are non-allergenic. It only means that no allergen has been reported in the database searched.

Potential allergenicity could then be investigated computationally using properties such as:

• sequence similarity to known allergens;
• membership in protein families containing known allergens;
• conserved domains or motifs;
• structural similarity to known allergens;
• protein stability and physicochemical properties; and
• predicted antibody-binding regions or other immunologically relevant features.

A simple starting point is therefore to compare a protein against known allergens using sequence similarity.

• Q2: Is there any other tool beyond NetAllergen that can be used as a proxy to predict allergenicity in silico?

Answer: Yes. One simple approach is BLAST. A protein sequence can be compared with experimentally characterized allergens to determine whether it is similar to a known allergen.

Sequence similarity may indicate that two proteins share a common evolutionary origin, protein family, structure or conserved regions. These similarities can make immunological cross-reactivity plausible.

However, sequence similarity alone does not demonstrate allergenicity or cross-reactivity.

Another possibility is to compare protein structures. Proteins can retain similar three-dimensional structures even when their sequence identity is relatively low. A predicted structure from tools such as AlphaFold could therefore be compared with structures of known allergens.

Thus, sequence similarity, protein-family membership and structural similarity can all be used as proxies for potential allergenicity, but experimental evidence is required to demonstrate an allergic response.

• Q3: Could immunotherapy potentially reduce allergic responses to proteins found in brown algae?

Answer: Potentially, but only if the brown-algae protein shares immunologically relevant features with the allergen targeted by the immunotherapy.

For cross-reactivity to be plausible, the proteins would need to contain sufficiently similar IgE-binding epitopes. High sequence similarity, structural similarity and membership in the same protein family could provide evidence supporting this hypothesis.

However, proteins can belong to the same family without showing clinically relevant cross-reactivity. Therefore, a similar brown-algae protein should be considered a candidate for further investigation, rather than a demonstrated cross-reactive allergen.

Experimental IgE-binding or inhibition experiments would be required to demonstrate cross-reactivity.

PART B: Can you beat BLAST?

• Q4: What will the model see? How is the data structured?

Answer: Each row of the dataset represents one protein. The Target column specifies whether the protein is an allergen (1) or non-allergen (0), while the Partition column specifies the sequence cluster-based partition used for model evaluation.

The Random Forest does not directly see the protein sequence. It receives numerical features calculated from the sequence.

The dataset contains several types of features, including:

• physicochemical properties, such as average residue weight, charge and isoelectric point;
• amino-acid group composition, such as polar, non-polar, aromatic, acidic and basic residues;
• hydrophobicity;
• predicted solvent accessibility;
• predicted disorder;
• predicted secondary structure;
• predicted MHC-binding properties;
• amino-acid composition; and
• autocorrelation-derived sequence features.

The Target and Partition columns are excluded from the input features.

There are 3894 proteins in the supplied dataset:

3206 non-allergens
688 allergens

The dataset therefore contains substantially more non-allergens than allergens.

• Q5: How well does it work?

Answer: The exact results depend slightly on the random seed and number of trees.

Using random_state = 42, 10 trees and evaluating all ten sequence partitions (0–9), the Random Forest gives approximately:

Metric	Score
Accuracy	0.835
Precision	0.616
Recall (Sensitivity)	0.137
Specificity	0.982
MCC	0.234
ROC-AUC	0.559

The confusion matrix is:

	Predicted non-allergen	Predicted allergen
Actual non-allergen	TN = 3152	FP = 54
Actual allergen	FN = 591	TP = 97

The high accuracy is therefore somewhat misleading. The model correctly identifies most non-allergens but misses a large fraction of the allergens.

• Q6: Where does the model make mistakes?

Answer: The main problem is false negatives.

Using the example above, 591 allergens are incorrectly predicted to be non-allergens, compared with only 54 non-allergens incorrectly predicted to be allergens.

For a screening tool for novel food proteins, a false negative is particularly problematic because:

allergen → predicted non-allergen

A potentially allergenic protein could therefore incorrectly be considered safe.

This problem is reflected by the model's low recall/sensitivity.

The model has very high specificity (~0.98) but low sensitivity (~0.14). It is therefore much better at recognizing non-allergens than at detecting allergens.

The class imbalance in the dataset (3206 non-allergens versus 688 allergens) may contribute to this behaviour.

• Q7: Why can AUC be more informative than accuracy?

Answer: Accuracy measures the fraction of correct classifications at a particular classification threshold. It can be misleading when the classes are imbalanced.

For example, because approximately 82% of the proteins in this dataset are non-allergens, a model could obtain relatively high accuracy by predicting most proteins as non-allergens while still failing to detect many allergens.

ROC-AUC instead measures how well the model separates allergens from non-allergens across different classification thresholds.

For the example model:

ROC-AUC ≈ 0.835

Thus, although the model has poor recall at the default classification threshold, its predicted probabilities contain useful information for separating the two classes.

An AUC of:

0.5 ≈ random discrimination
1.0 = perfect discrimination

Importantly, ROC-AUC should be calculated using the predicted probabilities rather than the final binary predictions.

• Q8: How does the number of estimators (10, 100, 1000) affect model performance?

Answer: Increasing the number of estimators means increasing the number of decision trees in the Random Forest.

Using random_state = 42, the results show that increasing the number of trees initially improves and stabilizes the model.

For example:

Number of trees	Accuracy	Precision	Recall	Specificity	MCC	ROC-AUC
10	0.834	0.642	0.141	0.983	0.245	0.835
100	0.839	0.746	0.137	0.990	0.273	0.892
1000	Results may vary slightly; run the notebook

Increasing the number of trees does not necessarily continue to improve the model indefinitely. Random Forest performance generally becomes more stable as more trees are added and eventually reaches a plateau.

The important conclusion is therefore not that "more trees are always better", but that enough trees are needed for stable predictions. Beyond this point, adding trees mainly increases computation time.

PART C: Can you beat the baseline?

There is no single correct answer to this section.

One possible solution is shown below.

• Q9: Provide a rationale for the change and report the new performance.

Example answer:

The baseline model has very low recall compared with specificity. One possible explanation is the strong class imbalance in the dataset:

3206 non-allergens
688 allergens

I therefore modified the Random Forest to give greater weight to the minority allergen class:

<syntaxhighlight lang="python"> rf_classifier = RandomForestClassifier(

   n_estimators=100,
   class_weight="balanced",
   random_state=42

) </syntaxhighlight>

The hypothesis is that increasing the importance of allergens during training may reduce the number of false negatives and improve sensitivity.

Other valid approaches include:

• tuning Random Forest parameters;
• testing another machine-learning algorithm;
• changing or selecting features;
• resampling the training data;
• using alternative sequence representations; or
• combining similarity-based and machine-learning predictions.

Students should report the same performance measures used for the baseline:

Metric	Baseline	Improved model
Accuracy
Precision
Recall
Specificity
MCC
ROC-AUC

---

• Q10: Did your modification improve allergenicity prediction?

Answer: This depends on the modification and the evaluation results.

A model should not be considered improved simply because its accuracy increased.

For example, a useful modification for allergen screening might increase recall and reduce false negatives while maintaining reasonable specificity. Improvements in MCC and ROC-AUC can provide additional evidence that overall discrimination has improved.

It is also possible to obtain a mixed result. For example:

Recall increases
Specificity decreases

This means that more allergens are detected, but more non-allergens are incorrectly classified as allergens.

Such a trade-off may be acceptable for a screening application, but the choice depends on the intended use of the predictor.

Conclusion

There is no single computational method that can demonstrate that a protein is an allergen.

Database and similarity-based approaches can identify experimentally characterized allergens and proteins related to known allergens.

Machine-learning approaches can identify combinations of protein properties associated with known allergens and use these patterns to predict new proteins.

Both approaches can therefore identify potential allergens that should be prioritized for further investigation.

However:

similarity ≠ cross-reactivity
prediction ≠ experimental evidence
high accuracy ≠ necessarily a useful model

The quality of an allergenicity predictor depends strongly on the training data, sequence redundancy, class balance, features and evaluation strategy.

Ultimately, experimental evidence is required to demonstrate allergenicity.