Regular expressions

Previous: Dict techniques

Next: Python object model

Required course material for the lesson

Powerpoint: Regular expressions in Python
Video: An (unfortunately) true story
Resource: Example code - Regex
Video: Live Coding
PDF: Regular Expressions Cheat Sheet
WWW: Web page where you can test your regular expressions
Resource: Example code - exam form

Subjects covered

• Regular expressions, duh.
• Patterns, how to design and use them - and not use them.

Exercises to be handed in

Important - read this before starting
This exercise set will be similar to the format of the exam. The content will obviously be different.
You have to download this python file. It contains some frame work code, but mostly some unfinished functions. Each exercise is about finishing one of the function groups in the file. You can write the functions directly in the python file, or use VScode or other Jupyter Notebook editor to write it, but then it has to be copied over to the python file. You must hand in the finished python file, not a .ipynb file.
Inability to understand or perform this process will make you fail the exam, so it is worth spending some time on the procedure.
Exercise 6 to 8 has strong taste of something I would do at an exam. It is also an interesting beginning of the making a HIV vaccine. The data is real and the methods are real.

1. Make a function that accepts a string (X) as parameter. Use regular expressions (RE) to determine if the string is a number and outputs either "X is a number" or "X is not a number" depending on the string content. The goal is to do this with a SINGLE regex.
   These should all be considered as numbers: "4" "-7" "0.656" "-67.35555"
   These are not numbers: "5." "56F" ".32" "-.04" "1+1" "1-1"
   Note: The program is very simple, but it is likely the most difficult regular expression, you will have to make in this set of exercises. Perhaps you should do the following exercises before attempting this one - just to get some experience first.
   
   
2. Regex is often used for validation. This time make a function that takes a string as input, and checks if it is in a proper format for an email address. Output is "X is a valid email address" or "X is not a valid email address". Quite similar to the previous exercise, just a different thing to validate.
   
   
3. In the Functions lesson you made a fastaread() function in exercise 5. Start with inserting that function in the python file at the designated place. Now a function that accepts a filename as parameter and reads and verifies a fasta file. Use your fastaread() for the reading part. The verification here means that the function prints "filename is DNA fasta" or "filename is protein fasta" if the file is successfully verified for either dna or protein sequence, and "filename is not fasta" if unsuccessfully verified. Test the function with dna7.fsa and dnanoise.fsa. You can find a description of fasta format in Biological knowledge needed in the course. You are expected to know which symbols are used for DNA and protein sequence - otherwise look it up.
   You are welcome to use the fastaread() function in other exercises if appropriate.
   
   
4. In the Functions lesson you made a fastawrite() function in exercise 6. Start with inserting that function in the python file at the designated place. Building on your experience with the previous exercise, make a function that takes as input an input-fasta-filename and and output-fastafilename. The function reads the input fasta file, discard entries that do not conform to DNA or protein sequence, and rewrite (using your fastawrite() function) the acceptable entries in the output fasta file, in such a way that the normal 60 chars per line is followed with no spaces in between. The function must inform the user how many entries was kept and how many discarded. Hint: Test on dnanoise.fsa, which contains 3 entries that should be discarded (V00179, J00265, J02989).
   You are welcome to use the fastawrite() function in other exercises if appropriate.
   
   

5. 

   In the file alignment.fsa is a protein alignment of part of the insulin gene from different organisms. Make a function that reads the input fasta file and determine the consensus sequence of the alignment, which you return. The consensus sequence is simply the most frequent occurring amino acid on each position.
   Consensus: MALWMRLLPLLALLALWEPDPAGAFVNGHLCGSHLVEALYLVCGERGFFYTPKSRREVEDPGVGGLELGGGP
   
   
   
6. All HIV envelope proteins from various HIV strains in SwissProt have been identified and collected in the file HIVenvelope.txt. Make a function that accepts an input SwissProt filename and and output fasta filename, and uses regular repressions to extract the ID and the protein sequence from each entry in the input file and save them in an output fasta file named HIVenv.fsa. You did something similar back in the Stateful parsing lesson, exercise 6.
   
   
7. Continuing the investigation in HIV be creating a function that accepts an input fasta filename and and output epitope file name. The functions reads the input HIVenv.fsa fasta file and creates a single set consisting of all possible epitopes for all sequences. Save the epitopes in the output file HIVepitopes.txt - one epitope per line. An epitope is simply a k-mer 9 residues long, which can possibly elicit a immune system response. So save all unique 9-mers in the sequences in the file. To generate a file that you can verify against my file, you must sort the epitopes alphabetically before saving them.
   
   
8. You must prepare the epitopes in HIVepitopes.txt for machine learning. A common ML problem is when you have many data points (here epitopes) that look like each other. This introduces an unwanted bias in the ML predictions. Your job is to eliminate epitopes that are too similar using the Hobohm-1 algorithm. Hobohm-1 works like this: Look a the first epitope in the list. Compare it sequentially with the rest of the epitopes. If an epitope is too similar with the first, throw it away. Now no epitopes looks like the first. Proceed to the second epitope and repeat the comparing and possibly throwing away of the subsequent epitopes. Proceed to the third epitope in the list. Repeat this pattern until you have reached the end of the list. Now all epitopes left are dissimilar to each other. Create a function that accepts an input epitope file and an output epitope file to perform the job, possibly also creating other helper functions in the process.
   How to determine if two epitopes are too similar? Easy - you earlier learned about the Hamming distance, Functions exercise 1. Just compute the Hamming distance between two epitopes and if the distance is 3 or less, they are too similar. Save the "surviving" epitopes in the file HIVepitopesML.txt.
   Hint: Think about the Hobohm-1 algorithm before you implement it. You can easily run into problems.
   You should see a reduction from 15909 epitopes to 3842 in 30-60 seconds.