22113 - User contributions [en]

QT clustering

2025-05-15T12:34:15Z

WikiSysop: /* References */

__NOTOC__
===Description===
The program reads a number of data points (multi-dimensional vectors) from a file and partitions those into clusters. Clustering is important in discovering patterns or modes in multi-dimensional data sets. It is also a method of organizing data examples into similar groups (clusters). In this particular case, QT clustering partitions the data set such that each example (data point) is assigned to exactly one cluster. QT clustering is superior to K-means clustering in that the number of clusters is not given beforehand and it yields the same result in repeated runs. It requires more CPU time, though.

QT (Quality Threshold) has its name from the user-determined threshold (distance) of the maximal diameter of the clusters that the method computes.

While not strictly required, you could make your QT algorithm into a class. That would make it very easy to include and use in the future.

===Input and output===
The input is a tab separated file containing one data point on each line. Each data point is a vector consisting of a number of numbers. The program should handle any given vector size, but the vector size is constant in any data file. Input file example:

ex01 8.76 3.29 1.05
ex02 12.3 2.33 3.53
ex03 -0.54 -3.56 1.45
.
.

The output is all data points, partitioned in the clusters they belong to. Output example where each cluster starts with the cluster and is proceeded by the the members of that cluster:

Cluster-1
ex10 1.04 2.98 1.34
ex12 1.23 2.34 1.69
.
.
Cluster-2
ex04 -0.34 3.51 9.02
ex07 -8.56 5.12 12.5
.
.

===Examples of program execution===
cluster.py vectors.txt 500
The 500 is the maximum diameter for a cluster in the data set. An interesting twist could be to automatically decide the cluster diameter like this: X % of the distance between the two data point furthest away from each other. Called like this
cluster.py vectors.txt 30%

===Details===
The method works for any type of data set where it is possible to calculate a ''distance'' between any two points. In this project we are just considering euclidean distances, as they are simple.
[https://en.wikipedia.org/wiki/Pythagorean_theorem Pythagoras's theorem].

The algorithm works like this.
# For each point in the data set, calculate the ''candidate cluster'' with that point as the starting point. With '''n''' points in the data set, there are '''n''' candidate clusters, obviously.
# Choose the candidate cluster that contains most points as the primary cluster and remove those points from the data set. If two or more candidate clusters have equally most points, pick the cluster with the smallest diameter. If they are still equal, pick the first you found.
# Repeat step 1 and 2 until there are no points in the data set or a set limit has been reached; like all remaining candidate clusters has less than, say, 10 points and are therefore not true clusters, but noise.
# Print the resulting clusters.

A ''candidate cluster'' for a point is calculated using "complete linkage" like this:
# Consider the starting point as the beginning of the candidate cluster for that point. This is trivially seen as a subset of your data set.
# Add one point from your data set at a time in such a way that you extend the candidate cluster diameter the least. Again, if two points would extend the diameter the least, pick the first one you find.
# Continue adding points - that is repeat step 2 - to your candidate cluster until the diameter exceeds the Quality Threshold (hence the name QT clustering). The point that makes the diameter exceed the QT is not part of the candidate cluster.
Important definition: The diameter of a data set (or candidate cluster) is the distance of the two points furthest from each other. 
Note: All points in the data set can participate in multiple candidate clusters. Any point is not permanently assigned to a candidate cluster, before you actually pick the largest one and remove the "winners" points from the data set. 
Note: Building a candidate cluster according to above method is '''NOT''' the same method as just adding the nearest point to the starting point - or any point in the growing candidate cluster.

A fairly large part of this project is optimizing the algorithm just described. This is done by gaining insight in the algorithm - not calculating what does not need to be calculated, not calculating the same thing again and again.

Various data sets: [https://teaching.healthtech.dtu.dk/material/22113/point100.lst 100 data points], [https://teaching.healthtech.dtu.dk/material/22113/point1000.lst 1000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point3000.lst 3000 data points],
[https://teaching.healthtech.dtu.dk/material/22113/point4169.lst 4169 data points], [https://teaching.healthtech.dtu.dk/material/22113/point5000.lst 5000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point6000.lst 6000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point10000.lst 10000 data points].

Checking the correctness of your program. 
[https://teaching.healthtech.dtu.dk/material/22113/QT-small.lst Result] of clustering the small list (100 points) with QT being 30% of the diameter. 
[https://teaching.healthtech.dtu.dk/material/22113/result1000.lst Result] of clustering 1000 points with QT being 30% of the diameter.

The algorithm is deterministic - meaning that an implementation will yield the same result on the same data set every time. However, in the description there are two places, where "you pick the first one" you find. This is implementation dependent and therefore two different implementations of QT can give rise to different results. The data sets given here are constructed in such a way, that this will NOT happen for them, i.e. no matter how you implement your method you should get the same result.

===References===
# [http://genome.cshlp.org/content/9/11/1106.full.pdf+html Exploring expression data: identification and analysis of coexpressed genes. LJ Heyer, S Kruglyak, S Yooseph - Genome research, 1999 - genome.cshlp.org]
# [https://www.chem.agilent.com/cag/bsp/products/gsgx/Downloads/pdf/qt_clustering.pdf QT clustering in industry - Agilent]
Peter's speed reference using 30% as the threshold, using pure python:
Points Time (seconds) Improved
100 0 0
1000 2 1
3000 32 16
4169 65 33
5000 115 53
6000 149 74
10000 505 223
It is not required to achieve these numbers, but it is important to have a reference - and maybe a goal. The improved times have been reached by using efficient data types, not by any change in the method or computer.

QT clustering

2025-05-15T12:33:37Z

WikiSysop: /* References */

QT clustering

2025-04-22T13:12:58Z

WikiSysop: /* References */

__NOTOC__
===Description===
The program reads a number of data points (multi-dimensional vectors) from a file and partitions those into clusters. Clustering is important in discovering patterns or modes in multi-dimensional data sets. It is also a method of organizing data examples into similar groups (clusters). In this particular case, QT clustering partitions the data set such that each example (data point) is assigned to exactly one cluster. QT clustering is superior to K-means clustering in that the number of clusters is not given beforehand and it yields the same result in repeated runs. It requires more CPU time, though.

QT (Quality Threshold) has its name from the user-determined threshold (distance) of the maximal diameter of the clusters that the method computes.

While not strictly required, you could make your QT algorithm into a class. That would make it very easy to include and use in the future.

===Input and output===
The input is a tab separated file containing one data point on each line. Each data point is a vector consisting of a number of numbers. The program should handle any given vector size, but the vector size is constant in any data file. Input file example:

ex01 8.76 3.29 1.05
ex02 12.3 2.33 3.53
ex03 -0.54 -3.56 1.45
.
.

The output is all data points, partitioned in the clusters they belong to. Output example where each cluster starts with the cluster and is proceeded by the the members of that cluster:

Cluster-1
ex10 1.04 2.98 1.34
ex12 1.23 2.34 1.69
.
.
Cluster-2
ex04 -0.34 3.51 9.02
ex07 -8.56 5.12 12.5
.
.

===Examples of program execution===
cluster.py vectors.txt 500
The 500 is the maximum diameter for a cluster in the data set. An interesting twist could be to automatically decide the cluster diameter like this: X % of the distance between the two data point furthest away from each other. Called like this
cluster.py vectors.txt 30%

===Details===
The method works for any type of data set where it is possible to calculate a ''distance'' between any two points. In this project we are just considering euclidean distances, as they are simple.
[https://en.wikipedia.org/wiki/Pythagorean_theorem Pythagoras's theorem].

The algorithm works like this.
# For each point in the data set, calculate the ''candidate cluster'' with that point as the starting point. With '''n''' points in the data set, there are '''n''' candidate clusters, obviously.
# Choose the candidate cluster that contains most points as the primary cluster and remove those points from the data set. If two or more candidate clusters have equally most points, pick the cluster with the smallest diameter. If they are still equal, pick the first you found.
# Repeat step 1 and 2 until there are no points in the data set or a set limit has been reached; like all remaining candidate clusters has less than, say, 10 points and are therefore not true clusters, but noise.
# Print the resulting clusters.

A ''candidate cluster'' for a point is calculated using "complete linkage" like this:
# Consider the starting point as the beginning of the candidate cluster for that point. This is trivially seen as a subset of your data set.
# Add one point from your data set at a time in such a way that you extend the candidate cluster diameter the least. Again, if two points would extend the diameter the least, pick the first one you find.
# Continue adding points - that is repeat step 2 - to your candidate cluster until the diameter exceeds the Quality Threshold (hence the name QT clustering). The point that makes the diameter exceed the QT is not part of the candidate cluster.
Important definition: The diameter of a data set (or candidate cluster) is the distance of the two points furthest from each other. 
Note: All points in the data set can participate in multiple candidate clusters. Any point is not permanently assigned to a candidate cluster, before you actually pick the largest one and remove the "winners" points from the data set. 
Note: Building a candidate cluster according to above method is '''NOT''' the same method as just adding the nearest point to the starting point - or any point in the growing candidate cluster.

A fairly large part of this project is optimizing the algorithm just described. This is done by gaining insight in the algorithm - not calculating what does not need to be calculated, not calculating the same thing again and again.

Various data sets: [https://teaching.healthtech.dtu.dk/material/22113/point100.lst 100 data points], [https://teaching.healthtech.dtu.dk/material/22113/point1000.lst 1000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point3000.lst 3000 data points],
[https://teaching.healthtech.dtu.dk/material/22113/point4169.lst 4169 data points], [https://teaching.healthtech.dtu.dk/material/22113/point5000.lst 5000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point6000.lst 6000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point10000.lst 10000 data points].

Checking the correctness of your program. 
[https://teaching.healthtech.dtu.dk/material/22113/QT-small.lst Result] of clustering the small list (100 points) with QT being 30% of the diameter. 
[https://teaching.healthtech.dtu.dk/material/22113/result1000.lst Result] of clustering 1000 points with QT being 30% of the diameter.

The algorithm is deterministic - meaning that an implementation will yield the same result on the same data set every time. However, in the description there are two places, where "you pick the first one" you find. This is implementation dependent and therefore two different implementations of QT can give rise to different results. The data sets given here are constructed in such a way, that this will NOT happen for them, i.e. no matter how you implement your method you should get the same result.

===References===
# [http://genome.cshlp.org/content/9/11/1106.full.pdf+html Exploring expression data: identification and analysis of coexpressed genes. LJ Heyer, S Kruglyak, S Yooseph - Genome research, 1999 - genome.cshlp.org]
# [https://www.chem.agilent.com/cag/bsp/products/gsgx/Downloads/pdf/qt_clustering.pdf QT clustering in industry - Agilent]
Peter's speed reference using 30% as the threshold:
Points Time (seconds) Improved
100 0 0
1000 2 1
3000 32 20
4169 65 44
5000 115 70
6000 149 106
10000 505 342
It is not required to achieve these numbers, but it is important to have a reference - and maybe a goal.

QT clustering

2025-04-22T12:23:38Z

WikiSysop: /* Description */

__NOTOC__
===Description===
The program reads a number of data points (multi-dimensional vectors) from a file and partitions those into clusters. Clustering is important in discovering patterns or modes in multi-dimensional data sets. It is also a method of organizing data examples into similar groups (clusters). In this particular case, QT clustering partitions the data set such that each example (data point) is assigned to exactly one cluster. QT clustering is superior to K-means clustering in that the number of clusters is not given beforehand and it yields the same result in repeated runs. It requires more CPU time, though.

QT (Quality Threshold) has its name from the user-determined threshold (distance) of the maximal diameter of the clusters that the method computes.

While not strictly required, you could make your QT algorithm into a class. That would make it very easy to include and use in the future.

===Input and output===
The input is a tab separated file containing one data point on each line. Each data point is a vector consisting of a number of numbers. The program should handle any given vector size, but the vector size is constant in any data file. Input file example:

ex01 8.76 3.29 1.05
ex02 12.3 2.33 3.53
ex03 -0.54 -3.56 1.45
.
.

The output is all data points, partitioned in the clusters they belong to. Output example where each cluster starts with the cluster and is proceeded by the the members of that cluster:

Cluster-1
ex10 1.04 2.98 1.34
ex12 1.23 2.34 1.69
.
.
Cluster-2
ex04 -0.34 3.51 9.02
ex07 -8.56 5.12 12.5
.
.

===Examples of program execution===
cluster.py vectors.txt 500
The 500 is the maximum diameter for a cluster in the data set. An interesting twist could be to automatically decide the cluster diameter like this: X % of the distance between the two data point furthest away from each other. Called like this
cluster.py vectors.txt 30%

===Details===
The method works for any type of data set where it is possible to calculate a ''distance'' between any two points. In this project we are just considering euclidean distances, as they are simple.
[https://en.wikipedia.org/wiki/Pythagorean_theorem Pythagoras's theorem].

The algorithm works like this.
# For each point in the data set, calculate the ''candidate cluster'' with that point as the starting point. With '''n''' points in the data set, there are '''n''' candidate clusters, obviously.
# Choose the candidate cluster that contains most points as the primary cluster and remove those points from the data set. If two or more candidate clusters have equally most points, pick the cluster with the smallest diameter. If they are still equal, pick the first you found.
# Repeat step 1 and 2 until there are no points in the data set or a set limit has been reached; like all remaining candidate clusters has less than, say, 10 points and are therefore not true clusters, but noise.
# Print the resulting clusters.

A ''candidate cluster'' for a point is calculated using "complete linkage" like this:
# Consider the starting point as the beginning of the candidate cluster for that point. This is trivially seen as a subset of your data set.
# Add one point from your data set at a time in such a way that you extend the candidate cluster diameter the least. Again, if two points would extend the diameter the least, pick the first one you find.
# Continue adding points - that is repeat step 2 - to your candidate cluster until the diameter exceeds the Quality Threshold (hence the name QT clustering). The point that makes the diameter exceed the QT is not part of the candidate cluster.
Important definition: The diameter of a data set (or candidate cluster) is the distance of the two points furthest from each other. 
Note: All points in the data set can participate in multiple candidate clusters. Any point is not permanently assigned to a candidate cluster, before you actually pick the largest one and remove the "winners" points from the data set. 
Note: Building a candidate cluster according to above method is '''NOT''' the same method as just adding the nearest point to the starting point - or any point in the growing candidate cluster.

A fairly large part of this project is optimizing the algorithm just described. This is done by gaining insight in the algorithm - not calculating what does not need to be calculated, not calculating the same thing again and again.

Various data sets: [https://teaching.healthtech.dtu.dk/material/22113/point100.lst 100 data points], [https://teaching.healthtech.dtu.dk/material/22113/point1000.lst 1000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point3000.lst 3000 data points],
[https://teaching.healthtech.dtu.dk/material/22113/point4169.lst 4169 data points], [https://teaching.healthtech.dtu.dk/material/22113/point5000.lst 5000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point6000.lst 6000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point10000.lst 10000 data points].

Checking the correctness of your program. 
[https://teaching.healthtech.dtu.dk/material/22113/QT-small.lst Result] of clustering the small list (100 points) with QT being 30% of the diameter. 
[https://teaching.healthtech.dtu.dk/material/22113/result1000.lst Result] of clustering 1000 points with QT being 30% of the diameter.

The algorithm is deterministic - meaning that an implementation will yield the same result on the same data set every time. However, in the description there are two places, where "you pick the first one" you find. This is implementation dependent and therefore two different implementations of QT can give rise to different results. The data sets given here are constructed in such a way, that this will NOT happen for them, i.e. no matter how you implement your method you should get the same result.

===References===
# [http://genome.cshlp.org/content/9/11/1106.full.pdf+html Exploring expression data: identification and analysis of coexpressed genes. LJ Heyer, S Kruglyak, S Yooseph - Genome research, 1999 - genome.cshlp.org]
# [https://www.chem.agilent.com/cag/bsp/products/gsgx/Downloads/pdf/qt_clustering.pdf QT clustering in industry - Agilent]
Peter's speed reference using 30% as the threshold:
Points Time (seconds) Improved
100 0 0
1000 2 2
3000 32 20
4169 65 44
5000 115 70
6000 149 106
10000 505 342
It is not required to achieve these numbers, but it is important to have a reference - and maybe a goal.

QT clustering

2025-04-22T12:21:31Z

WikiSysop: /* Details */

__NOTOC__
===Description===
The program reads a number of data points (multi-dimensional vectors) from a file and partitions those into clusters. Clustering is important in discovering patterns or modes in multi-dimensional data sets. It is also a method of organizing data examples into similar groups (clusters). In this particular case, QT clustering partitions the data set such that each example (data point) is assigned to exactly one cluster. QT clustering is superior to K-means clustering in that the number of clusters is not given beforehand and it yields the same result in repeated runs. It requires more CPU time, though.

QT (Quality Threshold) has its name from the user-determined threshold (distance) of the maximal diameter of the clusters that the method computes.

===Input and output===
The input is a tab separated file containing one data point on each line. Each data point is a vector consisting of a number of numbers. The program should handle any given vector size, but the vector size is constant in any data file. Input file example:

ex01 8.76 3.29 1.05
ex02 12.3 2.33 3.53
ex03 -0.54 -3.56 1.45
.
.

The output is all data points, partitioned in the clusters they belong to. Output example where each cluster starts with the cluster and is proceeded by the the members of that cluster:

Cluster-1
ex10 1.04 2.98 1.34
ex12 1.23 2.34 1.69
.
.
Cluster-2
ex04 -0.34 3.51 9.02
ex07 -8.56 5.12 12.5
.
.

===Examples of program execution===
cluster.py vectors.txt 500
The 500 is the maximum diameter for a cluster in the data set. An interesting twist could be to automatically decide the cluster diameter like this: X % of the distance between the two data point furthest away from each other. Called like this
cluster.py vectors.txt 30%

===Details===
The method works for any type of data set where it is possible to calculate a ''distance'' between any two points. In this project we are just considering euclidean distances, as they are simple.
[https://en.wikipedia.org/wiki/Pythagorean_theorem Pythagoras's theorem].

The algorithm works like this.
# For each point in the data set, calculate the ''candidate cluster'' with that point as the starting point. With '''n''' points in the data set, there are '''n''' candidate clusters, obviously.
# Choose the candidate cluster that contains most points as the primary cluster and remove those points from the data set. If two or more candidate clusters have equally most points, pick the cluster with the smallest diameter. If they are still equal, pick the first you found.
# Repeat step 1 and 2 until there are no points in the data set or a set limit has been reached; like all remaining candidate clusters has less than, say, 10 points and are therefore not true clusters, but noise.
# Print the resulting clusters.

A ''candidate cluster'' for a point is calculated using "complete linkage" like this:
# Consider the starting point as the beginning of the candidate cluster for that point. This is trivially seen as a subset of your data set.
# Add one point from your data set at a time in such a way that you extend the candidate cluster diameter the least. Again, if two points would extend the diameter the least, pick the first one you find.
# Continue adding points - that is repeat step 2 - to your candidate cluster until the diameter exceeds the Quality Threshold (hence the name QT clustering). The point that makes the diameter exceed the QT is not part of the candidate cluster.
Important definition: The diameter of a data set (or candidate cluster) is the distance of the two points furthest from each other. 
Note: All points in the data set can participate in multiple candidate clusters. Any point is not permanently assigned to a candidate cluster, before you actually pick the largest one and remove the "winners" points from the data set. 
Note: Building a candidate cluster according to above method is '''NOT''' the same method as just adding the nearest point to the starting point - or any point in the growing candidate cluster.

A fairly large part of this project is optimizing the algorithm just described. This is done by gaining insight in the algorithm - not calculating what does not need to be calculated, not calculating the same thing again and again.

Various data sets: [https://teaching.healthtech.dtu.dk/material/22113/point100.lst 100 data points], [https://teaching.healthtech.dtu.dk/material/22113/point1000.lst 1000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point3000.lst 3000 data points],
[https://teaching.healthtech.dtu.dk/material/22113/point4169.lst 4169 data points], [https://teaching.healthtech.dtu.dk/material/22113/point5000.lst 5000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point6000.lst 6000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point10000.lst 10000 data points].

Checking the correctness of your program. 
[https://teaching.healthtech.dtu.dk/material/22113/QT-small.lst Result] of clustering the small list (100 points) with QT being 30% of the diameter. 
[https://teaching.healthtech.dtu.dk/material/22113/result1000.lst Result] of clustering 1000 points with QT being 30% of the diameter.

The algorithm is deterministic - meaning that an implementation will yield the same result on the same data set every time. However, in the description there are two places, where "you pick the first one" you find. This is implementation dependent and therefore two different implementations of QT can give rise to different results. The data sets given here are constructed in such a way, that this will NOT happen for them, i.e. no matter how you implement your method you should get the same result.

===References===
# [http://genome.cshlp.org/content/9/11/1106.full.pdf+html Exploring expression data: identification and analysis of coexpressed genes. LJ Heyer, S Kruglyak, S Yooseph - Genome research, 1999 - genome.cshlp.org]
# [https://www.chem.agilent.com/cag/bsp/products/gsgx/Downloads/pdf/qt_clustering.pdf QT clustering in industry - Agilent]
Peter's speed reference using 30% as the threshold:
Points Time (seconds) Improved
100 0 0
1000 2 2
3000 32 20
4169 65 44
5000 115 70
6000 149 106
10000 505 342
It is not required to achieve these numbers, but it is important to have a reference - and maybe a goal.

QT clustering

2025-04-22T12:17:15Z

WikiSysop: /* References */

__NOTOC__
===Description===
The program reads a number of data points (multi-dimensional vectors) from a file and partitions those into clusters. Clustering is important in discovering patterns or modes in multi-dimensional data sets. It is also a method of organizing data examples into similar groups (clusters). In this particular case, QT clustering partitions the data set such that each example (data point) is assigned to exactly one cluster. QT clustering is superior to K-means clustering in that the number of clusters is not given beforehand and it yields the same result in repeated runs. It requires more CPU time, though.

QT (Quality Threshold) has its name from the user-determined threshold (distance) of the maximal diameter of the clusters that the method computes.

===Input and output===
The input is a tab separated file containing one data point on each line. Each data point is a vector consisting of a number of numbers. The program should handle any given vector size, but the vector size is constant in any data file. Input file example:

ex01 8.76 3.29 1.05
ex02 12.3 2.33 3.53
ex03 -0.54 -3.56 1.45
.
.

The output is all data points, partitioned in the clusters they belong to. Output example where each cluster starts with the cluster and is proceeded by the the members of that cluster:

Cluster-1
ex10 1.04 2.98 1.34
ex12 1.23 2.34 1.69
.
.
Cluster-2
ex04 -0.34 3.51 9.02
ex07 -8.56 5.12 12.5
.
.

===Examples of program execution===
cluster.py vectors.txt 500
The 500 is the maximum diameter for a cluster in the data set. An interesting twist could be to automatically decide the cluster diameter like this: X % of the distance between the two data point furthest away from each other. Called like this
cluster.py vectors.txt 30%

===Details===
The method works for any type of data set where it is possible to calculate a ''distance'' between any two points. In this project we are just considering euclidean distances, as they are simple.
[https://en.wikipedia.org/wiki/Pythagorean_theorem Pythagoras's theorem].

The algorithm works like this.
# For each point in the data set, calculate the ''candidate cluster'' with that point as the starting point. With '''n''' points in the data set, there are '''n''' candidate clusters, obviously.
# Choose the candidate cluster that contains most points as the primary cluster and remove those points from the data set. If two or more candidate clusters have equally most points, pick the cluster with the smallest diameter. If they are still equal, pick the first you found.
# Repeat step 1 and 2 until there are no points in the data set or a set limit has been reached; like all remaining candidate clusters has less than, say, 10 points and are therefore not true clusters, but noise.
# Print the resulting clusters.

A ''candidate cluster'' for a point is calculated using "complete linkage" like this:
# Consider the starting point as the beginning of the candidate cluster for that point. This is trivially seen as a subset of your data set.
# Add one point from your data set at a time in such a way that you extend the candidate cluster diameter the least. Again, if two points would extend the diameter the least, pick the first one you find.
# Continue adding points - that is repeat step 2 - to your candidate cluster until the diameter exceeds the Quality Threshold (hence the name QT clustering). The point that makes the diameter exceed the QT is not part of the candidate cluster.
Important definition: The diameter of a data set (or candidate cluster) is the distance of the two points furthest from each other. 
Note: All points in the data set can participate in multiple candidate clusters. Any point is not permanently assigned to a candidate cluster, before you actually pick the largest one and remove the "winners" points from the data set. 
Note: Building a candidate cluster according to above method is '''NOT''' the same method as just adding the nearest point to the starting point - or any point in the growing candidate cluster.

A fairly large part of this project is optimizing the algorithm just described. This is done by gaining insight in the algorithm - not calculating what does not need to be calculated, not calculating the same thing again and again.

Various data sets: [https://teaching.healthtech.dtu.dk/material/22113/point100.lst 100 data points], [https://teaching.healthtech.dtu.dk/material/22113/point1000.lst 1000 data points],
[https://teaching.healthtech.dtu.dk/material/22113/point4169.lst 4169 data points], [https://teaching.healthtech.dtu.dk/material/22113/point5000.lst 5000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point6000.lst 6000 data points], [https://teaching.healthtech.dtu.dk/material/22113/point10000.lst 10000 data points].

Checking the correctness of your program. 
[https://teaching.healthtech.dtu.dk/material/22113/QT-small.lst Result] of clustering the small list (100 points) with QT being 30% of the diameter. 
[https://teaching.healthtech.dtu.dk/material/22113/result1000.lst Result] of clustering 1000 points with QT being 30% of the diameter.

The algorithm is deterministic - meaning that an implementation will yield the same result on the same data set every time. However, in the description there are two places, where "you pick the first one" you find. This is implementation dependent and therefore two different implementations of QT can give rise to different results. The data sets given here are constructed in such a way, that this will NOT happen for them, i.e. no matter how you implement your method you should get the same result.

===References===
# [http://genome.cshlp.org/content/9/11/1106.full.pdf+html Exploring expression data: identification and analysis of coexpressed genes. LJ Heyer, S Kruglyak, S Yooseph - Genome research, 1999 - genome.cshlp.org]
# [https://www.chem.agilent.com/cag/bsp/products/gsgx/Downloads/pdf/qt_clustering.pdf QT clustering in industry - Agilent]
Peter's speed reference using 30% as the threshold:
Points Time (seconds) Improved
100 0 0
1000 2 2
3000 32 20
4169 65 44
5000 115 70
6000 149 106
10000 505 342
It is not required to achieve these numbers, but it is important to have a reference - and maybe a goal.

Unit test - start of reverse polish notation class

2025-03-24T13:52:16Z

WikiSysop: Created page with "These are the two files, I made in class for unit test demo purposes. My original class in the file: ReversePolishCalc.py <pre> class ReversePolishCalc: def __init__(self): self.stack = list() def _checkstack(self, count): if len(self.stack) < count: raise IndexError("Stack does not contain enough elements to perform operaation") def push(self, vector): if isinstance(vector, (int, float, str)): vector = [..."

These are the two files, I made in class for unit test demo purposes.

My original class in the file: ReversePolishCalc.py
<pre>
class ReversePolishCalc:
def __init__(self):
self.stack = list()

def _checkstack(self, count):
if len(self.stack) < count:
raise IndexError("Stack does not contain enough elements to perform operaation")

def push(self, vector):
if isinstance(vector, (int, float, str)):
vector = [vector]
if not isinstance(vector, (list, tuple)):
raise ValueError("Input can not be understood as numbers")
for number in vector:
if isinstance(number, (int, float)):
self.stack.append(number)
elif isinstance(number, str):
try:
self.stack.append(int(number))
except ValueError:
try:
self.stack.append(float(number))
except ValueError:
raise ValueError("Input can not be understood as numbers")
else:
raise ValueError("Input can not be understood as numbers")

def pop(self):
self._checkstack(1)
return self.stack.pop()

def add(self):
self._checkstack(2)
self.stack[-2] += self.stack[-1]
del self.stack[-1]

def subtract(self):
self._checkstack(2)
self.stack[-2] -= self.stack[-1]
del self.stack[-1]

def multiply(self):
self._checkstack(2)
self.stack[-2] *= self.stack[-1]
del self.stack[-1]

def divide(self):
self._checkstack(2)
if self.stack[-1] == 0:
raise ZeroDivisionError
self.stack[-2] /= self.stack[-1]
del self.stack[-1]

def factorial(self):
self._checkstack(1)
no = int(self.stack[-1])
if no != self.stack[-1]:
raise ValueError("Factorial with floats is invalid")
if no < 0:
raise ValueError("Factorial can not be calcuated with negatives")
res = 1
for i in range(2, no+1):
res *= i
self.stack[-1] = res
</pre>

My test file of the class: test_ReversePolishCalc.py
<pre>
import pytest
from ReversePolishCalc import ReversePolishCalc as rpc

def test_push1():
# Arrange
calc = rpc()
# Act
calc.push(1)
# Assert
assert calc.stack[-1] == 1, "Simple push of integer 1"

def test_push2():
# Arrange
calc = rpc()
# Act
calc.push(1)
# Assert
assert calc.stack[0] == 1, "Simple push of integer 2"

def test_push3():
# Arrange
calc = rpc()
# Act
calc.push(1.2)
# Assert
assert calc.stack[0] == 1.2, "Push of float"

def test_push4():
# Arrange
calc = rpc()
# Act
calc.push("1")
# Assert
assert calc.stack[0] == 1, "Push of 1 as string"

def test_push5():
# Arrange
calc = rpc()
# Act
calc.push([1, 1.5, "2.5"])
# Assert
assert calc.stack == [1, 1.5, 2.5], "Advanced list push of ints, floats and strings"

@pytest.mark.parametrize("x,y", [(1,1), (1.2, 1.2), ("1.2", 1.2), ("1", 1)])
def test_push(x,y):
# Arrange
calc = rpc()
# Act
calc.push(x)
# Assert
assert calc.stack[0] == y, "Push of 1 as string"

def test_pop1():
# Arrange
calc = rpc()
calc.push(1)
# Act
num = calc.pop()
# Assert
assert num == 1

def test_failpop1():
# Arrange
calc = rpc()
# Act
with pytest.raises(IndexError):
num = calc.pop()
# Assert

</pre>

Unit test

2025-03-24T13:48:12Z

WikiSysop: /* Required course material for the lesson */

__NOTOC__
{| width=500 style="font-size: 10px; float:right; margin-left: 10px; margin-top: -56px;"
|Previous: [[Classes]]
|Next: [[Scientific Libraries, Pandas, Numpy]]
|}
== Required course material for the lesson ==
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22113/22113_08-Testing.ppt Testing] 
Online: [https://docs.pytest.org/ pytest documentation] 
Resource: [[Example code - Unit test]] 
Resource: [[Unit test - start of reverse polish notation class]] 
Blog: [https://www.joelonsoftware.com/2000/04/30/top-five-wrong-reasons-you-dont-have-testers/ On testing], by the founder of StackExchange. 

== Subjects covered ==
Overview of test methods 
Unit test using pytest framework.

== Exercises to be handed in ==
You should make a special folder for the exercises. I will refer to my special folder as ''unittest'' in these exercises. You will also see some ''__pycache__'' folders appear in places. This is Pythons cache for "compiled" programs. It is safe to ignore and also to delete, because it may become outdated.
# Use your factorial function from exercise 2 in [[Making Functions]]. If you did not do so already, change it to use exceptions instead of '''sys.exit()''', when an error occurs. Now make simple unit tests for the following test cases: 12, 2, 1, 0, -1, 3.0, 3.4, "3", "3.1.", "ABC". The factorial function and all test functions must be in one single file (''factorial_test.py'' in ''unittest''), which you can run ''pytest'' on.
# Now remove the factorial function from ''factorial_test.py'' and put it in its own file ''factorial.py''. Import it from the ''factorial_test.py'' like '''from factorial import factorial'''. The first factorial is the name of the .py file, the second factorial is the name of your factorial function. Just run ''pytest'' (no file name) in the folder to check it works. It is more normal to have test and function separated.
# Above we removed test code from function code by creating two files. Next, put the files in their own folder in ''unittest''. I would put my ''factorial.py'' in ''unittest/src'' and ''factorial_test.py'' in ''unittest/test''. This way there is a very clear separation between function and test. The problem is making sure the test code loads the function code. Do it wrong a couple of times - it is very instructive.
# Follow the file structure of having a ''code'' (or ''src'') folder for programs, a ''test'' folder for tests, and now a ''testdata'' folder for files containing test data. Now make unit tests and appropriate test data files for your '''fasta''' class from last week. In this exercise you just need to make unit test for the method '''load'''. You need to hand in both tests and test data. Maybe you should zip it all. Learn to zip :-)
# Add unit tests for the method '''save''' in your '''fasta''' class.
# Add unit tests for the method '''delete''' in your '''fasta''' class.
I would not be surprised if you find errors in your '''fasta''' class based on these tests. I found flaws in my code.

== Exercises for extra practice ==
# Add unit tests for all methods in your '''fasta''' class. That will be a bit of work.

Classes

2025-03-17T17:24:50Z

WikiSysop: /* Exercises to be handed in */

__NOTOC__
{| width=500 style="font-size: 10px; float:right; margin-left: 10px; margin-top: -56px;"
|Previous: [[Comprehension, Generators, Functions and Methods]]
|Next: [[Unit test]]
|}
== Required course material for the lesson ==
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22113/22113_07-Classes.ppt Classes] 
Resource: [[Example code - Classes]] 

== Subjects covered ==
Classes 
Object Oriented Programming

== Exercises to be handed in ==
The exercises will be about the making of a class. A the exercises goes on the class can do more and more. Remember to add code outside the class, which test the new methods you make as you make them. The class methods need some decent input control and you need to think about what is required. Notice how versatile the class becomes, as you progress through the exercises.
# Create a class called '''Fasta'''. The purpose of the class is to do various kinds of manipulation of fasta files and sequences. Start with adding 3 class methods: '''load(filename)''', which gets a file name/path and loads the fasta file into internal instance lists with header and sequence. '''save(filename)''', which writes the internal instance header/sequence lists into a fasta file. '''content()''', which returns two lists; the headers and the sequences. You can likely reuse part of your functions in exercise 3 and 4 from [[Making Functions]]. Example use of the Fasta class: <div style="font-family:'Courier New'">myfasta = Fasta() myfasta.load("dna7.fsa") print(myfasta.content()) myfasta.save("newfile.fsa")</div>
# Add a method '''delete([start[,end]])''', which deletes entries in the headers and sequences. If called with no arguments like '''delete()''' it deletes all headers/sequences. If called with one argument like '''delete(start)''' it deletes the header and sequence at position ''start''. If called with two arguments, it deletes headers and sequences from position ''start'' up to but not including position ''end''. ''start'' and ''end'' can also be negative numbers, and in such case we count from the end of the lists, just like it works in normal list manipulations.
# Modify the '''content()''' method, so it works similar to '''delete()'''. If called with no arguments like '''content()''' it returns 2 lists with all headers and sequences. If called with one argument like '''content(start)''' it returns the header and sequence at position ''start''. If called with two arguments, it returns the headers and sequences from position ''start'' up to but not including position ''end'' as two lists. Be careful that you return a copy of the lists and not the lists themselves as then the headers and sequences in the instance can be modified unintended outside the instance.
# Add a method '''insert(header,sequence[,position])''', which adds header and sequence to the instance lists. ''header'' and ''sequence'' can either be simple strings (single header and sequence) or lists of headers and sequences. If ''position'' is not given, then the addition takes place at the end of the existing headers/sequences. If ''position'' is given then insertion takes places at that position.
# Add a method '''verify(alphabet,[start[,end]])''', which verifies sequence entries according to an alphabet. It works in a way similar to '''delete()''' and '''content()''' for the range of what should be verified. The method returns True if all entries in the range are verified (belonging to the alphabet), False otherwise. The alphabet could be a DNA alphabet (ATCG), a protein alphabet or something derived from those. You should put your alphabets into class variables, as they are common for all instances, and not subject to change.
# Add a method '''discard(alphabet,[start[,end]])''', which discards sequence entries and corresponding headers if they do not match the given alphabet. It works in a way similar to '''delete()''' and '''content()''' for the range of what should be verified.
# Add iteration and length evaluation (__len__ magic method) to the '''Fasta''' class using magic methods. Do this so you can write code like <div style="font-family:'Courier New'">if len(MyFastaInstance) > 0:     for header, sequence in MyFastaInstance:         # Do something with header and sequence         print(header, sequence)</div>The function '''len''' returns the number of sequences.
# Add the methods '''deletethis()''', '''insertthis(header, sequence)''', '''verifythis(alphabet)''' and '''discardthis(alphabet)''' to the '''Fasta''' class. The methods should only work when ''iterating over an instance at the current item'', i.e. they work when you are iterating over the fasta sequences on the ''current'' sequence and header, like this:<div style="font-family:'Courier New'">for header, sequence in MyFastaInstance:     if not MyFastaInstance.verifythis("DNA"):         MyFastaInstance.deletethis()         continue     # Do something with header and sequence</div>As some may remember, it is normally impossible to successfully iterate straightforward through a list and delete and/or add elements to the list during the iteration. You have to make this possible, maybe by changing the way your iteration works in the previous exercise.

== Exercises for extra practice ==

Classes

2025-03-17T17:22:56Z

WikiSysop: /* Exercises to be handed in */

__NOTOC__
{| width=500 style="font-size: 10px; float:right; margin-left: 10px; margin-top: -56px;"
|Previous: [[Comprehension, Generators, Functions and Methods]]
|Next: [[Unit test]]
|}
== Required course material for the lesson ==
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22113/22113_07-Classes.ppt Classes] 
Resource: [[Example code - Classes]] 

== Subjects covered ==
Classes 
Object Oriented Programming

== Exercises to be handed in ==
The exercises will be about the making of a class. A the exercises goes on the class can do more and more. Remember to add code outside the class, which test the new methods you make as you make them. The class methods need some decent input control and you need to think about what is required. Notice how versatile the class becomes, as you progress through the exercises.
# Create a class called '''Fasta'''. The purpose of the class is to do various kinds of manipulation of fasta files and sequences. Start with adding 3 class methods: '''load(filename)''', which gets a file name/path and loads the fasta file into internal instance lists with header and sequence. '''save(filename)''', which writes the internal instance header/sequence lists into a fasta file. '''content()''', which returns two lists; the headers and the sequences. You can likely reuse part of your functions in exercise 3 and 4 from [[Making Functions]]. Example use of the Fasta class: <div style="font-family:'Courier New'">myfasta = Fasta() myfasta.load("dna7.fsa") print(myfasta.content()) myfasta.save("newfile.fsa")</div>
# Add a method '''delete([start[,end]])''', which deletes entries in the headers and sequences. If called with no arguments like '''delete()''' it deletes all headers/sequences. If called with one argument like '''delete(start)''' it deletes the header and sequence at position ''start''. If called with two arguments, it deletes headers and sequences from position ''start'' up to but not including position ''end''. ''start'' and ''end'' can also be negative numbers, and in such case we count from the end of the lists, just like it works in normal list manipulations.
# Modify the '''content()''' method, so it works similar to '''delete()'''. If called with no arguments like '''content()''' it returns 2 lists with all headers and sequences. If called with one argument like '''content(start)''' it returns the header and sequence at position ''start''. If called with two arguments, it returns the headers and sequences from position ''start'' up to but not including position ''end'' as two lists. Be careful that you return a copy of the lists and not the lists themselves as then the headers and sequences in the instance can be modified unintended outside the instance.
# Add a method '''insert(header,sequence[,position])''', which adds header and sequence to the instance lists. ''header'' and ''sequence'' can either be simple strings (single header and sequence) or lists of headers and sequences. If ''position'' is not given, then the addition takes place at the end of the existing headers/sequences. If ''position'' is given then insertion takes places at that position.
# Add a method '''verify(alphabet,[start[,end]])''', which verifies sequence entries according to an alphabet. It works in a way similar to '''delete()''' and '''content()''' for the range of what should be verified. The method returns True if all entries in the range are verified (belonging to the alphabet), False otherwise. The alphabet could be a DNA alphabet (ATCG), a protein alphabet or something derived from those. You should put your alphabets into class variables, as they are common for all instances, and not subject to change.
# Add a method '''discard(alphabet,[start[,end]])''', which discards sequence entries and corresponding headers if they do not match the given alphabet. It works in a way similar to '''delete()''' and '''content()''' for the range of what should be verified.
# Add iteration and length evaluation to the '''Fasta''' class using magic methods. Do this so you can write code like <div style="font-family:'Courier New'">if len(MyFastaInstance) > 0:     for header, sequence in MyFastaInstance:         # Do something with header and sequence         print(header, sequence)</div>The function '''len'' returns the number of sequences.
# Add the methods '''deletethis()''', '''insertthis(header, sequence)''', '''verifythis(alphabet)''' and '''discardthis(alphabet)''' to the '''Fasta''' class. The methods should only work when ''iterating over an instance at the current item'', i.e. they work when you are iterating over the fasta sequences on the ''current'' sequence and header, like this:<div style="font-family:'Courier New'">for header, sequence in MyFastaInstance:     if not MyFastaInstance.verifythis("DNA"):         MyFastaInstance.deletethis()         continue     # Do something with header and sequence</div>As some may remember, it is normally impossible to successfully iterate straightforward through a list and delete and/or add elements to the list during the iteration. You have to make this possible, maybe by changing the way your iteration works in the previous exercise.

== Exercises for extra practice ==

Classes

2025-03-17T17:20:44Z

WikiSysop: /* Exercises to be handed in */

__NOTOC__
{| width=500 style="font-size: 10px; float:right; margin-left: 10px; margin-top: -56px;"
|Previous: [[Comprehension, Generators, Functions and Methods]]
|Next: [[Unit test]]
|}
== Required course material for the lesson ==
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22113/22113_07-Classes.ppt Classes] 
Resource: [[Example code - Classes]] 

== Subjects covered ==
Classes 
Object Oriented Programming

== Exercises to be handed in ==
The exercises will be about the making of a class. A the exercises goes on the class can do more and more. Remember to add code outside the class, which test the new methods you make as you make them. The class methods need some decent input control and you need to think about what is required. Notice how versatile the class becomes, as you progress through the exercises.
# Create a class called '''Fasta'''. The purpose of the class is to do various kinds of manipulation of fasta files and sequences. Start with adding 3 class methods: '''load(filename)''', which gets a file name/path and loads the fasta file into internal instance lists with header and sequence. '''save(filename)''', which writes the internal instance header/sequence lists into a fasta file. '''content()''', which returns two lists; the headers and the sequences. You can likely reuse part of your functions in exercise 3 and 4 from [[Making Functions]]. Example use of the Fasta class: <div style="font-family:'Courier New'">myfasta = Fasta() myfasta.load("dna7.fsa") print(myfasta.content()) myfasta.save("newfile.fsa")</div>
# Add a method '''delete([start[,end]])''', which deletes entries in the headers and sequences. If called with no arguments like '''delete()''' it deletes all headers/sequences. If called with one argument like '''delete(start)''' it deletes the header and sequence at position ''start''. If called with two arguments, it deletes headers and sequences from position ''start'' up to but not including position ''end''. ''start'' and ''end'' can also be negative numbers, and in such case we count from the end of the lists, just like it works in normal list manipulations.
# Modify the '''content()''' method, so it works similar to '''delete()'''. If called with no arguments like '''content()''' it returns 2 lists with all headers and sequences. If called with one argument like '''content(start)''' it returns the header and sequence at position ''start''. If called with two arguments, it returns the headers and sequences from position ''start'' up to but not including position ''end'' as two lists. Be careful that you return a copy of the lists and not the lists themselves as then the headers and sequences in the instance can be modified unintended outside the instance.
# Add a method '''insert(header,sequence[,position])''', which adds header and sequence to the instance lists. ''header'' and ''sequence'' can either be simple strings (single header and sequence) or lists of headers and sequences. If ''position'' is not given, then the addition takes place at the end of the existing headers/sequences. If ''position'' is given then insertion takes places at that position.
# Add a method '''verify(alphabet,[start[,end]])''', which verifies sequence entries according to an alphabet. It works in a way similar to '''delete()''' and '''content()''' for the range of what should be verified. The method returns True if all entries in the range are verified (belonging to the alphabet), False otherwise. The alphabet could be a DNA alphabet (ATCG), a protein alphabet or something derived from those. You should put your alphabets into class variables, as they are common for all instances, and not subject to change.
# Add a method '''discard(alphabet,[start[,end]])''', which discards sequence entries and corresponding headers if they do not match the given alphabet. It works in a way similar to '''delete()''' and '''content()''' for the range of what should be verified.
# Add iteration and length evaluation to the '''Fasta''' class using magic methods. Do this so you can write code like <div style="font-family:'Courier New'">if len(MyFastaInstance) > 0:     for header, sequence in MyFastaInstance:         # Do something with header and sequence         print(header, sequence)</div>The method '''len'' returns the number of sequences.
# Add the methods '''deletethis()''', '''insertthis(header, sequence)''', '''verifythis(alphabet)''' and '''discardthis(alphabet)''' to the '''Fasta''' class. The methods should only work when ''iterating over an instance at the current item'', i.e. they work when you are iterating over the fasta sequences on the ''current'' sequence and header, like this:<div style="font-family:'Courier New'">for header, sequence in MyFastaInstance:     if not MyFastaInstance.verifythis("DNA"):         MyFastaInstance.deletethis()         continue     # Do something with header and sequence</div>As some may remember, it is normally impossible to successfully iterate straightforward through a list and delete and/or add elements to the list during the iteration. You have to make this possible, maybe by changing the way your iteration works in the previous exercise.

== Exercises for extra practice ==

Classes

2025-03-17T17:17:38Z

WikiSysop: /* Exercises to be handed in */

__NOTOC__
{| width=500 style="font-size: 10px; float:right; margin-left: 10px; margin-top: -56px;"
|Previous: [[Comprehension, Generators, Functions and Methods]]
|Next: [[Unit test]]
|}
== Required course material for the lesson ==
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22113/22113_07-Classes.ppt Classes] 
Resource: [[Example code - Classes]] 

== Subjects covered ==
Classes 
Object Oriented Programming

== Exercises to be handed in ==
The exercises will be about the making of a class. A the exercises goes on the class can do more and more. Remember to add code outside the class, which test the new methods you make as you make them. The class methods need some decent input control and you need to think about what is required. Notice how versatile the class becomes, as you progress through the exercises.
# Create a class called '''Fasta'''. The purpose of the class is to do various kinds of manipulation of fasta files and sequences. Start with adding 3 class methods: '''load(filename)''', which gets a file name/path and loads the fasta file into internal instance lists with header and sequence. '''save(filename)''', which writes the internal instance header/sequence lists into a fasta file. '''content()''', which returns two lists; the headers and the sequences. You can likely reuse part of your functions in exercise 3 and 4 from [[Making Functions]]. Example use of the Fasta class: <div style="font-family:'Courier New'">myfasta = Fasta() myfasta.load("dna7.fsa") print(myfasta.content()) myfasta.save("newfile.fsa")</div>
# Add a method '''delete([start[,end]])''', which deletes entries in the headers and sequences. If called with no arguments like '''delete()''' it deletes all headers/sequences. If called with one argument like '''delete(start)''' it deletes the header and sequence at position ''start''. If called with two arguments, it deletes headers and sequences from position ''start'' up to but not including position ''end''. ''start'' and ''end'' can also be negative numbers, and in such case we count from the end of the lists, just like it works in normal list manipulations.
# Modify the '''content()''' method, so it works similar to '''delete()'''. If called with no arguments like '''content()''' it returns 2 lists with all headers and sequences. If called with one argument like '''content(start)''' it returns the header and sequence at position ''start''. If called with two arguments, it returns the headers and sequences from position ''start'' up to but not including position ''end'' as two lists. Be careful that you return a copy of the lists and not the lists themselves as then the headers and sequences in the instance can be modified unintended outside the instance.
# Add a method '''insert(header,sequence[,position])''', which adds header and sequence to the instance lists. ''header'' and ''sequence'' can either be simple strings (single header and sequence) or lists of headers and sequences. If ''position'' is not given, then the addition takes place at the end of the existing headers/sequences. If ''position'' is given then insertion takes places at that position.
# Add a method '''verify(alphabet,[start[,end]])''', which verifies sequence entries according to an alphabet. It works in a way similar to '''delete()''' and '''content()''' for the range of what should be verified. The method returns True if all entries in the range are verified (belonging to the alphabet), False otherwise. The alphabet could be a DNA alphabet (ATCG), a protein alphabet or something derived from those. You should put your alphabets into class variables, as they are common for all instances, and not subject to change.
# Add a method '''discard(alphabet,[start[,end]])''', which discards sequence entries and corresponding headers if they do not match the given alphabet. It works in a way similar to '''delete()''' and '''content()''' for the range of what should be verified.
# Add iteration and length evaluation to the '''Fasta''' class using magic methods. Do this so you can write code like <div style="font-family:'Courier New'">if len(MyFastaInstance) > 0:     for header, sequence in MyFastaInstance:         # Do something with header and sequence         print(header, sequence)</div>
# Add the methods '''deletethis()''', '''insertthis(header, sequence)''', '''verifythis(alphabet)''' and '''discardthis(alphabet)''' to the '''Fasta''' class. The methods should only work when ''iterating over an instance at the current item'', i.e. they work when you are iterating over the fasta sequences on the ''current'' sequence and header, like this:<div style="font-family:'Courier New'">for header, sequence in MyFastaInstance:     if not MyFastaInstance.verifythis("DNA"):         MyFastaInstance.deletethis()         continue     # Do something with header and sequence</div>As some may remember, it is normally impossible to successfully iterate straightforward through a list and delete and/or add elements to the list during the iteration. You have to make this possible, maybe by changing the way your iteration works in the previous exercise.

== Exercises for extra practice ==

Classes

2025-03-17T17:15:59Z

WikiSysop: /* Exercises to be handed in */

__NOTOC__
{| width=500 style="font-size: 10px; float:right; margin-left: 10px; margin-top: -56px;"
|Previous: [[Comprehension, Generators, Functions and Methods]]
|Next: [[Unit test]]
|}
== Required course material for the lesson ==
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22113/22113_07-Classes.ppt Classes] 
Resource: [[Example code - Classes]] 

== Subjects covered ==
Classes 
Object Oriented Programming

== Exercises to be handed in ==
The exercises will be about the making of a class. A the exercises goes on the class can do more and more. Remember to add code outside the class, which test the new methods you make as you make them. The class methods need some decent input control and you need to think about what is required. Notice how versatile the class becomes, as you progress through the exercises.
# Create a class called '''Fasta'''. The purpose of the class is to do various kinds of manipulation of fasta files and sequences. Start with adding 3 class methods: '''load(filename)''', which gets a file name/path and loads the fasta file into internal instance lists with header and sequence. '''save(filename)''', which writes the internal instance header/sequence lists into a fasta file. '''content()''', which returns two lists; the headers and the sequences. You can likely reuse part of your functions in exercise 3 and 4 from [[Making Functions]]. Example use of the Fasta class: <div style="font-family:'Courier New'">myfasta = Fasta() myfasta.load("dna7.fsa") print(myfasta.content()) myfasta.save("newfile.fsa")</div>
# Add a method '''delete([start[,end]])''', which deletes entries in the headers and sequences. If called with no arguments like '''delete()''' it deletes all headers/sequences. If called with one argument like '''delete(start)''' it deletes the header and sequence at position ''start''. If called with two arguments, it deletes headers and sequences from position ''start'' up to but not including position ''end''. ''start'' and ''end'' can also be negative numbers, and in such case we count from the end of the lists, just like it works in normal list manipulations.
# Modify the '''content()''' method, so it works similar to '''delete()'''. If called with no arguments like '''content()''' it returns 2 lists with all headers and sequences. If called with one argument like '''content(start)''' it returns the header and sequence at position ''start''. If called with two arguments, it returns the headers and sequences from position ''start'' up to but not including position ''end'' as two lists. Be careful that you return a copy of the lists and not the lists themselves as then the headers and sequences in the instance can be modified unintended outside the instance.
# Add a method '''insert(header,sequence[,position])''', which adds header and sequence to the instance lists. ''header'' and ''sequence'' can either be simple strings (single header and sequence) or lists of headers and sequences. If ''position'' is not given, then the addition takes place at the end of the existing headers/sequences. If ''position'' is given then insertion takes places at that position.
# Add a method '''verify(alphabet,[start[,end]])''', which verifies sequence entries according to an alphabet. It works in a way similar to '''delete()''' and '''content()''' for the range of what should be verified. The method returns True if all entries in the range are verified, False otherwise. The alphabet could be a DNA alphabet (ATCG), a protein alphabet or something derived from those. You should put your alphabets into class variables, as they are common for all instances, and not subject to change.
# Add a method '''discard(alphabet,[start[,end]])''', which discards sequence entries and corresponding headers if they do not match the given alphabet. It works in a way similar to '''delete()''' and '''content()''' for the range of what should be verified.
# Add iteration and length evaluation to the '''Fasta''' class using magic methods. Do this so you can write code like <div style="font-family:'Courier New'">if len(MyFastaInstance) > 0:     for header, sequence in MyFastaInstance:         # Do something with header and sequence         print(header, sequence)</div>
# Add the methods '''deletethis()''', '''insertthis(header, sequence)''', '''verifythis(alphabet)''' and '''discardthis(alphabet)''' to the '''Fasta''' class. The methods should only work when ''iterating over an instance at the current item'', i.e. they work when you are iterating over the fasta sequences on the ''current'' sequence and header, like this:<div style="font-family:'Courier New'">for header, sequence in MyFastaInstance:     if not MyFastaInstance.verifythis("DNA"):         MyFastaInstance.deletethis()         continue     # Do something with header and sequence</div>As some may remember, it is normally impossible to successfully iterate straightforward through a list and delete and/or add elements to the list during the iteration. You have to make this possible, maybe by changing the way your iteration works in the previous exercise.

== Exercises for extra practice ==

Classes

2025-03-17T17:13:18Z

WikiSysop: /* Exercises to be handed in */

__NOTOC__
{| width=500 style="font-size: 10px; float:right; margin-left: 10px; margin-top: -56px;"
|Previous: [[Comprehension, Generators, Functions and Methods]]
|Next: [[Unit test]]
|}
== Required course material for the lesson ==
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22113/22113_07-Classes.ppt Classes] 
Resource: [[Example code - Classes]] 

== Subjects covered ==
Classes 
Object Oriented Programming

== Exercises to be handed in ==
The exercises will be about the making of a class. A the exercises goes on the class can do more and more. Remember to add code outside the class, which test the new methods you make as you make them. The class methods need some decent input control and you need to think about what is required. Notice how versatile the class becomes, as you progress through the exercises.
# Create a class called '''Fasta'''. The purpose of the class is to do various kinds of manipulation of fasta files and sequences. Start with adding 3 class methods: '''load(filename)''', which gets a file name/path and loads the fasta file into internal instance lists with header and sequence. '''save(filename)''', which writes the internal instance header/sequence lists into a fasta file. '''content()''', which returns two lists; the headers and the sequences. You can likely reuse part of your functions in exercise 3 and 4 from [[Making Functions]]. Example use of the Fasta class: <div style="font-family:'Courier New'">myfasta = Fasta() myfasta.load("dna7.fsa") print(myfasta.content()) myfasta.save("newfile.fsa")</div>
# Add a method '''delete([start[,end]])''', which deletes entries in the headers and sequences. If called with no arguments like '''delete()''' it deletes all headers/sequences. If called with one argument like '''delete(start)''' it deletes the header and sequence at position ''start''. If called with two arguments, it deletes headers and sequences from position ''start'' up to but not including position ''end''. ''start'' and ''end'' can also be negative numbers, and in such case we count from the end of the lists, just like it works in normal list manipulations.
# Modify the '''content()''' method, so it works similar to '''delete()'''. If called with no arguments like '''content()''' it returns 2 lists with all headers and sequences. If called with one argument like '''content(start)''' it returns the header and sequence at position ''start''. If called with two arguments, it returns the headers and sequences from position ''start'' up to but not including position ''end'' as two lists. Be careful that you return a copy of the lists and not the lists themselves as then the headers and sequences in the instance can be modified unintended outside the instance.
# Add a method '''insert(header,sequence[,position])''', which adds header and sequence to the instance lists. ''header'' and ''sequence'' can either be simple strings (single header and sequence) or lists of headers and sequences. If ''position'' is not given, then the addition takes place at the end of the existing headers/sequences. If ''position'' is given then insertion takes places at that position.
# Add a method '''verify(alphabet,[start[,end]])''', which verifies sequence entries according to an alphabet. It works in a way similar to '''delete()''' and '''content()''' for the range of what should be verified. The method returns True if all entries in the range are verified, False otherwise. The alphabet could be a DNA alphabet (ATCG), a protein alphabet or something derived from those. You should put your alphabets into class variables, as they are common for all instances, and not subject to change.
# Add a method '''discard(alphabet,[start[,end]])''', which discards sequence entries and corresponding headers if they do not match the given alphabet. It works in a way similar to '''delete()''' and '''content()''' for the range of what should be verified.
# Add iteration and length evaluation to the '''Fasta''' class using magic methods. Do this so you can write code like <div style="font-family:'Courier New'">if len(MyFastaInstance) > 0:     for header, sequence in MyFastaInstance:         # Do something with header and sequence         print(header, sequence)</div>
# Add the methods '''deletethis()''', '''insertthis(header, sequence)''', '''verifythis(alphabet)''' and '''discardthis(alphabet)''' to the '''Fasta''' class. The methods should only work when ''iterating over an instance at the current item'', i.e. they work when you are iterating over the fasta sequences on the ''current'' sequence and header, like this:<div style="font-family:'Courier New'">for header, sequence in MyFastaInstance:     if not MyFastaInstance.verifythis("DNA"):         MyFastaInstance.deletethis()         continue     # Do something with sequence and header</div>As some may remember, it is normally impossible to successfully iterate straightforward through a list and delete and/or add elements to the list during the iteration. You have to make this possible, maybe by changing the way your iteration works in the previous exercise.

== Exercises for extra practice ==

Regular Expressions

2025-03-17T17:06:11Z

WikiSysop: /* Exercises to be handed in */

__NOTOC__
{| width=500 style="font-size: 10px; float:right; margin-left: 10px; margin-top: -56px;"
|Previous: [[Python Recap and Objects]]
|Next: [[Making Functions]]
|}
== Required course material for the lesson ==
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22113/22113_03-Regex.ppt Regular expressions in Python] 
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=04cb2b80-d941-42a5-a632-af27012cd0d7 Regular Expressions] Monday 
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=738d525b-a6c1-4973-b6cc-af27012ca86e An (unfortunately) true story] Monday 
Resource: [[Example code - Regex]] 
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=2952d382-7059-4691-8170-af27012bc9a1 Live Coding] 
PDF: [https://teaching.healthtech.dtu.dk/material/22113/regular-expressions-cheat-sheet-v2.pdf Regular Expressions Cheat Sheet] 
WWW: [http://regex101.com/ Web page where you can test your regular expressions] 

== Subjects covered ==
* Regular expressions, duh.
* Patterns, how to design and use them.

== Exercises to be handed in ==
'''You might recognize some of these exercises. You must ONLY use regex for your pattern recognition and extraction of single data points (like an accession number).''' 
If you don't know what '''stateful parsing''' is, [https://teaching.healthtech.dtu.dk/22101/index.php/Stateful_Parsing look here]
# Make a program that accepts a string as input from the keyboard. Use regular expressions (RE) to determine if the input is a number. The goal is to do this with a SINGLE regex. These should all be considered as numbers: "4" "-7" "0.656" "-67.35555" These are not numbers: "5." "56F" ".32" "-.04" "1+1" Note: The program is very simple, but it is likely the most difficult regular expression, you will have to make in this set of exercises. Perhaps you should do the following exercises before attempting this one - just to get some experience first.
# Make a program that can read and verify a fasta file. Test with ''dna7.fsa'' and ''dnanoise.fsa''. Verification here means that the program prints "DNA fasta" or "Protein fasta" if the file is successfully verified for either dna or protein sequence, and "Not fasta" if unsuccessfully verified. You can find a description of fasta format in [[Biological knowledge needed in the course]]. You are expected to know which symbols are used for DNA and protein sequence - or that you are able to look it up. Hint: If you have made a program before (previous course) that reads a fasta file, this and the following exercise is not too hard, but otherwise you can consider doing them last.
# Change exercise 2 in the following way: Make the program discard entries that can not conform to DNA or protein sequence, and rewrite the acceptable entries in the output file ''fastaout.fsa'', in such a way that the normal 60 chars per line is followed with no spaces in between. The program must inform the user how many entries was kept and how many discarded. Test on ''dnanoise.fsa'', which contain 3 entries that should be discarded - this is a strong hint.
# The last exercises will all have to do with the files ''data1-4.gb'', which are various Genbank entries of genes. First you should study the files, notice the structure of the data. In all exercises you will have to parse (read and find the wanted data) the files using RE's which are very well designed for that purpose. This is a build-up process, so every exercise is added to the previous ones, so the final program can do a lot. Your program should be able to handle all files (so test them), but just one at a time.
# Extract the accession number, the definition and the organism (and print it).
# Extract and print all MEDLINE article numbers which are mentioned in the entries.
# Extract and print the translated gene (the amino acid sequence). Look for the line starting with /translation=. Generalize; An amino acid sequence can be short, i.e. only one line in the feature table, or long, i.e. more than one line in the feature table. Use stateful parsing.
# Extract and print the DNA (whole base sequence in the end of the file). Use stateful parsing.
# Extract and print ONLY the coding DNA. That is described in FEATURES - CDS (Coding DNA Sequence). As an example, the line in ''data1.gb'' says 'join(2424..2610,3397..3542)' and means that the coding sequence are bases 2424-2610 followed by bases 3397-3542. The bases in between are an intron and not a part of the coding DNA. Remember to generalize; there can be more (or less) than two exons, and the 'join' line can continue on the next line. Use stateful parsing.

== Exercises for extra practice ==

Comprehension, Generators, Functions and Methods

2025-03-16T06:12:51Z

WikiSysop: /* Exercises to be handed in */

__NOTOC__
{| width=500 style="font-size: 10px; float:right; margin-left: 10px; margin-top: -56px;"
|Previous: [[Advanced Data Structures and New Data Types]]
|Next: [[Classes]]
|}
== Required course material for the lesson ==
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22113/22113_06-Comprehension.ppt Comprehension, Generators, Functions and Methods] 
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ef476c74-cc82-4478-afd2-af270128c92f Comprehension] Monday 
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ad2b345c-4c9f-4fbb-8593-af270128ae40 Generators] Monday 
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ff23a72f-a1ee-46fc-a434-af2701281405 Iteration in detail, use of lambda function, libraries] Monday 
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=83c5ef87-3ed8-4069-b691-af130047ec9f How to parse bio files with many entries] 
Resource: [[Example code - Comprehension]] 
Resource: [[Example code - Misc]] 

Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=56f5c2c8-a3fe-4847-bb94-af2701286138 Live Coding 2] 

== Subjects covered ==
''Comprehension'', which is a way of manipulation/selecting data with a hidden loop. 
''Lambda'', the small anonymous function. 
''Generators'', which is like a function with memory of previous calls. 
More theoretical iteration. 
New functions and methods.

== Exercises to be handed in ==
# Make a program that calculates the product of two matrices and prints it on the screen (which is STDOUT, remember unix). The matrices are in the files ''mat1.dat'' and ''mat2.dat''. Numbers in the files are tab separated. A matrix should be stored as an list of lists. Advice: The program should have a function that reads a matrix from a given file (to be used twice), a function that calculates the product, and a function that prints a matrix. This way ensures that your program is easy to change to other forms of matrix calculations. Here are two links to the definition of matrix multiplication. [https://www.mathsisfun.com/algebra/matrix-multiplying.html Math is Fun] [http://mathworld.wolfram.com/MatrixMultiplication.html Math world]
# The purpose of this exercise is to find the 10 genes that has the biggest difference in expression between cancer and control patients in the ''dna-array.dat'' file after a linear transformation of the numbers in the columns. In order to not start from the beginning, use the file ''dna-array-norm.dat'' created in exercise 4 in [[Advanced Data Structures and New Data Types]] as input. The other tab-separated input file ''lineartransform.dat'' has an '''A''' (slope) and a '''B''' (intersection) - one AB pair for each number column in the ''dna-array-norm.dat'' file. For each line in ''dna-array-norm.dat'' you first linear transform the numbers according to the A & B in ''lineartransform.dat'' - first number uses first AB pair, second number uses second AB pair, and so forth. If your number is X, then the transformed number is A*X+B. When the entire line is transformed, you calculate the average of the cancer patients and the average of the controls. From that, find the 10 genes with the biggest difference in expression. There are a number of ways, but a simple one is to create a list of tuples with every tuple consisting of (gene name, cancer average, control average), and then sort the list according to the difference in cancer and control average. Using a lambda function when sorting springs to mind. Display the top 10 in the sorted list.
# Make a moving average generator: '''moving_avg(List_of_numbers, Window_size)'''. The generator calculates the average number in a window moving across the list. Try it on the numbers in ''ex1.dat'', i.e. load the numbers column-based into a single list first, i.e. first all the numbers in column 1, then the numbers in column 2, and so forth in the list.
# Make a trend discoverer generator: '''trend(List_of_numbers)'''. It looks at a list of numbers in a moving window way and emits 1, if the next number is higher than the previous, and 0 otherwise. Any longer sequence of 0's or 1's in the generator output is a trend in the data. Check with ''ex1.dat'' (load same way a previous exercise) or another file of your choosing.
# Changing the previous exercise: Make a '''find_trend(List_of_numbers, Minimum_trend_size)''' generator, which return a tuple '''(Position_Start, Size, Direction)''' of where and how big the trends in List_of_numbers are. '''Direction''' is 0 or 1 as you want to know which direction the trend is going. '''Position_start''' is the position in the (zero-based) list, where the trend starts. '''Size''' is how long the trend of ascending/descending numbers is. This is surprisingly difficult. Test with a simple file of your own making to check your results.
# Make a generator '''combinations()''', that takes a list of strings as input, e.g. '''combinations(["GAVIL", "ST", "NQ", "FWY", "D", "HKR"])''', and generates all possible combinations. A combination is formed by choosing 1 letter from the first string, 1 letter from the second string, and so forth, in that order, until a letter from all strings is chosen. The input list can have any number of strings and the strings can have any length (greater than 0). There must be NO REPEATS - random is not an acceptable library to use. As is obvious, the example has 5*2*2*3*1*3 = 180 different combinations, the first being GSNFDH. Print them all on the screen. If your input is ['0123456789', '0123456789', '0123456789'], then you will print the numbers from 000 to 999. Hint: A list of counters, 1 per string, could be useful in iterating through the combinations. When can such a generator be useful? If you want to generate a list of antigens, which needs certain amino acids to be in certain positions.

== Exercises for extra practice ==

Python Recap and Objects

2025-03-16T06:04:50Z

WikiSysop: /* Exercises to be handed in */

__NOTOC__
{| width=500 style="font-size: 10px; float:right; margin-left: 10px; margin-top: -56px;"
|Previous: [[Unix]]
|Next: [[Regular Expressions]]
|}
== Required course material for the lesson ==
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22113/22113_02-RecapObjects.ppt Python Recap and Objects] 
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22113/22113_02-Random.ppt Random numbers] 


== Subjects covered ==
* A short essential recap of Python learned course 22101.
* F-string formatting
* Command line arguments with Python
* The Python Object Model, and how it influences Python.
* Identity versus Equality

== Exercises to be handed in ==
'''In exercises 2-9 the job is to ''select'' the lines in the input file, i.e. the exercises are about which lines to select in various ways. The output is just a few of the lines in the inputfile.'''
# Make a handy little calculator, '''calc.py''', which takes 2 numbers and a operation from command line and displays the result: Here is an example of how it works:<pre>./calc.py 3 + 6</pre>The output should just be the result. You need to have at least the basic 4 operators working (+,-,/,*). If you want to, you can extend it to have more numbers than 2, like<pre>./calc.py 5 + 12 / 4</pre>An issue with operator precedence might appear. You might have difficulty with *, remember what you were just taught about unix.
# The input file ''scores.txt'' is a tab-separated file with an accession number in first column followed by 6 numbers (scores) between 0 and 1. You must find the accession numbers and scores (that means the entire line) of the 10 highest and 10 lowest "combined scores" (combined score is the metric for selection) and save the output in the file ''scoresextreme.txt''. The combined score is simply the 6 numbers added together. The order of the output must be from high to low. Take the name of input file and the output file from the command line, so the program is flexible.
# Change exercise 2 in the following way: There is an input file, ''negative_list.txt'', which is a list of genes which can NOT be part of the output. They are banned from your analysis. As can be seen, the genes are identified by their swissprot id. In order to translate from swissprot id to accession number so you can relate it to the ''scores.txt'', you must use the input file ''translation.txt'', where the first item on the line is a accession number, second item is the corresponding swissprot id.
# Change exercise 2 in the following way: Make the program work no matter how many numbers there are on every line. It must be the same number of numbers, i.e. in one file it could be 10 numbers on every line, in another file it could be 7 numbers per line.
# Change exercise 2 in the following way: Instead of using the combined score as the metric for selecting the accession numbers and scores, then use the average score as the metric. That will allow for having a different number of numbers on each line in the input file.
# Change exercise 2 in the following way: When you calculate the combined score the first number should weigh 50% more than the other numbers and the last should weigh 50% less.
# Change exercise 2 in the following way: When you calculate the combined score the numbers should be weighted after a linear sliding scale with the first number count for 50% more than its real value, sliding linearly down to the last number which is weighted 50% less. The weight is thus dynamically calculated according to how many numbers there are on the line and which position the number is on the line. '''N''' is the number of numbers, '''P''' is the position of the number, then the weight '''W''' is calculated as: W = 1.5 - (P-1)/(N-1) A more generic expression for the sliding weighting scale - '''B''' is the beginning weight, '''E''' is the ending weight: W = B - (B-E)*(P-1)/(N-1)
# Change exercise 2 in the following way: Just find the 10 lines in the input file with the highest combined scores. This should be easier than the original exercise.
# This is the same exercise as the previous (ex 8), however imagine that the input file is enormous - so big that you can not have it in memory. You still need to solve the problem, and this is done by having a running list of the best scores.

== Exercises for extra practice ==

22113/22163 - Unix & Python Programming for Bioinformaticians

2025-01-20T10:43:01Z

WikiSysop: /* Prepare for the course */

__NOTOC__
== Prepare for the course ==
You must read and follow the [[Course preparation]] before the you show up on the first day of the course. 
You are '''required''' to read at least the first part of [[Aligning expectations]] when the course starts and whenever you have a question related to the conduction of the course. 
Resources can be good to check out during the course, or when you need something more.

'''Teacher:''' [https://www.inside.dtu.dk/da/dtuinside/generelt/telefonbog/person?id=816&cpid=214027&tab=2&qt=dtupublicationquery Peter Wad Sackett], pwsa@dtu.dk 
'''Language:''' The course is taught in English. 
'''Tools:''' There is [[Course preparation]]. 
'''Textbooks:''' There are no text books for the course. I will make do with powerpoints and references to online resources. You can find the material under the individual lessons in the [[programme]]. 
'''Location:''' Building 116, aud. 83 

== Resources ==
'''Unix/Linux'''
* Youtube: [https://www.youtube.com/watch?v=HbgzrKJvDRw Linux File System/Structure Explained]
* Youtube: [https://www.youtube.com/watch?v=wBp0Rb-ZJak The Complete Linux Course: Beginner to Power User]
* Youtube: [https://www.youtube.com/playlist?list=PLIhvC56v63IJIujb5cyE13oLuyORZpdkL Linux series] by the very entertaining NetworkChuck
* Online: [http://www.oliverelliott.org/article/computing/tut_unix/ Online tutorial on unix]

'''Python 3'''
* Online: [https://www.coursera.org/learn/python Coursera course: Programming for Everybody] is a beginner course in Python. Everyone who wants to prepare more for course 22113 can start here. The [https://teaching.healthtech.dtu.dk/material/22113/CourseraPythonBook_270.pdf Coursera textbook]
* Online: [https://pynative.com/ PYnative] Good site for learning about Python. Information, tutorials, exercises and even online editor, all well explained in an accessible way.
* Youtube: [https://www.youtube.com/watch?v=rfscVS0vtbw Python beginner course]
* Online: [https://teaching.healthtech.dtu.dk/material/22113/clean_code.html Clean Code] by Lukasz Dynowski. An amazing read that is mandatory. Read it once around lesson 3 and once more around lesson 6.
* Online: [https://rosalind.info/problems/locations/ Rosalind project] Python exercises at different levels for practicing
* Book: ''Learning Python'', 5th ed. by Mark Lutz (O'Reilly) ISBN: 978-1-449-35573-9. This is the best Python book I have read. It covers all the basics and then some. All from the perspective of being a novice programmer. However, it is a brick; big, heavy and unwieldy. If you only want one Python book, then this should be the one. The course will not be taught from this book, but it could be good to have as a Python reference manual.
* Book: ''Python Crash Course: A Hands-On, Project-Based Introduction to Programming'' by Eric Matthes (No Starch Press) ISBN: 1593276036, 9781593276034. A pretty OK book which leads you into the Python world without too many distracting points and theoretical contemplation.
* Online: [https://docs.python.org/3/tutorial/ Official Python 3 tutorial]
* Online: [https://docs.python.org/3/reference/index.html Python 3 reference manual]
* Online: [https://docs.python.org/3/library/index.html Python 3 standard library]

'''Biological'''
* Info: [[Biological knowledge needed in the course]]

'''Writing reports, articles, thesis at university level'''
* PDF: [https://teaching.healthtech.dtu.dk/material/22113/Vejledning_i_opbygning_og_skrivning_af_rapporter_v.2013.3.pdf Vejledning i opbygning og skrivning af rapporter]
* PDF: [https://teaching.healthtech.dtu.dk/material/22113/how_to_write_reports.pdf Guide on how to write a report]
* Online: [https://www.elsevier.com/connect/11-steps-to-structuring-a-science-paper-editors-will-take-seriously How to structure a science paper]
* PDF: [https://www.imm.dtu.dk/~janba/MastersThesisAdvice.pdf Master Thesis Advice]
* Online: [https://thereader.mitpress.mit.edu/umberto-eco-how-to-write-a-thesis/ How to write a thesis]

'''Interesting but less teaching oriented material'''
* Blog: [https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ Why rewriting software projects can be bad]
* Online: [http://ivory.idyll.org/blog/big-data-biology.html Top 12 reasons you know you are a Big Data biologist]
* Online: [http://lifehacker.com/six-life-lessons-ive-learned-from-programming-1502077380 How programming and your life is similar]
* Youtube: [http://www.youtube.com/watch?v=nKIu9yen5nc What most schools don't teach - how to think]

== Archive of old course programmes ==
[[Programme - Spring 2023]] 
[[Programme - Spring 2024]]

Programme

2025-01-17T15:37:29Z

WikiSysop:

'''[[Collection of files]]''' used in the exercises and lessons - all gathered here.

* M 03/02 Lesson 1: [[Unix]]
* T 06/02 More lecture on [[Unix]]
* M 10/02 Lesson 2: [[Python Recap and Objects]]
* T 13/02 Official talk about Random numbers
* M 17/02 Lesson 3: [[Regular Expressions]]
* T 20/02 Continuing lesson
* M 24/02 Lesson 4: [[Making Functions]]
* T 27/02 Unofficial talk about Garbage Collection in Python
* M 03/03 Lesson 5: [[Advanced Data Structures and New Data Types]]
* T 06/03 Unofficial talk about how Data Structures work in Python. [https://evaluering.dtu.dk/ Midterm evaluation] - part 1 (this is an evaluation of the course, not of you)
* M 10/03 Lesson 6: [[Comprehension, Generators, Functions and Methods]]
* T 13/03 [https://evaluering.dtu.dk/ Midterm evaluation] - part 2
* M 17/03 Lesson 7: [[Classes]]
* T 20/03 More lecture on [[Classes]], Project introduction
* M 24/03 Lesson 8: [[Unit test]] and start of project
* T 27/03 More lecture on [[Unit test]], project work
* M 31/03 Lesson 9: [[Scientific Libraries, Pandas, Numpy]]
* T 03/04 More lecture on [[Scientific Libraries, Pandas, Numpy]], project work
* M 07/04 Lesson 11: [[Scientific Libraries, Statistics]] SciPy
* T 10/04 Continuing lesson, project work
* Easter holidays - Easter messes with the lesson order. Notice the switch between 10 & 11
* T 24/04 Lesson 10: [[Runtime evaluation of algorithms]]
* M 28/04 Lesson 12: [[Scientific Libraries, Plotting]] Matplotlib, Seaborn
* T 01/05 Continuing lesson, project work
* M 05/05 Lesson 13, [[Last words]], Biopython
* T 08/05 Q/A session, project work
* M 12/05 The project is handed in at 15.00 on DTU Learn
* T 13/05 Start of project evaluation
* M 20/05 The project evaluation is handed in at 15.00 on DTU Learn
* T 16/05 Exam - [http://eksamensplan.dtu.dk/ the official DTU exam plan]

== Curious about the exam ==
[https://teaching.healthtech.dtu.dk/22113/index.php/Aligning_expectations#Passing_the_course Read this], and if you fail or are afraid of failing [https://teaching.healthtech.dtu.dk/22113/index.php/Aligning_expectations#Failing_the_course read here].

Programme

2025-01-17T15:35:35Z

WikiSysop:

'''[[Collection of files]]''' used in the exercises and lessons - all gathered here.

* M 03/02 Lesson 1: [[Unix]]
* T 06/02 More lecture on [[Unix]]
* M 10/02 Lesson 2: [[Python Recap and Objects]]
* T 13/02 Official talk about Random numbers
* M 17/02 Lesson 3: [[Regular Expressions]]
* T 20/02 Continuing lesson
* M 24/02 Lesson 4: [[Making Functions]]
* T 27/02 Unofficial talk about Garbage Collection in Python
* M 03/03 Lesson 5: [[Advanced Data Structures and New Data Types]]
* T 06/03 Unofficial talk about how Data Structures work in Python. [https://evaluering.dtu.dk/ Midterm evaluation] - part 1 (this is an evaluation of the course, not of you)
* M 10/03 Lesson 6: [[Comprehension, Generators, Functions and Methods]]
* T 13/03 [https://evaluering.dtu.dk/ Midterm evaluation] - part 2
* M 17/03 Lesson 7: [[Classes]]
* T 20/03 More lecture on [[Classes]], Project introduction
* M 24/03 Lesson 8: [[Unit test]] and start of project
* T 27/03 More lecture on [[Unit test]], project work
* M 31/03 Lesson 9: [[Scientific Libraries, Pandas, Numpy]]
* T 03/04 More lecture on [[Scientific Libraries, Pandas, Numpy]], project work
* M 07/04 Lesson 11: [[Scientific Libraries, Statistics]] SciPy
* T 10/04 Continuing lesson, project work
* Easter holidays - Easter messes with the lesson order
* T 24/04 Lesson 10: [[Runtime evaluation of algorithms]]
* M 28/04 Lesson 12: [[Scientific Libraries, Plotting]] Matplotlib, Seaborn
* T 01/05 Continuing lesson, project work
* M 05/05 Lesson 13, [[Last words]], Biopython
* T 08/05 Q/A session, project work
* M 12/05 The project is handed in at 15.00 on DTU Learn
* T 13/05 Start of project evaluation
* M 20/05 The project evaluation is handed in at 15.00 on DTU Learn
* T 16/05 Exam - [http://eksamensplan.dtu.dk/ the official DTU exam plan]

== Curious about the exam ==
[https://teaching.healthtech.dtu.dk/22113/index.php/Aligning_expectations#Passing_the_course Read this], and if you fail or are afraid of failing [https://teaching.healthtech.dtu.dk/22113/index.php/Aligning_expectations#Failing_the_course read here].

Programme

2025-01-17T15:34:27Z

WikiSysop:

'''[[Collection of files]]''' used in the exercises and lessons - all gathered here.

* M 03/02 Lesson 1: [[Unix]]
* T 06/02 More lecture on [[Unix]]
* M 10/02 Lesson 2: [[Python Recap and Objects]]
* T 13/02 Official talk about Random numbers
* M 17/02 Lesson 3: [[Regular Expressions]]
* T 20/02 Continuing lesson
* M 24/02 Lesson 4: [[Making Functions]]
* T 27/02 Unofficial talk about Garbage Collection in Python
* M 03/03 Lesson 5: [[Advanced Data Structures and New Data Types]]
* T 06/03 Unofficial talk about how Data Structures work in Python. [https://evaluering.dtu.dk/ Midterm evaluation] - part 1 (this is an evaluation of the course, not of you)
* M 10/03 Lesson 6: [[Comprehension, Generators, Functions and Methods]]
* T 13/03 [https://evaluering.dtu.dk/ Midterm evaluation] - part 2
* M 17/03 Lesson 7: [[Classes]]
* T 20/03 More lecture on [[Classes]], Project introduction
* M 24/03 Lesson 8: [[Unit test]] and start of project
* T 27/03 More lecture on [[Unit test]], project work
* M 31/03 Lesson 9: [[Scientific Libraries, Pandas, Numpy]]
* T 03/04 More lecture on [[Scientific Libraries, Pandas, Numpy]], project work
* M 07/04 Lesson 11: [[Scientific Libraries, Statistics]] SciPy
* T 10/04 Continuing lesson, project work
* Easter holidays - Easter messes with the lesson order
* T 24/04 Lesson 10: [[Runtime evaluation of algorithms]]
* M 28/04 Lesson 12: [[Scientific Libraries, Plotting]] Matplotlib, Seaborn
* T 01/05 Continuing lesson, project work
* M 05/05 Lesson 13, [[Last words]], Biopython
* T 08/05 Q/A session, project work
* M 12/05 The project is handed in at 15.00 on DTU Learn
* T 13/05 Start of project evaluation
* M 20/05 The project evaluation is handed in at 15.00 on DTU Learn
* T 15/05 Exam - [http://eksamensplan.dtu.dk/ the official DTU exam plan]

== Curious about the exam ==
[https://teaching.healthtech.dtu.dk/22113/index.php/Aligning_expectations#Passing_the_course Read this], and if you fail or are afraid of failing [https://teaching.healthtech.dtu.dk/22113/index.php/Aligning_expectations#Failing_the_course read here].

Programme - Spring 2024

2025-01-17T15:12:07Z

WikiSysop: Created page with "'''Collection of files''' used in the exercises and lessons - all gathered here. * M 29/01 Lesson 1: Unix * T 01/02 More lecture on Unix * M 05/02 Lesson 2: Python Recap and Objects * T 08/02 Official talk about Random numbers * M 12/02 Lesson 3: Regular Expressions * T 15/02 Continuing lesson * M 19/02 Lesson 4: Making Functions * T 22/02 Unofficial talk about Garbage Collection in Python * M 26/02 Lesson 5: Advanced Data Structures and New Da..."

'''[[Collection of files]]''' used in the exercises and lessons - all gathered here.

* M 29/01 Lesson 1: [[Unix]]
* T 01/02 More lecture on [[Unix]]
* M 05/02 Lesson 2: [[Python Recap and Objects]]
* T 08/02 Official talk about Random numbers
* M 12/02 Lesson 3: [[Regular Expressions]]
* T 15/02 Continuing lesson
* M 19/02 Lesson 4: [[Making Functions]]
* T 22/02 Unofficial talk about Garbage Collection in Python
* M 26/02 Lesson 5: [[Advanced Data Structures and New Data Types]]
* T 29/02 Unofficial talk about how Data Structures work in Python. [https://evaluering.dtu.dk/ Midterm evaluation] - part 1 (this is an evaluation of the course, not of you)
* M 04/03 Lesson 6: [[Comprehension, Generators, Functions and Methods]]
* T 07/03 [https://evaluering.dtu.dk/ Midterm evaluation] - part 2
* M 11/03 Lesson 7: [[Classes]]
* T 14/03 More lecture on [[Classes]], Project introduction
* M 18/03 Lesson 8: [[Unit test]] and start of project
* T 21/03 More lecture on [[Unit test]], project work
* Easter holidays - Easter messes with the lesson order
* T 04/04 Lesson 10: [[Runtime evaluation of algorithms]]
* M 08/04 Lesson 9: [[Scientific Libraries, Pandas, Numpy]]
* T 11/04 More lecture on [[Scientific Libraries, Pandas, Numpy]], project work
* M 15/04 Lesson 11: [[Scientific Libraries, Statistics]] SciPy
* T 18/04 Continuing lesson, project work
* M 22/04 Lesson 12: [[Scientific Libraries, Plotting]] Matplotlib, Seaborn
* T 25/04 Continuing lesson, project work
* M 29/04 Lesson 13, [[Last words]], Biopython
* T 02/05 Q/A session, project work
* M 06/05 The project is handed in at 15.00 on DTU Learn
* T 07/05 Start of project evaluation
* M 13/05 The project evaluation is handed in at 15.00 on DTU Learn
* T 15/05 Exam - [http://eksamensplan.dtu.dk/ the official DTU exam plan]

== Curious about the exam ==
[https://teaching.healthtech.dtu.dk/22113/index.php/Aligning_expectations#Passing_the_course Read this], and if you fail or are afraid of failing [https://teaching.healthtech.dtu.dk/22113/index.php/Aligning_expectations#Failing_the_course read here].

22113/22163 - Unix & Python Programming for Bioinformaticians

2025-01-17T15:11:37Z

WikiSysop: /* Archive of old course programmes */

__NOTOC__
== Prepare for the course ==
You must read and follow the [[Course preparation]] before the you show up on the first day of the course. 
You are '''required''' to read at least the first part of [[Aligning expectations]] when the course starts and whenever you have a question related to the conduction of the course. 
Resources can be good to check out during the course, or when you need something more.

'''Teacher:''' [https://www.inside.dtu.dk/da/dtuinside/generelt/telefonbog/person?id=816&cpid=214027&tab=2&qt=dtupublicationquery Peter Wad Sackett], pwsa@dtu.dk 
'''Language:''' The course is taught in English. 
'''Tools:''' There is [[Course preparation]]. 
'''Textbooks:''' There are no text books for the course. I will make do with powerpoints and references to online resources. You can find the material under the individual lessons in the [[programme]]. 
'''Location:''' Building 116, aud. 82 NOTICE THIS LOCATION CHANGE 
'''Time:''' Monday 13:00 - 17:00, Thursday 9:00 - 12:00, module F2-A and F2-B. 
'''Deadlines for project work and exam:''' See the [[Programme]].

== Course details ==
There are no plans for streaming the lectures as there already are recorded video lectures for first half of the course. Discord is used for online help and discussion - if necessary.

{|
| [[Programme]] || Spring 2024
|-
| [[Aligning expectations]] || Required reading
|-
| [[Code construction]] || Required reading for peer evaluation
|-
| [[Project list]] || Of projects to do
|-
| [[Mini projects]] || For practicing programming
|}


== Resources ==
'''Unix/Linux'''
* Youtube: [https://www.youtube.com/watch?v=HbgzrKJvDRw Linux File System/Structure Explained]
* Youtube: [https://www.youtube.com/watch?v=wBp0Rb-ZJak The Complete Linux Course: Beginner to Power User]
* Youtube: [https://www.youtube.com/playlist?list=PLIhvC56v63IJIujb5cyE13oLuyORZpdkL Linux series] by the very entertaining NetworkChuck
* Online: [http://www.oliverelliott.org/article/computing/tut_unix/ Online tutorial on unix]

'''Python 3'''
* Online: [https://www.coursera.org/learn/python Coursera course: Programming for Everybody] is a beginner course in Python. Everyone who wants to prepare more for course 22113 can start here. The [https://teaching.healthtech.dtu.dk/material/22113/CourseraPythonBook_270.pdf Coursera textbook]
* Online: [https://pynative.com/ PYnative] Good site for learning about Python. Information, tutorials, exercises and even online editor, all well explained in an accessible way.
* Youtube: [https://www.youtube.com/watch?v=rfscVS0vtbw Python beginner course]
* Online: [https://teaching.healthtech.dtu.dk/material/22113/clean_code.html Clean Code] by Lukasz Dynowski. An amazing read that is mandatory. Read it once around lesson 3 and once more around lesson 6.
* Online: [https://rosalind.info/problems/locations/ Rosalind project] Python exercises at different levels for practicing
* Book: ''Learning Python'', 5th ed. by Mark Lutz (O'Reilly) ISBN: 978-1-449-35573-9. This is the best Python book I have read. It covers all the basics and then some. All from the perspective of being a novice programmer. However, it is a brick; big, heavy and unwieldy. If you only want one Python book, then this should be the one. The course will not be taught from this book, but it could be good to have as a Python reference manual.
* Book: ''Python Crash Course: A Hands-On, Project-Based Introduction to Programming'' by Eric Matthes (No Starch Press) ISBN: 1593276036, 9781593276034. A pretty OK book which leads you into the Python world without too many distracting points and theoretical contemplation.
* Online: [https://docs.python.org/3/tutorial/ Official Python 3 tutorial]
* Online: [https://docs.python.org/3/reference/index.html Python 3 reference manual]
* Online: [https://docs.python.org/3/library/index.html Python 3 standard library]

'''Biological'''
* Info: [[Biological knowledge needed in the course]]

'''Writing reports, articles, thesis at university level'''
* PDF: [https://teaching.healthtech.dtu.dk/material/22113/Vejledning_i_opbygning_og_skrivning_af_rapporter_v.2013.3.pdf Vejledning i opbygning og skrivning af rapporter]
* PDF: [https://teaching.healthtech.dtu.dk/material/22113/how_to_write_reports.pdf Guide on how to write a report]
* Online: [https://www.elsevier.com/connect/11-steps-to-structuring-a-science-paper-editors-will-take-seriously How to structure a science paper]
* PDF: [https://www.imm.dtu.dk/~janba/MastersThesisAdvice.pdf Master Thesis Advice]
* Online: [https://thereader.mitpress.mit.edu/umberto-eco-how-to-write-a-thesis/ How to write a thesis]

'''Interesting but less teaching oriented material'''
* Blog: [https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ Why rewriting software projects can be bad]
* Online: [http://ivory.idyll.org/blog/big-data-biology.html Top 12 reasons you know you are a Big Data biologist]
* Online: [http://lifehacker.com/six-life-lessons-ive-learned-from-programming-1502077380 How programming and your life is similar]
* Youtube: [http://www.youtube.com/watch?v=nKIu9yen5nc What most schools don't teach - how to think]

== Archive of old course programmes ==
[[Programme - Spring 2023]] 
[[Programme - Spring 2024]]

22113/22163 - Unix & Python Programming for Bioinformaticians

2025-01-17T15:11:23Z

WikiSysop: /* Archive of old course programmes */

__NOTOC__
== Prepare for the course ==
You must read and follow the [[Course preparation]] before the you show up on the first day of the course. 
You are '''required''' to read at least the first part of [[Aligning expectations]] when the course starts and whenever you have a question related to the conduction of the course. 
Resources can be good to check out during the course, or when you need something more.

'''Teacher:''' [https://www.inside.dtu.dk/da/dtuinside/generelt/telefonbog/person?id=816&cpid=214027&tab=2&qt=dtupublicationquery Peter Wad Sackett], pwsa@dtu.dk 
'''Language:''' The course is taught in English. 
'''Tools:''' There is [[Course preparation]]. 
'''Textbooks:''' There are no text books for the course. I will make do with powerpoints and references to online resources. You can find the material under the individual lessons in the [[programme]]. 
'''Location:''' Building 116, aud. 82 NOTICE THIS LOCATION CHANGE 
'''Time:''' Monday 13:00 - 17:00, Thursday 9:00 - 12:00, module F2-A and F2-B. 
'''Deadlines for project work and exam:''' See the [[Programme]].

== Course details ==
There are no plans for streaming the lectures as there already are recorded video lectures for first half of the course. Discord is used for online help and discussion - if necessary.

{|
| [[Programme]] || Spring 2024
|-
| [[Aligning expectations]] || Required reading
|-
| [[Code construction]] || Required reading for peer evaluation
|-
| [[Project list]] || Of projects to do
|-
| [[Mini projects]] || For practicing programming
|}


== Resources ==
'''Unix/Linux'''
* Youtube: [https://www.youtube.com/watch?v=HbgzrKJvDRw Linux File System/Structure Explained]
* Youtube: [https://www.youtube.com/watch?v=wBp0Rb-ZJak The Complete Linux Course: Beginner to Power User]
* Youtube: [https://www.youtube.com/playlist?list=PLIhvC56v63IJIujb5cyE13oLuyORZpdkL Linux series] by the very entertaining NetworkChuck
* Online: [http://www.oliverelliott.org/article/computing/tut_unix/ Online tutorial on unix]

'''Python 3'''
* Online: [https://www.coursera.org/learn/python Coursera course: Programming for Everybody] is a beginner course in Python. Everyone who wants to prepare more for course 22113 can start here. The [https://teaching.healthtech.dtu.dk/material/22113/CourseraPythonBook_270.pdf Coursera textbook]
* Online: [https://pynative.com/ PYnative] Good site for learning about Python. Information, tutorials, exercises and even online editor, all well explained in an accessible way.
* Youtube: [https://www.youtube.com/watch?v=rfscVS0vtbw Python beginner course]
* Online: [https://teaching.healthtech.dtu.dk/material/22113/clean_code.html Clean Code] by Lukasz Dynowski. An amazing read that is mandatory. Read it once around lesson 3 and once more around lesson 6.
* Online: [https://rosalind.info/problems/locations/ Rosalind project] Python exercises at different levels for practicing
* Book: ''Learning Python'', 5th ed. by Mark Lutz (O'Reilly) ISBN: 978-1-449-35573-9. This is the best Python book I have read. It covers all the basics and then some. All from the perspective of being a novice programmer. However, it is a brick; big, heavy and unwieldy. If you only want one Python book, then this should be the one. The course will not be taught from this book, but it could be good to have as a Python reference manual.
* Book: ''Python Crash Course: A Hands-On, Project-Based Introduction to Programming'' by Eric Matthes (No Starch Press) ISBN: 1593276036, 9781593276034. A pretty OK book which leads you into the Python world without too many distracting points and theoretical contemplation.
* Online: [https://docs.python.org/3/tutorial/ Official Python 3 tutorial]
* Online: [https://docs.python.org/3/reference/index.html Python 3 reference manual]
* Online: [https://docs.python.org/3/library/index.html Python 3 standard library]

'''Biological'''
* Info: [[Biological knowledge needed in the course]]

'''Writing reports, articles, thesis at university level'''
* PDF: [https://teaching.healthtech.dtu.dk/material/22113/Vejledning_i_opbygning_og_skrivning_af_rapporter_v.2013.3.pdf Vejledning i opbygning og skrivning af rapporter]
* PDF: [https://teaching.healthtech.dtu.dk/material/22113/how_to_write_reports.pdf Guide on how to write a report]
* Online: [https://www.elsevier.com/connect/11-steps-to-structuring-a-science-paper-editors-will-take-seriously How to structure a science paper]
* PDF: [https://www.imm.dtu.dk/~janba/MastersThesisAdvice.pdf Master Thesis Advice]
* Online: [https://thereader.mitpress.mit.edu/umberto-eco-how-to-write-a-thesis/ How to write a thesis]

'''Interesting but less teaching oriented material'''
* Blog: [https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ Why rewriting software projects can be bad]
* Online: [http://ivory.idyll.org/blog/big-data-biology.html Top 12 reasons you know you are a Big Data biologist]
* Online: [http://lifehacker.com/six-life-lessons-ive-learned-from-programming-1502077380 How programming and your life is similar]
* Youtube: [http://www.youtube.com/watch?v=nKIu9yen5nc What most schools don't teach - how to think]

== Archive of old course programmes ==
[[Programme - Spring 2023]]
[[Programme - Spring 2024]]

2024-04-04T13:42:14Z

WikiSysop:

Remember to [https://teaching.healthtech.dtu.dk/22113/index.php/Aligning_expectations#Projects,_general_information read about projects here].

The list is arranged approximately from lowest to highest difficulty.
# [[Text mining MEDLINE abstracts]]
# [[K-means clustering]]
# [[Data mining in NCBI databases]]
# [[Searching for motifs in sequences]]
# [[Data analysis]]
# [[Resistance to antibiotics]]
# [[k-nearest neighbor (k-NN) continuous variable estimation]]
# [[Read trimmer for Next-Generation-Sequencing data]]
# [[QT clustering]]
# [[Pairwise alignment]]
# [[Artificial Neural Network]]

Example code - Unit test

2024-03-20T10:04:54Z

WikiSysop:

I decided to make some unit tests for my prime number generator, I used as an example last week, see [[Example code - Classes]]
I make two files - one file containing the class ''PrimeGenerator.py'' and one file containing the tests, ''test_PrimeGenerator.py''. The files are supposed to be in the same folder.
I will not show the code from last week in ''PrimeGenerator.py'', if you want to see the code, click the above link.


This is my unit test file. I want to test the following:
* It can generate different series of primes in a not-ascending order (i.e 10, 20, 15)
* It can figure out if a number is a prime, especially around 0.
* It reacts correctly on wrong (nonsense) input
'''File:''' test_PrimeGenerator.py
<pre>
import pytest
from PrimeGenerator import PrimeGenerator

# Parametric test of the generator part
@pytest.mark.parametrize("x, y", [(10,(2,3,5,7)), (20,(2,3,5,7,11,13,17,19)), (15,(2,3,5,7,11,13))])

def test_generator(x, y):
result = tuple(PrimeGenerator(x))
assert result == y, "Generator test"

# Parametric test of the giving illegal data to the generator
@pytest.mark.parametrize("x", [1.3, 6.5, 'Cat', [1,2,3,4], {1,2,3,4}, {1:1, 2:2, 3:3, 4:4}])

def test_generator_illegal(x):
with pytest.raises(ValueError):
PrimeGenerator(x)

# Parametric test of the isprime part
@pytest.mark.parametrize("x, y", [(-3, False),(-2, False),(0, False),(1, False),(2, True),(3, True),(4, False),(5, True),(100, False),(47, True)])

def test_isprime_compute(x, y):
assert PrimeGenerator().isprime(x) == y, "isprime test"

# Parametric test of the giving illegal data to isprime
@pytest.mark.parametrize("x", [1.3, 6.5, 'Cat', [1,2,3,4], {1,2,3,4}, {1:1, 2:2, 3:3, 4:4}])

def test_isprime_illegal(x):
with pytest.raises(ValueError):
PrimeGenerator().isprime(x)
</pre>
When running the unit tests with '''pytest test_PrimeGenerator.py''', I got the following problems.
FAILED test_PrimeGenerator.py::test_generator_illegal[1.3] - Failed: DID NOT RAISE <class 'ValueError'>
FAILED test_PrimeGenerator.py::test_generator_illegal[6.5] - Failed: DID NOT RAISE <class 'ValueError'>
FAILED test_PrimeGenerator.py::test_generator_illegal[x3] - TypeError: int() argument must be a string, a bytes-like ...
FAILED test_PrimeGenerator.py::test_generator_illegal[x4] - TypeError: int() argument must be a string, a bytes-like ...
FAILED test_PrimeGenerator.py::test_generator_illegal[x5] - TypeError: int() argument must be a string, a bytes-like ...
FAILED test_PrimeGenerator.py::test_isprime_illegal[1.3] - Failed: DID NOT RAISE <class 'ValueError'>
FAILED test_PrimeGenerator.py::test_isprime_illegal[6.5] - Failed: DID NOT RAISE <class 'ValueError'>
FAILED test_PrimeGenerator.py::test_isprime_illegal[Cat] - TypeError: '<=' not supported between instances of 'str' a...
FAILED test_PrimeGenerator.py::test_isprime_illegal[x3] - TypeError: '<=' not supported between instances of 'list' a...
FAILED test_PrimeGenerator.py::test_isprime_illegal[x4] - TypeError: '<=' not supported between instances of 'set' an...
FAILED test_PrimeGenerator.py::test_isprime_illegal[x5] - TypeError: '<=' not supported between instances of 'dict' a...
Despite my best effort in making the PrimeGenerator class, I overlooked something, not in the core code, but in the validation of input.
The truth is that I did not think of my test_generator_illegal unit test to begin with, but when I made the test_isprime_illegal and it failed, then I realized the other input validation problem. This just shows that is it valuable to make the unit tests, even with the most stupid and inane test you can think of.

Having fixed my code issues, I here present the new PrimeGenerator code.
<pre>
#!/usr/bin/env python3
# Prime number generator

class PrimeGenerator:
# Class varible, known primes in consecutive order, can be extended, but must contain these
knownprimes = [2, 3]
# Highest tested number for prime
highesttested = 3

# Instatiation
def __init__(self, number=None):
if number is not None:
if not isinstance(number, int): # New code
raise ValueError("Integer expected") # New code
self.target = number

# Initializing iteration
def __iter__(self):
if self.target is None:
raise ValueError("No number specified")
self.pos = 0
return self

# Find next prime
def __next__(self):
# Can we use the list of known primes to find the next?
if self.pos < len(self.knownprimes):
nextprime = self.knownprimes[self.pos]
if nextprime >= self.target:
raise StopIteration
self.pos += 1
return nextprime
# No, start computing the next prime
while self.target > PrimeGenerator.highesttested+1:
PrimeGenerator.highesttested += 1
if self._isprime(PrimeGenerator.highesttested):
self.knownprimes.append(PrimeGenerator.highesttested)
self.pos += 1
return self.highesttested
raise StopIteration

# Private method for identifying a prime
def _isprime(self, number):
factor = 0
pos = 0
while factor*factor <= number:
# find next potential factor either in known primes or odd numbers above last known prime
if pos < len(self.knownprimes):
factor = self.knownprimes[pos]
pos += 1
else:
factor += 2
# test if it truly is a factor
if number % factor == 0:
return False
return True

# It is nice be able to ask if an number is a prime
def isprime(self, number=None):
if number is None:
number = self.target
if not isinstance(number, int): # New code
raise ValueError("Integer expected")
if number in PrimeGenerator.knownprimes:
return True
if number <= PrimeGenerator.highesttested:
return False
return self._isprime(number)

if __name__ == "__main__":
# Small testing
for i in PrimeGenerator(1000):
print(i)

print(PrimeGenerator().isprime(1000000016531)) # True
print(PrimeGenerator().isprime(1000000016521)) # False

# Big prime, don't try
# 999296950101072104250052714631
</pre>