WikiSysop at 09:29, 17 June 2024

2024-06-17T09:29:51Z

← Older revision		Revision as of 11:29, 17 June 2024
Line 1:		Line 1:
	{\| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"		{\| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"
	\|Previous: [[Parallel programming]]		\|Previous: [[Parallel programming]]
	\|Next: [[~~MapReduce and~~ Binary representation]]		\|Next: [[Binary representation]]
	\|}		\|}
	== Material for the lesson ==		== Material for the lesson ==

WikiSysop: /* Material for the lesson */

2024-06-17T09:28:56Z

Material for the lesson

← Older revision		Revision as of 11:28, 17 June 2024
Line 6:		Line 6:
	Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=cfc600a9-86b0-4e3b-9ece-af170077af38 Concurrency and semaphores]<br>		Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=cfc600a9-86b0-4e3b-9ece-af170077af38 Concurrency and semaphores]<br>
	Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=57cec219-787d-4ece-8ade-af1700778547 Semaphore demonstration - see video to prepare]<br>		Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=57cec219-787d-4ece-8ade-af1700778547 Semaphore demonstration - see video to prepare]<br>
	Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/~~HPCLife07~~-Semaphores.ppt Semaphores and Concurrency]<br>		Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_10-Semaphores.ppt Semaphores and Concurrency]<br>
	Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=16e3a478-8428-4529-a297-af1700775ce8 Fair distribution]<br>		Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=16e3a478-8428-4529-a297-af1700775ce8 Fair distribution]<br>
	Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/~~HPCLife~~-LoadBalancing.ppt Fair distribution]<br>		Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_10-LoadBalancing.ppt Fair distribution]<br>
	Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=20d9f85c-0785-4c60-81de-af17007736b8 Exercises]		Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=20d9f85c-0785-4c60-81de-af17007736b8 Exercises]

WikiSysop: Created page with "{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;" |Previous: Parallel programming |Next: MapReduce and Binary representation |} == Material for the lesson == Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=cfc600a9-86b0-4e3b-9ece-af170077af38 Concurrency and semaphores]
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=57cec219-787d-4ece-8ade-af1700778547 Semaphore demonstration - see video to prepare]
Powerpoi..."

2024-03-06T10:55:58Z

Created page with "{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;" |Previous: Parallel programming |Next: MapReduce and Binary representation |} == Material for the lesson == Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=cfc600a9-86b0-4e3b-9ece-af170077af38 Concurrency and semaphores] Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=57cec219-787d-4ece-8ade-af1700778547 Semaphore demonstration - see video to prepare] Powerpoi..."

New page

{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"
|Previous: [[Parallel programming]]
|Next: [[MapReduce and Binary representation]]
|}
== Material for the lesson ==
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=cfc600a9-86b0-4e3b-9ece-af170077af38 Concurrency and semaphores] 
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=57cec219-787d-4ece-8ade-af1700778547 Semaphore demonstration - see video to prepare] 
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/HPCLife07-Semaphores.ppt Semaphores and Concurrency] 
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=16e3a478-8428-4529-a297-af1700775ce8 Fair distribution] 
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/HPCLife-LoadBalancing.ppt Fair distribution] 
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=20d9f85c-0785-4c60-81de-af17007736b8 Exercises]

== Exercises ==
'''1)'''
Looking at human.fsa we have the human genome as as fasta file – one entry per chromosome.
Let us search for functional (could be regulatory) elements in the human DNA. A candidate for such an element is an over-represented k-mer. 
A k-mer is a stretch of DNA or protein sequence of size k. Hence a 10-mer represents 10 nucleotides or amino acids in a row.

Find which 5-mers, 6-mers and 7-mers are over-represented. This is simply done by counting all the different k-mers, and counting the different nucleotides. 
Example for k-mer: ’atgca’ , P for probability
P(a) = number of a/number of nucleotides
count(atgca-mers)/count(5-mers) > P(a)*P(t)*P(g)*P(c)*P(a)
If this inequality is true, there is an over-representation of 'atgca'.
We only want to find the significantly over-represented k-mers, so we arbitrarily decide that a k-mer is significantly overrepresented when the frequency with which it appears is double what we expect from the probability calculation.

Issues: In the fasta file there are stretches of unknown bases (n and other letters than atcg). All k-mers in these regions are disregarded.

'''2)'''
Parallelize the counting. The main process finds stretches of DNA that should be given to worker processes, which return counting results to the main process, that sums them up.
This could be done on entry basis, but does not have to be. 
Idea for parallelizing (only one program using joblib):
# count function/worker
def countme (file, start, end):
find sequence at start to end in file
count everything
return results
# main program
read fasta file and index sequence positions (start/end).
for positions in seqindex:
allcounts = Parallel(n_jobs=4)(delayed(countme)(file, x[0], x[1]) for x in seqindex)
combine allcounts
compute over-representaion

Issues to consider: 
* If you split DNA in twain for different processes to handle, there are k-mers that won’t be counted in the split, unless you handle it in some way. 
* What happens with memory usage – especially when using 8, 9, 10-mers. How to deal with it? 
*Some sequences are perhaps larger than others. How will you spread the load on the worker processes, so they all end at approximately the same time. This is a hard problem to solve optimally, but rather easy to solve well, but not optimal. This problem is called '''load balancing'''.

More parallelism - Revision history

WikiSysop at 09:29, 17 June 2024

WikiSysop: /* Material for the lesson */