https://teaching.healthtech.dtu.dk:443/22112/api.php?action=feedcontributions&feedformat=atom&user=WikiSysop22112 - User contributions [en]2026-05-02T09:09:57ZUser contributionsMediaWiki 1.41.0https://teaching.healthtech.dtu.dk:443/22112/index.php?title=22112_High_Performance_Computing_in_Life_Science&diff=15622112 High Performance Computing in Life Science2025-12-04T06:56:33Z<p>WikiSysop: /* Course details */</p>
<hr />
<div>__NOTOC__<br />
'''Teacher:''' [https://www.inside.dtu.dk/da/dtuinside/generelt/telefonbog/person?id=816&cpid=214027&tab=2&qt=dtupublicationquery Peter Wad Sackett], pwsa@dtu.dk <!--, [https://www.inside.dtu.dk/en/dtuinside/generelt/telefonbog/person?id=130596&tab=0 Gisle Alberg Vestergaard], gisves@dtu.dk --> <br><br />
'''Language:''' The course is taught in English.<br><br />
'''Tools:''' Various servers will be introduced.<br><br />
'''Textbooks:''' There are no text books for the course. I will make do with powerpoints, videos and references to online resources. You can find the material under the individual lessons in the [[Programme]].<br><br />
'''Location:''' Building 208, room 001.<br><br />
'''Time:''' Thursday 13:00 - 17:00, module E1-B.<br><br />
<br />
== Course details ==<br />
The course is a practical introduction to working with supercomputers in Life Science. Using both "ordinary" laptops and computer clusters as a basis for lectures and exercises, we will cover subjects such as computer design, queueing systems, distributed computing, algorithms, parallel programming, performance and performance dependencies, database design and use, and practical organization of projects.<br />
<br />
{| <br />
|-<br />
| [[Aligning expectations]] || Mandatory read at course start<br />
|-<br />
| [[Programme]] || Fall 2024<br />
|-<br />
| [https://teaching.healthtech.dtu.dk/22112/progress/ Exercise progression] || How well are you doing?<br />
|-<br />
| [[Computerome]] || Some info on Computerome<br />
|-<br />
| [https://teaching.healthtech.dtu.dk/material/22112/StudentAssistantLundbeck.pdf Job] || Student job at Lundbeck<br />
|}<br />
<br />
== Earlier programmes ==<br />
[[Programme Fall 2023]]<br><br />
[[Programme Fall 2024]]</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Parallel_programming&diff=155Parallel programming2025-11-06T13:29:37Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Algorithms]]<br />
|Next: [[More parallelism]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=6d186b7e-f5a7-4e1f-b613-af270122f263 Parallel Programming - problems]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=bb4fc4d2-c0af-44bd-b181-af270122cf03 Libraries to use and how they work]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=5aa86527-f501-44b6-9ee4-af270122a685 Parent/Child relationships]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_09-Programming.ppt Programming]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=8ff003cd-f906-4d42-9f3d-af2701228c6d What is expected from exercises]<br />
<br />
== Exercises ==<br />
''Read a fasta file, find the complement strand of each entry, save result in new fasta file.''<br><br />
Note: The python you should use on '''pupil1''' server is /home/ctools/anaconda3-2024.10-1/bin/python3 - there are 2 pythons, so it is possible to choose wrong.<br><br />
'''1)'''<br><br />
In lesson 6, Distributed computing, you created 2-3 programs and used the Queueing System to submit your sub tasks. You are going to repeat this exercise, but this time do not use the QS to do the sub tasks, but use<br />
'''subprocess''' and '''joblib''' as shown in the powerpoint example. The code is the same as second exercise except for the part that do the sub tasks, and waits for them to finish.<br><br />
You must still use the QS to submit the initial main job.<br />
<br />
'''2)'''<br><br />
If you feel this was too easy, then the administrator should only index the fasta file and launch workers with the indexed info using '''joblib''' and an internal worker function.<br />
The workers will create the complement strand entry as a file and the administrator will collect the pieces as in above. This is using techniques we have already covered. The result will be the same as above, but faster.<br />
<br />
'''3)'''<br><br />
For an extra challenge, then create a different worker function that writes the complement stand directly back into the file the right place as has been discussed earlier.<br />
Here you need your own "private" fasta file, because you can not write in mine in /home/projects/pr_course.<br />
<br />
<br />
Time everything to see the difference. Don't use more than 8 workers at a time - '''joblib''' helps with that.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Queueing_System&diff=154Queueing System2025-11-05T12:19:59Z<p>WikiSysop: /* Material for the lesson */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Computer design]]<br />
|Next: [[Distributed computing]]<br />
|}<br />
== Material for the lesson ==<br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_05-Queue.ppt Queueing System]<br><br />
IUPAC nucleotide codes: [http://www.dnabaser.com/articles/IUPAC%20ambiguity%20codes.html Read before doing exercise]<br><br />
<!--<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=d679e5c8-9372-4af7-9260-af270124ddf6 Introduction to the Queueing System]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=bca57da6-2c60-4218-96ac-af270124b673 Submitting jobs]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=e6e59e86-2928-4caf-b5c3-af2701248a3d Queue control and practical advice]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=c82ed125-4664-4c5b-b23b-af1700719969 Exercises]<br />
--><br />
<br />
== Exercises ==<br />
The exercises must work with the human genome in the fastafile human.fsa and you can develop/test your program against the small scale humantest.fsa file.<br />
''You must use the Queueing System and '''sbatch'''''.<br />
<br />
# Make (or reuse) a program that reads a fasta file and finds the complement strand for each entry, and saves the result in a new file. Keep it simple. Make sure it works. Speed is not important in this step.<br />
# Speed is still not important. Add this functionality to your program: Count the bases and unknowns in the entry and add the counts to the header line, like >seq01 A:3450 T:45665 C:34576 G:142345 N:5462<br />
# You need to increase the performance of your program. Experiment with various ideas of how to increase the speed. You got some ideas last lecture and you should also be using your Python knowledge. Document your experiments with a line or two as comments in your program. This exercise is likely the one that takes the most time to complete and the shortest to run.<br />
<br />
I solved this problem in 222 seconds on a server, however time vary, as I had also a run using 527 seconds with the same code - IO from other users can really affect your time. I had a "slow" version run on a server in over 1000 seconds. On my laptop with SSD I solved the problem in 100 seconds.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=22112_High_Performance_Computing_in_Life_Science&diff=15322112 High Performance Computing in Life Science2025-10-29T06:23:07Z<p>WikiSysop: /* Course details */</p>
<hr />
<div>__NOTOC__<br />
'''Teacher:''' [https://www.inside.dtu.dk/da/dtuinside/generelt/telefonbog/person?id=816&cpid=214027&tab=2&qt=dtupublicationquery Peter Wad Sackett], pwsa@dtu.dk <!--, [https://www.inside.dtu.dk/en/dtuinside/generelt/telefonbog/person?id=130596&tab=0 Gisle Alberg Vestergaard], gisves@dtu.dk --> <br><br />
'''Language:''' The course is taught in English.<br><br />
'''Tools:''' Various servers will be introduced.<br><br />
'''Textbooks:''' There are no text books for the course. I will make do with powerpoints, videos and references to online resources. You can find the material under the individual lessons in the [[Programme]].<br><br />
'''Location:''' Building 208, room 001.<br><br />
'''Time:''' Thursday 13:00 - 17:00, module E1-B.<br><br />
<br />
== Course details ==<br />
The course is a practical introduction to working with supercomputers in Life Science. Using both "ordinary" laptops and computer clusters as a basis for lectures and exercises, we will cover subjects such as computer design, queueing systems, distributed computing, algorithms, parallel programming, performance and performance dependencies, database design and use, and practical organization of projects.<br />
<br />
{| <br />
|-<br />
| [[Aligning expectations]] || Mandatory read at course start<br />
|-<br />
| [[Programme]] || Fall 2024<br />
|-<br />
| [https://teaching.healthtech.dtu.dk/22112/progress/ Exercise progression] || How well are you doing?<br />
|-<br />
| [[Computerome]] || Some info on Computerome<br />
|}<br />
<br />
== Earlier programmes ==<br />
[[Programme Fall 2023]]<br><br />
[[Programme Fall 2024]]</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=152Computer design2025-09-25T17:52:36Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Computers.ppt Computer design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
Note that the videos are no longer completely aligned with the powerpoints due to a shift in focus.<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
Motto: Less fear - more fun<br />
<br />
'''1) Measuring IO speed'''<br><br />
Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.<br />
<br />
'''2) The Chat Board'''<br><br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called '''chatboard''' that is common to you all. In that database there will be just one table, '''messages''' with the fields;<br />
<pre><br />
id primary key auto_increment<br />
mesg varchar(500)<br />
name varchar(15) default NULL<br />
channel varchar(12) default 'common'<br />
stamp timestamp default CURRENT_TIMESTAMP<br />
</pre><br />
The last field, stamp, is automatically updated to the current time when the message is inserted. You only have to insert mesg, name and channel when inserting a row. You can only read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality. You are to make a python program that runs in the terminal and interacts with you (your commands). If confused, start with making a program that simply pulls all messages from the database and show them. Build up with selecting a channel to show, then the ability to post a message.<br><br />
Something that I found to be a challenge was to figure out when to pull new messages from the database and which messages. If you are overwhelmed, then just auto pull the last 5-10 messages every time - but you get a lot redundant information that way. You could rely on the ''/last'' command - you command the pull yourself. I made a pull for only new messages when I pressed <enter> without any command/message.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b> This is minimum useful functionality/requirement.<br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you", ''/trueid'', which sets your name to your login name, ''/activity'', which shows when the channels had the latest messages.<br><br />
You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
Hint: Something I found useful is that you can get the ''id'', i.e. the auto incremented primary key of you last inserted message like this:<br />
<pre><br />
cursor.execute("""INSERT INTO messages SET mesg=%s, name=%s, channel=%s""", (chat, nick, channel))<br />
msgID = cursor.lastrowid<br />
</pre><br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?<br />
<br />
'''Extra optional fun:''' ChatBots<br><br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
<br />
* behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
* find the latest message and add 'That is what she said yesterday'.<br><br />
* be more clever and consult external AI and generate a one-line answer/comment.<br><br />
* find the quote of the day and post on the common channel.<br><br />
* camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
* copy other posters - simply repeating their message.<br><br />
* advertise an event across all channels.<br><br />
* have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
* If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=151Computer design2025-09-25T17:51:05Z<p>WikiSysop: /* Material for the lesson */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Computers.ppt Computer design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
Note that the videos are no longer completely aligned with the powerpoints due to a shift in focus.<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
Motto: Less fear - more fun<br />
<br />
'''1) Measuring IO speed'''<br><br />
Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.<br />
<br />
'''2) The Chat Board'''<br><br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called '''chatboard''' that is common to you all. In that database there will be just one table, '''messages''' with the fields;<br />
<pre><br />
id primary key auto_increment<br />
mesg varchar(500)<br />
name varchar(15) default NULL<br />
channel varchar(12) default 'common'<br />
stamp timestamp default CURRENT_TIMESTAMP<br />
</pre><br />
The last field, stamp, is automatically updated to the current time when the message is inserted. You only have to insert mesg, name and channel when inserting a row. You can only read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality. You are to make a python program that runs in the terminal and interacts with you (your commands). If confused, start with making a program that simply pulls all messages from the database and show them. Build up with selecting a channel to show, then the ability to post a message.<br><br />
Something that I found to be a challenge was to figure out when to pull new messages from the database and which messages. If you are overwhelmed, then just auto pull the last 5-10 messages every time - but you get a lot redundant information that way. You could rely on the ''/last'' command - you command the pull yourself. I made a pull for only new messages when I pressed <enter> without any command/message.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b> This is minimum useful functionality/requirement.<br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you", ''/trueid'', which sets your name to your login name, ''/activity'', which shows when the channels had the latest messages.<br><br />
You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
Hint: Something I found useful is that you can get the ''id'', i.e. the auto incremented primary key of you last inserted message like this:<br />
<pre><br />
cursor.execute("""INSERT INTO messages SET mesg=%s, name=%s, channel=%s""", (chat, nick, channel))<br />
msgID = cursor.lastrowid<br />
</pre><br />
<br />
'''Extra optional fun:''' ChatBots<br><br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
<br />
* behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
* find the latest message and add 'That is what she said yesterday'.<br><br />
* be more clever and consult external AI and generate a one-line answer/comment.<br><br />
* find the quote of the day and post on the common channel.<br><br />
* camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
* copy other posters - simply repeating their message.<br><br />
* advertise an event across all channels.<br><br />
* have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
* If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=150Computer design2025-09-25T17:50:30Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Computers.ppt Computer design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
Note that the videos are no longer completely aligned with the powerpoints due to a shift in focus.<br />
<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
Motto: Less fear - more fun<br />
<br />
'''1) Measuring IO speed'''<br><br />
Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.<br />
<br />
'''2) The Chat Board'''<br><br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called '''chatboard''' that is common to you all. In that database there will be just one table, '''messages''' with the fields;<br />
<pre><br />
id primary key auto_increment<br />
mesg varchar(500)<br />
name varchar(15) default NULL<br />
channel varchar(12) default 'common'<br />
stamp timestamp default CURRENT_TIMESTAMP<br />
</pre><br />
The last field, stamp, is automatically updated to the current time when the message is inserted. You only have to insert mesg, name and channel when inserting a row. You can only read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality. You are to make a python program that runs in the terminal and interacts with you (your commands). If confused, start with making a program that simply pulls all messages from the database and show them. Build up with selecting a channel to show, then the ability to post a message.<br><br />
Something that I found to be a challenge was to figure out when to pull new messages from the database and which messages. If you are overwhelmed, then just auto pull the last 5-10 messages every time - but you get a lot redundant information that way. You could rely on the ''/last'' command - you command the pull yourself. I made a pull for only new messages when I pressed <enter> without any command/message.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b> This is minimum useful functionality/requirement.<br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you", ''/trueid'', which sets your name to your login name, ''/activity'', which shows when the channels had the latest messages.<br><br />
You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
Hint: Something I found useful is that you can get the ''id'', i.e. the auto incremented primary key of you last inserted message like this:<br />
<pre><br />
cursor.execute("""INSERT INTO messages SET mesg=%s, name=%s, channel=%s""", (chat, nick, channel))<br />
msgID = cursor.lastrowid<br />
</pre><br />
<br />
'''Extra optional fun:''' ChatBots<br><br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
<br />
* behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
* find the latest message and add 'That is what she said yesterday'.<br><br />
* be more clever and consult external AI and generate a one-line answer/comment.<br><br />
* find the quote of the day and post on the common channel.<br><br />
* camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
* copy other posters - simply repeating their message.<br><br />
* advertise an event across all channels.<br><br />
* have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
* If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=149Computer design2025-09-25T17:48:57Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Computers.ppt Computer design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
Note that the videos are no longer completely aligned with the powerpoints due to a shift in focus.<br />
<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
'''1) Measuring IO speed'''<br><br />
Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.<br />
<br />
'''2) The Chat Board'''<br><br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called '''chatboard''' that is common to you all. In that database there will be just one table, '''messages''' with the fields;<br />
<pre><br />
id primary key auto_increment<br />
mesg varchar(500)<br />
name varchar(15) default NULL<br />
channel varchar(12) default 'common'<br />
stamp timestamp default CURRENT_TIMESTAMP<br />
</pre><br />
The last field, stamp, is automatically updated to the current time when the message is inserted. You only have to insert mesg, name and channel when inserting a row. You can only read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality. You are to make a python program that runs in the terminal and interacts with you (your commands). If confused, start with making a program that simply pulls all messages from the database and show them. Build up with selecting a channel to show, then the ability to post a message.<br><br />
Something that I found to be a challenge was to figure out when to pull new messages from the database and which messages. If you are overwhelmed, then just auto pull the last 5-10 messages every time - but you get a lot redundant information that way. You could rely on the ''/last'' command - you command the pull yourself. I made a pull for only new messages when I pressed <enter> without any command/message.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b> This is minimum useful functionality/requirement.<br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you", ''/trueid'', which sets your name to your login name, ''/activity'', which shows when the channels had the latest messages.<br><br />
You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
Hint: Something I found useful is that you can get the ''id'', i.e. the auto incremented primary key of you last inserted message like this:<br />
<pre><br />
cursor.execute("""INSERT INTO messages SET mesg=%s, name=%s, channel=%s""", (chat, nick, channel))<br />
msgID = cursor.lastrowid<br />
</pre><br />
<br />
'''Extra optional fun:''' ChatBots<br><br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
<br />
* behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
* find the latest message and add 'That is what she said yesterday'.<br><br />
* be more clever and consult external AI and generate a one-line answer/comment.<br><br />
* find the quote of the day and post on the common channel.<br><br />
* camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
* copy other posters - simply repeating their message.<br><br />
* advertise an event across all channels.<br><br />
* have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
* If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=148Computer design2025-09-25T17:46:55Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Computers.ppt Computer design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
Note that the videos are no longer completely aligned with the powerpoints due to a shift in focus.<br />
<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
'''1) Measuring IO speed'''<br><br />
Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.<br />
<br />
'''2) The Chat Board'''<br><br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called '''chatboard''' that is common to you all. In that database there will be just one table, '''messages''' with the fields;<br />
<pre><br />
id primary key auto_increment<br />
mesg varchar(500)<br />
name varchar(15) default NULL<br />
channel varchar(12) default 'common'<br />
stamp timestamp default CURRENT_TIMESTAMP<br />
</pre><br />
The last field, stamp, is automatically updated to the current time when the message is inserted. You only have to insert mesg, name and channel when inserting a row. You can only read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality. You are to make a python program that runs in the terminal and interacts with you (your commands). If confused, start with making a program that simply pulls all messages from the database and show them. Build up with selecting a channel to show, then the ability to post a message.<br><br />
Something that I found to be a challenge was to figure out when to pull new messages from the database and which messages. If you are overwhelmed, then just auto pull the last 5-10 messages every time - but you get a lot redundant information that way. You could rely on the ''/last'' command - you command the pull yourself. I made a pull for only new messages when I pressed <enter> without any command/message.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b> This is minimum useful functionality/requirement.<br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you", ''/trueid'', which sets your name to your login name, ''/activity'', which shows when the channels had the latest messages.<br><br />
You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
Hint: Something I found useful is that you can get the ''id'', i.e. the auto incremented primary key of you last inserted message like this:<br />
<pre><br />
cursor.execute("""INSERT INTO messages SET mesg=%s, name=%s, channel=%s""", (chat, nick, channel))<br />
msgID = cursor.lastrowid<br />
</pre><br />
<br />
'''Extra optional fun:''' ChatBots<br><br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=147Computer design2025-09-25T17:39:24Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Computers.ppt Computer design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
Note that the videos are no longer completely aligned with the powerpoints due to a shift in focus.<br />
<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
'''1) Measuring IO speed'''<br><br />
Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.<br />
<br />
'''2) The Chat Board'''<br><br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called '''chatboard''' that is common to you all. In that database there will be just one table, '''messages''' with the fields;<br />
<pre><br />
id primary key auto_increment<br />
mesg varchar(500)<br />
name varchar(15) default NULL<br />
channel varchar(12) default 'common'<br />
stamp timestamp default CURRENT_TIMESTAMP<br />
</pre><br />
The last field, stamp, is automatically updated to the current time when the message is inserted. You only have to insert mesg, name and channel when inserting a row. You can only read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality. You are to make a python program that runs in the terminal and interacts with you (your commands). If confused, start with making a program that simply pulls all messages from the database and show them. Build up with selecting a channel to show, then the ability to post a message.<br><br />
Something that I found to be a challenge was to figure out when to pull new messages from the database and which messages. If you are overwhelmed, then just auto pull the last 5-10 messages every time - but you get a lot redundant information that way. You could rely on the ''/last'' command - you command the pull yourself. I made a pull for only new messages when I pressed <enter> without any command/message.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b> This is minimum useful functionality/requirement.<br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you", ''/trueid'', which sets your name to your login name, ''/activity'', which shows when the channels had the latest messages.<br><br />
You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
'''Extra optional fun:''' ChatBots<br><br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=146Computer design2025-09-25T17:21:10Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Computers.ppt Computer design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
Note that the videos are no longer completely aligned with the powerpoints due to a shift in focus.<br />
<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
'''1) Measuring IO speed'''<br><br />
Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.<br />
<br />
'''2) The Chat Board'''<br><br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called '''chatboard''' that is common to you all. In that database there will be just one table, ''messages'' with the fields;<br />
<pre><br />
id primary key auto_increment<br />
mesg varchar(500)<br />
name varchar(15) default NULL<br />
channel varchar(12) default 'common'<br />
stamp timestamp default CURRENT_TIMESTAMP<br />
</pre><br />
The last field, stamp, is automatically updated to the current time when the message is inserted. You only have to insert mesg, name and channel when inserting a row. You can only read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you", ''/trueid'', which sets your name to your login name, ''/activity'', which shows when the channels had the latest messages.<br><br />
You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=145Computer design2025-09-25T17:17:57Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Computers.ppt Computer design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
Note that the videos are no longer completely aligned with the powerpoints due to a shift in focus.<br />
<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
'''1) Measuring IO speed'''<br><br />
Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.<br />
<br />
'''2) The Chat Board'''<br><br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called '''chatboard''' that is common to you all. In that database there will be just one table, ''messages'' with the fields;<br />
<pre><br />
id primary key auto_increment<br />
mesg varchar(500)<br />
name varchar(15) default NULL<br />
channel varchar(12) default 'common'<br />
stamp timestamp default CURRENT_TIMESTAMP<br />
</pre><br />
The last field, stamp, is automatically updated to the current time when the message is inserted. You only have to insert mesg, name and channel. You can only read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=144Computer design2025-09-25T17:12:47Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Computers.ppt Computer design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
Note that the videos are no longer completely aligned with the powerpoints due to a shift in focus.<br />
<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
'''1)''' Measuring IO speed<br><br />
Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.<br />
<br />
'''2)''' The Chat Board<br><br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called '''chatboard''' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key auto_increment), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), stamp (timestamp) default CURRENT_TIMESTAMP, which means the timestamp is automatically created when the message is inserted. You only have the ability to read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=143Computer design2025-09-25T17:12:03Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Computers.ppt Computer design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
Note that the videos are no longer completely aligned with the powerpoints due to a shift in focus.<br />
<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
'''1)''' Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.<br />
<br />
'''2)''' The Chat Board<br><br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key auto_increment), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), stamp (timestamp) default CURRENT_TIMESTAMP, which means the timestamp is automatically created when the message is inserted. You only have the ability to read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=142Coding against databases2025-09-25T17:10:23Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br><br />
According to my results: 262 people are born out of wedlock. That is amazing as it is out of 500 people.<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional - really simple once it works'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
===Chatboard exercise moved to next lesson===</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Exercises_and_Organization&diff=141Exercises and Organization2025-09-25T06:38:15Z<p>WikiSysop: /* Material for the lesson */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Hash usage]]<br />
|Next: [https://teaching.healthtech.dtu.dk/22112/index.php/22112_High_Performance_Computing_in_Life_Science Main page]<br />
|}<br />
== Material for the lesson ==<br />
<!--Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=0ccf3b1b-52b7-4d91-9314-af17007069b8 Organization and Exercises]<br>--><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_13-Project.ppt Projects - organize your work life]<br><br />
PDF: [https://teaching.healthtech.dtu.dk/material/22112/22112_13-Organization.pdf Organization of projects] Article by someone like you - just older and more experienced.<br><br />
PDF: [https://teaching.healthtech.dtu.dk/material/22112/guide-to-reproducible-code.pdf Guide to Reproducible Code]<br><br />
PDF: [https://teaching.healthtech.dtu.dk/material/22112/HPC_analysis_framework_Leon.pdf HPC Analysis Framework] by Leon Jessen<br><br />
Tweet: [https://twitter.com/jessenleon/status/964236119658418176 Use Twitter for academic networking. You needn’t tweet anything yourself, but it’s a great way to follow bioinformatics influencers and news], Leon Jessen<br><br />
Video: [https://www.youtube.com/watch?v=8xnm_RKkycQ Alexander Schönhuth: "Snakemake: Reproducible and Scalable Data Analysis"] Optional, but good to know about. Just get the idea.<br />
<br />
== Exercises ==<br />
'''1)'''<br />
In this exercise we will work with an Illumina sequencing file: ngs1.fastq.gz. There is a test version, ngs1test.fastq.gz, and an even smaller version, ngs1small.fastq.gz, of this file, which you should use when developing your programs. You need to use the queueing system, or log in on an interactive node if you are just doing "small stuff".<br />
<br />
All NGS fastq files are horrendously big. This is why they are always compressed. When working with these kind of files, you always use (de)compression, simply to save disk space. If you can manage to (de)compress on the fly (this means decompressing while reading) you might even save time since the file is being read compressed (i.e. small) and unfolds directly in RAM on the computer. The (de)compression takes CPU time and the reading/writing takes IO time.<br />
<br />
The content of the file looks like these 4 lines repeated a lot:<br />
<br />
@ST-K00127:16:H32JKBBXX:8:1101:1111:1246 1:N:0:<font color="FF0000">AGCCATGC</font><br />
NAAAAGGCTCGACGCCTGTTATTATCGGAGTAGAACCGGAAATATCCGGATTAAGTTTTTCAGATCGGAAGAGCACACGTCTGAACTCCAGTCACAGCCA<br />
+<br />
#AAFFKKKKKKKKKKFK<KFKKKKKKKKFKFFKKKKKKKKKKKKKKKKK7FKKKKKKKKK<FF<KKKKKKKFKKKKKKKKKKKKKKKKKKKKKKKKKKKK<br />
This is called a read. The lines are in order: header line, sequence, comment line (usually empty, just a +) and quality score. There is a char on the quality score for each base in the sequence, i.e. sequence and quality lines have the same length.<br />
<br />
The file contains microbiome sequencing data from a number of persons. This is common practice. Each person has a unique barcode, which is the red DNA string in the header.<br />
<br />
The act of splitting the file up in several files - one for each person - is called demultiplexing. We will be doing this in next exercise.<br />
<br />
Any current sequencing technology creates sequencing errors. The barcode can actually be sequenced wrong, not just the dna in the read. This means that some of the reads in the file belongs to people we can not identify.<br />
<br />
Your job in this exercise is therefore to identify which barcodes are real, and which are due to sequencing errors. The erronous barcodes and their data are simply discarded. You can assume that ”right” barcodes occur much more frequently than ”wrong” barcodes, but it is also true that there can be a significant difference in the frequency of "right" barcodes.<br><br />
Another strong/useful observation is that barcodes are "spread" across the spectrum. The distance from a real barcode to another real barcode is big. The distance is counted as bases that do not match at the same position in the barcodes.<br />
Real: CACTTCGA<br />
Real: TGAAGAGA<br />
Distance: 6<br />
The distance between a real barcode and one that is sequenced wrong in one base is obviously 1.<br />
Real: CACTTCGA<br />
Error: TACTTCGA<br />
Distance: 1<br />
This means that it is possible to assign some of the sequences to the right barcode even if the barcode is sequenced wrong. You are not required to do so in these exercises, but it is good to remember this in real life.<br />
<br />
Be careful in how much CPU time you are using to solve the task.<br />
<br />
You must report<br />
The program<br />
A list (file) of correct barcodes.<br />
Report the number of persons.<br />
<br />
For your benefit, I have made a function that opens a gzipped file, like the ngs file.<br />
<pre><br />
import gzip, sys<br />
def openfile(filename):<br />
try:<br />
if filename.endswith('.gz'):<br />
fh = gzip.open(filename, mode='rb')<br />
else:<br />
fh = open(filename, 'rb')<br />
except:<br />
print("Can't open file:", filename)<br />
return fh<br />
<br />
myfilehandle = openfile('mygzippedfile.gz')<br />
for line in myfilehandle:<br />
print(line)<br />
myfilehandle.close()<br />
</pre><br />
<br />
'''2)'''<br />
Now that you have a list of correct barcodes, your task is to write a program that can demultiplex the ngs1.fastq.gz and save the resulting fastq files.<br />
This could be combined with the previous exercise.<br />
<br />
You need to focus on performance and disk space. <br />
This means that the solution to the problem contains a jobscript (for submission to the queueing system) and likely a program or two, which handles the various aspects of the job.<br><br />
Hint: you might look into the '''Popen''' call in '''subprocess'''.<br />
<br />
You need to report the following:<br />
<br />
Your programs.<br />
How much disk space are you using to store your results?<br />
How long time did it take to complete the job?<br />
<br />
When you are done, delete your results to conserve disk space.<br />
My numbers are: 30 GB and 97 minutes.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=140Coding against databases2025-09-23T11:45:00Z<p>WikiSysop: /* Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br><br />
According to my results: 262 people are born out of wedlock. That is amazing as it is out of 500 people.<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key auto_increment), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), stamp (timestamp) default CURRENT_TIMESTAMP, which means the timestamp is automatically created when the message is inserted. You only have the ability to read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=139Coding against databases2025-09-21T11:40:42Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br><br />
According to my results: 262 people are born out of wedlock. That is amazing as it is out of 500 people.<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key auto_increment), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), stamp (timestamp). You only have the ability to read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=138Coding against databases2025-09-21T11:35:49Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br><br />
According to my results: 262 people are born out of wedlock.<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key auto_increment), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), stamp (timestamp). You only have the ability to read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=137Coding against databases2025-09-21T09:55:53Z<p>WikiSysop: /* Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key auto_increment), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), stamp (timestamp). You only have the ability to read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is something:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
As you hopefully can see, this structure makes it easy to start small with simple things and build up with new expanded functionality.<br />
<br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
/last <X> Pull last X messages from the chosen channel and display.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=136Coding against databases2025-09-21T08:03:04Z<p>WikiSysop: /* Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key auto_increment), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), stamp (timestamp). You only have the ability to read the table and insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is nothing:<br />
pass<br />
else:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=135Coding against databases2025-09-21T07:54:48Z<p>WikiSysop: /* Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key auto_increment), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), stamp (timestamp). You only have the ability to insert new rows in the table - that is - create new messages. Only the moderator can delete messages.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is nothing:<br />
pass<br />
else:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=134Coding against databases2025-09-21T06:38:30Z<p>WikiSysop: /* Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key auto_increment), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), stamp (timestamp). You only have the ability to insert new rows in the table - that is - create new messages. Only the moderator can delete messages. '''This database and table is NOT CREATED YET.'''<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''stamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is nothing:<br />
pass<br />
else:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=133Coding against databases2025-09-21T06:26:58Z<p>WikiSysop: /* Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key autoincrement), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), timestamp (timestamp). You only have the ability to insert new rows in the table - that is - create new messages. Only the moderator can delete messages. '''This database and table is NOT CREATED YET.'''<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''timestamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is nothing:<br />
pass<br />
else:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br><br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=132Coding against databases2025-09-21T06:26:34Z<p>WikiSysop: /* Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key autoincrement), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), timestamp (timestamp). You only have the ability to insert new rows in the table - that is - create new messages. Only the moderator can delete messages. '''This database and table is NOT CREATED YET.'''<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''timestamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is nothing:<br />
pass<br />
else:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br />
If anybody posted a calculation, it could compute the result and post it - just like google.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=131Coding against databases2025-09-21T06:24:30Z<p>WikiSysop: /* Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key autoincrement), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), timestamp (timestamp). You only have the ability to insert new rows in the table - that is - create new messages. Only the moderator can delete messages. '''This database and table is NOT CREATED YET.'''<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''timestamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is nothing:<br />
pass<br />
else:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course. Once you get the the SSH tunneling set up, you just run your client from home/your laptop. How can it get any better?</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=130Coding against databases2025-09-21T06:22:52Z<p>WikiSysop: /* Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key autoincrement), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), timestamp (timestamp). You only have the ability to insert new rows in the table - that is - create new messages. Only the moderator can delete messages. '''This database and table is NOT CREATED YET.'''<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''timestamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is nothing:<br />
pass<br />
else:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br />
<br />
I think that was enough ideas for bots.<br />
<br />
The fun part of this exercise is you can use the Chat Board for the rest of the course, once you made the client. You can even develop your client and/or bot more in the progress of the course.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=129Coding against databases2025-09-21T06:17:21Z<p>WikiSysop: /* Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key autoincrement), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), timestamp (timestamp). You only have the ability to insert new rows in the table - that is - create new messages. Only the moderator can delete messages. '''This database and table is NOT CREATED YET.'''<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''timestamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is nothing:<br />
pass<br />
else:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
''Your dream, pal, your vision.''<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br />
<br />
I think that was enough ideas.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=128Coding against databases2025-09-21T06:12:00Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key autoincrement), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), timestamp (timestamp). You only have the ability to insert new rows in the table - that is - create new messages. Only the moderator can delete messages. '''This database and table is NOT CREATED YET.'''<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''timestamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is nothing:<br />
pass<br />
else:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
Your dream, pal, your vision.<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"<br><br />
It could copy other posters - simply repeating their message.<br><br />
It could advertise an event across all channels.<br><br />
It could have a list of people's birthdays, and when the day came about, post "Today is X's birthday" on the common channel.<br />
<br />
I think that was enough ideas.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=127Coding against databases2025-09-21T06:05:40Z<p>WikiSysop: /* Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key autoincrement), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), timestamp (timestamp). You only have the ability to insert new rows in the table - that is - create new messages. Only the moderator can delete messages. '''This database and table is NOT CREATED YET.'''<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''timestamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is nothing:<br />
pass<br />
else:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.<br />
<br />
When you have done your ChatClient you have accumulated enough experience to create a ChatBot. What should the bot do?<br><br />
Your dream, pal, your vision.<br><br />
It could behave like a dog; search for a random or specific user, and bark at them or pee on them.<br><br />
It could find the latest message and add 'That is what she said yesterday'.<br><br />
It could be more clever and consult external AI and generate a one-line answer/comment.<br><br />
It could find the quote of the day and post on the common channel.<br><br />
It could camp on the 'Dreams' channel and say to every new poster: "Tell me about your dreams!"</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=126Coding against databases2025-09-21T05:49:58Z<p>WikiSysop: /* Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key autoincrement), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), timestamp (timestamp). This database and table is NOT CREATED YET.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''timestamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is nothing:<br />
pass<br />
else:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit Quits the program, possibly leaving a message.<br />
/channel <channelname> Selects/creates the channel for your messages.<br />
/showchannels Shows possible channels.<br />
/trueid Sets your name to your login name.<br />
/nick <nickname> Sets your name to <nickname>.<br />
/activity Which channels had the latest messages, and when.<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search for messages by the person. ''/who'', which would show who has messages. ''/dance'', which displays a message "<yourname> is dancing for you". You dream it up - it is your chat client and the more powerful and funny it is, the better chances of you being the next Mark Zuckerberg.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=125Coding against databases2025-09-21T05:40:14Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board ===<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key autoincrement), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default 'common'), timestamp (timestamp). This database and table is NOT CREATED YET.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. Ask google if you don't know what a chat channel is. If NULL, it is the common channel.<br><br />
''timestamp'' is when the message is created. This is used for selecting the newest messages and deleting the old.<br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last messages and show<br />
get a one-line input<br />
while input is not quit:<br />
if input is command1:<br />
do commmand1 code<br />
elif input is command2:<br />
do command2 code<br />
elif input is nothing:<br />
pass<br />
else:<br />
input was a message, so post it to the chat board<br />
pull chat board for new messages and display them if any<br />
get next online input<br />
close database connection<br />
</pre><br />
A command is an input where the first word starts with slash possibly followed by a parameter. Anything else is just a message to be posted on the selected channel.<br><br />
<b>List of commands:</b><br><br />
<pre><br />
/quit<br />
/channel <channelname><br />
/showchannels<br />
/trueid<br />
/nick <nickname><br />
/activity<br />
</pre><br />
You can implement your own commands, like ''/search <name>'', which would search</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=124Coding against databases2025-09-21T05:12:09Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.<br />
<br />
=== Totally optional extra exercise I invented in my dreams Sunday morning: The Chat Board<br />
This is a fun cooperative exercise with databases. You should all make your own Chat client, where you can communicate with the other course participants. The exercise utilizes the concurrent features of databases - the easy update without worry if somebody else is accessing the database at the same time.<br />
<br />
You use a database called ''chatboard'' that is common to you all. In that database there will be just one table, ''messages'' with the fields; id (primary key autoincrement), mesg (varchar(500)), name (varchar(15), default NULL), channel (varchar(12), default NULL), timestamp (timestamp). This database and table is NOT CREATED YET.<br><br />
''mesg'' is the message on the chat.<br><br />
''name'' is name of the message sender. If NULL, then it is anonymous.<br><br />
''channel'' is the channel for this message. You make a subject a group of people with this. If NULL, it is the common channel.<br><br />
''timestamp'' is when the message is created. <br />
<br />
The Chat client you make should have the following structure.<br />
<pre><br />
connect to database chatboard<br />
pull last</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Distributed_computing&diff=123Distributed computing2025-09-20T20:05:34Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Queueing System]]<br />
|Next: [[What affects performance]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a67ca717-9598-4169-9b52-af2701246519 Distributed computing]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=e99121a6-df37-4d45-9891-af2701243c8a Profiling and subprocess]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_06-Distributed.ppt Distributed computing]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=88d328c4-990a-4ce7-80cb-af1700719959 Exercises]<br />
<br />
[https://evaluering.dtu.dk/ Do midterm evaluation] <span style="color:red"><-- Important to do this week - Friday latest</span><br />
<br />
== Exercises ==<br />
Using the Queueing System to do distributed computing. '''Warning''': The most difficult part of these exercises is actually using the Queueing System. The python code itself is fairly easy, but getting the QS to work requires patience and experience. Check the QS examples in the powerpoint from last lecture. Don't be afraid to consult the ''man'' pages on a slurm command. Great options are hidden there, f.ex how to get rid of header on output.<br />
<br />
'''1)'''<br><br />
Return to last time’s exercise; read a fasta file, find the reverse complement for each entry and count the bases in the entry putting the numbers in the header line and save it all in one file.<br />
Now solve this using the method on slide 5, i.e. distributed programming in embarrassingly parallel fashion.<br><br />
'''Test your programs on the small scale humantest.fsa file.''' When ready try the real human.fsa.<br />
<br />
You have to make several programs; the administrator, the worker and the collector.<br />
<br />
The administrator splits up the original input fasta file into several pieces (one fasta sequence per piece) and submits a job per piece (the worker) with the relevant file piece as input.<br />
The worker which reads a file with one fasta sequence (given), computes the complement strand and base count and outputs the result to a file (given).<br />
The collector program that collects all the result pieces and put them together in the original order in one file. This you run by yourself after the worker jobs finished. The structure of the administrator is like<br />
foreach fastasequence in inputfile<br />
save fastasequence in file.x<br />
submit job with file.x<br />
By naming/numbering the files in some systematic way, it is easier to collect them afterwards. Realize that you can test your code without using the QS, by simply running the worker directly. Also understand that this is an exercise in using the Queueing System, not in simple programming.<br />
<br />
<!--<br />
In order to lighten your workload and lessen the confusion and grief the Queueing System can give you, I have made a a fairly general '''submit to QS''' python function for you to use if you wish.<br />
The command you submit is a string (which you can construct), and the command itself. The command must be a "simple" one meaning a program with options and parameters as you would write it on the command line.<br />
It does not offer the the convenience of the shell, meaning that piping, IO-redirection and wildcards does not work. Realize that a great source of error is not using full path to programs and files.<br />
<pre><br />
def submit(command, directory='', modules='', runtime, cores, ram, group='pr_course',<br />
jobscript='jobscript', output='/dev/null', error='/dev/null'):<br />
"""<br />
Function to submit a job to the Queueing System - with jobscript file<br />
Parameters are:<br />
command: The command/program you want executed together with any parameters.<br />
Must use full path unless the directory is given and program is there. <br />
directory: Working directory - where should your program run, place of your data.<br />
If not specified, uses current directory.<br />
modules: String of space separated modules needed for the run.<br />
runtime: Time in minutes set aside for execution of the job.<br />
cores: How many cores are used for the job.<br />
ram: How much memory in GB is used for the job.<br />
group: Accounting - which group pays for the compute.<br />
jobscript: Standard name for the jobscript that needs to be made.<br />
You should number your jobscripts if you submit more than one.<br />
output: Output file of your job.<br />
error: Error file of your job.<br />
"""<br />
runtime = int(runtime)<br />
cores = int(cores)<br />
ram = int(ram)<br />
if cores > 10:<br />
print("Can't use more than 10 cores on a node")<br />
sys.exit(1)<br />
if ram > 120:<br />
print("Can't use more than 120 GB on a node")<br />
sys.exit(1)<br />
if runtime < 1:<br />
print("Must allocate at least 1 minute runtime")<br />
sys.exit(1)<br />
minutes = runtime % 60<br />
hours = int(runtime/60)<br />
walltime = "{:d}:{:02d}:00".format(hours, minutes)<br />
if directory == '':<br />
directory = os.getcwd()<br />
# Making a jobscript<br />
script = '#!/bin/sh\n'<br />
script += '#PBS -A ' + group + ' -W group_list=' + group + '\n'<br />
script += '#PBS -e ' + error + ' -o ' + output + '\n'<br />
script += '#PBS -d ' + directory + '\n'<br />
script += '#PBS -l nodes=1:ppn=' + str(cores) + ',mem=' + str(ram) + 'GB' + '\n'<br />
script += '#PBS -l walltime=' + walltime + '\n'<br />
if modules != '':<br />
script += 'module load ' + modules + '\n'<br />
script += command + '\n'<br />
if not jobscript.startswith('/'):<br />
jobscript = directory + '/' + jobscript<br />
with open(jobscript, 'wt') as jobfile:<br />
jobfile.write(script)<br />
# The submit<br />
job = subprocess.run(['qsub', jobscript],stdout=subprocess.PIPE, universal_newlines=True) <br />
jobid = job.stdout.split('.')[0]<br />
return jobid<br />
<br />
def submit2(command, directory='', modules='', runtime, cores, ram, group='pr_course',<br />
output='/dev/null', error='/dev/null'):<br />
"""<br />
Function to submit a job to the Queueing System - without jobscript file<br />
Parameters are:<br />
command: The command/program you want executed together with any parameters.<br />
Must use full path unless the directory is given and program is there. <br />
directory: Working directory - where should your program run, place of your data.<br />
If not specified, uses current directory.<br />
modules: String of space separated modules needed for the run.<br />
runtime: Time in minutes set aside for execution of the job.<br />
cores: How many cores are used for the job.<br />
ram: How much memory in GB is used for the job.<br />
group: Accounting - which group pays for the compute.<br />
output: Output file of your job.<br />
error: Error file of your job.<br />
"""<br />
runtime = int(runtime)<br />
cores = int(cores)<br />
ram = int(ram)<br />
if cores > 10:<br />
print("Can't use more than 10 cores on a node")<br />
sys.exit(1)<br />
if ram > 120:<br />
print("Can't use more than 120 GB on a node")<br />
sys.exit(1)<br />
if runtime < 1:<br />
print("Must allocate at least 1 minute runtime")<br />
sys.exit(1)<br />
minutes = runtime % 60<br />
hours = int(runtime/60)<br />
walltime = "{:d}:{:02d}:00".format(hours, minutes)<br />
if directory == '':<br />
directory = os.getcwd()<br />
# Making a jobscript<br />
script = '#!/bin/sh\n'<br />
script += '#PBS -A ' + group + ' -W group_list=' + group + '\n'<br />
script += '#PBS -e ' + error + ' -o ' + output + '\n'<br />
script += '#PBS -d ' + directory + '\n'<br />
script += '#PBS -l nodes=1:ppn=' + str(cores) + ',mem=' + str(ram) + 'GB' + '\n'<br />
script += '#PBS -l walltime=' + walltime + '\n'<br />
if modules != '':<br />
script += 'module load ' + modules + '\n'<br />
script += command + '\n'<br />
# The submit<br />
job = subprocess.run(['qsub'], input=script, stdout=subprocess.PIPE, universal_newlines=True) <br />
jobid = job.stdout.split('.')[0]<br />
return jobid<br />
<br />
<br />
# Is used like<br />
jobid = submit('myprogram myfile.fsa', directory='/home/projects/pr_course/people/pwsa',<br />
modules='tools anaconda3/4.0.0', jobscript='job.1', runtime=10, cores=1, ram=2)<br />
jobid = submit2('myprogram myfile.fsa', directory='/home/projects/pr_course/people/pwsa',<br />
modules='tools anaconda3/4.0.0', runtime=10, cores=1, ram=2)<br />
</pre><br />
You can make almost the entire program on your own laptop if you substitute the below function with above function. Below just calls the worker sequentially not using the QS. You loose all the distribution the QS offers,<br />
but avoid using computerome (and waiting for time). It is great in the development phase.<br />
<pre><br />
# Submit to the queueing system, runtime in minutes, ram in GB<br />
def unix_call(command):<br />
job = subprocess.run(command.split()) <br />
<br />
# Is used like<br />
unix_call('myprogram myfile.fsa')<br />
</pre><br />
--><br />
<br />
'''2)'''<br><br />
Make the administrator and collector into one program. <br />
foreach fastasequence in inputfile<br />
save fastasequence in file.x<br />
submit job with file.x<br />
wait for all jobs to finish<br />
collect data<br />
<br />
It is more difficult to solve this exercise - so do number 1 first. You need to find a way of waiting for your worker jobs to be done before your start collecting. However, you can see if this distributed method is faster than last week’s sequential method.<br />
<br />
Some ideas to wait for the jobs to be done.<br />
* Waiting and checking for all output files to appear. Cons; If a worker job breaks during execution you wait forever since the output file does not appear. If the output file is big, The worker might not have finished writing to it before collection starts. A trick to avoid this is for the worker to make an extra empty file at the end of the job and check for the presence of that file. An alternative is to write the file using a temporary filename, and rename the file to the correct name, when done writing.<br />
* Using '''squeue''' to check that the jobs are gone from the queue. You need to find a way to recognize your jobs. That can be to name them or get the jobid from when you submit the job or perhaps just show your jobs (not everybody’s). When the list is empty, you are done. Cons; If you submit fast enough, the jobs might not have had time enough to show up in the queue, misleading you to think you are done. If you just use the ”your own jobs” method, you can only run one main job at a time, i.e. compute on one project only.<br />
* Using '''sacct''' to show completed jobs. When your jobs are in the list, they are done.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=122Coding against databases2025-09-19T14:19:57Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3306 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Programme&diff=121Programme2025-09-18T17:21:28Z<p>WikiSysop: </p>
<hr />
<div>* 04/09 Lesson 1: [[Databases]]<br />
* 11/09 Lesson 2: [[MySQL]]<br />
* 18/09 Lesson 3: [[Coding against databases]]<br />
* 25/09 Lesson 4: [[Computer design]]<br />
* 02/10 Lesson 5: [[Queueing System]]<br />
* 09/10 Lesson 6: [[Distributed computing]], [https://evaluering.dtu.dk/ Midterm evaluation] - part 1<br />
* Fall holidays<br />
* 23/10 Lesson 7: [[What affects performance]], [https://evaluering.dtu.dk/ Midterm evaluation] - part 2<br />
* 30/10 Lesson 8: [[Algorithms]]<br />
* 06/11 Lesson 9: [[Parallel programming]]<br />
* 13/11 Lesson 10: [[More parallelism]]<br />
* 20/11 Lesson 11: [[Binary representation]]<br />
* 27/11 Lesson 12: [[Hash usage]]<br />
* 04/12 Lesson 13: [[Exercises and Organization]]<br />
* 09/12 Exam - [http://eksamensplan.dtu.dk/ the official DTU exam plan] [https://student.dtu.dk/eksamen/eksamensdatoer Exam dates]<br />
<br />
== Curious about the exam ==<br />
[https://teaching.healthtech.dtu.dk/22112/index.php/Aligning_expectations#About_the_exam Read here]</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=120Coding against databases2025-09-18T12:43:41Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3-2024.10-1/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3603 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Coding_against_databases&diff=119Coding against databases2025-09-17T19:27:06Z<p>WikiSysop: /* Exercises */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[MySQL]]<br />
|Next: [[Computer design]]<br />
|}<br />
== Material for the lesson ==<br />
Note: The powerpoint has been updated with new library and transactions, but not the video.<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=a891e09a-bec1-4ab1-8869-af2701258352 Using Python with MySQL]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_03-CodingDB.ppt Coding with Databases]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=68d743a1-9691-4690-b3a9-af170071ba0d Exercises]<br />
<br />
== Exercises ==<br />
Create one or two programs that can answer the following questions about the database you created last lesson. The data is from 2016 and all records are "complete" for the time period they were collected in. Specifically, all children's parents are in the database until we go so far back that records did not exists (the information is lost in the mists of time).<br><br />
The programs should run on the server '''pupil1''', not your own machine.<br><br />
The mysql connect call looks like this.<br><br />
cnx = mysql.connector.connect(user='DTUusername', passwd='YOUR_PASSWORD', db='DTUusername', host='localhost')<br><br />
You can find your password in the .my.cnf file in your home.<br><br />
You will mostly use python to solve the exercises. It is not intended that you use advanced mysql queries, but feel free.<br><br />
'''Note, use this python: /home/ctools/anaconda3_2021.11/bin/python3'''<br />
<br />
'''1)'''<br><br />
Who are born out of wedlock? Definition: A child is born out of wedlock if the biological father and mother is not married when the child is born. In the absence of information, you can not claim a child is born out of wedlock (that means if you only know the mother from the database, you can not claim the child is a bastard).<br />
<br />
'''2)'''<br><br />
Are there families who have had the same disease through 3 generations? Who and what disease in such case? It is not a trick question - you will find some.<br><br />
In answering this question, then any type of cancer should just be regarded as cancer, i.e. the same disease (this is not true in reality). 3 generations should be understood as a ”straight line of inheritance”, i.e. child, father, father’s father/mother. Child, mother and father’s father is not straight.<br />
<br />
If you look closely at your results, you will find some very interesting family patterns - such is the curse of (badly) generated data.<br />
<br />
'''3) Somewhat optional'''<br><br />
Run your programs again, but this time locally on your laptop, while you use the database on the server.<br><br />
This is an interesting challenge, which can be solved by SSH tunneling the mysql port 3603 to the server. The requirements for solving the exercise is understanding and setting up proper SSH tunneling as taught in lesson 1, and that mysql.connector is locally installed in your python distribution.<br />
In principle, this is done by 1 unix command setting up the SSH tunnel.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Programme&diff=118Programme2025-08-11T09:14:42Z<p>WikiSysop: </p>
<hr />
<div>* 04/09 Lesson 1: [[Databases]]<br />
* 11/09 Lesson 2: [[MySQL]]<br />
* 18/09 Lesson 3: [[Coding against databases]]<br />
* 25/09 Lesson 4: [[Computer design]]<br />
* 02/10 Lesson 5: [[Queueing System]]<br />
* 09/10 Lesson 6: [[Distributed computing]], [https://evaluering.dtu.dk/ Midterm evaluation] - part 1<br />
* Fall holidays<br />
* 23/10 Lesson 7: [[What affects performance]], [https://evaluering.dtu.dk/ Midterm evaluation] - part 2<br />
* 30/10 Lesson 8: [[Algorithms]]<br />
* 06/11 Lesson 9: [[Parallel programming]]<br />
* 13/11 Lesson 10: [[More parallelism]]<br />
* 20/11 Lesson 11: [[Binary representation]]<br />
* 27/11 Lesson 12: [[Hash usage]]<br />
* 04/12 Lesson 13: [[Exercises and Organization]]<br />
* 09/12 Exam - [http://eksamensplan.dtu.dk/ the official DTU exam plan]<br />
<br />
== Curious about the exam ==<br />
[https://teaching.healthtech.dtu.dk/22112/index.php/Aligning_expectations#About_the_exam Read here]</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=22112_High_Performance_Computing_in_Life_Science&diff=11722112 High Performance Computing in Life Science2025-08-11T09:11:57Z<p>WikiSysop: /* Earlier programmes */</p>
<hr />
<div>__NOTOC__<br />
'''Teacher:''' [https://www.inside.dtu.dk/da/dtuinside/generelt/telefonbog/person?id=816&cpid=214027&tab=2&qt=dtupublicationquery Peter Wad Sackett], pwsa@dtu.dk <!--, [https://www.inside.dtu.dk/en/dtuinside/generelt/telefonbog/person?id=130596&tab=0 Gisle Alberg Vestergaard], gisves@dtu.dk --> <br><br />
'''Language:''' The course is taught in English.<br><br />
'''Tools:''' Various servers will be introduced.<br><br />
'''Textbooks:''' There are no text books for the course. I will make do with powerpoints, videos and references to online resources. You can find the material under the individual lessons in the [[Programme]].<br><br />
'''Location:''' Building 208, room 001.<br><br />
'''Time:''' Thursday 13:00 - 17:00, module E1-B.<br><br />
<br />
== Course details ==<br />
The course is a practical introduction to working with supercomputers in Life Science. Using both "ordinary" laptops and computer clusters as a basis for lectures and exercises, we will cover subjects such as computer design, queueing systems, distributed computing, algorithms, parallel programming, performance and performance dependencies, database design and use, and practical organization of projects.<br />
<br />
{| <br />
|-<br />
| [[Aligning expectations]] || Mandatory read at course start<br />
|-<br />
| [[Programme]] || Fall 2024<br />
|-<br />
| [[Computerome]] || Some info on Computerome<br />
|}<br />
<br />
== Earlier programmes ==<br />
[[Programme Fall 2023]]<br><br />
[[Programme Fall 2024]]</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Programme_Fall_2024&diff=116Programme Fall 20242025-08-11T09:11:41Z<p>WikiSysop: Created page with "* 05/09 Lesson 1: Databases * 12/09 Lesson 2: MySQL * 19/09 Lesson 3: Coding against databases * 26/09 Lesson 4: Computer design * 03/10 Lesson 5: Queueing System * 10/10 Lesson 6: Distributed computing, [https://evaluering.dtu.dk/ Midterm evaluation] - part 1 * Fall holidays * 24/10 Lesson 7: What affects performance, [https://evaluering.dtu.dk/ Midterm evaluation] - part 2 * 31/10 Lesson 8: Algorithms * 07/11 Lesson 9: Parallel program..."</p>
<hr />
<div>* 05/09 Lesson 1: [[Databases]]<br />
* 12/09 Lesson 2: [[MySQL]]<br />
* 19/09 Lesson 3: [[Coding against databases]]<br />
* 26/09 Lesson 4: [[Computer design]]<br />
* 03/10 Lesson 5: [[Queueing System]]<br />
* 10/10 Lesson 6: [[Distributed computing]], [https://evaluering.dtu.dk/ Midterm evaluation] - part 1<br />
* Fall holidays<br />
* 24/10 Lesson 7: [[What affects performance]], [https://evaluering.dtu.dk/ Midterm evaluation] - part 2<br />
* 31/10 Lesson 8: [[Algorithms]]<br />
* 07/11 Lesson 9: [[Parallel programming]]<br />
* 14/11 Lesson 10: [[More parallelism]]<br />
* 21/11 Lesson 11: [[Binary representation]]<br />
* 28/11 Lesson 12: [[Hash usage]]<br />
* 05/12 Lesson 13: [[Exercises and Organization]]<br />
* 09/12 Exam - [http://eksamensplan.dtu.dk/ the official DTU exam plan]<br />
<br />
== Curious about the exam ==<br />
[https://teaching.healthtech.dtu.dk/22112/index.php/Aligning_expectations#About_the_exam Read here]</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=22112_High_Performance_Computing_in_Life_Science&diff=11522112 High Performance Computing in Life Science2025-08-11T09:11:08Z<p>WikiSysop: /* Earlier programmes */</p>
<hr />
<div>__NOTOC__<br />
'''Teacher:''' [https://www.inside.dtu.dk/da/dtuinside/generelt/telefonbog/person?id=816&cpid=214027&tab=2&qt=dtupublicationquery Peter Wad Sackett], pwsa@dtu.dk <!--, [https://www.inside.dtu.dk/en/dtuinside/generelt/telefonbog/person?id=130596&tab=0 Gisle Alberg Vestergaard], gisves@dtu.dk --> <br><br />
'''Language:''' The course is taught in English.<br><br />
'''Tools:''' Various servers will be introduced.<br><br />
'''Textbooks:''' There are no text books for the course. I will make do with powerpoints, videos and references to online resources. You can find the material under the individual lessons in the [[Programme]].<br><br />
'''Location:''' Building 208, room 001.<br><br />
'''Time:''' Thursday 13:00 - 17:00, module E1-B.<br><br />
<br />
== Course details ==<br />
The course is a practical introduction to working with supercomputers in Life Science. Using both "ordinary" laptops and computer clusters as a basis for lectures and exercises, we will cover subjects such as computer design, queueing systems, distributed computing, algorithms, parallel programming, performance and performance dependencies, database design and use, and practical organization of projects.<br />
<br />
{| <br />
|-<br />
| [[Aligning expectations]] || Mandatory read at course start<br />
|-<br />
| [[Programme]] || Fall 2024<br />
|-<br />
| [[Computerome]] || Some info on Computerome<br />
|}<br />
<br />
== Earlier programmes ==<br />
[[Programme Fall 2023]]<br />
[[Programme Fall 2024]]</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=114Computer design2025-08-06T21:00:25Z<p>WikiSysop: /* Material for the lesson */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Computers.ppt Computer design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
Note that the videos are no longer completely aligned with the powerpoints due to a shift in focus.<br />
<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computerome&diff=113Computerome2025-08-06T20:59:08Z<p>WikiSysop: /* Information on Computerome 2 and the Queueing System there */</p>
<hr />
<div>== Information on Computerome 2 and the Queueing System there ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=debcebae-c594-4daf-be11-af2701252f77 Computerome modules]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_Computerome2.ppt Computerome 2]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=d679e5c8-9372-4af7-9260-af270124ddf6 Introduction to the Queueing System]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=bca57da6-2c60-4218-96ac-af270124b673 Submitting jobs]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=e6e59e86-2928-4caf-b5c3-af2701248a3d Queue control and practical advice]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_ComputeromeQueue.ppt Queueing System]<br><br />
Resource: [https://www.computerome.dk/ Computerome 2 general information]<br><br />
<br />
Note that only the powerpoints are up to date. The videos are not likely to be updated as this material has been removed from the course.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=112Computer design2025-08-06T20:58:43Z<p>WikiSysop: /* Material for the lesson */</p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Computers.ppt Computer design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computerome&diff=111Computerome2025-08-06T20:57:22Z<p>WikiSysop: </p>
<hr />
<div>== Information on Computerome 2 and the Queueing System there ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=debcebae-c594-4daf-be11-af2701252f77 Computerome modules]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_Computerome2.ppt Computerome 2]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=d679e5c8-9372-4af7-9260-af270124ddf6 Introduction to the Queueing System]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=bca57da6-2c60-4218-96ac-af270124b673 Submitting jobs]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=e6e59e86-2928-4caf-b5c3-af2701248a3d Queue control and practical advice]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_ComputeromeQueue.ppt Queueing System]<br><br />
<br />
Note that only the powerpoints are up to date. The videos are not likely to be updated as this material has been removed from the course.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computer_design&diff=110Computer design2025-08-06T20:54:18Z<p>WikiSysop: </p>
<hr />
<div>{| width=500 style="float:right; margin-left: 10px; margin-top: -56px;"<br />
|Previous: [[Coding against databases]]<br />
|Next: [[Queueing System]]<br />
|}<br />
== Material for the lesson ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=87e6e835-22fc-4eb8-b5df-af27012554f9 Computer architecture and design]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=1171f7c8-74ea-43be-aff5-af2701250c46 Python IO performance & file buffers]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_04-Supercomputers.ppt Supercomputers]<br><br />
Resource: [https://www.computerome.dk/ Computerome 2 general information]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=ed903123-66d6-4dd8-ac09-af1700719d93 Exercises]<br />
<br />
<!--<br />
== Tasks to do ==<br />
* Set up the two-factor authentication with Entrust IdentityGuard soft token for logging in to Computerome 2.<br />
* Familiarize yourself with connecting to Computerome. Simple videos are avaliable on the web site.<br />
* Examine the modules system mentioned in the powerpoint and the web site.<br />
* Find the course directory /home/projects/pr_course. All files that I refer to during the course will be in that folder.<br />
* Set up a personal folder in the ''people'' folder, which is in the course directory. This personal folder is not your home folder, but the place where you put the files/exercises you create during the course. Set permissions so the group can read, otherwise we am unable to read your stuff. Make sure we can easily see what exercise you are solving.<br />
* Get a student account from me on '''DTU Healthtech''' local systems if you don't have an account already. You need such an account for database exercises later. Not urgent.<br />
<br />
== Prices on Computerome ==<br />
As of October 1, 2024 the prices will be changed to the following:<br />
CPU node hour: 5,25kr. (0,13125 kr. per core hour)<br />
GPU node hour: 11,30kr. (0,2825 kr. per core hour)<br />
Storage per TB/month: 59,00kr.<br />
--><br />
<br />
== Exercises ==<br />
Create four python scripts which copies the file human.fsa to your own home, using the four different methods mentioned in the powerpoint. Time the programs using ’time’, e.g.<br />
time python mycopy1.py<br />
Also time the simple Unix cp doing the same task. Delete your copy afterwards.<br><br />
Rank the programs according to performance and explain the difference in execution time.<br />
<br />
This is actually a bit hard to do correctly. Considering what you learned about file buffering, then reading the same file twice (or more) in a row will use the file buffers instead of real IO the second time.<br />
This means that the first time is always slowest, no matter how efficient the different programs are. You can't even be sure if another student is reading the same file at the same time and hence "disturbs" your measurements.<br />
<br />
A way around this problem is simply to use a computer with so little memory, that the file buffer system does not come into play. That is your laptop.<br><br />
Try to run the scripts on different machines. As minimum you have your laptop and pupil1. DTU's HPC system is also an option.</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=Computerome&diff=109Computerome2025-08-06T20:53:02Z<p>WikiSysop: Created page with "== Information on Computerome 2 and the Queueing System there == Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=d679e5c8-9372-4af7-9260-af270124ddf6 Introduction to the Queueing System]<br> Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=bca57da6-2c60-4218-96ac-af270124b673 Submitting jobs]<br> Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=e6e59e86-2928-4caf-b5c3-af2701248a3d Queue control and practical advice]<br> Powerpoint: [http..."</p>
<hr />
<div>== Information on Computerome 2 and the Queueing System there ==<br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=d679e5c8-9372-4af7-9260-af270124ddf6 Introduction to the Queueing System]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=bca57da6-2c60-4218-96ac-af270124b673 Submitting jobs]<br><br />
Video: [https://panopto.dtu.dk/Panopto/Pages/Viewer.aspx?id=e6e59e86-2928-4caf-b5c3-af2701248a3d Queue control and practical advice]<br><br />
Powerpoint: [https://teaching.healthtech.dtu.dk/material/22112/22112_05-Queue.ppt Queueing System]<br></div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=22112_High_Performance_Computing_in_Life_Science&diff=10822112 High Performance Computing in Life Science2025-08-06T20:51:00Z<p>WikiSysop: /* Course details */</p>
<hr />
<div>__NOTOC__<br />
'''Teacher:''' [https://www.inside.dtu.dk/da/dtuinside/generelt/telefonbog/person?id=816&cpid=214027&tab=2&qt=dtupublicationquery Peter Wad Sackett], pwsa@dtu.dk <!--, [https://www.inside.dtu.dk/en/dtuinside/generelt/telefonbog/person?id=130596&tab=0 Gisle Alberg Vestergaard], gisves@dtu.dk --> <br><br />
'''Language:''' The course is taught in English.<br><br />
'''Tools:''' Various servers will be introduced.<br><br />
'''Textbooks:''' There are no text books for the course. I will make do with powerpoints, videos and references to online resources. You can find the material under the individual lessons in the [[Programme]].<br><br />
'''Location:''' Building 208, room 001.<br><br />
'''Time:''' Thursday 13:00 - 17:00, module E1-B.<br><br />
<br />
== Course details ==<br />
The course is a practical introduction to working with supercomputers in Life Science. Using both "ordinary" laptops and computer clusters as a basis for lectures and exercises, we will cover subjects such as computer design, queueing systems, distributed computing, algorithms, parallel programming, performance and performance dependencies, database design and use, and practical organization of projects.<br />
<br />
{| <br />
|-<br />
| [[Aligning expectations]] || Mandatory read at course start<br />
|-<br />
| [[Programme]] || Fall 2024<br />
|-<br />
| [[Computerome]] || Some info on Computerome<br />
|}<br />
<br />
== Earlier programmes ==<br />
[[Programme Fall 2023]]</div>WikiSysophttps://teaching.healthtech.dtu.dk:443/22112/index.php?title=22112_High_Performance_Computing_in_Life_Science&diff=10722112 High Performance Computing in Life Science2025-08-06T20:50:47Z<p>WikiSysop: /* Course details */</p>
<hr />
<div>__NOTOC__<br />
'''Teacher:''' [https://www.inside.dtu.dk/da/dtuinside/generelt/telefonbog/person?id=816&cpid=214027&tab=2&qt=dtupublicationquery Peter Wad Sackett], pwsa@dtu.dk <!--, [https://www.inside.dtu.dk/en/dtuinside/generelt/telefonbog/person?id=130596&tab=0 Gisle Alberg Vestergaard], gisves@dtu.dk --> <br><br />
'''Language:''' The course is taught in English.<br><br />
'''Tools:''' Various servers will be introduced.<br><br />
'''Textbooks:''' There are no text books for the course. I will make do with powerpoints, videos and references to online resources. You can find the material under the individual lessons in the [[Programme]].<br><br />
'''Location:''' Building 208, room 001.<br><br />
'''Time:''' Thursday 13:00 - 17:00, module E1-B.<br><br />
<br />
== Course details ==<br />
The course is a practical introduction to working with supercomputers in Life Science. Using both "ordinary" laptops and computer clusters as a basis for lectures and exercises, we will cover subjects such as computer design, queueing systems, distributed computing, algorithms, parallel programming, performance and performance dependencies, database design and use, and practical organization of projects.<br />
<br />
{| <br />
|-<br />
| [[Aligning expectations]] || Mandatory read at course start<br />
|-<br />
| [[Programme]] || Fall 2024<br />
|-<br />
| [[Computerome] || Some info on Computerome<br />
|}<br />
<br />
== Earlier programmes ==<br />
[[Programme Fall 2023]]</div>WikiSysop