Unix pipes - Revision history

Gabre at 14:04, 17 December 2024

2024-12-17T14:04:52Z

@@ Line 565: / Line 565: @@
 </pre>
-Which makes total of nearly 3 minutes and 30 seconds, `without coding time.` Please note that the prime checker works way much slower than the random integer generator.
+Which makes a total of nearly 3 minutes and 30 seconds, `without coding time.` Please note that the prime checker works way much slower than the random integer generator.
 'Using Pipes'
@@ Line 582: / Line 582: @@
 </pre>
 It made a difference, yes? A minute down seems not that big but imagine much bigger tasks. We always prefer lower time consumptions with also `lower coding times.`

Gabre at 21:58, 16 December 2024

2024-12-16T21:58:42Z

← Older revision		Revision as of 23:58, 16 December 2024
Line 553:		Line 553:
	The runtime of both are, respectively:		The runtime of both are, respectively:

	~~```~~		<pre>
	real 0m44.270s		real 0m44.270s
	user 0m42.046s		user 0m42.046s
	sys 0m2.200s		sys 0m2.200s
			</pre>
	and		and
			<pre>
	real 2m39.985s		real 2m39.985s
	user 2m6.337s		user 2m6.337s
	sys 0m25.831s		sys 0m25.831s
	~~```~~		</pre>

	Which makes total of nearly 3 minutes and 30 seconds, `without coding time.` Please note that the prime checker works way much slower than the random integer generator.		Which makes total of nearly 3 minutes and 30 seconds, `without coding time.` Please note that the prime checker works way much slower than the random integer generator.

	~~###~~ Using Pipes		'Using Pipes'

	Let's pipe them together! We will use the code below:		Let's pipe them together! We will use the code below:

	~~```bash~~
	time python3 random_int_generator.py 50000000 --min 100 --max 1000000 \| python3 prime_checker.py		time python3 random_int_generator.py 50000000 --min 100 --max 1000000 \| python3 prime_checker.py
	~~```~~

	The total runtime of this code is:		The total runtime of this code is:
	~~```~~		<pre>
	real 2m41.284s		real 2m41.284s
	user 2m21.816s		user 2m21.816s
	sys 0m15.455s		sys 0m15.455s
	~~```~~		</pre>
	It made a difference, yes? A minute down seems not that big but imagine much bigger tasks. We always prefer lower time consumptions with also `lower coding times.`		It made a difference, yes? A minute down seems not that big but imagine much bigger tasks. We always prefer lower time consumptions with also `lower coding times.`

Gabre: /* RSA Checker */

2024-12-16T21:56:36Z

RSA Checker

← Older revision		Revision as of 23:56, 16 December 2024
Line 526:		Line 526:
	But since we do not have the prime numbers in files, we need to utilize `file descriptors`! A way to use two file descriptors at the same time is by bundling commands together with parenthesis. It will bundle the codes together and redirects the output of all code inside the parentheses. Let's break it down, using an example:		But since we do not have the prime numbers in files, we need to utilize `file descriptors`! A way to use two file descriptors at the same time is by bundling commands together with parenthesis. It will bundle the codes together and redirects the output of all code inside the parentheses. Let's break it down, using an example:

	python3 /home/projects/22126_NGS/exercises/pipes/RSAcompute.py <(python3 /home/projects/22126_NGS/exercises/pipes/~~random_integer_generator~~.py \| python3 /home/projects/22126_NGS/exercises/pipes/prime_checker.py) <(python3 /home/projects/22126_NGS/exercises/pipes/~~random_integer_generator~~.py \| python3 /home/projects/22126_NGS/exercises/pipes/prime_checker.py)		python3 /home/projects/22126_NGS/exercises/pipes/RSAcompute.py <(python3 /home/projects/22126_NGS/exercises/pipes/random_int_generator.py --min 1000000 --max 10000000 10000 \| python3 /home/projects/22126_NGS/exercises/pipes/prime_checker.py) <(python3 /home/projects/22126_NGS/exercises/pipes/random_int_generator.py --min 1000000 --max 10000000 10000\| python3 /home/projects/22126_NGS/exercises/pipes/prime_checker.py)

	We knew the part inside the parentheses, it outputs a list of prime numbers. Now, we do it two times since we need ~~a pair~~ of prime numbers. We bundled the parts that output prime numbers and redirected them to the `~~RSAchecker~~.py`. `<~~` indicates that~~ the ~~input goes~~ into ~~the~~ file, ~~so that is the reverse of what~~ we ~~did in the previous section~~.		We knew the part inside the parentheses, it outputs a list of prime numbers. Now, we do it two times since we need 2 files of prime numbers. We bundled the parts that output prime numbers and redirected them to the `RSAcompute.py`. "<( PROCESS HERE )" will take the stdout of PROCESS HERE and turn it into a file descriptor. Essentially, we have saved writing 4 files here using file descriptors and, RSAcompute.py goes as fast as random_int_generator and prime_checker can go.

	~~And voila!~~ It worked perfectly, and we have ~~valid prime number~~ pairs for encryption.		It worked perfectly, and we have RSA pairs for encryption.

	~~Congrats! Your encryption works!~~

	# Benchmarking		== Benchmarking ==

	Love to see all codes in action, but checking if they are working optimized is another concern since we need everything (ideally) low-cost at the means of time, calculations, and such. So we need to benchmark our pipeline to see if some code bottlenecks or raises errors during the pipeline. For this benchmarking, we are going to use the `time` function of Linux (see Linux Concepts Section, if you already forgot :D.) Let's start building our pipeline!		Love to see all codes in action, but checking if they are working optimized is another concern since we need everything (ideally) low-cost at the means of time, calculations, and such. So we need to benchmark our pipeline to see if some code bottlenecks or raises errors during the pipeline. For this benchmarking, we are going to use the `time` function of Linux (see Linux Concepts Section, if you already forgot :D.) Let's start building our pipeline!

	# Time Efficiency Benchmarking		=== Time Efficiency Benchmarking ===

	## Random Integer Generator and Prime Checker		'Random Integer Generator and Prime Checker'

	Based on our knowledge from the previous section, we know that we can achieve this pipeline with various methods, like using intermediate files, file descriptors, or pipes. So when we need to pick any of them, the concern is cost efficiency, and in this case, it is time efficiency. Let's try every method and check if it really changes that much. We are going to generate 50.000.000 numbers in every test, which are between 100 and 1.000.000. All tests are undergone with 6GB RAM and 2GB Swap Memory.		Based on our knowledge from the previous section, we know that we can achieve this pipeline with various methods, like using intermediate files, file descriptors, or pipes. So when we need to pick any of them, the concern is cost efficiency, and in this case, it is time efficiency. Let's try every method and check if it really changes that much. We are going to generate 50.000.000 numbers in every test, which are between 100 and 1.000.000. All tests are undergone with 6GB RAM and 2GB Swap Memory.

	~~###~~ Using Intermediate Files		'Using Intermediate Files'

	We will, for testing intermediate files, generate a file consists all random integers and feed `prime checker` with them. In order to achieve that, we will check them separately and add up later. We will use the code:		We will, for testing intermediate files, generate a file consists all random integers and feed `prime checker` with them. In order to achieve that, we will check them separately and add up later. We will use the code:

	~~```bash~~		time python3 random_int_generator.py 50000000 --min 100 --max 1000000 > random_integers.txt
	time python3 random_int_generator.py 50000000 --min 100 --max 1000000 > random_integers.txt		time python3 prime_checker.py random_integers.txt > prime_list_first.txt
	time python3 prime_checker.py random_integers.txt > prime_list_first.txt
	~~```~~

	The runtime of both are, respectively:		The runtime of both are, respectively:

Gabre: /* RSA Checker */

2024-12-16T21:51:35Z

RSA Checker

← Older revision		Revision as of 23:51, 16 December 2024
Line 509:		Line 509:
	public_key, private_key = generate_rsa_keys(p, q)		public_key, private_key = generate_rsa_keys(p, q)
	# Print keys to stdout		# Print keys to stdout
	print(f"Public Key: {public_key[0]}\n{public_key[1]}")		print(f"Public Key: {public_key[0]}\t{public_key[1]}\tPrivate Key: {private_key[0]}\t{private_key[1]}")
	~~print(f"Private~~ Key: {private_key[0]}\n{private_key[1]}")
	except ValueError as e:		except ValueError as e:
	print(f"Error generating keys for primes {p} and {q}: {e}")		print(f"Error generating keys for primes {p} and {q}: {e}")

Gabre: /* RSA Checker */

2024-12-16T21:49:02Z

RSA Checker

← Older revision		Revision as of 23:49, 16 December 2024
Line 499:		Line 499:
	primes2 = read_primes_from_file(prime_file_2)		primes2 = read_primes_from_file(prime_file_2)

	if len(primes1) != len(primes2):		# Take the minimum length of the two prime lists
	~~print("Error~~: ~~Both files must contain the same number of primes.")~~		min_length = min(len(primes1), len(primes2))
	~~sys.exit(1)~~		primes1 = primes1[:min_length]
			primes2 = primes2[:min_length]

	# Generate RSA keys for each pair of primes		# Generate RSA keys for each pair of primes
Line 508:		Line 509:
	public_key, private_key = generate_rsa_keys(p, q)		public_key, private_key = generate_rsa_keys(p, q)
	# Print keys to stdout		# Print keys to stdout
	print(f"Public ~~Keys~~: {public_key[0]}\t{public_key[1]}~~\tPrivate Keys~~: {private_key[0]}\t{private_key[1]}")		print(f"Public Key: {public_key[0]}\n{public_key[1]}")
			print(f"Private Key: {private_key[0]}\n{private_key[1]}")
	except ValueError as e:		except ValueError as e:
	print(f"Error generating keys for primes {p} and {q}: {e}")		print(f"Error generating keys for primes {p} and {q}: {e}")
Line 514:		Line 516:
	if __name__ == "__main__":		if __name__ == "__main__":
	main()		main()

	</pre>		</pre>

Gabre: /* RSA Checker */

2024-12-16T21:42:31Z

RSA Checker

Show changes

Gabre: /* Prime Checker (Naive) */

2024-12-16T21:38:29Z

Prime Checker (Naive)

← Older revision		Revision as of 23:38, 16 December 2024
Line 411:		Line 411:
	if num <= 1:		if num <= 1:
	return False		return False
	for i in range(2, num):		for i in range(2, int(math.sqrt(num))+1):
	if num % i == 0:		if num % i == 0:
	return False		return False

Gabre: /* RSA Checker */

2024-12-16T21:36:43Z

RSA Checker

← Older revision		Revision as of 23:36, 16 December 2024
Line 512:		Line 512:
	if __name__ == "__main__":		if __name__ == "__main__":
	main()		main()

	</pre>		</pre>


	The RSAchecker.py script is designed to ~~generate~~ RSA key pairs from two lists of prime numbers.		The RSAchecker.py script is designed to check RSA key pairs from two lists of prime numbers. An RSA key pair consists of a public key and a private key. These key pairs are fundamental to the RSA encryption algorithm, which is widely used in secure communications. If you want to learn about it further, you can find information about RSA encryption further on the internet.

	It reads prime numbers from two files, computes the RSA key pair for each pair of primes, and checks if the encryption and decryption process is successful~~. If you want to learn about it further, you can find information about RSA encryption further on the internet~~.		The script reads prime numbers from two files, computes the validity of the RSA key pair for each pair of primes, and checks if the encryption and decryption process is successful.

	To use this script, you need to provide two files containing prime numbers, one is so-called `public keys`, and the other one is `private keys`. Each file should have one prime number per line. The script will read these files, compute RSA key pairs, and verify their validity.		To use this script, you need to provide two files containing prime numbers, one is so-called `public keys`, and the other one is `private keys`. Each file should have one prime number per line. The script will read these files, compute RSA key pairs, and verify their validity.

	The correct way to use this script follows:		The correct way to use this script follows:
	python RSAchecker.py [file_with_primes_1] [file_with_primes_2]
			python RSAchecker.py [file_with_primes_1] [file_with_primes_2]

	But since we do not have the prime numbers in files, we need to utilize `file descriptors`! A way to use two file descriptors at the same time is by bundling commands together with parenthesis. It will bundle the codes together and redirects the output of all code inside the parentheses. A little confusing, right? Let's break it down, using an example:		But since we do not have the prime numbers in files, we need to utilize `file descriptors`! A way to use two file descriptors at the same time is by bundling commands together with parenthesis. It will bundle the codes together and redirects the output of all code inside the parentheses. A little confusing, right? Let's break it down, using an example:

Gabre: /* Prime Checker (Naive) */

2024-12-16T15:42:06Z

Prime Checker (Naive)

← Older revision		Revision as of 17:42, 16 December 2024
Line 444:		Line 444:
	As we specified earlier, the random integer generator generates 100 numbers between 10 and 100, so our prime checker would be fed with them. It will then print out only `prime ones`. That means, the original output of `random_int_generator.py` would be omitted since it has been redirected to the `prime_checker.py`. Also this prime checker code provides the runtime to the user, for assessing the performance of this code.		As we specified earlier, the random integer generator generates 100 numbers between 10 and 100, so our prime checker would be fed with them. It will then print out only `prime ones`. That means, the original output of `random_int_generator.py` would be omitted since it has been redirected to the `prime_checker.py`. Also this prime checker code provides the runtime to the user, for assessing the performance of this code.

	# RSA Checker		=== RSA Checker ===

	Here comes the code first:		Here comes the code first:

	~~```python~~		<pre>
	import sys		import sys

Line 511:		Line 512:
	if __name__ == "__main__":		if __name__ == "__main__":
	main()		main()
	~~```~~
			</pre>


	The RSAchecker.py script is designed to generate RSA key pairs from two lists of prime numbers.		The RSAchecker.py script is designed to generate RSA key pairs from two lists of prime numbers.

	It reads prime numbers from two files, computes the RSA key pair for each pair of primes, and checks if the encryption and decryption process is successful. If you want to learn about it further, you can find information about RSA encryption further on the internet.		It reads prime numbers from two files, computes the RSA key pair for each pair of primes, and checks if the encryption and decryption process is successful. If you want to learn about it further, you can find information about RSA encryption further on the internet.

Gabre: /* Prime Checker (Naive) */

2024-12-16T15:41:03Z

Prime Checker (Naive)

← Older revision		Revision as of 17:41, 16 December 2024
Line 395:		Line 395:


	A number is prime if it can be divided by 1 and itself. 19 and 17 are prime but not 14 (divide by 2,7) nor 16 (divided by 2,4,8). The most naive way to check if a number is prime is to check every number between 1 and the number to see if it can be divided. A slightly smarter way is to check ~~all the way~~ to the square root of the number.		A number is prime if it can be divided by 1 and itself. 19 and 17 are prime but not 14 (divide by 2,7) nor 16 (divided by 2,4,8). The most naive way to check if a number is prime is to check every number between 1 and the number to see if it can be divided. A slightly smarter way is to check up to the square root of the number.

	The code for the naive approach seems like this:		The code for the naive approach seems like this:
Line 404:		Line 404:
	import itertools		import itertools

	parser = argparse.ArgumentParser(description="Prime Number ~~Checker for Internship Task~~. This program checks if the input numbers are prime and writes the primes to stdout.")		parser = argparse.ArgumentParser(description="Prime Number checker . This program checks if the input numbers are prime and writes the primes to stdout.")
	args = parser.parse_args()		args = parser.parse_args()

Line 411:		Line 411:
	if num <= 1:		if num <= 1:
	return False		return False
	~~if num <= 3:~~		for i in range(2, num):
	~~return True~~		if num % i == 0:
	~~if num % 2 == 0 or num % 3 == 0:~~
	~~return False~~
	for i in range(5, ~~int(math.sqrt(~~num~~)) + 1, 6~~):
	if num % i ~~== 0 or num % (i + 2)~~ == 0:
	return False		return False
	return True		return True