Example code - Dicts
Files used in example
Web server statistics
It is of interest to the BOSS to see how many visitors a web site has and how many web pages has been seen. So therefore someone (you) has to make some statistics from the web server log file.
Example of a log entry
This is one line split in 3, so it can be seen.
52d3ccde.dynamic-ip.k-net.dk - - [07/Mar/2017:19:48:44 +0100] "GET /teaching/index.php?title=-&action=raw&gen=js&useskin=monobook HTTP/1.1" 200 290 "http://wiki.bio.dtu.dk/teaching/index.php/Course27617Spring2017" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0.2 Safari/602.3.12"
In this example, we are only interested in the red part , which is the IP-number or host name of the visitor, and the green part, which is the file served to the visitor. The problem is that the web is filled with search engines indexing the web sites, and they don't count as humans - yet. So they must be excluded from your statistics. Most of the indexing robots look at a file called /robots.txt, for what they are allowed to index. So here is a way to identify the non-humans. Also, many web pages uses pictures, meaning that the web server also serves picture files to the visitors. These pictures do not count as a page view (since they are being part of a page). Picture files mostly have the extensions .gif, .jpg, .png and .jpeg, so they can also be identified.
The program
#!/usr/bin/python3 # makes statistics on a apache webserver log file import sys # get file try: logfile = open('apachewiki.log', 'r') except IOError as err: print('There seems to be a problem with the file:', str(err)) sys.exit(1) # Dict for counting page views hosts = dict() # Set of search engine crawlers crawler = set() for line in logfile: field = line.split() # Is this a crawler if field[6] == '/robots.txt': crawler.add(field[0]) # Is this a pic? if not field[6].endswith(('.gif','.jpg','.png','.jpeg')): # Must be a page view then if field[0] in hosts: hosts[field[0]] += 1 else: hosts[field[0]] = 1 logfile.close() # Remove page views made by crawlers crawlCount = 0 for item in crawler: crawlCount += hosts[item] del hosts[item] # Sort page views by size # Print top ten print('Top 10 viewers') for host in sorted(hosts.keys(), reverse=True, key=hosts.get)[:10]: print(host, hosts[host]) # Get total page views total = 0 for val in list(hosts.values()): total += val print('Unique visitors:', len(hosts)) print('Unique crawlers:', len(crawler)) print('Total visitor page views:', total) print('Total crawler page views:', crawlCount)