5/10/2017

The summary of the letters problem

The last approach of letters problem I put here, on the Anaconda Cloud.

Telegraph vs Guardian


And now we change the letters_2.py
program to make it more flexible. First of all, we define a function get_text that takes some text (a string) as an argument and returns three lists:
  • a list of small letters (for x axis),
  • list with the amount of given letters found in the text,
  • list of their rates.

Then we read the home pages of chosen web sites (to strings) and pass them as arguments respectively to the function get_text.. The returned lists we pass to bokeh figure function and line metkod to generate html page with the plot. From this page we can save the plot to the png image.
#letters_3.py
# -*- coding: utf-8 -*-

def get_text(s):
    chars = []
    for i in range(255):
        chars.append(0)
    
    for letter in s:
        indeks=ord(letter)-1
        chars[indeks]+=1          
    
    d = len(chars)
    A = []
    B = []
    C = []
    
    for i in range(d):
        if chars[i]>0 and (i+1)>=97 and (i+1)<=122:                
            A.append(chr(i+1))
            B.append(chars[i])
            C.append(0)
    
    sum_b = sum(B)
        
    for i in range(len(A)):
        C[i] = round(100.0*B[i]/sum_b,1)
    return A, B, C


url1="http://www.telegraph.co.uk/"
url2="http://www.guardian.co.uk/"


import urllib
                                        
sock = urllib.urlopen(url1) 
htmlSource1 = sock.read()
sock.close() 

sock = urllib.urlopen(url2) 
htmlSource2 = sock.read()
sock.close() 

X, Y1, Y = get_text(htmlSource1)

X, Z1, Z = get_text(htmlSource2)

print 'number of letters'

print 'Telegraph:', sum(Y1)
print 'Guardian:', sum(Z1)

from bokeh.plotting import figure, output_file, show

p = figure(x_range = X, title = 'Telegraph letters vs Guardian letters')
output_file("letters3.html")
p.line(X, Y, legend="telegraph", line_width=3)
p.line(X, Z, legend="guardian", color = 'red', line_width=3)

show(p)
The chart tells everything

Counting letters on a web page

Now we can change the previous program (letters_1.py). We'll take the text from a web page.

#letters_2.py
# -*- coding: utf-8 -*-

url="http://www.telegraph.co.uk/"

import urllib                                        
sock = urllib.urlopen(url) 
htmlSource = sock.read() 

s = htmlSource

chars = []
for i in range(255):
    chars.append(0)

for letter in s:
    indeks=ord(letter)-1
    chars[indeks]+=1          

d = len(chars)
X = []
Y = []

for i in range(d):
    if chars[i]>0 and (i+1)>=97 and (i+1)<=122:                
        X.append(chr(i+1))
        Y.append(chars[i])

sum_y = sum(Y)
print 'All small letters on (the home page)', url, ' ', sum_y
print '\nThe frequency of letters in %:\n '

for i in range(len(X)):
    Y[i] = round(100.0*Y[i]/sum_y,1)
    print '%5s %10.1f' %(X[i], Y[i])

And the results are:

All small letters on (the home page) http://www.telegraph.co.uk/   355470

The frequency of letters in %:
 
    a        9.1
    b        1.2
    c        3.8
    d        3.8
    e        9.7
    f        2.1
    g        3.1
    h        2.7
    i        7.3
    j        1.4
    k        0.7
    l        4.3
    m        4.3
    n        6.1
    o        5.1
    p        3.6
    q        1.0
    r        5.8
    s        6.8
    t        9.0
    u        2.0
    v        2.6
    w        1.4
    x        0.8
    y        1.5
    z        0.9

We can compare these results with the ones from letters.py. If we add some piece of code we can produce a bar chart that visualizes the frequency of letters. We'll use bokeh charts and data frame from pandas package. So we append such a code:

import pandas as pd

df = pd.DataFrame(
    {'letters': X,
     'freq': Y     
    })

from bokeh.charts import Bar, output_file, show


p = Bar(df, 'letters', values='freq',
        title="The frequency of letters in English texts", 
        bar_width=0.4, ylabel = "%",
        color = "green", legend = False)

output_file("letters.html")
show(p)

5/09/2017

Counting letters in English text.


Let's write a program that determines the frequency of occurring of different letters in English.
We start with that short text (from a story about Sherlock Holmes - 'A scandal in Bohemia').

I will comment this code tomorrow.
#letters_1.py
# -*- coding: utf-8 -*-

s = '''
To Sherlock Holmes she is always the woman. I have seldom heard him mention her under 
any other name. In his eyes she eclipses and predominates the whole of her sex. 
It was not that he felt any emotion akin to love for Irene Adler. All emotions, 
and that one particularly, were abhorrent to his cold, precise but admirably balanced mind. 
He was, I take it, the most perfect reasoning and observing machine that the world has seen, 
but as a lover he would have placed himself in a false position. '''

chars = []
for i in range(255):
    chars.append(0)

for letter in s:
    indeks=ord(letter)-1
    chars[indeks]+=1          

d = len(chars)
X = []
Y = []

for i in range(d):
    if chars[i]>0 and (i+1)>=97 and (i+1)<=122:                
        X.append(chr(i+1))
        Y.append(chars[i])

sum_y = sum(Y)
print 'All small letters in the text: ', sum_y
print '\nThe frequency of letters in %:\n '

for i in range(len(X)):
    Y[i] = round(100.0*Y[i]/sum_y,1)
    print '%5s %10.1f' %(X[i], Y[i])
And the program prints something like this.
All small letters in the text:  382

The frequency of letters in %:

    a        9.2
    b        1.6
    c        2.4
    d        3.9
    e       14.4
    f        1.6
    g        0.5
    h        6.8
    i        5.8
    k        0.8
    l        5.8
    m        3.7
    n        7.3
    o        7.9
    p        1.8
    r        5.8
    s        6.8
    t        7.6
    u        1.3
    v        1.3
    w        2.1
    x        0.3
    y        1.6

As we can see the most common letter as the letter 'e'.