List Comprehensions is the most interesting feature of Python i have came across. It allows to make a new list with certain operation done within a list. We can actually write a list with simple operations in it.

[1, 2 + 5, 3 + 7, 10 - 8] #This will give [1, 7, 10, 2]

Similarly i can write a for loop in a list. Like [(output of loop) (loop)]

[x for x in range(5)] # This will give [0, 1, 2, 3, 4]

See this list had two parts embedded in it. “for x in range(5)”, the second part and “x” which is the output of the for loop as the first part. We can also make the first part an operation with the for loop output.

[x * 10 for x in range(5)]          # Which is [0, 10, 20, 30, 40]

["Agni-%d" % x for x in [1,2,3]]    # Which is ["Agni-1", "Agni-2", "Agni-3"]

[x % 2 == 0 for x in [1, 2, 3, 4]]  # Which is [False, True, False, True]

So this time it became [(operation) (loop)]. Now we can also add a condition part to it.

a =  ["Washington", "Roosvelt", "Ronald", "Nixon"]
[x for x in a if "a" in x] # Which gives ["Washington", "Ronald"]

Now we have three parts in our list comprehension syntax. [(output if condition satisfied) (for loop) (condition)].

Let me conclude this with an example. We have two lists a and b. We need to make a new list with elements common to both the lists. This can be easily done with set. For our example i am not using set. First let me solve it in the flat way with out list comprehension.

Without List Comprehension

a = [1,2,3,4,5,6,7]
b = [8,9,3,1,10,11]
c = []

for x in a:
    if x in b:
        c.append(x)

With List Comprehension

a = [1,2,3,4,5,6,7]
b = [8,9,3,1,10,11]
c = [x for x in a if x in b]

This makes me feel list comprehension is cute.

Python Storm is perhaps the best Python ORM. In an ORM we will have to define a class for a table that we are going to use. Thus the name Object Relational Mapping as we are relating an object of a class with a table in a database. With the object of the class we defined for a table we do the database transactions.

Import

from storm.locals import *

To explain the basic db operations with Storm, Lets consider a table named Generals. The table contains two fields, name and nationality of few WW2 Generals. Lets consider that the table is in the db names WW2.

Table generals

+----+-------------------+----------+
| id | name              | country  |
+----+-------------------+----------+
|  1 | Erwin Rommel      | Germany  |
|  2 | Montgomery        | Britiain |
|  3 | Friedrich Paulus  | Germany  |
|  4 | George Patton     | USA      |
|  5 | Douglas MacArthur | USA      |
|  6 | Dwight Eisenhower | USA      |
+----+-------------------+----------+

Here id is the primary key and fileds name and country are varchar. First we are going to define a class to represent the table.

Class for Table Generals

class Generals(object):
    __storm_table__ = "generals"   # Setting the actual table name
    id = Int(primary=True)         # Setting id is primary key
    topic = Unicode()              # All varchar types are defined as Unicode in Storm
    definition = Unicode()

Once we have the class, our program can understand the structure of the table. But still there is no connection with the database. Here i am going to explain Storm with Mysql. Our database adapter is mysql, database name is ‘WW2’, mysql user name is ‘historian’, password is ‘secret’ and our mysql server runs in localhost. The following is the connection procedure.

Database Connection

database = create_database("mysql://historian:secret@localhost/WW2") # Defining Connection parameters
store = Store(database)                                              # Connection established

Now store is the connection variable. Lets try a few SELECT operations with Storm. Will try selecting general ‘Montgomery’ and finding out his nationality.

SELECT a single row

result = store.find(Generals, Generals.name == u"Montgomery").one()
print result.country # Will print 'Britain'

SELECT multiple rows

results = store.find(Generals, Generals.country == u"USA")
# This for loop will print id, name, country of all generals who's country is 'USA'. The return value of storm is a result set
for result in results:
    print result.id
    print result.name
    print result.country

SELECT with LIKE condition

results = store.find(Generals, Generals.name == u"D%")
# This for loop will print 'Douglas MacArthur' and 'Dwight Eisenhower'
for result in results:
    print result.name

SELECT with id

store.get(Generals, 1).name    # Will return 'Erwin Rommel'
store.get(Generals, 1).country # Will return 'Germany'

Now time for Insert operation. Lets insert a new gerneral, Vasily Chuikov to table generals. This is done by making on object of class Generals, assigning values to the fields and adding it to table.

INSERT

obj = Generals()                # Creating a new object of Generals

obj.name = u"Vasily Chuikov"    # Setting name as 'Vasily Chuikov'
obj.country = u"Germany"        # Setting country as 'Germany'

store.add(obj)                  # Adding updated object to store
store.flush()                   # The updation is written to table

Note that store.add() will only add the new updated objedt to store. This wont reflect in the db. store.flush() will write the data to db.

Lets try Updation operation. We inserted wrong data last time. ‘Vasily Chuikov’ is not German instead from Soviet Union. We need to update this change.

UPDATE

store.find(Generals, Generals.name == u"Vasily Chuikov").set(country = u"Soviet Union")

Executing a query with Storm. Suppose we dont know the structure of the table and hence we havent defined the class for the table. Sill we can do any db operation with Storm.

EXECUTION of raw db Qery

store.execute("SELECT * FROM tanks").get_all() # Will select all data in table tanks, and return a list of tuples.
store.execute("SELECT * FROM ships").get_one() # Will select first data from ships.

Storm has a lot more features that makes db transactions easier. At any point in a transaction we can rollback with store.rollback() or save the whole transaction with store.commit() Once you have commited a transaction, flush is no more required.

MySQLdb is a Python library to use MySQL database. MySQLdb is the first database wrapper i ever used. Its quite simple and good. Anyone who knows MySQL commands can use MySQLdb with ease.

First step is to import MySQLdb to your code

Importing

import MySQLdb

Connecting to MySQL. The parameters we need to connect for DB are db user name, password for that user, host address and the db name. With these data was make a connection to the db from our code.

Connection

host_ip = "localhost" # This is to used database in the system itself. In case of remote db, use the ip of the system.
user_name = "admin"
password = "secret"
db_name = "dictionary" # Name of the db you intent to use.

db_connection = MySQLdb.connect(host = host_ip, user = user_name, passwd = password, db = db_name)

Now that we have the connection, we can perform db operations. For using the db operation methods we need to create a cursor of the db connection. The cursor class provides the methods that we acn use to do db operations.

Cursor

db_cursor = db_connection.cursor()

Lets try a SELECT operation with the cursor. Lets say that the db dictionary has a table named word_meanings. And lets try selecting one meaning for a word. The table has three feilds namely id, word, meaning.

SELECT one row

db_cursor.execute("SELECT * FROM word_meanings WHERE word = 'data'")
result = db_cursor.fetchone()

The result in variable meaning is a tuple. The tuple contains all the data in that particular row selected

Result

result = (1273, 'data', 'Facts and statistics collected together for reference or analysis') #id, word, meaning

meaning = result[2]

What if you want to select more than one row. If the word has more than one menings. In that case we can use fetchmany(n) where the n number of rows will be selected. Or fetchall() will select all rows that satisfies your query.

SELECT multiple rows

db_cursor.execute("SELECT * FROM word_meanings WHERE word = 'data'")
result = db_cursor.fetchall()

In this case the variable result will be a tuple of tuples.

Result

result = (
          (1273, 'data', 'Facts and statistics collected together for reference or analysis')
          (1274, 'data', 'These data represent the results of our analyses')
          (1275, 'data', 'information in numerical form that can be digitally transmitted or processed ')
          )

For any MySQL query like SELECT, SHOW, DESC that returns a set of data, we need to do fetchone(), fetchall() or fetchmany(n). For those query dont return any data such as INSERT, UPDATE, CREATE, we just need to do execute() alone.

Now lets try an INSERT query. We have just made a new contribution to English dictionary. So gonna insert the word and its meaning to table word_meanings.

INSERT

db_cursor.execute("INSERT INTO word_meanings(word, meaning) VALUES ('indictionariate', 'A word that is not in the dictionary')")

But as we are programmers, we dont want to pass a string query to execute().Instead we would prefer to pass only the required data to the query and not making the whole query all the time.

Passing Data to Query

new_word = "indictionariate"
word_meaning = "A word that is not in the dictionary"
db_cursor.execute("INSERT INTO word_meanings(word, meaning) VALUES (%s, %s)", (new_word, word_meaning))

Which is equal to

Updating Query

query = "INSERT INTO word_meanings(word, meaning) VALUES (%s, %s)" % (new_word, word_meaning)
db_cursor.execute(query)

What if you want to make multiple entries with same query. I know you must be thinking about using a loop over the above code to do this. Well its not a bad idea. But MySQLdb has its own method to do the same.

Multiple Querying

db_cursor.execute("INSERT INTO word_meanings(word, meaning) VALUES (%s, %s)", 
 [(word1, meaning1),
  (word2, meaning2),
  (word3, meaning3)]
)

So this is how you used MySQLdb. A vey important thing is to close a db connection once usage is done. You dont wanna leave a connetion open. Bad for your MySQL server. So always the final step of your db operation is close the db commection.

Closing DB Connection

db_connection.close()

Even though MySQLdb is good and simple, i wont advise it to be used for a long running program, which is very dynamic. It is good for a one shot code. For long running processes use some ORMs.

I had been filling up my blog with tiny and fancy Python stuff for sometime. Now i am thinking of posting something useful. I have no idea where and what to start.

All programmers has to deal with data. Even though i am not an expert in it, i had been using quite a few of them for last 2 years. MySQL, Scribe, Redis, TokyoCabinet etc. And always end up at writing custom ETL for individual projects. So form my next post i will be dealing with them.

Now i wish if i had taken DBMS seriously at college. :D

Also i have thought of making this blog a mix of Python and Ruby.

Today i stumbled upon something really simple but cool. I was on IRC just going through the talks in Ruby channel. A guy known as maasha in ruby channel was asking a doubt.

Doubt: How to print in the same line again and again clearing the first print?

Its my habit to tryout things in Python first. This seemed to be interesting since it would be cool if i could print 1% to 100% if my program is running something, like loading data to a database. After a few tries and reading i found the solution :) .

Python

import time
from sys import stdout

for x in range(0, 101):
    stdout.write("\r%d%s" % (x,"%"))
    stdout.flush()
    time.sleep(0.2)

stdout.write("\rComplete !")
stdout.flush()

Ruby

1.upto(100) do |i|
  STDOUT.write("\r#{i}%")
  STDOUT.flush
  sleep(0.2)
end

STDOUT.write("\rComplete !")

This piece of code will print from 1% to 100% with a time delay of 0.2 sec.

I remember my college classes in which threading was explained to us. I dint had any idea what a thread meant. In fact i never tried to understand it. At this point i consider a thread as a worker who dont interfere with the execution of my program when its give a task, who will get back to me when its task is done.

Consider a scenario in which i want to execute a function that will take some time to complete and i dont want to wait for this and want to continue with further operations. In this case i would prefer a thread to execute that function.

Remember the FizzBuzz problem i mentioned in the last post. Will try to run using a thread. Here i am going to run an infinite while loop that will print a global variable ‘status’ until it gets the value ‘Done’.

LOOP

status = None

while True:
    if status == "Done":
        break
    else:
        print status

The above loop will never get terminated. So let me define a function that updates the value of the variable ‘status’ according to FizzBuzz iteration. And i am going to run the function with a thread. So that the while loop will be running independently.

THREADING METHOD fizzbuzz

import thread
import threading
import time

status = None

def fizzbuzz(n, sleeptime):
    """This function runs a for loop and updates status
       as Fizz when i is a multiple of 3, updates status
       as Buzz when i is a multiple of 5 and FizzBuzz
       when i is a multiple of 3 and 5"""

    global status

    for i in range (1, n):
        if i % 15 == 0: status = "%d : FizzBuzz" % i
        elif i % 3 == 0: status = "%d : Fizz" % i
        elif i % 5 == 0: status = "%d : Buzz" % i
        else: status = i
        time.sleep(sleeptime)

    statu = "Done"

    thread.interrupt_main() # Kills the tread

# Starting a thread that executes the method fizzbuzz
thread.start_new_thread(fizzbuzz, (100, 0.5))

while True:
    if status == "Done":
        break
    else:
        print status #Prints the status that is being updated by the thread.

You will notice that the while loop and the tread is running at the same time. And this is what a thread is meant for.

The above code is only meant for an example. And i am using the global variable ‘status’ only to show that the thread is updating the variable. Its not a good practice to mix global variables and threads.

I had always wondered how to print in same line continuously. This is a visually pleasing idea when your script is uploading something. Also i can be sure that the script is running well. Today i had to write something similar and also added the this continuous printing in same line to it. Here is how continuous printing in same line works.

EXAMPLE

from sys import stdout

    stdout.write("Loading ")
    for x in range(1000):
        stdout.write(">")

As i did it i got this fancy idea of solving Fizz Buzz with this along with color printing. Fizz Buzz was the problem given to me on my interview two years before.

Fizz Buzz:

1. Iterate from 1 to 100
2. Print Fizz if number is a multiple of 3
3. Print Buzz if number is a multiple of 5
4. Print Fizz Buzz if number is multiple of both

I solved it then, perhaps the in the worst method.

The following code will continuously print numbers from 1 to 100. If the number is a multiple of 3 the number will be printed in red, for 5 blue and for multiple of both green.

Fizz Buzz

from sys import stdout

for x in range(1, 100):
    if x % 3 == 0 and x % 5 == 0:
        stdout.write(" \033[1;32m%d\033[1;m" % x)
    elif x % 5 == 0:
        stdout.write(" \033[1;34m%d\033[1;m" % x)
    elif x % 3 == 0:
        stdout.write(" \033[1;31m%d\033[1;m" % x)
    else:
        stdout.write(" %d" % x)

But i dont advice you to do this on your interview. Interview is not about being fancy.

Here is Python class that lets you print a string in color.

CODE

class ColorPrint:

    def __init__(self, text):
        self.text = text
        self.color_codes = {
            'green' : '\033[1;32m%s\033[1;m',
            'blue' : '\033[1;34m%s\033[1;m',
            'gray' :'\033[1;30m%s\033[1;m',
            'red' : '\033[1;31m%s\033[1;m',
            'yellow' : '\033[1;33m%s\033[1;m',
            'magenta' : '\033[1;35m%s\033[1;m',
            'cyan' : '\033[1;36m%s\033[1;m',
            'white' : '\033[1;37m%s\033[1;m',
            'crimson' : '\033[1;38m%s\033[1;m',
            'RED' : '\033[1;41m%s\033[1;m',
            'GREEN' : '\033[1;42m%s\033[1;m',
            'BROWN' : '\033[1;43m%s\033[1;m',
            'BLUE' : '\033[1;44m%s\033[1;m',
            'MAGENTA' : '\033[1;45m%s\033[1;m',
            'CYAN' : '\033[1;46m%s\033[1;m',
            'GRAY' : '\033[1;47m%s\033[1;m',
            'CRIMSON' : '\033[1;48m%s\033[1;m'
            }

    def __getattr__(self, color):
        if (self.color_codes.has_key(color)):
            print self.color_codes[color] % self.text
        else:
            print self.text

You can copy this class to your code and use it. But i would prefer that you use it as a module. For that copy the code and save it as a python file, example color.py. Use it as in the following example.

Use Example

from color import ColorPrint

ColorPrint("hello").red
ColorPrint("hello").green

And if you want to highlight the color over your string

Use Example

from color import ColorPrint

ColorPrint("hello").RED
ColorPrint("hello").GREEN

The class will print the string normally in case the color dosent exist in python color codes. Hope this piece of code will be of use to you.

Its time to mention in this blog that i can do Ruby also. I started with Python which i still love. Its been a while since i have been going around Ruby. I know its a bit early for this. I have my first gem published.

https://rubygems.org/gems/rubykeyword

Like C C++ and Java, Python also has the feature inheritance. I dont find more convincing example than a mother and daughter to explain inheritance. Lets say that a daughter is inheriting eye colour and hair colour from her mother. But they both have different date of birth.

Example

class Mother:

    def hair(self):
        return "Black"

    def eyes(self):
        return "Brown"

    def date_of_birth(self):
        return "21 January 1962"

class Daughter(Mother):

    def date_of_birth(self):
        return "19 May 1986"

daughter_object = Daughter()

print obj.eyes() # This will print "Brown"
print obj.hair() # This will print "Black"
print obj.date_of_birth() # This will print "19 May 1986"

mother_object = Daughter()
print obj.date_of_birth() # This will print "21 January 1962"

Its possible for a class to inherit from many classes. Lets consider a son inheriting his facial features from his mother and mental features from his father. But he cant inherit same features from both at a time.

Example

class Mother:

    def hair(self):
        return "Black"

    def eyes(self):
        return "Brown"

    def date_of_birth(self):
        return "5 April 1962"

class Father:

    def iq(self):
        return "Superior"

    def decision_making(self):
        return "Fast"

    def temper(self):
        return "Short Tempered"

    def date_of_birth(self):
        return "11 June 1954"

class Son(Father, Mother):

    def date_of_birth(self):
        return "6 May 1991"

son_object = Son()
print son_object.hair() # Will print "Black"
print son_object.iq() # Will print "Superior"
print son_object.date_of_birth() # Will print "6 May 1991"