Archive for May, 2009

Howto run a sub thread/process to monitor the system status in Python?

Posted on May 22, 2009. Filed under: Linux, Python |

Howto run a sub thread/process to monitor the system status in Python?

 1. popen  or popen2

 This will call another process by /bin/sh, so if you want get the pid, the result pid maybe the pid of /bin/sh

 Examples:

1.1   without log

>>>Import popen2, os

>>>Cur = popen2.Popen4(“vmstat –n 100”)

>>>Cur.pid

123

>>> 

 

ps aux|grep vmstat

root       123    0.0  0.1   1808   564 pts/9    S+   10:57   0:00 vmstat -n 100

 

1.2    With log

>>>import popen2, os

>>>cur = popen2.Popen4(“vmstat –n 100 > vmstat.log”)

>>>cur.pid

124

>>> 

 

ps aux|grep vmstat

root       124  0.1  0.2   4488   996 pts/9    S+   11:01   0:00 /bin/sh -c vmstat -n 100 > vmstat.log

root       125  0.0  0.1   1804   560 pts/9    S+   11:01   0:00 vmstat -n 100

 

>>>os.kill(cur.pid, 9)

>>>os.waitpid(cur.pid,0)

(124, 9)

>>> 

 

ps aux|grep vmstat

root       125  0.0  0.1   1804   560 pts/9    S+   11:01   0:00 vmstat -n 100

 

From the examples, you can see that, if you want the sub-thread record some log from the command, you’d better do not use this kind of popen2 lib, you can use following method

 

2. subprocess

 

2.1 Start the subprocess:

>>>Outlog = open(outputlogname, “w”)

>>>errlog=open(errlogname, “w”)

>>>try:

>>>    process=subprocess.Popen([“vmstat”,”-n”,”100”], stdout=outlog, stderr=errlog)

>>>except  OSError,e:

>>>    print “The OSError is:”,e

>>>    print “Maybe  this command is not exist”

>>>except Exception,e2:

>>>    print e

>>>pid = process.pid

134

 

ps aux|grep vmstat

root       134  0.0  0.1   1804   560 pts/9    S+   11:01   0:00 vmstat -n 100

 

You can see here: there is no “/bin/sh” process, the Popen have a parameter “shell=True/False”:

On Unix, with shell=False (default): In this case, the Popen class uses os.execvp() to execute the child program. args should normally be a sequence. A string will be treated as a sequence with the string as the only item (the program to execute).

On Unix, with shell=True: If args is a string, it specifies the command string to execute through the shell. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional shell arguments.

 

2.2 Kill the sub process

>>>os.kill(process.pid, 9)

134

>>>os.waitpid(process.pid, 0)

(134, 9)

>>>outlog.close()

>>>errlog.close()

 

 

 

Reference:

http://mail.python.org/pipermail/python-list/1999-May/002214.html

http://docs.python.org/library/subprocess.html

http://docs.python.org/library/popen2.html

http://mail.python.org/pipermail/python-list/2004-May/260937.html

http://code.activestate.com/recipes/496960/

http://www.daniweb.com/forums/thread36752.html#

http://groups.google.com/group/comp.lang.python/msg/9fa3a3c287e8e2a3?hl=en&

http://code.activestate.com/recipes/52296/

Read Full Post | Make a Comment ( None so far )

Python http upload script

Posted on May 15, 2009. Filed under: Python |

Python http upload script

 Httplib module and some useful methods

 Note   The httplib module has been renamed to http.client in Python 3.0. The 2to3 tool will automatically adapt imports when converting your sources to 3.0.

class httplib.HTTPConnection(host[, port[, strict[, timeout]]])

 HTTPConnection.request(method, url[, body[, headers]])

 HTTPConnection.getresponse()

Note   Note that you must have read the whole response before you can send a new request to the server

 HTTPConnection.set_debuglevel(level)

Set the debugging level (the amount of debugging output printed). The default debug level is 0, meaning no debugging output is printed.

HTTPConnection.connect()

Connect to the server specified when the object was created.

HTTPConnection.close()

Close the connection to the server.

As an alternative to using the request() method described above, you can also send your request step by step, by using the four functions below.

HTTPConnection.putrequest(request, selector[, skip_host[, skip_accept_encoding]])

This should be the first call after the connection to the server has been made. It sends a line to the server consisting of the request string, the selector string, and the HTTP version (HTTP/1.1). To disable automatic sending of Host: or Accept-Encoding: headers (for example to accept additional content encodings), specify skip_host or skip_accept_encoding with non-False values.

Changed in version 2.4: skip_accept_encoding argument added.

HTTPConnection.putheader(header, argument[, ])

Send an RFC 822-style header to the server. It sends a line to the server consisting of the header, a colon and a space, and the first argument. If more arguments are given, continuation lines are sent, each consisting of a tab and an argument.

HTTPConnection.endheaders()

Send a blank line to the server, signalling the end of the headers.

HTTPConnection.send(data)

Send data to the server. This should be used directly only after the endheaders() method has been called and before getresponse() is called.

 

 Example of upload file by PUT

 import httplib

conn = httplib.HTTPConnection(“192.168.1.1”)

conn.set_debuglevel(10)

conn.putrequest(‘PUT’,’http://192.168.1.1/cgi-bin/upload.php’)

conn.putheader(“Content-Length”, “32”)

conn.endheaders()

conn.send(“Hello world, I am uploading 32 bytes, if the length is less than 32 bytes, the script will halt here, if more than 32 bytes, the upload.php will only read the first 32 bytes”)

resps = conn.getresponse()

data = resps.read()

print “response is the webpage which upload.php shows after PUT upload:”, data

 

References:

httplib — HTTP protocol client

http://docs.python.org/library/httplib.html

 urllib — Open arbitrary resources by URL

http://www.python.org/doc/2.6/library/urllib.html

Http client to POST using multipart/form-data

http://code.activestate.com/recipes/146306/

Big File Upload

http://webpython.codepoint.net/cgi_big_file_upload

httplib HTTPConnection request problem

http://bytes.com/groups/python/30065-httplib-httpconnection-request-problem

http://mail.python.org/pipermail/python-list/2004-July/272313.html

PyCURL interface – Uploading large binary files

http://curl.haxx.se/mail/archive-2004-02/0043.html

http://pycurl.sourceforge.net/

urlgrabber

http://linux.duke.edu/projects/urlgrabber/

http://linux.duke.edu/projects/urlgrabber/comparison.ptml

HTTP Upload — An Overview

http://www.chilkatsoft.com/p/p_200.asp

HTTP Upload using a Proxy Server

http://www.example-code.com/python/upload_proxy_server.asp

Read Full Post | Make a Comment ( None so far )

Launch putty from browser

Posted on May 12, 2009. Filed under: Linux, Windows |

Launch PuTTY from Your Browser – Adding Procotols to Windows

For awhile now, I have wanted the ability to launch putty from a simple hyperlink.

PuTTY enables Windows users to access *nix machines through SSH. For many in the tech industry this tool is absolutely crucial. While it is fairly full featured, it is quite cumbersome for generic SSH use. One may say “Jon, all you need to do is run PuTTY, enter the host name you wish to connect to and provide a user name and password.” This process is certainly fine for a single SSH operation. This becomes quite a pain with multiple hosts at once.

What if there was a way to create a hyperlink which launches PuTTY? One would be able to launch PuTTY from email clients, browsers, spreadsheets, etc…
An example link would be: ssh://google.com

What do we need to do?

  1. Register a new protocol (in our case, let’s call it “ssh”) to Windows’s registry.
  2. Create a batch file that can take in strings such as “ssh://google.com” and massage them and pass them on to putty.exe
  3. Point the registry modification to that batch file.

One method of editing the registry is to create a .reg file, save it, right click it and select merge. The .reg file will then be merged with the registry. We will use this method. The key created can always be deleted by later from regedit.

Here is an example .reg file which points the protocol “ssh” to a batch file located at “C:/putty_util.bat” :

REGEDIT4
[HKEY_CLASSES_ROOT\ssh]
@="URL:ssh Protocol"
"URL Protocol"=""
[HKEY_CLASSES_ROOT\ssh\shell]
[HKEY_CLASSES_ROOT\ssh\shell\open]
[HKEY_CLASSES_ROOT\ssh\shell\open\command]
@="\"C:\\putty_util.bat\" %1"

Now save ssh.reg locally, right click it, and select merge.

Note: You can create any new protocol by changing each “ssh” in this .reg file to whatever abbreviation you wish to use for your protocol. You will likewise want to change the string @=”\”C:\\putty_util.bat\” %1″ to point at whatever application you wish to use. In this case, we are using the application C:\putty_util.bat and passing in the first argument we receive to it. In the example of ssh://google.com, we will be passing in the string “ssh://google.com”

Unfortunately, PuTTY does not understand arguments which start off with “ssh://” . As a result, we must use a batch file (C:\putty_util.bat) to remove the “ssh://” and any other oddities. Here is the batch file for my box:

@echo off
set var=%1
set extract=%var:~6,-1%
"C:\Program Files\PuTTY\putty.exe" %extract%

set extract=%var:~6,-1% is telling us to remove the first 6 characters and the last character from var and place that value in extract. The removal of the first 6 characters is the removal of “ssh://”. We remove the last character as there seems to be a trailing slash present in the input to our batch file.

With extract holding just the hostname, we now execute putty.exe and pass in extract as an argument. In this case, putty.exe is located at C:\Program Files\PuTTY\putty.exe. Please change this to the location of your putty.exe. Save this batch file to C:\putty_util.bat

Once this is done, anytime you have a link such as ssh://google.com you should be able to simply click it and launch PuTTY.

I will have a followup post with instructions on how to accomplish the same with ssh in other operating systems.

Sources:

http://kb.mozillazine.org/Register_protocol

External Links:

http://www.putty.org/

 

References:

http://jonmicklos.com/blog/?p=77

http://kb.mozillazine.org/Register_protocol

http://msdn.microsoft.com/en-us/library/aa767914(VS.85).aspx

http://www.chiark.greenend.org.uk/~sgtatham/putty/wishlist/url-launching.html

http://forums.whirlpool.net.au/forum-replies.cfm?t=326109

http://forums.cacti.net/about22862.html

Read Full Post | Make a Comment ( 7 so far )

Howto upload file by PHP

Posted on May 9, 2009. Filed under: PHP |

Howto upload file by PHP

 

  1. Upload by POST

http://us3.php.net/manual/en/features.file-upload.post-method.php

 

  1. Upload multi-files by POST

http://us3.php.net/manual/en/features.file-upload.multiple.php

 

  1. Upload by PUT

http://us3.php.net/manual/en/features.file-upload.put-method.php

 

  1. Other references

http://us3.php.net/manual/en/features.file-upload.php

http://us3.php.net/manual/en/function.move-uploaded-file.php

 

http://us3.php.net/manual/en/reserved.variables.php

http://us3.php.net/manual/en/reserved.variables.globals.php

http://us3.php.net/manual/en/reserved.variables.request.php

http://us3.php.net/manual/en/reserved.variables.server.php

 

http://us3.php.net/manual/en/reserved.variables.files.php

 

Use the function  print_r()  can print out the whole structure of the variable.

Read Full Post | Make a Comment ( None so far )

How To Use Linux epoll with Python

Posted on May 5, 2009. Filed under: Linux, Python | Tags: , , , , |

Contents

Introduction

As of version 2.6, Python includes an API for accessing the Linux epoll library. This article uses Python 3 examples to briefly demonstrate the API. Questions and feedback are welcome.

Blocking Socket Programming Examples

Example 1 is a simple Python 3.0 server that listens on port 8080 for an HTTP request message, prints it to the console, and sends an HTTP response message back to the client.

  • Line 9: Create the server socket.
  • Line 10: Permits the bind() in line 11 even if another program was recently listening on the same port. Otherwise this program could not run until a minute or two after the previous program using that port had finished.
  • Line 11: Bind the server socket to port 8080 of all available IPv4 addresses on this machine.
  • Line 12: Tell the server socket to start accepting incoming connections from clients.
  • Line 14: The program will stop here until a connection is received. When this happens, the server socket will create a new socket on this machine that is used to talk to the client. This new socket is represented by the clientconnection object returned from the accept() call. The address object indicates the IP address and port number at the other end of the connection.
  • Lines 15-17: Assemble the data being transmitted by the client until a complete HTTP request has been transmitted. The HTTP protocol is described at HTTP Made Easy.
  • Line 18: Print the request to the console, in order to verify correct operation.
  • Line 19: Send the response to the client.
  • Lines 20-22: Close the connection to the client as well as the listening server socket.

The official HOWTO has a more detailed description of socket programming with Python.

Example 1 (All examples use Python 3)

 1  import socket
 2
 3  EOL1 = b'\n\n'
 4  EOL2 = b'\n\r\n'
 5  response  = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
 6  response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
 7  response += b'Hello, world!'
 8
 9  serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
10  serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
11  serversocket.bind(('0.0.0.0', 8080))
12  serversocket.listen(1)
13
14  connectiontoclient, address = serversocket.accept()
15  request = b''
16  while EOL1 not in request and EOL2 not in request:
17     request += connectiontoclient.recv(1024)
18  print(request.decode())
19  connectiontoclient.send(response)
20  connectiontoclient.close()
21
22  serversocket.close()

Example 2 adds a loop in line 15 to repeatedly processes client connections until interrupted by the user (e.g. with a keyboard interrupt). This illustrates more clearly that the server socket is never used to exchange data with the client. Rather, it accepts a connection from a client, and then creates a new socket on the server machine that is used to communicate with the client.

The finally statement block in lines 23-24 ensures that the listening server socket is always closed, even if an exception occurs.

Example 2

 1  import socket
 2
 3  EOL1 = b'\n\n'
 4  EOL2 = b'\n\r\n'
 5  response  = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
 6  response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
 7  response += b'Hello, world!'
 8
 9  serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
10  serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
11  serversocket.bind(('0.0.0.0', 8080))
12  serversocket.listen(1)
13
14  try:
15     while True:
16        connectiontoclient, address = serversocket.accept()
17        request = b''
18        while EOL1 not in request and EOL2 not in request:
19            request += connectiontoclient.recv(1024)
20        print('-'*40 + '\n' + request.decode()[:-2])
21        connectiontoclient.send(response)
22        connectiontoclient.close()
23  finally:
24     serversocket.close()

Benefits of Asynchronous Sockets and Linux epoll

The sockets shown in Example 2 are called blocking sockets, because the Python program stops running until an event occurs. The accept() call in line 16 blocks until a connection has been received from a client. The recv() call in line 19 blocks until data has been received from the client (or until there is no more data to receive). The send() call in line 21 blocks until all of the data being returned to the client has been queued by Linux in preparation for transmission.

When a program uses blocking sockets it often uses one thread (or even a dedicated process) to carry out the communication on each of those sockets. The main program thread will contain the listening server socket which accepts incoming connections from clients. It will accept these connections one at a time, passing the newly created socket off to a separate thread which will then interact with the client. Because each of these threads only communicates with one client, it is ok if it is blocked from proceeding at certain points. This blockage does not prohibit any of the other threads from carrying out their respective tasks.

The use of blocking sockets with multiple threads results in straightforward code, but comes with a number of drawbacks. It can be difficult to ensure the threads cooperate appropriately when sharing resources. And this style of programming can be less efficient on computers with only one CPU.

The C10K Problem discusses some of the alternatives for handling multiple concurrent sockets. One is the use of asynchronous sockets. These sockets don’t block until some event occurs. Instead, the program performs an action on an asynchronous socket and is immediately notified as to whether that action succeeded or failed. This information allows the program to decide how to proceed. Since asynchronous sockets are non-blocking, there is no need for multiple threads of execution. All work may be done in a single thread. This single-threaded approach comes with its own challenges, but can be a good choice for many programs. It can also be combined with the multi-threaded approach: asynchronous sockets using a single thread can be used for the networking component of a server, and threads can be used to access other blocking resources, e.g. databases.

Linux 2.6 has a number of mechanisms for managing asynchronous sockets, three of which are exposed by the Python API’s select, poll and epoll.  epoll and poll are better than select because the Python program does not have to inspect each socket for events of interest. Instead it can rely on the operating system to tell it which sockets may have these events. And epoll is better than poll because it does not require the operating system to inspect all sockets for events of interest each time it is queried by the Python program. Rather Linux tracks these events as they occur, and returns a list when queried by Python. So epoll is a more efficient and scalable mechanism for large numbers (thousands) of concurrent socket connections, as shown in these graphs.

Asynchronous Socket Programming Examples with epoll

Programs using epoll often perform actions in this sequence:

  1. Create an epoll object
  2. Tell the epoll object to monitor specific events on specific sockets
  3. Ask the epoll object which sockets may have had the specified event since the last query
  4. Perform some action on those sockets
  5. Tell the epoll object to modify the list of sockets and/or events to monitor
  6. Repeat steps 3 through 5 until finished
  7. Destroy the epoll object

Example 3 duplicates the functionality of Example 2 while using asynchronous sockets. The program is more complex because a single thread is interleaving the communication with multiple clients.

  • Line 1: The select module contains the epoll functionality.
  • Line 13: Since sockets are blocking by default, this is necessary to use non-blocking (asynchronous) mode.
  • Line 15: Create an epoll object.
  • Line 16: Register interest in read events on the server socket. A read event will occur any time the server socket accepts a socket connection.
  • Line 19: The connection dictionary maps file descriptors (integers) to their corresponding network connection objects.
  • Line 21: Query the epoll object to find out if any events of interest may have occurred. The parameter “1” signifies that we are willing to wait up to one second for such an event to occur. If any events of interest occurred prior to this query, the query will return immediately with a list of those events.
  • Line 22: Events are returned as a sequence of (fileno, event code) tuples. fileno is a synonym for file descriptor and is always an integer.
  • Line 23: If a read event occurred on the socket server, then a new socket connection may have been created.
  • Line 25: Set new socket to non-blocking mode.
  • Line 26: Register interest in read (EPOLLIN) events for the new socket.
  • Line 31: If a read event occurred then read new data sent from the client.
  • Line 33: Once the complete request has been received, then unregister interest in read events and register interest in write (EPOLLOUT) events. Write events will occur when it is possible to send response data back to the client.
  • Line 34: Print the complete request, demonstrating that although communication with clients is interleaved this data can be assembled and processed as a whole message.
  • Line 35: If a write event occurred on a client socket, it’s able to accept new data to send to the client.
  • Lines 36-38: Send the response data a bit at a time until the complete response has been delivered to the operating system for transmission.
  • Line 39: Once the complete response has been sent, disable interest in further read or write events.
  • Line 40: A socket shutdown is optional if a connection is closed explicitly. This example program uses it in order to cause the client to shutdown first. The shutdown call informs the client socket that no more data should be sent or received and will cause a well-behaved client to close the socket connection from it’s end.
  • Line 41: The HUP (hang-up) event indicates that the client socket has been disconnected (i.e. closed), so this end is closed as well. There is no need to register interest in HUP events. They are always indicated on sockets that are registered with the epoll object.
  • Line 42: Unregister interest in this socket connection.
  • Line 43: Close the socket connection.
  • Lines 18-45: The try-catch block is included because the example program will most likely be interrupted by a KeyboardInterrupt exception
  • Lines 46-48: Open socket connections don’t need to be closed since Python will close them when the program terminates. They’re included as a matter of good form.
Example 3

 1  import socket, select
 2
 3  EOL1 = b'\n\n'
 4  EOL2 = b'\n\r\n'
 5  response  = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
 6  response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
 7  response += b'Hello, world!'
 8
 9  serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
10  serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
11  serversocket.bind(('0.0.0.0', 8080))
12  serversocket.listen(1)
13  serversocket.setblocking(0)
14
15  epoll = select.epoll()
16  epoll.register(serversocket.fileno(), select.EPOLLIN)
17
18  try:
19     connections = {}; requests = {}; responses = {}
20     while True:
21        events = epoll.poll(1)
22        for fileno, event in events:
23           if fileno == serversocket.fileno():
24              connection, address = serversocket.accept()
25              connection.setblocking(0)
26              epoll.register(connection.fileno(), select.EPOLLIN)
27              connections[connection.fileno()] = connection
28              requests[connection.fileno()] = b''
29              responses[connection.fileno()] = response
30           elif event & select.EPOLLIN:
31              requests[fileno] += connections[fileno].recv(1024)
32              if EOL1 in requests[fileno] or EOL2 in requests[fileno]:
33                 epoll.modify(fileno, select.EPOLLOUT)
34                 print('-'*40 + '\n' + requests[fileno].decode()[:-2])
35           elif event & select.EPOLLOUT:
36              byteswritten = connections[fileno].send(responses[fileno])
37              responses[fileno] = responses[fileno][byteswritten:]
38              if len(responses[fileno]) == 0:
39                 epoll.modify(fileno, 0)
40                 connections[fileno].shutdown(socket.SHUT_RDWR)
41           elif event & select.EPOLLHUP:
42              epoll.unregister(fileno)
43              connections[fileno].close()
44              del connections[fileno]
45  finally:
46     epoll.unregister(serversocket.fileno())
47     epoll.close()
48     serversocket.close()

epoll has two modes of operation, called edge-triggered and level-triggered. In the edge-triggered mode of operation a call to epoll.poll() will return an event on a socket only once after the read or write event occurred on that socket. The calling program must process all of the data associated with that event without further notifications on subsequent calls to epoll.poll(). When the data from a particular event is exhausted, additional attempts to operate on the socket will cause an exception. Conversely, in the level-triggered mode of operation, repeated calls to epoll.poll() will result in repeated notifications of the event of interest, until all data associated with that event has been processed. No exceptions normally occur in level-triggered mode.

For example, suppose a server socket has been registered with an epoll object for read events. In edge-triggered mode the program would need to accept() new socket connections until a socket.error exception occurs. Whereas in the level-triggered mode of operation a single accept() call can be made and then the epoll object can be queried again for new events on the server socket indicating that additional calls to accept() should be made.

Example 3 used level-triggered mode, which is the default mode of operation. Example 4 demonstrates how to use edge-triggered mode. In Example 4, lines 25, 36 and 45 introduce loops that run until an exception occurs (or all data is otherwise known to be handled). Lines 32, 38 and 48 catch the expected socket exceptions. Finally, lines 16, 28, 41 and 51 add the EPOLLET mask which is used to set edge-triggered mode.

Example 4

 1  import socket, select
 2
 3  EOL1 = b'\n\n'
 4  EOL2 = b'\n\r\n'
 5  response  = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
 6  response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
 7  response += b'Hello, world!'
 8
 9  serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
10  serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
11  serversocket.bind(('0.0.0.0', 8080))
12  serversocket.listen(1)
13  serversocket.setblocking(0)
14
15  epoll = select.epoll()
16  epoll.register(serversocket.fileno(), select.EPOLLIN | select.EPOLLET)
17
18  try:
19     connections = {}; requests = {}; responses = {}
20     while True:
21        events = epoll.poll(1)
22        for fileno, event in events:
23           if fileno == serversocket.fileno():
24              try:
25                 while True:
26                    connection, address = serversocket.accept()
27                    connection.setblocking(0)
28                    epoll.register(connection.fileno(), select.EPOLLIN | select.EPOLLET)
29                    connections[connection.fileno()] = connection
30                    requests[connection.fileno()] = b''
31                    responses[connection.fileno()] = response
32              except socket.error:
33                 pass
34           elif event & select.EPOLLIN:
35              try:
36                 while True:
37                    requests[fileno] += connections[fileno].recv(1024)
38              except socket.error:
39                 pass
40              if EOL1 in requests[fileno] or EOL2 in requests[fileno]:
41                 epoll.modify(fileno, select.EPOLLOUT | select.EPOLLET)
42                 print('-'*40 + '\n' + requests[fileno].decode()[:-2])
43           elif event & select.EPOLLOUT:
44              try:
45                 while len(responses[fileno]) > 0:
46                    byteswritten = connections[fileno].send(responses[fileno])
47                    responses[fileno] = responses[fileno][byteswritten:]
48              except socket.error:
49                 pass
50              if len(responses[fileno]) == 0:
51                 epoll.modify(fileno, select.EPOLLET)
52                 connections[fileno].shutdown(socket.SHUT_RDWR)
53           elif event & select.EPOLLHUP:
54              epoll.unregister(fileno)
55              connections[fileno].close()
56              del connections[fileno]
57  finally:
58     epoll.unregister(serversocket.fileno())
59     epoll.close()
60     serversocket.close()

Since they’re similar, level-triggered mode is often used when porting an application that was using the select or poll mechanisms, while edge-triggered mode may be used when the programmer doesn’t need or want as much assistance from the operating system in managing event state.

In addition to these two modes of operation, sockets may also be registered with the epoll object using the EPOLLONESHOT event mask. When this option is used, the registered event is only valid for one call to epoll.poll(), after which time it is automatically removed from the list of registered sockets being monitored.

Performance Considerations

Listen Backlog Queue Size

In Examples 1-4, line 12 has shown a call to the serversocket.listen() method. The parameter for this method is the listen backlog queue size. It tells the operating system how many TCP/IP connections to accept and place on the backlog queue before they are accepted by the Python program. Each time the Python program calls accept() on the server socket, one of the connections is removed from the queue and that slot can be used for another incoming connection. If the queue is full, new incoming connections are silently ignored causing unnecessary delays on the client side of the network connection. A production server usually handles tens or hundreds of simultaneous connections, so a value of 1 will usually be inadequate. For example, when using ab to perform load testing against these sample programs with 100 concurrent HTTP 1.0 clients, any backlog value less than 50 would often produce performance degradation.

TCP Options

The TCP_CORK option can be used to “bottle up” messages until they are ready to send. This option, illustrated in lines 34 and 40 of Examples 5, might be a good option to use for an HTTP server using HTTP/1.1 pipelining.

Example 5

 1  import socket, select
 2
 3  EOL1 = b'\n\n'
 4  EOL2 = b'\n\r\n'
 5  response  = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
 6  response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
 7  response += b'Hello, world!'
 8
 9  serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
10  serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
11  serversocket.bind(('0.0.0.0', 8080))
12  serversocket.listen(1)
13  serversocket.setblocking(0)
14
15  epoll = select.epoll()
16  epoll.register(serversocket.fileno(), select.EPOLLIN)
17
18 try:
19     connections = {}; requests = {}; responses = {}
20     while True:
21        events = epoll.poll(1)
22        for fileno, event in events:
23           if fileno == serversocket.fileno():
24              connection, address = serversocket.accept()
25              connection.setblocking(0)
26              epoll.register(connection.fileno(), select.EPOLLIN)
27              connections[connection.fileno()] = connection
28              requests[connection.fileno()] = b''
29              responses[connection.fileno()] = response
30           elif event & select.EPOLLIN:
31              requests[fileno] += connections[fileno].recv(1024)
32              if EOL1 in requests[fileno] or EOL2 in requests[fileno]:
33                 epoll.modify(fileno, select.EPOLLOUT)
34                 connections[fileno].setsockopt(socket.IPPROTO_TCP, socket.TCP_CORK, 1)
35                 print('-'*40 + '\n' + requests[fileno].decode()[:-2])
36           elif event & select.EPOLLOUT:
37              byteswritten = connections[fileno].send(responses[fileno])
38              responses[fileno] = responses[fileno][byteswritten:]
39              if len(responses[fileno]) == 0:
40                 connections[fileno].setsockopt(socket.IPPROTO_TCP, socket.TCP_CORK, 0)
41                 epoll.modify(fileno, 0)
42                 connections[fileno].shutdown(socket.SHUT_RDWR)
43           elif event & select.EPOLLHUP:
44              epoll.unregister(fileno)
45              connections[fileno].close()
46              del connections[fileno]
47  finally:
48     epoll.unregister(serversocket.fileno())
49     epoll.close()
50     serversocket.close()

On the other hand, the TCP_NODELAY option can be used to tell the operating system that any data passed to socket.send() should immediately be sent to the client without being buffered by the operating system. This option, illustrated in line 14 of Example 6, might be a good option to use for an SSH client or other “real-time” application.

Example 6

 1  import socket, select
 2
 3  EOL1 = b'\n\n'
 4  EOL2 = b'\n\r\n'
 5  response  = b'HTTP/1.0 200 OK\r\nDate: Mon, 1 Jan 1996 01:01:01 GMT\r\n'
 6  response += b'Content-Type: text/plain\r\nContent-Length: 13\r\n\r\n'
 7  response += b'Hello, world!'
 8
 9  serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
10  serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
11  serversocket.bind(('0.0.0.0', 8080))
12  serversocket.listen(1)
13  serversocket.setblocking(0)
14  serversocket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
15
16  epoll = select.epoll()
17  epoll.register(serversocket.fileno(), select.EPOLLIN)
18
19 try:
20     connections = {}; requests = {}; responses = {}
21     while True:
22        events = epoll.poll(1)
23        for fileno, event in events:
24           if fileno == serversocket.fileno():
25              connection, address = serversocket.accept()
26              connection.setblocking(0)
27              epoll.register(connection.fileno(), select.EPOLLIN)
28              connections[connection.fileno()] = connection
29              requests[connection.fileno()] = b''
30              responses[connection.fileno()] = response
31           elif event & select.EPOLLIN:
32              requests[fileno] += connections[fileno].recv(1024)
33              if EOL1 in requests[fileno] or EOL2 in requests[fileno]:
34                 epoll.modify(fileno, select.EPOLLOUT)
35                 print('-'*40 + '\n' + requests[fileno].decode()[:-2])
36           elif event & select.EPOLLOUT:
37              byteswritten = connections[fileno].send(responses[fileno])
38              responses[fileno] = responses[fileno][byteswritten:]
39              if len(responses[fileno]) == 0:
40                 epoll.modify(fileno, 0)
41                 connections[fileno].shutdown(socket.SHUT_RDWR)
42           elif event & select.EPOLLHUP:
43              epoll.unregister(fileno)
44              connections[fileno].close()
45              del connections[fileno]
46  finally:
47     epoll.unregister(serversocket.fileno())
48     epoll.close()
49     serversocket.close()

Source Code

The examples on this page are in the public domain and available for download.

References:

http://scotdoyle.com/python-epoll-howto.html

http://mail.python.org/pipermail/python-dev/2003-August/037614.html

http://ionelmc.wordpress.com/2008/04/01/curious-difference-between-epoll-poll-kqueue-and-select/

http://ilab.cs.byu.edu/python/select/echoserver.html

http://docs.python.org/library/select.html

Read Full Post | Make a Comment ( None so far )

Liked it here?
Why not try sites on the blogroll...