Posted on May 7, 2008 by Mikko Ohtamaa
Filed Under python
I wrote this little script as a mental exercise and to prove the power of Python programming language. If anyone accepts the challenge, I’d like to see submissions in other programming langauges 
For the foreigners: this is the best comic in Finland, so I hope you’ll get translations soon! It tells about the relationship of a woman and a pig (sic) reflecting the deepest shadows of Finnish social life.
"""
Creats local mirror from Viivi & Wagner strips by fetching all of them from hs.fi.
Will create downloaded strips as
2004/1.1.2004.gif
2004/2.1.2004.gif
...
until today
Try this in C++!
Motivation: No one has build Viivi & Wagner search engine with speech bubble OCR support
and I desperately wanted to find "Kottarainen lentaa korvaan" strip for my gf.
Time to complete: 20 min.
"""
__docformat__ = "epytext"
__author__ = "Mikko Ohtamaa"
__license__ = "BSD"
__copyright__ = "2008 Mikko Ohtamaa"
import os
import re
import urllib
from BeautifulSoup import BeautifulSoup
# 1.1.2004 start page
url = "http://www.hs.fi/viivijawagner/1073386660690"
# Loop until there is no longer next link
while True:
stream = urllib.urlopen(url)
html = stream.read()
stream.close()
soup = BeautifulSoup(html)
# Parse strip date from contents
date = None
# Find strip date, which is next to a title
h1 = soup.findAll(text="Viivi ja Wagner")
# Should be present always
date = h1[0].parent.parent.p.string
print “Fetching ” + date
# Scrape strip
strip = soup.findAll(”div” , { “class” : “strip” })
img = strip[0].img
stream = urllib.urlopen(img["src"])
data = stream.read()
stream.close()
# For each year, give a new folder to avoid file system stress
# (lotsa files in a folder kill poor Gnome)
day, month, year = date.split(”.”)
folder = year
if not os.path.exists(folder):
os.mkdir(folder)
# Store contents
fname = os.path.join(folder, date + “.gif”)
f = open(fname, “wb”)
f.write(data)
f.close()
# Find next url, it is a containing one img tag
img = soup.findAll(alt=”seuraava”)
if len(img) == 0:
break
a = img[0].parent
url = a["href"]
See preview
Share This
Leave a Comment
Posted on April 19, 2008 by Mikko Ohtamaa
Filed Under pys60, python, series 60, symbian
A common question for Python for Series 60 newcomers is how to build standalone Symbian applications from Python source code. We have been using Makefile based toolchain internally. I describe it in this picture, I didn’t bother to add thumbnail for the image, since it’s a 3400 pixels wide diagram.
The diagram describes building a PyS60 application with some Python extensions (Symbian native C++) mixed in and bundling it all to one downloadable SIS file. The application will appear as any first class S60 application in the menu and the user does not know it’s running Python internally, besides bad installation experience (it challenges Microsoft installers with all those unnecessary yes/no questions), extra uninstaller entries and slow start-up time.
The biggest problems are caused by embedded SISs (SIS inside other SIS files) which are not treaded very wel by several Symbian parties. In theory, it could be build one monolithic SIS, but you’d need to recompile PyS60 from scratch and patch UIDs inside it for your own UIDs received from symbiansigned.com. We are planning to explore SCons based build solution to address this problem, since Makefiles are a bit unflexible with tasks like PKG file and UID range generation.
Here is a PKG file example for final user distributable SIS file.
Also, see UIKludges project for additional details for PKG files of Python extensions.
You need to have
- Ensymble tool
- Series 60 SDK (contains some old GNU make)
You need to master
- A build tool (make)
- Symbian PKG file structure
- Lots of different command line tools
Pros
- It’s the best one we have for now
Cons
- Symbian signing and certification companies don’t understand embedded SIS files (all SIS files must be signed prior embedding) and may have hard time signing SIS files containing only an extension DLL for Pyton. Symbian Signed test criteria has been built only UI application based SIS files in the mind.
- You cannot cook your own patched PyS60 distribution without revamping some hardcoded UIDs and paths, since otherwise there are UID conflicts (EXE and DLL file UIDs are in Nokia’s protected range)
- S60 installers askes extra confirmation for every embedded SIS file, even in the middle of the progress bar, so the user experience of installation is screwed up
- There will extra uninstallation entry for every embedded SIS file in S60 application manager confusing the user
- As you can see, most cons come from Symbian and Symbian signing limitations and have nothing to do with Python
Ps. I would have put this thing to wiki.opensource.nokia.com, but their webmaster email address is non-functional and one cannot upload images to their Wiki.Share This
3 Comments
Posted on April 18, 2008 by Mikko Ohtamaa
Filed Under python, zope
I want to use Zope 3 interface package to write component architecture i.e. have a plug-ins easily in Python. Zope 3 interfaces are very handy and, which cannot be conducted from the name, are available outside Zope too. From my prior experiences I know that Zope 3 interfaces package is one of the best and most underrated Python packages out there. It even influenced to the new design of Python 3k.
Well then… I haven’t used Zope 3 interfaces standalone before, so the first thing what I do is writing “zope 3 interfaces” into my Google search this.
This page comes up.
It’s horrible - the very reason I write this quick blog entry. Some notes below (I have written things from the point of external visitor - I have hands deep in Zope myself, so you don’t need to clarify these things for me or teach anything)
- The information is tangled mess: please use subtitles and images
- You could mention that the package can be used outside Zope
- You could mention tat the package is a generic Python package
- The list of CamelCaseWordsWhichGoesOnAndOnUnexplained made me puke. Please use proper English and explain the meaning of the links.
- It has a comment box: I tried to comment the page but I will get permission error. Please do not show the comment box if the commenting is not actually possbile.
- Then I tried to register to Zope.org to comment or fix the basic things on the page. You have a join link in zope.org, but there is no registration form. Manual email conversation seem to be prerequisite for the registration. Is Zope a secret society or something…? The contribution barrier just rise too high.
- In this point, I give up. Even if you had the best interface package out there, I don’t care anymore. Looks like getting involved to fixing thing takes too much of my time. You had your 10 minutes of time to impress me and you failed. Not only that, but I got so frustrated that I want to learn smoking, fast.
In the post “No, you are not smart enough for Zope” Martjin Faassen highlights some problems of Zope community. “It’s hard to get good content written” Martjin claims. I disagree. Whoever created the page originally could have thought what people coming to the page want. They don’t want to decrypt the brain core dump of hardcore Zope developer. They want to know what is this thing, how this thing is beneficial for them, how do I get started with it and how do I use it.
You all know how Internet works. You all have visit on web pages. You all are customers for the same thing you also produce. So writing a basic web page is not something you couldn’t do.
Hints:
Pardon me the tone of this post. Zope is the 23th best thing out there, but the Zope community has stagnated badly in some aspects. Some things were acceptable ten years ago when web was still young and Python developers hardcore, but if you don’t keep with the pace you lose all the mindshare.Share This
5 Comments
Posted on March 27, 2008 by Mikko Ohtamaa
Filed Under Uncategorized, javascript
With Web 2.0, Javascript has gained a great foothold in the web development. Javascript, which was originally indented to be few lines of interactivity here and there, is now used for full-blown UI frameworks like Ext JS and Dojo. Because its toy background, Javascript is not the easiest language to debug. Also, the lovely phenomenon known as browser wars has ensured that professional Javascript development is PITA due to browser incompatibilities - It seconds the Symbian C++ embedded programming if you know what I mean.
Javascript is executed run-time. Thus, any errors cannot be cached until the code is run. JSLint is a verifier tool which checks Javascript syntax offline. Of course, it cannot detect application logic errors, but it makes sure that you don’t have cross-browser compability problems in your Javascript syntax. Maybe the most famous of these problems is the extra comma before the list termination which is ok for Firefox, but kills the page on Internet Explorer.
Some argue that this does not really catch bugs and instead a comprehensive unit-test suite should be used. This might be a requirement for a platform level Javascript library code, but it is often an overkill for your little site with some flashy dialog windows.
Installing JSLint
JSLint itself is written in Javascript. So you need to offline Javascript interpreter to run it. On Linux there exists Rhino. On Ubuntu, Rhino package is broken. and you need to download the orignal Rhino JAR from here and copy it to /usr/share/java.
sudo apt-get install rhino # Note https://bugs.launchpad.net/ubuntu/+source/rhino/+bug/93885
Then get JSLint offline version from here.
Running JSLint straight from the Eclipse
We use JSEclipse plug-in in Eclipse for Javascript development. Thus, it is convenient to to execute JSLint directly from the Eclipse IDE.
Drag and drop jslint.js to your project/scripts folder in Eclipse.
Create a new External tool like in the example below.
Then you can run it by just clicking “run external tool” icon. Output comes to the console window.
Checking all Javascript files during unit test
verify_javascript.bash can be executed from shell and it checks all JS files in your project. This is a nice feature to integrate to your unit test cycle.
Example:
$ bash scripts/verify_javascript.bash source/server/media/jssource/server/media/js/jquery.js
Lint at line 11 character 30: Expected '{' and instead saw 'var'.
...
Share This
1 Comment
Posted on March 14, 2008 by saarni
Filed Under Adobe AIR, Flex 3, sqlite
Learning new development environments and languages can be quite fun, but sometimes it turns out just plain frustrating. Especially so if you are trying to learn something that is still in its beta stage. This time it was Adobe AIR and Flex 3.
The program in question required storing date and time values in an sqlite3 database. Now, as the title of this entry says, I should’ve just read the AIR sqlite handling manual 1 more time (or 2-10 more times) and I could’ve learnt that AIR SQL handling accepts “Date” as column type and handles inserting and getting Date objects from the database transparently. I did not notice this while studying and used “TIMESTAMP” as column type instead. This is where things went wrong… so wrong…
Using “TIMESTAMP” column type and inserting datetime strings worked flawlessly in the betas along with using sqlite date functions. But AIR 1.0 release came and suddenly my application started producing the following SQL errors everywhere:
SQLError: 'Error #3115: SQL Error.', details:'could not convert text value to numeric value.', operation:'execute'
Naturally for a few days I went through my SQL code and tables trying different things on columns that had types like “INTEGER”, “REAL”, “NUMERIC” and so on. After trying and trying I finally decided to read the AIR documentation again. The essential part turned out to be how AIR handles the column affinities in the release version: everything not recognized by AIR is given “NUMERIC” column affinity. This and the fact that AIR is now stricter about types is what changed between the betas and release version and broke the columns with “TIMESTAMP” as type since I was trying to insert text.
So, again, I should’ve just rtfm…Share This
1 Comment
Posted on March 14, 2008 by Mikko Ohtamaa
Filed Under signing, symbian
Edit: Made correction to iPhone SDK price/added self-signed option
I have been recently involved in Symbian development. Unfortunately, I could say. Symbian has made the life of Symbian developers headacheful. The main problem is that the application signing is made unnecessary cumbersome. Getting a required developer certificate to just to run your one liner test code is a long road paved with curses, stress and ponder how a world class company can screw up things so badly.
I write this post to vent out the frustration. Hopefully someone picks up the feedback presented here. Personally, I don’t care anymore really. After explaining to the customer once “I am afraid that we cannot have the demo in this meeting, since we were unable to get the developer certificates in time. It was nice you did summon all of your executes, though.” my sympathy towards Symbian has fallen to zero.
In the long run, I believe, this will hurt Symbian when developers leave the platform. When this happens Symbian will miss 1) developer base 2) the wow value of all new cool things when hackers move to happier platforms - after all you don’t want to work with things causing a headache even if you are paid generously. Professional Symbian capable developers are already rare and scaring away the potential candidates (students, open source developers, freeware developers) won’t really help the matter. Of course, for us, who are already in the business, it’s good since we can raise our hourly fee and die as rich dinosaurs (remember COBOL).
Looks like the root problem is the cocky attitude: “We are the biggest mobile operating system in the world. You are not our customer, the biggest mobile phone brands are. Thus, you are an insignicant fly and we don’t need to listen to you. Please go away.” The big boys tell that this very same reason lead to Motorola’s fall from the number one mobile phone vendor position - so let’s see if the history is repeating itself.
The problems of Symbian development process, especially signing, are well known. Symbian Signed forum category has 120 000 views. Symbian signed support requests has 57 000 views and 2000 messages, since November 2007. Over a thousand support requests in four months. Symbian has money. It could fix the problem if it has will. If there is will in the organization and the work just does not progress, the corporate world has a simple solution: fire the management and hire capable responsible persons.
I won’t go the details of the problems in this post. I have prepared a post containing over 50 bulletins with detailed steps to get the developer certification process done and it lists every little mistake in Symbian Signed process. Unsurprisingly, the post mentions fuck and hell over twenty times, so I am reluctant to post it on our company site. So I sketched a manager comprehensible overview of the state of the process in this post. To justify my criticism, I compare Symbian to Apple iPhone and make some suggestion how to fix things.
Certificate costs
Symbian: 200$/1.5 years (you need a publisher id to get a developer certificate) + 20$ per signing round
Update: You can still make self-signing certificates yourself with limited capabilities (no GPS), but symbiansigned.com doesn’t bother to mention it.
Apple: 99$/year (for testing and distribution applications)
Suggestion: Make the developer certificate free. It used to be free, but based on this post, Symbian signed claims that they do not have enough CPU power, because of all kind of nasties are knocking their server. Come on! it’s Internet. You are hosted by Cidercone and by knowing how much they charge for the hosting I believe you can get every possible server power in the world to generate your keys.
This development start up cost automatically excludes third world people, freeware makers and students i.e. the future Symbian professionals. It’s hard to maintain a platform if there is not enough skilled people to work with it. There won’t be meat into Nokia’s and Sony-Ericsson’s innovation machines.
Development start lead time
How long it takes to get “Hello World” code to the phone.
Symbian: 8 workdays
Apple: 6 hours
Complete Guide to Symbian Signed guide (manual) is 20 pages PDF. How often you need a user manual to use a web site? Even with this page amount, the manual lacks crucial pieces of information which you need to hunt and ask help from the Symbian Signed support forums. I had to make 3 support request for Symbian Signed and 1 for TrustCenter.
Set-up process
Symbian: Create 3 user accounts (symbiansigned.com, developers.symbian.com, TrustCenter). Fax your passport and some signed papers to TrustCenter Germany. Wait when your request is manually processed. Ask TrustCenter what I do with the link they send me (hint: the only working web browser for their site is the very same browser you orignally submitted your application from). Convert TrustCenter key to Symbian specific format using obscure BAT tool. Use half-baked Windows EXE to generate files. Ask at the support forums permission to be allowed to create the Publisher id.
Apple: Download SDK. Click “I Agree.”
Symbiansigned management team must be either incomplete or inexistent, or they just hate their users. Looks like no one didn’t bother model or test this process, since the flaws are so apparent. Why do we need three user accounts? Why do we need a user account at all, isn’t my credit card payment enough? Why there can’t be a single web page on symbiansigned.com where I could do these all things without? Also, Symbian could do these things without third parties (TrustCenter) who are badly integrated to the process.
Web site and information reachability
Symbian: Two sites with different user accounts. Information is spread around and hidden behind various for registred users only pages. Often Symbian PDFs (yes they seem to prefer zipped PDFs) and web pages cross-link to other sites and these links are broken. Their internal site search engine does not give meaningful results. Google cannot search pages, since they are hidden behind the registration. Web forms don’t work. Navigation paths are unclear and you often need to guess where a certain piece of information could be.
Apple: One site - and it just works.
Why Symbian hides a crucial BAT tool in the page labyrinth of developers.symbian.com and behind login? Does the 100 lines of BAT code contains a very hidden trade secret? I doubt. (I also doubt that a BAT tool is a good way to solve a problem which could be fixed with a web form or Javascript, but that’s another matter.)
I think this is the part of the attidude problem. SymbianSigned.com mission is not to “secure” the development and control the developers of the platform. It’s mission should be easily enable developers to start development and release their software for Symbian. The meaning of “easily” includes that the web site is working and up to the standards. Please hire a usability expert (we are here) and get it fixed. This problem is not solvable by writing yet-another user manual PDF.
Giving feedback
Symbian: Support like goes to the Symbian developer forum. A bulleting board with unorganized, angry, posts. They don’t give a feedback form to report the problems on Symbian Signed.
Apple: “There are three great ways to get answers to your questions about ADC membership, products, and services: visit our Frequently Asked Questions web pages, submit your questions or feedback using the form below, or call us to speak directly to a worldwide support agent. We look forward to assisting you.”
I’d help symbiansigned.com to get its site better if I could. When there is a broken link, I can hit feedback and ask them to fix it. But looks like they want to be ignorant about the matter - if I report a problem on their forum I bet no one will ever fix it. There won’t be even “thank you.” Probably no one ever reads my post. On the other hand, when having such non-functional service, this feedback channel would be flooded in overnight.
Conclusion
Symbian has unforgiveable screwed up with its signing process and doesn’t seem to get the feedback from the developers who suffer from it. Steve Ballmer’s famous quote “developers, developers, developers” refer to what’s the important factor making the platform succesfully in long term. This is especially crude when you know that things used to be good.Share This
9 Comments
Posted on March 7, 2008 by Mikko Ohtamaa
Filed Under iphone, python
I just read waffle’s blog entry about iPhone SDK release. Looks like Objective C is the only supported language by default (I am just downloading SDK).The comments speculated that embedding Python is not possible due to size constraints. Bollocks I say =) Python for Series 60 phones is 500 kb download without trimming. It’s less than the size of HTML page you are viewing now - RAM footprint is even smaller) If Series 60 phones, which have much more modest hardware specifications, can run Python it shouldn’t be a problem for iPhones either.
Why Apple didn’t add additional language support by default? Well they seem to have their hands full to get SDK out at all (delays) so we shouldn’t expect to have perfect set in 1.0 release.
Now, who wants start a porting project with me? Share This
2 Comments
Posted on March 7, 2008 by Mikko Ohtamaa
Filed Under django, python
I run Django in a standalone long-running application (video encoding server). It leaked memory severely. By using htop, one was seeing two gigabytes reserved for /usr/bin/python after a while. Before starting the debugging session, I had no faintest idea what could be the cause of the problem. Django is robust technology - this kind of things haven’t happened for me before. Since I was running Django in standalone mode, I suspected that some query cache does not get cleared. But random poking around the source code didn’t give any clues.
It was time to do some serious memory debugging for Python.
Python as is doesn’t leak memory, since it’s garbage collected virtual machine. All “leaks” are design problems in the application logic.I found a good primer here what’s going inside Python’s memory management.
First I tried this nice TrackRefs class from Zope. It relies on Python’s own in-interpreter functions to monitor objects.
class TrackRefs:
"""Object to track reference counts across test runs."""
def __init__(self, limit=40):
self.type2count = {}
self.type2all = {}
self.limit = limit
def update(self):
obs = sys.getobjects(0)
type2count = {}
type2all = {}
for o in obs:
all = sys.getrefcount(o)
if type(o) is str and o == '<dummy key>':
# avoid dictionary madness
continue
t = type(o)
if t in type2count:
type2count[t] += 1
type2all[t] += all
else:
type2count[t] = 1
type2all[t] = all
ct = [(type2count[t] - self.type2count.get(t, 0),
type2all[t] - self.type2all.get(t, 0),
t)
for t in type2count.iterkeys()]
ct.sort()
ct.reverse()
printed = False
logger.debug(”———————-”)
logger.debug(”Memory profiling”)
i = 0
for delta1, delta2, t in ct:
if delta1 or delta2:
if not printed:
logger.debug(”%-55s %8s %8s” % (”, ‘insts’, ‘refs’))
printed = True
logger.debug(”%-55s %8d %8d” % (t, delta1, delta2))
i += 1
if i >= self.limit:
break
self.type2count = type2count
self.type2all = type2all
You need to have Python compiled in debug mode to have sys.getobjects() method. Luckily this beefed up Python binary is availalble from Ubuntu’s stock repository:
sudo apt-get install python-dbg python-mysqldb-dbg
Note that native Python extensions don’t work unless they are specifically compiled against the Python debug build (python-mysqldb-dbg)..
Then I add TrackRefs to my main loop:
def run(self):
self.running = True
logger.info("Started worker " + self.get_worker_id_string())
# Memory leak tracking
tracker = TrackRefs()
while self.running:
self.mark_for_download()
self.process_downloads()
self.process_encodings()
tracker.update() # Dump memory here
time.sleep(settings.WORKER_POLL_DELAY)
And after running a while I start getting interesting results:
7956 [2008-03-07 02:59:28,767] INFO Jobs needing sources to download 0
7956 [2008-03-07 02:59:28,768] DEBUG Processable jobs: 0
7956 [2008-03-07 02:59:29,754] DEBUG ———————-
7956 [2008-03-07 02:59:29,754] DEBUG Memory profiling
7956 [2008-03-07 02:59:29,754] DEBUG insts refs
7956 [2008-03-07 02:59:29,754] DEBUG <type ‘int’> 150 137406
7956 [2008-03-07 02:59:29,755] DEBUG <type ‘tuple’> 117 130211
7956 [2008-03-07 02:59:29,755] DEBUG <type ‘dict’> 5 8331
7956 [2008-03-07 02:59:29,755] DEBUG <type ’str’> 3 27643
7956 [2008-03-07 02:59:29,755] DEBUG <type ‘unicode’> 3 4606
7956 [2008-03-07 02:59:29,755] DEBUG <type ‘list’> 3 3492
7956 [2008-03-07 02:59:29,756] DEBUG <type ‘frame’> 1 962
7956 [2008-03-07 02:59:29,756] DEBUG <type ‘cell’> 0 12948
7956 [2008-03-07 02:59:29,756] DEBUG <type ‘function’> 0 9479
Woah! Who reserved 130 000 ints and tuples? No wonder that soon python gulps 1 gigabytes of memory. Since this is the only number which grows during the main loop cycling and there is no references to classes or objects debugging becomes a bit more difficult. I need to try to cross-reference the difficult tuple objects.
This didn’t go well - with gc.get_referrers() recurive parsing I got some results (example below). But it became soom clearthat debug references from the system itself was impossible: the memory debugging code will always create nasty cyclic references to the system, since it needs to track the objects. I gave up. There had to be something better.
9154 [2008-03-07 04:05:23,571] DEBUG /var/lib/python-support/python2.5/MySQLdb/connections.pyc
9154 [2008-03-07 04:05:23,571] DEBUG <type ‘function’>
9154 [2008-03-07 04:05:23,572] DEBUG defaulterrorhandler
9154 [2008-03-07 04:05:23,572] DEBUG <type ‘function’>
9154 [2008-03-07 04:05:23,572] DEBUG string_literal
9154 [2008-03-07 04:05:23,572] DEBUG <type ‘function’>
9154 [2008-03-07 04:05:23,573] DEBUG unicode_literal
9154 [2008-03-07 04:05:23,573] DEBUG <type ‘function’>
9154 [2008-03-07 04:05:23,573] DEBUG string_decoder
9154 [2008-03-07 04:05:23,573] DEBUG <type ‘function’>
9154 [2008-03-07 04:05:23,573] DEBUG __exit__
9154 [2008-03-07 04:05:23,574] DEBUG <type ‘function’>
9154 [2008-03-07 04:05:23,574] DEBUG begin
9154 [2008-03-07 04:05:23,574] DEBUG <type ‘function’>
9154 [2008-03-07 04:05:23,574] DEBUG __init__
9154 [2008-03-07 04:05:23,574] DEBUG <type ‘function’>
9154 [2008-03-07 04:05:23,575] DEBUG show_warnings
There was: Guppy. Thank you Sverker Nilsson! You saved my day.
Since the API of Guppy is a little eccentric, here are some examples for you:
# init heapy
heapy = guppy.hpy()
# Print memory statistics
def update():
print heapy.heap()
# Print relative memory consumption since last sycle
def update():
print heapy.heap()
heapy.setref()
# Print relative memory consumption w/heap traversing
def update()
print heapy.heap().get_rp(40)
heapy.setref()
With heapy.heap() ; heapy.setref() I got this output:
Partition of a set of 12 objects. Total size = 3544 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 3 25 2244 63 2244 63 unicode
1 2 17 708 20 2952 83 types.FrameType
2 3 25 432 12 3384 95 dict (no owner)
3 3 25 120 3 3504 99 str
4 1 8 40 1 3544 100 list
One adds get_rp() travelsal magic and everything becomes clear:
Reference Pattern by <[dict of] class>.
0: _ — [-] 14 (dict (no owner) | list | str | types.FrameType | types.Gene…
1: a [-] 3 dict (no owner): 0×8c11f34*2, 0×8c1bd54*2, 0×8c1f854*2
2: aa —- [-] 1 list: 0×833c504*18
3: a3 [-] 1 dict of django.db.backends.mysql.base.DatabaseWrapper: 0×8…
4: a4 —— [-] 1 dict (no owner): 0×83a65d4*2
5: a5 [R] 1 guppy.heapy.heapyc.RootStateType: 0xb787c7a8L
6: a3b —– [-] 1 django.db.backends.mysql.base.DatabaseWrapper: 0×8356a34
7: a3ba [S] 7 dict of module: ..db, ..models, ..query, ..transaction…
8: b —- [S] 1 types.FrameType: <<lambda> at 0×8b16ecc>
9: c [-] 2 list: 0×833c504*18, 0xb7dafe6cL*5
<Type e.g. ‘_.more’ for more.>
What there could in DatabaseWrapper object which is growing and growing… query debugger. Django keeps track of all queries for debugging purposes (connection.queries). This list is reseted at the end of HTTP request. But in standalone mode, there are no requests. So you need to manually reset to queries list after each working cycle.
while self.running:
self.mark_for_download()
self.process_downloads()
self.process_encodings()
tracker.update()
time.sleep(settings.WORKER_POLL_DELAY)
# Clear database connection ad reset query debugger
# between cycles to make sure that
# related resources get released
reset_queries()
connection.close()
print str(connection.queries)
But even after this fix, I got increase in tuple and int usage when monitoring with TrackRefs. But when I run heapy.heap() alone, there is no increase. So the tuple and int consumption must have been caused by TrackRef, sys.getobjects, gc, etc. magic itself.Share This
5 Comments
Posted on February 21, 2008 by Mikko Ohtamaa
Filed Under pys60
Python programming language, created in 1991 by Guido van Rossum, is the sixth most popular programming language and “the programming language of year 2007“. Nokia Series 60 is a Symbian-based mobile phone platform for high end smart phones which is the most widespread smart phone platform. Python for Series 60 is Python ported to S60. PyS60 is open source and there already exists a notable community around it - a sign of a successful software platform.
This essay is a post-mortem for creating a commercial grade PyS60 application. By “commercial” I mean something which is publicly available for end users and meets the criteria of Symbian Signed certification. I hope this essay will shed light on the real world challenges, thus aiding Nokia, mobile application vendors and PyS60 community to know when to use PyS60, to strengthen PyS60 as a platform and to guide people make most of it.
Description of the project
I was given a task to create a mobile phone client for a web service. The client runs on the the background, posting information from the phone (files, location) to the web site. Two developers were assigned for the job. In the end of the project, almost 100 Python modules and classes had been written (just for the mobile side of the things). I believe this is the most complex PyS60 development effort this far.
The application is divided in two parts: user interface process and background process. The background process is running all the time, monitoring the phone state and posting changes to the server. The UI process can be launched from the phone menu and it commands the background process. Two processes communicate over localhost TCP/IP sockets. The application communicates with the Internet server by HTTP multipart posts. The background daemon had four threads running: main thread, server communication thread, interprocess communication thread and GPS thread.
The core modules of the application were written portability in mind. In fact, one can run the core as standalone Python application on a normal PC. This makes unit testing and debugging much easier. Some supporting libraries and files were ported from the normal Python 2.2.2 distritbution to PyS60.
Python for Series 60 and the alternative platforms
When we started the project, I was not aware of commercial applications built on PyS60, besides Nokia’s own Mobile Web Server. I knew it was used as in-house in few large software companies. Python has traditionally had strong foothold in the academic world. So it does not come with a surprise that Finnish universities use PyS60 on mobile application software development courses due to Python’s low learning curve.
Python itself has been around longer than the most popular programming language, Java. Python’s suitability for the complex development jobs was largely untested until the rise of modern web. The dynamic, agile, nature of Python does not come for free. So called abstraction penalty (interpreting) slows down Python execution speed. It was not until the 00s when processor were fast enough to make this irrelevant matter: software development costs outrun hardware costs and more agile development tools had to be found.
When building S60 applications, you need to chose the progamming language between three candidates: Good old fashioned C++, Java Micro Edition (Java ME) and Python for Series 60.
Currently, most of Series 60, and Symbian, platform specific applications are written in C++, with a twist. One cannot consider this as “normal” C++ code, but the bastard child of this language. In their quest for the ultimate embedded efficiency (read: the smallest memory footprint) Symbian architectures placed an enormous mental burden on the poor Symbian developers. For example, Symbian C++ has custom pointer schemes, manual error handling, wicked threading and a weird build process. Or to put it bluntly: writing C++ for Symbian is far more difficult than writing C++ for platform X. You cannot really use any standard libraries on Symbian (this is changing, however). Thus, building S60 applications based on C++ technologies is costly, and painful. I have not yet met a happy Symbian C++ developer and I have met hundreds of them.
Then, we have Java ME. Sun made a good move by creating a unified mobile application run-time environment. Java ME is the only way to get your mobile application running regardless of the mobile phone vendor. But Java’s infamous slogan “write once, run everywhere” doesn’t hold a candle.Java ME implementations have internal differences and they are buggy as two weeks dead squirrel (seen one squirrel once and followed Sun’s Java ME mailing list long enough). Java ME gets the new features last due to standardization process. Java ME applications are heavily sandboxed and they cannot access half of the phone features.
And as the last option is Python for Series 60. Started as a Nokia’s research effort, Nokia brings this agile environment to mobile software development. Python for Series 60 has also a sister project, PyUIQ, which targets Sony-Ericsson phones.
PyS60 has had the reputation to be “prototype only” technology and it’s use has been counter-argumeted by “serious development needs C++”. This is slowly changing. Nokia has shown serious commitment to Python. With the launch of Mobile Web Server, Nokia proved that Python is feasible in commercial grade applications.
This is how my brain worked when I chose PyS60: Symbian C++ is notorious difficult and no sane person wants to ever touch it. Java ME might have not enough access to the phone features to fulfill the application requirements My client already used Python on embedded Linux and web servers. Why not give PyS60 a shot?
Arguments for Python for Series 60
Python is agile
This is maybe the most important argument to pick PyS60. Python tends to reduce development time compared to C++ or Java. The language itself is very easy to learn. Syntax is expressive and developers can achieve much more functionality in less lines of codes: You can focus on getting the task itself done. With higher level concepts and less hassle with lower level things, changes to introduce bugs radically decreases.
Python development does not have traditional compiling phase, though byte code files can be built for the final release to obfuscate the code and optimize loading times. Changes made into Python code are immediately (in seconds) visible in the running application. Even if the application runs on the target device, since code changes can be automatically synchronized over a Bluetooth connection. No wonder Python enjoys legendary fast prototyping language fame - the developers have less time to surf in web when they don’t need to wait compilers to bake yet another test build.
PyS60 is open source
I assume you are familiar with the benefits of open source already: vendor freedom, no licensing cost, not being mercy on others and so on. This all holds true for PyS60 as well.
PyS60 license (Python license) does not place restrictions for creating closed source software. You can have very good development tools for free: Series 60 SDK, Eclipse IDE with PyDev plug-in and PUTools bluetooth synchronization kit for free. This makes the barrier to enter PyS60 development very low.
Since PyS60 is fully available in the source code form, you can customize and fix it for your needs. The traditional horror story from Java ME development is that your application won’t run in the mobile phone model X, since there is a bug in the vendor’s Java Virtual Machine implementation. Unless you are A Very Big Player, don’t ever think about going to talk with the vendor and ask it to fix its bug - hitting one of these issues might prevent you to release product at all. With PyS60 you can control all aspects of the software execution up to the closed parts of the device. No more being mercy on others!
Being open is pop now. Embracing open source gives much for your company: good citizen reputation, hiring pool and free marketing.
Python is portable
Python runs on Linux, Windows mobile and Symbian - Python runs on almost every smart phone known to man. And not just the phones, it runs on every computer as well: Windows, OSX and Linux server and desktops. Moving your code between these environments is easy. Maybe there isn’t yet “cross-platform Python SDK for mobile phones”, but it might be just a matter of time when something leveraging the idea comes out.
In our project, core files, independent from graphical user interface, were built and unit tested on Linux desktop environment. No emulator was needed. It was not until the late development stages when they had to be moved to Symbian environment. Since you can develop and debug your code in such “easy” environment, you can work more effectively.
Because the full Python is available in the phone and it can run the exactly same modules as your servers, sharing code between these two environments opens possibility for making “smart” clients. Think about the situation where your phone is offline. It can have a subset of the full database loaded on a memory card. Since moving Python code from the server to the mobile is easy, the mobile application can have the same database functions and application logic modules loaded with it. Without the extra duplication of the logic code, one can operate on the small subset of the data and later synchronize it with the main database.
Python is extendable
You can mix C++ and Python. When you run out of ready components or you need unleash all available CPU cycles for CPU sensitive calculation tasks, you can write your own native extensions (DLL files) for PyS60 in Symbian C++. They enjoy all the same privileges as any native built code or application. Actually standalone PyS60 applications are just normal Symbian executables which just happen to have Python interpreter started inside them.
If you are working with phone features which are not yet very common and lack working Java ME implementations, say acceleration sensors, instead of creating a native C++ application you can create a Python application. You need to mix-in low level C++ only for the the critical part of the applications which operate with the native platform API.
PyS60 community is vivid
Surf to forum.nokia.com Python discussion boards and you can witness the activity around PyS60. A lot of folks are answering the beginners’ questions and guiding them. If your case is not totally unique, you can expect help to be found. As a matter of fact, you, reading this text, is a sole proof of powerful community. I was not directly paid to write this and I am not affiliated with Nokia any way - this writing flows from the bottom of my heart to promote PyS60 and open some window for the (hireable) talents of our company.
You love it
For some things, we just feel impulsive attachment. They are beautiful, aesthetic, sexy. When you see it, you need to have it. PyS60 is one of these things.
Arguments against Python for Series 60
PyS60 has not been around for long
PyS60 is a young project. It still has some maturity issues. If you don’t do preparations well and test that all needed modules work as you want them to work, you might need to do some extra work to get around the issues.
Especially PyS60 user interface lacks some crucial components and polish which may force you to create UI components yourself.
By “getting around” the issues I mean leaping to the native Symbian code world and writing embedded C++. Since this leap is all but a little step, in the terms of development skills and complexity, I recommend preparing for the worst before the roof is falling on your head. I have seen too much “plz reply me asap urgent my boss kills me this afternoon” PyS60 discussion board messages from persons who haven’t done their homework. For example, you could make sure that you have some hired Symbian guns available in the case your own effort was not enough.
Python is for high-end phones only
PyS60 and PyUIQ are available for the high end smart phones only. Nokia’s Series 40 devices, and other devices built around in-house operating systems, cannot run Python. If you are targetting to e.g. consumers markets and Java ME has all the required capabilities to implement the application, it is the only real way to go.
Symbian is dreadful platform
Symbian, as a software development platform, is very challenging. Due to its history, the feeling of developing for Symbian differs greatly from other mobile, desktop and server platforms. Things simple elsewhere, like installing an application, might become a development nightmare on Symbian where one has to fight his/her way through capability limitations, artificial device restrictions and signing process just to get the application to the phone.
PyS60 naturally inherits all these challenges. Java ME goes around the issue by having its own custom installation, signing, etc. schemes. On Symbian, even accessing some of the device features - a device you own and hold in your hand - requires you to throw in few hundred bucks cash to get the publisher id - to get the developer certificate - to sign the application - to have all required capabilities - to access the specific features (read: GPS). More cash and third party testing is needed to publish the application. This is the price of virus free DRM’ed platform.
Lack of entrepreneurship spirit
If your company states “we are a Java house” or “we are a C++ house” and you miss courage to look for alternatives, then maybe you shouldn’t. Sailing into the brave new world takes more resources than staying near your home harbor. On the other hand, if you are a technology company, it’s quite important follow technologies outside your mainstream, unless you want to suffer the extinction when the new kids on the block gain power (remember the day when COBOL died).
Conclusion
Nokia has done a great deed and effort by creating PyS60 and then giving it out for us to play. Now its own turn to act as a community to give back something we can.
Building commercial grade PyS60 applications might not be straightforward yet, but it is possible. Working with PyS60 is definitely more productive than using complex Symbian C++. It is not always straightforward as Java ME, but eventually it will be. Not only that, but PyS60 can surpass Java ME development with all the Python’s agility.
If you consider PyS60 for your commercial grade product, it all boils down to the feature set of the application, your existing team skill set and the hacker spirit. And now there exists at least one commercial vendor, besides mighty Nokia, who advertises commercial PyS60 services (hint: our company), so you won’t be alone.
We could look things in a larger perspective. In the mobile world, PyS60 is the pack leader of Python-in-your-pocket solutions. Windows CE has Python support available, but it is not officially backed up by Microsoft. This may change quickly when mobile .NET virtual machine gets DLR support making it possible to run IronPython. On Linux platforms, Python is a natural part of the platforms (OLPC, OpenMoko). Google Android and Apple iPhone will roll out their own software development platforms soon.
In the dream world, we could have a common mobile Python platform akin to Java ME of today. Java is not the only way to provide vendor neutral mobile applications: there already exists effort to get mobile cross-platform .NET implementation (Mono on embedded Linux, Red Five Labs .NET on Symbian). The future popularity of Linux phones, iPhone and Android might decide the faith of mobile Python. Could we have Python as a batteries-included feature in the future phones?Share This
9 Comments
Posted on November 27, 2007 by Mikko Ohtamaa
Filed Under Plone, eclipse, php, python
Currently I work in quite wide field of software development: Python (standalone, Plone, Zope, Django), PHP, Java, Symbian and embedded Linux. I am using Eclipse for development, since it’s pretty much the only consistent platform filling my needs. The nature of work also forces me to use different computers (Mac/Windows/Linux) with different clients. This drives me to reinstall Eclipse now and then.
Below are my personal notes what plug-ins are needed to get “perfect” Eclipse set-up. Basically they are just my own notes so that I don’t need to Google everything all over again every time I reinstall. I hope the readers can find new pearls here or suggest improvements.
Eclipse setup
Eclipse has internal updater/web installer. All plug-ins are downloaded as ZIP files and extracted to Eclipse folder or installed through the internal updater. Paste Eclipse update site URLs to menu Help -> Software updates -> Find and Install, New Remote Location. You can use dummy text as the name of update site.
Eclipse Web Tools Platform bundles Eclipse, Java development tools, HTML editor, CSS editor and some other generic useful stuff.
Python
PyDev is a plug-in for Python and Jython development.
Site URL: http://pydev.sourceforge.net
Eclipse update site URL: http://pydev.sourceforge.net/updates/
PDT
PDT download provides Eclipse, HTML editor, PHP editor and CSS editor.
Site URL: http://www.eclipse.org
Eclipse update site URL: http://download.eclipse.org/tools/pdt/updates/
Subclipse
Subclipse provides Subversion version control integration to Eclipse.
Eclipse update site URL: http://subclipse.tigris.org/update_1.2.x
In the installer, uncheck the integration modules checkbox or the installer will complain about missing modules.
JSEclipse
JSEclipse provides a better editor (over WTP) for Javascript files, with impressive outlining and autofill capabilities.
Download requires Adobe developer account or similar fill-in-the-fields crap.
Site URL: http://labs.adobe.com/technologies/jseclipse/
ShellEd
Syntax coloring for Unix shell scripts
Project site: http://sourceforge.net/projects/shelled
SQL Explorer
SQL editor with limited GUI capabilities. Based on Eclipse platform. Comes standalone and as Eclipse plug-in.
needs MySQL JDBC driver
Technorati tags: Python Plone Django PHP Eclipse Web development Subclipse Javascript SQLShare This
1 Comment
Next Page →
RSS
