planetmobile.us up and running

planetmobile.us is a blog aggregator focused on mobile software development.The purpose of the aggregator is to give all good blogs in one packet, without the need to hunt them individually.

planetmobile.us was managed by Christopher Schmidt until I resurrected it few weeks ago.

Please feel free to subscribe your feed to the planet!

How to encode h264 video files for Nokia Series 60 standalone playback

Bored with Spiderman 3 which came with your Nokia N95 8 GB? This guide shortly tells how to get movies into your N95 on Ubuntu Linux using ffmpeg video encoder. The aim is to encode video suitable for playback from Nokia N-series (N95, N78, others) mobile phone memory card. We use h264 + AAC codecs which provides the best quality/compression rate for Nokia phones currently.

Ubuntu does not distribute proprietary codes. First thing you need to do is to rebuild ffmpeg.  Since Ubuntu 8.04 Hardy Heron ships with ffmpeg from 2007, which is aeons old in video codec years, you need to build libx264 and ffmpeg from SVN sources. Here are detailed, valid, instructions. Note that FFMPEG trunk is not currently stable (September 2008), so you need to use revision 15261 which needs this little patch. Indeed, this is a very difficult month to start your career in the dark world of video encoders.

To make it legal and support open source codec development,  please pay for your codecs.

Then we use this guide by Robert Swain. We have a tiny sub 2,4″ screen, we do not care about the quality and do one pass encoding. By empirical research, I have found that the following MPEG-4 profile parameters are compatible with N95 8 GB and provide the optimal result. You can vary video and audio bitrate depending on your taste.

Here is a script which recursivelu encodes all detected video files suitable for mobile format:

#!/bin/sh
#
# Optimal movie encoding for Nokia N-series mobile phones
#
# Copyright 2008 Red Innovation Ltd.
#
# Say hi if you find this useful.
# We do some professional mobile video publishing, so if you
# need a helping hand please call us.
#
# Usage: Run encode.sh in any folder and all video files are recursively converted to mobile phone suitable format
#
# Note: We expect all the source material be in 16:9 aspect ration
#
# Also see http://www.nseries.com/index.html#l=support,search,faq,general,video%20encoding,53848
#

VIDEO_BITRATE=300k

AUDIO_BITRATE=72k

# Assume locally build ffmpeg + x264 in /usr/local/bin
# http://ubuntuforums.org/showthread.php?t=786095
export LD_LIBRARY_PATH=/usr/local/lib

# Search all source AVI, MPG and WMV video files
# Place all encoded files to the same folder with the source, with added .mp4 extension
find . -iname "*.avi" -or -iname "*.wmv" -or -iname "*.mpg" | while read src ; do
        srcfile=`basename "$src"`
	srcfolder=`dirname "$src"`
	dstfile="$srcfolder"/"$srcfile".mp4

	# The magical string!
	# Size and cropping is for 16:9 source material, so that 320:240 display will have black bars.
	# Fex pixels off... note that h264 sizes must be multiplies of 16, use 256x144 for streaming
	# N95 RealMedia player does not seem to respect MPEG-4 embedded aspect ration info.
	/usr/local/bin/ffmpeg -y -i "$srcfile" -acodec libfaac -ab $AUDIO_BITRATE -s 320x176 -aspect 16:9 -vcodec libx264 -b $VIDEO_BITRATE -qcomp 0.6 -qmin 16 -qmax 51 -qdiff 4 -flags +loop -cmp +chroma -subq 7 -refs 6 -g 250 -keyint_min 25 -rc_eq 'blurCplx^(1-qComp)' -sc_threshold 40 -me_range 12 -i_qfactor 0.71 -directpred 3 "$dstfile"

done

Introducing Python for Series 60 Community Edition

This blog post will introduce Python for Series 60 Community Edition.

Pythor for Series 60 Community Edition is a new open source effort to push Python for high quality mobile phone development. It aims to provide a maintained software stack for creating real mobile applications. The codebase is derived from the original Nokia’s Python for Series 60 codebase, but has been refactored for better integration with third party extensions and patches and commercial grade application deployment.

Motivation

Building and distribution

It is difficult to distribute Python for Series 60 applications to the end user with the current Nokia’s PyS60 distribution. You probably want to modify or extend PyS60 in some way. Since the build chain and deployment model is not designed for changes this would collide with the other PyS60 installations. Symbian Platform security prevents installing conflicting binaries. Thus, one can effectively have one Nokia PyS60 application in the phone once.

There are other problems: Nokia PyS60 distribution has UIDs in Nokia protected range. Embedded SIS file cause extra installation dialog and an application manager uninstall entry. Trimming down Nokia PyS60 distribution is a little bit difficult.

To overcome all these issues we created a build chain which spits out monolithic PyS60 distributions. We build only one DLL whose name and UID can be decided. Also the build chain is Scons for Symbian, scrapping the obscure, inflexible and difficult to understand Symbian ABLD once for all.

Evolution towards higher quality

Currently there is no centralized authority to co-ordinate PyS60 open source developers and maintain the repository of all the extension and patches. This effectively prevents the biggest benefit of open source: open innovation and gradual evolution of the product. It would be very nice having all those third party extensions, now scattered around the internet, under one maintained source - making PyS60 more functional out of the box.

The community maintained repositories do not have the same restrictions as ones managed by a big public corporations. It is not a probable target of a trigger happy lawyer action and ungentlemanly competition: the discussion and plans can be public and due dilugence check of the code more relax.

We started the project in the Launchpad. Launchpad provide a distributed version control system (Bazaar) which streamlines the process of integrating third party commits and patches. This should encourage contribution. The standard build system makes it easy to roll out applications and extensions from bare  C++ source up to the end user distributable SIS files.

It is yet to see what kind of co-operation possibilities between the community and Nokia exists. In the future, it should be possible to cherry pick patches from PyS60 community edition to Nokia’s own version.

Showing the commercial potential of PyS60 in the mobile application development

On Python you can write native Series 60 applications with very little effort compared to hardcore C++ banging, lowering the barries to enter the mobile application development.

We do not deny that we have an extrinct motivation called money. Of course we have also instrict motivations like thinking Python is the best programming language in the world and we all want to be most respectable gurus in it. Gurus need to eat still, though. We hope that our effort does not go unnoticed in the mobile application development world and good subcontract offers fill our inboxes.

Also, there is the John McClane effect. Unless we had done it, no one had. Somebody has to save the world, despite the hangover.

It runs on Linux

Since we are no longer dependend on .BAT/Perl/Windows hindered ABLD buildchain, we can (almost) crosscompile and build native Symbian binaries in Linux and  Python applications. All good hackers use Linux - but currently there are still kinks and you need to use WINE for some parts - all sane Symbian developers are tied to Windows based tools for now and so are these instructions.

Prerequisitements

You need all this stuff to get things running.

Install Bazaar

You need Bazaar distributed version control client. We are not planning to have fixed releases for Python for Series 60 community edition any time soon. This is because 1) the most magic happens at a compiler level and we are providing a buildchain 2) we hope this fosters incoming patches.

Why Bazaar?

Install ActiveState Perl

Series 60 SDK has ActiveState as a prerequisitement for running its installer.

http://www.activestate.com/Products/activeperl/index.mhtml

Install Series 60 SDK

Use only Series 60 3.0 maintenance release. Other releases have SDK bugs preventing correct Python compilation.

Get the Windows installer from http://forum.nokia.com. Forum Nokia Registration is required.

Please use the default installation location C:\Symbian\9.1\S603rd_MR.

Install Carbide.c++ express

Carbide.c++ comes with a Windows compiler to compile the emulator binaries. You need this only if you indend to develop and test your applications on Series 60 emulator.

http://www.forum.nokia.com/info/sw.nokia.com/id/dbb8841d-832c-43a6-be13-f78119a2b4cb.html

Forum Nokia Registration is required.

Use Software updater in Carbide.c++ to install PyDev, Python developer extensions for Eclipse.

http://pydev.sourceforge.net/

Install Python 2.5

Scons build chain and our utility scripts use Python.

http://www.python.org/download/releases/2.5.2/

Use the installer EXE and the default installation location C:\Python25. If you want to use advanced Bluetooth shell (PUTools) you also need wxPython and pyserial packages.

Install SCons

Python for Series 60 build script are based on SCons. It is a build system using Python as a recipe langauge.

http://sourceforge.net/project/showfiles.php?group_id=30337

Install Subversion

Install Subversion client for Windows. This is needed for checking out Scons for Series 60.

http://www.collab.net/downloads/subversion/

Registration to CollabNet is needed to download Windows binaries.

Scons for Series 60

SCons for Series 60 is available as a separate project. SCons for Symbian is a build toolchain intended as a replacement for Perl and MMP files used on regular Symbian projects. SCons for Symbian is not limited to build Python - You can use it to build any Series 60 C++ application.

http://code.google.com/p/scons-for-symbian/

This is later checked out during to the environment construction, so you do not need to install it now.

We have included a workaround for a problem with limited command line length on Windows.

Included tools

The following tools are included in the trunk tools folder:

These tools are not licensed under Apache license. Some of them are under GPL license. However, we believe that distribution them is ok, since this falls under GPL’s mere aggregation clause. However if you indent to distribute commercial applications built from PyS60 Community codebase, make sure that you understand the set of different licenses involved.

Set up build environment

Ensure that Bazaar is properly in your Windows path.

Create a workspace folder

First you need to subst (make a folder appear as a driver letter) in Windows. Open command line. Go to SDK folder.

C:\Symbian\9.1>subst t: S60_3rd_MR
T:
mkdir workspace

Now choose this folder as a workspace folder in Carbide C++ and create an Empty Symbian C++ project called ”pys60” there.

Checkout PyS60 community edition

The go to this folder

T:
cd workspace\pys60
bzr branch lp:pys60community
cd pys60community\src

Preparing the build

This needs to be done only once.

We need to pacth the existing Series 60 SDK headers which have some bugs.

T:
cd \epoc32\include
\workspace\pys60\pys60community\src\tools\patch.exe -p1 < \workspace\pys60\pys60community\src\pys60-fix-3rded-sdk.diff

EPOCROOT must be set for some Series 60 SDK tools to work. We point to T: drive root.

T:
cd workspace\pys60\src
set EPOCROOT=\

As we still have some dependencies to the legacy system, one needs to configure the build system using PyS60 setup. This will generate some files and defines for Series 60 versio 3.0. 

c:\Python25\python.exe setup.py configure 30

Do not run bldmake bldfiles.

You need to convert legacy MMP build files to SCons based. First we need to possibly fix up PATH, since Carbide C++ might break it.

set PATH=c:\program files\bazaar;c:\program files\CSL Arm Toolchain\arm-none-symbian elf\bin;c:\program files\CSL Arm Toolchain\libexec\gcc\arm-none-symbianelf\3.4.3;C:\program files\CSL Arm Toolchain\bin;t:\epoc32\gcc\bin;t:\epoc32\tools;t:\epo c32\tools;C:\program files\CSL Arm Toolchain\bin;C:\Program Files\Common Files\Symbian\Tools;C:\Perl\site\bin;C:\Perl\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem

c:\Python25\python tools\mmp2scons.py
ext\\calendar\\calendar.mmp:60:7: warning: no newline at end of file
Creating recipe ext\miso\build.py
Creating recipe ext\socket\build.py
...
Done!

Checkout SCons for Symbian. We assume it lives in src tree.

"c:\Program Files\CollabNet Subversion"\svn checkout http://scons-for-symbian.googlecode.com/svn/trunk/ scons_symbian

Running the build

Now we can execute the actual Python build script. This will create one monolithic emulator DLL which has almost all the PyS60 extensions built in - some extensions need manual building, since they rely on headers not found from standard Series 60 SDK.  You might need to reset PATH to default to Carbide C++ after the previous mmp2cons step by reopening the console window.

c:\Python25\Scripts\scons

You should see the following output. As you can see, UIDs are being allocated dynamically as instructed in Scontruct UID_BASE argument.

scons: Reading SConscript files ...
EPOCROOT=\
Info: SIS creation disabled
Building winscw udeb
Defines []
Getting dependencies for e32socket.pyd
Getting dependencies for _topwindow.pyd
Getting dependencies for zlib.pyd
Getting dependencies for _locationacq.pyd
Getting dependencies for _location.pyd
Getting dependencies for _graphics.pyd
Getting dependencies for _sysinfo.pyd
Getting dependencies for Python222Config.lib
Getting dependencies for Python222.dll
Getting dependencies for Python_appui.dll
Allocated UID:0xE1000000
Getting dependencies for Python.exe
Allocated UID:0xE1000001
Getting dependencies for Python_launcher.exe
Allocated UID:0xE1000002
scons: done reading SConscript files.
scons: Building targets ...

...

scons: done building targets.

Now you should be able to start a custom built Python shell in the emulator. You should see the following applications in Installation menu: helloworld, btconsole and filebrowser. Try launch helloworld and if it opens a pop up dialog the build has been succesful.

Building a SIS file for mobile phones

To build a target distribution type:

scons release=urel compiler=gcce dosis=true

This should yield to the result:

scons: Building targets ...
ensymble(["MyPythonForSymbian.sis"], [])
scons: warning: no package version given, using 1.0.0
scons: warning: no certificate given, using insecure built-in one
scons: done building targets.

If you want to build a SIS file signed with your developer certificate:

c:\python25\scripts\scons compiler=gcce release=urel dosis=true simplesis={'--privkey':'C:\\Certificates\\PrivateKeyNoPassphrase.pem','--cert':'C:\\Certificates\\MyApp.cer','--passphrase':''}

For now, installing the SIS file works only for C drive  - we’ll fix this little issue soon.

If you do not have a symbiansigned.com developer certificate you can sign the SIS file online for one phone (one IMEI code).

  1. Go to symbiansigned.com
  2. Register
  3. Enter your IMEI and upload the SIS file to OpenSigned Beta

Building your own application

The purpose of this project is to make rolling out your PyS60 applications possible - so here we go. Currently we make a quite bad assumption that all the application live in the same source tree with PyS60 community edition due to problems with absolute file paths with Symbian build tools - we will figure out a long term solution for this later.

PyS60 applications are stub Symbian executables which boostrap Python virtual machine and start the Python code execution. Executables are linked against a custom PyS60 DLL and they are restricted by capabilities given to the EXE file.

PyS60 applications live in applications source tree. The source tree comes with Bluetooth console, Hello world and File browser sample applications.

Scons build scripts takes as applications parameter a comma separated list which applications are included in the build.

scons builtin=all applications=helloworld,filebrowser

Applications consist of

See applications\helloworld folder to examine what files are needed to build an application. All application Python modules go to the private application folder (/private/myapplicationuid). Default.py must boostrap PYTHONPATH (sys.path) for this folder - PYTHONPATH defines where Python interpreter looks for the code. Application UIDs can be chosen manually or they are picked automatically by Scons for the unprotected test range.

Note that Python Script Shell application is handled out of this flow due to its legacy heritage.

Adding in your own extension

If you have development an PyS60 extension you can drop in into the buildchain easily. Each extension is defined in ext subfolder. It consists of necessary CPP, H and Python files. The building structure is defined in build.py using SConstruct command PyS60Extension().

Build.py files can be automatically generated from legacy code using mmp2scons.py converter.

c:\Python25\python.exe tools\mmp2scons.py

ext\calendar\calendar.mmp:60:7: warning: no newline at end of file
ext\progressnotes\progressnotes.mmp:38:7: warning: no newline at end of file
ext\uikludges\uikludges.mmp:37:7: warning: no newline at end of file
Creating recipe ext\\socket\build.py
Creating recipe ext\\glcanvas\build.py
Creating recipe ext\\graphics\build.py
...
...Done!
c:\Python25\scripts\scons
>> import applicationmanager

If your extension is using thread local storage (Dll::Tls()) you might need to figure out how to workaround with it. See socket  module for example. You may also need to play around with the init function of the Python extension - it must be init + module name.

Developing on target

If you want to develop your application on a mobile phone, you do not need to go through the full development cycle for every little change. It is possible to update Python files on a phone without SIS installation. You can either automatically synchronize changed files from your PC to Phone (the example below) or you can edit files in-place on the Phone either using PCSuite or Series 60 SMB server.

Here are short instructions how to update files using PUTools console (btconsole). PUTools is wxPython based remote Python shell which allows you to run Python console commands over a Bluetooth connection from your PC. PUTools also has a file syncrhonization feature - after editing source code on the PC changes are reflected automatically to the phone.

  1. Put application Python files to a shared folder on the phone e.g. the memory card root (E:\). Add startup.py to E:\ which will modify sys.path to include your files.
  2. Add incoming Bluetooth serial port on windows (Control panel -> Bluetooth -> COM ports)
  3. Start Bluetooth shell on the computer (tools\putools\pcfiles\console.bat)
    4. T:\workspace\pys60\pys60community\src\tools\putools\pcfiles>c:\python25\python.exe putools.py com5
  4. Start Bluetooth shell on the phone (btconsole icon)
  5. Edit PUTools sync.config file on PC and run sync command on Bluetooth terminal to update changes made on PC to phone
  6. Run application launcher in the console (depends on the application structure how it is best to bootstrap in the shell):
>> import filebrowser
>> filebrowser.FileBrowser.run()

Release notes

Here is the short summary of differences with the current PyS60 community edition and one available from Nokia. This information is also available in divergence.txt file in the source folder.

2008-08-29 Mikko Ohtamaa <mikko@redinnovation.com>

    * PyS60 general

        New build chain and static config generation

        Migration tool for MMP -> Scons based extensions

        Added several tools included in the core distribution: sisinfo, ensymble, cog, patch

        Patched py2sis tool

        Contains extension: applicationmanager

        Contains extension: uikludges

        Contains extension: progressnotes

        Contains extension: miso

        Contains application: Bluetooth shell

        Contains example applications: filebrowser, helloworld

        Changed Bluetooth console bootstrap to e:\startup.py         

2008-08-15 Antti Haapala <antti@redinnovation.com>

    * e32socketmodule.cpp:

        socket.access_points has more information, two new
        fields is given per access point: isptype and bearertype,
        whose values are integers corresponding to values returned
        by CApSelect::Type and CApSelect::BearerType respectively.
        No symbolic constants are yet exported.

    * appuifwmodule.cpp:

        multi_select_list has a new argument, selected, which defaults
        to None. Given a list of integers, the items with the given indices
        are initially selected.

Conclusion

We hope this helps you to get started with PyS60 community edition. It’s still a bit complicated, since setting up the build environment on Windows is a such a pain. In the future, when the Linux based build system is reading settings up the development environment should be easier - all those boring steps happen automatically.

This might be still too difficult for some of the readers, since a lot of prerequirement work must be done before anything useful can be done. Feel free to comment the article in this blog, but we hope that you use Answers section in Launchpad to ask help and technical questions related to PyS60 community edition.

PyS60 application release build toolchain

A common question for Python for Series 60 newcomers is how to build standalone Symbian applications from Python source code. We have been using Makefile based toolchain internally. I describe it in this picture, I didn’t bother to add thumbnail for the image, since it’s a 3400 pixels wide diagram.

The diagram describes building a PyS60 application with some Python extensions (Symbian native C++) mixed in and bundling it all to one downloadable SIS file. The application will appear as any first class S60 application in the menu and the user does not know it’s running Python internally, besides bad installation experience (it challenges Microsoft installers with all those unnecessary yes/no questions), extra uninstaller entries and slow start-up time.

The biggest problems are caused by embedded SISs (SIS inside other SIS files) which are not treaded very wel by several Symbian parties.  In theory, it could be build one monolithic SIS, but you’d need to recompile PyS60 from scratch and patch UIDs inside it for your own UIDs received from symbiansigned.com. We are planning to explore SCons based build solution to address this problem, since Makefiles are a bit unflexible with tasks like PKG file and UID range generation.

Here is a PKG file example for final user distributable SIS file.

Also, see UIKludges project for additional details for PKG files of Python extensions.

You need to have

You need to master

Pros

Cons

Ps. I would have put this thing to wiki.opensource.nokia.com, but their webmaster email address is non-functional and one cannot upload images to their Wiki.

Symbian digging its own grave with symbiansigned.com

Edit: Made correction to iPhone SDK price/added self-signed option

I have been recently involved in Symbian development. Unfortunately, I could say. Symbian has made the life of Symbian developers headacheful. The main problem is that the application signing is made unnecessary cumbersome. Getting a required developer certificate to just to run your one liner test code is a long road paved with curses, stress and ponder how a world class company can screw up things so badly.

I write this post to vent out the frustration. Hopefully someone picks up the feedback presented here. Personally, I don’t care anymore really. After explaining to the customer once “I am afraid that we cannot have the demo in this meeting, since we were unable to get the developer certificates in time. It was nice you did summon all of your executes, though.” my sympathy towards Symbian has fallen to zero.

In the long run, I believe, this will hurt Symbian when developers leave the platform. When this happens Symbian will miss 1) developer base 2) the wow value of all new cool things when hackers move to happier platforms - after all you don’t want to work with things causing a headache even if you are paid generously. Professional Symbian capable developers are already rare and scaring away the potential candidates (students, open source developers, freeware developers) won’t really help the matter. Of course, for us, who are already in the business, it’s good since we can raise our hourly fee and die as rich dinosaurs (remember COBOL).

Looks like the root problem is the cocky attitude: “We are the biggest mobile operating system in the world. You are not our customer, the biggest mobile phone brands are. Thus, you are an insignicant fly and we don’t need to listen to you. Please go away.” The big boys tell that this very same reason lead to Motorola’s fall from the number one mobile phone vendor position - so let’s see if the history is repeating itself.

The problems of Symbian development process, especially signing, are well known. Symbian Signed forum category has 120 000 views. Symbian signed support requests has 57 000 views and 2000 messages, since November 2007. Over a thousand support requests in four months. Symbian has money. It could fix the problem if it has will. If there is will in the organization and the work just does not progress, the corporate world has a simple solution: fire the management and hire capable responsible persons.

I won’t go the details of the problems in this post. I have prepared a post containing over 50 bulletins with detailed steps to get the developer certification process done and it lists every little mistake in Symbian Signed process. Unsurprisingly, the post mentions fuck and hell over twenty times, so I am reluctant to post it on our company site. So I sketched a manager comprehensible overview of the state of the process in this post. To justify my criticism, I compare Symbian to Apple iPhone and make some suggestion how to fix things.

Certificate costs

Symbian: 200$/1.5 years (you need a publisher id to get a developer certificate) + 20$ per signing round

Update: You can still make self-signing certificates yourself with limited capabilities (no GPS), but symbiansigned.com doesn’t bother to mention it.

Apple: 99$/year (for testing and distribution applications)

Suggestion: Make the developer certificate free. It used to be free, but based on this post, Symbian signed claims that they do not have enough CPU power, because of all kind of nasties are knocking their server. Come on! it’s Internet. You are hosted by Cidercone and by knowing how much they charge for the hosting I believe you can get every possible server power in the world to generate your keys.

This development start up cost automatically excludes third world people, freeware makers and students i.e. the future Symbian professionals. It’s hard to maintain a platform if there is not enough skilled people to work with it. There won’t be meat into Nokia’s and Sony-Ericsson’s innovation machines.

Development start lead time

How long it takes to get “Hello World” code to the phone.

Symbian: 8 workdays

Apple: 6 hours

Complete Guide to Symbian Signed guide (manual) is 20 pages PDF. How often you need a user manual to use a web site? Even with this page amount, the manual lacks crucial pieces of information which you need to hunt and ask help from the Symbian Signed support forums. I had to make 3 support request for Symbian Signed and 1 for TrustCenter.

Set-up process

Symbian: Create 3 user accounts (symbiansigned.com, developers.symbian.com, TrustCenter). Fax your passport and some signed papers to TrustCenter Germany. Wait when your request is manually processed. Ask TrustCenter what I do with the link they send me (hint: the only working web browser for their site is the very same browser you orignally submitted your application from). Convert TrustCenter key to Symbian specific format using obscure BAT tool. Use half-baked Windows EXE to generate files. Ask at the support forums permission to be allowed to create the Publisher id.

Apple: Download SDK. Click “I Agree.”

Symbiansigned management team must be either incomplete or inexistent, or they just hate their users. Looks like no one didn’t bother model or test this process, since the flaws are so apparent. Why do we need three user accounts? Why do we need a user account at all, isn’t my credit card payment enough? Why there can’t be a single web page on symbiansigned.com where I could do these all things without? Also, Symbian could do these things without third parties (TrustCenter) who are badly integrated to the process.

Web site and information reachability

Symbian: Two sites with different user accounts. Information is spread around and hidden behind various for registred users only pages. Often Symbian PDFs (yes they seem to prefer zipped PDFs) and web pages cross-link to other sites and these links are broken. Their internal site search engine does not give meaningful results. Google cannot search pages, since they are hidden behind the registration. Web forms don’t work. Navigation paths are unclear and you often need to guess where a certain piece of information could be.

Apple: One site - and it just works.

Why Symbian hides a crucial BAT tool in the page labyrinth of developers.symbian.com and behind login? Does the 100 lines of BAT code contains a very hidden trade secret? I doubt. (I also doubt that a BAT tool is a good way to solve a problem which could be fixed with a web form or Javascript, but that’s another matter.)

I think this is the part of the attidude problem. SymbianSigned.com mission is not to “secure” the development and control the developers of the platform. It’s mission should be easily enable developers to start development and release their software for Symbian. The meaning of “easily” includes that the web site is working and up to the standards. Please hire a usability expert (we are here) and get it fixed. This problem is not solvable by writing yet-another user manual PDF.

Giving feedback

Symbian: Support like goes to the Symbian developer forum. A bulleting board with unorganized, angry, posts. They don’t give a feedback form to report the problems on Symbian Signed.

Apple: “There are three great ways to get answers to your questions about ADC membership, products, and services: visit our Frequently Asked Questions web pages, submit your questions or feedback using the form below, or call us to speak directly to a worldwide support agent. We look forward to assisting you.”

I’d help symbiansigned.com to get its site better if I could. When there is a broken link, I can hit feedback and ask them to fix it. But looks like they want to be ignorant about the matter - if I report a problem on their forum I bet no one will ever fix it. There won’t be even “thank you.” Probably no one ever reads my post. On the other hand, when having such non-functional service, this feedback channel would be flooded in overnight.

Conclusion

Symbian has unforgiveable screwed up with its signing process and doesn’t seem to get the feedback from the developers who suffer from it. Steve Ballmer’s famous quote “developers, developers, developers” refer to what’s the important factor making the platform succesfully in long term. This is especially crude when you know that things used to be good.

SDK released - Python in iPhone?

I just read waffle’s blog entry about iPhone SDK release. Looks like Objective C is the only supported language by default (I am just downloading SDK).The comments speculated that embedding Python is not possible due to size constraints. Bollocks I say =) Python for Series 60 phones is 500 kb download without trimming. It’s less than the size of HTML page you are viewing now - RAM footprint is even smaller) If Series 60 phones, which have much more modest hardware specifications, can run Python it shouldn’t be a problem for iPhones either.

Why Apple didn’t add additional language support by default? Well they seem to have their hands full to get SDK out at all (delays) so we shouldn’t expect to have perfect set in 1.0 release.

Now, who wants start a porting project with me? ;)

Building commercial grade mobile applications with Python for Series 60

Python programming language, created in 1991 by Guido van Rossum, is the sixth most popular programming language and “the programming language of year 2007“. Nokia Series 60 is a Symbian-based mobile phone platform for high end smart phones which is the most widespread smart phone platform. Python for Series 60 is Python ported to S60. PyS60 is open source and there already exists a notable community around it - a sign of a successful software platform.

This essay is a post-mortem for creating a commercial grade PyS60 application. By “commercial” I mean something which is publicly available for end users and meets the criteria of Symbian Signed certification. I hope this essay will shed light on the real world challenges, thus aiding Nokia, mobile application vendors and PyS60 community to know when to use PyS60, to strengthen PyS60 as a platform and to guide people make most of it.

Description of the project
I was given a task to create a mobile phone client for a web service. The client runs on the the background, posting information from the phone (files, location) to the web site. Two developers were assigned for the job. In the end of the project, almost 100 Python modules and classes had been written (just for the mobile side of the things). I believe this is the most complex PyS60 development effort this far.

The application is divided in two parts: user interface process and background process. The background process is running all the time, monitoring the phone state and posting changes to the server. The UI process can be launched from the phone menu and it commands the background process. Two processes communicate over localhost TCP/IP sockets. The application communicates with the Internet server by HTTP multipart posts. The background daemon had four threads running: main thread, server communication thread, interprocess communication thread and GPS thread.

The core modules of the application were written portability in mind. In fact, one can run the core as standalone Python application on a normal PC. This makes unit testing and debugging much easier. Some supporting libraries and files were ported from the normal Python 2.2.2 distritbution to PyS60.

Python for Series 60 and the alternative platforms

When we started the project, I was not aware of commercial applications built on PyS60, besides Nokia’s own Mobile Web Server. I knew it was used as in-house in few large software companies. Python has traditionally had strong foothold in the academic world. So it does not come with a surprise that Finnish universities use PyS60 on mobile application software development courses due to Python’s low learning curve.

Python itself has been around longer than the most popular programming language, Java. Python’s suitability for the complex development jobs was largely untested until the rise of modern web. The dynamic, agile, nature of Python does not come for free. So called abstraction penalty (interpreting) slows down Python execution speed. It was not until the 00s when processor were fast enough to make this irrelevant matter: software development costs outrun hardware costs and more agile development tools had to be found.

When building S60 applications, you need to chose the progamming language between three candidates: Good old fashioned C++, Java Micro Edition (Java ME) and Python for Series 60.

Currently, most of Series 60, and Symbian, platform specific applications are written in C++, with a twist. One cannot consider this as “normal” C++ code, but the bastard child of this language. In their quest for the ultimate embedded efficiency (read: the smallest memory footprint) Symbian architectures placed an enormous mental burden on the poor Symbian developers. For example, Symbian C++ has custom pointer schemes, manual error handling, wicked threading and a weird build process. Or to put it bluntly: writing C++ for Symbian is far more difficult than writing C++ for platform X. You cannot really use any standard libraries on Symbian (this is changing, however). Thus, building S60 applications based on C++ technologies is costly, and painful. I have not yet met a happy Symbian C++ developer and I have met hundreds of them.

Then, we have Java ME. Sun made a good move by creating a unified mobile application run-time environment. Java ME is the only way to get your mobile application running regardless of the mobile phone vendor. But Java’s infamous slogan “write once, run everywhere” doesn’t hold a candle.Java ME implementations have internal differences and they are buggy as two weeks dead squirrel (seen one squirrel once and followed Sun’s Java ME mailing list long enough). Java ME gets the new features last due to standardization process. Java ME applications are heavily sandboxed and they cannot access half of the phone features.

And as the last option is Python for Series 60. Started as a Nokia’s research effort, Nokia brings this agile environment to mobile software development. Python for Series 60 has also a sister project, PyUIQ, which targets Sony-Ericsson phones.

PyS60 has had the reputation to be “prototype only” technology and it’s use has been counter-argumeted by “serious development needs C++”. This is slowly changing. Nokia has shown serious commitment to Python. With the launch of Mobile Web Server, Nokia proved that Python is feasible in commercial grade applications.

This is how my brain worked when I chose PyS60: Symbian C++ is notorious difficult and no sane person wants to ever touch it. Java ME might have not enough access to the phone features to fulfill the application requirements My client already used Python on embedded Linux and web servers. Why not give PyS60 a shot?

Arguments for Python for Series 60

Python is agile

This is maybe the most important argument to pick PyS60. Python tends to reduce development time compared to C++ or Java. The language itself is very easy to learn. Syntax is expressive and developers can achieve much more functionality in less lines of codes: You can focus on getting the task itself done. With higher level concepts and less hassle with lower level things, changes to introduce bugs radically decreases.

Python development does not have traditional compiling phase, though byte code files can be built for the final release to obfuscate the code and optimize loading times. Changes made into Python code are immediately (in seconds) visible in the running application. Even if the application runs on the target device, since code changes can be automatically synchronized over a Bluetooth connection. No wonder Python enjoys legendary fast prototyping language fame - the developers have less time to surf in web when they don’t need to wait compilers to bake yet another test build.

PyS60 is open source

I assume you are familiar with the benefits of open source already: vendor freedom, no licensing cost, not being mercy on others and so on. This all holds true for PyS60 as well.

PyS60 license (Python license) does not place restrictions for creating closed source software. You can have very good development tools for free: Series 60 SDK, Eclipse IDE with PyDev plug-in and PUTools bluetooth synchronization kit for free. This makes the barrier to enter PyS60 development very low.

Since PyS60 is fully available in the source code form, you can customize and fix it for your needs. The traditional horror story from Java ME development is that your application won’t run in the mobile phone model X, since there is a bug in the vendor’s Java Virtual Machine implementation. Unless you are A Very Big Player, don’t ever think about going to talk with the vendor and ask it to fix its bug - hitting one of these issues might prevent you to release product at all. With PyS60 you can control all aspects of the software execution up to the closed parts of the device. No more being mercy on others!

Being open is pop now. Embracing open source gives much for your company: good citizen reputation, hiring pool and free marketing.

Python is portable

Python runs on Linux, Windows mobile and Symbian - Python runs on almost every smart phone known to man. And not just the phones, it runs on every computer as well: Windows, OSX and Linux server and desktops. Moving your code between these environments is easy. Maybe there isn’t yet “cross-platform Python SDK for mobile phones”, but it might be just a matter of time when something leveraging the idea comes out.

In our project, core files, independent from graphical user interface, were built and unit tested on Linux desktop environment. No emulator was needed. It was not until the late development stages when they had to be moved to Symbian environment. Since you can develop and debug your code in such “easy” environment, you can work more effectively.

Because the full Python is available in the phone and it can run the exactly same modules as your servers, sharing code between these two environments opens possibility for making “smart” clients. Think about the situation where your phone is offline. It can have a subset of the full database loaded on a memory card. Since moving Python code from the server to the mobile is easy, the mobile application can have the same database functions and application logic modules loaded with it. Without the extra duplication of the logic code, one can operate on the small subset of the data and later synchronize it with the main database.

Python is extendable

You can mix C++ and Python. When you run out of ready components or you need unleash all available CPU cycles for CPU sensitive calculation tasks, you can write your own native extensions (DLL files) for PyS60 in Symbian C++. They enjoy all the same privileges as any native built code or application. Actually standalone PyS60 applications are just normal Symbian executables which just happen to have Python interpreter started inside them.

If you are working with phone features which are not yet very common and lack working Java ME implementations, say acceleration sensors, instead of creating a native C++ application you can create a Python application. You need to mix-in low level C++ only for the the critical part of the applications which operate with the native platform API.

PyS60 community is vivid

Surf to forum.nokia.com Python discussion boards and you can witness the activity around PyS60. A lot of folks are answering the beginners’ questions and guiding them. If your case is not totally unique, you can expect help to be found. As a matter of fact, you, reading this text, is a sole proof of powerful community. I was not directly paid to write this and I am not affiliated with Nokia any way - this writing flows from the bottom of my heart to promote PyS60 and open some window for the (hireable) talents of our company.

You love it

For some things, we just feel impulsive attachment. They are beautiful, aesthetic, sexy. When you see it, you need to have it. PyS60 is one of these things.

Arguments against Python for Series 60

PyS60 has not been around for long

PyS60 is a young project. It still has some maturity issues. If you don’t do preparations well and test that all needed modules work as you want them to work, you might need to do some extra work to get around the issues.

Especially PyS60 user interface lacks some crucial components and polish which may force you to create UI components yourself.

By “getting around” the issues I mean leaping to the native Symbian code world and writing embedded C++. Since this leap is all but a little step, in the terms of development skills and complexity, I recommend preparing for the worst before the roof is falling on your head. I have seen too much “plz reply me asap urgent my boss kills me this afternoon” PyS60 discussion board messages from persons who haven’t done their homework. For example, you could make sure that you have some hired Symbian guns available in the case your own effort was not enough.

Python is for high-end phones only

PyS60 and PyUIQ are available for the high end smart phones only. Nokia’s Series 40 devices, and other devices built around in-house operating systems, cannot run Python. If you are targetting to e.g. consumers markets and Java ME has all the required capabilities to implement the application, it is the only real way to go.

Symbian is dreadful platform

Symbian, as a software development platform, is very challenging. Due to its history, the feeling of developing for Symbian differs greatly from other mobile, desktop and server platforms. Things simple elsewhere, like installing an application, might become a development nightmare on Symbian where one has to fight his/her way through capability limitations, artificial device restrictions and signing process just to get the application to the phone.

PyS60 naturally inherits all these challenges. Java ME goes around the issue by having its own custom installation, signing, etc. schemes. On Symbian, even accessing some of the device features - a device you own and hold in your hand - requires you to throw in few hundred bucks cash to get the publisher id - to get the developer certificate - to sign the application - to have all required capabilities - to access the specific features (read: GPS). More cash and third party testing is needed to publish the application. This is the price of virus free DRM’ed platform.

Lack of entrepreneurship spirit

If your company states “we are a Java house” or “we are a C++ house” and you miss courage to look for alternatives, then maybe you shouldn’t. Sailing into the brave new world takes more resources than staying near your home harbor. On the other hand, if you are a technology company, it’s quite important follow technologies outside your mainstream, unless you want to suffer the extinction when the new kids on the block gain power (remember the day when COBOL died).

Conclusion

Nokia has done a great deed and effort by creating PyS60 and then giving it out for us to play. Now its own turn to act as a community to give back something we can.

Building commercial grade PyS60 applications might not be straightforward yet, but it is possible. Working with PyS60 is definitely more productive than using complex Symbian C++. It is not always straightforward as Java ME, but eventually it will be. Not only that, but PyS60 can surpass Java ME development with all the Python’s agility.

If you consider PyS60 for your commercial grade product, it all boils down to the feature set of the application, your existing team skill set and the hacker spirit. And now there exists at least one commercial vendor, besides mighty Nokia, who advertises commercial PyS60 services (hint: our company), so you won’t be alone.

We could look things in a larger perspective. In the mobile world, PyS60 is the pack leader of Python-in-your-pocket solutions. Windows CE has Python support available, but it is not officially backed up by Microsoft. This may change quickly when mobile .NET virtual machine gets DLR support making it possible to run IronPython. On Linux platforms, Python is a natural part of the platforms (OLPC, OpenMoko). Google Android and Apple iPhone will roll out their own software development platforms soon.

In the dream world, we could have a common mobile Python platform akin to Java ME of today. Java is not the only way to provide vendor neutral mobile applications: there already exists effort to get mobile cross-platform .NET implementation (Mono on embedded Linux, Red Five Labs .NET on Symbian). The future popularity of Linux phones, iPhone and Android might decide the faith of mobile Python. Could we have Python as a batteries-included feature in the future phones?

Copyright © Red Innovation Ltd. 2008 All Rights Reserved. | Log in | XHTML
Close
E-mail It