How to unit test security declarations in Plone and Zope

Security is hard. Unit testing security in Plone seems to be even harder. Here is a fool proof example how to do it. After comments I plan to release this as plone.org How to. I hope some of these ideas could get into PloneTestCase itself, so there wouldn’t be need to reinvent the wheel on every product.

Since 2004, when I was first introduced to Plone, it has been great mystery to me how to properly unit test your content type and workflow security declarations. Archetypes itself uses ugly hack where it creates secure Python Scripts from strings in Zope and then executes them. There had to be something better, but after asking questions no one seem to know what.

Function security declarations (security.declareProtected & co.) are only effective when Python is run in restricted mode. Entering to “restricted Zope Python” has not been very well documented anywhere, until RestrictedPython package Read me got revamped. This finally gave a clue how to one could hit Unauthorized exceptions in unit testing.

To enter the promised world of sandboxed Python you need to do following

Zope get_safe_globals() will overwrite __getattr__ with guarded_getattr, etc. providing automatic code execution level security. This information is not usable only for unit testing, but for scripting purposes also - it is a developer heaven to be able to give a sandboxed template environment to the users to play around withoutworry that they can escalate privileges.

But getting into restricted mode was not enough…  after that all kind of kinks started to hit me. Namely, in some places of Plone items are cached over the request lifecycle. Since unit tests do not create new requests, the cache will contain invalid values. Here borg.localroles bit me badly - I had to dig through the security management layers manually to see why the unit test code was giving bad results. Maybe it would be wise to have a flag for caches and disable them when running on a test layer?

Below is the my example code for normal Document content type and simple_publication_workflow. All sandboxed code are declared in independend functions, but it is easy to pass arguments for them. If there is no need to reuse the sandboxed functions, I recommend use Python lambda: function declaration.

Functions which should succesfully pass sandbox testing are evaluated using self.execUntrusted(). Functions which are expect to fail are evaluated using self.assertUnauthorized().

import unittest

# Zope security imports
from AccessControl import getSecurityManager
from AccessControl.SecurityManagement import newSecurityManager
from AccessControl.SecurityManagement import noSecurityManager
from AccessControl.SecurityManager import setSecurityPolicy
from AccessControl import ZopeGuards
from AccessControl.ZopeGuards import guarded_getattr, get_safe_globals
from AccessControl.ImplPython import ZopeSecurityPolicy
from AccessControl import Unauthorized

# Restricted Python imports
from RestrictedPython import compile_restricted
from RestrictedPython.Guards import safe_builtins
from RestrictedPython.SafeMapping import SafeMapping

from zope.component import getUtility, getMultiAdapter, getSiteManager
from Products.CMFCore.tests.base.security import UserWithRoles
from Products.CMFCore.WorkflowCore import WorkflowException
from Products.CMFCore.utils import getToolByName

__docformat__ = "epytext"
__author__ = "Mikko Ohtamaa <mikko@redinnovation.com>"
__license__ = "BSD"

class WorkflowTestCase(PloneTestCase):
    """ Test workflow access rights. """

    def afterSetUp(self):
        self.workflow = getToolByName(self.portal, 'portal_workflow')
        self.acl_users = getToolByName(self.portal, 'acl_users')
        self.types = getToolByName(self.portal, 'portal_types')
        self.registration =  getToolByName(self.portal, 'portal_registration')
        self.membership =  getToolByName(self.portal, 'portal_membership') 

        # Create a normal registered portal member
        # to be used in tests
        self.registration.addMember("testmember", "secret", ["Member",], properties={ 'username': "testmember", 'email' : "foobar@foobar.com" })

        # Set verbose security policy, making debugging Unauthorized
        # exceptions great deal easier in unit tests
        setSecurityPolicy(ZopeSecurityPolicy(verbose=True))

    def clearLocalRolesCache(self):
        """ Clear borg.localroles cache.

        borg.localroles check role implementation caches user/request combinations.
        If we edit the roles for a user we need to clear this cache,
        """
        from zope.annotation.interfaces import IAnnotations
        ann = IAnnotations(self.app.REQUEST)
        for key in ann.keys():
            del ann[key]

    def loginAsPortalMember(self, id):
        ''' Login as a normal portal member.

        @param id. username
        '''
        self.login(id)

    def _execUntrusted(self, debug, func, *args, **kwargs):
        """ Sets up a sandboxed Python environment with Zope security in place.

        Calls func() in an sandboxed environment. The security mechanism
        should catch all unauthorized function calls (declared
        with a class SecurityManager).

        Security is effective only inside the function itself -
        The function security declarations themselves are ignored.

        @param func: Function object
        @param args: Parameters delivered to func
        @param kwargs: Parameters delivered to func
        @param debug: If True, break into pdb debugger just before evaluation
        @return: Function return value
        """

        # Create global variable environment for the sandbox
        globals = get_safe_globals()
        globals['__builtins__'] = safe_builtins
        globals['_getattr_'] = guarded_getattr

        # Create variable context available in the restricted Python
        data = { "func" : func,
                "args" : ZopeGuards.SafeIter(args),
                "kwargs" : kwargs } # TODO: Do we need to map this to SafeMappings?

        globals.update(data)

        # Our magic code
        body = """func(*args, **kwargs)"""

        # The following will replace all function calls
        # in the code with Zope call guard proxies
        code = compile_restricted(body, "<string>", "eval")

        # Here is a good place to break in
        # if you need to do some ugly permission debugging
        if debug:
            import pdb
            pdb.set_trace()

        return eval(code, globals)

    def execUntrusted(self, func, *args, **kwargs):
        """ Sets up a sandboxed Python environment with Zope security in place. """
        return self._execUntrusted(False, func, *args, **kwargs)

    def execUntrustedDebug(self, func, *args, **kwargs):
        """ Sets up a sandboxed Python debug environment with Zope security in place. """
        return self._execUntrusted(True, func, *args, **kwargs)

    def assertUnauthorized(self, func, *args, **kwargs):
        """ Check that calling func with currently effective roles will raise Unauthroized error. """
        try:
            self.execUntrusted(func, *args, **kwargs)
        except Unauthorized, e:
            return

        raise AssertionError, 'Unauthorized exception was expected'

    def test_document_workflow_access(self):
        """ Check that anonymous users cannot access diagnosis in unwanted state. """

        def check_set_access(doc, text="foobar"):
            """ This is executed as RestrictedPython, print might not be available """

            # Try do a call which should hit Zope and Archetypes field security mechanisms
            doc.setText(text)

        def check_read_access(doc):
            """ This is executed as RestrictedPython, print might not be available """

            # Try do a call which should hit Zope and Archetypes field security mechanisms
            return doc.getText()

        def check_workflow_action(portal, action):
            """ Publish the document.

            Stresses secure workflow execution
            """
            portal.portal_workflow.doActionFor(portal.doc, action)

        # Login as a manager and create
        # an item which is initially private page to play around with
        self.loginAsPortalOwner()
        self.portal.invokeFactory("Document", "doc")
        doc = self.portal.doc
        # Item is private by default and editably by creator
        self.execUntrusted(check_set_access, doc)
        self.logout()

        # Anonymous cannot access the document when it's private
        self.assertUnauthorized(check_read_access, doc)
        self.assertUnauthorized(check_set_access, doc)       

        # Relogin as a normal member and see we cannot access the item
        self.loginAsPortalMember("testmember")
        self.assertUnauthorized(check_set_access, doc)
        self.logout()

        # Now relogin as the manager and share manager role with a member
        self.loginAsPortalOwner()
        self.membership.setLocalRoles(obj=doc,
                      member_ids=["testmember"],
                      member_role="Owner",
                      reindex=True)
        # IMPORTANT: This is a very invisible feature of Plone 3.1 -
        # setLocalRoles is ineffective in unit tests unless the cache is cleared
        self.clearLocalRolesCache()
        self.logout()

        # Relogin as a normal member and now we should be able to edit the document
        self.loginAsPortalMember("testmember")
        doc = self.portal.doc
        # Rich text is automatically paragraphed unless it
        # begins with HTML element
        self.assertEqual(self.execUntrusted(check_read_access, doc), "<p>foobar</p>")
        self.execUntrusted(check_set_access, doc)
        self.execUntrusted(check_workflow_action, self.portal, "submit")
        # Only site manager can publish items
        try:
            self.execUntrusted(check_workflow_action, self.portal, "publish")
            raise AssertionError("Publishing as normal member should not be possible")
        except WorkflowException:
            # WorkflowException: No workflow provides the '${action_id}' action.
            pass
        self.logout()           

        # Now the portal owner publishes the document
        self.loginAsPortalOwner()
        self.execUntrusted(check_workflow_action, self.portal, "publish")
        self.logout()

        # Anonymous should now have read access/no edit
        self.execUntrusted(check_set_access, doc)
        self.assertEqual(self.execUntrusted(check_read_access, doc), "<p>foobar</p>")      

        # Member should be still able to read and edit the document
        self.loginAsPortalMember("testmember")
        self.assertEqual(self.execUntrusted(check_read_access, doc), "<p>foobar</p>")
        self.logout()

def test_suite():
    suite = unittest.TestSuite()
    suite.addTest(unittest.makeSuite(WorkflowTestCase))
    return suite

Mysterious buildout error - missing docs/HISTORY.txt file

I was getting the following error with Plone buildout

…

Develop: '/home/moo/workspace/collective.easytemplate'
Traceback (most recent call last):
  File "/tmp/tmp_G8621", line 11, in ?
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 655, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 931, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.4/site-packages/setuptools/command/easy_install.py", line 919, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 26, in run_setup
    DirectorySandbox(setup_dir).run(
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 63, in run
    return func()
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 29, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 9, in ?
    return open(os.path.join(os.path.dirname(__file__), *rnames)).read()
  File "/usr/lib/python2.4/site-packages/setuptools/sandbox.py", line 166, in _open
    return _open(path,mode,*args,**kw)
IOError: [Errno 2] No such file or directory: 'docs/HISTORY.txt'
An internal error occured due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 1477, in main
    getattr(buildout, command)(args)
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 324, in install
    installed_develop_eggs = self._develop()
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/buildout.py", line 556, in _develop
    zc.buildout.easy_install.develop(setup, dest)
  File "/home/moo/workspace/Plone-3.1/eggs/zc.buildout-1.1.1-py2.4.egg/zc/buildout/easy_install.py", line 868, in develop
    assert os.spawnl(os.P_WAIT, executable, _safe_arg (executable), *args) == 0
AssertionError

My product had docs folder. HISTORY.txt was there properly. This made me scratch my head for a while.

Buildout calls easy_install as an external process. If easy_install eggs have dependencies in their setup.py easy_install tries to download and install these eggs.

There is no reported progress what eggs are installed in easy_install process created from buildout. Looks like buildout verbosity (-v) switch does not reach easy_install.

So the problem was not in my product, but in its dependency. However the debug output did not reveal that we were dealing with a dependency. Is there easy means to solve this kind of problems? I bluntly put debug prints inside my server wide setuptools Python files to known which was the faulty dependency.

It turned out that easy_install was trying to execute setup.py against a downloaded source distribution (.tar.gz). I had the same egg as a local source code copy. The source code contains docs folder, the egg doesn’t.

The solution was to change buildout.cfg develop directive to be the same as the flattened dependency order of the eggs (dependencies come top). This way setup.py was evaluated correctly against the source code folder.

planetmobile.us up and running

planetmobile.us is a blog aggregator focused on mobile software development.The purpose of the aggregator is to give all good blogs in one packet, without the need to hunt them individually.

planetmobile.us was managed by Christopher Schmidt until I resurrected it few weeks ago.

Please feel free to subscribe your feed to the planet!

MySQL bind_address workaround

MySQL has an ugly design fault preventing it to listen more than one interface in its bind_address my.conf directive. Thus, you usually cannot connect to the same MySQL instance using localhost and external IP sources.

Here is a workaround based on xinetd daemon. These are sample commands for Ubuntu/Debian.

Go to root

sudo -i

Install xinetd

apt-get install xinetd

Add a new xinetd mapping

pico /etc/xinetd.d/mysql

service mysql
{
    only_from	   = localhost mansikki.redinnovation.com 80.75.108.108 server213-171-218-5.livedns.org.uk 213.171.218.5
    flags          = REUSE
    socket_type    = stream
    wait           = no
    user           = root
    redirect       = 127.0.0.1 3306
    log_on_failure += USERID
    interface 	   = 84.34.147.68
}

Restart xinetd

/etc/init.d/xinetd restart

To debug xinetd:

/etc/init.d/xinetd stop
xinetd -d

xinetd only_from directive also gives an access control by allowed source IP addresses. This protects your MySQL against bots and brute force attacks.

Note that iptables DNAT translation doesn’t work (easily). Localhost packets don’t travel PREROUTING and POSTROUTING chains.

How to encode h264 video files for Nokia Series 60 standalone playback

Bored with Spiderman 3 which came with your Nokia N95 8 GB? This guide shortly tells how to get movies into your N95 on Ubuntu Linux using ffmpeg video encoder. The aim is to encode video suitable for playback from Nokia N-series (N95, N78, others) mobile phone memory card. We use h264 + AAC codecs which provides the best quality/compression rate for Nokia phones currently.

Ubuntu does not distribute proprietary codes. First thing you need to do is to rebuild ffmpeg.  Since Ubuntu 8.04 Hardy Heron ships with ffmpeg from 2007, which is aeons old in video codec years, you need to build libx264 and ffmpeg from SVN sources. Here are detailed, valid, instructions. Note that FFMPEG trunk is not currently stable (September 2008), so you need to use revision 15261 which needs this little patch. Indeed, this is a very difficult month to start your career in the dark world of video encoders.

To make it legal and support open source codec development,  please pay for your codecs.

Then we use this guide by Robert Swain. We have a tiny sub 2,4″ screen, we do not care about the quality and do one pass encoding. By empirical research, I have found that the following MPEG-4 profile parameters are compatible with N95 8 GB and provide the optimal result. You can vary video and audio bitrate depending on your taste.

Here is a script which recursivelu encodes all detected video files suitable for mobile format:

#!/bin/sh
#
# Optimal movie encoding for Nokia N-series mobile phones
#
# Copyright 2008 Red Innovation Ltd.
#
# Say hi if you find this useful.
# We do some professional mobile video publishing, so if you
# need a helping hand please call us.
#
# Usage: Run encode.sh in any folder and all video files are recursively converted to mobile phone suitable format
#
# Note: We expect all the source material be in 16:9 aspect ration
#
# Also see http://www.nseries.com/index.html#l=support,search,faq,general,video%20encoding,53848
#

VIDEO_BITRATE=300k

AUDIO_BITRATE=72k

# Assume locally build ffmpeg + x264 in /usr/local/bin
# http://ubuntuforums.org/showthread.php?t=786095
export LD_LIBRARY_PATH=/usr/local/lib

# Search all source AVI, MPG and WMV video files
# Place all encoded files to the same folder with the source, with added .mp4 extension
find . -iname "*.avi" -or -iname "*.wmv" -or -iname "*.mpg" | while read src ; do
        srcfile=`basename "$src"`
	srcfolder=`dirname "$src"`
	dstfile="$srcfolder"/"$srcfile".mp4

	# The magical string!
	# Size and cropping is for 16:9 source material, so that 320:240 display will have black bars.
	# Fex pixels off... note that h264 sizes must be multiplies of 16, use 256x144 for streaming
	# N95 RealMedia player does not seem to respect MPEG-4 embedded aspect ration info.
	/usr/local/bin/ffmpeg -y -i "$srcfile" -acodec libfaac -ab $AUDIO_BITRATE -s 320x176 -aspect 16:9 -vcodec libx264 -b $VIDEO_BITRATE -qcomp 0.6 -qmin 16 -qmax 51 -qdiff 4 -flags +loop -cmp +chroma -subq 7 -refs 6 -g 250 -keyint_min 25 -rc_eq 'blurCplx^(1-qComp)' -sc_threshold 40 -me_range 12 -i_qfactor 0.71 -directpred 3 "$dstfile"

done

Designing a high usability Plone theme

This is my brain dump of instructions for artists how to design good Plone themes. I hope I can receive some comments, some feedback from the artists itself and then publish this as a plone.org tutorial.

Often external artist is used to design a site theme. Artists might or might not have seen Plone, artists might or might not have any basic usability know how. This article should explain the elements which “must be there” to make a match between the theme and Plone easily.

The basic layout

This document describes the elements of multilingual high usability Plone theme. It is based on fluid div layout, meaning that the content stays very readable on small screens or when CSS is not loaded (screen readers). See the example layout.

The layout must not break down when user is using non-default font size. E.g. all element accept two rows text, even if the default case is usually one row.

Plone layout

Here we are designing a “normal” site theme where Plone is used to publish textual content for external readers. This might not always be the case - for example if Plone is used as a professional tool one might want to use all available screen space to display as much as possible action shortcuts to make the tool to quick to use. The latter is actual case I have seen in medical applications.

Plone layout is formed by seven main parts.

Left body padding (auto width)

Header (780 – 1280 pixels)

Left portlets Content (780 – 1000 pixels) Right portlets
Footer (780 – 1280 pixels)
Right body padding (auto width)

The layout must be designed so that

Alternative layout cases

The layout must be formed from such a blocks that left or right portlets can be easily dropped without breaking the layout.

Right portlets missing:

Header

Left portlets Content
Footer

Both portlets missing (front page view):

Header

Content
Footer

Header elements

The header should have the following elements

The header must scale between 780 – 1280 px. Section navigation tabs may trigger drop down menus, see http://www.jyu.fi/

Content area elements

The content area contains

The whole portlet section can be dropped, making space for the content.

Portlets

Portlets are boxes on the left and right side of the content containing section specific actions.

Example portlets:

The portlet consists of

See http://www.siggraph.org/ for creative use of portlets.

Footer elements

The footer has

The footer must scale between 780 – 1280 px.

Complete picture

Left body padding

  • Texture
  • Drop shadow

  • Logo
  • Search box
  • Site actions
  • Language selector
  • Section tabs
  • Personal bar

  • Navigation
  • Breadcrumbs
  • Content actions
  • Notification messages
  • Title
  • Description
  • Text body /images
  • Calendar
  • News
  • Recently added
  • Copyright
  • Link to about page
  • Contact information
Right body padding

Special cases

These are often required and might shoot you into a foot

Colors

Plone uses mechanism to have color variables in CSS. See base_properties.prop to get an idea what colors there are and try to guess how they are used.

Icons

Plone uses generic icon mechanism to apply icons to any action available on site. Icons are 16×16, transparent with one pixel transparent border leaving 14×14 pixels for the content.

The icons should preferably be suitable for dark and light backgrounds. This might be hard to achieve, though, so it is suitable to use ligh background icons, since this is the Plone default.

Actions

Language flags

Plone default flags can be used

Content icons

These reflect different Plone content types

Link icons

Plone uses Javascript to apply special icons for external links

Favorite icon

Introducing Python for Series 60 Community Edition

This blog post will introduce Python for Series 60 Community Edition.

Pythor for Series 60 Community Edition is a new open source effort to push Python for high quality mobile phone development. It aims to provide a maintained software stack for creating real mobile applications. The codebase is derived from the original Nokia’s Python for Series 60 codebase, but has been refactored for better integration with third party extensions and patches and commercial grade application deployment.

Motivation

Building and distribution

It is difficult to distribute Python for Series 60 applications to the end user with the current Nokia’s PyS60 distribution. You probably want to modify or extend PyS60 in some way. Since the build chain and deployment model is not designed for changes this would collide with the other PyS60 installations. Symbian Platform security prevents installing conflicting binaries. Thus, one can effectively have one Nokia PyS60 application in the phone once.

There are other problems: Nokia PyS60 distribution has UIDs in Nokia protected range. Embedded SIS file cause extra installation dialog and an application manager uninstall entry. Trimming down Nokia PyS60 distribution is a little bit difficult.

To overcome all these issues we created a build chain which spits out monolithic PyS60 distributions. We build only one DLL whose name and UID can be decided. Also the build chain is Scons for Symbian, scrapping the obscure, inflexible and difficult to understand Symbian ABLD once for all.

Evolution towards higher quality

Currently there is no centralized authority to co-ordinate PyS60 open source developers and maintain the repository of all the extension and patches. This effectively prevents the biggest benefit of open source: open innovation and gradual evolution of the product. It would be very nice having all those third party extensions, now scattered around the internet, under one maintained source - making PyS60 more functional out of the box.

The community maintained repositories do not have the same restrictions as ones managed by a big public corporations. It is not a probable target of a trigger happy lawyer action and ungentlemanly competition: the discussion and plans can be public and due dilugence check of the code more relax.

We started the project in the Launchpad. Launchpad provide a distributed version control system (Bazaar) which streamlines the process of integrating third party commits and patches. This should encourage contribution. The standard build system makes it easy to roll out applications and extensions from bare  C++ source up to the end user distributable SIS files.

It is yet to see what kind of co-operation possibilities between the community and Nokia exists. In the future, it should be possible to cherry pick patches from PyS60 community edition to Nokia’s own version.

Showing the commercial potential of PyS60 in the mobile application development

On Python you can write native Series 60 applications with very little effort compared to hardcore C++ banging, lowering the barries to enter the mobile application development.

We do not deny that we have an extrinct motivation called money. Of course we have also instrict motivations like thinking Python is the best programming language in the world and we all want to be most respectable gurus in it. Gurus need to eat still, though. We hope that our effort does not go unnoticed in the mobile application development world and good subcontract offers fill our inboxes.

Also, there is the John McClane effect. Unless we had done it, no one had. Somebody has to save the world, despite the hangover.

It runs on Linux

Since we are no longer dependend on .BAT/Perl/Windows hindered ABLD buildchain, we can (almost) crosscompile and build native Symbian binaries in Linux and  Python applications. All good hackers use Linux - but currently there are still kinks and you need to use WINE for some parts - all sane Symbian developers are tied to Windows based tools for now and so are these instructions.

Prerequisitements

You need all this stuff to get things running.

Install Bazaar

You need Bazaar distributed version control client. We are not planning to have fixed releases for Python for Series 60 community edition any time soon. This is because 1) the most magic happens at a compiler level and we are providing a buildchain 2) we hope this fosters incoming patches.

Why Bazaar?

Install ActiveState Perl

Series 60 SDK has ActiveState as a prerequisitement for running its installer.

http://www.activestate.com/Products/activeperl/index.mhtml

Install Series 60 SDK

Use only Series 60 3.0 maintenance release. Other releases have SDK bugs preventing correct Python compilation.

Get the Windows installer from http://forum.nokia.com. Forum Nokia Registration is required.

Please use the default installation location C:\Symbian\9.1\S603rd_MR.

Install Carbide.c++ express

Carbide.c++ comes with a Windows compiler to compile the emulator binaries. You need this only if you indend to develop and test your applications on Series 60 emulator.

http://www.forum.nokia.com/info/sw.nokia.com/id/dbb8841d-832c-43a6-be13-f78119a2b4cb.html

Forum Nokia Registration is required.

Use Software updater in Carbide.c++ to install PyDev, Python developer extensions for Eclipse.

http://pydev.sourceforge.net/

Install Python 2.5

Scons build chain and our utility scripts use Python.

http://www.python.org/download/releases/2.5.2/

Use the installer EXE and the default installation location C:\Python25. If you want to use advanced Bluetooth shell (PUTools) you also need wxPython and pyserial packages.

Install SCons

Python for Series 60 build script are based on SCons. It is a build system using Python as a recipe langauge.

http://sourceforge.net/project/showfiles.php?group_id=30337

Install Subversion

Install Subversion client for Windows. This is needed for checking out Scons for Series 60.

http://www.collab.net/downloads/subversion/

Registration to CollabNet is needed to download Windows binaries.

Scons for Series 60

SCons for Series 60 is available as a separate project. SCons for Symbian is a build toolchain intended as a replacement for Perl and MMP files used on regular Symbian projects. SCons for Symbian is not limited to build Python - You can use it to build any Series 60 C++ application.

http://code.google.com/p/scons-for-symbian/

This is later checked out during to the environment construction, so you do not need to install it now.

We have included a workaround for a problem with limited command line length on Windows.

Included tools

The following tools are included in the trunk tools folder:

These tools are not licensed under Apache license. Some of them are under GPL license. However, we believe that distribution them is ok, since this falls under GPL’s mere aggregation clause. However if you indent to distribute commercial applications built from PyS60 Community codebase, make sure that you understand the set of different licenses involved.

Set up build environment

Ensure that Bazaar is properly in your Windows path.

Create a workspace folder

First you need to subst (make a folder appear as a driver letter) in Windows. Open command line. Go to SDK folder.

C:\Symbian\9.1>subst t: S60_3rd_MR
T:
mkdir workspace

Now choose this folder as a workspace folder in Carbide C++ and create an Empty Symbian C++ project called ”pys60” there.

Checkout PyS60 community edition

The go to this folder

T:
cd workspace\pys60
bzr branch lp:pys60community
cd pys60community\src

Preparing the build

This needs to be done only once.

We need to pacth the existing Series 60 SDK headers which have some bugs.

T:
cd \epoc32\include
\workspace\pys60\pys60community\src\tools\patch.exe -p1 < \workspace\pys60\pys60community\src\pys60-fix-3rded-sdk.diff

EPOCROOT must be set for some Series 60 SDK tools to work. We point to T: drive root.

T:
cd workspace\pys60\src
set EPOCROOT=\

As we still have some dependencies to the legacy system, one needs to configure the build system using PyS60 setup. This will generate some files and defines for Series 60 versio 3.0. 

c:\Python25\python.exe setup.py configure 30

Do not run bldmake bldfiles.

You need to convert legacy MMP build files to SCons based. First we need to possibly fix up PATH, since Carbide C++ might break it.

set PATH=c:\program files\bazaar;c:\program files\CSL Arm Toolchain\arm-none-symbian elf\bin;c:\program files\CSL Arm Toolchain\libexec\gcc\arm-none-symbianelf\3.4.3;C:\program files\CSL Arm Toolchain\bin;t:\epoc32\gcc\bin;t:\epoc32\tools;t:\epo c32\tools;C:\program files\CSL Arm Toolchain\bin;C:\Program Files\Common Files\Symbian\Tools;C:\Perl\site\bin;C:\Perl\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem

c:\Python25\python tools\mmp2scons.py
ext\\calendar\\calendar.mmp:60:7: warning: no newline at end of file
Creating recipe ext\miso\build.py
Creating recipe ext\socket\build.py
...
Done!

Checkout SCons for Symbian. We assume it lives in src tree.

"c:\Program Files\CollabNet Subversion"\svn checkout http://scons-for-symbian.googlecode.com/svn/trunk/ scons_symbian

Running the build

Now we can execute the actual Python build script. This will create one monolithic emulator DLL which has almost all the PyS60 extensions built in - some extensions need manual building, since they rely on headers not found from standard Series 60 SDK.  You might need to reset PATH to default to Carbide C++ after the previous mmp2cons step by reopening the console window.

c:\Python25\Scripts\scons

You should see the following output. As you can see, UIDs are being allocated dynamically as instructed in Scontruct UID_BASE argument.

scons: Reading SConscript files ...
EPOCROOT=\
Info: SIS creation disabled
Building winscw udeb
Defines []
Getting dependencies for e32socket.pyd
Getting dependencies for _topwindow.pyd
Getting dependencies for zlib.pyd
Getting dependencies for _locationacq.pyd
Getting dependencies for _location.pyd
Getting dependencies for _graphics.pyd
Getting dependencies for _sysinfo.pyd
Getting dependencies for Python222Config.lib
Getting dependencies for Python222.dll
Getting dependencies for Python_appui.dll
Allocated UID:0xE1000000
Getting dependencies for Python.exe
Allocated UID:0xE1000001
Getting dependencies for Python_launcher.exe
Allocated UID:0xE1000002
scons: done reading SConscript files.
scons: Building targets ...

...

scons: done building targets.

Now you should be able to start a custom built Python shell in the emulator. You should see the following applications in Installation menu: helloworld, btconsole and filebrowser. Try launch helloworld and if it opens a pop up dialog the build has been succesful.

Building a SIS file for mobile phones

To build a target distribution type:

scons release=urel compiler=gcce dosis=true

This should yield to the result:

scons: Building targets ...
ensymble(["MyPythonForSymbian.sis"], [])
scons: warning: no package version given, using 1.0.0
scons: warning: no certificate given, using insecure built-in one
scons: done building targets.

If you want to build a SIS file signed with your developer certificate:

c:\python25\scripts\scons compiler=gcce release=urel dosis=true simplesis={'--privkey':'C:\\Certificates\\PrivateKeyNoPassphrase.pem','--cert':'C:\\Certificates\\MyApp.cer','--passphrase':''}

For now, installing the SIS file works only for C drive  - we’ll fix this little issue soon.

If you do not have a symbiansigned.com developer certificate you can sign the SIS file online for one phone (one IMEI code).

  1. Go to symbiansigned.com
  2. Register
  3. Enter your IMEI and upload the SIS file to OpenSigned Beta

Building your own application

The purpose of this project is to make rolling out your PyS60 applications possible - so here we go. Currently we make a quite bad assumption that all the application live in the same source tree with PyS60 community edition due to problems with absolute file paths with Symbian build tools - we will figure out a long term solution for this later.

PyS60 applications are stub Symbian executables which boostrap Python virtual machine and start the Python code execution. Executables are linked against a custom PyS60 DLL and they are restricted by capabilities given to the EXE file.

PyS60 applications live in applications source tree. The source tree comes with Bluetooth console, Hello world and File browser sample applications.

Scons build scripts takes as applications parameter a comma separated list which applications are included in the build.

scons builtin=all applications=helloworld,filebrowser

Applications consist of

See applications\helloworld folder to examine what files are needed to build an application. All application Python modules go to the private application folder (/private/myapplicationuid). Default.py must boostrap PYTHONPATH (sys.path) for this folder - PYTHONPATH defines where Python interpreter looks for the code. Application UIDs can be chosen manually or they are picked automatically by Scons for the unprotected test range.

Note that Python Script Shell application is handled out of this flow due to its legacy heritage.

Adding in your own extension

If you have development an PyS60 extension you can drop in into the buildchain easily. Each extension is defined in ext subfolder. It consists of necessary CPP, H and Python files. The building structure is defined in build.py using SConstruct command PyS60Extension().

Build.py files can be automatically generated from legacy code using mmp2scons.py converter.

c:\Python25\python.exe tools\mmp2scons.py

ext\calendar\calendar.mmp:60:7: warning: no newline at end of file
ext\progressnotes\progressnotes.mmp:38:7: warning: no newline at end of file
ext\uikludges\uikludges.mmp:37:7: warning: no newline at end of file
Creating recipe ext\\socket\build.py
Creating recipe ext\\glcanvas\build.py
Creating recipe ext\\graphics\build.py
...
...Done!
c:\Python25\scripts\scons
>> import applicationmanager

If your extension is using thread local storage (Dll::Tls()) you might need to figure out how to workaround with it. See socket  module for example. You may also need to play around with the init function of the Python extension - it must be init + module name.

Developing on target

If you want to develop your application on a mobile phone, you do not need to go through the full development cycle for every little change. It is possible to update Python files on a phone without SIS installation. You can either automatically synchronize changed files from your PC to Phone (the example below) or you can edit files in-place on the Phone either using PCSuite or Series 60 SMB server.

Here are short instructions how to update files using PUTools console (btconsole). PUTools is wxPython based remote Python shell which allows you to run Python console commands over a Bluetooth connection from your PC. PUTools also has a file syncrhonization feature - after editing source code on the PC changes are reflected automatically to the phone.

  1. Put application Python files to a shared folder on the phone e.g. the memory card root (E:\). Add startup.py to E:\ which will modify sys.path to include your files.
  2. Add incoming Bluetooth serial port on windows (Control panel -> Bluetooth -> COM ports)
  3. Start Bluetooth shell on the computer (tools\putools\pcfiles\console.bat)
    4. T:\workspace\pys60\pys60community\src\tools\putools\pcfiles>c:\python25\python.exe putools.py com5
  4. Start Bluetooth shell on the phone (btconsole icon)
  5. Edit PUTools sync.config file on PC and run sync command on Bluetooth terminal to update changes made on PC to phone
  6. Run application launcher in the console (depends on the application structure how it is best to bootstrap in the shell):
>> import filebrowser
>> filebrowser.FileBrowser.run()

Release notes

Here is the short summary of differences with the current PyS60 community edition and one available from Nokia. This information is also available in divergence.txt file in the source folder.

2008-08-29 Mikko Ohtamaa <mikko@redinnovation.com>

    * PyS60 general

        New build chain and static config generation

        Migration tool for MMP -> Scons based extensions

        Added several tools included in the core distribution: sisinfo, ensymble, cog, patch

        Patched py2sis tool

        Contains extension: applicationmanager

        Contains extension: uikludges

        Contains extension: progressnotes

        Contains extension: miso

        Contains application: Bluetooth shell

        Contains example applications: filebrowser, helloworld

        Changed Bluetooth console bootstrap to e:\startup.py         

2008-08-15 Antti Haapala <antti@redinnovation.com>

    * e32socketmodule.cpp:

        socket.access_points has more information, two new
        fields is given per access point: isptype and bearertype,
        whose values are integers corresponding to values returned
        by CApSelect::Type and CApSelect::BearerType respectively.
        No symbolic constants are yet exported.

    * appuifwmodule.cpp:

        multi_select_list has a new argument, selected, which defaults
        to None. Given a list of integers, the items with the given indices
        are initially selected.

Conclusion

We hope this helps you to get started with PyS60 community edition. It’s still a bit complicated, since setting up the build environment on Windows is a such a pain. In the future, when the Linux based build system is reading settings up the development environment should be easier - all those boring steps happen automatically.

This might be still too difficult for some of the readers, since a lot of prerequirement work must be done before anything useful can be done. Feel free to comment the article in this blog, but we hope that you use Answers section in Launchpad to ask help and technical questions related to PyS60 community edition.

Catching silent Javascript exceptions with a function decorator

The one utterly annoying thing with the otherwise excellent Firefox/Firebug combo is that some exceptions are let silently through without being end up to be visible in the Firebug console. This makes debugging very difficult, unless you are aware of the phenomenon. I am not sure whether this is caused by some internal Firefox logic flow, since IE + Visual Web Developer doesn’t seem to be affected by this.

Since this problem pops up constantly, I decided to create an easy way to deal with the situation. I decorate all functions which made have silent exceptions (e.g. one called from document load event) with a custom logger function which will first log the exception and then rethrow it.

Thus instead of writing (JQuery example)

myfunction() {
    // crash here
    var i = foobar; // missing variable foobar
}

$(document).ready(myfunction);

write

myfunction() {
    // crash here
    var i = foobar; // missing variable foobar
}

$document.ready(logExceptions(myFunction));

or

// myFunction can be bind to many events and exceptions are logged always
myfunction = logExceptions(function() {
  // crash here
    var i = foobar; // missing variable foobar
});

$document.ready(logExceptions(myFunction));

The Javascript code for the decorator:

/**
 * Enhancd Javascript logging and exception handler.
 *
 * Copyright 2008 Red Innovation Ltd.
 *
 * @author Mikko Ohtamaa
 * @license 3-clause BSD
 *
 */

// Browser specific logging output initialization
// Supports Firefox/Firebug. Other (Opera) can be hooked in here.
if(!console) {
	// Install dummy functions, so that logging does not break the code if Firebug is not present
    var console = {};
    console.log = function(msg) {};
    console.info = function(msg) {};
    console.warn = function(msg) {};
} else {
    // console.log provided by Firefox + Firebug
}

/**
 * Try print human digestable exception stack trace to Firebug console.
 *
 * http://developer.mozilla.org/en/docs/Core_JavaScript_1.5_Reference:Global_Objects:Error
 *
 * @param e: Error
 */
function printStackTrace(e) {

	var msg = e.name + ":" + e.message;

	if (e.fileName) {
		msg += " at " + e.fileName + ":" + e.lineNumber;
	}
	console.log(msg);

	if (e.stack) {
		// Extract Firefox stack information. This tells how you ended up
		// to the exception in the first place. I didn't find
		// instructions how to parse this stuff.
		console.log(e.stack);
	}
}

/**
 * Decorate function so that exceptions falling through are printed always.
 *
 * Returns a decorated function which will be used instead of the normal function.
 * The decorated function has preplaced try ... catch block which will not let
 * through any exceptions silently or without logging. Even though there is an
 * exception it is normally throw upwards in the stack after logging.
 *
 * @param func: Javascript function reference
 */
function logExceptions(func) {

	var orignal = func;

	decorated = function() {
		try {
			orignal.apply(this, arguments);
		} catch(exception) {
			printStackTrace(exception);
			throw exception;
		}
	}

	return decorated;
}

Speeding up Plone loading with PTS_LANGUAGES

If you are not a Finnish speaker (like 99,9% of you) you might not want to (re)load Finnish and other unwanted language catalogs during the Plone start up. This is possible for Plone 3.1, as Reinout van Rees explains (found out afterwards).

For your Plone launcher, set environment variables (space separated list)

PTS_LANGUAGES=en mylanguagecodehere

If your Data.fs is not fresh (i.e. you have an existing Plone instance) there is still one task to do. Go to Placess Translation Service in Zope. Delete all translation catalogs. If there exists a translation catalog entry in ZODB a reload event seem to be triggered even though PTS_LANGUAGES settings is effective. Restart Zope. Maybe this is a bug? Do this on a development box only - this code seems to be quite new.
The magic code is in PlacelessTranslationService/load.py.

Facebook requests…

I love you all guys.

I just learnt that Firefox web page screen capture tools (any of them) can’t take web page screenshots higher than short 16-bit interger (32768) pixels. Crash crash crash. But I hope I am alone with my problem.

So, thanks for being so supportive… and it’s not fully rendered (over 250 of them), since after 3200 the image was cut. Excuse me if I am not willing to support your cause.

Facebook requetss

Next Page →

Copyright © Red Innovation Ltd. 2008 All Rights Reserved. | Log in | XHTML
Close
E-mail It