Plone, KSS, Javascript, field validation and the cup of WTF

Knowledge does hurt. Today I had my sweet cup of WTF. We are developing a medical database based on Plone 3.1. It uses very advanced AJAX framework called KSS - basically you can avoid the pain of writing pure Javascript by crafting special CSS like stylesheets which bind server-side Python code to any Javascript event. This makes AJAX programming a joy. You can easily combine server-side logic with user interface events, like field validation.

Well… then there was an error. KSS validation was not working for the text fields on a certain pages…. or it did sometimes. We were not sure. This is so called Heisenbug. I armed myself with sleepy eyes, Firebug and a lot of energy drinks.

I saw a KSS error in the Firebug log window and failed HTTP POST in the server logs.

Invalid request.

The parameter, value, was omitted from the request.

Looks like the field value was not properly posted for the field validation.

The first thing was locate the error and get function traceback for the messy situation. Unfortunately Firefox Javascript engine or Firebug cannot show tracebacks properly… the grass is so much greener on the other side of the fence. So I had to manually search through the codebase by manually plotting console.log() calls here and there.

Finally I thought I pinpointed the cause of the failure. By shaking finger (excitement, tireness and all that extra caffeinen from energy drinks), I opened the Javascript file just to realize why Javascript is utterly utterly shitty and why no sane person wants to do low level Javascript development. If ECMA standard committee had been clever and had been able to enforce anything long time ago, the following piece could be replaced with one function call.

fo.getValueOfFormElement = function(element) {
    // Returns the value of the form element / or null
    // First: update the field in case an editor is lurking
    // in the background
    this.fieldUpdateRegistry.doUpdate(element);
    if (element.disabled) {
        return null;
    }
    // Collect the data
    if (element.selectedIndex != undefined) {
        // handle single selects first
        if(!element.multiple) {
                if (element.selectedIndex < 0) {
                    value="";
                } else {
                    var option = element.options[element.selectedIndex];
                    // on FF and safari, option.value has the value
                    // on IE, option.text needs to be used
                    value = option.value || option.text;
                }
        // Now process selects with the multiple option set
        } else {
            var value = [];
            for(i=0; i<element.options.length; i++) {
                var option = element.options[i];
                if(option.selected) {
                    // on FF and safari, option.value has the value
                    // on IE, option.text needs to be used
                    value.push(option.value || option.text);
                }
            }
        }
    // Safari 3.0.3 no longer has “item”, instead it works
    // with direct array access []. Although other browsers
    // seem to support this as well, we provide checking
    // in both ways. (No idea if item is still needed.)
    } else if (typeof element.length != ‘undefined’ &&
        ((typeof element[0] != ‘undefined’ &&
        element[0].type == “radio”) ||
        (typeof element.item(0) != ‘undefined’ &&
        element.item(0).type == “radio”))) {
        // element really contains a list of input nodes,
        // in this case.
        var radioList = element;
        value = null;
        for (var i=0; i < radioList.length; i++) {
            var radio = radioList[i] || radioList.item(i);
            if (radio.checked) {
                value = radio.value;
            }
        }
    } else if (element.type == “radio” || element.type == “checkbox”) {
        if (element.checked) {
           value = element.value;
        } else {
            value = null;
        }
    } else if ((element.tagName.toLowerCase() == ‘textarea’)
               || (element.tagName.toLowerCase() == ‘input’ &&
                    element.type != ’submit’ && element.type != ‘reset’)
              ) {
        value = element.value;
    } else {
        value = null;
    }
    return value;
};

It turned out that the element in this case was an empty list of radio buttons. When you are tab keying through a radio button group without any value selected, like in the case a content object is just created, KSS validation is triggered even though there is no value in any of the radio buttons. This makes KSS think the value is null and it does not properly handle the situation. This does not cause any user visible effects unless you have Javascript debugging on (Firebug + debugging mode in Plone’s Javascript registry).

But this was not the bug I was looking for. It was just masking the original bug, because I had an empty radio button group next to the text field whose validation was not correctly done. More server side debugging…

I inserted some funky debug prints to Archetypes.Field.validate_validators():

validate_validators()
Calling validators:(('isEmptyNoError', V_SUFFICIENT), ('validDecRange03', V_REQUIRED))

We can see that not triggered validator, validDecRange03, is still with us. Then I add more debug prints to see where things go wrong, this time to to Products.validation.chain.__call__.

Calling validators:(('isEmptyNoError', V_SUFFICIENT), ('validDecRange03', V_REQUIRED))
Name:isEmptyNoError value:234234234234234234234234234232342342342344534232342344234534554 result:True

Ok - we have a case here. isEmptyNoError validator is executed before our custom validator. Since this validator is flagged as “sufficient” other validators are not evaluated. I think this has not been the case before and our validator have worked properly… maybe there was API change in Plone 3.1 which broke the things?

After digging and digging and digging I found this 4 years old bug. Let’s open the famous isEmptyNoError source code in Products.validation.validators.EmptyValidator:

class EmptyValidator:
    __implements__ = IValidator

    def __init__(self, name, title='', description='', showError=True):
        self.name = name
        self.title = title or name
        self.description = description
        self.showError = showError

    def __call__(self, value, *args, **kwargs):
        isEmpty  = kwargs.get('isEmpty', False)
        instance = kwargs.get('instance', None)
        field    = kwargs.get('field', None)

        # XXX: This is a temporary fix. Need to be fixed right for AT 2.0
        #      content_edit / BaseObject.processForm() calls
        #      widget.process_form a second time!
        if instance and field:
            widget  = field.widget
            request = getattr(instance, 'REQUEST', None)
            if request and request.form:
                form   = request.form
                result = widget.process_form(instance, field, form,
                                             empty_marker=_marker,
                                             emptyReturnsMarker=True)
                if result is _marker or result is None:
                    isEmpty = True

        if isEmpty:
            return True
        elif value == '' or value is None:
            return True
        else:
            if getattr(self, 'showError', False):
                return ("Validation failed(%(name)s): '%(value)s' is not empty." %
                       { 'name' : self.name, 'value': value})
            else:
                return False

There is my WTF. Or XXX - thanks for the kisses. My guess is that because KSS validation is executed in special context, the magical REQUEST might not be there. The “is sufficient” validator fails because of the some sort of god forgotten magic and thus all custom validators fail in KSS when the field is not required.

The workaround: I add my own greetings to the code:

        # XXX: This is a temporary fix. Need to be fixed right for AT 2.0
        #      content_edit / BaseObject.processForm() calls
        #      widget.process_form a second time!
        if instance and field:
            widget  = field.widget
            request = getattr(instance, 'REQUEST', None)

            # XXX: Whatever this all does, it does not work for KSS validation
            # requests and we should ignore this
            if "fieldname" in request:
              return False

            if request and request.form:
                form   = request.form
                result = widget.process_form(instance, field, form,
                                             empty_marker=_marker,
                                             emptyReturnsMarker=True)
                if result is _marker or result is None:
                    isEmpty = True

If Zope 3 drives you smoking Plone 3 drives me drinking. No wonder newbies steer away from Plone - if you hadn’t been armed with 8 years of web development experience you would never have figured out what’s going on with such a simple thing as adding a custom validator. A comment added to the bug tracker.

Eclipse web developer plug-in memo

Currently I work in quite wide field of software development: Python (standalone, Plone, Zope, Django), PHP, Java, Symbian and embedded Linux. I am using Eclipse for development, since it’s pretty much the only consistent platform filling my needs. The nature of work also forces me to use different computers (Mac/Windows/Linux) with different clients. This drives me to reinstall Eclipse now and then.

Below are my personal notes what plug-ins are needed to get “perfect” Eclipse set-up. Basically they are just my own notes so that I don’t need to Google everything all over again every time I reinstall. I hope the readers can find new pearls here or suggest improvements.

Eclipse setup

Eclipse has internal updater/web installer. All plug-ins are downloaded as ZIP files and extracted to Eclipse folder or installed through the internal updater. Paste Eclipse update site URLs to menu Help -> Software updates -> Find and Install, New Remote Location. You can use dummy text as the name of update site.

Eclipse WTP (Web Tools Platform)

Eclipse Web Tools Platform bundles Eclipse, Java development tools, HTML editor, CSS editor and some other generic useful stuff.

Python

PyDev is a plug-in for Python and Jython development.

Site URL: http://pydev.sourceforge.net

Eclipse update site URL: http://pydev.sourceforge.net/updates/

PDT

PDT download provides Eclipse, HTML editor, PHP editor and CSS editor.

Site URL: http://www.eclipse.org

Eclipse update site URL: http://download.eclipse.org/tools/pdt/updates/

Subclipse

Subclipse provides Subversion version control integration to Eclipse.

Eclipse update site URL: http://subclipse.tigris.org/update_1.2.x

In the installer, uncheck the integration modules checkbox or the installer will complain about missing modules.

JSEclipse

JSEclipse provides a better editor (over WTP) for Javascript files, with impressive outlining and autofill capabilities.

Download requires Adobe developer account or similar fill-in-the-fields crap.

Site URL: http://labs.adobe.com/technologies/jseclipse/

ShellEd

Syntax coloring for Unix shell scripts

Project site: http://sourceforge.net/projects/shelled

SQL Explorer

SQL editor with limited GUI capabilities. Based on Eclipse platform. Comes standalone and as Eclipse plug-in.

needs MySQL JDBC driver

Technorati tags:

Could it be possible to make viewlets simple again?

Viewlets were introduced in Plone 2.5 and they are extensively utilized in Plone 3.0. Viewlets are controlling the snippets of web pages and you must understand them if you wish to add new Plone UI elements or create new Plone themes. This new technology promises to separate Python based logic from page templates, thus making HTML authoring a cleaner process. On the other hand, it introduces new elements to Plone technology soup, further bumping already steep Plone learning curve. Could we avoid this learning bump and keeping simple changes simple.

I present not-so-rare scenario where a newbie wants to add a new piece of HTML code to every Plone page. I use a custom language bar as an example: instead of using the normal language selector, provided by Plone or LinguaPlone extension, our newbie developr wants to style a language bar suitable to his/her site theme and have it in non-standard location on the page.

Old way

In Plone 2.5, much of the Plone HTML output were controlled by two things

To add a new item to the portal header section, let’s call it a custom language bar, you had to do the following.

Voila! 7 steps. One had to understand HTML, CSS, Firebug, TAL and little bit Python now and then. This is nothing too difficult for an average Joe.

Plone 3.0 way

Plone 3.0 has a system with containers and viewlets. All viewlets are bind to a certain container, in which they can shuffled and hide (even through-the-web!) .

Pros:

Cons:

Below are steps what we need to create the imaginary custom language bar in Plone 3.0. This is not the exact recipe, but I guess you all get the point along the lines…

Phew. In this point we have at least 2 python modules, 1 page template file, 2 different XML dialects, 150 lines of Python code. You need to master following technologies: HTML, CSS, TAL, @@manage-viewlets, object-oriented Python, Zope Component Architecture, ZCML and GenericProfile XML. The five latter ones don’t actually have anything to do with the job, changing a bit of HTML code. Basically this means that anyone who wants to pimp his/her Plone must have PhD in software architectures.

Why would you need to be Enterprise Software Architecture to make a simple HTML change? I am afraid that the new Plone way of HTML authoring will drive people futher away from Plone, because they just can’t do it(tm) anymore. Plone is already enjoying bad ass learning curve. Making it even steeper and forcing people to learn things which are not really related to their jobs won’t make Plone more attractive. I am afraid that soon there might be only three people left in this world how actually have the skillset to create new Plone skins (Alexander Limi, David Convent and Denis Mishunov).

Could it be easier?

The bitter comments in viewlets tutorial indicate that not everyone loves the new way. Though we need a system like this to make Plone flexible, somewhere along the way architecture just got little too complex. Maybe we should be a bit more Grok‘ish and try to hide all this complexity from novices. These two principles help in designing a system:

My dream is that to accomplish something you need as few lines and few files as possible. Preferably one file for one thing. No changes in other files needed.

It could be something like this (indent problems below!):

myproduct/browser/viewlets/my_languagebar.pt

<html xmlns="http://www.w3.org/1999/xhtml">
<plone:viewlet="my_languagebar"  to-slot="portal-top"

default-visibility="true"
  default-order="after:logo"


bind-data=”showCurrentLanguage python:True”>


<ul>


<li>Choose Finnish</li>


<li>Choose English</li>

</ul>

</plone:viewlet>


</html>

Explanation: Let’s add viewlet my_languagebar to portal_top container and by default it should be placed after logo viewlet. The HTML code is all there. We override context variable showCurrentLanguage with inline Python script.

Now, let’s unleash the debate dogs and let them bite this blog :)

Technorati tags:

Tuning file system performance for Plone development

I recently read this article about tuning Ext3 file system for better performance. I was doing a fresh Ubuntu 7.10 install on my laptop, so I decided to see how much this would affect to my every day Plone development.

On Linux, every time a file is read, its access time attribute is rewritten. This causes a lot of unnecessary writes to file system. Since there are only few rare application needing this feature, turning of the feature can give a nice performance boost on systems dealing with large amount of files.

Plone 3.0 has 10000 files. A lot of them are read during the start-up. Maybe I am getting somewhere here…

When you are doing Plone development, you need to restart Plone often. I used this highly scientific method to measure Plone start-up time from issuing zopectl fg to getting the front page load completed in Firefox. I warmed the file system cache beforehand by doing two dry runs.

I also did some simple front page bombing with ab tool.

System setup

Out-of-the-box filesystem

Lap 1: 23s

Lap 2: 22s

Lap 3: 22s

ab stats:

Concurrency Level: 10
Time taken for tests: 11.805239 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 2058700 bytes
HTML transferred: 2030600 bytes
Requests per second: 8.47 [#/sec] (mean)

Tuned file system

Lap 1: 21s

Lap 2: 22s

Lap 3: Didn’t bother to do it…

ab stats:

Concurrency Level: 10
Time taken for tests: 12.102054 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 2058700 bytes
HTML transferred: 2030600 bytes
Requests per second: 8.26 [#/sec] (mean)

Conclusion

“Hooray.”

Though Plone/Zope crawls through of thousands of files during the start up (and thus touches their access times), the slow start-up process seem to be CPU bound. Magic file system tricks won’t make your everyday Plone development more effective.

Plone business, part II, conference in Shangai

Let me first frame this blog entry: I was Shangai’ed last night. I picked up this new wonderful term when I was eavesdropping a conversation in a Plone conference. “Did you Shangai’ed last night?” “What do you mean?” “Drink till 5 am.” I must admit that I Shangai’ed quite badly. But you don’t need to worry about me. Friends from my hotel, they were totally Finnish’ed.

So, I am sitting at Fiumicino airport on my way back from a Plone conference, enjoying a well deserved hangover. Since I really don’t have anything I could do, I finally find time for updating my blog. 6-7 months ago I wrote a blog entry about what it feels like starting one’s own software sweatshop. Things have progressed quite nicely and I’d like to give a status update.

First about the conference. Plone conference 2007 (Plone being the best open source content management system out there with a strong business oriented community) was held in Naples, Italy. It was my first big open source gathering - 350 participants were present from all over the world. My home city, Oulu, is a bit off from the international routes and travelling was expensive: my wallet got Finnish’ed too. Still, I believe the marvellous conference was worth every euro.

I finally met one of my US clients face-to-face. They are a nice senior couple, who run a family web business. We had a lot of wonderful discussions, and got some humane depth into our business relationships, which I believe greatly helps getting possible joint ventures done more smoothly in the future.

Getting faces and voices for IRC nicks was a little confusing, in a positive way. You were hit with thoughts like “Oh my, this guy was so tall” or “I expected a younger one”. I also noticed I have started referring the Plone scene as “we” instead of “them” - a clear sign that the community has stolen my soul.

What comes to presentations and workshop sessions, Plone developers are, quoting a sentence from Limi, “- geniuses”. Plone/Zope is currently very hard to master software-development-wise, but this is a recognized problem. People are working on it to make it easier and more like other agile web frameworks out there (Django, Ruby on Rails or TurboGears).

By the way, writing on my Macbook feels like being the Sex and The City main character. 90% of people in the Plone scene are using Macs. I wonder why we don’t ask for sponsorship from Apple, since being in touch with Plone folks is an ultra effective mouth-to-mouth marketing channel for Apple. I didn’t have a Macintosh laptop in the previous Plone event I participated. The social pressure to get one is enormous - I had to order a Macbook just for this trip =)

The business part

Currently, Red Innovation incorporates four employees and one contractor. I am managing things, doing less coding every day. Three of my friends work as flexible part timers until they finish their studies. One freelancer is doing contracting through me for bigger clients. Plone is not the only thing in our boat: we have been doing everything from embedded mobile phone Linux to video and social sites on every known programming language.

In the conference, there were really cool presentations on how to become a Plone consultant and how to work with Plone consultants. I hope I could send the latter presentation video for all of my clients. Two of its main points were to be open and to have a budget for consultants. It’s really hard to work outside the information loop and all feature requests come as granted from the Heaven without explanation of the logic behind them. You can implement the feature, but the result is what the client told you, not what the client was thinking. Small fixes and adjustments follow after the initial implementation when the client clarifies his/her wishes. Even more small adjustments follow and micro-management ensures. It’s stressful to work this way, since you have zero degree of creative freedom and the same thing must be fixed repeatedly.

In the conference presentation of consulting business they told that the first few years are hard. You are not paid well and you don’t have any cash reserve, which drives you to accept all the work you can get. This slowly drifts you away from developing a sustainable growing business and your original goal as an open source entrepreneur.
Sadly, I can confirm this to be very true. Currently I am working hard just to keep the company running. Contracts are too short. The continuance is missing. Average hourly fee is somewhere between 40€-50€/hour. As they mentioned in the presentation, in the UK Plone hour price level was more like 80€/hour, and this was for contracts shorter than a year. I can only dream about getting a Plone contract longer than a year. The most horrible thing to reveal is that our company is accepting PHP jobs whilst missing suitable projects for our core technology platform.

However, this is just a start. It was expected. On the bright side, our average hour price is rising, slowly but steadily. I have achieved almost everything I dreamed of when I finished high school. I have my own luxury top floor office and I can decide my own working hours. Most importantly, I am not doing Dilbert job as being a code monkey transforming design documents to code implementation whilst boring myself to death of insignificance in a grey corporate mass.

In the conference, a friend of mine asked something along the lines: “Mikko, How high you think you are in Plone developer ranking?” This sputtered the following truth: I won’t ever wave a flag at the very top of Mt. Plone-developers. When you want to run a business, be prepared to sacrifice the happiness of coding. You will write Word documents, emails, talk on the phone, and so on. Technology goes forward; you don’t have time to catch the train. Sooner or later you’ll see only the backlights of your prior superiour technical skills.

But, my dear Service Buying Friend, this doesn’t matter from your point of view as a client. Nowadays I offer something better over my coding thumbs. Instead being able to solve your problem, I can point a person to do it (and take my little slice in the middle). I want to change the world and make it happier place for all of us. I believe I achieve a far greater effect by establishing a role model of running open source business instead of just code, code and code. Even if I code very cool stuff.

User group Finland?

Finland lacks open source business scene. Contracting work business is generally very conservative here. Companies, with the exception of Nokia, neither have a clue how to be a part of a community nor why to contribute back to the community. Our public sector is enjoying high Microsoft-corruption where ASP and Sharepoint are the words.

There must be a change in this. Even for the sake of the national budget: I want my tax money spent on cost effective Finnish open source solutions instead of the crap produced in Redmond.

Maybe the Plone-Python user group, working with patience, could have an effect here. The evangelistic work of teaching customers, teaching public sector and keeping noise in media would eventually convert the hearts of non-techies. Besides Red Innovation, there is one known active Plone consulting shop in Finland. Two universities are using Plone in large scale. The great presentation of Roberto Allende about starting a user group just might have motivated me enough to gather some leadership and accept the task of bringing pieces together.

P.S. Plone 3.0 compatible DataGridField release will come soon! I promise!

P.P.S Posting this blog with Wordpress update and all took 7 days. DGF 1.6 is already out there…

A quick tryout: Documentation Generating for Plone products

Plone is a modular CMS, which can be expanded with additional products. That means new features are easy to install, and also to customize. However, quickly understanding code that other people wrote, might turn tricky as there are as each coder uses his own style. Therefore, it might be useful to get an overall picture of the system before diving into details.

Documentation generators are useful for giving a comprehensive view on code. These are applications that traverse through code and extract information out of it. They use the structured information then to produce a nice looking reference of the code. Ever heard about API? Yep. Ever seen that sort of documentation among any 3rd party Plone product? At least I haven’t.

Luckily, there a few choices suitable for Plone/Python:

Parsers: doxygen (generic), epydoc (defines ‘epytext’, parses also others), docutils (defines and parses ‘reStructuredText’)
Extensions: graphviz (builds visualization graphs)
Plugins: eclox (an Eclipse plugin that uses doxygen, which uses graphviz)

(Plone API’s at api.plone.org use epydoc btw.)

Out of these, I quickly tested doxygen on a Plone product called EasyShop. The result was interesting but without use. EasyShop does only little subclassing and therefore the documentation doxygen produced was basically listings of separate classes and methods. Doxygen uses graphviz to build graphical visualizations of class relations, but those were out of use also. The problem here is that Plone products are not common python packages: they have adapters, utilities, views, events, subscribers and such. Creating dececnt API out of these would need a specific solution targeted at the platform.

Documentation generating seems interesting, however, and graphviz the most providing out of the whole bunch. Unfortunately, I couldn’t produce anything useful on my first few tries, but the subject just needs a little more research. After all, think about it: an API-like documentation with UML-like graphs of any Plone product, wouldn’t that be nice?

Hello Internet

I am proud to present new Red Innovation Ltd. company blog. We are ditching our old and faithful Plone-based Quills blog in the favor of Wordpress. Now, when we have several people doing blogging, this change makes our life easier.

Though Plone is a wonderful CMS platform, it really lacks a finished blog product upon it. I had only seen Wordpress blogs before. When installing Wordpress, I must admit that I was impressed, something which is not easily achieved after all these years in Internet. It’s so easy. Hundreds of times easier than playing around with Plone/Zope. My natural skeptism towards PHP products just got hit hard. Though PHP might not be the sharpest sword in programming language armory, it doesn’t seem to prevent creating nice end user products.

Copyright © Red Innovation Ltd. 2008 All Rights Reserved. | Log in | XHTML
Close
E-mail It