A Pythonic Way to Measure and Improve Your Programming Skills?

Those who know me know I am no fan of the programmer certification industry and I agree that your typical certificate arrangement, e.g. to show HR to get that box checked in your candidate file, so a company can skip responsibility in vetting a candidate, is un-Pythonic. However, I do think we need a good answer to the engineering manager or team leader who asks, "how do I know this person knows Python?", and to the candidate who thinks, "I believe I'm good at Python but am I as good as I think?".

The first case primarily cares about the reputation of the testing facility within the industry, so that if they choose wrong, they are absolved of blame. The latter two are often honestly looking for some guidance on bringing good people in-house or to step-up their programming skills.

As Python use spreads, it becomes more important for the community to provide benchmarks for measuring skill levels, to avoid Python being relegated to the, respectfully, lightweight programmer category e.g. PHP or perhaps Ruby on Rails (not Ruby itself).

Prior to PyCon I'd been thinking about some kind of campaign, service or documents, that I call "So you think you know Python...". My initial idea was for use by Python programmers, who are honest with themselves, to have a way to measure their knowledge. We all run into or are programmers who think they are hot-stuff, especially those of us who work solo as many in our community do, but when put side-by-side against others, fall a bit short. PyCon can have that effect, with so many smart people in a small area. ;-) But the objective isn't to denegrate these programmers but to provide a measuring stick, along with an educational roadmap, so they can make wise choices.

The rapid growth in use of Python is going to lead to a shortage in Python programmers, unless we manage the supply side as well. The manager wanting to find and hire good Python programmers is not an adversary seeking to use certificates to commoditize our community, but an ally in helping us get industry respect and career opportunities using our favorite language. We need to behind what he asks for to see what he needs.

I've been carefully watching Crunchy, about which a talk was given at PyCon, for writing tutorials that, with its "doctests" feature, could be used to propose tests that pass and require a candidate to write an acceptable program. This week I was prototyping a "rst2crunchy" tool, that let's you write Crunchy scripts in reStructured Text, rather than HTML. Crunchy needs more features, such as a user login facility and measurement facilities to store test submissions for review by experienced Python programmers after they pass. From what I hear on the edu-sig list, André Roberge and Johannes Woolard are tackling these but could use more help. Johannes in particular was a student for Google's Summer of Code 2006 and I believe is looking for sponsorship for 2007.

But perhaps Crunchy is not the ready answer in this case, so I'm looking for suggestions.

With regard to how a manager can know a person is good at Python, besides looking at their portfolio of work, we could stake out conceptual areas of the language or libraries about which a programmer should be conversant. This might be a document, with example questions but carefully focused on what lies behind those questions, so an interviewer can come up with his own questions and avoid rote memorization. This conceptual map could also eventually form the basis of advanced-placement in schools that goes beyond those terrible Visual BASIC test questions. And for the Python programmer who wonders if his knowledge is broad enough and wants to improve him or herself, the conceptual map could have links/recommendations to training materials on specific topics.

Join me on the advocacy list and let's discuss this further.

Jeff Rush
Python Advocacy Coordinator

Personal Schedule Application for PyCon 2007 Now Available

Doug Napoleone's really cool Personal Schedule Application is now up and running on the PyCon site.

Features include:

• displays a color-coded, up-to-date schedule
• displays pop-ups with descriptive information about each event
• allows selecting events to create your own personal schedule
• supports iCalendar output of both conference and personal schedules

If you have a PyCon login account, the application also:

• stores your schedule on the server instead of in a cookie.
• highlights talks that you're presenting or sessions you're chairing.

If you store your schedule on the server, your selections are included and summarized in a report for the PyCon organizers. We can therefore see which talks are the most popular, and this will help us schedule them in rooms with enough capacity.

A print-specific stylesheet is in development.

We think the schedule application is very neat, and hope it's useful in planning your PyCon attendance.

To easily create your own PyCon login account, visit the Site Signup Form, which is independent of your conference registration.

And speaking of your conference registration, if you haven't yet registered, the deadline for early bird registration is approaching fast - January 15th. We have some really great tutorials with limited seating so sign up for your favorites today or you might miss out.

A Call to Arms for Python Advocacy

As the Python Advocacy Coordinator, I've put up some wiki pages on the Python website for which I'm soliciting ideas, writing and graphics. Some of the material exists scattered about and just needs locating and organizing.

http://wiki.python.org/moin/Advocacy

First there is a need for whitepapers and flyers - I've put up a list of suggested topics w/notes at:

http://wiki.python.org/moin/AdvocacyWritingTasks

And there are fields for signing up for specific documents. We also have a page of possible magazine articles if that's more your style:

http://wiki.python.org/moin/ArticleIdeas.

These works are to be formed into advocacy kits for various audiences. So far we have the following ideas for kits:

• College Student's Python Advocacy Kit
• IT Department Python Advocacy Kit
• University Educator's Python Advocacy Kit
• K-12 Educator's Python Advocacy Kit
• Research Lab Python Advocacy Kit

The table of contents for the various kits can be found at:

http://wiki.python.org/moin/Advocacy#AdvocacyKits

Did we miss your kit? And what would you include in any of these?

Next, we are seeking reusable/retargetable teaching materials, such as those under a Creative Commons license. We need slide presentations and class handouts. Now I know there are a great many slide presentations on the web about Python. I can google them all but we're not looking for just any presentation, we're looking for the best of field. You can find the collection point at:

http://wiki.python.org/moin/Advocacy#TeachingMaterials

Last, perhaps you can dash off an idea or two for promotional merchandise. This could be the usuals like shirts but also posters, bumper stickers and buttons. What would you like to have? And with what graphics or slogans? We can reuse some of the slogans from the PyCon Slogan Contest, but are there others?

Our collection point for promo item ideas is at:

http://wiki.python.org/moin/Advocacy/WearablesGadgets

All materials will be credited to the authors to the extent possible by the delivery media. Make your mark.

Jeff Rush
Python Advocacy Coordinator

Get Involved in Python Advocacy

I'd like to extend an invitation to those who would like to get involved in advocating the use of Python. In August, the PSF hired me, for a 6-mo contract, to coordinate the Python advocacy effort. Since then I've been working to make the next PyCon one of the best conferences yet, and putting in place the infrastructure of a newcomer portal focused on drawing in those people who don't know much about Python but have developed an interest for various reasons. The portal also has an entry point to organize the materials and activies of those already in the Python community who want to get involved. I've also established a new mailing list on which to discuss advocacy, replacing the marketing@wingide.com list, and a blog for keeping the Python community up-to-date about advocacy goings on. The blog is aggregated into the official Planet Python but I've been unable to reach the organizer for the unofficial Planet Python.
A bit about the newcomer portal to place it in context; the portal is designed to help someone who has just become aware of Python decide if the language is right for them. It seeks to quickly direct visitors to the information they want, and bring to their attention how diverse and vibrant the support for Python is. The audience is not only programmers but also journalists, project managers, scientists/engineers, recruiters and educators. Different audiences come at Python with different needs and often need different explanations. And some are indeed programmers, but using other languages, who wonder how Python compares to what they are using now.

For the portal we have a need for content writers to focus on specific problem domains, for the various subcommunities to provide technology roadmaps and representative samples of source code that would entice someone to check them out. As one example, the SciPy/NumPy group could write about what makes their software attractive to the scientific community and provide one-page sources that illustrates certain common operations, to show off the clarify and expressiveness of Python. I've found plenty of material on the SciPy website that I'm weaving into the newcomer portal.

The portal is not designed to replace what we have at www.python.org but to complement it, and to serve as an organizing point for the extensive content already on www.python.org and elsewhere. The portal also specifically supports dynamic content, relational database storage of information and easy plug-in of new components to add new features. Such features will eventually include, among other things, a searchable roster of user groups, a registry of speakers and trainers and a catalog of books about Python, each with RSS feeds where appropriate.

And for the curious, the portal is written using the Zope 3 component system, building on the underlying Twisted subsystem for internal background scheduling and hooked to a PostgreSQL database. The site makes use of Zope 3 viewlets to provide pluggable display elements, reStructured text documents for a clean separation between content and infrastructure, and Nabu for synchronization of document collections into the indexing engine and persisting of the reST DOMs to enable content manipulation at presentation
time according to what is to be viewed (biblio data, abstracts, content). The portal is located on the python.org servers and all software and content are checked into svn.python.org.

In closing, I am greatly honored by the trust of the foundation in me and hope to serve the community well. As coordinator, I invite others to get involved and will strive to provide an assistive environment within which everyone can be productive. The primary discussion area is the new mailing list which can be joined at:

http://mail.python.org/mailman/listinfo/advocacy

and a list of what is needed is at:

http://advocacy.python.org/getinvolved

I am also maintaining a list of accomplishments and near-term To-Do's for myself at:

http://wiki.python.org/moin/PythonAdvocacyCoordinator

Jeff Rush
Python Advocacy Coordinator

O'Reilly Seeking Ideas on Partnering with the Community

The PSF (Python Software Foundation) was contacted by Shane at O'Reilly some ten-days ago, to open a dialog on how the Python community and O'Reilly could work together. As Python Advocacy Coordinator, this was given to me as a possible opportunity. I managed to sit down and chat with Shane this past Tuesday.

O'Reilly is trying to connect with the various open-source communities, such as the Python and Perl language groups, but also groups like the BSD Foundation. The criteria for whom to contact was partially based on those groups having some form of formal organization, similar to the Apache Foundation.

O'Reilly is in the very early stages of trying to discover how the groups can work together, with no actual kick-off until January 2007. The phone call was primarily to solicit ideas than to propose specific actions.

A Common Documentation Workflow

Shane said that O'Reilly was hearing a lot about how one problem that open-source communities face is poor documentation and that there is a strong need for documents being available in multiple languages. Teachers in certain countries are saying, for example, they could not use the BSD Book because their students lacked sufficient english to follow it.

So one goal might be for O'Reilly to provide translation coordination, and perhaps push for a single common documentation format, along with some workflow coordination of the global documentation scene, to try to raise the quality of documentation. Details are vague and certainly getting the entire open-source community to agree on one format would be a herculean challenge. Still, O'Reilly said they have a history of embracing challenges.

Part of the idea is to try to involve documentation contributors beyond the original developers, by adopting a format/mechanism that is friendly to technical writers, educators and college professors. I suggested something like reStructured text, which Shane was already familiar with. After researching the One Laptop Per Child project, I might now propose the derivative, Crossmark text, which supports multi-page documents.

It came up that people will often choose a programming language by whether there is documentation in their native language rather than purely on the merits of the programming language itself. This led to ideas of other ways to advocate the use of a programming language.

A Problem-Centered Tool Selector

One idea I proposed was a web portal of programming language concepts, with very good explanations and links to source code examples in a variety of programming languages. Shane mentioned the scriptome, a cookbook of of Perl one-liners for bioinformatics data processing tasks.

My work in advocacy has brought to my attention the need to focus on the domain or problem first, and then show how a technology can meet those needs. So I proposed a portal with some kind of sophisticated lookup/navigation engine, where someone with a problem could identify those programming languages with strengths in their area, perform comparisons among languages/frameworks and eventually decide on the toolset to use. I've already done a bit of work along these lines.

Shane thought this might have some possibilities, that we could identify a handful of problem areas and, working with a few members from each open-source community, come up with a handful of recipes to present. Then if it really is a good idea, encourage it to take off from there, else let it fade away.

It also occurred to me that such an arrangement might encourage the improvement of languages, by shining a light on select areas. A bit of friendly rivalry where upon discovering a gap in the solution space of language A but not in language B, that the community for language A might then work to fill that gap.

Registry of Speakers

Continuing our brainstorming on ways for O'Reilly and the open-source community to work together, I brought up the recurring problem that user groups often have of finding speakers. And while some speakers/topics are language-specific there are also those that cross language boundaries. So how about putting together some form of speaker registry, keyed both my open-source project and geographical area. This way user group organizers can search for someone to visit their group. And since there are also traveling speakers looking for engagements, such a solution should allow a speaker to visit the portal and indicate the areas in which he can speak and the dates when he will be in certain cities, to locate an audience. This way he can optimize his time, find receptive audiences of which he was unaware, and perhaps expense his trip on his taxes.

Targeted Book Release Notices Filtered by Community

Another idea is for the publishing industry to improve the way in which it communicates with its readers. It was unclear to Shane if O'Reilly already had such a system in place, or if so, whether it had all the functionality I'm about to describe.

The idea is for publishers to provide RSS feeds of current and future releases, appropriate tagged with publication date, programming language and so forth. Then the various Python, Perl and other websites could subscribe to the feeds and stay current on releases. At the moment many rely upon volunteer labor to manually post book information on their sites. Such a feed system should also provide for book reviews, and the ability of a person to subscribe to a feed by a particular author. Basically something a bit like a distributed form of the review system on Amazon.com's website.

This is nothing new and we're sure bits and pieces of this exist to some degree. It just needs to be standardized and mainstreamed. The approach helps publishers spread the word about their product by leveraging the open-source portals which have their own, focused traffic. And the portals benefit because their visitors want to know what new releases are coming in the specific area of interest.

Certainly publishers, including O'Reilly publish newsletters, including a mailing list of releases, but that material often mixes it all together, sending Java book notices to Python programmers and vice versa. It also usually is complete PR spin and lacks reviews by the community.

Alternative Ways of Publishing

We also discussed O'Reilly's move toward print-on-demand, and how it will increase the diversity in printed reading material, providing an outlet for subcommunities too small for a traditional book run. One trick will be to find ways to market those creations, since your normal publicity effort would be too costly. It will be important for O'Reilly to cheaply spread the word to the specific audience for a topic, and being able to RSS-stream release, review and author information to specific open-source portals would be a step in this direction.

And I brought up the challange of open-source projects like Zope 3, where the underlying software changes faster than a printed book can track. A solution is 'iterative editing', where a publisher keeps a book hot and streamlines the publication of future editions every 3-6 mos. However it also requires the open-source community to feed updates to the author/publisher on a near continual basis so the frequent editions are current.

Expanding the Pool of 'Non-Programming' Programmers

We continued on talking about how to achieve "programming for non-programmers", to draw in those who are not (and do not want to be) professional programmers, such as scientists and teachers. I don't have any specific project suggestions in my notes, other than my own strong desire to reach out to non-programmers, the establish basic software literacy. Shane mentioned that O'Reilly believes there is an incredible audience of open-source that is being ignored by traditional IT marketing.

In Closing - A Few Guiding Principles

• focus PR campaigns on those who are ignored, such as non-programmers
• the community will never agree on one format, so adopt several good ones and support mechanical conversion.
• to enable iterative editing/release, try to avoid the use of one-way conversion tools re print layout

O'Reilly will digest these ideas, test their endorseability with the other open-source communities and we'll see where it all leads. I'd also like to solicit additional ideas from the Python community. Please join the Advocacy mailing list and let's talk about it.

Tech Walkthru: D-BUS, HAL, Notify and Avahi LAN Services

As a result of researching topics for PyCon keynotes, I've been digging a bit into the One Laptop Per Child project. Being based on Python, essentially seeking to provide a 'Python operating environment', it looked quite interesting. I noticed that they plan to make heavy use of localized wireless grids for children to interact, essentially multicast service-locating DNS the software for which is called Avahi. The Avahi software implements the zeroconf or Bonjour/Rendevous functionality and sits on top of the D-BUS, a Unix desktop bus for interprocess communications within the same machine. There is a Python module for interfacing to D-BUS, both as a client and a server.

And so this past Saturday at the DFW Pythoneers hands-on session, I pulled together some demo code and we walked through this technology, D-BUS, HAL, Avahi, notification-daemon, in Python.

About D-BUS

D-BUS is both a library and a daemon. The library implements point-to-point message-based communication between processes on the same machine. The daemon adds a service registration ability and can start dormant services upon the arrival of requests. Now it turns out that you can start as many daemons and therefore buses as you wish. Usually there are two: a system bus started at boot time for exchanging messages about hardware availability and such, and a session bus started at user login time for desktop objects to talk among themselves.

Python programs can participate on either of these buses, with quite seamless handling of datatypes and asynchronous I/O using glib (wishing for a Twisted reactor myself). Buses aren't of much use though unless there is something to which to talk which leads us to HAL.

About HAL

One of the services that resides on the system bus is HAL, the Hardware Abstraction Layer. Using Python, it appears as a Device Manager mapping (keys, values, items) of device names to device instances, and each device instance has a mapping of properties (name -> value) and capabilities (set of well-known strings). Both the manager and device objects support callbacks upon the addition/removal/change of devices and properties.

In my demo code, I show how to monitor and manipulate the brightness of my laptop LCD display, as well as to detect the insertion of a USB flash drive. But if you're going to be responding to hardware events, you often want to notify and perhaps interact with the user, which means we need something like the notification daemon.

About the Notification Daemon

Another service that resides on the bus, although on the session but not the system bus, is the notification daemon. Not a catchy name it comes associated with the 'libnotify' component. Using it your Python-based services can quietly display one of those non-focus-stealing balloons in the lower-right corner of the screen. The balloons can have lifetimes or require user acknowledgement. Also supported are icons/graphics and sound although I didn't play with those in my demo code.

About Avahi

But getting back to my original goal, the Avahi multicast DNS service. Our group began to run out of time but Avahi is also accessible on the system bus. This means a Python program could stay aware of what machines are in the neighborhood and, combined with the notification daemon alert you.

I hope to further investigate tying these services together, along with the Festival/Cepstral speech synthesis interface I'm developing for Python and it would be really cool if we could have something working in time for PyCon. Something like the groupware programs the Mac users have to share live notes on talks but this time for us Linux people as well. ;-)
The source for the demo, rough code sketched out in an evening, is available in the DFW Pythoneers subversion repository.

-Jeff

Group Discussion: About Dynamic Languages and Python

This month at the DFW Unix User Group our main presentation was given by Dave Thomas about Ruby and Rails. Seeing as I run the Linux SIG afterward and am often short on topic ideas, I figured we could make an evening of dynamic languages, discussing what is a dynamic language, why you would want one and how Python fits into the picture. It was a bit of a walk-through of loosely related programming concepts, attempting to explain various concepts that are commonly tossed about but loosely defined.

The group of about 15-20 people was quite interactive as we discussed (and debated) the meanings of data typing, covering the dimension of dynamic versus static (whether variables can change their type, once set), and the independent dimension of weak versus strong (whether type mismatches are implicitly fixed).

	dynamic	static
strong	Python, Ruby	Java, FORTRAN
weak	Perl, PHP	COBOL, C

Also introduced was the concept of duck typing, yet another aspect of dynamic languages, and brief overviews of continuations and closures, two very different ideas that people often confuse. We also went through the meanings of introspection and reflection, finding it hard to separate the two. We covered the distinction between first-class, second-class and other-class language concepts, showing how functions in C/C++ and classes in C++ are second-class constructs but first-class in Python.

This led to definitions of paradigms (algorithms for computers) versus methodologies (algorithms for people), also frequently confused in conversation. We covered the common methodologies of flowcharting, structured and top-down programming, object-oriented and extreme programming. And the well-known procedural, functional, object-oriented and declarative paradigms. The audience noticed that object-oriented was on both lists, being both a paradigm and a methodology. And we wondered why we call it 'functional' programming instead of function-oriented' programming, similar to OO. It was also pointed out that 'object-oriented' and 'dynamically typed' are independent traits of a language and that you can have one without the other.

At this point we went a bit into the history of object-oriented programming, beginning with Simula-67, which had only non-instanced data and lacked inheritance until those ideas where introduced by Smalltalk. And it was noted that there are (at least) three object-oriented flavors; the usual split between C++ and Smalltalk but also yet another approach with Oberon.

As a group we tackled the definition of a scripting language, concluding it was the use to which the language was put more than anything about the syntax itself.

Moving into Python, we began defining the criterial for choosing a dynamic language; ease of expression, ease of handing maintenance off to others, richness of libraries, quality of documentation, license, and the size, friendliness and skill level of the supporting community. Python excels in all these areas and the history and feature set of of Python was introduced.

It was also pointed out that Python is not an either-or language, in that you can combine it with various environments such as Java/JVM, C#/CLR and C++/CPython.

The talk was not our usual fare of Linux applications but seemed to be well received. Our membership consists of programmers and sysadmins. This talk was also my first use of reStructured text combined with the S5 slide format as I usually produce slides using OpenOffice.org, exported to PDF. The slideshow is available for online viewing as well as the raw reStructured text making it up.

-Jeff