The TurboGears community is pleased to announced that TurboGears 1.0.6 has been released. This version is a maintenance release that fixes bugs & glitches found since the last two months.

We are really active at the moment working on the 1.1 branch which should enter beta tomorrow. The 1.1 release will be fully retro-compatible with the 1.0 branch but will propose important new things:

• SQLAlchemy as the default ORM
• Genshi has the default templating engine
• A new test system that is forward compatible with branch 1.5 & 2.0 (this is an important step toward migrating old applications written in 1.0 to 2.0 without breaking everything in the process)
• Drop the use of RuleDispatch and use Peak Rules instead (no more C compilation)
• Enhanced mod_wsgi compatibility to avoid past hacks that we had to do to deploy on Apache

Once 1.1 is out has a stable version we will encourage all 1.x users to go with this new version and to update their unit tests by following the TestMigration procedure.

A big work is undergoing also in the 1.5 branch to bring you CherryPy 3.1 support and further move TG into the WSGI realm.

Enjoy this new release and give feed-back in our mailing list!

Florent Aide.

I’ve had the great pleasure of being invited by the TurboGears team to become an official committer for the great TurboGears framework.

This is because I’ve been contributing patches for TurboGears 2 and other packages used by Animador (a TurboGears 2 application), since I started its development, in order to fix bugs and/or add new features that I want in Animador. So now I can apply my changes by myself!

And stay tunned, because very soon it’s going to be very easy to add OpenId support to any WSGI application by means of a plugin for the framework-independent repoze.who package!

We just cut another release for 1.9.7a3, and it’s even more backwards compatable with TG1.

I occasionally get questions about why we are working on version 2.0 of TurboGears and what it means for users.

• The ability to retrieve whole graphs of objects in a single query through the ORM
• Commit entire graphs of object changes back to the database in one step
• The ability to support multiple databases easily
• Out of the box support for a powerful web-based interactive debugger
• Full WSGI support
  • Your app is a WSGI app out of the box
  • You can run multiple TG2 apps in a single process
  • You easily call a any WSGI app from inside TG2
  • You can easily create and add middleware to your TG2 app
• Easy access to a large library of helper functions in your templates
• Out of the box support for using Routes to overide object dispatch for unusual URL’s
• Flexible out of the box caching for pages, intermediary data, etc
• Improved object dispatch, to better support resource oriented URL’s
• Support of Dojo, jquery, ext.js, mootools, and other javascript libraries via ToscaWidgets

This means that TG2 apps have more flexibility and can can scale better than TG1, and we’re working on trying to make TurboGears 2 better documented than any other framework. Because we have full WSGI support you can easily mount existing WSGI applications in your TurboGears site, and you can also get things like profiling middleware, middleware that helps you find memory leaks, or any of a whole host of other interesting middleware almost for free.

There’s lots more to be done before we hit 2.0 final, we need to transparently support attomic commits across database boundaries (when the underlying stores support it). We need to make it even easier to build reusable site-components with TG2, and we need to continue to improve the TG2 documentaiton.

But I think we’re making huge progress, and I’m looking forward to the next release. The current plan is to release a 1.9.7 stable release in the next 4-8 weeks, and to release 2.0 (with the above mentioned extra features) later this year.

While I do not advertise it directly on this site, in addition to my “day job” I also provide Python training services for companies and individuals that want a comprehensive introduction to the Python programming language on a compressed schedule. One of the most difficult topics to condense and explain to a group of new Python programmers is metaclasses. Over the last year or so I’ve spent a lot of time refining my teaching methods to make this complex topic approachable.

A few months ago, I presented a version of my slides on metaclasses at a PyATL meeting. Following the meeting, I was approached by Doug Hellmann about the possibility of writing an article on metaclasses for Python Magazine. Well, I am pleased to say that my article, entitled Metaclasses Demystified appears in the July 2008 issue of Python Magazine.

This is my first professionally published technical article, and I am really excited that my work will be read by many Python programmers. Depending on feedback from readers, expect to see more contributions to Python Magazine from me in the future. It was a lot of fun crafting the article!

As promised, I have created a screencast which shows one how to document a project using sphinx.

http://showmedo.com/videos/video?name=2910020&fromSeriesID=291

The video runs about 16 minutes, I go through a complete example, from installing sphinx, to inserting a doctest in your code, checking it with nosetests, and finally publishing it to the html pages.

Source code for this demo can be found at pythontutorials.googlecode.com.  

I think virtualenv is horribly useful, because it is very, very common that I want to use more than one version of a library. Sometimes this is just because I want to test TurboGears 2.0 against some new version of some dependency, sometimes it’s because I’m running some old version of some project with out-dated requirements, and other times it’s just because I want to key my OS’s system python clean. Because there’s lots of virtualenvironments on my machine, and because I’m always switching between them I was very happy to find Doug Helman’s article on how to more easily manage switching between the various virtual environments that I’m working on at any given point in time.

If you use virtualenvs — and if you’re a python developer you should ;) — take a look at this article, it might make your life easier.

I’ve not posted to this blog in over a month, mainly because I have been busy. Busy is a good thing I think. I have been helping clearwired get their newest web application off the ground, working on a number of screencasts, helping Mark with the latest batches of TG2 releases, learning about new and interesting OSS Python projects.

One of the more interesting things I have started to use with some frequency is Sphinx. Sphinx is a documentation system whichallows you to create webpages (among other things) with .rst files. It also grabs directly from your modules and inserts your docstrings (including your doctests.) It is great to have code which is not only documented, but also tested using nose. Look for a screencast from me in the near future on this one.

Two of the recent screencasts I have written show how to use virtualenv, PasteScript, and Nose. I go through a ground-up example showing how to create a virtualenv, a project package, and finally how to test that package and provide robustness with code coverage. The final screencast is about how to install TG2, which has become much easier in the advent of TG2’s first alpha release.

TG2 is actually nearing it’s second alpha release. I think This is the best release yet. I spent some time moving DBSprockets to pep-8 compliance, and the two things TG2 depends on (tg.ext.silverplate and tgcrud) are now both using the newest release of DBSprockets. I also released basketweaver which allows you to create a simple local pypi made of static HTML files. Special thanks to Chris McDonough, who wrote the makeindex.py script. All I did was to fixed it up a little, and package using PasteScript, and provide a console_script for easy usage.

Along with Sphinx, I have been trying to wrap my head around Rum, which is a new project to generate forms from database schema, much like DBSprockets. Although the internals of the system are somewhat mystifying, the API is squeaky clean, and in my mind what DBSprockets was eventually planned to be. The great news is that Alberto has done such a good job laying the groundwork for Rum I should be able to jump right in and apply lessons learned from DBSprockets. At this point it is safe to say that I will continue to maintain DBSprockets for bug fixes, but that the internals of it are going to be converted over to using Rum, before being deprecated altogether. DBSprockets had a great run, but it is time to move on and move forward with a superior design which promises to integrate so many WSGI technologies.

Here’s a quick note on how to use IIS to serve up your TurboGears applications. This is something we need to document better, if we want to increase python/turbogears penetration in the windows market.

It’s not like asp.net is so good that they don’t need TurboGears. And TurboGears multi-threaded+multiprocess model works better on windows than many of the other “dynamic language web-frameworks which depend solely on the multi-process model.

TurboGears has used a tree of controller objects to do URL dispatch since 1.0. Which is nice and easy to understand, and makes getting started very quick. But, it wasn’t always apparent how you should use it to do RESTful dispatch, since the HTTP verbs all ended up going to the same controller method.

You could dispatch within that method, but that never felt totally clean to me. And I’ve been thinking and talking about better ways to do this. The good news is that Rick Copland just “made it happen” after a conversation in atlanta last week.

Using his trick you can write code like this:

class Root(controllers.RootController):    class person(RestMethod):        @expose('json')        def get(self, **kw):            #...do stuff...            return dict(method='GET', args=kw)        @expose('json')        def post(self, **kw):             #...do stuff...            return dict(method='POST', args=kw)        # NOT exposed        def delete(self, **kw):            return dict(method='DELETE', args=kw)

And then HTTP GET and POST verbs will be routed to their respective methods. This makes working with RESTFul api’s easier. And on that front I’m very much looking forward to Dojo 1.2 which has all of kinds of restful json data store goodness.

Hopefully Rick’s recipe can find it’s way into the 1.x core code somewhere so that it’s even easier to do. But for now a couple dozen lines of code gives you everything you need for RESTful goodness.

Looks like the TG 1.x team has been doing good things. CherryPy 3 is support is a very, very good to hear that it’s been integrated into TG1, because this sets the stage for a much longer support cycle for TG1.

CherryPy 3 is a more solid base to work with, and it’s easier to invision supporting it for several years than it was with CherryPy 2.x. And the test-refactoring work that has gone into making this happen is a huge benefit as well, because you will be able to write application level tests that work in tg 1.5 and will still work in 2.0, easing that migration path.

Florent Aide may be on vacation, but things are still moving forward very quickly on the 1.5 front. I’m looking forward to seeing a release with all this great stuff in it.

We’re hard at work trying to make the TurboGears 2 docs into something best in class. There’s a long way to go, but the toolchain we’re using keeps getting better and better thanks to Sphinx and Bruno José de Moraes Melo and his GSoC work.

Sphinx all by itself provides a great system for turning ReStructured Text (ReST) files into usable documentation. Since the TG docs were in MoinMoin already, and we’d been careful to use ReST, getting started with sphinx was pretty easy. We updated the doc strings in TG2 to take advantage of the sphinx autodoc features to do API level documentiaton.

But the missing piece was pulling example code in from working sample projects. Sphinx had some include helpers, but they weren’t flexible enough for our needs. So, Bruno has created a set of sphinx extensions to help us. The first of these imports working files from projects checked into subversion. This allows us to show progressive enhancement of projects, and it allows us to keep the working example code under the same source control system as the docs.

So now our Wiki20 source doc has sections that look like this:

.. code:: Wiki-20/wiki20/controllers/root.py   :revision: 4831

Thanks to the sphinx goodness the code this pulls in is going to be color coded for automatically. Bruno has also added the ability to mark of specific snipits of source code so that you can import just the bit you want to show off like this:

.. code:: Wiki-20/wiki20/templates/page.html    :revision: 4831    :section: PageName

and where the PageName section is marked off like this:

<!-- ##{PageName} -->    Page Name Goes Here<!-- ## -->

Bruno is working on a few more extensions to sphinx, that automatically test project code before adding it to the doc, and that automatically zip up the sample project and make it available so that documentation uses can easily download it.

I’m pretty excited about how the docs are shaping up, and I’m hoping that a few more people get involved in writing good docs, because the TG team is very much committed to trying to have the best possible docs. Docs aren’t sexy, but good docs can make or break a the use of a framework for new users getting started, and advanced students who are trying to do complex things.

When I first heard about Apple’s new MobileMe service, I was excited by the prospect of being able to keep all of my devices in sync instantly, with no waiting. I was so excited, that I purchased a boxed version of .Mac so that I’d be one of the first to get my hands on the service when it debuted.

One of the main reasons I purchased MobileMe was because of Apple’s beautiful marketing site describing the features of MobileMe. The site states:

MobileMe stores all your email, contacts, and calendars in the cloud and pushes them down to your iPhone, iPod touch, Mac, and PC. When you make a change on one device, the cloud updates the others. Push happens automatically, instantly, and continuously. You don’t have to wait for it or remember to do anything — such as docking your iPhone and syncing manually — to stay up to date.

As it turns out, this is not only misleading, but is essentially a bold-faced lie. While the “push” does occur from your iPhone to the cloud and vice-versa, your Mac only receives and sends updates once every 15 minutes, even if you instruct it to sync “Automatically” in the MobileMe preferences pane on your Mac. Honestly, I expected better from Apple.

In addition to this very irritating issue, I have several other problems with the service, which is billed as “Exchange for the rest of us,” that greatly diminish its usefulness.

I would love to utilize the push email service to manage my personal email. However, I have a personal domain where all of my email is delivered. In addition, I subscribe to many mailing lists, and receive several hundred emails a day. All of my email is neatly organized into folders automatically, using server-side mail filters through the excellent procmail utility.

MobileMe email has two massive shortcomings that make it useless for my situation:

1. You cannot set up MobileMe to handle email for a personal domain, unless you set up your existing mail server to forward email onto MobileMe. I could live without this one, if it were not for the second problem.
2. While rules that you set up in Apple mail “sync” to MobileMe, they are not applied server side. In order for your rules to be applied, you must have a version of Apple Mail running on some computer connected to the internet to apply those rules client-side.

MobileMe is a very compelling service if you just read the marketing, but in the end, in falls extremely short of my expectations from a company like Apple. I have sent my complaints onto MobileMe customer support, and am awaiting a response. Hopefully, they’ll at least acknowledge the misleading marketing.

I’ve been doing package management related stuff a lot recently, and thanks to a nice little script Chris McDonough showed me, it’s easy to turn a directory full of eggs into a PyPI (Python Package Index) style index page.

And now I’m pretty much sold on the the idea of having separate indexes for all our major releases so that there’s a known, tested, set of versions and dependencies, that can be easy_installed with one command to get a working install. When combined with Ian Bicking’s virtualenv package, this pretty much solves the package management problem for users, because they can keep a lightweight virtual environment for each version of the software they want to have, and because they can have a reliable and repeatable way to get a specific version of TurboGears along with all of it’s dependencies.

For projects like TurboGears, Pylons, or Zope, where there’s a pretty large set of dependencies it makes good sense for the development team to take on the burden of managing packages and versions for users. This is of course conceptually similar to creating a Linux distribution, but fortunately it’s orders of magnitude easier. We have around 40 packages in the turbogears 2 index, but Ubuntu has well over 10,000 packages. And in a world of interconnected componets one of the core value-adding propositions that a “megaframework” adds is the ability to get a pre-vetted set of packages that are known to work together.

And it’s in that light that I’m thinking about a time based release schedule makes sense. It’s not so much that we need to change the core TG2 code all that often, but there are lots of components and interactions, and it seems like there’s a value in providing regular “distributions” where we’ve tested everything and proved that it all works together well.

It’s the framework maker’s job to make doing the right thing easy. I was reading an article this morning, and came across this quote:

The right thing should be the right thing partly because it’s easy and natural to do. If the right thing is unnatural, that is kind of an environment smell. It’s beyond a code smell. It’s telling you something.

– Rod Johnson (Founder of the Java Spring framework).

Which is another way of saying:

There should be one — and preferably only one — obvious way to do it.

– The Zen of Python

I think TurboGears is actually pretty good about making the right thing seem natural. Heck Bruce Eckel has said:

I think this is the first time a web framework’s functionality has been so obvious”.

But just because we’ve done a pretty good job in the past does not mean that we shouldn’t be working to do better. And there’s one area which has been bothering me a bit recently.

I want RESTful web services to feel more natural to write in TurboGears. The new TG2 lookup method allows you to use the URL as a way to instantiate a resource object and call methods on it, which is an important first step. We probably should also extend the object dispatch mechanism to call different methods based on HTTP verbs, which I think will help.

And of course in TG2 you can just use Routes to create more restful resources, but this feels a unnatural to some TG users who are very invested in object dispatch.

Either way, REST is something that is often (though not always) the right thing to do, and we need to make it so that it needs to feel natural to TG programmers.

On July first we cut out another 1.0.x TurboGears release.

I see this release as an ongoing effort to maintain our 1.0.x users and provide support and enhancements to existing applications.

While I am working with Mark and others on the 2.0 branch to make a release happen, I also work on the 1.5 front (formerly known as the 1.1 branch) to get this intermediary release out.

The 1.5 release aims to be 100% compatible with 1.0.x applications (running on python >= 2.4). I test regularly my development version against production applications that are running on top of TG 1.0 at the moment.

The real differences in 1.5 compared to 1.0 are:

• Different defaults for quickstarted projects, SQLAlchemy, Genshi instead of SQLObject and Kid.
• Testing Framework compatible with 2.0 to aid refactoring an application from 1.5 to 2.0 by using your test suit.
• RuleDispatch dropped from the required TurboJson version and replaced by PeakRules. Should not change too much for end users, but will help maintainers because we won’t need to precompile binaries.

At the end of the day I could say the TG scene is in a pretty good shape and we begin to see some attraction from the 2.0 branch, which is a good thing IMHO. And last but not least, releases always get more attention than simple SVN commits and we get users’ feedback in a much greater scale on such occasions, which is essential for an Open Source project such as ours.

TG2 has been a wild and crazy place with quite a few API changes over the last 4 months. Cutting a release makes installation easier, at least in a project like TurboGears 2, where dependency management is part of the release process.

It makes writing components easier–it’s easier to certify against a release than against a range of SVN checkouts.

On the other hand, releases seem to make promises about API stability, that we as developers aren’t always ready to make. I think that’s often a bad excuse for not doing the work needed to cut a release. After all we have limited time, and perhaps it is better if we spend that time working on “new stuff” rather than on packaging. The problem with that thinking is that releases provide a way for the development community to manage breaking API changes better, informing users about them in a central place, and providing users with tools to manage the pace of change.

So, releases are even more important when there’s lots of activity and potential API changes. . And they save time. Well, not my time personally, but every hour I spend doing releases makes getting started that much easier for hundreds, or thousands of people. Hopefully karma will result in some of those people contributing back.

Which is why we just cut a release, and why I think we should keep cutting releases on a regular basis from now to 2.0 final.

But how should we manage that process?

Open Source culture has a “we’ll release it when it’s done” philosophy which avoids all kinds of half baked stuff from getting released in a critically buggy state. And that’s a good thing.

On the other hand Ubuntu has taught us that it’s possible to have stable, complete, compelling software releases, in an open source world — and that it can be done on a predictable schedule.

So, I’m hopping that we can do time-based mini-releases of TG2 over the next few months, on our way to 2.0. I’m still thinking this through, but I think we will do a first alpha release this month, and follow that up with monthly releases until we hit 2.0.

2.0 final is very likely several months out, but having monthly releases between now and then will make that process easier, and can help us gather more testers, who are willing to put up with a bit of well managed API change on the way to 2.0.

Long term, I’m considering doing a 6 month time based release schedule, because it seems to have worked out well for Ubuntu.

And TG2 is a bit like Ubuntu or in that it brings together lots of little pieces to make something bigger, so may benefit from structuring our release process after theirs. But that’s just day-dreaming at this point, and I’m very interested in what other people think about how we can best manage the release process of TG.

At the same time, we probably should start thinking about version life-cycles and support timeliness. We’re planning on supporting 1.0 for at least the next a year after 2.0 final. But if there are people willing to help out I think that support could go on for longer (even forever, if enough people want it, that’s the beauty of open source!).

I’m not sure exactly how long is long enough… Heck, in general, what do you think? How should we manage releases and support-timelines in the TurboGears world. Is the Ubuntu system transferable to web-framework world?

We’re very serious about wanting to know what would make users most happy, so please let us know what you think.

I’d like to officially announce the release of TurboGears 1.9.7 alpha 1. We’ve been working on this for a while yet, and while there’s still quite a bit of work to be done, we’ve now got a solid base to work from.

There’s a whole new section of the turbogears website devoted to TurboGears 2. We’re still working on TG2, and on the docs, as is expected with an alpha release, but I’m really excited about the new docs, which are generated using Sphinx.

Thanks to some help from the folks at the Repoze project, we’ve even got out own package index to use for installing TG2, so If you’ve already got setuptools installed, the install process is as easy as:


$ easy_install -i http://www.turbogears.org/2.0/downloads/1.9.7a1/index tg.devtools


Of course, we’ve got more detailed install instrutions including information on how to install into a clean virtual environment, so that you eliminate any possibility that you’ll get version conflicts in your main python install:

http://www.turbogears.org/2.0/docs/main/DownloadInstall.html

There are a lot of new features in TG2, an a lot more to come, so be aware that this is an alpha release, and we’ll probably see some API changes between now and 2.0 final. We’ll document those changes and tell you what you need to know to upgrade your project, but we’re not going to be afraid to improve and refine our API’s between now and the 2.0 final. So, if you’re not afraid of change, and are interested in helping to shape the next generation of dynamic web frameworks, hop on in and let us know what you think.

I think it’s worth reaffirming our commitment to the 1.x users. This does not mean we’re dropping support for TG1, if anything it looks like 1.x development is accelerating these days. So, if you’re interested in a stable, well tested, growing environment TurboGears 1 is still a great choice.

I’ll blog more this afternoon about my thoughts on the TurboGears 2 release schedule, and the release process. But, I’m very grateful to all of the people who helped out in order to make this release possible. Thanks a million times!

I keep getting reports that TurboGears seems to be stagnant, and from the inside that just does not make sense: We’ve had 202 checkins in the the last 30 days. Which, by way of reference, is a tiny bit more than Django’s 183.

So, that seems to indicate that we’re definitely not quite dead yet. But, even that’s only half the story.

If you pull in checkins to a few components that are external to TG, but internal to Rails and Django (template language, ORM, forms + form validation, etc) that number grows significantly:

• 58 — SQLAlchemy — ORM
• 38 — Genshi — Template Language
• 113 — ToscaWidgets and tw.forms — Widgets, and Widget based Forms

Thats 353. Now if you pull in the changes to Pylons, related middleware and helpers, there’s another big jump:

• 56 — Pylons
• 106 — Paste
• 55— WebHelpers
• 36 — Routes
• 6 — Beaker

Now, of course there are other components which are actively contributing to the development of stuff that TurboGears users get as part of the package. But even ignoring all of that we’re looking at over 600 commits in the last month.

TurboGears ecosystem growth:

There’s also a growing ecosystem of turbogears stuff, and it’s all moving forward very quickly too.

Here’s a quick sample of projects I’m watching:

• 26 — TGTools
• 33 — tw.openlayers
• 31 — tw.dynforms
• 4— dbsprockets
• 7 — tw.dojo
• 16 — tw.jquery

There’s a new TG2 based CMS in the works too (http://code.google.com/p/lymon/). And there are a couple more very interesting projects that have not yet gone public.

Splitting our attention:

One thing that’s worth mentioning about all of this is that we’ve intentionally split our efforts over two areas:

1. Evolutionary improvements to TG 1.x
2. A new core in TG2.x

And that slows us down a bit. But it’s important because we want to take care of our installed userbase, and to try to grow into new areas at the same time. Doing one or the other would be way easier, but ignoring either one would be a huge mistake.

If it wasn’t for all the amazing stuff going on in the WSGi component world, TG2 would not be possible.

But because the wider python web community is developing new ways to work together around the WSGI model, tg2 development has been moving forward very well. And that’s one of the reasons why I’m so sold on the “component” model of framework development. Sure, it would be nice to have everything under one roof, and to have a stronger guiding hand on the whole process. But it’s not worth giving up all the innovation that happens “at the edges.”

TurboGears 1.x progress

On the evolutionary improvement front we’ve done lots to support SQLAlchemy, Genshi, ToscaWigets, DBSprockets, improve Json support, and created a brand new testing infrastructure, improved out authorization system, and otherwise made lots of positive changes. We’ve also had a half dozen new releases, with feature enhancements, averaging about a release a month.

TurboGears 2 progress

TG2 constitutes our revolutionary front, we’ve added many, many new features, and tools, and have maintained very significant API backwards compatibility, improved performance, and entirely redesigned the core of TurboGears. And we’re approaching our first alpha release in the next few days, so I don’t want to belabor the new stuff here.

What we need to do better:

I understand that not all of this work has been very visible, and that’s our fault, for not engaging the wider python community better, and it’s my fault in particular for not getting the TurboGears 2 work out into the wider world more quickly. Which is something I definitely intend to change in the very near future!

So, if you’re a part of the TG community, and you’re site/project needs to be better known, let me known, let me know. We need to raise the profile of some of the very interesting stuff that’s being done in the community, because outsiders seem to think that we’re not moving very fast, while insiders talk to me about the “blistering pace” of development.

Tomorrow Night I will be speaking about agile technologies with SQLAlchemy at the Front Range Pythoneers monthly meeting. I spent a few hours last night working out my presentation and creating a number of screencasts which show how to use tools like virtualenv, paster, and nosetests.  I also touch apon sqlalchemy, and how one would set up a test environment for their database schema.  Even if you don’t live in the Boulder/Denver area the information could be valuable to you, so I decided to set up a googlecode repository to store all of my tutorial-related materials.  It is called PythonTutorials.