Progress in open development

April 26, 2006 | Stephen | Process.

Just prior to this week’s news, Glenn had an all-hands for the Operating Platforms Group. Since the all-hands is performed once for the Pacific through European business day and again later for Asia through Pacific working hours, I thought I would compose my presentation in full in the hopes of being consistent, and knowing that I could share it here as well. I fear the later presentation will become known as the “apple juice” talk, as my not-quite-two-years-old son Nathaniel came to the phone a few times to inform the audience of his preferred remedy to a sudden thirst. Your comments are welcomed—we’ll be expanding and revising the talk and its material as the program and its tools mature. The talk benefitted from criticism and suggestions from Jim and Karyn.

Overview

Thank you for this opportunity to speak with you. Over the next thirty minutes, I thought that we could review how software development for the various consolidations will change into open development, on opensolaris.org. I will attempt to be clear about what you can do today and what’s yet to come. Along the way, I’ll also share what I believe are candid criticisms of what we have so far.

In the interests of lending some structure to this talk, we’ll look at typical tasks an individual developer tends to do in the course of working on an operating system component. On a given day, a developer might

read and comment on a modified bug report,
update a child source tree,
change some code in their source tree,
build the binaries in their source tree,
solicit or provide feedback for a code review, or
make or evaluate a request to integrate reviewed code into a consolidation’s gate,

as actions that involve small group of people. Our developer might also be taking actions that involve a larger number of people, up to the entire OpenSolaris community. For instance, they could be

asking or answering questions in a public forum,
proposing or commenting on a new interface or implementation choice, or
proposing or joining a new area of investigation.

I intend to touch briefly on each of these examples, which I hope will make it clear that we’re committed to making it feasible—even easy—to perform the complete set of development tasks in an open environment.

Bug reporting

The first part of my morning is still spent sipping a hot cup of tea and reading new and update bug reports in the categories I’m interested in. For any of the reports I might wish to comment on, I can use the Java client to connect to the bug database and make appropriate updates, perhaps making connections to related bugs or adding the email of a developer I suspect might be concerned about this issue. Unfortunately, none of these actions is presently convenient for a developer without that access: bug submission and manual querying is available at bugs.opensolaris.org, but ongoing reporting and updates are not.

There are two adjustments to make if you’re using the bug database. First, use the Description field for the bulk of your updates, as all other fields are not published. This means the end of the “See Comments” bug idiom. And also, check Hook 6, which will contain the OpenSolaris ID of the submitter; if you make a significant status change, send the submitter an update.

We’ve given input into one effort to examine the bug database’s role in open development, and we’re funding work to improve the functionality of bugs.opensolaris.org. But if you have Bugster access and can help a contributor who can’t make an update, please do so.

There are additional complications for the present bug database with respect to upstream and downstream relationships—with other communities whose software we rely upon or whose rely upon ours—but those aren’t part of a typical developer’s day.

One aspect of bug reporting we have noticed is that the bite sized bugs, with the keyword “oss-bite-size“, provide a very effective starting point for developers new to OpenSolaris. I encourage you all to start marking introductory bugs with this keyword; because this keyword has been so effective, we’re going to start tracking the marking and fixing rates as a metric for the program. For experienced developers, I appeal to you to respect some kind of rough law of “conservation of bite-sized bugs”; if you fix one, mark another one.

Source code management

Usually my tea is still warm when I’m deciding whether or not my current source tree is sufficiently up-to-date. Generally, the tea is tepid once my bringover is done.

If you’re following the tools-discuss list on opensolaris.org, you’ll know that we recently selected a source code management tool called Mercurial to be the distributed source code management utility for opensolaris consolidations. We will be working with the Mercurial community to get a few features implemented, and I expect that most of you will be pleased with the performance of this very nice utility. I expect in most cases my tea will still be pleasantly warm.

There will be an interim period where we’ll be publishing Mercurial bundles externally, but still using TeamWare inside Sun. This period will mostly be spent on tools updates, but in most cases we should see all developers on an equal footing, whether their primary consolidation uses Subversion or Mercurial.

Editing code

Once I’ve synced up, I’m in position to start making changes. I promise not to recommend a specific editor. Generally, expectations about code have not changed: the C style is expected and enforced, other conventions will be verified by your code reviewers.

Practices across internal and external contributors are common for this stage.

New files written by contributors to ON should contain the CDDL copyright block; other consolidations may have a different default license. An example is given for common types of files in

$SRC/prototypes/prototype.{Makefile, c, csh, h, java, ksh, man, s, sh}

in the ON consolidation. If your work involves integration into a Solaris release prior to 14 June, then the CDDL block should not be carried back into that source tree.

The “wx cddlchk” invocation, which is also part of “wx nits” and “wx pbchk”, will check that the license block is correct and up-to-date.

Building a consolidation

Typically, building your consolidation or a portion thereof takes some time, usually based on your system’s performance or how many people you share that system with. Once you’ve started your build, this is generally a good time to stretch or make a new cup of tea.

The various consolidations will tax build systems differently. With the recent integration of GCC shadow compilation, ON build system administrators need to be aware that there are now potentially two compiler invocations for each changed C file. On aggressively configured systems, this may result in paging.

For full builds, the experience of an internal and external developer should be equivalent.

A key difference at present is the availability of test suites. As you might suspect, there are many test suites that depend on the Sun corporate network environment; the result is that the suites are not presently available. Under the present sponsor process, a code submission is tested by the sponsor, or testing is arranged by the sponsor. There are a number of paths to balancing the test responsibility: the Test Community is exploring a hosted self-service test run, and would probably welcome discussions about what tests a developer should be able to run in their own environment to validate their changes.

Code review

With validated changes in hand, a developer is ready to start down the path towards integration. This commences with getting one or more developers to review your changes to the code. Of course, because you want your reviewers to find actual problems, you already made certain that your changes build cleanly and have no obvious style errors.

Finding a code reviewer often involves checking who’s around, either physically or by various network protocols. The webrev tool allows you to construct a small web tree that contains your changes in a variety of formats so that a reviewer can work through your modifications and make appropriate suggestions. This makes it more convenient to have reviewers who aren’t physically contiguous to you.

Since webrev generates a tree of HTML documents, all you need to publish your code review materials is a place to put them. This is easiest if you have an external site, and is an area where a developer primarily using the Sun corporate environment is at a disadvantage.

In the interim, Dan Price and Steve Lau have provided a “code review swamp” hosted at cr.grommit.com to offer a place to host your webrevs. The swamp is set up to handle publishing current format webrevs today; instructions are present on that site.

Now, I’ve also asked the Tools community to examine enhancing webrev to put the changes into a single document and use browser features to make the review experience equivalent. In this case, publication becomes substantially easier, and even blogs become a possible location for publishing code reviews.

Even with publication eased, finding reviewers can still end up relying on personal networks, which is difficult for a developer new to the community. I also believe that the request for a code review needs to be narrowcasted to those fellow developers who are interested in the area that is changing. So I believe we’ll need to develop code review workflow support on the opensolaris.org site. If you’re interested in how this feature might work, we will be discussing it in the Tools community.

Code integration

With support for your changes from your code reviewers, and after building and testing your possibly modified code once more, you are ready to make your request to integrate—which has long been known by its acronym, RTI.

At present, the “request to integrate” is made via a tool only accessible on the Sun network. That, and the current source code management situation, led to the current sponsor process.

It’s worth noting that each time I mention an aspect where Sun corporate network access is required to access a resource, the sponsor is performing the work on behalf of the contributor. That can add up to a substantial amount of work for some changes. Please be supportive and respectful of both the contributor and the sponsor; if you’ve the time to be a sponsor, or assist a sponsor-contributor interaction by participating in a review or running tests or shepherding a build, please consider helping out.

Once the various consolidations move to external writable consolidations, we’ll see direct integration by all committers, but for new developers, the sponsor process will remain in place. Aspects will become simpler: the new source code tools make passing patches around substantially easier than before. As new capabilities are brought online on opensolaris.org, like code review hosting and self-service testing, we’ll see the sponsor-contributor process distilled down to its core: mentoring on technical participation in operating systems development.

Forum participation

Of course, technical guidance isn’t only provided on a one-on-one basis, and not everyone is intending that their changes target mainline. In the various discussion forums, you’ll see questions on how best to do something, or where it’s presently done, or whether a particular choice has drawbacks.

You should engage in these conversations. Questions, and the context behind them, is valuable to all community members, whether they are planning to develop code or merely consider that feature in their deployment. A considered answer has value beyond the immediate conversation; all accurate technical content, be it a forum response, project documents, or blog entries, will increase the search engine cross section for opensolaris.org. As with any public forum, your participation should be confined to the technical aspects of the discussion; private information concerning your organization is not suitable as a topic for public discussion. That’s not a new restriction—merely common sense.

I’ll talk about forum creation in a moment. But first, I want to talk about a few special forums on opensolaris.org.

opensolaris-discuss is, in an extremely loose metaphor, the town square of opensolaris.org. The various procedures that the community’s Governing Board has developed generally involve making an announcement or request for discussion on opensolaris-discuss. But, like any public place, there are conversations on pretty much any topic; it’s a high traffic alias.

opensolaris-code has in recent days had its role clarified: it’s a purely technical discussion alias on any aspect of the various components of OpenSolaris, architectural, implementation, or otherwise.

Architectural process

The third forum I want to mention is the ARC community and its associated mailing list, because it brings us back to our typical development activities.

Occasionally, in the course of fixing a bug or considering a problem, we realize we need to add a new interface to some OpenSolaris component. The process for doing this is to take the interface through architectural review. We’ve had a couple of contributors take their new interfaces through the fast track process, and reviewed some of the difficulties associated with each of those cases.

We also have found that the site isn’t well structured for the frequent publication required to track the currently active ARC cases, or to publish key historical cases.

So, while we have an awkward way of making the process participatory, the correct course appears to be to take the fast track process for open consolidations and modify it such that it can be hosted on opensolaris.org. If you’re interested in this topic, we’ll be starting this discussion in the ARC community.

Proposals

But sometimes, you realize that the changes you’re envisioning are sufficiently large that they will require a group effort. This effort usually needs a forum of its own, plus a tree of documents, plus a source code repository.

opensolaris.org can handle this with its project hosting support, which will shortly offer the source code hosting feature. To get a project, you just propose the project on opensolaris-discuss. Over time, your project may garner interested individuals and communities, whose association will become a way for community members to identify projects they may find interesting. In the meanwhile, your project is well underway.

New projects and even existing projects for Nevada should use opensolaris.org for hosting their project; please don’t wait until you’re ready to integrate. That’s far too late. As you might suspect, we’re developing a metric to monitor the progression towards external hosting.

The other kind of forum that we can host on opensolaris.org is called a community, which has been confusing in a manifold sense. The term itself is overloaded: we can talk of the OpenSolaris membership as a whole as a community, or of a specific OpenSolaris community. And the role of a community in OpenSolaris governance is still being debated by the governance working group.

But I can tell you a few aspects of communities that might help you determine whether your initiative is a potential community. First, a community is not a program, strategy, or lifestyle. If you haven’t heard these terms in reference to software development, you may wish to plug your ears for a little while. The programs, and so forth, are mechanisms for organizing projects with respect to the business priorities associated with the development of one distribution of OpenSolaris. Second, those distribution groupings are likely to have a finite period of applicability whereas a community, because it will be identifying its contributors to the larger OpenSolaris membership, is expected to be a long lived group.

Finally, a community is expected to appeal to a fairly wide subset of the membership; it attempts to gather combine a collection of interests into a coherent group. That’s fuzzy and it’s made a bit worse by the fact that there are some communities that were grandfathered from being proposed prior to the introduction of a community proposal process or prior to the introduction of project hosting. Reorganizing some of the communities will be an area of effort over the next few months, but the basic guideline is propose your effort as a project.

Summary

I hope this review of typical developer activities, how they are changing as we develop in an open community, and how certain aspects of these activities are simply hard to do if your workstation is not on Sun’s network. None of these deficits is likely new to those Sun developers who work from a remote location or from a campus distant from California. And of course there are other aspects to collaboration that we need to explore, but that would be another talk entirely. It’s my intent that we get to a situation where a developer, without reference to affiliation or location—but possibly dependent on their available bandwidth—can participate equivalently in opensolaris development. That will require ideas, feedback, code, and action from all of us.

Thanks.

Your questions, please.