To make it easy to have a redis-server running on an OpenSolaris-derived system, I’ve written an smf(5) service manifest:

• redis.xml

The default configuration of Redis is good enough for most prototyping scenarios, so this manifest assumes (a) that you’ve built and installed Redis to /usr/local, its default install location, and (b) are happy with the default configuration. In its default configuration, redis-server does not daemonize, and writes a log message every 5 seconds—you’ll very much want to change the latter if you move to production.

Exercises

1. Add a property group and property to store a configuration location, and modify the start method appropriately. This enhancement should be on the service, such that it can be easily overridden on each instance. (*)

• supervisord.xml

If you don’t provide a supervisor.conf in one of the standard locations, enabling this service instance will result in it heading immediately to the maintenance state, as the start method will fail repeatedly. You can use svcs -x to perform this diagnosis:

$ svcs -x                                                         ~
svc:/site/supervisord:default (supervisor process control system)
 State: maintenance since Sat Sep 04 16:34:14 2010
Reason: Start method failed repeatedly, last exited with status 2.
   See: http://sun.com/msg/SMF-8000-KS
   See: utmpd(1M)
   See: utmpx(4)
   See: /var/svc/log/site-supervisord:default.log
Impact: This service is not running.

The log file will contain a message, with some amount of repetition, like

[ Sep  4 16:34:13 Enabled. ]
[ Sep  4 16:34:13 Rereading configuration. ]
[ Sep  4 16:34:13 Executing start method ("/usr/bin/supervisord"). ]
Error: No config file found at default paths (/usr/etc/supervisord.conf, /usr/supervisord.conf, supervisord.conf, etc/supervisord.conf, /etc/supervisord.conf); use the -c option to specify a config file at a different path
For help, use /usr/bin/supervisord -h
[ Sep  4 16:34:13 Method "start" exited with status 2. ]

It’s worth noting that all of the programs run by a single instance of supervisord will be in the same process contract. If you know the fault characteristics of your programs, you may wish to use multiple instances of supervisord to keep programs with “sympathetic” failure modes and frequencies. You may also need to ignore core dumps and external signals, depending on the programs you are running; on recent systems, you can see /var/svc/manifest/network/http-apache22.xml for an example of a startd property group that does so. Alternatively, you could modify your configuration to run each of the programs to be started in independent contracts using ctrun(1).

Exercises

1. We should really provide a property group that contains the key invocation settings as properties. I’ve omitted it here, particularly for the configuration file, because the method token expansion outlined in smf_method(5) lacks handling for unset property values. (*)
2. Extend supervisord to understand process contracts. This exercise would include constructing a Python module to interact with the contract filesystem. (***)

A trivial example of the failure you might see would be caused by the following program:

#include <sys/types.h>
#include <dirent.h>
#include <err.h>
#include <stdio.h>

int
main(int argc, char *argv[])
{
        DIR *d;
        struct dirent *e;

        if ((d = opendir("/")) == NULL)
                err(1, "opendir failed");

        for (e = readdir(d); e != NULL; e = readdir(d)) {
                    if (e->d_type != DT_UNKNOWN)
                        (void) printf("recognized filetype for '%s'\n",
                            e->d_name);
        }

        (void) closedir(d);

        return (0);
 }

When we attempt to compile this program with gcc, we get something like

$ gcc a.c
a.c: In function `main':
a.c:16: error: structure has no member named `d_type'
a.c:16: error: `DT_UNKNOWN' undeclared (first use in this function)
a.c:16: error: (Each undeclared identifier is reported only once
a.c:16: error: for each function it appears in.)

Studio cc will give similar output:

$ /opt/SunStudioExpress/bin/cc a.c
"a.c", line 16: undefined struct/union member: d_type
"a.c", line 16: undefined symbol: DT_UNKNOWN
cc: acomp failed for a.c

The addition of d_type to struct dirent came first for the BSD Unixes and was later added to Linux. Because it’s not easy to add members to well-known structures and preserve binary compatibility, OpenSolaris and Solaris lack this field, as well as the DT_* constant definitions. (If d_type were to become part of the Unix standards, Solaris would likely have to introduce a second family of opendir()/readdir()/closedir() functions and a second version of the structure, similar to how large files were introduced for 32-bit programs.)

Because we fail at compilation time, our workaround has to modify either the program’s source code or its build environment. (Preloading is too late.) It’s probably possible to combine a few definitions and a shared object that we include via LD_PRELOAD but it seems easier to just provide a C wrapper around readdir(3C) and an alternate struct dirent. We develop this approach in the next section.

DIRENT and READDIR

The approach we take is

1. Introduce DIRENT and READDIR via adirent.h.
2. Change the source program such that each call to readdir() is replaced by READDIR() and each use of struct dirent is replaced by DIRENT. In each file so modified, add a #include <adirent.h>.
3. Compile adirent.c via gcc -I. -O2 -c adirent.c or equivalent.
4. Add adirent.o to the link line for each binary that includes one of the files modified in step 2.

If we apply these steps to our example above, we get

#include <sys/types.h>
#include <adirent.h>
#include <err.h>
#include <stdio.h>

int
main(int argc, char *argv[])
{
        DIR *d;
        DIRENT *e;

        if ((d = opendir("/")) == NULL)
                err(1, "opendir failed");

        for (e = READDIR(d); e != NULL; e = READDIR(d)) {
                    if (e->d_type != DT_UNKNOWN)
                        (void) printf("recognized filetype for '%s'\n",
                            e->d_name);
        }

        (void) closedir(d);

        return (0);
 }

with the result that compilation and execution now work

$ gcc -O2 -I. -c adirent.c
$ gcc -I. a.c adirent.o
$ ./a.out

This shim function and definitions should be sufficient for most ports around this incompatibility, but there are some additional comments worth making.

Performance. Because many programs expect d_type to be one of DT_REG or DT_DIR to save on a stat(2) call, this shim will force those programs into an alleged “slow” path. The actual impact of returning DT_UNKNOWN on every call will be program- and situation-dependent; it didn’t seem to affect my mail indexing.

Multithreaded programs. The current implementation does not protect the static structure defined in adirent.c. Programs with multiple threads performing readdir(3C) calls through READDIR() will get unexpected results. It should be relatively straightforward to dynamically allocate one struct adirent for each thread coming through READDIR() for the first time.

Downloads

I suppose these should be in a repository on Bitbucket or GitHub. For now, they’re just simple downloads:

• adirent.c
• adirent.h

Acknowledgments

I discussed this problem with Dan, who in particular noted that DT_UNKNOWN was always a legal return value for d_type. Bart looked over my shoulder and spied at least one error during the debugging phase.

(I found the actual result: 4237MiB sent up, so almost 4GiB.)

1. Enter about:config into the URL field.
2. Enter sessionstore into the displayed filter field and press Enter.
3. Double-click the value cell of the browser.sessionstore.enabled row, so that the value field reads False.
4. Restart Firefox. One click saved.

[ T: Firefox ]

That leaves Priority 1, the deployment of a public bug tracking system. Bug tracking has loomed over the OpenSolaris effort for pretty much its entire implementation phase; we’ve known that aspects of the current bug tracking methodologies impact many parts of Sun’s business, and that any solution will require the identification of which entanglements are strategic—meaning that there’s a requirement for any new system—and which are accidental—meaning that there’s only some transition cost, as the entangled system can be adjusted to consume information in some designed fashion. So, as part of getting a set of draft requirements together for discussion, I thought I would work through some of the issues facing defect tracking as we move to open development. Most of these points are about the developer–distribution (or upstream–downstream) interface that a defect tracking system (DTS) or defect tracking architecture represents.

The primary requirement that a distribution has on their DTS is some ability to maintain confidential data associated with each tracked defect. Let’s call that database a proprietary annotation system (PAS)—the data within it capture the customer histories associated with various defects or collections of defects (“products”?) represented in the system. The DTS, meanwhile, is meant to be neutral across all participants, developers and distribution assemblers, and unconcerned with non-technical characteristics of the defect.

This contrast allows us to postulate a set of relationships among various active DTSs and PASs for an open development community.

The association of confidential information with an existing defect looks something like

Of course, in the OpenSolaris case, SMI’s annotations become just one potential PAS; other distributions may also choose to annotate publicly known (“community tracked”) defects:

One requirement that we’ve heard during preliminary discussions with the various teams is there must be some ability to search the entirety of the product-relevant portion of the DTS. One possibility is that each PAS operator builds a search corpus that combines the upstream DTS with the PAS content. A potentially more economic alternative would be to allow the the associations to be bidirectional (so that an indexer with authorizations allowing it to access one or more proprietary annotation systems can present a complete defect corpus). Making the existence of the annotations public does not seem like a significant leak of proprietary information, while the existence of annotations might be a useful measure of defect significance. (It is probably worth explicitly stating that having a complete defect corpus for searching does not imply that use of a single DTS is the only means of obtaining such a corpus.)

These associations are more complex objects than the current See Also links in typical monolithic DTSes, in that they carry one or more mappings of status and responsibility between the DTS and each PAS. Potentially, this capability could lead to more precise handling of release readiness, in that a query involving a group of PAS-tracked defects could indicate that one or more stoppers are blocking the release of a certain version of the tracked product. Of course, prior to open development, the Solaris organization has historically managed that fairly well for its products, so why is it worth discussing?

Well, for OpenSolaris today, the set of defects in our own DTS isn’t the complete set of relevant defects for product release. In fact, as examples, Solaris tracks the defects, branches, and patches for GNOME, Mozilla Firefox, and Perl, among others. That is, Sun’s current combined DTS/PAS, Bugster, is in the following relationships with an upstream source

when the product is affected by an upstream-sourced component, and

when a customer is affected by an upstream-sourced component. There’s also an awkward local cost since upstream state is usually tracked manually (or possibly via custom scripting).

We can choose to apply this tracking need to our second defect/annotation figure above, in two ways: we can track the OpenSolaris consolidation defects as a peer of other upstream DTSs

or, more interestingly, we can allow community management of the relevance of upstream defects, like the following

(A distribution may not choose to annotate the public DTS record on an upstream bug, but may instead choose to annotate on the upstream DTS record directly. Ignoring the community signal when it has a mismatch with distribution priorities seems likely; using it as a guide when no conflicting principle exists seems like a safe course as well.)

I think that introducing these kinds of associations allows us to solve a large class of defect tracking problems that are currently impacting the OpenSolaris space. (Obviously there are issues, too–for instance, one could introduce association cycles (that clients must detect).) I would expect that the confidentiality issue impacts all of the distributions pulling from OpenSolaris consolidations (and probably, more generally, all groups looking at open development): I believe every distribution has customers of some kind. The importance of upstream relationships may presently be operating environment-specific, although I know other groups are also dependent on some number of OSS components.

I’ve had a small email toolset built around the Mutt email client (or mail user agent (MUA) in email-speak). My .muttrc includes $HOME/.mailboxes, which is a list of the mail inboxes into which my procmail filter delivers. The m and check commands use .mailboxes to determine where new messages have arrived. Happily, this infrastructure can easily give me a few hours of reduced interrupts, so that I can get some thinking and writing done.

The idea is to strongly reduce the number of significant inboxes during what we’ll label as “prime” work hours, between lunch and dinner on weekdays. Given the use of .mailboxes outlined above, the solution is pretty easy:

1. Move .mailboxes to .mailboxes.offprime.
2. Take a copy of .mailboxes.offprime as .mailboxes.prime. Prune the prime mailboxes severely.
3. Install these files in $HOME. (My Makefiles do this, and all the mailbox lists and the Makefiles are version-controlled.)
4. Add something like the following using crontab -e:

   # We want to eliminate mail testing on all mailboxes during prime work
   # hours.  This change means that between 12pm - 6pm, mailboxes is
   # pointed to a reduced mailboxes file.
   0 12 * * 1,2,3,4,5 /usr/bin/ln -sf /home/sch/.mailboxes.prime /home/sch/.mailboxes
   0 18 * * 1,2,3,4,5 /usr/bin/ln -sf /home/sch/.mailboxes.offprime /home/sch/.mailboxes
   

5. Run whichever of the two ln(1) invocations above correspond to the current time.

You’re now ready to relax and enjoy reduced interrupt rates, leaving you time to focus on real work… or write blog postings about managing your interrupt rates.

[ T: OpenSolaris Solaris mutt procmail ]

I’ve had a small email toolset built around the Mutt email client (or mail user agent (MUA) in email-speak). My .muttrc includes $HOME/.mailboxes, which is a list of the mail inboxes into which my procmail filter delivers. The m and check commands use .mailboxes to determine where new messages have arrived. Happily, this infrastructure can easily give me a few hours of reduced interrupts, so that I can get some thinking and writing done.

The idea is to strongly reduce the number of significant inboxes during what we’ll label as “prime” work hours, between lunch and dinner on weekdays. Given the use of .mailboxes outlined above, the solution is pretty easy:

1. Move .mailboxes to .mailboxes.offprime.
2. Take a copy of .mailboxes.offprime as .mailboxes.prime. Prune the prime mailboxes severely.
3. Install these files in $HOME. (My Makefiles do this, and all the mailbox lists and the Makefiles are version-controlled.)
4. Add something like the following using crontab -e:

   # We want to eliminate mail testing on all mailboxes during prime work
   # hours.  This change means that between 12pm - 6pm, mailboxes is
   # pointed to a reduced mailboxes file.
   0 12 * * 1,2,3,4,5 /usr/bin/ln -sf /home/sch/.mailboxes.prime /home/sch/.mailboxes
   0 18 * * 1,2,3,4,5 /usr/bin/ln -sf /home/sch/.mailboxes.offprime /home/sch/.mailboxes
   

5. Run whichever of the two ln(1) invocations above correspond to the current time.

You’re now ready to relax and enjoy reduced interrupt rates, leaving you time to focus on real work… or write blog postings about managing your interrupt rates.

[ T: OpenSolaris Solaris mutt procmail ]

I’m a creature of Vim, and am quite happy with the recently released Vim 7. Tying the external editor feature of Wikipedia to Vim is straightforward, if you install Erik Möller’s ee script—and tweak it a little.

Below you’ll find a modified version that respects the http_proxy setting, handles the case where no matching site is present in your ee.ini file, and adds a configuration option to not append the “using an external editor” string to your article’s change summary.

• ee.pl
• ee.pl Hg export

[ T: OpenSolaris Solaris genunix Mediawiki ]