Category Archives: Software

Building node.js on OpenIndiana

More specifically, these instructions should let you build node 0.8.16, the current stable version, on oi_151a5:

$ uname -a
SunOS cooler 5.11 oi_151a5 i86pc i386 i86pc Solaris

(cooler is in its seventh year of service, having run many builds of Solaris, OpenSolaris, and, now, OpenIndiana.) First, you’ll need a GCC 4.x compiler. If you attempt to use the 3.4.3 gcc compiler, you’ll get

cc1: error: unrecognized command line option "-fno-tree-vrp"
cc1: error: unrecognized command line option "-fno-tree-sink"

in the output from your failed build. So, use the Illumos GCC 4.4.4 build, which you can install via

$ sudo pkg install developer/illumos-gcc developer/gnu-binutils

which led to the installation on my system, of 3 packages, and a total of 56.9MiB of content downloaded. Include these new tools in your path for the build:

$ export PATH=/opt/gcc/4.4.4/bin:/usr/gnu/bin:$PATH

To help the node build find the appropriate Standard C++ library for linking, we set the linker run path, via the environment. (By having a correct run path, our node binary won’t need LD_LIBRARY_PATH to be set to pick up libstdc++.so.6.) We can then configure, and issue the (GNU) make to start a build:

$ export LD_RUN_PATH=/opt/gcc/4.4.4/lib
$ CC=gcc ./configure --prefix=$HOME
$ CC=gcc gmake

You can test the resulting binary

$ ./node
> process.version;
'v0.8.16'
> ^D

and install the node platform to the configured location.

$ CC=gcc gmake install

And now you have a working node.js for your OpenIndiana system. npm is installed as well, so you can begin downloading the modules needed for your development. (If you’re running OmniOS, it looks like the “managed services” repository includes a pkg(5)-installable node.js package, so you can install that interpreter directly. Maybe that’s what cooler should run next.)

post-review returns HTTP 500 with cribbed repository configuration

We run ReviewBoard at work; we set up each hosted Git repository as new projects are started. I made a silly error a few weeks ago: when I created a new repository, I filled in the Mirror Path setting with that of a previous repository. This mistake leads to client output like

$ post-review 7fdd345e22783b289be128205cc0c47935057e20
Error creating review request: HTTP 500

Your administrator mail address should receive a message with a body like

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/Django-1.3.3-py2.7.egg/django/core/handlers/base.py", line 111, in get_response
  response = callback(request, *callback_args, **callback_kwargs)

File "/usr/lib/python2.7/site-packages/Django-1.3.3-py2.7.egg/django/views/decorators/cache.py", line 79, in _wrapped_view_func
  response = view_func(request, *args, **kwargs)

File "/usr/lib/python2.7/site-packages/Django-1.3.3-py2.7.egg/django/views/decorators/vary.py", line 22, in inner_func
  response = func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/Djblets-0.6.22-py2.7.egg/djblets/webapi/resources.py", line 397, in __call__
  result = view(request, api_format=api_format, *args, **kwargs)

File "/usr/lib/python2.7/site-packages/Djblets-0.6.22-py2.7.egg/djblets/webapi/resources.py", line 581, in post
  return self.create(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/ReviewBoard-1.6.11-py2.7.egg/reviewboard/webapi/decorators.py", line 127, in _check
  return view_func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/Djblets-0.6.22-py2.7.egg/djblets/webapi/decorators.py", line 88, in _checklogin
  return view_func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/Djblets-0.6.22-py2.7.egg/djblets/webapi/decorators.py", line 62, in _call
  return view_func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/Djblets-0.6.22-py2.7.egg/djblets/webapi/decorators.py", line 231, in _validate
  return view_func(*args, **new_kwargs)

File "/usr/lib/python2.7/site-packages/ReviewBoard-1.6.11-py2.7.egg/reviewboard/webapi/resources.py", line 5984, in create
  Q(local_site=local_site))

File "/usr/lib/python2.7/site-packages/Django-1.3.3-py2.7.egg/django/db/models/manager.py", line 132, in get
  return self.get_query_set().get(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/Django-1.3.3-py2.7.egg/django/db/models/query.py", line 351, in get
  % (self.model._meta.object_name, num, kwargs))

MultipleObjectsReturned: get() returned more than one Repository -- it returned 2! Lookup parameters were {}

each time a review is submitted against one of these repositories.

You can fix this condition by correcting the Mirror Path of the new repository; it’s not a ReviewBoard issue.

Bespoke services: site/redis

For prototyping web applications, I have recently come to rely on having Redis handy. In various sketches or early versions, I’ve used it to store event logs, to persist a collection of simple objects, or to conveniently manage a particularly large dictionary.

To make it easy to have a redis-server running on an OpenSolaris-derived system, I’ve written an smf(5) service manifest:

The default configuration of Redis is good enough for most prototyping scenarios, so this manifest assumes (a) that you’ve built and installed Redis to /usr/local, its default install location, and (b) are happy with the default configuration. In its default configuration, redis-server does not daemonize, and writes a log message every 5 seconds—you’ll very much want to change the latter if you move to production.

Exercises

  1. Add a property group and property to store a configuration location, and modify the start method appropriately. This enhancement should be on the service, such that it can be easily overridden on each instance. (*)

Irritating retrograde

A government agency I interact with has updated their web-based client software. The original application was a basic sequence of web forms. Its replacement? An approximately ~50MiB Silverlight-based application. In the process of the update, they discarded my original web account and password. The backend service that the application must communicate with is still slow, operating costs now include the bandwidth to update cached copies (for performance reasons), and the application itself has new usability issues. Because of the switch from standardized Web technologies to Silverlight, the majority of their customers can’t run the application on their phone or tablet. (If it were Flash, iPads would still be excluded.) How was this change an upgrade, again?

Bespoke services: site/supervisord

Recently, I’ve been experimenting with supervisor, which is a Python-based process restarter for Unix/Linux. Lincoln Loop recently offered instructions on running supervisor under upstart, which is applicable to some of the current Linux distributions. On OpenSolaris and related systems, the service management facility, smf(5), can be used to ensure your supervisors stay online. Below is a simple manifest that starts (and restarts) supervisord after a small set of services becomes available.

If you don’t provide a supervisor.conf in one of the standard locations, enabling this service instance will result in it heading immediately to the maintenance state, as the start method will fail repeatedly. You can use svcs -x to perform this diagnosis:

$ svcs -x                                                         ~
svc:/site/supervisord:default (supervisor process control system)
 State: maintenance since Sat Sep 04 16:34:14 2010
Reason: Start method failed repeatedly, last exited with status 2.
   See: http://sun.com/msg/SMF-8000-KS
   See: utmpd(1M)
   See: utmpx(4)
   See: /var/svc/log/site-supervisord:default.log
Impact: This service is not running.

The log file will contain a message, with some amount of repetition, like

[ Sep  4 16:34:13 Enabled. ]
[ Sep  4 16:34:13 Rereading configuration. ]
[ Sep  4 16:34:13 Executing start method ("/usr/bin/supervisord"). ]
Error: No config file found at default paths (/usr/etc/supervisord.conf, /usr/supervisord.conf, supervisord.conf, etc/supervisord.conf, /etc/supervisord.conf); use the -c option to specify a config file at a different path
For help, use /usr/bin/supervisord -h
[ Sep  4 16:34:13 Method "start" exited with status 2. ]

It’s worth noting that all of the programs run by a single instance of supervisord will be in the same process contract. If you know the fault characteristics of your programs, you may wish to use multiple instances of supervisord to keep programs with “sympathetic” failure modes and frequencies. You may also need to ignore core dumps and external signals, depending on the programs you are running; on recent systems, you can see /var/svc/manifest/network/http-apache22.xml for an example of a startd property group that does so. Alternatively, you could modify your configuration to run each of the programs to be started in independent contracts using ctrun(1).

Exercises

  1. We should really provide a property group that contains the key invocation settings as properties. I’ve omitted it here, particularly for the configuration file, because the method token expansion outlined in smf_method(5) lacks handling for unset property values. (*)
  2. Extend supervisord to understand process contracts. This exercise would include constructing a Python module to interact with the contract filesystem. (***)

adirent.[ch]: Adding d_type to struct dirent on OpenSolaris

An occasional porting problem you may encounter when compiling programs for OpenSolaris is the absence of d_type in the directory entry structure returned by readdir(3C). I hit this issue when experimenting with mu as a search solution for my accumulated email.

A trivial example of the failure you might see would be caused by the following program:

#include <sys/types.h>
#include <dirent.h>
#include <err.h>
#include <stdio.h>

int
main(int argc, char *argv[])
{
        DIR *d;
        struct dirent *e;

        if ((d = opendir("/")) == NULL)
                err(1, "opendir failed");

        for (e = readdir(d); e != NULL; e = readdir(d)) {
                    if (e->d_type != DT_UNKNOWN)
                        (void) printf("recognized filetype for '%s'\n",
                            e->d_name);
        }

        (void) closedir(d);

        return (0);
 }

When we attempt to compile this program with gcc, we get something like

$ gcc a.c
a.c: In function `main':
a.c:16: error: structure has no member named `d_type'
a.c:16: error: `DT_UNKNOWN' undeclared (first use in this function)
a.c:16: error: (Each undeclared identifier is reported only once
a.c:16: error: for each function it appears in.)

Studio cc will give similar output:

$ /opt/SunStudioExpress/bin/cc a.c
"a.c", line 16: undefined struct/union member: d_type
"a.c", line 16: undefined symbol: DT_UNKNOWN
cc: acomp failed for a.c

The addition of d_type to struct dirent came first for the BSD Unixes and was later added to Linux. Because it’s not easy to add members to well-known structures and preserve binary compatibility, OpenSolaris and Solaris lack this field, as well as the DT_* constant definitions. (If d_type were to become part of the Unix standards, Solaris would likely have to introduce a second family of opendir()/readdir()/closedir() functions and a second version of the structure, similar to how large files were introduced for 32-bit programs.)

Because we fail at compilation time, our workaround has to modify either the program’s source code or its build environment. (Preloading is too late.) It’s probably possible to combine a few definitions and a shared object that we include via LD_PRELOAD but it seems easier to just provide a C wrapper around readdir(3C) and an alternate struct dirent. We develop this approach in the next section.

DIRENT and READDIR

The approach we take is

  1. Introduce DIRENT and READDIR via adirent.h.
  2. Change the source program such that each call to readdir() is replaced by READDIR() and each use of struct dirent is replaced by DIRENT. In each file so modified, add a #include <adirent.h>.
  3. Compile adirent.c via gcc -I. -O2 -c adirent.c or equivalent.
  4. Add adirent.o to the link line for each binary that includes one of the files modified in step 2.

If we apply these steps to our example above, we get

#include <sys/types.h>
#include <adirent.h>
#include <err.h>
#include <stdio.h>

int
main(int argc, char *argv[])
{
        DIR *d;
        DIRENT *e;

        if ((d = opendir("/")) == NULL)
                err(1, "opendir failed");

        for (e = READDIR(d); e != NULL; e = READDIR(d)) {
                    if (e->d_type != DT_UNKNOWN)
                        (void) printf("recognized filetype for '%s'\n",
                            e->d_name);
        }

        (void) closedir(d);

        return (0);
 }

with the result that compilation and execution now work

$ gcc -O2 -I. -c adirent.c
$ gcc -I. a.c adirent.o
$ ./a.out

This shim function and definitions should be sufficient for most ports around this incompatibility, but there are some additional comments worth making.

Performance. Because many programs expect d_type to be one of DT_REG or DT_DIR to save on a stat(2) call, this shim will force those programs into an alleged “slow” path. The actual impact of returning DT_UNKNOWN on every call will be program- and situation-dependent; it didn’t seem to affect my mail indexing.

Multithreaded programs. The current implementation does not protect the static structure defined in adirent.c. Programs with multiple threads performing readdir(3C) calls through READDIR() will get unexpected results. It should be relatively straightforward to dynamically allocate one struct adirent for each thread coming through READDIR() for the first time.

Downloads

I suppose these should be in a repository on Bitbucket or GitHub. For now, they’re just simple downloads:

Acknowledgments

I discussed this problem with Dan, who in particular noted that DT_UNKNOWN was always a legal return value for d_type. Bart looked over my shoulder and spied at least one error during the debugging phase.

~4100MiB

I seeded the 2008.05 release candidate for about 45 hours, ultimately shipping a little over 4100 megabytes. I’m going to take a break, because I want to update my DP2-based workstation and get some work done, but, once we have new bits, I’ll getting seeding again.

(I found the actual result: 4237MiB sent up, so almost 4GiB.)

Stopping Firefox's restore session dialog

I’m sure that there are Firefox users out there who want to restore their previous session; I never do, and so deactivating that dialog is a big timesaver. If you search, you’ll find a few writeups on how to adjust the configuration, but I want to document the minimum steps to suppress the restore session dialog.

  1. Enter about:config into the URL field.
  2. Enter sessionstore into the displayed filter field and press Enter.
  3. Double-click the value cell of the browser.sessionstore.enabled row, so that the value field reads False.
  4. Restart Firefox. One click saved.

[ T: Firefox ]