BSD Planet * BSD People

Introducing the ATF nofork branch

Despite my time for free software being virtually zero these days, I have managed to implement a prototype of what ATF would look like if it didn't implement forking and isolation in test programs. This feature has been often requested by users to simplify their life when debugging test cases.<div><div><br /></div><div>I shouldn't repeat everything I posted on the atf-devel mailing list regarding this announcement, so please <a href="http://mail-index.netbsd.org/atf-devel/2010/03/05/msg000070.html">refer to that email for details</a>. But I must say that the results look promising: the overall code of ATF is much simpler and also faster. (An execution I just tried cuts the run time of the ATF test suite from 1m 41s to 1m 16s.) Expect more simplifications and speed-ups!</div></div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-1501363386005733392?l=blog.julipedia.org' alt='' /></div></content>

by Julio Merino at March 06, 2010 08:48 PM

February 28, 2010

by jdarnold at February 28, 2010 06:30 PM

Do It Sudo

I've come across a couple sudo tricks during the past few days and thought I would pass them along. The first thing you need to know before using sudo is that you should use the visudo command to edit the sudo config file. This script does a number of useful...

February 22, 2010

Google Summer of Code 2010 NetBSD swcryptX Project Suggestion (Updated)

I've been thinking of a neat-o project for this year's Google Summer of Code:

Abstract:

Overview of operation

The opencrypto(9) framework exists to coordinate hardware acceleration in NetBSD. Applications of the framework can be inside the kernel like the FAST_IPSEC IPsec implementation, or in userland like OpenSSL with the "cryptodev" engine. Crypto drivers can be realized in software or in hardware. Hardware drivers can be used to instruct e.g. the AMD Geode LX's AES block or a HIFN chip to perform cryptographic operations. Upon system startup, the crypto drivers at the opencrypto(9) framework, telling what operations they can perform. When an operation is required later, the framework will look which crypto device is currently not busy, and offload the operation to that device. Upon completion, the result is fed back to the application.

The following image illustrates the components and their interaction.

Limitations

Offloading the cryptographic requests involves some overhead. Data needs to be transferred to the hardware and back. On systems with a slow CPU, this overhead is relatively small compared to the operation speed of the CPU. On faster CPUs, the overhead becomes more of a burden, making the benefit of the crypto hardware negligible.

As examples, while a hifn(4) chip can provide worthwhile speedups on 500MHz and 1GHz CPUs, no performance win is experienced on a 2.4GHz CPU.

Proposal

The communication overhead involves data transfers over a PCI bus, which is of relatively low speed compared to today's modern CPUs. Preventing the data transfer is a worthwhile goal. In coordination with today's modern multi-core CPUs, using one or more CPUs solely for the purpose of crypto acceleration, a measurable improvement of crypto performance is expected. At the same time, no special hardware requirements beyond the CPU exist. This allows turning standard contemporary systems into fast crypto systems easily.

The following image illustrates the idea of interoperation between a CPU core that runs the kernel and application codes and three cores that are dedicated to crypto code.

Implementation Roadmap

This is where it gets fishy. ;) ~~The existing opencrypto(4) framework probably needs to be make MP-aware at the same time, employing proper use of NetBSD's locking framework.~~ (Already done) The existing swcrypto(4) needs to be adjusted for operation on multiple CPUs at the same time. A way to decide how many CPUs are dedicated to run swcrypto(4) instances. CPUs that run swcrypto(4) need to be taken out from the usual NetBSD CPU scheduling so that they are available exclusively for crypto.

Requirements

In no particular order:

Know how to build and install a kernel
Understanding of fine grained SMP and locking
How to use NetBSD's kernel threads, code-wise
How to interact with NetBSD's scheduler, code-wise
Tell the scheduler to pin a specific kernel thread to a specific CPU
Interaction between applications (IPsec, OpenSSL) with opencrypto(9), code-wise
Interaction of crypto providers with opencrypto(9), code-wise
Hardware! You won't be able to do this without at least two CPU cores in your machine. The more the better.
Benchmarking & a test setup for it

Project Applications

Please follow the NetBSD Project Application/Proposal HowTo if you're serious to work on this project.

If you have any questions let me know, public discussion should be led on the tech-crypto@ list.

February 22, 2010 10:57 PM

February 21, 2010

More news from NetBSD land: inside and outside the kernel

My inbox is still overflowing with NetBSD related news, so here is the next chunk for you:

People owning a Marvell PCMCIA WiFi card can how have a look at the malo(4) driver. If you use this on a gumstix ARM board, have a look as some fiddling with drvctl(8) is needed to get the card recognized properly.
Speaking about Gumstix, KIYOHARA Takashi has declared the porting effort as finished. Support for the baex, connex, verdex and verdex-pro modules is available, as is support for the support modules.
Staying in the "embedded" corner, KIYOHARA Takashi has announced that Plathome's OpenBlockS600 (AMCC 405EX) can now boot NetBSD via NFS. See the posting for dmesg output. The OpenBlockS600 comes with a AMCC 405EX PowerPC CPU, two GigE ethernet ports and a bunch of other goodies for a price of about $600US.
Coming from hardware to software, pkgsrc-2009Q4 was released some time ago, and of course binary packages are available for a number of platforms: 5.0/macppc, 4.0.1/sparc and 5.0.1/sparc, 4.0.1/i386, 5.0.1/i386 and the same for 4.0.1/amd64 5.0.1/amd64. Also, binaries of pkgsrc-2009Q3 are available for 5.0/shark.
Google's Summer of Code was a big success for NetBSD and all of the Open Source community, and it seems there will be one again this summer.
Preparations are in an early stage, but there are already a FAQ and a timeline as well as the Program Terms of Service. From the NetBSD side, we're always happy for project suggestions (please use our mailing lists for discussions), and in NetBSD, we are currently working on out projects page. If you plan to submit a proposal for a project with NetBSD, please see our project application/proposal form If you plan to submit a proposal for a project with NetBSD, please see our Project Application/Proposal HowTo.
Getting back to the NetBSD code, a number of interesting changes were made in the previous weeks. The first to mention is that David Young has continued is work on the new shutdown order for device drivers: ``cgd, dk, dm, md, raid, and vnd gracefully detach from the device tree during shutdown. I believe that ccd is the only virtual disk that does not detach.'' This allows having arbitrary stack of file systems, and still have them unconfigured properly in the right order on system shutdown.
Another major change that went into NetBSD recently is that terminfo was imported into NetBSD-current. Terminfo replaces termcap, but provides a backward compatible termcap interface. This move follows discussion from last summer, and docs by The Open Group (the people who make things like the POSIX standard and the Single Unix Specification), which indicates that the termcap specification will be withdrawn in the future.
Moving from userland inside the kernel, David Holland has proceeded with work to unhook LFS from UFS. Historically, the Log structured File System was written after the Berkeley Fast File System. With the idea of sharing the core "Unix File System" code for both file systems, this resulted in a strong relationship between LFS and FFS, which was/is not always the best for the advantage and stability of either one: ``sharing ufs between both ffs and lfs has made all three entities (but particularly lfs) gross. ffs and lfs are not similar enough structurally for this sharing to really be a good design.''
Another major addition to the NetBSD kernel was made recently by Darren Hunt: ``Courtesy of CoyotePoint Systems, I've been working on a port of DTrace [...] to NetBSD for i386.'' Citing Wikipedia, ``DTrace is a comprehensive dynamic tracing framework created by Sun Microsystems for troubleshooting kernel and application problems on production systems in real time. Originally developed for Solaris, it has since been released under the free Common Development and Distribution License (CDDL) and has been ported to several other Unix-like systems.
DTrace can be used to get a global overview of a running system, such as the amount of memory, CPU time, filesystem and network resources used by the active processes. It can also provide much more fine-grained information, such as a log of the arguments with which a specific function is being called, or a list of the processes accessing a specific file. ''
The code is available in NetBSD-current. I haven't looked into this yet, but I'm looking forward of reports and blog postings if the wikipedia command line examples work.
(I think like with ZFS, Dtrace could use a hand with documenting the NetBSD side of things. Any takers?)
The last kernel change to mention is related to security: mapping the address 0 from userland was disabled. This issue went through the press late last year, and it this is now addressed in NetBSD, too. Those that still beed to map address 0 can do so via the USER_VA0_DISABLED_DEFAULT kernel option or the vm.user_va0_disable sysctl.
So much about about the NetBSD code for now. Of course having all those fine features added screams for an immediate (*cough*) release, which brings me to the fact that NetBSD 5.0.2 has been released: `` NetBSD 5.0.2 is the second critical/security update of the NetBSD 5.0 release branch. It represents a selected subset of fixes deemed critical for security or stability reasons.
Please note that all fixes in critical/security updates (i.e., NetBSD 5.0.1, 5.0.2, etc.) are cumulative, so the latest update contains all such fixes since the corresponding minor release. These fixes will also appear in future minor releases (i.e., NetBSD 5.1, 5.2, etc.), together with other less-critical fixes and feature enhancements. ''

I'm closing for today by pointing to three NetBSD-related events:

There's a hackathon going this weekend (Feb 20/21 2010)
Volunteers are wanted to setup & man a NetBSD booth at FrOSCamp 2010 Zurich, Switzerland, on Sep 17/18 2010
pkgsrcCon 2010 will be held in Basel, Switzerland, from May 28ths to 30ths 2010.

Have fun meeting the gang!

February 21, 2010 02:20 PM

February 15, 2010

Setting up a PPPoE server with NetBSD

I had an urge to configure PPPoE today, but didn't have a DSL line handy to test it. So, with some help from martin@, I got to prod NetBSD into becoming a PPPoE server.

I've put my notes here.

Open issues:

Whenever I terminate a client session ("ifconfig pppoe0 down"), the server's pppoe0 interface goes down as well. Before new connections are served, I have to do a manual "ifconfig pppoe0 up".
When I add "query-dns=3" to the client side, the connection is terminated immediately after it's established. I don't know how to offer DNS servers via PPPoE/LCP, but at least just leaving the connection open instead of terminating it would be nice.

I understand this method is only for one/few PPPoE connections. If more sessions are needed, this may not be the best option. It still served my purpose well. YMMV.

February 15, 2010 09:56 PM

by jdarnold at February 15, 2010 03:54 PM

Org-ing in Emacs

I have been using emacs since the beginning of computer time, way back in the mid-80s. My .emacs file has been carried around for almost as long. Each major rev causes me headaches, as I never know what archaic options are going to break things. And yet, I'm still learning...

February 12, 2010

Musing about git's object store efficiency

I'm currently looking at git to see what it can and cannot do, and one thing I've looked today is how effective the backing store mechanism is. To recall: CVS stores a list of patches between versions in a single file, and git stores each new revision in full in a separate file in the so-called object store. Is that an issue for NetBSD? Let's see;

One of the more frequently updated files is the i386 port's GENERIC kernel config file, which is at revision 1.963 right now. This means that since it's import into CVS, 963 different revisions have been made. In CVS, all those files are kept in a single GENERIC,v file. In git, this puts 963 files on the file system. A bit of a difference.

Looking at the space requirements for storing the repository data itself, the GENERIC,v file is 883,233 bytes[1]. Extracting all 963 versions from revision 1.1 to revision 963 results in disk space usage of 32,805,828 bytes[2,3]. And that's not counting the overhead of 962 inodes and the related directory bookkeeping.

In other words, the git model requires about 37 times the space that CVS does.

Sure the example file is not exactly one with an average number of revisions, and I know that git offers some more efficient storage methods via "pack" files, but investigating those is left as an exercise to the reader. :-)

[1] Obtained via rsync from cvs.netbsd.org:

% ls -la GENERIC,v 
-r--r--r--  1 feyrer  wheel  883233 Feb 12 16:57 GENERIC,v

[2]

% mkdir extracted
% chdir extracted
% sh -c 'for i in `jot 964`; do echo $i ; co -p -r1.$i ../GENERIC >GENERIC-`printf %04d $i` ; done'

[3]

% cat extracted/* | wc -c
 32805828

February 12, 2010 05:26 PM

February 08, 2010

by jdarnold at February 08, 2010 06:07 PM

Tumblings, Jan 24

A couple of links posted to my Linux Tumblr last week: Wicked Cheap Hosting - some great web hosting deals, courtesy of All About Linux TermBuilder - a simple web app to build a commandline command...

February 07, 2010

by jdarnold at February 07, 2010 03:37 PM

Arched

So early in December, I went on a quest for a new Linux distro. It wasn't so much that I was unhappy with my openSUSE 11.0 installation, but I knew I was probably going to install the new 11.2 version and so I figured I would cast about to see...

February 05, 2010

Hackathon, February 19-22 2010

Aleksej Saushev writes on a bunch of lists:

We're running 13th Hackathon February 19-22 2010,
come and join us on IRC channel #netbsd-code at FreeNode (irc.freenode.net).
You may choose other ways to participate, if you find it more convenient.
See Hackathon page for updates: http://wiki.netbsd.se/Hackathon13


Goal

Fix as many bugs as possible, close as many PRs as possible.


Details

In previous years we have accumulated a lot of problem reports.
Some of them relate to no longer supported branches (e.g. 2.0)
and were occasionally fixed during regular code work.
Some of them relate to hardware developers don't have access to.
Some of them may be too easy to fix, but noone looks at that part
of code (e.g. documentation).
Some of them relate to packages already removed or heavily reworked.
Some of them relate to packages in exotic environment (Solaris, Interix)
and developers cannot test if the bug is present or not.

You can find more in PR database at http://netbsd.org/support/query-pr.html


More specifically, David Holland maintains categorized lists of PRs at

  http://www.netbsd.org/~dholland/buglists/

of which "Wanted for 5.1" 

(http://www.netbsd.org/~dholland/buglists/51-WANTED.html)
and "Stuck" (http://www.netbsd.org/~dholland/buglists/STUCK.html)
are of particular interest.

Another hot area is pkgsrc 
(http://www.netbsd.org/~dholland/buglists/pkgsrc.html).


IMPORTANT NOTE: You don't have to be programmer to be able to help us,
you can help us with generic testing support.

Come, bring your favourite booze, and join in!

February 05, 2010 11:00 PM

February 02, 2010

by jdarnold at February 02, 2010 08:46 PM

Linux Love Links

A few things I have recently thrown down on my Linux Love Tumblr blog, which I use for quick little Linux links and notes: 7 Best Linux Apps for Ripping CDs and DVDs | Maximum PC 50+ Ultimate Collections of Planet Wallpapers TermBuilder: a graphical Linux command line generator...

Turn your ALIX into KITT (Updated)

Drop the following script into /etc/rc.d/alixkitt, put "alixkitt=yes" into /etc/rc.conf and enjoy:

#!/bin/sh
# Turn ALIX2c front LEDs into running LED
# http://www.gifninja.com/Workspace/59f3f22e-5f40-4937-936c-1dc0d6fbe690/output.gif
#
# Copyright (c) 2010 Hubert Feyrer <hubertf@NetBSD.org>

# PROVIDE: alixkitt

$_rc_subr_loaded . /etc/rc.subr

name="alixkitt"
start_cmd="alixkitt_start"
stop_cmd="alixkitt_stop"
pidfile="/var/run/${name}.pid"

alixkitt_sleeptime=1.0          # 0.5

alixkitt_setled()
{
        gpioctl -q -d /dev/gpio 6 $1
        gpioctl -q -d /dev/gpio 25 $2
        gpioctl -q -d /dev/gpio 27 $3

        sleep $alixkitt_sleeptime
}

alixkitt_start()
{
	touch $pidfile
        (
                alixkitt_setled 0 0 0

                while [ -f $pidfile ] ; do
                        alixkitt_setled 0 1 1
                        alixkitt_setled 1 0 1
                        alixkitt_setled 1 1 0
                        alixkitt_setled 1 0 1
                done
        ) &;
        echo $! >$pidfile
        echo Started pid $!
}

alixkitt_stop()
{
        if [ -f /var/run/alixkitt.pid ]; then
                read pid <$pidfile
                echo Stopped pid $pid
                kill $pid
                rm $pidfile
                alixkitt_setled 1 1 1
        fi
        exit 0
}

load_rc_config $name
run_rc_command "$1"

P.S.: There's still a bug left in that stopping immediately re-starts the process. Got a patch? Update: Axel Scheepers suggested doing the while-loop only as long as the PID-file exists. Changed above.

February 02, 2010 10:18 AM

February 01, 2010

by jdarnold at February 01, 2010 02:28 PM

On Being Persistent

Those of us who have multiple hard drives in our computers will inevitably boot up one morning to find the naming scheme for these drives has changed. What was once /dev/sda is now /dev/sdb and vice versa. Your computer won't boot and fsck complains about an uknown or mismatched filesystem...

January 31, 2010

January 31, 2010 11:50 PM

Toggling Alix2c LEDs

I've found that the three LEDs at the front of a PC Engines Alix2C board can be toggled on and off via their attachment to the GPIO bus. Here are the commands for this:

left:	gpioctl -d /dev/gpio 6 2
middle:	gpioctl -d /dev/gpio 25 2
right:	gpioctl -d /dev/gpio 27 2

Unfilling my inbox: NetBSD news from the past few weeks - ACPI, NUMA, Xen, and more

Herre are some more things that I've caught in my inbox for too long, and I'm finally finding some time to sum them up here:

NetBSD's "let's move kernel parts to the userland" RUMP project is still under heavy development, and in order to make testing of compatibility after kernel changes easier, a new command "rumptest" was added to build.sh: ``Basically you say:
./build.sh ${yourargs} tools ; ./build.sh ${yourargs} rumptest
Where yourargs are what have you, e.g. '-U -u -o -O /objs'.
The latter builds only the rump kernel libs and uses some ld+awk magic to figure out if things go right or not. This is to avoid having to install headers and build libs (which is too slow since a full build is too slow). The magic is not a substitute for a full build, but it is n+1 times faster and works probably 99.9% of the time.
The scheme uses a number of predefined component sets (e.g. tmpfs+vfs+rumpkern) to test linkage. They are currently listed in build.sh. This area probably needs some work in the future. It would be nice to autogenerate the combinations somehow.
If things go well, you get something like this:

===> Rump build&link tests successful ===> build.sh ended: Wed Nov 18 20:10:59 EET 2009
'' See Antti's Antti's mail to tech-kern: on how to tell if things didn't go so well, and what to do in that case.
According to Wikipedia, ``Non-Uniform Memory Access or Non-Uniform Memory Architecture (NUMA) is a computer memory design used in multiprocessors, where the memory access time depends on the memory location relative to a processor. Under NUMA, a processor can access its own local memory faster than non-local memory, that is, memory local to another processor or memory shared between processors.''
Supporting NUMA in a contemporary (i.e.: Intel centric) SMP-enabled operating system requires following a bunch of standards, two of which are parsing of two tables, the System Resource Affinity Table (SRAT) and the System Locality Information Table (SLIT). Both tables are accessible via the Advanced Configuration and Power Interface (ACPI), and according to the German-language Wikipedia, the SRAT is used to assign local memory to local threads to boost their performance, and the SLIT defines the "distance" of the nodes among themselves, which is used to determine the "nearest" memory if local memory is not enough.
Now, Christop Egger has posted patches to add an ACPI SLIT parser and an ACPI SRAT parser. See the two postings for dmesg pr0n from his tests on an 8-node system.
Staying with ACPI and Christoph Egger, he found that even though the ACPI spec defines an ACPI device for fans, BIOS vendors and OEMs do their own thing. To accommodate things like the fan sensor found in the ACPI Thermal Zone in his HP Pavillion DV9700 laptop he has proposed a driver to extend the acpitz(4) driver with fan information. That way, envstat(8) can be used to display the ran's RPMs:
```
[acpitz0]
  Processor Thermal Zone:     56.000   95.000                       degC
                     fan:       2840                                 RPM 
```
Staying with driver games, iMil writes me that there's documentation on getting DRI, AIGLX, Composite and Compiz going with NetBSD 5.0 available in the O(ther)NetBSD Wiki now.
The documentation covers how to enable the Direct Rendering Manager (DRI), setting up and configuring Modular X.org, assuring that everything's in place, and how to get Compitz going. Mmm, wobbly windows at last! :-)
While we're talking funky desktop stuff: Marc Balmer has submitted a patch to get touchpanel support for ums(4). ums(4) is for USB mice, and in contrast to mice, touch panels need to deal with absolute numbers, not relative numbers.
Back to the guts of the kernel, another patch suggested by Christop Egger was for adding x2apic. What is x2apic? X2APIC is ``an Intel-only feature but can also be found in virtual environments with support for CPU apic id's > 0xff.
I.e. Xen 4.0 (not yet released) supports 128 CPUs in HVM guests with the CPUs enumerated with even apic id's. That means you need x2apic for the 128th CPU :) ''
While speaking of Xen: Xen 4.0 is coming soon, and there's a call to help testing it on NetBSD!
Install Mercurial, check out latest Xen sources, apply a bunch of patches, build and install. Examples of commands are given, in addition to changes required for /boot.cfg etc.
Report your findings to port-xen!
Last one for today: Michal Gladecki, Editor-in-Chief of BSD Magazine writes: ``We are happy to announce that BSD Magazine is transforming into a free monthly online publication. The online version of BSD Magazine will stay in the same quality and form. It will look like the BSD magazine one is familiar and comfortable with. Please sign up to our newsletter at www.bsdmag.org and get every issue straight to your inbox. Also, you can now download any of the previous issues from our website. The first online issue -- 2/2010 -- is coming out in February. Please spread the word about BSD Magazine. '' Click!

So much for today. I still have a bunch of news items in my inbox for next time, but let's call it good for today.

Unrelated, I've been playing with git a bit over the past few days, and wile I have a number of questions building up (which will be subject to tech-repository or so), what I can say today is that the speed of "git pull" with NetBSD's git repository and my 1MBit DSL line reminds me a lot of the times when I used SUP with my 56k modem - it took forever, too. :-(

January 31, 2010 04:54 PM

January 26, 2010

January 26, 2010 01:48 AM

cssh - cluster ssh

Sometimes you need more than one

January 24, 2010

by Julio Merino at January 24, 2010 10:48 PM

set -e and set -x

If you write shell scripts, you definitely need to know about two nice features that can be enabled through the <tt>set</tt> builtin:<div><ul><li><tt>set -e</tt>: Enables checking of all commands. If a command exits with an error and the caller does not check such error, the script aborts immediately. Enabling this will make your scripts more robust. But don't wait until your script is "complete" to set the flag as an afterthought, because it will be a nightmare to fix the scrip to work with this feature enabled. Just write <tt>set -e</tt> as the very first line of your code; well... after the shell bang.</li><li><tt>set -x</tt>: If you are writing simple scripts that are meant to, well, script the execution of a few tasks (as opposed of being full-flown programs written in shell), set this flag to trace the execution of all commands. This will make the interpreter print each command right before it is executed, so it will aid you in knowing what is happening at any point in time.</li></ul></div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-8947551254574129862?l=blog.julipedia.org' alt='' /></div></content>

January 23, 2010

Problems starting a jail with ezjail

Things change. We must evolve.

January 23, 2010 12:48 AM

January 19, 2010

A colorful collection of NetBSD news from the past few weeks

AKA "I've been slacking again, and there's a whole pile of stuff here now that I'm putthing into one blog posting". Here we go:

Initial support for the FriendlyArm Mini2440 board has been announced by Paul Fleischer. In a later update, most of the hardware is reported working, and the patch is available for review & comments. Furthermore, the touch screen is usable, and Qt/Embedded was built on top of wscons.
Force10 Networks Receives Common Criteria Security Certification for Its High-Performance Ethernet Switch/Router Products. According to the article, ``Common Criteria evaluations entail formal rigorous analysis and testing to examine security aspects of a product or system. Extensive testing activities involve a comprehensive and formally repeatable process, confirming that the security product functions as claimed by the manufacturer. Security weaknesses and potential vulnerabilities are specifically examined during wide-ranging evaluation and testing.
FTOS is the operating system software that runs on Force10 switch/router product lines, including the E-Series, C-Series and S-Series platforms. Based on NetBSD, FTOS leverages a distributed, multiprocessor architecture that delivers highly scalable protocols and reliability. By delivering the same OS across its entire switch/router line, Force10 ensures that customers benefit from stable code, a consistent configuration environment and simpler software management. ''
While there: Force10 Networks Delivers Ethernet-Optimized Platform for MPLS Core Networks: ``Force10 Networks, Inc. [...] announced the immediate availability of MPLS (multi-protocol label switching) functionality for its ExaScale E-Series core switch/routers. [...] The ExaScale platform combines high-density, non-blocking, line-rate 10 GbE switching and routing with robust MPLS LSR support at 1/5th of the cost of a traditional core router, enabling carriers to fully capitalize on the economic advantages of Ethernet.''
Create Bootable Live Linux USB Drives with UNetbootin: ``UNetbootin allows you to create bootable Live USB drives for a variety of Linux distributions from Windows or Linux, without requiring you to burn a CD. You can either let it download one of the many distributions supported out-of-the-box for you, or supply your own Linux .iso file if you've already downloaded one or your preferred distribution isn't on the list.''
And of course we all know that NetBSD is Linux, right? See the list of supported distributions:

The homepage mentions that NetBSD 4.0 is supported, maybe someone wants to give them an update on what's up with NetBSD 5.0? Would be nice to see that on the list!
Ever wondered what happened with the BSD Certification recently? There's a video from the talk BSD Certification Group: A Case Study in Open Source Certification available that talks about the goal of the project, the two exams (BSD Associate, BSD Professional), and what's going on behind the scenes.
With the move from XFree to X.org, the X server for the DNARD Shark's NetBSD/shark lost support for accelerated X. Thanks to Michael 'macallan' Lorenz, hardware-accelerated X for NetBSD/shark is back now: ``I finally got around to start working on an Xorg driver for the IGS CyberPro 20x0 family found in rev. 5 Sharks, Netwinder etc. - currently the driver is built only on shark and supports only the VL variant found there. Adding support for PCI chips is trivial though, just needs extra probing. The driver supports autoconfiguration ( X -configure should yield something almost usable, only DefaultDepth needs to be adjusted).''
Staying with cool platforms, here's a quick procedure to run NetBSD/sun2 5.0.1 on The Machine Emulator (TME) (see pkgsrc/emulators/tme), compiled by Izumi Tsutsui. Who's first do get a pkgsrc bulk build done? :-)
Jed Davis has committed the RAIDframe parity Summer-of-Code project. See his posting for the details. The project ``drastically reduces the time RAIDframe spends rewriting parity after an unclean shutdown by keeping better track of outstanding writes (thus, "parity map"). The tech-kern archives have more details [...]
This feature is enabled by default on all sets (other than RAID 0). It can be administratively disabled with the new "raidctl -M" flag, which is described in the changes to the raidctl(8) man page; however, the I/O overhead for updating the parity map is expected to be unnoticeable in practice.''

So much for now. There is more in the pipe, but that will have to wait for now. Good night!

January 19, 2010 03:17 PM

Integrating the Tape Library with an existing Bacula installation

Adding a new tape drive isn't always simple

January 19, 2010 12:48 AM

January 18, 2010

January 18, 2010 12:48 AM

Tape Libraries & Bacula

Tape libraries can be complex to set up

January 17, 2010

January 17, 2010 10:47 PM

NetBSD and ISO 8859-2

Flying by one one of our mailing lists, I found this gem: ``This short article aims to help people that want to use ISO8859-2 character encoding in NetBSD/i386's wscons(4) console. It is very brief, so if you understand Slovak language, I recommend you to read NetBSD Slovak & Czech HOWTO instead. ''

The article describes how to teach wscons to use a ISO 8859-2 font, and also how to adjust the wscons keymap to either a czech or slovak keyboard mapping.

I think that gives support for ISO 8859-1, -2 and -15, so now where are your patches for the rest?

January 13, 2010

Hardware crypto with Geode LX based ALIX board

I'm using an Alix 2d13 machine as home router for quite some time now - uptime was 158 days today. The board has a 500MHz AMD Geode LX800 CPU, which can do AES crypto in hardware. Today, I've found time to play with the glxsb(4) driver, and get some numbers:

The 'numbers' are in 1000s of bytes per second processed.
crypto   type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
sw       aes-128-cbc       3583.29k     3931.51k     4037.32k     8113.75k     8205.61k
hw       aes-128-cbc       1200.70k     4470.18k    11729.65k    17328.05k    34006.33k

The command "openssl speed -evp aes-128-cbc -elapsed" was ran each time, in the first case with glxsb(4) disabled (boot -c, disable glxsb), in the second case with the driver enabled.

Still, I find those numbers interesting in comparison to those of a Soekris 5501, esp. as the machines have the same CPU & clock rate.

I couldn't find a way to switch use of hardware crypto off in software, anyone got a hint? openssl(1)'s "-engine cryptodev" seems to always use hardware crypto when it's there, and leaving out that switch uses the same engine, so no difference. I've found a few sysctls under kern.*, but apparently none seems relevant to my goal (not to speak of the lack of documentation...)

January 13, 2010 04:55 PM

January 12, 2010

sysupdate - an addition for etcupdate(8) (Update #2)

I wanted to update a few machines with latest netbsd-5 and -current today, and found that there's a certain lack of tools to aid in the process: we have etcupdate(8) and postinstall(8), but nothing to help extract the userland sets (let alone do the same thing for the kernel, esp. now that modules must be put into place, too).

To save me from re-typing the same commands, I've hacked a script "sysupdate" that's intended to run with a new kernel booted, and which will then 1) extract all the sets where mtree files are in /etc/mtree, and 2) run etcupdate for etc (and xetc, if in /etc/mtree).

Here's an example run:

    % sh ~/tmp/sysupdate -s /usr/cvs/src-current/obj.i386/releasedir/i386/binary/sets -n
    ---------------------------------------------------------------------------
    Updating your userland now!
    ---------------------------------------------------------------------------

    New userland: NetBSD 5.99.23 (GENERIC) #0: Sun Jan 3 02:13:00 CET 2010
    Current kernel: NetBSD vmnetbsd6 5.99.23 NetBSD 5.99.23 (GENERIC) #0:
     Sun Jan 3 02:13:00 CET 2010 feyrer%mini.local@localhost
     :/usr/cvs/src-current/obj.i386/sys/arch/i386/compile/GENERIC i386

    Really update? (y/N) y
    Updating sets...
    Extracting base.tgz
    tar -plzx -C / -f base.tgz
    Extracting comp.tgz
    tar -plzx -C / -f comp.tgz
    Extracting games.tgz
    tar -plzx -C / -f games.tgz
    Extracting man.tgz
    tar -plzx -C / -f man.tgz
    Extracting misc.tgz
    tar -plzx -C / -f misc.tgz
    Extracting modules.tgz
    tar -plzx -C / -f modules.tgz
    Extracting tests.tgz
    tar -plzx -C / -f tests.tgz
    Extracting text.tgz
    tar -plzx -C / -f text.tgz
    Extracting xbase.tgz
    tar -plzx -C / -f xbase.tgz
    Extracting xcomp.tgz
    tar -plzx -C / -f xcomp.tgz
    Extracting xfont.tgz
    tar -plzx -C / -f xfont.tgz
    Extracting xserver.tgz
    tar -plzx -C / -f xserver.tgz
    Updating etc files...
    etcupdate -s /usr/cvs/src-current/obj.i386/releasedir/i386/binary/sets/etc.tgz \
     -s /usr/cvs/src-current/obj.i386/releasedir/i386/binary/sets/xetc.tgz
    Done.
    %

The script is available in the current-users@ mailing list archive.

Update: Chavdar Ivanov tells me that there's a package pkgsrc/sysutils/etcmanage that comes with a script INSTALL-NetBSD which does the same as mine (plus more). Seems that's not established enough that it's the communicated way for updating (yet?). Given that the script uses perl, maybe that's some time off. :-P

Another option that uses toosl that come with NetBSD out of the box (ksh!) is available here - thanks to Dalibor Gudzic for the hint and the script!

Update #2: Asmodehn sent me mail that he also has made a script for performing system updates. How about someone collect all these into one, and import something into NetBSD that at least provides minimal functionality?

January 12, 2010 12:54 PM

January 11, 2010

Installing NetBSD/macppc on a Mac Mini G4

Yesterday, I spent a while installing NetBSD/macppc 5.0.1 on a Mac Mini G4. The process wasn't easy, as it involved the following steps. I'm omitting many details, as they are "common knowledge" to Mac users (or otherwise can be easily found on the net):<br /><ul><li>After booting the installer from the CD image, drop into the shell.</li><li>Use <tt>pdisk</tt> to create an <tt>Apple_HFS</tt> partition for the boot loader and two <tt>Apple_UNIX_SVR2</tt> partitions, one for the root file system and another for swap.</li><li>Run <tt>sysinst</tt> and install the system. When asked to repartition the disk, just say <i>Use existing partition sizes</i>.</li><li>Once the system is installed, drop again into the shell before rebooting.</li><li>Mount your hard disk into <tt>/mnt</tt> and chroot into it.</li><li>Fetch a copy of pkgsrc.</li><li>Install the <tt>sysutils/hfsutils</tt> package.</li><li>Use <tt>hformat</tt> to create a new HFS file system in the <tt>Apple_HFS</tt> partition we created.</li><li>Mount the installation CD.</li><li>Copy, using <tt>hcopy</tt>, the <tt>ofwboot.xcf</tt> file from the CD to the boot partition.</li><li>Reboot.</li><li>Drop into the OpenFirmware setup (Command+Option+P+R).</li><li>Set <tt>boot-device</tt> to <tt>hd:,\ofwboot.xcf</tt>.</li><li>Set <tt>boot-file</tt> to <tt>netbsd</tt>.</li><li>And here is the tricky thing to get the machine to auto-boot: Set <tt>boot-command</tt> to <tt>." hello" cr " screen" output boot</tt>, not <tt>mac-boot</tt>.</li></ul><div>I found the last command somewhere on the Internet (dunno where now), but, supposedly, a regular <tt>mac-boot</tt> should have worked. In fact, it works if you call this command from the prompt, but not during automatic boot. (It turns out to be a problem with the version of OpenFirmware I have.)</div><div><br /></div><div>Just writing down the steps in case I need them later on. Installing Debian stable was much, much easier, but the installer for testing crashes every day with a different error, so I gave up.</div><div><br /></div><div>(Oh, by the way, I did the same installation into an old PowerMac G3 and that was really painful. The machine refused to boot from any of the CDs I tried and the prebuilt kernels hang during initialization due to a bogus driver. In the end: netbooting and using custom kernels.)</div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-1422463512138756573?l=blog.julipedia.org' alt='' /></div></content>

by Julio Merino at January 11, 2010 11:48 PM

December 28, 2009

Mounting fixed and removable disks with KDE

If you're among the people running KDE, and if you want to be able to have KDE handle removable and fixed disks properly during the HAL subsystem, there's a suggestion by David Brownlee to achieve this.

To get removable devices automatically mounted and a file browser pop up, add the following to /usr/pkg/etc/PolicyKit/PolicyKit.conf:

        <match action="org.freedesktop.hal.storage.mount-removable">
                <return result="yes" />
        </match>

David writes that ``[w]ith the above plugging in a USB flash drive into my amd64 laptop pops up a notification window, clicking on it opens dolphin, and clicking on the device in dolphin correctly mounted it.''

A similar thing can be achieved for fixed disks, by adding the following data:

        <match action="org.freedesktop.hal.storage.mount-fixed">
                <return result="yes" />
        </match>

There are still discussions if/what part of this should be enabled by default, but I think it's nevertheless useful to document this here. Enjoy!

December 28, 2009 12:50 PM

December 21, 2009

December 21, 2009 12:48 AM

Upgrading PostgreSQL

Upgrading the client, then the server

December 10, 2009

Catching up on source changes

I've dug through my source-changes archive once more, and there are entries from February(?!?) to mid-October 2009 in there that I think may be of interest to casual followers of NetBSD.

Before I go into the details, let me add a hint on the "source-changes-full" mailing list, which is like source-changes, but the mails there also contain diffs of the changes (as long as they are below 1MB in size). The -full list is not archived on mail-index.netbsd.org, and I must have missed the announcement. It's definitely useful!

So, what interesting changes have there been to NetBSD-current recently? Here we go:

New / updated drivers:
- sdtemp(4) reads the on-DIMM temperature sensors following the JEDEC Standard No. 21-C Section 4-7.
- gpio(4) got support for Intel ICH southbridges as found on Intel SS4220-E (ICH7) and Acorp 6A815EPD (ICH2) motherboards
- lom(4) supports the LOMlite "Lights Out Management" boards found on Sun hardware like the LOMlite found on the Sun Netra t1 and the LOMlite2 found on Sun Netra T1/X1 and Sun Fire V100/V120.
- An initial version of a SD/MMC driver for the Winbond W83L518D (and probably W83L519D) Integrated Media Reader with PNP bus attachment glue was added.
- acpiwmi(4) was added to added, to further support ACPI development. See the specs for more details.
- piixide(4) now supports Intel 3400 SATA
New kernel interfaces and library functions include:
- getdate(3) and getdate_err(3) were added according to POSIX.1 and the Open Group
- RAIDframe was sped up massively when checking parity after a system crash with the help of a parity map. This work was done during this year's Google Summer-of-Code.
- A "netbsd.inet.icmp.bmcastecho" sysctl was added to disable replies to the broadcast address.
- Floating point functions f{min,max,dim}{,l,f}(3) were added
- udl(4) adds support for DisplayLink DL-1x0/1x5 based USB LCDs and USB-VGA adaptors
Random other additions and changes include:
- hdaudioctl(8) can be used to manipulate hdaudio(4) devices
- The NetBSD/playstation2 port was removed due to a lack of developer support. It's still available in CVS.
- The NetBSD/i386 bootloader now knows a "dev" command to list available boot devices and their size.
Newly imported and updated 3rd party software includes:
- pfsync from OpenBSD 4.2, adopted in this year's Google Summer of Code.
- PF from OpenBSD 4.2 was imported.
- Multicast DNS ("Bonjour") support was added, based on Apple's mDNSResponder. Enable with "dns: mdns" in /etc/nsswitch.conf.
- GNU binutils were updated to version 2.19
- Many of the bazillion of X.org components were updated - ways too many to note in detail.
- BIND was updated to 9.7.0b1
- Flex 2.5.35 was imported
- A new m4(1) was imported

December 10, 2009 12:30 AM

December 09, 2009

Logical Volume Management (LVM)

Adam Hamsik has been working on Logical Volume Management (LVM) support in NetBSD during the Google Summer of Code and beyond. The code is a from-scratch implementation of the same interfaces found in the Linux kernel, and as a result, the same userland tools can be used. The code is now enabled in NetBSD by default, i.e. it will be part of NetBSD 6.0 (quite some time down the road, but time to test and shake out the bugs is *now*).

Besides the code, there's also documentation available in the NetBSD Guide.

Happy Volume Managing!

December 09, 2009 11:37 PM

December 02, 2009

by jdarnold at December 02, 2009 01:27 PM

Glorious LXF126 Contest

If you are the winner in my glorious Linux Format 2009 Christmas Issue giveaway, here's what you have to look forward to: Ultimate eye candy - a pretty interesting article on how to get the most dazzling display, whether you use Compiz, KDE, or GNOME. Despite my general disdain...

November 30, 2009

by jdarnold at November 30, 2009 03:15 PM

Contest and Projects

Due to a mix-up, I ended up with two copies of Linux Format #126, the Christmas 2009 issue. I figure I'll run my first ever contest here at Daemon Dancing - after over 6 years of writing on this blog, why not? Let's make it a simple one - just...

mailwrapper exited on signal 11

Sometimes, Postfix gets clobbered.

November 30, 2009 12:48 AM

November 29, 2009

logcheck - a log file scanner

Automate the scanning of your log files

November 29, 2009 12:48 AM

November 27, 2009

by jdarnold at November 27, 2009 02:29 PM

Holiday Projects

So on this long weekend in the US, I have a few technology projects I hope to work on: Fix Girls' Computer Their computer finally gave up the ghost a few weeks ago and they've been using one of my work computers while I figured out what to do. I...

November 24, 2009

November 24, 2009 12:48 AM

spamd-setup gives 404

Lists change. Find out which one.

November 22, 2009

c partition goes away in FreeBSD 8

Sometimes, things change.

November 22, 2009 12:48 AM

November 16, 2009

Processing Makefile.am with M4

ATF's Makefile.am, which is a single Makefile for the whole tree, was already at the 1300 lines mark and growing. At this size, it is unmanageable, and a quick look at its contents reveals tons of repeated delicate code.<br /><br />Why so much repeated code, you ask, if the whole point of Automake is to simplify Makefiles? Automake does in fact simplify Makefile code when you define targets known by Automake, such as binaries and/or libraries. However, as soon as you start doing fancy things with documentation, building tons of small programs or messing with shell scripts, things get out of control because you are left on your own to define their targets and their necessary build logic.<br /><br />Up until now, I had just kept up with the boilerplate code... but now that I'm starting to add pretty complex rules to generate HTML and plain text documentation out of XML files, the complexity must go. And here comes my solution:<br /><br />I've just committed an experiment to <a href="http://mtn-host.prjek.net/viewmtn/atf/revision/info/aca66742e00b236d0b6dcb22ae70468736b612bc">process Makefile.am with M4</a>. I've been trying to look for prior art behind this idea and couldn't find any, so I'm not sure how well this will work. But, so far, this has cut down 350 lines of Makefile.am code.<br /><br />How does this work? First of all, I've written a script to generate the Makefile.am from the Makefile.am.m4 and put it in admin/generate-makefile.sh. All this script does is call M4, but I want to keep this logic in a single place because it has to be used from two call sites as described below.<br /><br />Then, I've added an autogen.sh script to the top-level directory that generates Makefile.am (using the previous script) and calls autoreconf -is. I'm against autogen.sh scripts that pretend to be smart instead of just calling autoreconf, but in this case I see no other way around it.<br /><br />At last, I've modified Makefile.am to add an extra rule to generate itself based on the M4 version. This, of course, also uses generate-makefile.sh.<br /><br />We'll see how this scales, but I'm so far happy with the results.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-5706562320936440325?l=blog.julipedia.org' alt='' /></div></content>

Extending sudo credentials

If you use sudo for, e.g. pkgsrc's just-in-time su, you may have often bitten by the problem that some compilations are slow and the build process stops right in the middle to ask you for a root password. If you go away while the system compiles, you'll be frustrated when you come back, as the process may still well be at the very beginning.<br /><br />This happens because, unless disabled by the system administrator, your sudo credentials last for 5 minutes. If you hadn't used sudo for those 5 minutes, it will ask you for your password again. A simple workaround for the problem is to automatically renew your credentials, say, every 2 minutes. You can do this by running the following command (from the same console you are using later on!) right before starting a pkgsrc build:<pre>$ ( while :; do sudo -v; sleep 120; done ) &</pre><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-190748834219167977?l=blog.julipedia.org' alt='' /></div></content>

Best config setting ever

<pre>echo 'set editing-mode vi' >>~/.inputrc</pre>This will enable vi-editing mode for all commands that use the GNU readline library (e.g. bash, python, bc, etc.), not only the shell. For the shell only (including non-bash shells), add 'set -o vi' to your shrc file.<br /><br />I don't know why I didn't do this before given that I'm a pretty hard vi user. Still, for some reason, I kept using emacs-like key bindings for command-line editing. Not any more! However, be careful: if you are used to vim's visual editing mode, you'll keep hitting 'v' in the command line and getting super annoyed.<br /><br />Enjoy!<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-1338186479268297089?l=blog.julipedia.org' alt='' /></div></content>

1000 revisions for ATF

Mmm! Revision <a href="http://mtn-host.prjek.net/viewmtn/atf/revision/info/7ca234b9aceabcfe9a8a1340baa07d6fdc9e3d33">7ca234b9aceabcfe9a8a1340baa07d6fdc9e3d33</a>, committed about an hour ago, marks the 1000th revision in the ATF repository. Thanks for staying with me if you are following the project :)<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-681908603179050098?l=blog.julipedia.org' alt='' /></div></content>

Books by Joel Spolsky

I just finished reading the third book in a row from Joel Spolsky, titled Joel on Software. Before this one, I read More Joel on Software and The Best Software Writing 1, all in a bit over a month. Note: I hadn't read any book cover-to-cover for a loooong while. Very interesting and entertaining books; highly recommended.<div><br /></div><div>Oh, and his writing style is really enjoyable. We, crappy blog writers, can learn a lot from him!</div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-9020484380911179497?l=blog.julipedia.org' alt='' /></div></content>

Rearchitecting ATF

During the last few weeks, I've been doing some ATF coding and, well... I'm not happy. At all. I keep implementing features but I feel, more and more, that ATF is growing out of control and that it is way too sluggish. It oughtn't be so slow. About 6 minutes to run the whole test suite in a Mac G3 I just got? HA! I bet I can do much, much, much better than that. Come on, we are targeting NetBSD, so we should support all those niche platforms rather well, and speed matters.<div><br /></div><div>The thing is, the current code base grew out of a prototype that didn't have that much of a design. Well, it had a design but, in my opinion, it has turned to be a bad design. I couldn't imagine that we could hit the bottlenecks (speed) and user-interface issues (for example, the huge difficulties that involve debugging a failing test case) that we are hitting. So...<div><br /></div><div><i>IT IS TIME FOR A CHANGE!!!</i></div><div><br /></div><div>I'm currently working on a written specification of what ATF will look like, hopefully, in the not-so-distant future. It will take a while to get there, but with enough effort, we soon will. And life will be better. And no, I'm not talking about a from-scratch rewrite; that'd only hurt the project. I plan to take incremental and safe steps, keeping the code base running all the time, but I will do a major face-lift of everything. (I wish I could say "we" instead of "I" here. But not there yet.)</div><div><br /></div><div>Why am I writing a specification, you ask? Well, because that forces me (or ANY developer) to think how I want the thing to look like and to decide, exactly, on what the design will be, which technologies will be used, which languages will be involved and in what components, etc. And no, I'm not talking of a class model design; I'm just talking about the main design of the whole picture, which is quite hard by itself. Plus having a spec will allow me to show it to you before I start coding and you will say "oh, wonderful, this new design sucks so much that I'm not going to bother with the new version". Or maybe hell will freeze and you will think, "mmm, this looks interesting, maybe it will solve these issues I'm having as regards speed, ease of debugging and ease of use".</div><div><br /></div><div>Anyway, I hope to have a draft "soon" and to hear any of the two possible comments as a result!</div><div><br /></div><div><b>Edit (July 29th)</b>: Alright, I have uploaded an <a href="http://www.netbsd.org/~jmmv/rearchitecting-atf/spec.html">extremely preliminary copy of the specification</a> just so that you can see where my current ideas are headed. Expect many more changes to this document, so don't pay too much attention to the tiny details (most of which aren't there anyway yet).</div></div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-4743147252783979430?l=blog.julipedia.org' alt='' /></div></content>

The mess of ATF's code

Yes. ATF's code is a "bit" messy, to put it bluntly. I'm quite happy with some of the newest bits but there are some huge parts in it that stink. The main reason for this is that the "ugly" parts were the ones that were written first, and they were basically a prototype; we didn't know all the requirements for the code at that point... and we still don't know them, but we know we can do much better. Even though I'm writing in plural... I'm afraid we = I at the moment :-P<div><br /></div><div>So, is it time for the big-rewrite-from-scratch? NO! Joel Spolsky wrote about why <a href="http://www.joelonsoftware.com/articles/fog0000000348.html">this is a bad idea</a> and I have to agree with him. Yeah, I'm basically the only developer of the code so everything is in my head, and I'd do a rewrite with a fresh mind, but... I'd lose tons of work and, specially, I'd lose tons of code that deals with tricky corner-cases that are hard to remember.</div><div><br /></div><div>Sure, I want to clean things up but they'll happen incrementally. And preferably concurrently with feature additions. These two things could definitely happen at the same time if only I had infinite spare time...</div><div><br /></div><div>Anyway, the major point of this post is to describe what I don't like about the current code base and how I'd like to see it changing:</div><div><ul><li>A completely revamped C++ API for test cases. The current one sucks. It is not consistent with the C API. It lacks important functionality. It uses exceptions for test-case status reporting (yuck!). And it's ugly.</li><li>Clear separation of "internal/helper" APIs from the test APIs. You'll agree that the "fs" module, which provides path abstraction and other file system management routines, is something that cannot be part of ATF's API. ATF is about testing. Period. Either that fs module should be in a separate library or should be completely hidden from the public. Otherwise, it'll suffer from abuse and, what scares me, will have to become part of ATF's API. And likewise, most &mdash; really &mdash; most of the modules in the current code are internal.</li><li>Less dependencies from the C++ API to the C API. Most of the current C++ modules are wrappers of their corresponding C counterparts. This is nice for code reuse but makes the code extremely fragile. In C++, things like RAII can provide really robust code with minimum effort, but intermixing such C++ code with C makes things ugly really quickly. I'd like to find a way to keep the two libraries separate from each other (and thus keep the C++ binding "pure"), but at the same time I don't want to duplicate code... an interesting problem.</li><li>Split the tarball into smaller pieces. People writing test cases for C applications don't want to pull in a huge package that depends on C++ and whatnot. And ATF is huge. It takes forever to compile. And this is a serious issue for broad adoption. Note: whether the tools are written in C++ or not is a separate issue, because these are not a dependency for anything!</li><li>The shell binding is slow. Really slow compared to the other ones. Optimizations would be nice, but those do not address the root of the problem: it's costly to query information from shell-based tests at run time. I.e. it takes a long time to get the full list of test cases available in a test suite because you have to run every single test program with the -l flag. Keeping a separate file with test-case metadata alongside the binary could resolve this and allow more flexibility at run time.</li><li>And some other things.</li></ul><div>Those are the major things I'd like to see addressed soon, but they involve tons of work. Of course, I'd like to be able to work on some features expected by other developers: easier debugging, DOCUMENTATION!...</div><div><br /></div><div>So, helpers welcome :-)</div></div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-8981481737885658422?l=blog.julipedia.org' alt='' /></div></content>

Technicians and schedules

Here I am, on the afternoon of a work day, sitting at home waiting for an eircom technician to come set it up my phone line. How nice. The story goes like this:<div><br /></div><div>Two weeks ago, I placed an online order to request a phone line, explicitly specifying that the physical installation is already done (even though I don't know if it works or not, but that should be fairly easy for them to check). A few days later, the technician called me saying that he'd come today (two weeks after), anytime from 12.00 to 15.00, but that I'd call the company the same day to get a more accurate schedule.</div><div><br /></div><div>Fine, I'll wait until the 23rd to do that call. But you know what happened, right? I called them this morning and they said that, effectively, the technician was coming today, from 12.30 <i>onwards</i> but they were unable to provide me any more specific information because the technicians have <i>multiple appointments</i>. What? Again, WHAT? At this age of technology, can't we implement a system to track technicians and their schedules? Can't we make some approximations of how long each visit will take? I bet it's trivial if you put in just some common sense.</div><div><br /></div><div>People have jobs, and they can't leave anytime for unknown periods of time; granted, I have some more freedom at Google, but that is absolutely not the case for most other companies. If you have to be at home at 12.30 sharp, and the appointment will last 30 minutes approximately, that is one thing, but having to be at home from 12.30 for an unexpected period of time, that is a very different thing.</div><div><br /></div><div>Just wondering... couldn't they just make the technician call you about 20-30 minutes before arrival so that you could make the same arrangements as him and be there at the same time? It doesn't seem such an insane request.</div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-2459937406467822931?l=blog.julipedia.org' alt='' /></div></content>

Child-process management in C for ATF

Let's face it: spawning child processes in Unix is a "mess". Yes, the interfaces involved (fork, wait, pipe) are really elegant and easy to understand, but every single time you need to spawn a new child process to, later on, execute a random command, you have to write quite a bunch of error-prone code to cope with it. If you have ever used any other programming language with higher-level abstraction layers — just check Python's subprocess.Popen — you surely understand what I mean.<div><br /><div><div>The current code in ATF has many places were child processes have to be spawned. I recently had to add yet another case of this, and... enough was enough. Since then, I've been working on a C API to spawn child processes from within ATF's internals and just pushed it to the repository. It's still fairly incomplete, but with minor tweaks, it'll keep all the dirty details of process management contained in a single, one-day-to-be-portable module.</div><div><br /></div><div>The interface tries to mimic the one that was designed on my Boost.Process Summer of Code project, but in C, which is quite painful. The main idea is to have a fork function to which you pass the subroutine you want to run on the child, the behavior you want for the stdout stream and the behavior you want for the stderr steam. These behaviors can be any of capture (aka create pipes for IPC communcations), silence (aka redirect to /dev/null), redirect to file descriptor and redirect to file. For simplicity, I've omitted stdin. With all this information, the fork function returns you an opaque structure representing the child, from which you can obtain the IPC channels if you requested them and on which you can wait for finalization.</div><div><br /></div><div>Here is a little example, with tons of details such as error handling or resource finalization removed for simplicity. The code below would spawn "/bin/ls" and store its output in two files named ls.out and ls.err:</div><pre>static<br />atf_error_t<br />run_ls(const void *v)<br />{<br /> system("/bin/ls");<br /> return atf_no_error();<br />}<br /><br />static<br />void<br />some_function(...)<br />{<br /> atf_process_stream_t outsb, errsb;<br /> atf_process_child_t child;<br /> atf_process_status_t status;<br /><br /> atf_process_status_init_redirect_path(&amp;outsb, "ls.out");<br /> atf_process_status_init_redirect_path(&amp;errsb, "ls.err");<br /><br /> atf_process_fork(&amp;child, run_ls, &amp;outsb, &amp;errsb, NULL);<br /> ... yeah, here comes the concurrency! ...<br /> atf_process_child_wait(&amp;child, &amp;status);<br /><br /> if (atf_process_status_exited(&amp;status))<br /> printf("Exit: %d\n", atf_process_status_exitstatus(&amp;status));<br /> else<br /> printf("Error!");<br />}</pre><div>Yeah, quite verbose, huh? Well, it's the price to pay to simulate namespaces and similar other things in C. I'm not too happy with the interface yet, though, because I've already encountered a few gotchas when trying to convert some of the existing old fork calls to the new module. But, should you want to check the whole mess, <a href="http://mtn-host.prjek.net/viewmtn/atf/revision/info/3ba2af4ab1cad67108c3fb1aea15e8e0168667ff">check out the corresponding revision</a>.</div></div></div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-1880013018173418859?l=blog.julipedia.org' alt='' /></div></content>

How to find an apartment in Dublin

It has been three weeks since I moved to Dublin, Ireland, and I finally have settled into my new apartment. It has taken me two weeks (I was pretty busy during the first one) to go through ads, visits and offers to finally get a place that is cozy, nicely decorated and decently located, all at a quite reasonable price. I could have gotten nicer places for a bit more money, but I'm happy with this one so far.<br /><br />If you are looking forward to finding a place to stay in Dublin, this post contains some suggestions based on my experience:<br /><br />First of all, keep in mind that Dublin is outrageously expensive. The prices for housing here are insane at the moment (OK, not as expensive as NYC or SF, but really expensive anyway). Be prepared to spend around 1K EUR for a nice 1 bedroom apartment, and 1.5K EUR for a nice 2 bedroom apartment. Things may improve in the next months, as they just did for the first quarter of the year.<br /><br />With that said, your first point of reference should be <a href="http://daft.ie/">daft</a>. This is the place where all landlords and agencies put their ads, and the place where everyone is looking for apartments. To get started, you need to know where you want to live. Get a rough idea and then locate that place in one of the <a href="http://en.wikipedia.org/wiki/Dublin_postal_districts">Dublin postal districts</a> and the ones surrounding it. Given that public transportation is... well... suboptimal, you don't want to live too far from your workplace. Then, hunt for places within your budget... and a budget a bit higher: you can always <i>try to</i> negotiate the rent down and get a nicer apartment than you would otherwise, still staying within your initial budget.<br /><br />Once you have selected some of the apartments you want to check, call the landlords or agents and ask for an appointment as soon as possible. And, during the visit, check a few basic stuff:<br /><div><ul><li>Whether the house is old or new: if it's new, it'll probably be in nicer condition overall.</li><li>Water pressure: old houses have poor water pressure.</li><li>Electric shower: this is really scary to me, but it is what most old houses have to deal with poor water pressure.</li><li>Carpet: nice, but a horrible mess to clean up.</li><li>Garbage collection service: if the building does not do this for you, you'll have to pay for garbage collection separately. I just bought 3 bin tags and those were almost 9 EUR. Yes: 9 EUR to pay for the collection of THREE garbage bags.</li><li>Location of supermarkets: Dublin is basically a big town, so most roads don't have shops. Make sure that you have a supermarket nearby where you can walk to to get basic stuff.</li><li>Availability of cable/phone: you'll need this for Internet.</li><li>Furniture: most apartments in Dublin are provided fully-furnished, so make sure to pick one with furniture that you like. Ask if you are allowed to replace some. Pay special attention to the mattress and couches!!</li><li>Cutlery: OK, this is part of the furniture, but check what you have. Your landlord may provide you additional stuff for free upon request.</li><li>Washer and <i>dryer</i>: you want to have a dryer, as most lease contracts state you cannot hung clothes on public places.</li><li>Heating and double-windows: you'll need this during the winter.</li></ul>And, at last, don't hurry! The housing market has improved during the last months, so if you see a place that you like, you'll <i>most likely</i> have a few days to decide whether you want it or not (in the past, you had to decide during viewing time, or otherwise it'd be gone afterwards). Think well about your decision and negotiate; don't show yourself as impatient or you'd get worse deals!</div><div><br /></div><div>I think that's all for know. If there is anything else, the post will be updated :)</div><div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-6501743084580398314?l=blog.julipedia.org' alt='' /></div></content>

Trying AdSense

I've just decided to enable AdSense on this blog and see what the results are. If they are not worth it (what I'm expecting), I'll disable ads after a while. But who knows, maybe I get a nice surprise!<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-7459230551314143568?l=blog.julipedia.org' alt='' /></div></content>

Paella in NYC

These days, I'm starting to cook by myself (aka learning) and yesterday I made paella for 6 people while staying in NYC (leaving on Sunday...). This is the third time in two weeks that I cook this Spanish dish, but I think the results were pretty good despite the lack of ingredients. After all, cooking is not as hard as I originally thought! And it's pretty fun too!<br /><br />Just blogging this because the results look nice:<br /><br /><div style='text-align:center;margin:0px auto 10px;'><a href='http://2.bp.blogspot.com/_xLbGV919cEE/SgxglPObEeI/AAAAAAAAC5Q/na0_iEsVnAY/s1600-h/IMG_1329.JPG'><img src='http://2.bp.blogspot.com/_xLbGV919cEE/SgxglPObEeI/AAAAAAAAC5Q/na0_iEsVnAY/s400/IMG_1329.JPG' border='0' alt='' /></a>&nbsp;</div>P.S. I'm now eating the leftovers from yesterday. Yummm! :-)<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-4620782319169725789?l=blog.julipedia.org' alt='' /></div></content>

Mailing lists for commit notifications

The project I'm currently working on at university uses Subversion as its version control system. Unfortunately, the project itself has no mailing list to receive notifications on every commit, and the managers refuse to set this up. They do not see the value of such a list and they are scared of it because they probably assume that <i>everyone</i> ought to be subscribed to it.<br /><br />Having worked on projects that have a commit notification mailing list available, I strongly advise to have such a list anytime you have more than one developer working on a project[1]. Bonus points if every commit message comes with a bundled copy of the change's diff (in unified form!). This list <i>must be independent from the regular development mailing list</i> and it <i>must be opt-in</i>: i.e. never subscribe anyone by default, let themselves subscribe if they want to! Not everyone will need to receive this information, but it comes very useful... and it's extremely valuable for the project managers themselves!<br /><br />Why is this useful? Being subscribed to the commit notification mailing list, it is extremely easy to know what is going on on the project[2]. It is also really easy to review the code submissions as soon as they are made which, with proper reviews by other developers, <i>trains the authors</i> and improves their skills. And if the revision diff is inlined, it is trivial to pinpoint mistakes in it (be them style errors, subtle bugs, or serious design problems) by replying to the email.<br /><br />So, to my current project managers: if you read me, here is a wish-list item. And, for everyone else, if you need to set up a new project, consider creating this mailing list as soon as possible. Maybe few developers will subscribe to it, but those that do will pay attention and will provide very valuable feedback in the form of replies.<br /><br />1: Shame on me for not having such a mailing list for ATF. Haven't investigated how to do so with Monotone.<br /><br />2: Of course, the developers must be conscious to commit early and often, and to provide well-formed changesets: i.e. self-contained and with descriptive logs.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-4782427967638867417?l=blog.julipedia.org' alt='' /></div></content>

DEBUG.EXE dropped in Windows 7

Wow. <a href="http://tech.slashdot.org/article.pl?sid=09/05/08/169226&from=rss">DEBUG.EXE is being finally phased out in Windows 7</a>. I can't believe it was still there.<br /><br />This brings me back two different memories. I had used this program in the past (a long while ago!) and it caused me both pain and joy.<br /><br />Regarding pain: I had an MS-DOS 5.x book that spent a whole section on DEBUG.EXE, and one of the examples in it contained a command that caused the program in memory to be written to some specific sectors of the floppy disk. Guess what I tried? I executed that same command but told it to use my hard disk instead of the floppy drive. Result: a corrupted file system. Had to run scandisk (remember it?), which marked some sectors as faulty and I thought I had ruined my precious 125MB WD Caviar hard disk. It wasn't until much, much, much later that I learnt that such a thing was not possible, and that really formatting the disk with a tool that had no memory of "bad" sectors (aka, Linux's newfs) could revert the disk to a clean state. (Actually, I kept that hard disk until very recently.)<br /><br />Regarding joy: On a boring weekend away from home, I used DEBUG.EXE on an old portable machine without internet connection to hack a version of PacMan. I disassembled the code until I found where it kept track of the player's lives and tweaked the counter to be infinite (or extra large, can't remember). <i>That</i> was fun. I could get to levels me and my father (who used to be an avid player) had never seen before!<br /><br />It's a pity this tool is going, but it must go. It is way too outdated compared to current debuggers. I wonder if anyone is still using it.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-446058185749523279?l=blog.julipedia.org' alt='' /></div></content>

Using C++ templates to optimize code

As part of the project I'm currently involved in at university, I started (re)writing a <a href="http://www.pintool.org/">Pin</a> tool to gather run-time traces of applications parallelized with OpenMP. This tool has to support two modes: one to generate a single trace for the whole application and one to generate one trace per parallel region of the application.<br /><br />In the initial versions of my rewrite, I followed the idea of the previous version of the tool: have a <tt>-split</tt> flag in the frontend that enables or disables the behavior described above. This flag was backed by an abstract class, <tt>Tracer</tt>, and two implementations: <tt>PlainTracer</tt> and <tt>SplittedTracer</tt>. The thread-initialization callback of the tool then allocated one of these objects for every new thread and the <i>per-instruction injected code</i> used a pointer to the interface to call the appropriate specialized instrumentation routine. This pretty much looked like this:<pre>void<br />thread_start_callback(int tid, ...)<br />{<br /> if (splitting)<br /> tracers[tid] = new SplittedTracer();<br /> else<br /> tracers[tid] = new PlainTracer();<br />}<br /><br />void<br />per_instruction_callback(...)<br />{<br /> Tracer* t = tracers[PIN_ThreadId()];<br /> t->instruction_callback(...);<br />}</pre>I knew from the very beginning that such an implementation was going to be inefficient due to the pointer dereference at each instruction and the vtable lookup for the correct virtual method implementation. However, it was a very quick way to move forward because I could reuse some small parts of the old implementation.<br /><br />There were two ways to optimize this: the first one involved writing different versions of <tt>per_instruction_callback</tt>, one for plain tracing and the other for splitted tracing, and then deciding which one to insert depending on the flag. The other way was to use template metaprogramming.<br /><br />As you can imagine, this being C++, I opted to use template metaprogramming to heavily abstract the code in the Pin tool. Now, I have an abstract core parametrized on the Tracer type. When instantiated, I provide the correct Tracer class and the compiler does all the magic for me. With this design, there is no need to have a parent Tracer class &mdash; though I'd welcome having C++0x concepts available &mdash;, and the callbacks can be easily inlined because there is no run-time vtable lookup. It looks something like this:<pre>template< class Tracer ><br />class BasicTool {<br /> Tracer* tracers[MAX_THREADS];<br /><br /> Tracer* allocate_tracer(void) const = 0;<br /><br />public:<br /> Tracer*<br /> get_tracer(int tid)<br /> {<br /> return tracers[tid];<br /> }<br />};<br /><br />class PlainTool : public BasicTool< PlainTracer > {<br /> PlainTracer*<br /> allocate_tracer(void) const<br /> {<br /> return new PlainTracer();<br /> }<br /><br />public:<br /> ...<br />} the_plain_tool;<br /><br />// This is tool-specific, non-templated yet.<br />void<br />per_instruction_callback(...)<br />{<br /> the_plain_tool.get_tracer(PIN_ThreadId()).instruction_callback(...);<br />}</pre>What this design also does is force me to have two different Pin tools: one for plain tracing and another one for splitted tracing. Of course, I chose it to be this way because I'm not a fan of run-time options (the <tt>-split</tt> flag). Having two separate tools with well-defined, non-optional features makes testing much, much easier and... follows the Unix philosophy of having each tool do exactly one thing, but doing it right!<br /><br />Result: around a 15% speedup. And C++ was supposed to be slow? ;-) You just need to know what the language provides you and choose wisely. (Read: my initial, naive prototype had a run-time of 10 minutes to trace part of a small benchmark; after several rounds of optimizations, it's down to 1 minute and 50 seconds to trace the <i>whole</i> benchmark!)<br /><br />Disclaimer: The code above is an oversimplification of what the tool contains. It is completely fictitious and obviates many details. I will admit, though, that the real code is too complex at the moment. I'm looking for ways to simplify it.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-4155033642151738049?l=blog.julipedia.org' alt='' /></div></content>

Numeric limits in C++

By pure chance when trying to understand a build error of some C++ code I'm working on, I came across the correct C++ way of checking for numeric limits. Here is how.<br /><br />In C, when you need to check for the limits of native numeric types, such as <tt>int</tt> or <tt>unsigned long</tt>, you include the <tt>limits.h</tt> header file and then use the <tt>INT_MIN</tt>/<tt>INT_MAX</tt> and <tt>ULONG_MAX</tt> macros respectively. In the C++ world, there is a corresponding <tt>climits</tt> header file to get the definition of these macros, so I always thought this was the way to follow.<br /><br />However, it turns out that the C++ standard defines a <tt>limits</tt> header file too, which provides the <tt>numeric_limits&lt;T&gt;</tt> template. This template class is specialized in <lt>T</lt> for every numeric type and provides a set of static methods to query properties about the corresponding type. The simplest ones are <tt>min()</tt> and <tt>max()</tt>, which are what we need to replace the old-style <tt>*_MIN</tt> and <tt>*_MAX</tt> macros.<br /><br />As an example, this C code:<pre>#include &lt;limits.h&gt;<br />#include &lt;stdio.h&gt;<br />#include &lt;stdlib.h&gt;<br /><br />int<br />main(void)<br />{<br /> printf("Integer range: %d to %d\n", INT_MIN, INT_MAX);<br /> return EXIT_SUCCESS;<br />}</pre>becomes the following in C++:<pre>#include &lt;cstdlib&gt;<br />#include &lt;iostream&gt;<br />#include &lt;limits&gt;<br /><br />int<br />main(void)<br />{<br /> std::cout &lt;&lt; "Integer range: "<br /> &lt;&lt; std::numeric_limits&lt; int &gt::min()<br /> &lt;&lt; " to "<br /> &lt;&lt; std::numeric_limits&lt; int &gt::max()<br /> &lt;&lt; "\n";<br /> return EXIT_SUCCESS;<br />}</pre>Check out the <a href="http://www.cplusplus.com/reference/std/limits/numeric_limits/">documentation</a> for more details on additional methods!<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-8028620398342481804?l=blog.julipedia.org' alt='' /></div></content>

The NetBSD Blog

The NetBSD Project recently launched a new <a href="http://blog.netbsd.org/">official blog for NetBSD</a>. From here, I'd like to invite you to visit it and <a href="http://blog.netbsd.org/tnf/feed/entries/atom">subscribe to it</a>. It's only with your support (through reading and, specially, <i>commenting</i>) that developers will post more entries! Enjoy :-)<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-5467658448048251016?l=blog.julipedia.org' alt='' /></div></content>

NetBSD-SoC needs your application!

The Google Summer of Code 2009 application deadline for students is tomorrow and NetBSD has got very few applications so far. If you have the interest in working on a cool operating system project, where almost any project idea can fit, take the time to read <a href="http://www.netbsd.org/contrib/soc-projects.html">our proposals</a> and apply! New, original ideas not listed there will also be considered.<br /><br />It'd be a pity if the number of assigned slots to NetBSD was small due to the low number of applications! We did much better past year.<br /><br />Note that there are a couple of <a href="http://www.NetBSD.org/~jmmv/atf/">ATF</a>-related proposals in there. Help will be certainly welcome (by me ;-) in those areas!<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-4081034348232814307?l=blog.julipedia.org' alt='' /></div></content>

Returning to Google

I've been holding back this announcement until all affected parties knew in advance. They do know now, so I'm happy to announce that I'll be joining Google Dublin on May 25th as a Google.com Software Engineer!<br /><br />Thanks to everyone who made that possible.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-898144593109126185?l=blog.julipedia.org' alt='' /></div></content>

Comments for old posts now moderated

After waking up today and finding 80+ spam comments all around old posts in this blog, I have decided to set all new comments for posts older than 14 days old to be moderated. Took half an hour to clean them all. Thank you, spammers.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-5724407113147228616?l=blog.julipedia.org' alt='' /></div></content>

What are unnamed namespaces for in C++?

In the past, I had come by some C++ code that used unnamed namespaces everywhere as the following code shows, and I didn't really know what the meaning of it was:<pre>namespace {<br /><br />class something {<br />...<br />};<br /><br />} // namespace</pre>Until now.<br /><br />Not using unnamed namespaces in my own code bit me with name clash errors. How? Take ATF. Some of its files declare classes in .cpp files (not headers). I just copy/pasted some ATF code in another project and linked the libraries produced by each project together. Boom! Link error because of duplicate symbols. And the linker is quite right in saying so!<br /><br />For some reason, I always assumed that classes declared in the .cpp files would be private to the module. But if you just think a little bit about it, just a little, this cannot ever be the case: how could the compiler tell the difference between a class definition in a header file and a class definition in a source file? The compiler sees preprocessed sources, not what the programmer wrote, so all class definitions look the same!<br /><br />So how do you resolve this problem? Can you have a static class, pretty much like you can have a static variable or function? No, you cannot. Then, how do you declare implementation-specific classes private to a module? Put them in an unnamed namespace as the code above shows and you are all set. Every translation unit has its own unnamed namespace and everything you put in it will not conflict with any other translation unit.<div class="blogger-post-footer"><img width='1' height='1' src='https://blogger.googleusercontent.com/tracker/17885055-4053902384158212165?l=blog.julipedia.org' alt='' /></div></content>

November 13, 2009

Scratching the Linux Itch, pt. II

Finally, we get started building the system in Chapter 5. As mentioned in the last important point here, for every package done in Chapter 5, you need to do the following commands, as the 'lfs' user: $ cd $LFS/sources lfs:/mnt/lfs/sources$ tar xjvf binutils-2.20.tar.bz2 lfs:/mnt/lfs/sources$ cd binutils-2.20 If the downloaded package...

by jdarnold at November 13, 2009 03:41 PM

November 12, 2009

Scratching the Linux Itch, pt. I

So I'm going to try a Linux From Scratch build and installation, just because I need another pointless project to write about :) Here is the result of my version-check.sh script on my openSUSE 11.0 system: bash, version 3.2.39(1)-release /bin/sh -> /bin/bash Binutils: (GNU Binutils; openSUSE 11.0) 2.18.50.20080409-11.1 bison (GNU...

by jdarnold at November 12, 2009 03:36 PM

November 10, 2009

by jdarnold at November 10, 2009 03:53 PM

Linux Format 125

My favorite Linux magazine, by far, is Linux Format, a UK based magazine that is just chock-a-block full of great Linux info. They also run the informative Tux Radar blog. So I thought I would just give a rundown each month of the highlights from the most recent issue....

November 09, 2009

TeX Live 2009 - including NetBSD binaries

Today, TeX Live 2009 was released. After teTeX is no longer maintained, TeX Live is *the* Unix (and other) TeX distribution. And with the binary release come binaries for NetBSD: ``Executables for the cygwin and i386-netbsd platforms are now included, while the other BSD distributions have been dropped; we were advised that OpenBSD and FreeBSD users get TEX through their package systems, plus there were difficulties in making binaries that have a chance of working on more than one version. ''.

The NetBSD binaries were provided by Jukka Salmi. Mmm, community involvement. Thanks Jukka!

November 09, 2009 11:32 PM

November 07, 2009

NetBSD vs. Microkernels - From Kernel to Userspace and Abroad

A few weeks ago I made a comment about NetBSD not being a "microkernel" despite it's move to kernel modules. Antti Kantee wrote back to me (Hi Antti!) reminding me of his work on RUMP, PUFFS and the like, which I think does definitely deserve mentioning in that context:

With RUMP, PUFFS and the like, Antti is set on a mission to define interfaces between kernel components, and move those kernel components from inside the kernel to the userland. As a result, the (once) kernel-code is now running as "simple" userland processes that have all the benefits like memory protection and not taking down the whole system in case of an error. Examples that Antti has been bashing on started with moving file systems to the userland, which resulted in PUFFS and ReFUSE, an implementation of the Linux inferface of File systems in USErland (FUSE). Not stopping at file systems (which are traditionally used for accessing data on storage media, like e.g. for ntfs-3g), Antti went further and moved into getting network services like SSHFS going - in userland, based on FUSE.

Not stopping there, more recent works include:

USB device driver support, to run NetBSD's USB stack in userland - and with it a bunch of USB device drivers as well, e.g. storage (umass) and wireless adapters.
Running NFS in userspace, and with it the underlying TCP/IP stack? Well, we've had nfsd(8) for quite some time, but this time it's the same code running in userland as in kernel. The thread also answers questions on why we want to virtualize nfsd+networking with a process and why do we want to do that by reusing kernel code.

Remember when NetBSD got (re)FUSE? All of a sudden we got a bazillion of filesystems back then. So why not turn this development around, and make RUMP available as an interface for all sort of drivers to other operating systems. That way, they can get our file systems, usb stack and drivers, etc., and run them in userspace as well. Developing a driver on one operating system, and using it on many - a wet dream would come true!

Sounds impossible? Ye fear not, it's been done! Arnaud 'stacktic' Ysmal already has ported RUMP to FreeBSD and Linux, the work is available via pkgsrc/misc/rump, and there is also Arnaud's page on Rump on non-NetBSD Operating Systems for more information.

Whew... lots of research and development going on in this area, and - getting back to the initial topic - we may well see an operating system in the future that moves from the monolithic to a microkernel approach, and it may or may not be called NetBSD. Fact is, that a lot of research is going on in that area, on NetBSD, here and now. Hats off, Antti!

November 07, 2009 12:59 AM

On the difference between "data" and "information"

Thanks xkcd, from the information scientist inside me!

November 07, 2009 12:11 AM

October 25, 2009

Harddisk image cloning for Unix - g4u 2.4 released

g4u ("ghosting for unix") version 2.4 has been released. g4u is a NetBSD-based bootfloppy/CD-ROM that allows easy cloning of PC harddisks to deploy a common set up on a number of PCs using FTP. The floppy/CD offers two functions. The first is to upload the compressed image of a local harddisk to a FTP server, the other is to restore that image via FTP, uncompress it and write it back to disk. Network configuration is fetched via DHCP. As the harddisk is processed as an image, any filesystem and operating system can be deployed using g4u. Easy cloning of local disks as well as partitions is also supported.

Three years of time have passed since the last full release of g4u. Here's a list of what's new / changes in g4u 2.4:

Major new supported device types include bluetooth keyboards and SD/MMC cards - feedback highly appreciated!
Lots of new drivers. Too many to list, please see the g4u section of my blog at http://www.feyrer.de/NetBSD/blog.html?-tags=g4u for details.
Based on the NetBSD development version from Sep 2009
Source builds native and without root privileges on NetBSD 5.0 and crossbuilds also without root privileges from Mac OS X (tested) and probably others (untested; expected: Solaris, Linux).

The g4u 2.4 release is available on the g4u homeage at

http://www.feyrer.de/g4u/

October 25, 2009 11:05 PM

October 24, 2009