.\" Hey, Emacs, edit this file in -*- nroff-fill -*- mode .\"- .\" Copyright (c) 1997, 1998 .\" Nan Yang Computer Services Limited. All rights reserved. .\" .\" This software is distributed under the so-called ``Berkeley .\" License'': .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" 3. All advertising materials mentioning features or use of this software .\" must display the following acknowledgement: .\" This product includes software developed by Nan Yang Computer .\" Services Limited. .\" 4. Neither the name of the Company nor the names of its contributors .\" may be used to endorse or promote products derived from this software .\" without specific prior written permission. .\" .\" This software is provided ``as is'', and any express or implied .\" warranties, including, but not limited to, the implied warranties of .\" merchantability and fitness for a particular purpose are disclaimed. .\" In no event shall the company or contributors be liable for any .\" direct, indirect, incidental, special, exemplary, or consequential .\" damages (including, but not limited to, procurement of substitute .\" goods or services; loss of use, data, or profits; or business .\" interruption) however caused and on any theory of liability, whether .\" in contract, strict liability, or tort (including negligence or .\" otherwise) arising in any way out of the use of this software, even if .\" advised of the possibility of such damage. .\" .\" $Id: vinum.4,v 1.3 2003/10/21 16:21:55 wiz Exp $ .\" $NetBSD: vinum.4,v 1.3 2003/10/21 16:21:55 wiz Exp $ .\" .Dd October 16, 2003 .Dt VINUM 4 .Sh NAME .Nm vinum .Nd Logical Volume Manager .Sh SYNOPSIS .Cd "modload vinum" .Cd "modload Vinum" .Sh DESCRIPTION .Nm is a logical volume manager inspired by, but not derived from, the Veritas Volume Manager. It provides the following features: .Bl -bullet .It It provides device-independent logical disks, called .Em volumes . Volumes are not restricted to the size of any disk on the system. .It The volumes consist of one or more .Em plexes , each of which contain the entire address space of a volume. This represents an implementation of RAID-1 (mirroring). Multiple plexes can also be used for .\" XXX What about sparse plexes? Do we want them? .if t .sp .Bl -bullet .It Increased read throughput. .Nm will read data from the least active disk, so if a volume has plexes on multiple disks, more data can be read in parallel. .Nm reads data from only one plex, but it writes data to all plexes. .It Increased reliability. By storing plexes on different disks, data will remain available even if one of the plexes becomes unavailable. In comparison with a RAID-5 or RAID-4 plex (see below), using multiple plexes requires more storage space, but gives better performance, particularly in the case of a drive failure. .It Additional plexes can be used for on-line data reorganization. By attaching an additional plex and subsequently detaching one of the older plexes, data can be moved on-line without compromising access. .It An additional plex can be used to obtain a consistent dump of a file system. By attaching an additional plex and detaching at a specific time, the detached plex becomes an accurate snapshot of the file system at the time of detachment. .\" Make sure to flush! .El .It Each plex consists of one or more logical disk slices, called .Em subdisks . Subdisks are defined as a contiguous block of physical disk storage. A plex may consist of any reasonable number of subdisks (in other words, the real limit is not the number, but other factors, such as memory and performance, associated with maintaining a large number of subdisks). .It A number of mappings between subdisks and plexes are available: .Bl -bullet .It .Em Concatenated plexes consist of one or more subdisks, each of which is mapped to a contiguous part of the plex address space. .It .Em Striped plexes consist of two or more subdisks of equal size. The file address space is mapped in .Em stripes , integral fractions of the subdisk size. Consecutive plex address space is mapped to stripes in each subdisk in .if n turn. .if t \{\ turn. .ig .\" FIXME .br .ne 1.5i .PS move right 2i down SD0: box SD1: box SD2: box .br "plex 0" at SD0.n+(0,.2) "subdisk 0" rjust at SD0.w-(.2,0) "subdisk 1" rjust at SD1.w-(.2,0) "subdisk 2" rjust at SD2.w-(.2,0) .PE .. .\} The subdisks of a striped plex must all be the same size. .It .Em RAID-5 plexes require at least three equal-sized subdisks. They resemble striped plexes, except that in each stripe, one subdisk stores parity information. This subdisk changes in each stripe: in the first stripe, it is the first subdisk, in the second it is the second subdisk, etc. In the event of a single disk failure, .Nm will recover the data based on the information stored on the remaining subdisks. This mapping is particularly suited to read-intensive access. The subdisks of a RAID-5 plex must all be the same size. .It .Nm also supports .Em RAID-4 plexes . They have no advantage over RAID-5. The only difference is that the parity information resides on the same subdisk for each stripe. The subdisks of a RAID-4 plex must all be the same size. .\" Make sure to flush! .El .It .Nm Drives are the lowest level of the storage hierarchy. They represent disk special devices. .It .Nm offers automatic startup. Unlike .Ux file systems, .Nm volumes contain all the configuration information needed to ensure that they are started correctly when the subsystem is enabled. This is also a significant advantage over the Veritas\(tm File System. This feature regards the presence of the volumes. It does not mean that the volumes will be mounted automatically, since the standard startup procedures with .Pa /etc/fstab perform this function. .El .Sh KERNEL CONFIGURATION .Nm is intended to be used as a loadable kernel loadable module (LKM). Currently it must be compiled into the kernel. To do so, add this line to the kernel configuration file: .Bd -literal -offset indent pseudo-device vinum .Ed .Ss DEBUG OPTIONS The current version of .Nm vinum , both the kernel module and the user program .Xr vinum 8 , include significant debugging support. Enable it with the following entry in the kernel configuration file: .Bd -literal -offset indent options VINUMDEBUG .Ed .Sh RUNNING VINUM .Nm is part of the base .Nx system. It does not require installation. To start it, start the .Nm vinum program, which will load the LKM if it is not already present. Before using .Nm vinum , it must be configured. See .Xr vinum 8 for information on how to create a .Nm configuration. .Pp Normally, you start a configured version of .Nm at boot time. Set the variable .Ar start_vinum in .Pa /etc/rc.conf to .Ar YES to start .Nm at boot time. .Pp If .Nm is loaded as a LKM (the recommended way), the .Nm vinum Ar stop command will unload it. You can also do this with the .Nm modunload command. .Pp The LKM can only be unloaded when idle, in other words when no volumes are mounted and no other instances of the .Nm program are active. Unloading the LKM does not harm the data in the volumes. .Ss CONFIGURING AND STARTING OBJECTS Use the .Xr vinum 8 utility to configure and start .Nm objects. .Sh IOCTL CALLS .Pa ioctl calls are intended for the use of the .Nm configuration program only. They are described in the header file .Pa /usr/src/sys/dev/vinum/vinumio.h . .Ss DISK LABELS Conventional disk special devices have a .Em disk label in the second sector of the device. See .Xr disklabel 5 for more details. This disk label describes the layout of the partitions within the device. .Nm does not subdivide volumes, so volumes do not contain a physical disk label. For convenience, .Nm implements the ioctl calls DIOCGDINFO (get disk label), DIOCGPART (get partition information), DIOCWDINFO (write partition information) and DIOCSDINFO (set partition information). DIOCGDINFO and DIOCGPART refer to an internal representation of the disk label which is not present on the volume. As a result, the .Fl r option of .Xr disklabel 8 , which reads the .if t ``raw disk'', .if n "raw disk", will fail. .Pp In general, .Xr disklabel 8 serves no useful purpose on a vinum volume. If you run it, it will show you three partitions, a, b and c, all the same except for the fstype, for example: .br .ne 1i .Bd -literal -offset 3 partitions: # size offset fstype [fsize bsize bps/cpg] a: 2048 0 4.2BSD 1024 8192 0 # (Cyl. 0 - 0) b: 2048 0 swap # (Cyl. 0 - 0) c: 2048 0 unused 0 0 # (Cyl. 0 - 0) .Ed .Pp .Nm ignores the DIOCWDINFO and DIOCSDINFO ioctls, since there is nothing to change. As a result, any attempt to modify the disk label will be silently ignored. .Sh MAKING FILE SYSTEMS Since .Nm volumes do not contain partitions, the names do not need to conform to the standard rules for naming disk partitions. For a physical disk partition, the last letter of the device name specifies the partition identifier (a to h). .Nm volumes need not conform to this convention, but if they do not, .Nm newfs will complain that it cannot determine the partition. To solve this problem, use the .Fl v flag to .Nm newfs . For example, if you have a volume .Pa concat , use the following command to create a ufs file system on it: .Pp .Bd -literal # newfs -v /dev/vinum/concat .Ed .Sh OBJECT NAMING .Nm assigns default names to plexes and subdisks, although they may be overridden. We do not recommend overriding the default names. Experience with the .if t Veritas\(tm .if n Veritas(tm) volume manager, which allows arbitary naming of objects, has shown that this flexibility does not bring a significant advantage, and it can cause confusion. .sp Names may contain any non-blank character, but it is recommended to restrict them to letters, digits and the underscore characters. The names of volumes, plexes and subdisks may be up to 64 characters long, and the names of drives may up to 32 characters long. When choosing volume and plex names, bear in mind that automatically generated plex and subdisk names are longer than the name from which they are derived. .Bl -bullet .It When .Xr vinum 8 creates or deletes objects, it creates a directory .Pa /dev/vinum , in which it makes device entries for each volume. It also creates the subdirectories .Pa /dev/vinum/plex and .Pa /dev/vinum/sd , in which it stores device entries for the plexes and subdisks. In addition, it creates two more directories, .Pa /dev/vinum/vol and .Pa /dev/vinum/drive , in which it stores hierarchical information for volumes and drives. .It In addition, .Nm creates two super-devices, .Pa /dev/vinum/control and .Pa /dev/vinum/controld . .Pa /dev/vinum/control is used by .Xr vinum 8 , and .Pa /dev/vinum/controld is used by the .Nm daemon. .It Unlike .Nm .Ux drives, .Nm volumes are not subdivided into partitions, and thus do not contain a disk label. Unfortunately, this confuses a number of utilities, notably .Nm newfs , which normally tries to interpret the last letter of a .Nm volume name as a partition identifier. If you use a volume name which does not end in the letters .Ar a to .Ar c , you must use the .Fl v flag to .Nm newfs in order to tell it to ignore this convention. .\" .It Plexes do not need to be assigned explicit names. By default, a plex name is the name of the volume followed by the letters .Em \&.p and the number of the plex. For example, the plexes of volume .Ar vol3 are called .Ar vol3.p0 , .Ar vol3.p1 and so on. These names can be overridden, but it is not recommended. .br .It Like plexes, subdisks are assigned names automatically, and explicit naming is discouraged. A subdisk name is the name of the plex followed by the letters .Em \&.s and a number identifying the subdisk. For example, the subdisks of plex .Ar vol3.p0 are called .Ar vol3.p0.s0 , .Ar vol3.p0.s1 and so on. .br .It By contrast, .Nm drives must be named. This makes it possible to move a drive to a different location and still recognize it automatically. Drive names may be up to 32 characters long. .El .Pp EXAMPLE .Pp Assume the .Nm objects described in the section CONFIGURATION FILE in .Xr vinum 8 . The directory .Ar /dev/vinum looks like: .Bd -literal -offset indent # ls -lR /dev/vinum total 5 crwxr-xr-- 1 root wheel 162, 2 Mar 30 16:08 concat crwx------ 1 root wheel 162, 0x40000000 Mar 30 16:08 control crwx------ 1 root wheel 162, 0x40000001 Mar 30 16:08 controld drwxrwxrwx 2 root wheel 512 Mar 30 16:08 drive drwxrwxrwx 2 root wheel 512 Mar 30 16:08 plex drwxrwxrwx 2 root wheel 512 Mar 30 16:08 rvol drwxrwxrwx 2 root wheel 512 Mar 30 16:08 sd crwxr-xr-- 1 root wheel 162, 3 Mar 30 16:08 strcon crwxr-xr-- 1 root wheel 162, 1 Mar 30 16:08 stripe crwxr-xr-- 1 root wheel 162, 0 Mar 30 16:08 tinyvol drwxrwxrwx 7 root wheel 512 Mar 30 16:08 vol crwxr-xr-- 1 root wheel 162, 4 Mar 30 16:08 vol5 /dev/vinum/drive: total 0 crw-r----- 1 root operator 4, 15 Oct 21 16:51 drive2 crw-r----- 1 root operator 4, 31 Oct 21 16:51 drive4 /dev/vinum/plex: total 0 crwxr-xr-- 1 root wheel 162, 0x10000002 Mar 30 16:08 concat.p0 crwxr-xr-- 1 root wheel 162, 0x10010002 Mar 30 16:08 concat.p1 crwxr-xr-- 1 root wheel 162, 0x10000003 Mar 30 16:08 strcon.p0 crwxr-xr-- 1 root wheel 162, 0x10010003 Mar 30 16:08 strcon.p1 crwxr-xr-- 1 root wheel 162, 0x10000001 Mar 30 16:08 stripe.p0 crwxr-xr-- 1 root wheel 162, 0x10000000 Mar 30 16:08 tinyvol.p0 crwxr-xr-- 1 root wheel 162, 0x10000004 Mar 30 16:08 vol5.p0 crwxr-xr-- 1 root wheel 162, 0x10010004 Mar 30 16:08 vol5.p1 /dev/vinum/sd: total 0 crwxr-xr-- 1 root wheel 162, 0x20000002 Mar 30 16:08 concat.p0.s0 crwxr-xr-- 1 root wheel 162, 0x20100002 Mar 30 16:08 concat.p0.s1 crwxr-xr-- 1 root wheel 162, 0x20010002 Mar 30 16:08 concat.p1.s0 crwxr-xr-- 1 root wheel 162, 0x20000003 Mar 30 16:08 strcon.p0.s0 crwxr-xr-- 1 root wheel 162, 0x20100003 Mar 30 16:08 strcon.p0.s1 crwxr-xr-- 1 root wheel 162, 0x20010003 Mar 30 16:08 strcon.p1.s0 crwxr-xr-- 1 root wheel 162, 0x20110003 Mar 30 16:08 strcon.p1.s1 crwxr-xr-- 1 root wheel 162, 0x20000001 Mar 30 16:08 stripe.p0.s0 crwxr-xr-- 1 root wheel 162, 0x20100001 Mar 30 16:08 stripe.p0.s1 crwxr-xr-- 1 root wheel 162, 0x20000000 Mar 30 16:08 tinyvol.p0.s0 crwxr-xr-- 1 root wheel 162, 0x20100000 Mar 30 16:08 tinyvol.p0.s1 crwxr-xr-- 1 root wheel 162, 0x20000004 Mar 30 16:08 vol5.p0.s0 crwxr-xr-- 1 root wheel 162, 0x20100004 Mar 30 16:08 vol5.p0.s1 crwxr-xr-- 1 root wheel 162, 0x20010004 Mar 30 16:08 vol5.p1.s0 crwxr-xr-- 1 root wheel 162, 0x20110004 Mar 30 16:08 vol5.p1.s1 /dev/vinum/vol: total 5 crwxr-xr-- 1 root wheel 162, 2 Mar 30 16:08 concat drwxr-xr-x 4 root wheel 512 Mar 30 16:08 concat.plex crwxr-xr-- 1 root wheel 162, 3 Mar 30 16:08 strcon drwxr-xr-x 4 root wheel 512 Mar 30 16:08 strcon.plex crwxr-xr-- 1 root wheel 162, 1 Mar 30 16:08 stripe drwxr-xr-x 3 root wheel 512 Mar 30 16:08 stripe.plex crwxr-xr-- 1 root wheel 162, 0 Mar 30 16:08 tinyvol drwxr-xr-x 3 root wheel 512 Mar 30 16:08 tinyvol.plex crwxr-xr-- 1 root wheel 162, 4 Mar 30 16:08 vol5 drwxr-xr-x 4 root wheel 512 Mar 30 16:08 vol5.plex /dev/vinum/vol/concat.plex: total 2 crwxr-xr-- 1 root wheel 162, 0x10000002 Mar 30 16:08 concat.p0 drwxr-xr-x 2 root wheel 512 Mar 30 16:08 concat.p0.sd crwxr-xr-- 1 root wheel 162, 0x10010002 Mar 30 16:08 concat.p1 drwxr-xr-x 2 root wheel 512 Mar 30 16:08 concat.p1.sd /dev/vinum/vol/concat.plex/concat.p0.sd: total 0 crwxr-xr-- 1 root wheel 162, 0x20000002 Mar 30 16:08 concat.p0.s0 crwxr-xr-- 1 root wheel 162, 0x20100002 Mar 30 16:08 concat.p0.s1 /dev/vinum/vol/concat.plex/concat.p1.sd: total 0 crwxr-xr-- 1 root wheel 162, 0x20010002 Mar 30 16:08 concat.p1.s0 /dev/vinum/vol/strcon.plex: total 2 crwxr-xr-- 1 root wheel 162, 0x10000003 Mar 30 16:08 strcon.p0 drwxr-xr-x 2 root wheel 512 Mar 30 16:08 strcon.p0.sd crwxr-xr-- 1 root wheel 162, 0x10010003 Mar 30 16:08 strcon.p1 drwxr-xr-x 2 root wheel 512 Mar 30 16:08 strcon.p1.sd /dev/vinum/vol/strcon.plex/strcon.p0.sd: total 0 crwxr-xr-- 1 root wheel 162, 0x20000003 Mar 30 16:08 strcon.p0.s0 crwxr-xr-- 1 root wheel 162, 0x20100003 Mar 30 16:08 strcon.p0.s1 /dev/vinum/vol/strcon.plex/strcon.p1.sd: total 0 crwxr-xr-- 1 root wheel 162, 0x20010003 Mar 30 16:08 strcon.p1.s0 crwxr-xr-- 1 root wheel 162, 0x20110003 Mar 30 16:08 strcon.p1.s1 /dev/vinum/vol/stripe.plex: total 1 crwxr-xr-- 1 root wheel 162, 0x10000001 Mar 30 16:08 stripe.p0 drwxr-xr-x 2 root wheel 512 Mar 30 16:08 stripe.p0.sd /dev/vinum/vol/stripe.plex/stripe.p0.sd: total 0 crwxr-xr-- 1 root wheel 162, 0x20000001 Mar 30 16:08 stripe.p0.s0 crwxr-xr-- 1 root wheel 162, 0x20100001 Mar 30 16:08 stripe.p0.s1 /dev/vinum/vol/tinyvol.plex: total 1 crwxr-xr-- 1 root wheel 162, 0x10000000 Mar 30 16:08 tinyvol.p0 drwxr-xr-x 2 root wheel 512 Mar 30 16:08 tinyvol.p0.sd /dev/vinum/vol/tinyvol.plex/tinyvol.p0.sd: total 0 crwxr-xr-- 1 root wheel 162, 0x20000000 Mar 30 16:08 tinyvol.p0.s0 crwxr-xr-- 1 root wheel 162, 0x20100000 Mar 30 16:08 tinyvol.p0.s1 /dev/vinum/vol/vol5.plex: total 2 crwxr-xr-- 1 root wheel 162, 0x10000004 Mar 30 16:08 vol5.p0 drwxr-xr-x 2 root wheel 512 Mar 30 16:08 vol5.p0.sd crwxr-xr-- 1 root wheel 162, 0x10010004 Mar 30 16:08 vol5.p1 drwxr-xr-x 2 root wheel 512 Mar 30 16:08 vol5.p1.sd /dev/vinum/vol/vol5.plex/vol5.p0.sd: total 0 crwxr-xr-- 1 root wheel 162, 0x20000004 Mar 30 16:08 vol5.p0.s0 crwxr-xr-- 1 root wheel 162, 0x20100004 Mar 30 16:08 vol5.p0.s1 /dev/vinum/vol/vol5.plex/vol5.p1.sd: total 0 crwxr-xr-- 1 root wheel 162, 0x20010004 Mar 30 16:08 vol5.p1.s0 crwxr-xr-- 1 root wheel 162, 0x20110004 Mar 30 16:08 vol5.p1.s1 .Ed .Pp In the case of unattached plexes and subdisks, the naming is reversed. Subdisks are named after the disk on which they are located, and plexes are named after the subdisk. .\" XXX .Em This mapping is still to be determined . .Ss OBJECT STATES Each .Nm object has a .Em state associated with it. .Nm uses this state to determine the handling of the object. .Ss VOLUME STATES Volumes may have the following states: .sp .Bl -hang -width 14n .It Li down The volume is completely inaccessible. .It Li up The volume is up and at least partially functional. Not all plexes may be available. .El .Ss "PLEX STATES" Plexes may have the following states: .sp .ne 1i .Bl -hang -width 14n .It Li referenced A plex entry which has been referenced as part of a volume, but which is currently not known. .It Li faulty A plex which has gone completely down because of I/O errors. .It Li down A plex which has been taken down by the administrator. .It Li initializing A plex which is being initialized. .sp The remaining states represent plexes which are at least partially up. .It Li corrupt A plex entry which is at least partially up. Not all subdisks are available, and an inconsistency has occurred. If no other plex is uncorrupted, the volume is no longer consistent. .It Li degraded A RAID-5 plex entry which is accessible, but one subdisk is down, requiring recovery for many I/O requests. .It Li flaky A plex which is really up, but which has a reborn subdisk which we don't completely trust, and which we don't want to read if we can avoid it. .It Li up A plex entry which is completely up. All subdisks are up. .El .sp 2v .Ss "SUBDISK STATES" Subdisks can have the following states: .sp .ne 1i .Bl -hang -width 14n .It Li empty A subdisk entry which has been created completely. All fields are correct, and the disk has been updated, but the on the disk is not valid. .It Li referenced A subdisk entry which has been referenced as part of a plex, but which is currently not known. .It Li initializing A subdisk entry which has been created completely and which is currently being initialized. .sp The following states represent invalid data. .It Li obsolete A subdisk entry which has been created completely. All fields are correct, the config on disk has been updated, and the data was valid, but since then the drive has been taken down, and as a result updates have been missed. .It Li stale A subdisk entry which has been created completely. All fields are correct, the disk has been updated, and the data was valid, but since then the drive has been crashed and updates have been lost. .sp The following states represent valid, inaccessible data. .It Li crashed A subdisk entry which has been created completely. All fields are correct, the disk has been updated, and the data was valid, but since then the drive has gone down. No attempt has been made to write to the subdisk since the crash, so the data is valid. .It Li down A subdisk entry which was up, which contained valid data, and which was taken down by the administrator. The data is valid. .It Li reviving The subdisk is currently in the process of being revived. We can write but not read. .sp The following states represent accessible subdisks with valid data. .It Li reborn A subdisk entry which has been created completely. All fields are correct, the disk has been updated, and the data was valid, but since then the drive has gone down and up again. No updates were lost, but it is possible that the subdisk has been damaged. We won't read from this subdisk if we have a choice. If this is the only subdisk which covers this address space in the plex, we set its state to up under these circumstances, so this status implies that there is another subdisk to fulfil the request. .It Li up A subdisk entry which has been created completely. All fields are correct, the disk has been updated, and the data is valid. .El .sp 2v .Ss "DRIVE STATES" Drives can have the following states: .sp .ne 1i .Bl -hang -width 14n .It Li referenced At least one subdisk refers to the drive, but it is not currently accessible to the system. No device name is known. .It Li down The drive is not accessible. .It Li up The drive is up and running. .El .sp 2v .Sh SEE ALSO .Xr disklabel 5 , .Xr disklabel 8 , .Xr newfs 8 , .Xr vinum 8 .Sh HISTORY .Nm vinum first appeared in .Fx 3.0 . The RAID-5 component of .Nm was developed by Cybernet Inc. .Pa www.cybernet.com for its NetMAX product. .Sh AUTHORS .An Greg Lehey Aq grog@lemis.com .Sh BUGS .Bl -enum .It Detection of differences between the version of the kernel and the LKM is not yet implemented. .El .Ss Reporting problems with Vinum If you find any bugs in .Nm vinum , please report them to Greg Lehey .Aq grog@NetBSD.org . Supply the following information: .Pp .Bl -bullet .It The output of the .Nm vinum list command. .It Any messages printed in .Pa /var/log/messages . All such messages will be identified by the text .Nm at the beginning. .It If you have a panic, a stack trace. .El