MAN.9FRONT.ORG RTFM


     VENTI-FMT(8)                                         VENTI-FMT(8)

     NAME
          buildindex, checkarenas, checkindex, conf, fmtarenas,
          fmtbloom, fmtindex, fmtisect, syncindex - prepare and
          maintain a venti server

     SYNOPSIS
          venti/fmtarenas [ -Z ] [ -a arenasize ] [ -b blocksize ]
          name file

          venti/fmtisect [ -1Z ] [ -b blocksize ] name file

          venti/fmtbloom [ -n nblocks | -N nhash ] [ -s size ] file

          venti/fmtindex [ -a ] venti.conf

          venti/conf [ -w ] partition [ configfile ]

          venti/buildindex [ -bd ] [ -i isect ] ... [ -M imemsize ]
          venti.conf

          venti/checkindex [ -f ] [ -B blockcachesize ] venti.conf tmp

          venti/checkarenas [ -afv ] file

     DESCRIPTION
          These commands aid in the setup, maintenance, and debugging
          of venti servers.  See venti(6) for an overview of the venti
          system and venti(8) for an overview of the data structures
          used by the venti server.

          Note that the units for the various sizes in the following
          commands can be specified by appending `k', `m', or `g' to
          indicate kilobytes, megabytes, or gigabytes respectively.

        Formatting
          To prepare a server for its initial use, the arena parti-
          tions and the index sections must be formatted individually,
          with fmtarenas and fmtisect. Then the collection of index
          sections must be combined into a venti index with fmtindex.

          Fmtarenas formats the given file, typically a disk parti-
          tion, into an arena partition.  The arenas in the partition
          are given names of the form name%d, where %d is replaced
          with a sequential number starting at 0.

          Options to fmtarenas are:

          -a arenasize
               The arenas are of arenasize bytes.  The default is
               512M, which was selected to provide a balance between

     VENTI-FMT(8)                                         VENTI-FMT(8)

               the number of arenas and the ability to copy an arena
               to external media such as recordable CDs and tapes.

          -b blocksize
               The size, in bytes, for read and write operations to
               the file.  The size is recorded in the file, and is
               used by applications that access the arenas.  The
               default is 8k.

          -4   Create a `version 4' arena partition for backwards com-
               patibility with old servers.  The default is version 5,
               used by the current venti server.

          -Z   Do not zero the data sections of the arenas.  Using
               this option reduces the formatting time but should only
               be used when it is known that the file was already
               zeroed.  (Version 4 only; version 5 sections are not
               and do not need to be zeroed.)

          Fmtisect formats the given file, typically a disk partition,
          as a venti index section with the specified name. Each of
          the index sections in a venti configuration must have a
          unique name.

          Options to fmtisect are:

          -b bucketsize
               The size of an index bucket, in bytes.  All the index
               sections within a index must have the same bucket size.
               The default is 8k.

          -1   Create a `version 1' index section for backwards com-
               patibility with old servers.  The default is version 2,
               used by the current venti server.

          -Z   Do not zero the index.  Using this option reduces the
               formatting time but should only be used when it is
               known that the file was already zeroed.  (Version 1
               only; version 2 sections are not and do not need to be
               zeroed.)

          Fmtbloom formats the given file as a Bloom filter (see
          venti(6)). The options are:

          -n nblock | -N nhash
                   The number of blocks expected to be indexed by the
                   filter or the number of hash functions to use.  If
                   the -n option is given, it is used, along with the
                   total size of the filter, to compute an appropriate
                   nhash.

     VENTI-FMT(8)                                         VENTI-FMT(8)

          -s size  The size of the Bloom filter.  The default is the
                   total size of the file.  In either case, size is
                   rounded down to a power of two.

          The file argument in the commands above can be of the form
          file:lo-hi to specify a range of the file. Lo and hi are
          specified in bytes but can have the usual k, m, or g suf-
          fixes.  Either lo or hi may be omitted.  This notation elim-
          inates the need to partition raw disks on non-Plan 9 sys-
          tems.

          Fmtindex reads the configuration file venti.conf and ini-
          tializes the index sections to form a usable index struc-
          ture.  The arena files and index sections must have previ-
          ously been formatted using fmtarenas and fmtisect respec-
          tively.

          The function of a venti index is to map a SHA1 fingerprint
          to a location in the data section of one of the arenas.  The
          index is composed of blocks, each of which contains the map-
          ping for a fixed range of possible fingerprint values.
          Fmtindex determines the mapping between SHA1 values and the
          blocks of the collection of index sections.  Once this map-
          ping has been determined, it cannot be changed without
          rebuilding the index. The basic assumption in the current
          implementation is that the index structure is sufficiently
          empty that individual blocks of the index will rarely over-
          flow.  The total size of the index should be about 2% to 10%
          of the total size of the arenas, but the exact percentage
          depends both on the index block size and the compressed size
          of blocks stored.  See the discussion in venti(8) for more.

          Fmtindex also computes a mapping between a linear address
          space and the data section of the collection of arenas.  The
          -a option can be used to add additional arenas to an index.
          To use this feature, add the new arenas to venti.conf after
          the existing arenas and then run fmtindex -a.

          A copy of the above mappings is stored in the header for
          each of the index sections.  These copies enable buildindex
          to restore a single index section without rebuilding the
          entire index.

          To make it easier to bootstrap servers, the configuration
          file can be stored in otherwise empty space at the beginning
          of any venti partitions using conf. A partition so branded
          with a configuration file can be used in place of a configu-
          ration file when invoking any of the venti commands.  By
          default, conf prints the configuration stored in partition.
          When invoked with the -w flag, conf reads a configuration
          file from configfile (or else standard input) and stores it
          in partition.

     VENTI-FMT(8)                                         VENTI-FMT(8)

        Checking and Rebuilding
          Buildindex populates the index for the Venti system
          described in venti.conf. The index must have previously been
          formatted using fmtindex. This command is typically used to
          build a new index for a Venti system when the old index
          becomes too small, or to rebuild an index after media fail-
          ure.  Small errors in an index can usually be fixed with
          checkindex, but checkindex requires a large temporary
          workspace and buildindex does not.

          Options to buildindex are:

          -b           Reinitialise the Bloom filter, if any.

          -d           `Dumb' mode; run all three passes.

          -i isect     Only rebuild index section isect; may be
                       repeated to rebuild multiple sections.  The
                       name `none' is special and just reads the are-
                       nas.

          -M imemsize  The amount of memory, in bytes, to use for
                       caching raw disk accesses while running
                       buildindex. (This is not a property of the cre-
                       ated index.)  The usual suffices apply.  The
                       default is 256M.

          Checkindex examines the Venti index described in venti.conf.
          The program detects various error conditions including:
          blocks that are not indexed, index entries for blocks that
          do not exist, and duplicate index entries.  If requested, an
          attempt can be made to fix errors that are found.

          The tmp file, usually a disk partition, must be large enough
          to store a copy of the index.  This temporary space is used
          to perform a merge sort of index entries generated by read-
          ing the arenas.

          Options to checkindex are:

          -B blockcachesize
               The amount of memory, in bytes, to use for caching raw
               disk accesses while running checkindex. The default is
               8k.

          -f   Attempt to fix any errors that are found.

          Checkarenas examines the Venti arenas contained in the given
          file. The program detects various error conditions, and
          optionally attempts to fix any errors that are found.

     VENTI-FMT(8)                                         VENTI-FMT(8)

          Options to checkarenas are:

          -a   For each arena, scan the entire data section.  If this
               option is omitted, only the end section of the arena is
               examined.

          -f   Attempt to fix any errors that are found.

          -v   Increase the verbosity of output.

     SOURCE
          /sys/src/cmd/venti/srv

     SEE ALSO
          venti(6), venti(8)

     BUGS
          Buildindex should allow an individual index section to be
          rebuilt.