Refactor configuration management in bhyve.

Replace the existing ad-hoc configuration via various global variables
with a small database of key-value pairs.  The database supports
heirarchical keys using a MIB-like syntax to name the path to a given
key.  Values are always stored as strings.  The API used to manage
configuation values does include wrappers to handling boolean values.
Other values use non-string types require parsing by consumers.

The configuration values are stored in a tree using nvlists.  Leaf
nodes hold string values.  Configuration values are permitted to
reference other configuration values using '%(name)'.  This permits
constructing template configurations.

All existing command line arguments now set configuration values.  For
devices, the "-s" option parses its option argument to generate a list
of key-value pairs for the given device.

A new '-o' command line option permits setting an individual
configuration variable.  The key name is always given as a full path
of dot-separated components.

A new '-k' command line option parses a simple configuration file.
This configuration file holds a flat list of 'key=value' lines where
the 'key' is the full path of a configuration variable.  Lines
starting with a '#' are comments.

In general, bhyve starts by parsing command line options in sequence
and applying those settings to configuration values.  Once this is
complete, bhyve then begins initializing its state based on the
configuration values.  This means that subsequent configuration
options or files may override or supplement previously given settings.

A special 'config.dump' configuration value can be set to true to help
debug configuration issues.  When this value is set, bhyve will print
out the configuration variables as a flat list of 'key=value' lines.

Most command line argments map to a single configuration variable,
e.g.  '-w' sets the 'x86.strictmsr' value to false.  A few command
line arguments have less obvious effects:

- Multiple '-p' options append their values (as a comma-seperated
  list) to "vcpu.N.cpuset" values (where N is a decimal vcpu number).

- For '-s' options, a pci.<bus>.<slot>.<function> node is created.
  The first argument to '-s' (the device type) is used as the value of
  a "device" variable.  Additional comma-separated arguments are then
  parsed into 'key=value' pairs and used to set additional variables
  under the device node.  A PCI device emulation driver can provide
  its own hook to override the parsing of the additonal '-s' arguments
  after the device type.

  After the configuration phase as completed, the init_pci hook
  then walks the "pci.<bus>.<slot>.<func>" nodes.  It uses the
  "device" value to find the device model to use.  The device
  model's init routine is passed a reference to its nvlist node
  in the configuration tree which it can query for specific
  variables.

  The result is that a lot of the string parsing is removed from
  the device models and centralized.  In addition, adding a new
  variable just requires teaching the model to look for the new
  variable.

- For '-l' options, a similar model is used where the string is
  parsed into values that are later read during initialization.
  One key note here is that the serial ports use the commonly
  used lowercase names from existing documentation and examples
  (e.g. "lpc.com1") instead of the uppercase names previously
  used internally in bhyve.

Reviewed by:	grehan
MFC after:	3 months
Differential Revision:	https://reviews.freebsd.org/D26035

48 files changed, 2593 insertions, 1071 deletions

diff --git a/usr.sbin/bhyve/Makefile b/usr.sbin/bhyve/Makefile
index a4fc3deea77e..e35d528ab605 100644
--- a/usr.sbin/bhyve/Makefile
+++ b/usr.sbin/bhyve/Makefile
@@ -10,7 +10,7 @@ CFLAGS+=-I${SRCTOP}/sys
 PROG=	bhyve
 PACKAGE=	bhyve
 
-MAN=	bhyve.8
+MAN=	bhyve.8 bhyve_config.5
 
 BHYVE_SYSDIR?=${SRCTOP}
 
@@ -22,6 +22,7 @@ SRCS=	\
 	bhyverun.c		\
 	block_if.c		\
 	bootrom.c		\
+	config.c		\
 	console.c		\
 	ctl_util.c		\
 	ctl_scsi_all.c		\
@@ -83,7 +84,7 @@ CFLAGS.kernemu_dev.c+=	-I${SRCTOP}/sys/amd64
 .PATH:  ${BHYVE_SYSDIR}/sys/amd64/vmm
 SRCS+=	vmm_instruction_emul.c
 
-LIBADD=	vmmapi md pthread z util sbuf cam 9p
+LIBADD=	vmmapi md nv pthread z util sbuf cam 9p
 
 .if ${MK_CASPER} != "no"
 LIBADD+=	casper
diff --git a/usr.sbin/bhyve/bhyve.8 b/usr.sbin/bhyve/bhyve.8
index 115727a136a7..04e22302d9d7 100644
--- a/usr.sbin/bhyve/bhyve.8
+++ b/usr.sbin/bhyve/bhyve.8
@@ -24,7 +24,7 @@
 .\"
 .\" $FreeBSD$
 .\"
-.Dd January 18, 2021
+.Dd March 18, 2021
 .Dt BHYVE 8
 .Os
 .Sh NAME
@@ -46,6 +46,7 @@
 .Oc
 .Sm on
 .Op Fl G Ar port
+.Op Fl k Ar file
 .Oo Fl l
 .Sm off
 .Cm help | Ar lpcdev Op Cm \&, Ar conf
@@ -59,6 +60,7 @@
 .Oc
 .Sm on
 .Oc
+.Op Fl o Ar var Ns Cm = Ns Ar value
 .Op Fl p Ar vcpu Ns Cm \&: Ns Ar hostcpu
 .Op Fl r Ar file
 .Oo Fl s
@@ -149,6 +151,17 @@ Print help message and exit.
 .It Fl H
 Yield the virtual CPU thread when a HLT instruction is detected.
 If this option is not specified, virtual CPUs will use 100% of a host CPU.
+.It Fl k Ar file
+Set configuration variables from a simple, key-value config file.
+Each line of the config file is expected to consist of a config variable
+name, an equals sign
+.Pq Sq = ,
+and a value.
+No spaces are permitted between the variable name, equals sign, or
+value.
+Blank lines and lines starting with
+.Sq #
+are ignored.
 .It Fl l Op Ar help|lpcdev Ns Op , Ns Ar conf
 Allow devices behind the LPC PCI-ISA bridge to be configured.
 The only supported devices are the TTY-class devices
@@ -174,6 +187,11 @@ If no suffix is given, the value is assumed to be in megabytes.
 .Pp
 .Ar memsize
 defaults to 256M.
+.It Fl o Ar var Ns Cm = Ns Ar value
+Set the configuration variable
+.Ar var
+to
+.Ar value .
 .It Fl p Ar vcpu:hostcpu
 Pin guest's virtual CPU
 .Em vcpu
@@ -594,6 +612,32 @@ Alphanumeric name of the guest.
 This should be the same as that created by
 .Xr bhyveload 8 .
 .El
+.Sh CONFIGURATION VARIABLES
+.Nm
+uses an internal tree of configuration variables to describe global and
+per-device settings.
+When
+.Nm
+starts,
+it parses command line options (including config files) in the order given
+on the command line.
+Each command line option sets one or more configuration variables.
+For example,
+the
+.Fl s
+option creates a new tree node for a PCI device and sets one or more variables
+under that node including the device model and device model-specific variables.
+Variables may be set multiple times during this parsing stage with the final
+value overriding previous values.
+.Pp
+Once all of the command line options have been processed,
+the configuration values are frozen.
+.Nm
+then uses the value of configuration values to initialize device models
+and global settings.
+.Pp
+More details on configuration variables can be found in
+.Xr bhyve_config 5 .
 .Sh DEBUG SERVER
 The current debug server provides limited support for debuggers.
 .Ss Registers
@@ -717,6 +761,7 @@ bhyve -c 2 -m 4G -w -H \\
 .Xr ng_socket 4 ,
 .Xr nmdm 4 ,
 .Xr vmm 4 ,
+.Xr bhyve_config 5 ,
 .Xr ethers 5 ,
 .Xr bhyvectl 8 ,
 .Xr bhyveload 8
diff --git a/usr.sbin/bhyve/bhyve_config.5 b/usr.sbin/bhyve/bhyve_config.5
new file mode 100644
index 000000000000..4e200a779d50
--- /dev/null
+++ b/usr.sbin/bhyve/bhyve_config.5
@@ -0,0 +1,560 @@
+.\" SPDX-License-Identifier: BSD-2-Clause
+.\"
+.\" Copyright (c) 2021 John H. Baldwin <jhb@FreeBSD.org>
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\"    notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\"    notice, this list of conditions and the following disclaimer in the
+.\"    documentation and/or other materials provided with the distribution.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\" SUCH DAMAGE.
+.\"
+.Dd March 18, 2021
+.Dt BHYVE_CONFIG 5
+.Os
+.Sh NAME
+.Nm bhyve_config
+.Nd "bhyve configuration variables"
+.Sh DESCRIPTION
+.Xr bhyve 8
+uses a hierarchical tree of configuration variables to describe global and
+per-device settings.
+Internal nodes in this tree do not have a value,
+only leaf nodes have values.
+This manual describes the configuration variables understood by
+.Xr bhyve 8 .
+If additional variables are defined,
+.Xr bhyve 8
+will ignore them and will not emit errors for unknown variables.
+However, these additional variables can be referenced by other
+variables as described below.
+.Sh VARIABLE VALUES
+Configuration variable values are stored as strings.
+A configuration variable value may refer to one or more other
+configuration values by name.
+Instances of the pattern
+.Sq % Ns Pq Ar var
+are replaced by the value of the configuration variable
+.Va var .
+To avoid unwanted expansion,
+.Sq %
+characters can be escaped by a leading
+.Sq % .
+For example,
+if a configuration variable
+.Va disk
+uses the value
+.Pa /dev/zvol/bhyve/%(name) ,
+then the final value of the
+.Va disk
+variable will be set to the path of a ZFS volume whose name matches
+the name of the virtual machine on the pool
+.Pa bhyve .
+.Pp
+Some configuration variables may be interpreted as a boolean value.
+For those variables the following case-insensitive values may be used to
+indicate true:
+.Pp
+.Bl -bullet -offset indent -compact
+.It
+true
+.It
+on
+.It
+yes
+.It
+1
+.El
+.Pp
+The following values may be used to indicate false:
+.Pp
+.Bl -bullet -offset indent -compact
+.It
+false
+.It
+off
+.It
+no
+.It
+0
+.El
+.Pp
+Some configuration variables may be interperted as an integer.
+For those variables,
+any syntax supported by
+.Xr strtol 3
+may be used.
+.Sh GLOBAL SETTINGS
+.Ss Architecture Neutral Settings
+.Bl -column "memory.guest_in_core" "integer" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va name Ta string Ta Ta
+The name of the VM.
+.It Va cpus Ta integer Ta 1 Ta
+The total number of virtual CPUs.
+.It Va cores Ta integer Ta 1 Ta
+The number of virtual cores in each virtual socket.
+.It Va threads Ta integer Ta 1 Ta
+The number of virtual CPUs in each virtual core.
+.It Va sockets Ta integer Ta 1 Ta
+The number of virtual sockets.
+.It Va memory.guest_in_core Ta bool Ta false Ta
+Include guest memory in core file.
+.It Va memory.size Ta string Ta 256M Ta
+Guest physical memory size in bytes.
+The value must be formatted as described in
+.Xr expand_number 3 .
+.It Va memory.wired Ta bool Ta false Ta
+Wire guest memory.
+.It Va acpi_tables Ta bool Ta false Ta
+Generate ACPI tables.
+.It Va destroy_on_poweroff Ta bool Ta false Ta
+Destroy the VM on guest-initiated power-off.
+.It Va gdb.port Ta integer Ta 0 Ta
+TCP port number for the debug server.
+If this is set to a non-zero value, a debug server
+will listen for connections on this port.
+.It Va gdb.wait Ta bool Ta false Ta
+If the debug server is enabled, wait for a debugger to connect
+before starting the guest.
+.It Va rtc.use_localtime Ta bool Ta true Ta
+The real time clock uses the local time of the host.
+If this is set to false, the real time clock uses UTC.
+.It Va uuid Ta string Ta Ta
+The universally unique identifier (UUID) to use in the guest's
+System Management BIOS System Information structure.
+If an explicit value is not set, a valid UUID is generated from
+the host's hostname and the VM name.
+.It Va virtio_msix Ta bool Ta true Ta
+Use MSI-X interrupts for PCI VirtIO devices.
+If set to false, MSI interrupts are used instead.
+.It Va config.dump Ta bool Ta false Ta
+If this value is set to true,
+then
+.Xr bhyve 8
+will write all of its configuration variables to stdout and exit
+after it has finished parsing command line options.
+.El
+.Ss x86-Specific Settings
+.Bl -column "x86.vmexit_on_pause" "integer" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va x86.mptable Ta bool Ta true Ta
+Generate an MPTable.
+.It Va x86.x2apic Ta bool Ta false Ta
+Configure guest's local APICs in x2APIC mode.
+.It Va x86.strictio Ta bool Ta false Ta
+Exit if a guest accesses an I/O port that is not emulated.
+By default, writes are ignored and reads return all bits set.
+.It Va x86.strictmsr Ta bool Ta true Ta
+Inject a general protection fault if a guest accesses a Model Specific
+Register (MSR) that is not emulated.
+If this is false, writes are ignored and reads return zero.
+.It Va x86.vmexit_on_hlt Ta bool Ta false Ta
+Force a VM exit when a guest CPU executes the
+.Dv HLT
+instruction.
+This allows idle guest CPUs to yield the host CPU.
+.It Va x86.vmexit_on_pause Ta bool Ta false Ta
+Force a VM exit when a guest CPU executes the
+.Dv PAUSE
+instruction.
+.El
+.Sh DEVICE SETTINGS
+Device settings are stored under a device node.
+The device node's name is set by the parent bus of the device.
+.Ss PCI Device Settings
+PCI devices are described by a device node named
+.Dq pci Ns Ar bus . Ns Ar slot . Ns Ar function
+where each of
+.Ar bus ,
+.Ar slot ,
+and
+.Ar function
+are formatted as decimal values with no padding.
+All PCI device nodes must contain a configuration variable named
+.Dq device
+which specifies the device model to use.
+The following PCI device models are supported:
+.Bl -tag -indent
+.It Li hostbridge
+Provide a simple PCI-Host bridge device.
+This is usually configured at pci0:0:0 and is required by most guest
+operating systems.
+.It Li ahci
+AHCI storage controller.
+.It Li e1000
+Intel e82545 network interface.
+.It Li fbuf
+VGA framebuffer device attached to VNC server.
+.It Li lpc
+LPC PCI-ISA bridge with COM1-COM4 16550 serial ports,
+a boot ROM,
+and an optional debug/test device.
+This device must be configured on bus 0.
+.It Li hda
+High Definition audio controller.
+.It Li nvme
+NVM Express (NVMe) controller.
+.It Li passthru
+PCI pass-through device.
+.It Li uart
+PCI 16550 serial device.
+.It Li virtio-9p
+VirtIO 9p (VirtFS) interface.
+.It Li virtio-blk
+VirtIO block storage interface.
+.It Li virtio-console
+VirtIO console interface.
+.It Li virtio-net
+VirtIO network interface.
+.It Li virtio-rnd
+VirtIO RNG interface.
+.It Li virtio-scsi
+VirtIO SCSI interface.
+.It Li xhci
+Extensible Host Controller Interface (XHCI) USB controller.
+.El
+.Ss USB Device Settings
+USB controller devices contain zero or more child USB devices
+attached to slots.
+Each USB device stores its settings in a node named
+.Dq slot. Ns Va N
+under the controller's device node.
+.Va N
+is the number of the slot to which the USB device is attached.
+Note that USB slot numbers begin at 1.
+All USB device nodes must contain a configuration variable named
+.Dq device
+which specifies the device model to use.
+The following USB device models are supported:
+.Bl -tag -indent
+.It Li tablet
+A USB tablet device which provides precise cursor synchronization
+when using VNC.
+.El
+.Ss Block Device Settings
+Block devices use the following settings to configure their backing store.
+These settings are stored in the configuration node of the respective device.
+.Bl -column "sectorsize" "logical[/physical]" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It path Ta string Ta Ta
+The path of the file or disk device to use as the backing store.
+.It nocache Ta bool Ta false Ta
+Disable caching on the backing file by opening the backing file with
+.Dv O_DIRECT .
+.It nodelete Ta bool Ta false Ta
+Disable emulation of guest trim requests via
+.Dv DIOCGDELETE
+requests.
+.It sync Ta bool Ta false Ta
+Write changes to the backing file with synchronous writes.
+.It direct Ta bool Ta false Ta
+An alias for
+.Va sync .
+.It ro Ta bool Ta false Ta
+Disable writes to the backing file.
+.It sectorsize Ta Va logical Ns Op / Ns Va physical Ta Ta
+Specify the logical and physical sector size of the emulated disk.
+If the physical size is not specified,
+it is equal to the logical size.
+.El
+.Ss Network Backend Settings
+Network devices use the following settings to configure their backend.
+The backend is responsible for passing packets between the device model
+and a desired destination.
+Configuring a backend requires setting the
+.Va backend
+variable to one of the following values:
+.Bl -tag
+.It tap Ns Va N
+Use the named
+.Xr tap 4
+interface as the backend.
+.It vmnet Ns Va N
+Use the named
+.Xr vmnet 4
+interface as the backend.
+.It netgraph
+Use a
+.Xr netgraph 4
+socket hook as the backend.
+This backend uses the following additional variables:
+.Bl -column "peerhook" "Format" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va path Ta string Ta Ta
+The name of the
+.Xr netgraph 4
+destination node.
+.It Va peerhook Ta string Ta Ta
+The name of the destination hook.
+.It Va socket Ta string Ta Ta
+The name of the created
+.Xr ng_socket 4
+node.
+.It Va hook Ta string Ta vmlink Ta
+The name of the source hook on the created
+.Xr ng_socket 4
+node.
+.El
+.It netmap: Ns Va interface
+Use
+.Xr netmap 4
+on a network interface as the backend.
+.It vale Ns Va bridge : Ns Va port
+Use a port on a
+.Xr vale 4
+bridge as the backend.
+.El
+.Ss UART Device Settings
+.Bl -column "Name" "Format" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va path Ta path Ta Ta
+Backend device for the serial port.
+Either the pathname of a character device or
+.Dq stdio
+to use standard input and output of the
+.Xr bhyve 8
+process.
+.El
+.Ss Host Bridge Settings
+.Bl -column "vendor" "integer" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va vendor Ta integer Ta 0x1275 Ta
+PCI vendor ID.
+.It Va device Ta integer Ta 0x1275 Ta
+PCI device ID.
+.El
+.Ss AHCI Controller Settings
+AHCI controller devices contain zero or more ports each of which
+provides a storage device.
+Each port stores its settings in a node named
+.Dq port. Ns Va N
+under the controller's device node.
+The
+.Va N
+values are formatted as successive decimal values starting with 0.
+In addition to the block device settings described above, each
+port supports the following settings:
+.Bl -column "model" "integer" "generated"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va type Ta string Ta Ta
+The type of storage device to emulate.
+Must be set to either
+.Dq cd
+or
+.Dq hd .
+.It Va nmrr Ta integer Ta 0 Ta
+Nominal Media Rotation Rate, also known as RPM.
+A value 1 of indicates a device with no rate such as a Solid State Disk.
+.It Va ser Ta string Ta generated Ta
+Serial number of up to twenty characters.
+A default serial number is generated using a hash of the backing
+store's pathname.
+.It Va rev Ta string Ta 001 Ta
+Revision number of up to eight characters.
+.It Va model Ta string Ta Ta
+Model number of up to forty characters.
+Separate default model strings are used for
+.Dq cd
+and
+.Dq hd
+device types.
+.El
+.Ss e1000 Settings
+In addition to the network backend settings,
+Intel e82545 network interfaces support the following variables:
+.Bl -column "Name" "MAC address" "generated"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va mac Ta MAC address Ta generated Ta
+MAC address.
+If an explicit address is not provided,
+a MAC address is generated from a hash of the device's PCI address.
+.El
+.Ss Frame Buffer Settings
+.Bl -column "password" "[IP:]port" "127.0.0.1:5900"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va wait Ta bool Ta false Ta
+Wait for a remote connection before starting the VM.
+.It Va rfb Ta Oo Ar IP Ns : Oc Ns Ar port Ta 127.0.0.1:5900 Ta
+TCP address to listen on for remote connections.
+The IP address must be given as a numeric address.
+IPv6 addresses must be enclosed in square brackets and
+support scoped identifiers as described in
+.Xr getaddrinfo 3 .
+A bare port number may be given in which case the IPv4
+localhost address is used.
+.It Va vga Ta string Ta io Ta
+VGA configuration.
+More details are provided in
+.Xr bhyve 8 .
+.It Va w Ta integer Ta 1024 Ta
+Frame buffer width in pixels.
+.It Va h Ta integer Ta 768 Ta
+Frame buffer height in pixels.
+.It Va password Ta string Ta Ta
+Password to use for VNC authentication.
+This type of authentication is known to be cryptographically weak and is not
+intended for use on untrusted networks.
+.El
+.Ss High Definition Audio Settings
+.Bl -column "Name" "Format" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va play Ta path Ta Ta
+Host playback device,
+typically
+.Pa /dev/dsp0 .
+.It Va rec Ta path Ta Ta
+Host recording device,
+typically
+.Pa /dev/dsp0 .
+.El
+.Ss LPC Device Settings
+The LPC bridge stores its configuration under a top-level
+.Va lpc
+node rather than under the PCI LPC device's node.
+The following nodes are available under
+.Va lpc :
+.Bl -column "pc-testdev" "Format" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va bootrom Ta path Ta Ta
+Path to a boot ROM.
+The contents of this file are copied into the guest's
+memory ending just before the 4GB physical address.
+If a boot ROM is present, a firmware interface device is
+also enabled for use by the boot ROM.
+.It Va com1 Ta node Ta Ta
+Settings for the COM1 serial port device.
+.It Va com2 Ta node Ta Ta
+Settings for the COM2 serial port device.
+.It Va com3 Ta node Ta Ta
+Settings for the COM3 serial port device.
+.It Va com4 Ta node Ta Ta
+Settings for the COM4 serial port device.
+.It Va pc-testdev Ta bool Ta false Ta
+Enable the PC debug/test device.
+.El
+.Ss NVMe Controller Settings
+Each NVMe controller supports a single storage device.
+The device can be backed either by a memory disk described by the
+.Va ram
+variable, or a block device using the the block device settings described above.
+In addition, each controller supports the following settings:
+.Bl -column "ioslots" "Format" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va maxq Ta integer Ta 16 Ta
+Maximum number of I/O submission and completion queue pairs.
+.It Va qsz Ta integer Ta 2058 Ta
+Number of elements in each I/O queue.
+.It Va ioslots Ta integer Ta 8 Ta
+Maximum number of concurrent I/O requests.
+.It Va sectsz Ta integer Ta Ta
+Sector size.
+Can be one of 512, 4096, or 8192.
+Devices backed by a memory disk use 4096 as the default.
+Devices backed by a block device use the block device's sector size
+as the default.
+.It Va ser Ta string Ta Ta
+Serial number of up to twenty characters.
+A default serial number is generated using a hash of the device's PCI address.
+.It Va eui64 Ta integer Ta Ta
+IEEE Extended Unique Identifier.
+If an EUI is not provided, a default is generated using a checksum of the
+device's PCI address.
+.It Va dsm Ta string Ta auto Ta
+Whether or not to advertise DataSet Management support.
+One of
+.Dq auto ,
+.Dq enable ,
+or
+.Dq disable .
+The
+.Dq auto
+setting only advertises support if the backing store supports
+resource freeing, for example via TRIM.
+.It Va ram Ta integer Ta Ta
+If set, allocate a memory disk as the backing store.
+The value of this variable is the size of the memory disk in megabytes.
+.El
+.Ss PCI Passthrough Settings
+.Bl -column "Name" "integer" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va bus Ta integer Ta Ta
+Host PCI bus address of device to pass through.
+.It Va slot Ta integer Ta Ta
+Host PCI slot address of device to pass through.
+.It Va func Ta integer Ta Ta
+Host PCI function address of device to pass through.
+.El
+.Ss VirtIO 9p Settings
+Each VirtIO 9p device exposes a single filesystem from a host path.
+.Bl -column "sharename" "Format" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va sharename Ta string Ta Ta
+The share name exposed to the guest.
+.It Va path Ta path Ta Ta
+The path of a directory on the host to export to the guest.
+.It Va ro Ta bool Ta false Ta
+If true, the guest filesystem is read-only.
+.El
+.Ss VirtIO Console Device Settings
+Each VirtIO Console device contains one or more console ports.
+Each port stores its settings in a node named
+.Dq port. Ns Va N
+under the controller's device node.
+The
+.Va N
+values are formatted as successive decimal values starting with 0.
+Each port supports the following settings:
+.Bl -column "Name" "Format" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va name Ta string Ta Ta
+The name of the port exposed to the guest.
+.It Va path Ta path Ta Ta
+The path of a UNIX domain socket providing the host connection for the port.
+.El
+.Ss VirtIO Network Interface Settings
+In addition to the network backend settings,
+VirtIO network interfaces support the following variables:
+.Bl -column "Name" "MAC address" "generated"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va mac Ta MAC address Ta generated Ta
+MAC address.
+If an explicit address is not provided,
+a MAC address is generated from a hash of the device's PCI address.
+.It Va mtu Ta integer Ta 1500 Ta
+The largest supported MTU advertised to the guest.
+.El
+.Ss VirtIO SCSI Settings
+.Bl -column "Name" "integer" "Default"
+.It Sy Name Ta Sy Format Ta Sy Default Ta Sy Description
+.It Va dev Ta path Ta Ta
+The path of a CAM target layer (CTL) device to export:
+.Pa /dev/cam/ctl Ns Oo Ar pp . Ns Ar vp Oc .
+.It Va iid Ta integer Ta 0 Ta
+Initiator ID to use when sending requests to the CTL port.
+.El
+.Sh SEE ALSO
+.Xr expand_number 3 ,
+.Xr getaddrinfo 3 ,
+.Xr strtol 3 ,
+.Xr netgraph 4 ,
+.Xr netmap 4 ,
+.Xr ng_socket 4 ,
+.Xr tap 4 ,
+.Xr vale 4 ,
+.Xr vmnet 4 ,
+.Xr bhyve 8
diff --git a/usr.sbin/bhyve/bhyverun.c b/usr.sbin/bhyve/bhyverun.c
index aafab4af8d8c..a3e6ef3c4724 100644
--- a/usr.sbin/bhyve/bhyverun.c
+++ b/usr.sbin/bhyve/bhyverun.c
@@ -87,6 +87,7 @@ __FBSDID("$FreeBSD$");
 #include "acpi.h"
 #include "atkbdc.h"
 #include "bootrom.h"
+#include "config.h"
 #include "inout.h"
 #include "debug.h"
 #include "fwctl.h"
@@ -183,26 +184,11 @@ static const char * const vmx_exit_reason_desc[] = {
 typedef int (*vmexit_handler_t)(struct vmctx *, struct vm_exit *, int *vcpu);
 extern int vmexit_task_switch(struct vmctx *, struct vm_exit *, int *vcpu);
 
-const char *vmname;
-
 int guest_ncpus;
 uint16_t cores, maxcpus, sockets, threads;
 
-char *guest_uuid_str;
-
 int raw_stdio = 0;
 
-static int gdb_port = 0;
-static int guest_vmexit_on_hlt, guest_vmexit_on_pause;
-static int virtio_msix = 1;
-static int x2apic_mode = 0;	/* default is xAPIC */
-static int destroy_on_poweroff = 0;
-
-static int strictio;
-static int strictmsr = 1;
-
-static int acpi;
-
 static char *progname;
 static const int BSP = 0;
 
@@ -238,8 +224,8 @@ usage(int code)
         fprintf(stderr,
 		"Usage: %s [-aehuwxACDHPSWY]\n"
 		"       %*s [-c [[cpus=]numcpus][,sockets=n][,cores=n][,threads=n]]\n"
-		"       %*s [-l <lpc>]\n"
-		"       %*s [-m mem] [-p vcpu:hostcpu] [-s <pci>] [-U uuid] <vm>\n"
+		"       %*s [-k <file>] [-l <lpc>] [-m mem] [-o <var>=<value>]\n"
+		"       %*s [-p vcpu:hostcpu] [-s <pci>] [-U uuid] [<vm>]\n"
 		"       -a: local apic is in xAPIC mode (deprecated)\n"
 		"       -A: create ACPI tables\n"
 		"       -c: number of cpus and/or topology specification\n"
@@ -248,13 +234,15 @@ usage(int code)
 		"       -e: exit on unhandled I/O access\n"
 		"       -h: help\n"
 		"       -H: vmexit from the guest on hlt\n"
+		"       -k: key=value flat config file\n"
 		"       -l: LPC device configuration\n"
 		"       -m: memory size in MB\n"
+		"       -o: set config 'var' to 'value'\n"
+		"       -p: pin 'vcpu' to 'hostcpu'\n"
+		"       -P: vmexit from the guest on pause\n"
 #ifdef BHYVE_SNAPSHOT
 		"       -r: path to checkpoint file\n"
 #endif
-		"       -p: pin 'vcpu' to 'hostcpu'\n"
-		"       -P: vmexit from the guest on pause\n"
 		"       -s: <slot,driver,configinfo> PCI slot config\n"
 		"       -S: guest memory cannot be swapped\n"
 		"       -u: RTC keeps UTC time\n"
@@ -271,11 +259,8 @@ usage(int code)
 
 /*
  * XXX This parser is known to have the following issues:
- * 1.  It accepts null key=value tokens ",,".
- * 2.  It accepts whitespace after = and before value.
- * 3.  Values out of range of INT are silently wrapped.
- * 4.  It doesn't check non-final values.
- * 5.  The apparently bogus limits of UINT16_MAX are for future expansion.
+ * 1.  It accepts null key=value tokens ",," as setting "cpus" to an
+ *     empty string.
  *
  * The acceptance of a null specification ('-c ""') is by design to match the
  * manual page syntax specification, this results in a topology of 1 vCPU.
@@ -283,74 +268,40 @@ usage(int code)
 static int
 topology_parse(const char *opt)
 {
-	uint64_t ncpus;
-	int c, chk, n, s, t, tmp;
 	char *cp, *str;
-	bool ns, scts;
 
-	c = 1, n = 1, s = 1, t = 1;
-	ns = false, scts = false;
+	if (*opt == '\0') {
+		set_config_value("sockets", "1");
+		set_config_value("cores", "1");
+		set_config_value("threads", "1");
+		set_config_value("cpus", "1");
+		return (0);
+	}
+
 	str = strdup(opt);
 	if (str == NULL)
-		goto out;
+		errx(4, "Failed to allocate memory");
 
 	while ((cp = strsep(&str, ",")) != NULL) {
-		if (sscanf(cp, "%i%n", &tmp, &chk) == 1) {
-			n = tmp;
-			ns = true;
-		} else if (sscanf(cp, "cpus=%i%n", &tmp, &chk) == 1) {
-			n = tmp;
-			ns = true;
-		} else if (sscanf(cp, "sockets=%i%n", &tmp, &chk) == 1) {
-			s = tmp;
-			scts = true;
-		} else if (sscanf(cp, "cores=%i%n", &tmp, &chk) == 1) {
-			c = tmp;
-			scts = true;
-		} else if (sscanf(cp, "threads=%i%n", &tmp, &chk) == 1) {
-			t = tmp;
-			scts = true;
+		if (strncmp(cp, "cpus=", strlen("cpus=")) == 0)
+			set_config_value("cpus", cp + strlen("cpus="));
+		else if (strncmp(cp, "sockets=", strlen("sockets=")) == 0)
+			set_config_value("sockets", cp + strlen("sockets="));
+		else if (strncmp(cp, "cores=", strlen("cores=")) == 0)
+			set_config_value("cores", cp + strlen("cores="));
+		else if (strncmp(cp, "threads=", strlen("threads=")) == 0)
+			set_config_value("threads", cp + strlen("threads="));
 #ifdef notyet  /* Do not expose this until vmm.ko implements it */
-		} else if (sscanf(cp, "maxcpus=%i%n", &tmp, &chk) == 1) {
-			m = tmp;
+		else if (strncmp(cp, "maxcpus=", strlen("maxcpus=")) == 0)
+			set_config_value("maxcpus", cp + strlen("maxcpus="));
 #endif
-		/* Skip the empty argument case from -c "" */
-		} else if (cp[0] == '\0')
-			continue;
-		else
-			goto out;
-		/* Any trailing garbage causes an error */
-		if (cp[chk] != '\0')
+		else if (strchr(cp, '=') != NULL)
 			goto out;
+		else
+			set_config_value("cpus", cp);
 	}
 	free(str);
-	str = NULL;
-
-	/*
-	 * Range check 1 <= n <= UINT16_MAX all values
-	 */
-	if (n < 1 || s < 1 || c < 1 || t < 1 ||
-	    n > UINT16_MAX || s > UINT16_MAX || c > UINT16_MAX  ||
-	    t > UINT16_MAX)
-		return (-1);
-
-	/* If only the cpus was specified, use that as sockets */
-	if (!scts)
-		s = n;
-	/*
-	 * Compute sockets * cores * threads avoiding overflow
-	 * The range check above insures these are 16 bit values
-	 * If n was specified check it against computed ncpus
-	 */
-	ncpus = (uint64_t)s * c * t;
-	if (ncpus > UINT16_MAX || (ns && n != ncpus))
-		return (-1);
-
-	guest_ncpus = ncpus;
-	sockets = s;
-	cores = c;
-	threads = t;
-	return(0);
+	return (0);
 
 out:
 	free(str);
@@ -358,8 +309,81 @@ out:
 }
 
 static int
+parse_int_value(const char *key, const char *value, int minval, int maxval)
+{
+	char *cp;
+	long lval;
+
+	errno = 0;
+	lval = strtol(value, &cp, 0);
+	if (errno != 0 || *cp != '\0' || cp == value || lval < minval ||
+	    lval > maxval)
+		errx(4, "Invalid value for %s: '%s'", key, value);
+	return (lval);
+}
+
+/*
+ * Set the sockets, cores, threads, and guest_cpus variables based on
+ * the configured topology.
+ *
+ * The limits of UINT16_MAX are due to the types passed to
+ * vm_set_topology().  vmm.ko may enforce tighter limits.
+ */
+static void
+calc_topolopgy(void)
+{
+	const char *value;
+	bool explicit_cpus;
+	uint64_t ncpus;
+
+	value = get_config_value("cpus");
+	if (value != NULL) {
+		guest_ncpus = parse_int_value("cpus", value, 1, UINT16_MAX);
+		explicit_cpus = true;
+	} else {
+		guest_ncpus = 1;
+		explicit_cpus = false;
+	}
+	value = get_config_value("cores");
+	if (value != NULL)
+		cores = parse_int_value("cores", value, 1, UINT16_MAX);
+	else
+		cores = 1;
+	value = get_config_value("threads");
+	if (value != NULL)
+		threads = parse_int_value("threads", value, 1, UINT16_MAX);
+	else
+		threads = 1;
+	value = get_config_value("sockets");
+	if (value != NULL)
+		sockets = parse_int_value("sockets", value, 1, UINT16_MAX);
+	else
+		sockets = guest_ncpus;
+
+	/*
+	 * Compute sockets * cores * threads avoiding overflow.  The
+	 * range check above insures these are 16 bit values.
+	 */
+	ncpus = (uint64_t)sockets * cores * threads;
+	if (ncpus > UINT16_MAX)
+		errx(4, "Computed number of vCPUs too high: %ju",
+		    (uintmax_t)ncpus);
+
+	if (explicit_cpus) {
+		if (guest_ncpus != ncpus)
+			errx(4, "Topology (%d sockets, %d cores, %d threads) "
+			    "does not match %d vCPUs", sockets, cores, threads,
+			    guest_ncpus);
+	} else
+		guest_ncpus = ncpus;
+}
+
+static int
 pincpu_parse(const char *opt)
 {
+	const char *value;
+	char *newval;
+	char key[16];
 	int vcpu, pcpu;
 
 	if (sscanf(opt, "%d:%d", &vcpu, &pcpu) != 2) {
@@ -379,17 +403,85 @@ pincpu_parse(const char *opt)
 		return (-1);
 	}
 
-	if (vcpumap[vcpu] == NULL) {
-		if ((vcpumap[vcpu] = malloc(sizeof(cpuset_t))) == NULL) {
-			perror("malloc");
-			return (-1);
-		}
-		CPU_ZERO(vcpumap[vcpu]);
+	snprintf(key, sizeof(key), "vcpu.%d.cpuset", vcpu);
+	value = get_config_value(key);
+
+	if (asprintf(&newval, "%s%s%d", value != NULL ? value : "",
+	    value != NULL ? "," : "", pcpu) == -1) {
+		perror("failed to build new cpuset string");
+		return (-1);
 	}
-	CPU_SET(pcpu, vcpumap[vcpu]);
+
+	set_config_value(key, newval);
+	free(newval);
 	return (0);
 }
 
+static void
+parse_cpuset(int vcpu, const char *list, cpuset_t *set)
+{
+	char *cp, *token;
+	int pcpu, start;
+
+	CPU_ZERO(set);
+	start = -1;
+	token = __DECONST(char *, list);
+	for (;;) {
+		pcpu = strtoul(token, &cp, 0);
+		if (cp == token)
+			errx(4, "invalid cpuset for vcpu %d: '%s'", vcpu, list);
+		if (pcpu < 0 || pcpu >= CPU_SETSIZE)
+			errx(4, "hostcpu '%d' outside valid range from 0 to %d",
+			    pcpu, CPU_SETSIZE - 1);
+		switch (*cp) {
+		case ',':
+		case '\0':
+			if (start >= 0) {
+				if (start > pcpu)
+					errx(4, "Invalid hostcpu range %d-%d",
+					    start, pcpu);
+				while (start < pcpu) {
+					CPU_SET(start, vcpumap[vcpu]);
+					start++;
+				}
+				start = -1;
+			}
+			CPU_SET(pcpu, vcpumap[vcpu]);
+			break;
+		case '-':
+			if (start >= 0)
+				errx(4, "invalid cpuset for vcpu %d: '%s'",
+				    vcpu, list);
+			start = pcpu;
+			break;
+		default:
+			errx(4, "invalid cpuset for vcpu %d: '%s'", vcpu, list);
+		}
+		if (*cp == '\0')
+			break;
+		token = cp + 1;
+	}
+}
+
+static void
+build_vcpumaps(void)
+{
+	char key[16];
+	const char *value;
+	int vcpu;
+
+	for (vcpu = 0; vcpu < guest_ncpus; vcpu++) {
+		snprintf(key, sizeof(key), "vcpu.%d.cpuset", vcpu);
+		value = get_config_value(key);
+		if (value == NULL)
+			continue;
+		vcpumap[vcpu] = malloc(sizeof(cpuset_t));
+		if (vcpumap[vcpu] == NULL)
+			err(4, "Failed to allocate cpuset for vcpu %d", vcpu);
+		parse_cpuset(vcpu, value, vcpumap[vcpu]);
+	}
+}
+
 void
 vm_inject_fault(void *arg, int vcpu, int vector, int errcode_valid,
     int errcode)
@@ -421,24 +513,10 @@ paddr_host2guest(struct vmctx *ctx, void *addr)
 #endif
 
 int
-fbsdrun_vmexit_on_pause(void)
-{
-
-	return (guest_vmexit_on_pause);
-}
-
-int
-fbsdrun_vmexit_on_hlt(void)
-{
-
-	return (guest_vmexit_on_hlt);
-}
-
-int
 fbsdrun_virtio_msix(void)
 {
 
-	return (virtio_msix);
+	return (get_config_bool_default("virtio_msix", true));
 }
 
 static void *
@@ -457,8 +535,7 @@ fbsdrun_start_thread(void *param)
 #ifdef BHYVE_SNAPSHOT
 	checkpoint_cpu_add(vcpu);
 #endif
-	if (gdb_port != 0)
-		gdb_cpu_add(vcpu);
+	gdb_cpu_add(vcpu);
 
 	vm_loop(mtp->mt_ctx, vcpu, vmexit[vcpu].rip);
 
@@ -546,7 +623,7 @@ vmexit_inout(struct vmctx *ctx, struct vm_exit *vme, int *pvcpu)
 		return (error);
 	}
 
-	error = emulate_inout(ctx, vcpu, vme, strictio);
+	error = emulate_inout(ctx, vcpu, vme);
 	if (error) {
 		fprintf(stderr, "Unhandled %s%c 0x%04x at 0x%lx\n",
 		    in ? "in" : "out",
@@ -570,7 +647,7 @@ vmexit_rdmsr(struct vmctx *ctx, struct vm_exit *vme, int *pvcpu)
 	if (error != 0) {
 		fprintf(stderr, "rdmsr to register %#x on vcpu %d\n",
 		    vme->u.msr.code, *pvcpu);
-		if (strictmsr) {
+		if (get_config_bool("x86.strictmsr")) {
 			vm_inject_gp(ctx, *pvcpu);
 			return (VMEXIT_CONTINUE);
 		}
@@ -596,7 +673,7 @@ vmexit_wrmsr(struct vmctx *ctx, struct vm_exit *vme, int *pvcpu)
 	if (error != 0) {
 		fprintf(stderr, "wrmsr to register %#x(%#lx) on vcpu %d\n",
 		    vme->u.msr.code, vme->u.msr.wval, *pvcpu);
-		if (strictmsr) {
+		if (get_config_bool("x86.strictmsr")) {
 			vm_inject_gp(ctx, *pvcpu);
 			return (VMEXIT_CONTINUE);
 		}
@@ -735,8 +812,7 @@ vmexit_mtrap(struct vmctx *ctx, struct vm_exit *vmexit, int *pvcpu)
 #ifdef BHYVE_SNAPSHOT
 	checkpoint_cpu_suspend(*pvcpu);
 #endif
-	if (gdb_port != 0)
-		gdb_cpu_mtrap(*pvcpu);
+	gdb_cpu_mtrap(*pvcpu);
 #ifdef BHYVE_SNAPSHOT
 	checkpoint_cpu_resume(*pvcpu);
 #endif
@@ -821,7 +897,7 @@ vmexit_suspend(struct vmctx *ctx, struct vm_exit *vmexit, int *pvcpu)
 	case VM_SUSPEND_RESET:
 		exit(0);
 	case VM_SUSPEND_POWEROFF:
-		if (destroy_on_poweroff)
+		if (get_config_bool_default("destroy_on_poweroff", false))
 			vm_destroy(ctx);
 		exit(1);
 	case VM_SUSPEND_HALT:
@@ -842,8 +918,7 @@ vmexit_debug(struct vmctx *ctx, struct vm_exit *vmexit, int *pvcpu)
 #ifdef BHYVE_SNAPSHOT
 	checkpoint_cpu_suspend(*pvcpu);
 #endif
-	if (gdb_port != 0)
-		gdb_cpu_suspend(*pvcpu);
+	gdb_cpu_suspend(*pvcpu);
 #ifdef BHYVE_SNAPSHOT
 	checkpoint_cpu_resume(*pvcpu);
 #endif
@@ -854,10 +929,6 @@ static int
 vmexit_breakpoint(struct vmctx *ctx, struct vm_exit *vmexit, int *pvcpu)
 {
 
-	if (gdb_port == 0) {
-		fprintf(stderr, "vm_loop: unexpected VMEXIT_DEBUG\n");
-		exit(4);
-	}
 	gdb_cpu_breakpoint(*pvcpu, vmexit);
 	return (VMEXIT_CONTINUE);
 }
@@ -947,7 +1018,7 @@ fbsdrun_set_capabilities(struct vmctx *ctx, int cpu)
 {
 	int err, tmp;
 
-	if (fbsdrun_vmexit_on_hlt()) {
+	if (get_config_bool_default("x86.vmexit_on_hlt", false)) {
 		err = vm_get_capability(ctx, cpu, VM_CAP_HALT_EXIT, &tmp);
 		if (err < 0) {
 			fprintf(stderr, "VM exit on HLT not supported\n");
@@ -958,7 +1029,7 @@ fbsdrun_set_capabilities(struct vmctx *ctx, int cpu)
 			handler[VM_EXITCODE_HLT] = vmexit_hlt;
 	}
 
-        if (fbsdrun_vmexit_on_pause()) {
+	if (get_config_bool_default("x86.vmexit_on_pause", false)) {
 		/*
 		 * pause exit support required for this mode
 		 */
@@ -973,7 +1044,7 @@ fbsdrun_set_capabilities(struct vmctx *ctx, int cpu)
 			handler[VM_EXITCODE_PAUSE] = vmexit_pause;
         }
 
-	if (x2apic_mode)
+	if (get_config_bool_default("x86.x2apic", false))
 		err = vm_set_x2apic_state(ctx, cpu, X2APIC_ENABLED);
 	else
 		err = vm_set_x2apic_state(ctx, cpu, X2APIC_DISABLED);
@@ -1077,16 +1148,68 @@ spinup_vcpu(struct vmctx *ctx, int vcpu)
 	fbsdrun_addcpu(ctx, BSP, vcpu, rip);
 }
 
+static bool
+parse_config_option(const char *option)
+{
+	const char *value;
+	char *path;
+
+	value = strchr(option, '=');
+	if (value == NULL || value[1] == '\0')
+		return (false);
+	path = strndup(option, value - option);
+	if (path == NULL)
+		err(4, "Failed to allocate memory");
+	set_config_value(path, value + 1);
+	return (true);
+}
+
+static void
+parse_simple_config_file(const char *path)
+{
+	FILE *fp;
+	char *line, *cp;
+	size_t linecap;
+	unsigned int lineno;
+
+	fp = fopen(path, "r");
+	if (fp == NULL)
+		err(4, "Failed to open configuration file %s", path);
+	line = NULL;
+	linecap = 0;
+	lineno = 1;
+	for (lineno = 1; getline(&line, &linecap, fp) > 0; lineno++) {
+		if (*line == '#' || *line == '\n')
+			continue;
+		cp = strchr(line, '\n');
+		if (cp != NULL)
+			*cp = '\0';
+		if (!parse_config_option(line))
+			errx(4, "%s line %u: invalid config option '%s'", path,
+			    lineno, line);
+	}
+	free(line);
+	fclose(fp);
+}
+
+static void
+set_defaults(void)
+{
+
+	set_config_bool("acpi_tables", false);
+	set_config_value("memory.size", "256M");
+	set_config_bool("x86.strictmsr", true);
+}
+
 int
 main(int argc, char *argv[])
 {
 	int c, error, err;
-	int max_vcpus, mptgen, memflags;
-	int rtc_localtime;
-	bool gdb_stop;
+	int max_vcpus, memflags;
 	struct vmctx *ctx;
 	uint64_t rip;
 	size_t memsize;
+	const char *value, *vmname;
 	char *optstr;
 #ifdef BHYVE_SNAPSHOT
 	char *restore_file;
@@ -1096,31 +1219,25 @@ main(int argc, char *argv[])
 	restore_file = NULL;
 #endif
 
+	init_config();
+	set_defaults();
 	progname = basename(argv[0]);
-	gdb_stop = false;
-	guest_ncpus = 1;
-	sockets = cores = threads = 1;
-	maxcpus = 0;
-	memsize = 256 * MB;
-	mptgen = 1;
-	rtc_localtime = 1;
-	memflags = 0;
 
 #ifdef BHYVE_SNAPSHOT
-	optstr = "aehuwxACDHIPSWYp:G:c:s:m:l:U:r:";
+	optstr = "aehuwxACDHIPSWYk:o:p:G:c:s:m:l:U:r:";
 #else
-	optstr = "aehuwxACDHIPSWYp:G:c:s:m:l:U:";
+	optstr = "aehuwxACDHIPSWYk:o:p:G:c:s:m:l:U:";
 #endif
 	while ((c = getopt(argc, argv, optstr)) != -1) {
 		switch (c) {
 		case 'a':
-			x2apic_mode = 0;
+			set_config_bool("x86.x2apic", false);
 			break;
 		case 'A':
-			acpi = 1;
+			set_config_bool("acpi_tables", true);
 			break;
 		case 'D':
-			destroy_on_poweroff = 1;
+			set_config_bool("destroy_on_poweroff", true);
 			break;
 		case 'p':
                         if (pincpu_parse(optarg) != 0) {
@@ -1135,14 +1252,17 @@ main(int argc, char *argv[])
 			}
 			break;
 		case 'C':
-			memflags |= VM_MEM_F_INCORE;
+			set_config_bool("memory.guest_in_core", true);
 			break;
 		case 'G':
 			if (optarg[0] == 'w') {
-				gdb_stop = true;
+				set_config_bool("gdb.wait", true);
 				optarg++;
 			}
-			gdb_port = atoi(optarg);
+			set_config_value("gdb.port", optarg);
+			break;
+		case 'k':
+			parse_simple_config_file(optarg);
 			break;
 		case 'l':
 			if (strncmp(optarg, "help", strlen(optarg)) == 0) {
@@ -1167,15 +1287,17 @@ main(int argc, char *argv[])
 			else
 				break;
 		case 'S':
-			memflags |= VM_MEM_F_WIRED;
+			set_config_bool("memory.wired", true);
 			break;
                 case 'm':
-			error = vm_parse_memsize(optarg, &memsize);
-			if (error)
-				errx(EX_USAGE, "invalid memsize '%s'", optarg);
+			set_config_value("memory.size", optarg);
+			break;
+		case 'o':
+			if (!parse_config_option(optarg))
+				errx(EX_USAGE, "invalid configuration option '%s'", optarg);
 			break;
 		case 'H':
-			guest_vmexit_on_hlt = 1;
+			set_config_bool("x86.vmexit_on_hlt", true);
 			break;
 		case 'I':
 			/*
@@ -1187,28 +1309,28 @@ main(int argc, char *argv[])
 			 */
 			break;
 		case 'P':
-			guest_vmexit_on_pause = 1;
+			set_config_bool("x86.vmexit_on_pause", true);
 			break;
 		case 'e':
-			strictio = 1;
+			set_config_bool("x86.strictio", true);
 			break;
 		case 'u':
-			rtc_localtime = 0;
+			set_config_bool("rtc.use_localtime", false);
 			break;
 		case 'U':
-			guest_uuid_str = optarg;
+			set_config_value("uuid", optarg);
 			break;
 		case 'w':
-			strictmsr = 0;
+			set_config_bool("x86.strictmsr", false);
 			break;
 		case 'W':
-			virtio_msix = 0;
+			set_config_bool("virtio_msix", false);
 			break;
 		case 'x':
-			x2apic_mode = 1;
+			set_config_bool("x86.x2apic", true);
 			break;
 		case 'Y':
-			mptgen = 0;
+			set_config_bool("x86.mptable", false);
 			break;
 		case 'h':
 			usage(0);			
@@ -1219,10 +1341,10 @@ main(int argc, char *argv[])
 	argc -= optind;
 	argv += optind;
 
-#ifdef BHYVE_SNAPSHOT
-	if (argc > 1 || (argc == 0 && restore_file == NULL))
+	if (argc > 1)
 		usage(1);
 
+#ifdef BHYVE_SNAPSHOT
 	if (restore_file != NULL) {
 		error = load_restore_file(restore_file, &rstate);
 		if (error) {
@@ -1230,24 +1352,32 @@ main(int argc, char *argv[])
 					"file: '%s'.\n", restore_file);
 			exit(1);
 		}
-	}
-
-	if (argc == 1) {
-		vmname = argv[0];
-	} else {
 		vmname = lookup_vmname(&rstate);
-		if (vmname == NULL) {
-			fprintf(stderr, "Cannot find VM name in restore file. "
-					"Please specify one.\n");
-			exit(1);
-		}
+		if (vmname != NULL)
+			set_config_value("name", vmname);
 	}
-#else
-	if (argc != 1)
+#endif
+
+	if (argc == 1)
+		set_config_value("name", argv[0]);
+
+	vmname = get_config_value("name");
+	if (vmname == NULL)
 		usage(1);
 
-	vmname = argv[0];
-#endif
+	if (get_config_bool_default("config.dump", false)) {
+		dump_config();
+		exit(1);
+	}
+
+	calc_topolopgy();
+	build_vcpumaps();
+
+	value = get_config_value("memory.size");
+	error = vm_parse_memsize(value, &memsize);
+	if (error)
+		errx(EX_USAGE, "invalid memsize '%s'", value);
+
 	ctx = do_open(vmname);
 
 #ifdef BHYVE_SNAPSHOT
@@ -1272,6 +1402,11 @@ main(int argc, char *argv[])
 
 	fbsdrun_set_capabilities(ctx, BSP);
 
+	memflags = 0;
+	if (get_config_bool_default("memory.wired", false))
+		memflags |= VM_MEM_F_WIRED;
+	if (get_config_bool_default("memory.guest_in_core", false))
+		memflags |= VM_MEM_F_INCORE;
 	vm_set_memflags(ctx, memflags);
 	err = vm_setup_memory(ctx, memsize, VM_MMAP_ALL);
 	if (err) {
@@ -1293,7 +1428,7 @@ main(int argc, char *argv[])
 	pci_irq_init(ctx);
 	ioapic_init(ctx);
 
-	rtc_init(ctx, rtc_localtime);
+	rtc_init(ctx);
 	sci_init(ctx);
 
 	/*
@@ -1308,11 +1443,13 @@ main(int argc, char *argv[])
 	 * Initialize after PCI, to allow a bootrom file to reserve the high
 	 * region.
 	 */
-	if (acpi)
+	if (get_config_bool("acpi_tables"))
 		vmgenc_init(ctx);
 
-	if (gdb_port != 0)
-		init_gdb(ctx, gdb_port, gdb_stop);
+	value = get_config_value("gdb.port");
+	if (value != NULL)
+		init_gdb(ctx, atoi(value), get_config_bool_default("gdb.wait",
+		    false));
 
 	if (lpc_bootrom()) {
 		if (vm_set_capability(ctx, BSP, VM_CAP_UNRESTRICTED_GUEST, 1)) {
@@ -1364,7 +1501,7 @@ main(int argc, char *argv[])
 	/*
 	 * build the guest tables, MP etc.
 	 */
-	if (mptgen) {
+	if (get_config_bool_default("x86.mptable", true)) {
 		error = mptable_build(ctx, guest_ncpus);
 		if (error) {
 			perror("error to build the guest tables");
@@ -1375,7 +1512,7 @@ main(int argc, char *argv[])
 	error = smbios_build(ctx);
 	assert(error == 0);
 
-	if (acpi) {
+	if (get_config_bool("acpi_tables")) {
 		error = acpi_build(ctx, guest_ncpus);
 		assert(error == 0);
 	}
diff --git a/usr.sbin/bhyve/bhyverun.h b/usr.sbin/bhyve/bhyverun.h
index 0177baca14e9..9b0c9fb533e2 100644
--- a/usr.sbin/bhyve/bhyverun.h
+++ b/usr.sbin/bhyve/bhyverun.h
@@ -37,8 +37,6 @@
 struct vmctx;
 extern int guest_ncpus;
 extern uint16_t cores, sockets, threads;
-extern char *guest_uuid_str;
-extern const char *vmname;
 
 void *paddr_guest2host(struct vmctx *ctx, uintptr_t addr, size_t len);
 #ifdef BHYVE_SNAPSHOT
@@ -47,9 +45,6 @@ uintptr_t paddr_host2guest(struct vmctx *ctx, void *addr);
 
 void fbsdrun_set_capabilities(struct vmctx *ctx, int cpu);
 void fbsdrun_addcpu(struct vmctx *ctx, int fromcpu, int newcpu, uint64_t rip);
-int  fbsdrun_muxed(void);
-int  fbsdrun_vmexit_on_hlt(void);
-int  fbsdrun_vmexit_on_pause(void);
-int  fbsdrun_disable_x2apic(void);
 int  fbsdrun_virtio_msix(void);
+
 #endif
diff --git a/usr.sbin/bhyve/block_if.c b/usr.sbin/bhyve/block_if.c
index 4c91038ca765..ae09bc71d622 100644
--- a/usr.sbin/bhyve/block_if.c
+++ b/usr.sbin/bhyve/block_if.c
@@ -61,8 +61,10 @@ __FBSDID("$FreeBSD$");
 #include <machine/vmm_snapshot.h>
 
 #include "bhyverun.h"
+#include "config.h"
 #include "debug.h"
 #include "mevent.h"
+#include "pci_emul.h"
 #include "block_if.h"
 
 #define BLOCKIF_SIG	0xb109b109
@@ -427,18 +429,38 @@ blockif_init(void)
 	(void) signal(SIGCONT, SIG_IGN);
 }
 
+int
+blockif_legacy_config(nvlist_t *nvl, const char *opts)
+{
+	char *cp, *path;
+
+	if (opts == NULL)
+		return (0);
+
+	cp = strchr(opts, ',');
+	if (cp == NULL) {
+		set_config_value_node(nvl, "path", opts);
+		return (0);
+	}
+	path = strndup(opts, cp - opts);
+	set_config_value_node(nvl, "path", path);
+	free(path);
+	return (pci_parse_legacy_config(nvl, cp + 1));
+}
+
 struct blockif_ctxt *
-blockif_open(const char *optstr, const char *ident)
+blockif_open(nvlist_t *nvl, const char *ident)
 {
 	char tname[MAXCOMLEN + 1];
 	char name[MAXPATHLEN];
-	char *nopt, *xopts, *cp;
+	const char *path, *pssval, *ssval;
+	char *cp;
 	struct blockif_ctxt *bc;
 	struct stat sbuf;
 	struct diocgattr_arg arg;
 	off_t size, psectsz, psectoff;
 	int extra, fd, i, sectsz;
-	int nocache, sync, ro, candelete, geom, ssopt, pssopt;
+	int ro, candelete, geom, ssopt, pssopt;
 	int nodelete;
 
 #ifndef WITHOUT_CAPSICUM
@@ -449,59 +471,62 @@ blockif_open(const char *optstr, const char *ident)
 	pthread_once(&blockif_once, blockif_init);
 
 	fd = -1;
+	extra = 0;
 	ssopt = 0;
-	nocache = 0;
-	sync = 0;
 	ro = 0;
 	nodelete = 0;
 
-	/*
-	 * The first element in the optstring is always a pathname.
-	 * Optional elements follow
-	 */
-	nopt = xopts = strdup(optstr);
-	while (xopts != NULL) {
-		cp = strsep(&xopts, ",");
-		if (cp == nopt)		/* file or device pathname */
-			continue;
-		else if (!strcmp(cp, "nocache"))
-			nocache = 1;
-		else if (!strcmp(cp, "nodelete"))
-			nodelete = 1;
-		else if (!strcmp(cp, "sync") || !strcmp(cp, "direct"))
-			sync = 1;
-		else if (!strcmp(cp, "ro"))
-			ro = 1;
-		else if (sscanf(cp, "sectorsize=%d/%d", &ssopt, &pssopt) == 2)
-			;
-		else if (sscanf(cp, "sectorsize=%d", &ssopt) == 1)
+	if (get_config_bool_node_default(nvl, "nocache", false))
+		extra |= O_DIRECT;
+	if (get_config_bool_node_default(nvl, "nodelete", false))
+		nodelete = 1;
+	if (get_config_bool_node_default(nvl, "sync", false) ||
+	    get_config_bool_node_default(nvl, "direct", false))
+		extra |= O_SYNC;
+	if (get_config_bool_node_default(nvl, "ro", false))
+		ro = 1;
+	ssval = get_config_value_node(nvl, "sectorsize");
+	if (ssval != NULL) {
+		ssopt = strtol(ssval, &cp, 10);
+		if (cp == ssval) {
+			EPRINTLN("Invalid sector size \"%s\"", ssval);
+			goto err;
+		}
+		if (*cp == '\0') {
 			pssopt = ssopt;
-		else {
-			EPRINTLN("Invalid device option \"%s\"", cp);
+		} else if (*cp == '/') {
+			pssval = cp + 1;
+			pssopt = strtol(pssval, &cp, 10);
+			if (cp == pssval || *cp != '\0') {
+				EPRINTLN("Invalid sector size \"%s\"", ssval);
+				goto err;
+			}
+		} else {
+			EPRINTLN("Invalid sector size \"%s\"", ssval);
 			goto err;
 		}
 	}
 
-	extra = 0;
-	if (nocache)
-		extra |= O_DIRECT;
-	if (sync)
-		extra |= O_SYNC;
+	path = get_config_value_node(nvl, "path");
+	if (path == NULL) {
+		EPRINTLN("Missing \"path\" for block device.");
+		goto err;
+	}
 
-	fd = open(nopt, (ro ? O_RDONLY : O_RDWR) | extra);
+	fd = open(path, (ro ? O_RDONLY : O_RDWR) | extra);
 	if (fd < 0 && !ro) {
 		/* Attempt a r/w fail with a r/o open */
-		fd = open(nopt, O_RDONLY | extra);
+		fd = open(path, O_RDONLY | extra);
 		ro = 1;
 	}
 
 	if (fd < 0) {
-		warn("Could not open backing file: %s", nopt);
+		warn("Could not open backing file: %s", path);
 		goto err;
 	}
 
         if (fstat(fd, &sbuf) < 0) {
-		warn("Could not stat backing file %s", nopt);
+		warn("Could not stat backing file %s", path);
 		goto err;
         }
 
@@ -615,7 +640,6 @@ blockif_open(const char *optstr, const char *ident)
 err:
 	if (fd >= 0)
 		close(fd);
-	free(nopt);
 	return (NULL);
 }
 
diff --git a/usr.sbin/bhyve/block_if.h b/usr.sbin/bhyve/block_if.h
index f3b5b6938ef1..87ae4d169969 100644
--- a/usr.sbin/bhyve/block_if.h
+++ b/usr.sbin/bhyve/block_if.h
@@ -38,6 +38,7 @@
 #ifndef _BLOCK_IF_H_
 #define _BLOCK_IF_H_
 
+#include <sys/nv.h>
 #include <sys/uio.h>
 #include <sys/unistd.h>
 
@@ -62,7 +63,8 @@ struct blockif_req {
 };
 
 struct blockif_ctxt;
-struct blockif_ctxt *blockif_open(const char *optstr, const char *ident);
+int	blockif_legacy_config(nvlist_t *nvl, const char *opts);
+struct blockif_ctxt *blockif_open(nvlist_t *nvl, const char *ident);
 off_t	blockif_size(struct blockif_ctxt *bc);
 void	blockif_chs(struct blockif_ctxt *bc, uint16_t *c, uint8_t *h,
     uint8_t *s);
diff --git a/usr.sbin/bhyve/config.c b/usr.sbin/bhyve/config.c
new file mode 100644
index 000000000000..1df750e9576c
--- /dev/null
+++ b/usr.sbin/bhyve/config.c
@@ -0,0 +1,431 @@
+/*-
+ * SPDX-License-Identifier: BSD-2-Clause-FreeBSD
+ *
+ * Copyright (c) 2021 John H. Baldwin <jhb@FreeBSD.org>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <sys/cdefs.h>
+__FBSDID("$FreeBSD$");
+
+#include <assert.h>
+#include <err.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "config.h"
+
+static nvlist_t *config_root;
+
+void
+init_config(void)
+{
+
+	config_root = nvlist_create(0);
+	if (config_root == NULL)
+		err(4, "Failed to create configuration root nvlist");
+}
+
+static nvlist_t *
+_lookup_config_node(nvlist_t *parent, const char *path, bool create)
+{
+	char *copy, *name, *tofree;
+	nvlist_t *nvl, *new_nvl;
+
+	copy = strdup(path);
+	if (copy == NULL)
+		errx(4, "Failed to allocate memory");
+	tofree = copy;
+	nvl = parent;
+	while ((name = strsep(&copy, ".")) != NULL) {
+		if (*name == '\0') {
+			warnx("Invalid configuration node: %s", path);
+			nvl = NULL;
+			break;
+		}
+		if (nvlist_exists_nvlist(nvl, name))
+			nvl = (nvlist_t *)nvlist_get_nvlist(nvl, name);
+		else if (nvlist_exists(nvl, name)) {
+			for (copy = tofree; copy < name; copy++)
+				if (*copy == '\0')
+					*copy = '.';
+			warnx(
+		    "Configuration node %s is a child of existing variable %s",
+			    path, tofree);
+			nvl = NULL;
+			break;
+		} else if (create) {
+			new_nvl = nvlist_create(0);
+			if (new_nvl == NULL)
+				errx(4, "Failed to allocate memory");
+			nvlist_move_nvlist(nvl, name, new_nvl);
+			nvl = new_nvl;
+		} else {
+			nvl = NULL;
+			break;
+		}
+	}
+	free(tofree);
+	return (nvl);
+}
+
+nvlist_t *
+create_config_node(const char *path)
+{
+
+	return (_lookup_config_node(config_root, path, true));
+}
+
+nvlist_t *
+find_config_node(const char *path)
+{
+
+	return (_lookup_config_node(config_root, path, false));
+}
+
+nvlist_t *
+create_relative_config_node(nvlist_t *parent, const char *path)
+{
+
+	return (_lookup_config_node(parent, path, true));
+}
+
+nvlist_t *
+find_relative_config_node(nvlist_t *parent, const char *path)
+{
+
+	return (_lookup_config_node(parent, path, false));
+}
+
+void
+set_config_value_node(nvlist_t *parent, const char *name, const char *value)
+{
+
+	if (strchr(name, '.') != NULL)
+		errx(4, "Invalid config node name %s", name);
+	if (parent == NULL)
+		parent = config_root;
+	if (nvlist_exists_string(parent, name))
+		nvlist_free_string(parent, name);
+	else if (nvlist_exists(parent, name))
+		errx(4,
+		    "Attemping to add value %s to existing node %s of list %p",
+		    value, name, parent);
+	nvlist_add_string(parent, name, value);
+}
+
+void
+set_config_value(const char *path, const char *value)
+{
+	const char *name;
+	char *node_name;
+	nvlist_t *nvl;
+
+	/* Look for last separator. */
+	name = strrchr(path, '.');
+	if (name == NULL) {
+		nvl = config_root;
+		name = path;
+	} else {
+		node_name = strndup(path, name - path);
+		if (node_name == NULL)
+			errx(4, "Failed to allocate memory");
+		nvl = create_config_node(node_name);
+		if (nvl == NULL)
+			errx(4, "Failed to create configuration node %s",
+			    node_name);
+		free(node_name);
+
+		/* Skip over '.'. */
+		name++;
+	}
+
+	if (nvlist_exists_nvlist(nvl, name))
+		errx(4, "Attempting to add value %s to existing node %s",
+		    value, path);
+	set_config_value_node(nvl, name, value);
+}
+
+static const char *
+get_raw_config_value(const char *path)
+{
+	const char *name;
+	char *node_name;
+	nvlist_t *nvl;
+
+	/* Look for last separator. */
+	name = strrchr(path, '.');
+	if (name == NULL) {
+		nvl = config_root;
+		name = path;
+	} else {
+		node_name = strndup(path, name - path);
+		if (node_name == NULL)
+			errx(4, "Failed to allocate memory");
+		nvl = find_config_node(node_name);
+		free(node_name);
+		if (nvl == NULL)
+			return (NULL);
+
+		/* Skip over '.'. */
+		name++;
+	}
+
+	if (nvlist_exists_string(nvl, name))
+		return (nvlist_get_string(nvl, name));
+	if (nvlist_exists_nvlist(nvl, name))
+		warnx("Attempting to fetch value of node %s", path);
+	return (NULL);
+}
+
+static char *
+_expand_config_value(const char *value, int depth)
+{
+	FILE *valfp;
+	const char *cp, *vp;
+	char *nestedval, *path, *valbuf;
+	size_t valsize;
+
+	valfp = open_memstream(&valbuf, &valsize);
+	if (valfp == NULL)
+		errx(4, "Failed to allocate memory");
+
+	vp = value;
+	while (*vp != '\0') {
+		switch (*vp) {
+		case '%':
+			if (depth > 15) {
+				warnx(
+		    "Too many recursive references in configuration value");
+				fputc('%', valfp);
+				vp++;
+				break;
+			}				
+			if (vp[1] != '(' || vp[2] == '\0')
+				cp = NULL;
+			else
+				cp = strchr(vp + 2, ')');
+			if (cp == NULL) {
+				warnx(
+			    "Invalid reference in configuration value \"%s\"",
+				    value);
+				fputc('%', valfp);
+				vp++;
+				break;
+			}
+			vp += 2;
+
+			if (cp == vp) {
+				warnx(
+			    "Empty reference in configuration value \"%s\"",
+				    value);
+				vp++;
+				break;
+			}
+
+			/* Allocate a C string holding the path. */
+			path = strndup(vp, cp - vp);
+			if (path == NULL)
+				errx(4, "Failed to allocate memory");
+
+			/* Advance 'vp' past the reference. */
+			vp = cp + 1;
+
+			/* Fetch the referenced value. */
+			cp = get_raw_config_value(path);
+			if (cp == NULL)
+				warnx(
+		    "Failed to fetch referenced configuration variable %s",
+				    path);
+			else {
+				nestedval = _expand_config_value(cp, depth + 1);
+				fputs(nestedval, valfp);
+				free(nestedval);
+			}
+			free(path);
+			break;
+		case '\\':
+			vp++;
+			if (*vp == '\0') {
+				warnx(
+			    "Trailing \\ in configuration value \"%s\"",
+				    value);
+				break;
+			}
+			/* FALLTHROUGH */
+		default:
+			fputc(*vp, valfp);
+			vp++;
+			break;
+		}
+	}
+	fclose(valfp);
+	return (valbuf);
+}
+
+const char *
+expand_config_value(const char *value)
+{
+	static char *valbuf;
+
+	if (strchr(value, '%') == NULL)
+		return (value);
+
+	free(valbuf);
+	valbuf = _expand_config_value(value, 0);
+	return (valbuf);
+}
+
+const char *
+get_config_value(const char *path)
+{
+	const char *value;
+
+	value = get_raw_config_value(path);
+	if (value == NULL)
+		return (NULL);
+	return (expand_config_value(value));
+}
+
+const char *
+get_config_value_node(const nvlist_t *parent, const char *name)
+{
+
+	if (strchr(name, '.') != NULL)
+		errx(4, "Invalid config node name %s", name);
+	if (parent == NULL)
+		parent = config_root;
+	if (nvlist_exists_nvlist(parent, name))
+		warnx("Attempt to fetch value of node %s of list %p", name,
+		    parent);
+	if (!nvlist_exists_string(parent, name))
+		return (NULL);
+
+	return (expand_config_value(nvlist_get_string(parent, name)));
+}
+
+bool
+_bool_value(const char *name, const char *value)
+{
+
+	if (strcasecmp(value, "true") == 0 ||
+	    strcasecmp(value, "on") == 0 ||
+	    strcasecmp(value, "yes") == 0 ||
+	    strcmp(value, "1") == 0)
+		return (true);
+	if (strcasecmp(value, "false") == 0 ||
+	    strcasecmp(value, "off") == 0 ||
+	    strcasecmp(value, "no") == 0 ||
+	    strcmp(value, "0") == 0)
+		return (false);
+	err(4, "Invalid value %s for boolean variable %s", value, name);
+}
+
+bool
+get_config_bool(const char *path)
+{
+	const char *value;
+
+	value = get_config_value(path);
+	if (value == NULL)
+		err(4, "Failed to fetch boolean variable %s", path);
+	return (_bool_value(path, value));
+}
+
+bool
+get_config_bool_default(const char *path, bool def)
+{
+	const char *value;
+
+	value = get_config_value(path);
+	if (value == NULL)
+		return (def);
+	return (_bool_value(path, value));
+}
+
+bool
+get_config_bool_node(const nvlist_t *parent, const char *name)
+{
+	const char *value;
+
+	value = get_config_value_node(parent, name);
+	if (value == NULL)
+		err(4, "Failed to fetch boolean variable %s", name);
+	return (_bool_value(name, value));
+}
+
+bool
+get_config_bool_node_default(const nvlist_t *parent, const char *name,
+    bool def)
+{
+	const char *value;
+
+	value = get_config_value_node(parent, name);
+	if (value == NULL)
+		return (def);
+	return (_bool_value(name, value));
+}
+
+void
+set_config_bool(const char *path, bool value)
+{
+
+	set_config_value(path, value ? "true" : "false");
+}
+
+void
+set_config_bool_node(nvlist_t *parent, const char *name, bool value)
+{
+
+	set_config_value_node(parent, name, value ? "true" : "false");
+}
+
+static void
+dump_tree(const char *prefix, const nvlist_t *nvl)
+{
+	const char *name;
+	void *cookie;
+	int type;
+
+	cookie = NULL;
+	while ((name = nvlist_next(nvl, &type, &cookie)) != NULL) {
+		if (type == NV_TYPE_NVLIST) {
+			char *new_prefix;
+
+			asprintf(&new_prefix, "%s%s.", prefix, name);
+			dump_tree(new_prefix, nvlist_get_nvlist(nvl, name));
+			free(new_prefix);
+		} else {
+			assert(type == NV_TYPE_STRING);
+			printf("%s%s=%s\n", prefix, name,
+			    nvlist_get_string(nvl, name));
+		}
+	}
+}
+
+void
+dump_config(void)
+{
+	dump_tree("", config_root);
+}
diff --git a/usr.sbin/bhyve/config.h b/usr.sbin/bhyve/config.h
new file mode 100644
index 000000000000..9f82afb267b2
--- /dev/null
+++ b/usr.sbin/bhyve/config.h
@@ -0,0 +1,119 @@
+/*-
+ * SPDX-License-Identifier: BSD-2-Clause-FreeBSD
+ *
+ * Copyright (c) 2021 John H. Baldwin <jhb@FreeBSD.org>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * $FreeBSD$
+ */
+
+#ifndef __CONFIG_H__
+#define	__CONFIG_H__
+
+#include <sys/nv.h>
+
+/*-
+ * Manages a configuration database backed by an nv(9) list.
+ *
+ * The database only stores string values.  Callers should parse
+ * values into other types if needed.  String values can reference
+ * other configuration variables using a '%(name)' syntax.  In this
+ * case, the name must be the the full path of the configuration
+ * variable.  The % character can be escaped with a preceding \ to
+ * avoid expansion.  Any \ characters must be escaped.
+ *
+ * Configuration variables are stored in a tree.  The full path of a
+ * variable is specified as a dot-separated name similar to sysctl(8)
+ * OIDs.
+ */ 
+
+/*
+ * Fetches the value of a configuration variable.  If the "raw" value
+ * contains references to other configuration variables, this function
+ * expands those references and returns a pointer to the parsed
+ * string.  The string's storage is only stable until the next call to
+ * this function.
+ *
+ * If no node is found, returns NULL.
+ *
+ * If 'parent' is NULL, 'name' is assumed to be a top-level variable.
+ */
+const char *get_config_value_node(const nvlist_t *parent, const char *name);
+
+/*
+ * Similar to get_config_value_node but expects a full path to the
+ * leaf node.
+ */
+const char *get_config_value(const char *path);
+
+/* Initializes the tree to an empty state. */
+void	init_config(void);
+
+/*
+ * Creates an existing configuration node via a dot-separated OID
+ * path.  Will fail if the path names an existing leaf configuration
+ * variable.  If the node already exists, this returns a pointer to
+ * the existing node.
+ */
+nvlist_t *create_config_node(const char *path);
+
+/*
+ * Looks for an existing configuration node via a dot-separated OID
+ * path.  Will fail if the path names an existing leaf configuration
+ * variable.
+ */
+nvlist_t *find_config_node(const char *path);
+
+/*
+ * Similar to the above, but treats the path relative to an existing
+ * 'parent' node rather than as an absolute path.
+ */
+nvlist_t *create_relative_config_node(nvlist_t *parent, const char *path);
+nvlist_t *find_relative_config_node(nvlist_t *parent, const char *path);
+
+/*
+ * Adds or replaces the value of the specified variable.
+ *
+ * If 'parent' is NULL, 'name' is assumed to be a top-level variable.
+ */
+void	set_config_value_node(nvlist_t *parent, const char *name,
+    const char *value);
+
+/*
+ * Similar to set_config_value_node but expects a full path to the
+ * leaf node.
+ */
+void	set_config_value(const char *path, const char *value);
+
+/* Convenience wrappers for boolean variables. */
+bool	get_config_bool(const char *path);
+bool	get_config_bool_node(const nvlist_t *parent, const char *name);
+bool	get_config_bool_default(const char *path, bool def);
+bool	get_config_bool_node_default(const nvlist_t *parent, const char *name,
+    bool def);
+void	set_config_bool(const char *path, bool value);
+void	set_config_bool_node(nvlist_t *parent, const char *name, bool value);
+
+void	dump_config(void);
+
+#endif /* !__CONFIG_H__ */
diff --git a/usr.sbin/bhyve/gdb.c b/usr.sbin/bhyve/gdb.c
index b0f0c034b2c7..c4c37ab80bc7 100644
--- a/usr.sbin/bhyve/gdb.c
+++ b/usr.sbin/bhyve/gdb.c
@@ -131,6 +131,7 @@ static int cur_fd = -1;
 static TAILQ_HEAD(, breakpoint) breakpoints;
 static struct vcpu_state *vcpu_state;
 static int cur_vcpu, stopped_vcpu;
+static bool gdb_active = false;
 
 const int gdb_regset[] = {
 	VM_REG_GUEST_RAX,
@@ -730,6 +731,8 @@ static void
 _gdb_cpu_suspend(int vcpu, bool report_stop)
 {
 
+	if (!gdb_active)
+		return;
 	debug("$vCPU %d suspending\n", vcpu);
 	CPU_SET(vcpu, &vcpus_waiting);
 	if (report_stop && CPU_CMP(&vcpus_waiting, &vcpus_suspended) == 0)
@@ -748,6 +751,8 @@ void
 gdb_cpu_add(int vcpu)
 {
 
+	if (!gdb_active)
+		return;
 	debug("$vCPU %d starting\n", vcpu);
 	pthread_mutex_lock(&gdb_lock);
 	assert(vcpu < guest_ncpus);
@@ -829,6 +834,8 @@ gdb_cpu_mtrap(int vcpu)
 {
 	struct vcpu_state *vs;
 
+	if (!gdb_active)
+		return;
 	debug("$vCPU %d MTRAP\n", vcpu);
 	pthread_mutex_lock(&gdb_lock);
 	vs = &vcpu_state[vcpu];
@@ -869,6 +876,10 @@ gdb_cpu_breakpoint(int vcpu, struct vm_exit *vmexit)
 	uint64_t gpa;
 	int error;
 
+	if (!gdb_active) {
+		fprintf(stderr, "vm_loop: unexpected VMEXIT_DEBUG\n");
+		exit(4);
+	}
 	pthread_mutex_lock(&gdb_lock);
 	error = guest_vaddr2paddr(vcpu, vmexit->rip, &gpa);
 	assert(error == 1);
@@ -1859,4 +1870,5 @@ init_gdb(struct vmctx *_ctx, int sport, bool wait)
 	limit_gdb_socket(s);
 #endif
 	mevent_add(s, EVF_READ, new_connection, NULL);
+	gdb_active = true;
 }
diff --git a/usr.sbin/bhyve/hda_codec.c b/usr.sbin/bhyve/hda_codec.c
index 41e8121ae278..7a6ba345d8ec 100644
--- a/usr.sbin/bhyve/hda_codec.c
+++ b/usr.sbin/bhyve/hda_codec.c
@@ -205,7 +205,7 @@ struct hda_codec_softc {
  * HDA Codec module function declarations
  */
 static int hda_codec_init(struct hda_codec_inst *hci, const char *play,
-    const char *rec, const char *opts);
+    const char *rec);
 static int hda_codec_reset(struct hda_codec_inst *hci);
 static int hda_codec_command(struct hda_codec_inst *hci, uint32_t cmd_data);
 static int hda_codec_notify(struct hda_codec_inst *hci, uint8_t run,
@@ -391,7 +391,7 @@ verb_func_t hda_codec_verb_handlers[HDA_CODEC_NODES_COUNT] = {
 
 static int
 hda_codec_init(struct hda_codec_inst *hci, const char *play,
-    const char *rec, const char *opts)
+    const char *rec)
 {
 	struct hda_codec_softc *sc = NULL;
 	struct hda_codec_stream *st = NULL;
@@ -400,8 +400,6 @@ hda_codec_init(struct hda_codec_inst *hci, const char *play,
 	if (!(play || rec))
 		return (-1);
 
-	DPRINTF("cad: 0x%x opts: %s", hci->cad, opts);
-
 	sc = calloc(1, sizeof(*sc));
 	if (!sc)
 		return (-1);
diff --git a/usr.sbin/bhyve/inout.c b/usr.sbin/bhyve/inout.c
index b460ee2988cc..1fba90945afb 100644
--- a/usr.sbin/bhyve/inout.c
+++ b/usr.sbin/bhyve/inout.c
@@ -48,6 +48,7 @@ __FBSDID("$FreeBSD$");
 #include <assert.h>
 
 #include "bhyverun.h"
+#include "config.h"
 #include "inout.h"
 
 SET_DECLARE(inout_port_set, struct inout_port);
@@ -103,7 +104,7 @@ register_default_iohandler(int start, int size)
 }
 
 int
-emulate_inout(struct vmctx *ctx, int vcpu, struct vm_exit *vmexit, int strict)
+emulate_inout(struct vmctx *ctx, int vcpu, struct vm_exit *vmexit)
 {
 	int addrsize, bytes, flags, in, port, prot, rep;
 	uint32_t eax, val;
@@ -124,7 +125,8 @@ emulate_inout(struct vmctx *ctx, int vcpu, struct vm_exit *vmexit, int strict)
 
 	handler = inout_handlers[port].handler;
 
-	if (strict && handler == default_inout)
+	if (handler == default_inout &&
+	    get_config_bool_default("x86.strictio", false))
 		return (-1);
 
 	flags = inout_handlers[port].flags;
diff --git a/usr.sbin/bhyve/inout.h b/usr.sbin/bhyve/inout.h
index 45ad5f4ac654..afbae34c2dd0 100644
--- a/usr.sbin/bhyve/inout.h
+++ b/usr.sbin/bhyve/inout.h
@@ -72,8 +72,7 @@ struct inout_port {
 	DATA_SET(inout_port_set, __CONCAT(__inout_port, __LINE__))
 	
 void	init_inout(void);
-int	emulate_inout(struct vmctx *, int vcpu, struct vm_exit *vmexit,
-		      int strict);
+int	emulate_inout(struct vmctx *, int vcpu, struct vm_exit *vmexit);
 int	register_inout(struct inout_port *iop);
 int	unregister_inout(struct inout_port *iop);
 
diff --git a/usr.sbin/bhyve/mevent.c b/usr.sbin/bhyve/mevent.c
index 649a6b09cb34..2e9059837011 100644
--- a/usr.sbin/bhyve/mevent.c
+++ b/usr.sbin/bhyve/mevent.c
@@ -63,8 +63,6 @@ __FBSDID("$FreeBSD$");
 
 #define	MEVENT_MAX	64
 
-extern const char *vmname;
-
 static pthread_t mevent_tid;
 static int mevent_timid = 43;
 static int mevent_pipefd[2];
diff --git a/usr.sbin/bhyve/mevent_test.c b/usr.sbin/bhyve/mevent_test.c
index 1a9c168da6de..3a7aac98a65c 100644
--- a/usr.sbin/bhyve/mevent_test.c
+++ b/usr.sbin/bhyve/mevent_test.c
@@ -56,8 +56,6 @@ static pthread_cond_t accept_condvar = PTHREAD_COND_INITIALIZER;
 
 static struct mevent *tevp;
 
-char *vmname = "test vm";
-
 
 #define MEVENT_ECHO
 
diff --git a/usr.sbin/bhyve/net_backends.c b/usr.sbin/bhyve/net_backends.c
index 884ffb824111..30c26aea458b 100644
--- a/usr.sbin/bhyve/net_backends.c
+++ b/usr.sbin/bhyve/net_backends.c
@@ -75,10 +75,12 @@ __FBSDID("$FreeBSD$");
 #include <netgraph.h>
 #endif
 
+#include "config.h"
 #include "debug.h"
 #include "iov.h"
 #include "mevent.h"
 #include "net_backends.h"
+#include "pci_emul.h"
 
 #include <sys/linker_set.h>
 
@@ -96,7 +98,7 @@ struct net_backend {
 	 * and should not be called by the frontend.
 	 */
 	int (*init)(struct net_backend *be, const char *devname,
-	    const char *opts, net_be_rxeof_t cb, void *param);
+	    nvlist_t *nvl, net_be_rxeof_t cb, void *param);
 	void (*cleanup)(struct net_backend *be);
 
 	/*
@@ -204,7 +206,7 @@ tap_cleanup(struct net_backend *be)
 
 static int
 tap_init(struct net_backend *be, const char *devname,
-	 const char *opts, net_be_rxeof_t cb, void *param)
+	 nvlist_t *nvl, net_be_rxeof_t cb, void *param)
 {
 	struct tap_priv *priv = (struct tap_priv *)be->opaque;
 	char tbuf[80];
@@ -398,18 +400,14 @@ DATA_SET(net_backend_set, vmnet_backend);
 
 static int
 ng_init(struct net_backend *be, const char *devname,
-	 const char *opts, net_be_rxeof_t cb, void *param)
+	 nvlist_t *nvl, net_be_rxeof_t cb, void *param)
 {
 	struct tap_priv *p = (struct tap_priv *)be->opaque;
 	struct ngm_connect ngc;
-	char *ngopts, *tofree;
-	char nodename[NG_NODESIZ];
+	const char *value, *nodename;
 	int sbsz;
 	int ctrl_sock;
 	int flags;
-	int path_provided;
-	int peerhook_provided;
-	int socket_provided;
 	unsigned long maxsbsz;
 	size_t msbsz;
 #ifndef WITHOUT_CAPSICUM
@@ -425,56 +423,27 @@ ng_init(struct net_backend *be, const char *devname,
 
 	memset(&ngc, 0, sizeof(ngc));
 
-	strncpy(ngc.ourhook, "vmlink", NG_HOOKSIZ - 1);
-
-	tofree = ngopts = strdup(opts);
-
-	if (ngopts == NULL) {
-		WPRINTF(("strdup error"));
-		return (-1);
-	}
-
-	socket_provided = 0;
-	path_provided = 0;
-	peerhook_provided = 0;
-
-	while (ngopts != NULL) {
-		char *value = ngopts;
-		char *key;
-
-		key = strsep(&value, "=");
-		if (value == NULL)
-			break;
-		ngopts = value;
-		(void) strsep(&ngopts, ",");
-
-		if (strcmp(key, "socket") == 0) {
-			strncpy(nodename, value, NG_NODESIZ - 1);
-			socket_provided = 1;
-		} else if (strcmp(key, "path") == 0) {
-			strncpy(ngc.path, value, NG_PATHSIZ - 1);
-			path_provided = 1;
-		} else if (strcmp(key, "hook") == 0) {
-			strncpy(ngc.ourhook, value, NG_HOOKSIZ - 1);
-		} else if (strcmp(key, "peerhook") == 0) {
-			strncpy(ngc.peerhook, value, NG_HOOKSIZ - 1);
-			peerhook_provided = 1;
-		}
-	}
-
-	free(tofree);
-
-	if (!path_provided) {
+	value = get_config_value_node(nvl, "path");
+	if (value == NULL) {
 		WPRINTF(("path must be provided"));
 		return (-1);
 	}
+	strncpy(ngc.path, value, NG_PATHSIZ - 1);
 
-	if (!peerhook_provided) {
+	value = get_config_value_node(nvl, "hook");
+	if (value == NULL)
+		value = "vmlink";
+	strncpy(ngc.ourhook, value, NG_HOOKSIZ - 1);
+
+	value = get_config_value_node(nvl, "peerhook");
+	if (value == NULL) {
 		WPRINTF(("peer hook must be provided"));
 		return (-1);
 	}
+	strncpy(ngc.peerhook, value, NG_HOOKSIZ - 1);
 
-	if (NgMkSockNode(socket_provided ? nodename : NULL,
+	nodename = get_config_value_node(nvl, "socket");
+	if (NgMkSockNode(nodename,
 		&ctrl_sock, &be->fd) < 0) {
 		WPRINTF(("can't get Netgraph sockets"));
 		return (-1);
@@ -664,7 +633,7 @@ netmap_set_cap(struct net_backend *be, uint64_t features,
 
 static int
 netmap_init(struct net_backend *be, const char *devname,
-	    const char *opts, net_be_rxeof_t cb, void *param)
+	    nvlist_t *nvl, net_be_rxeof_t cb, void *param)
 {
 	struct netmap_priv *priv = (struct netmap_priv *)be->opaque;
 
@@ -924,10 +893,29 @@ static struct net_backend vale_backend = {
 DATA_SET(net_backend_set, netmap_backend);
 DATA_SET(net_backend_set, vale_backend);
 
+int
+netbe_legacy_config(nvlist_t *nvl, const char *opts)
+{
+	char *backend, *cp;
+
+	if (opts == NULL)
+		return (0);
+
+	cp = strchr(opts, ',');
+	if (cp == NULL) {
+		set_config_value_node(nvl, "backend", opts);
+		return (0);
+	}
+	backend = strndup(opts, cp - opts);
+	set_config_value_node(nvl, "backend", backend);
+	free(backend);
+	return (pci_parse_legacy_config(nvl, cp + 1));
+}
+
 /*
  * Initialize a backend and attach to the frontend.
  * This is called during frontend initialization.
- *  @pbe is a pointer to the backend to be initialized
+ *  @ret is a pointer to the backend to be initialized
  *  @devname is the backend-name as supplied on the command line,
  * 	e.g. -s 2:0,frontend-name,backend-name[,other-args]
  *  @cb is the receive callback supplied by the frontend,
@@ -937,21 +925,19 @@ DATA_SET(net_backend_set, vale_backend);
  *	the argument for the callback.
  */
 int
-netbe_init(struct net_backend **ret, const char *opts, net_be_rxeof_t cb,
+netbe_init(struct net_backend **ret, nvlist_t *nvl, net_be_rxeof_t cb,
     void *param)
 {
 	struct net_backend **pbe, *nbe, *tbe = NULL;
+	const char *value;
 	char *devname;
-	char *options;
 	int err;
 
-	devname = options = strdup(opts);
-
-	if (devname == NULL) {
+	value = get_config_value_node(nvl, "backend");
+	if (value == NULL) {
 		return (-1);
 	}
-
-	devname = strsep(&options, ",");
+	devname = strdup(value);
 
 	/*
 	 * Find the network backend that matches the user-provided
@@ -985,7 +971,7 @@ netbe_init(struct net_backend **ret, const char *opts, net_be_rxeof_t cb,
 	nbe->fe_vnet_hdr_len = 0;
 
 	/* Initialize the backend. */
-	err = nbe->init(nbe, devname, options, cb, param);
+	err = nbe->init(nbe, devname, nvl, cb, param);
 	if (err) {
 		free(devname);
 		free(nbe);
diff --git a/usr.sbin/bhyve/net_backends.h b/usr.sbin/bhyve/net_backends.h
index b55437fc7bbc..bc7834546bc1 100644
--- a/usr.sbin/bhyve/net_backends.h
+++ b/usr.sbin/bhyve/net_backends.h
@@ -37,8 +37,9 @@ typedef struct net_backend net_backend_t;
 
 /* Interface between network frontends and the network backends. */
 typedef void (*net_be_rxeof_t)(int, enum ev_type, void *param);
-int	netbe_init(net_backend_t **be, const char *opts, net_be_rxeof_t cb,
+int	netbe_init(net_backend_t **be, nvlist_t *nvl, net_be_rxeof_t cb,
             void *param);
+int	netbe_legacy_config(nvlist_t *nvl, const char *opts);
 void	netbe_cleanup(net_backend_t *be);
 uint64_t netbe_get_cap(net_backend_t *be);
 int	 netbe_set_cap(net_backend_t *be, uint64_t cap,
diff --git a/usr.sbin/bhyve/net_utils.c b/usr.sbin/bhyve/net_utils.c
index d602cac3eb0a..8088fa89a50f 100644
--- a/usr.sbin/bhyve/net_utils.c
+++ b/usr.sbin/bhyve/net_utils.c
@@ -40,11 +40,12 @@ __FBSDID("$FreeBSD$");
 #include <string.h>
 
 #include "bhyverun.h"
+#include "config.h"
 #include "debug.h"
 #include "net_utils.h"
 
 int
-net_parsemac(char *mac_str, uint8_t *mac_addr)
+net_parsemac(const char *mac_str, uint8_t *mac_addr)
 {
         struct ether_addr *ea;
         char zero_addr[ETHER_ADDR_LEN] = { 0, 0, 0, 0, 0, 0 };
@@ -107,7 +108,7 @@ net_genmac(struct pci_devinst *pi, uint8_t *macaddr)
 	char nstr[80];
 
 	snprintf(nstr, sizeof(nstr), "%d-%d-%s", pi->pi_slot,
-	    pi->pi_func, vmname);
+	    pi->pi_func, get_config_value("name"));
 
 	MD5Init(&mdctx);
 	MD5Update(&mdctx, nstr, (unsigned int)strlen(nstr));
diff --git a/usr.sbin/bhyve/net_utils.h b/usr.sbin/bhyve/net_utils.h
index 1ca20ddb74ab..3dd44572588c 100644
--- a/usr.sbin/bhyve/net_utils.h
+++ b/usr.sbin/bhyve/net_utils.h
@@ -34,7 +34,7 @@
 #include "pci_emul.h"
 
 void	net_genmac(struct pci_devinst *pi, uint8_t *macaddr);
-int	net_parsemac(char *mac_str, uint8_t *mac_addr);
+int	net_parsemac(const char *mac_str, uint8_t *mac_addr);
 int	net_parsemtu(const char *mtu_str, unsigned long *mtu);
 
 #endif /* _NET_UTILS_H_ */
diff --git a/usr.sbin/bhyve/pci_ahci.c b/usr.sbin/bhyve/pci_ahci.c
index 64c1fe0b90ca..38ec87c34eed 100644
--- a/usr.sbin/bhyve/pci_ahci.c
+++ b/usr.sbin/bhyve/pci_ahci.c
@@ -58,6 +58,8 @@ __FBSDID("$FreeBSD$");
 #include <md5.h>
 
 #include "bhyverun.h"
+#include "config.h"
+#include "debug.h"
 #include "pci_emul.h"
 #include "ahci.h"
 #include "block_if.h"
@@ -2313,20 +2315,115 @@ pci_ahci_read(struct vmctx *ctx, int vcpu, struct pci_devinst *pi, int baridx,
 	return (value);
 }
 
+/*
+ * Each AHCI controller has a "port" node which contains nodes for
+ * each port named after the decimal number of the port (no leading
+ * zeroes).  Port nodes contain a "type" ("hd" or "cd"), as well as
+ * options for blockif.  For example:
+ *
+ * pci.0.1.0
+ *          .device="ahci"
+ *          .port
+ *               .0
+ *                 .type="hd"
+ *                 .path="/path/to/image"
+ */
+static int
+pci_ahci_legacy_config_port(nvlist_t *nvl, int port, const char *type,
+    const char *opts)
+{
+	char node_name[sizeof("XX")];
+	nvlist_t *port_nvl;
+
+	snprintf(node_name, sizeof(node_name), "%d", port);
+	port_nvl = create_relative_config_node(nvl, node_name);
+	set_config_value_node(port_nvl, "type", type);
+	return (blockif_legacy_config(port_nvl, opts));
+}
+
 static int
-pci_ahci_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts, int atapi)
+pci_ahci_legacy_config(nvlist_t *nvl, const char *opts)
+{
+	nvlist_t *ports_nvl;
+	const char *type;
+	char *next, *next2, *str, *tofree;
+	int p, ret;
+
+	if (opts == NULL)
+		return (0);
+
+	ports_nvl = create_relative_config_node(nvl, "port");
+	ret = 1;
+	tofree = str = strdup(opts);
+	for (p = 0; p < MAX_PORTS && str != NULL; p++, str = next) {
+		/* Identify and cut off type of present port. */
+		if (strncmp(str, "hd:", 3) == 0) {
+			type = "hd";
+			str += 3;
+		} else if (strncmp(str, "cd:", 3) == 0) {
+			type = "cd";
+			str += 3;
+		} else
+			type = NULL;
+
+		/* Find and cut off the next port options. */
+		next = strstr(str, ",hd:");
+		next2 = strstr(str, ",cd:");
+		if (next == NULL || (next2 != NULL && next2 < next))
+			next = next2;
+		if (next != NULL) {
+			next[0] = 0;
+			next++;
+		}
+
+		if (str[0] == 0)
+			continue;
+
+		if (type == NULL) {
+			EPRINTLN("Missing or invalid type for port %d: \"%s\"",
+			    p, str);
+			goto out;
+		}
+
+		if (pci_ahci_legacy_config_port(ports_nvl, p, type, str) != 0)
+			goto out;
+	}
+	ret = 0;
+out:
+	free(tofree);
+	return (ret);
+}
+
+static int
+pci_ahci_cd_legacy_config(nvlist_t *nvl, const char *opts)
+{
+	nvlist_t *ports_nvl;
+
+	ports_nvl = create_relative_config_node(nvl, "port");
+	return (pci_ahci_legacy_config_port(ports_nvl, 0, "cd", opts));
+}
+
+static int
+pci_ahci_hd_legacy_config(nvlist_t *nvl, const char *opts)
+{
+	nvlist_t *ports_nvl;
+
+	ports_nvl = create_relative_config_node(nvl, "port");
+	return (pci_ahci_legacy_config_port(ports_nvl, 0, "hd", opts));
+}
+
+static int
+pci_ahci_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	char bident[sizeof("XX:XX:XX")];
+	char node_name[sizeof("XX")];
 	struct blockif_ctxt *bctxt;
 	struct pci_ahci_softc *sc;
-	int ret, slots, p;
+	int atapi, ret, slots, p;
 	MD5_CTX mdctx;
 	u_char digest[16];
-	char *next, *next2;
-	char *bopt, *uopt, *xopts, *config;
-	FILE* fp;
-	size_t block_len;
-	int comma, optpos;
+	const char *path, *type, *value;
+	nvlist_t *ports_nvl, *port_nvl;
 
 	ret = 0;
 
@@ -2342,98 +2439,24 @@ pci_ahci_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts, int atapi)
 	sc->pi = 0;
 	slots = 32;
 
-	for (p = 0; p < MAX_PORTS && opts != NULL; p++, opts = next) {
+	ports_nvl = find_relative_config_node(nvl, "port");
+	for (p = 0; p < MAX_PORTS; p++) {
 		struct ata_params *ata_ident = &sc->port[p].ata_ident;
-		memset(ata_ident, 0, sizeof(struct ata_params));
-
-		/* Identify and cut off type of present port. */
-		if (strncmp(opts, "hd:", 3) == 0) {
-			atapi = 0;
-			opts += 3;
-		} else if (strncmp(opts, "cd:", 3) == 0) {
-			atapi = 1;
-			opts += 3;
-		}
+		char ident[AHCI_PORT_IDENT];
 
-		/* Find and cut off the next port options. */
-		next = strstr(opts, ",hd:");
-		next2 = strstr(opts, ",cd:");
-		if (next == NULL || (next2 != NULL && next2 < next))
-			next = next2;
-		if (next != NULL) {
-			next[0] = 0;
-			next++;
-		}
-
-		if (opts[0] == 0)
+		snprintf(node_name, sizeof(node_name), "%d", p);
+		port_nvl = find_relative_config_node(ports_nvl, node_name);
+		if (port_nvl == NULL)
 			continue;
 
-		uopt = strdup(opts);
-		bopt = NULL;
-		fp = open_memstream(&bopt, &block_len);
-		comma = 0;
-		optpos = 0;
-
-		for (xopts = strtok(uopt, ",");
-		     xopts != NULL;
-		     xopts = strtok(NULL, ",")) {
-
-			/* First option assume as block filename. */
-			if (optpos == 0) {
-				/*
-				 * Create an identifier for the backing file.
-				 * Use parts of the md5 sum of the filename
-				 */
-				char ident[AHCI_PORT_IDENT];
-				MD5Init(&mdctx);
-				MD5Update(&mdctx, opts, strlen(opts));
-				MD5Final(digest, &mdctx);
-				snprintf(ident, AHCI_PORT_IDENT,
-					"BHYVE-%02X%02X-%02X%02X-%02X%02X",
-					digest[0], digest[1], digest[2], digest[3], digest[4],
-					digest[5]);
-				ata_string((uint8_t*)&ata_ident->serial, ident, 20);
-				ata_string((uint8_t*)&ata_ident->revision, "001", 8);
-				if (atapi) {
-					ata_string((uint8_t*)&ata_ident->model, "BHYVE SATA DVD ROM", 40);
-				}
-				else {
-					ata_string((uint8_t*)&ata_ident->model, "BHYVE SATA DISK", 40);
-				}
-			}
-
-			if ((config = strchr(xopts, '=')) != NULL) {
-				*config++ = '\0';
-				if (!strcmp("nmrr", xopts)) {
-					ata_ident->media_rotation_rate = atoi(config);
-				}
-				else if (!strcmp("ser", xopts)) {
-					ata_string((uint8_t*)(&ata_ident->serial), config, 20);
-				}
-				else if (!strcmp("rev", xopts)) {
-					ata_string((uint8_t*)(&ata_ident->revision), config, 8);
-				}
-				else if (!strcmp("model", xopts)) {
-					ata_string((uint8_t*)(&ata_ident->model), config, 40);
-				}
-				else {
-					/* Pass all other options to blockif_open. */
-					*--config = '=';
-					fprintf(fp, "%s%s", comma ? "," : "", xopts);
-					comma = 1;
-				}
-			}
-			else {
-				/* Pass all other options to blockif_open. */
-				fprintf(fp, "%s%s", comma ? "," : "", xopts);
-				comma = 1;
-			}
-			optpos++;
-		}
-		free(uopt);
-		fclose(fp);
+		type = get_config_value_node(port_nvl, "type");
+		if (type == NULL)
+			continue;
 
-		DPRINTF("%s\n", bopt);
+		if (strcmp(type, "hd") == 0)
+			atapi = 0;
+		else
+			atapi = 1;
 
 		/*
 		 * Attempt to open the backing image. Use the PCI slot/func
@@ -2441,9 +2464,8 @@ pci_ahci_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts, int atapi)
 		 */
 		snprintf(bident, sizeof(bident), "%d:%d:%d", pi->pi_slot,
 		    pi->pi_func, p);
-		bctxt = blockif_open(bopt, bident);
-		free(bopt);
 
+		bctxt = blockif_open(port_nvl, bident);
 		if (bctxt == NULL) {
 			sc->ports = p;
 			ret = 1;
@@ -2454,6 +2476,38 @@ pci_ahci_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts, int atapi)
 		sc->port[p].port = p;
 		sc->port[p].atapi = atapi;
 
+		/*
+		 * Create an identifier for the backing file.
+		 * Use parts of the md5 sum of the filename
+		 */
+		path = get_config_value_node(port_nvl, "path");
+		MD5Init(&mdctx);
+		MD5Update(&mdctx, path, strlen(path));
+		MD5Final(digest, &mdctx);
+		snprintf(ident, AHCI_PORT_IDENT,
+			"BHYVE-%02X%02X-%02X%02X-%02X%02X",
+			digest[0], digest[1], digest[2], digest[3], digest[4],
+			digest[5]);
+
+		memset(ata_ident, 0, sizeof(struct ata_params));
+		ata_string((uint8_t*)&ata_ident->serial, ident, 20);
+		ata_string((uint8_t*)&ata_ident->revision, "001", 8);
+		if (atapi)
+			ata_string((uint8_t*)&ata_ident->model, "BHYVE SATA DVD ROM", 40);
+		else
+			ata_string((uint8_t*)&ata_ident->model, "BHYVE SATA DISK", 40);
+		value = get_config_value_node(port_nvl, "nmrr");
+		if (value != NULL)
+			ata_ident->media_rotation_rate = atoi(value);
+		value = get_config_value_node(port_nvl, "ser");
+		if (value != NULL)
+			ata_string((uint8_t*)(&ata_ident->serial), value, 20);
+		value = get_config_value_node(port_nvl, "rev");
+		if (value != NULL)
+			ata_string((uint8_t*)(&ata_ident->revision), value, 8);
+		value = get_config_value_node(port_nvl, "model");
+		if (value != NULL)
+			ata_string((uint8_t*)(&ata_ident->model), value, 40);
 		ata_identify_init(&sc->port[p], atapi);
 
 		/*
@@ -2507,20 +2561,6 @@ open_fail:
 	return (ret);
 }
 
-static int
-pci_ahci_hd_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
-{
-
-	return (pci_ahci_init(ctx, pi, opts, 0));
-}
-
-static int
-pci_ahci_atapi_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
-{
-
-	return (pci_ahci_init(ctx, pi, opts, 1));
-}
-
 #ifdef BHYVE_SNAPSHOT
 static int
 pci_ahci_snapshot_save_queues(struct ahci_port *port,
@@ -2802,7 +2842,8 @@ pci_ahci_resume(struct vmctx *ctx, struct pci_devinst *pi)
  */
 struct pci_devemu pci_de_ahci = {
 	.pe_emu =	"ahci",
-	.pe_init =	pci_ahci_hd_init,
+	.pe_init =	pci_ahci_init,
+	.pe_legacy_config = pci_ahci_legacy_config,
 	.pe_barwrite =	pci_ahci_write,
 	.pe_barread =	pci_ahci_read,
 #ifdef BHYVE_SNAPSHOT
@@ -2815,26 +2856,14 @@ PCI_EMUL_SET(pci_de_ahci);
 
 struct pci_devemu pci_de_ahci_hd = {
 	.pe_emu =	"ahci-hd",
-	.pe_init =	pci_ahci_hd_init,
-	.pe_barwrite =	pci_ahci_write,
-	.pe_barread =	pci_ahci_read,
-#ifdef BHYVE_SNAPSHOT
-	.pe_snapshot =	pci_ahci_snapshot,
-	.pe_pause =	pci_ahci_pause,
-	.pe_resume =	pci_ahci_resume,
-#endif
+	.pe_legacy_config = pci_ahci_hd_legacy_config,
+	.pe_alias =	"ahci",
 };
 PCI_EMUL_SET(pci_de_ahci_hd);
 
 struct pci_devemu pci_de_ahci_cd = {
 	.pe_emu =	"ahci-cd",
-	.pe_init =	pci_ahci_atapi_init,
-	.pe_barwrite =	pci_ahci_write,
-	.pe_barread =	pci_ahci_read,
-#ifdef BHYVE_SNAPSHOT
-	.pe_snapshot =	pci_ahci_snapshot,
-	.pe_pause =	pci_ahci_pause,
-	.pe_resume =	pci_ahci_resume,
-#endif
+	.pe_legacy_config = pci_ahci_cd_legacy_config,
+	.pe_alias =	"ahci",
 };
 PCI_EMUL_SET(pci_de_ahci_cd);
diff --git a/usr.sbin/bhyve/pci_e82545.c b/usr.sbin/bhyve/pci_e82545.c
index 2d09c024f258..d0b4fb9e466f 100644
--- a/usr.sbin/bhyve/pci_e82545.c
+++ b/usr.sbin/bhyve/pci_e82545.c
@@ -65,6 +65,7 @@ __FBSDID("$FreeBSD$");
 #include "mii.h"
 
 #include "bhyverun.h"
+#include "config.h"
 #include "debug.h"
 #include "pci_emul.h"
 #include "mevent.h"
@@ -2277,15 +2278,12 @@ e82545_reset(struct e82545_softc *sc, int drvr)
 }
 
 static int
-e82545_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+e82545_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	char nstr[80];
 	struct e82545_softc *sc;
-	char *optscopy;
-	char *vtopts;
-	int mac_provided;
-
-	DPRINTF("Loading with options: %s", opts);
+	const char *mac;
+	int err;
 
 	/* Setup our softc */
 	sc = calloc(1, sizeof(*sc));
@@ -2323,56 +2321,20 @@ e82545_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 	pci_emul_alloc_bar(pi, E82545_BAR_IO, PCIBAR_IO,
 		E82545_BAR_IO_LEN);
 
-	/*
-	 * Attempt to open the net backend and read the MAC address
-	 * if specified.  Copied from virtio-net, slightly modified.
-	 */
-	mac_provided = 0;
-	sc->esc_be = NULL;
-	if (opts != NULL) {
-		int err = 0;
-
-		optscopy = vtopts = strdup(opts);
-		(void) strsep(&vtopts, ",");
-
-		/*
-		 * Parse the list of options in the form
-		 *     key1=value1,...,keyN=valueN.
-		 */
-		while (vtopts != NULL) {
-			char *value = vtopts;
-			char *key;
-
-			key = strsep(&value, "=");
-			if (value == NULL)
-				break;
-			vtopts = value;
-			(void) strsep(&vtopts, ",");
-
-			if (strcmp(key, "mac") == 0) {
-				err = net_parsemac(value, sc->esc_mac.octet);
-				if (err)
-					break;
-				mac_provided = 1;
-			}
-		}
-
-		free(optscopy);
-
+	mac = get_config_value_node(nvl, "mac");
+	if (mac != NULL) {
+		err = net_parsemac(mac, sc->esc_mac.octet);
 		if (err) {
 			free(sc);
 			return (err);
 		}
-
-		err = netbe_init(&sc->esc_be, opts, e82545_rx_callback, sc);
-		if (err) {
-			free(sc);
-			return (err);
-		}
-	}
-
-	if (!mac_provided) {
+	} else
 		net_genmac(pi, sc->esc_mac.octet);
+
+	err = netbe_init(&sc->esc_be, nvl, e82545_rx_callback, sc);
+	if (err) {
+		free(sc);
+		return (err);
 	}
 
 	netbe_rx_enable(sc->esc_be);
@@ -2540,6 +2502,7 @@ done:
 struct pci_devemu pci_de_e82545 = {
 	.pe_emu = 	"e1000",
 	.pe_init =	e82545_init,
+	.pe_legacy_config = netbe_legacy_config,
 	.pe_barwrite =	e82545_write,
 	.pe_barread =	e82545_read,
 #ifdef BHYVE_SNAPSHOT
diff --git a/usr.sbin/bhyve/pci_emul.c b/usr.sbin/bhyve/pci_emul.c
index 803ab0be38bb..e498b14ef700 100644
--- a/usr.sbin/bhyve/pci_emul.c
+++ b/usr.sbin/bhyve/pci_emul.c
@@ -55,6 +55,7 @@ __FBSDID("$FreeBSD$");
 
 #include "acpi.h"
 #include "bhyverun.h"
+#include "config.h"
 #include "debug.h"
 #include "inout.h"
 #include "ioapic.h"
@@ -73,8 +74,8 @@ __FBSDID("$FreeBSD$");
 #define	MAXFUNCS	(PCI_FUNCMAX + 1)
 
 struct funcinfo {
-	char	*fi_name;
-	char	*fi_param;
+	nvlist_t *fi_config;
+	struct pci_devemu *fi_pde;
 	struct pci_devinst *fi_devi;
 };
 
@@ -114,7 +115,7 @@ SYSRES_MEM(PCI_EMUL_ECFG_BASE, PCI_EMUL_ECFG_SIZE);
 
 #define	PCI_EMUL_MEMLIMIT32	PCI_EMUL_ECFG_BASE
 
-static struct pci_devemu *pci_emul_finddev(char *name);
+static struct pci_devemu *pci_emul_finddev(const char *name);
 static void pci_lintr_route(struct pci_devinst *pi);
 static void pci_lintr_update(struct pci_devinst *pi);
 static void pci_cfgrw(struct vmctx *ctx, int vcpu, int in, int bus, int slot,
@@ -170,13 +171,56 @@ pci_parse_slot_usage(char *aopt)
 	EPRINTLN("Invalid PCI slot info field \"%s\"", aopt);
 }
 
+/*
+ * Helper function to parse a list of comma-separated options where
+ * each option is formatted as "name[=value]".  If no value is
+ * provided, the option is treated as a boolean and is given a value
+ * of true.
+ */
+int
+pci_parse_legacy_config(nvlist_t *nvl, const char *opt)
+{
+	char *config, *name, *tofree, *value;
+
+	if (opt == NULL)
+		return (0);
+
+	config = tofree = strdup(opt);
+	while ((name = strsep(&config, ",")) != NULL) {
+		value = strchr(name, '=');
+		if (value != NULL) {
+			*value = '\0';
+			value++;
+			set_config_value_node(nvl, name, value);
+		} else
+			set_config_bool_node(nvl, name, true);
+	}
+	free(tofree);
+	return (0);
+}
+
+/*
+ * PCI device configuration is stored in MIBs that encode the device's
+ * location:
+ *
+ * pci.<bus>.<slot>.<func>
+ *
+ * Where "bus", "slot", and "func" are all decimal values without
+ * leading zeroes.  Each valid device must have a "device" node which
+ * identifies the driver model of the device.
+ *
+ * Device backends can provide a parser for the "config" string.  If
+ * a custom parser is not provided, pci_parse_legacy_config() is used
+ * to parse the string.
+ */
 int
 pci_parse_slot(char *opt)
 {
-	struct businfo *bi;
-	struct slotinfo *si;
+	char node_name[sizeof("pci.XXX.XX.X")];
+	struct pci_devemu *pde;
 	char *emul, *config, *str, *cp;
 	int error, bnum, snum, fnum;
+	nvlist_t *nvl;
 
 	error = -1;
 	str = strdup(opt);
@@ -213,32 +257,33 @@ pci_parse_slot(char *opt)
 		goto done;
 	}
 
-	if (pci_businfo[bnum] == NULL)
-		pci_businfo[bnum] = calloc(1, sizeof(struct businfo));
-
-	bi = pci_businfo[bnum];
-	si = &bi->slotinfo[snum];
-
-	if (si->si_funcs[fnum].fi_name != NULL) {
-		EPRINTLN("pci slot %d:%d already occupied!",
-			snum, fnum);
+	pde = pci_emul_finddev(emul);
+	if (pde == NULL) {
+		EPRINTLN("pci slot %d:%d:%d: unknown device \"%s\"", bnum, snum,
+		    fnum, emul);
 		goto done;
 	}
 
-	if (pci_emul_finddev(emul) == NULL) {
-		EPRINTLN("pci slot %d:%d: unknown device \"%s\"",
-			snum, fnum, emul);
+	snprintf(node_name, sizeof(node_name), "pci.%d.%d.%d", bnum, snum,
+	    fnum);
+	nvl = find_config_node(node_name);
+	if (nvl != NULL) {
+		EPRINTLN("pci slot %d:%d:%d already occupied!", bnum, snum,
+		    fnum);
 		goto done;
 	}
+	nvl = create_config_node(node_name);
+	if (pde->pe_alias != NULL)
+		set_config_value_node(nvl, "device", pde->pe_alias);
+	else
+		set_config_value_node(nvl, "device", pde->pe_emu);
 
-	error = 0;
-	si->si_funcs[fnum].fi_name = emul;
-	si->si_funcs[fnum].fi_param = config;
-
+	if (pde->pe_legacy_config != NULL)
+		error = pde->pe_legacy_config(nvl, config);
+	else
+		error = pci_parse_legacy_config(nvl, config);
 done:
-	if (error)
-		free(str);
-
+	free(str);
 	return (error);
 }
 
@@ -714,7 +759,7 @@ pci_emul_add_capability(struct pci_devinst *pi, u_char *capdata, int caplen)
 }
 
 static struct pci_devemu *
-pci_emul_finddev(char *name)
+pci_emul_finddev(const char *name)
 {
 	struct pci_devemu **pdpp, *pdp;
 
@@ -755,7 +800,7 @@ pci_emul_init(struct vmctx *ctx, struct pci_devemu *pde, int bus, int slot,
 
 	pci_set_cfgdata8(pdi, PCIR_COMMAND, PCIM_CMD_BUSMASTEREN);
 
-	err = (*pde->pe_init)(ctx, pdi, fi->fi_param);
+	err = (*pde->pe_init)(ctx, pdi, fi->fi_config);
 	if (err == 0)
 		fi->fi_devi = pdi;
 	else
@@ -1084,11 +1129,14 @@ pci_ecfg_base(void)
 int
 init_pci(struct vmctx *ctx)
 {
+	char node_name[sizeof("pci.XXX.XX.X")];
 	struct mem_range mr;
 	struct pci_devemu *pde;
 	struct businfo *bi;
 	struct slotinfo *si;
 	struct funcinfo *fi;
+	nvlist_t *nvl;
+	const char *emul;
 	size_t lowmem;
 	uint64_t cpu_maxphysaddr, pci_emul_memresv64;
 	u_int regs[4];
@@ -1111,8 +1159,13 @@ init_pci(struct vmctx *ctx)
 	pci_emul_memlim64 = cpu_maxphysaddr;
 
 	for (bus = 0; bus < MAXBUSES; bus++) {
-		if ((bi = pci_businfo[bus]) == NULL)
+		snprintf(node_name, sizeof(node_name), "pci.%d", bus);
+		nvl = find_config_node(node_name);
+		if (nvl == NULL)
 			continue;
+		pci_businfo[bus] = calloc(1, sizeof(struct businfo));
+		bi = pci_businfo[bus];
+
 		/*
 		 * Keep track of the i/o and memory resources allocated to
 		 * this bus.
@@ -1125,10 +1178,34 @@ init_pci(struct vmctx *ctx)
 			si = &bi->slotinfo[slot];
 			for (func = 0; func < MAXFUNCS; func++) {
 				fi = &si->si_funcs[func];
-				if (fi->fi_name == NULL)
+				snprintf(node_name, sizeof(node_name),
+				    "pci.%d.%d.%d", bus, slot, func);
+				nvl = find_config_node(node_name);
+				if (nvl == NULL)
 					continue;
-				pde = pci_emul_finddev(fi->fi_name);
-				assert(pde != NULL);
+
+				fi->fi_config = nvl;
+				emul = get_config_value_node(nvl, "device");
+				if (emul == NULL) {
+					EPRINTLN("pci slot %d:%d:%d: missing "
+					    "\"device\" value", bus, slot, func);
+					return (EINVAL);
+				}
+				pde = pci_emul_finddev(emul);
+				if (pde == NULL) {
+					EPRINTLN("pci slot %d:%d:%d: unknown "
+					    "device \"%s\"", bus, slot, func,
+					    emul);
+					return (EINVAL);
+				}
+				if (pde->pe_alias != NULL) {
+					EPRINTLN("pci slot %d:%d:%d: legacy "
+					    "device \"%s\", use \"%s\" instead",
+					    bus, slot, func, emul,
+					    pde->pe_alias);
+					return (EINVAL);
+				}
+				fi->fi_pde = pde;
 				error = pci_emul_init(ctx, pde, bus, slot,
 				    func, fi);
 				if (error)
@@ -2040,14 +2117,12 @@ pci_find_slotted_dev(const char *dev_name, struct pci_devemu **pde,
 			si = &bi->slotinfo[slot];
 			for (func = 0; func < MAXFUNCS; func++) {
 				fi = &si->si_funcs[func];
-				if (fi->fi_name == NULL)
+				if (fi->fi_pde == NULL)
 					continue;
-				if (strcmp(dev_name, fi->fi_name))
+				if (strcmp(dev_name, fi->fi_pde->pe_emu) != 0)
 					continue;
 
-				*pde = pci_emul_finddev(fi->fi_name);
-				assert(*pde != NULL);
-
+				*pde = fi->fi_pde;
 				*pdi = fi->fi_devi;
 				return (0);
 			}
@@ -2169,7 +2244,7 @@ struct pci_emul_dsoftc {
 #define	PCI_EMUL_MSIX_MSGS	16
 
 static int
-pci_emul_dinit(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_emul_dinit(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	int error;
 	struct pci_emul_dsoftc *sc;
diff --git a/usr.sbin/bhyve/pci_emul.h b/usr.sbin/bhyve/pci_emul.h
index 3e9e95a74b47..2dbed07dfc14 100644
--- a/usr.sbin/bhyve/pci_emul.h
+++ b/usr.sbin/bhyve/pci_emul.h
@@ -34,6 +34,7 @@
 #include <sys/types.h>
 #include <sys/queue.h>
 #include <sys/kernel.h>
+#include <sys/nv.h>
 #include <sys/_pthreadtypes.h>
 
 #include <dev/pci/pcireg.h>
@@ -52,7 +53,9 @@ struct pci_devemu {
 
 	/* instance creation */
 	int       (*pe_init)(struct vmctx *, struct pci_devinst *,
-			     char *opts);
+			     nvlist_t *);
+	int	(*pe_legacy_config)(nvlist_t *, const char *);
+	const char *pe_alias;
 
 	/* ACPI DSDT enumeration */
 	void	(*pe_write_dsdt)(struct pci_devinst *);
@@ -77,6 +80,7 @@ struct pci_devemu {
 	int	(*pe_snapshot)(struct vm_snapshot_meta *meta);
 	int	(*pe_pause)(struct vmctx *ctx, struct pci_devinst *pi);
 	int	(*pe_resume)(struct vmctx *ctx, struct pci_devinst *pi);
+
 };
 #define PCI_EMUL_SET(x)   DATA_SET(pci_devemu_set, x);
 
@@ -236,6 +240,7 @@ int	pci_msix_enabled(struct pci_devinst *pi);
 int	pci_msix_table_bar(struct pci_devinst *pi);
 int	pci_msix_pba_bar(struct pci_devinst *pi);
 int	pci_msi_maxmsgnum(struct pci_devinst *pi);
+int	pci_parse_legacy_config(nvlist_t *nvl, const char *opt);
 int	pci_parse_slot(char *opt);
 void    pci_print_supported_devices();
 void	pci_populate_msicap(struct msicap *cap, int msgs, int nextptr);
diff --git a/usr.sbin/bhyve/pci_fbuf.c b/usr.sbin/bhyve/pci_fbuf.c
index 0bd740a0908c..215ce742edfb 100644
--- a/usr.sbin/bhyve/pci_fbuf.c
+++ b/usr.sbin/bhyve/pci_fbuf.c
@@ -47,6 +47,7 @@ __FBSDID("$FreeBSD$");
 
 #include "bhyvegc.h"
 #include "bhyverun.h"
+#include "config.h"
 #include "debug.h"
 #include "console.h"
 #include "inout.h"
@@ -116,15 +117,6 @@ static struct pci_fbuf_softc *fbuf_sc;
 #define	PCI_FBUF_MSI_MSGS	 4
 
 static void
-pci_fbuf_usage(char *opt)
-{
-
-	EPRINTLN("Invalid fbuf emulation option \"%s\"", opt);
-	EPRINTLN("fbuf: {wait,}{vga=on|io|off,}rfb=<ip>:port"
-	    "{,w=width}{,h=height}");
-}
-
-static void
 pci_fbuf_write(struct vmctx *ctx, int vcpu, struct pci_devinst *pi,
 	       int baridx, uint64_t offset, int size, uint64_t value)
 {
@@ -225,102 +217,108 @@ pci_fbuf_read(struct vmctx *ctx, int vcpu, struct pci_devinst *pi,
 }
 
 static int
-pci_fbuf_parse_opts(struct pci_fbuf_softc *sc, char *opts)
+pci_fbuf_parse_config(struct pci_fbuf_softc *sc, nvlist_t *nvl)
 {
-	char	*uopts, *uoptsbak, *xopts, *config;
-	char	*tmpstr;
-	int	ret;
-
-	ret = 0;
-	uoptsbak = uopts = strdup(opts);
-	while ((xopts = strsep(&uopts, ",")) != NULL) {
-		if (strcmp(xopts, "wait") == 0) {
-			sc->rfb_wait = 1;
-			continue;
-		}
-
-		if ((config = strchr(xopts, '=')) == NULL) {
-			pci_fbuf_usage(xopts);
-			ret = -1;
-			goto done;
-		}
-
-		*config++ = '\0';
-
-		DPRINTF(DEBUG_VERBOSE, ("pci_fbuf option %s = %s",
-		   xopts, config));
-
-		if (!strcmp(xopts, "tcp") || !strcmp(xopts, "rfb")) {
-			/*
-			 * IPv4 -- host-ip:port
-			 * IPv6 -- [host-ip%zone]:port
-			 * XXX for now port is mandatory.
-			 */
-			tmpstr = strsep(&config, "]");
-			if (config) {
-				if (tmpstr[0] == '[')
-					tmpstr++;
-				sc->rfb_host = strdup(tmpstr);
-				if (config[0] == ':')
-					config++;
-				else {
-					pci_fbuf_usage(xopts);
-					ret = -1;
-					goto done;
-				}
-				sc->rfb_port = atoi(config);
-			} else {
-				config = tmpstr;
-				tmpstr = strsep(&config, ":");
-				if (!config)
-					sc->rfb_port = atoi(tmpstr);
-				else {
-					sc->rfb_port = atoi(config);
-					sc->rfb_host = strdup(tmpstr);
+	const char *value;
+	char *cp;
+
+	sc->rfb_wait = get_config_bool_node_default(nvl, "wait", false);
+
+	/* Prefer "rfb" to "tcp". */
+	value = get_config_value_node(nvl, "rfb");
+	if (value == NULL)
+		value = get_config_value_node(nvl, "tcp");
+	if (value != NULL) {
+		/*
+		 * IPv4 -- host-ip:port
+		 * IPv6 -- [host-ip%zone]:port
+		 * XXX for now port is mandatory for IPv4.
+		 */
+		if (value[0] == '[') {
+			cp = strchr(value + 1, ']');
+			if (cp == NULL || cp == value + 1) {
+				EPRINTLN("fbuf: Invalid IPv6 address: \"%s\"",
+				    value);
+				return (-1);
+			}
+			sc->rfb_host = strndup(value + 1, cp - (value + 1));
+			cp++;
+			if (*cp == ':') {
+				cp++;
+				if (*cp == '\0') {
+					EPRINTLN(
+					    "fbuf: Missing port number: \"%s\"",
+					    value);
+					return (-1);
 				}
+				sc->rfb_port = atoi(cp);
+			} else if (*cp != '\0') {
+				EPRINTLN("fbuf: Invalid IPv6 address: \"%s\"",
+				    value);
+				return (-1);
 			}
-	        } else if (!strcmp(xopts, "vga")) {
-			if (!strcmp(config, "off")) {
-				sc->vga_enabled = 0;
-			} else if (!strcmp(config, "io")) {
-				sc->vga_enabled = 1;
-				sc->vga_full = 0;
-			} else if (!strcmp(config, "on")) {
-				sc->vga_enabled = 1;
-				sc->vga_full = 1;
+		} else {
+			cp = strchr(value, ':');
+			if (cp == NULL) {
+				sc->rfb_port = atoi(value);
 			} else {
-				pci_fbuf_usage(xopts);
-				ret = -1;
-				goto done;
+				sc->rfb_host = strndup(value, cp - value);
+				cp++;
+				if (*cp == '\0') {
+					EPRINTLN(
+					    "fbuf: Missing port number: \"%s\"",
+					    value);
+					return (-1);
+				}
+				sc->rfb_port = atoi(cp);
 			}
-	        } else if (!strcmp(xopts, "w")) {
-		        sc->memregs.width = atoi(config);
-			if (sc->memregs.width > COLS_MAX) {
-				pci_fbuf_usage(xopts);
-				ret = -1;
-				goto done;
-			} else if (sc->memregs.width == 0)
-				sc->memregs.width = 1920;
-		} else if (!strcmp(xopts, "h")) {
-			sc->memregs.height = atoi(config);
-			if (sc->memregs.height > ROWS_MAX) {
-				pci_fbuf_usage(xopts);
-				ret = -1;
-				goto done;
-			} else if (sc->memregs.height == 0)
-				sc->memregs.height = 1080;
-		} else if (!strcmp(xopts, "password")) {
-			sc->rfb_password = strdup(config);
+		}
+	}
+
+	value = get_config_value_node(nvl, "vga");
+	if (value != NULL) {
+		if (strcmp(value, "off") == 0) {
+			sc->vga_enabled = 0;
+		} else if (strcmp(value, "io") == 0) {
+			sc->vga_enabled = 1;
+			sc->vga_full = 0;
+		} else if (strcmp(value, "on") == 0) {
+			sc->vga_enabled = 1;
+			sc->vga_full = 1;
 		} else {
-			pci_fbuf_usage(xopts);
-			ret = -1;
-			goto done;
+			EPRINTLN("fbuf: Invalid vga setting: \"%s\"", value);
+			return (-1);
 		}
 	}
 
-done:
-	free(uoptsbak);
-	return (ret);
+	value = get_config_value_node(nvl, "w");
+	if (value != NULL) {
+		sc->memregs.width = atoi(value);
+		if (sc->memregs.width > COLS_MAX) {
+			EPRINTLN("fbuf: width %d too large", sc->memregs.width);
+			return (-1);
+		}
+		if (sc->memregs.width == 0)
+			sc->memregs.width = 1920;
+	}
+
+	value = get_config_value_node(nvl, "h");
+	if (value != NULL) {
+		sc->memregs.height = atoi(value);
+		if (sc->memregs.height > ROWS_MAX) {
+			EPRINTLN("fbuf: height %d too large",
+			    sc->memregs.height);
+			return (-1);
+		}
+		if (sc->memregs.height == 0)
+			sc->memregs.height = 1080;
+	}
+
+	value = get_config_value_node(nvl, "password");
+	if (value != NULL)
+		sc->rfb_password = strdup(value);
+
+	return (0);
 }
 
 
@@ -351,7 +349,7 @@ pci_fbuf_render(struct bhyvegc *gc, void *arg)
 }
 
 static int
-pci_fbuf_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_fbuf_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	int error, prot;
 	struct pci_fbuf_softc *sc;
@@ -391,7 +389,7 @@ pci_fbuf_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 
 	sc->fsc_pi = pi;
 
-	error = pci_fbuf_parse_opts(sc, opts);
+	error = pci_fbuf_parse_config(sc, nvl);
 	if (error != 0)
 		goto done;
 
diff --git a/usr.sbin/bhyve/pci_hda.c b/usr.sbin/bhyve/pci_hda.c
index 86e46a550aec..7491944fedd0 100644
--- a/usr.sbin/bhyve/pci_hda.c
+++ b/usr.sbin/bhyve/pci_hda.c
@@ -34,6 +34,7 @@ __FBSDID("$FreeBSD$");
 
 #include "pci_hda.h"
 #include "bhyverun.h"
+#include "config.h"
 #include "pci_emul.h"
 #include "hdac_reg.h"
 
@@ -147,13 +148,11 @@ static inline uint32_t hda_get_reg_by_offset(struct hda_softc *sc,
 static inline void hda_set_field_by_offset(struct hda_softc *sc,
     uint32_t offset, uint32_t mask, uint32_t value);
 
-static uint8_t hda_parse_config(const char *opts, const char *key, char *val);
-static struct hda_softc *hda_init(const char *opts);
+static struct hda_softc *hda_init(nvlist_t *nvl);
 static void hda_update_intr(struct hda_softc *sc);
 static void hda_response_interrupt(struct hda_softc *sc);
 static int hda_codec_constructor(struct hda_softc *sc,
-    struct hda_codec_class *codec, const char *play, const char *rec,
-    const char *opts);
+    struct hda_codec_class *codec, const char *play, const char *rec);
 static struct hda_codec_class *hda_find_codec_class(const char *name);
 
 static int hda_send_command(struct hda_softc *sc, uint32_t verb);
@@ -210,7 +209,7 @@ static uint64_t hda_get_clock_ns(void);
 /*
  * PCI HDA function declarations
  */
-static int pci_hda_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts);
+static int pci_hda_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl);
 static void pci_hda_write(struct vmctx *ctx, int vcpu, struct pci_devinst *pi,
     int baridx, uint64_t offset, int size, uint64_t value);
 static uint64_t pci_hda_read(struct vmctx *ctx, int vcpu, struct pci_devinst *pi,
@@ -318,67 +317,20 @@ hda_set_field_by_offset(struct hda_softc *sc, uint32_t offset,
 	hda_set_reg_by_offset(sc, offset, reg_value);
 }
 
-static uint8_t
-hda_parse_config(const char *opts, const char *key, char *val)
-{
-	char buf[64];
-	char *s = buf;
-	char *tmp = NULL;
-	size_t len;
-	int i;
-
-	if (!opts)
-		return (0);
-
-	len = strlen(opts);
-	if (len >= sizeof(buf)) {
-		DPRINTF("Opts too big");
-		return (0);
-	}
-
-	DPRINTF("opts: %s", opts);
-
-	strcpy(buf, opts);
-
-	for (i = 0; i < len; i++)
-		if (buf[i] == ',') {
-			buf[i] = 0;
-			tmp = buf + i + 1;
-			break;
-		}
-
-	if (!memcmp(s, key, strlen(key))) {
-		strncpy(val, s + strlen(key), 64);
-		return (1);
-	}
-
-	if (!tmp)
-		return (0);
-
-	s = tmp;
-	if (!memcmp(s, key, strlen(key))) {
-		strncpy(val, s + strlen(key), 64);
-		return (1);
-	}
-
-	return (0);
-}
-
 static struct hda_softc *
-hda_init(const char *opts)
+hda_init(nvlist_t *nvl)
 {
 	struct hda_softc *sc = NULL;
 	struct hda_codec_class *codec = NULL;
-	char play[64];
-	char rec[64];
-	int err, p, r;
+	const char *value;
+	char *play;
+	char *rec;
+	int err;
 
 #if DEBUG_HDA == 1
 	dbg = fopen("/tmp/bhyve_hda.log", "w+");
 #endif
 
-	DPRINTF("opts: %s", opts);
-
 	sc = calloc(1, sizeof(*sc));
 	if (!sc)
 		return (NULL);
@@ -386,19 +338,28 @@ hda_init(const char *opts)
 	hda_reset_regs(sc);
 
 	/*
-	 * TODO search all the codecs declared in opts
+	 * TODO search all configured codecs
 	 * For now we play with one single codec
 	 */
 	codec = hda_find_codec_class("hda_codec");
 	if (codec) {
-		p = hda_parse_config(opts, "play=", play);
-		r = hda_parse_config(opts, "rec=", rec);
+		value = get_config_value_node(nvl, "play");
+		if (value == NULL)
+			play = NULL;
+		else
+			play = strdup(value);
+		value = get_config_value_node(nvl, "rec");
+		if (value == NULL)
+			rec = NULL;
+		else
+			rec = strdup(value);
 		DPRINTF("play: %s rec: %s", play, rec);
-		if (p | r) {
-			err = hda_codec_constructor(sc, codec, p ?	\
-				play : NULL, r ? rec : NULL, NULL);
+		if (play != NULL || rec != NULL) {
+			err = hda_codec_constructor(sc, codec, play, rec);
 			assert(!err);
 		}
+		free(play);
+		free(rec);
 	}
 
 	return (sc);
@@ -470,7 +431,7 @@ hda_response_interrupt(struct hda_softc *sc)
 
 static int
 hda_codec_constructor(struct hda_softc *sc, struct hda_codec_class *codec,
-    const char *play, const char *rec, const char *opts)
+    const char *play, const char *rec)
 {
 	struct hda_codec_inst *hci = NULL;
 
@@ -493,7 +454,7 @@ hda_codec_constructor(struct hda_softc *sc, struct hda_codec_class *codec,
 		return (-1);
 	}
 
-	return (codec->init(hci, play, rec, opts));
+	return (codec->init(hci, play, rec));
 }
 
 static struct hda_codec_class *
@@ -1263,7 +1224,7 @@ static uint64_t hda_get_clock_ns(void)
  * PCI HDA function definitions
  */
 static int
-pci_hda_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_hda_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	struct hda_softc *sc = NULL;
 
@@ -1285,7 +1246,7 @@ pci_hda_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 	/* allocate an IRQ pin for our slot */
 	pci_lintr_request(pi);
 
-	sc = hda_init(opts);
+	sc = hda_init(nvl);
 	if (!sc)
 		return (-1);
 
diff --git a/usr.sbin/bhyve/pci_hda.h b/usr.sbin/bhyve/pci_hda.h
index 7b6bba92e496..65a85f6d603d 100644
--- a/usr.sbin/bhyve/pci_hda.h
+++ b/usr.sbin/bhyve/pci_hda.h
@@ -72,7 +72,7 @@ struct hda_codec_inst {
 struct hda_codec_class {
 	char *name;
 	int (*init)(struct hda_codec_inst *hci, const char *play,
-		const char *rec, const char *opts);
+		const char *rec);
 	int (*reset)(struct hda_codec_inst *hci);
 	int (*command)(struct hda_codec_inst *hci, uint32_t cmd_data);
 	int (*notify)(struct hda_codec_inst *hci, uint8_t run, uint8_t stream,
diff --git a/usr.sbin/bhyve/pci_hostbridge.c b/usr.sbin/bhyve/pci_hostbridge.c
index 559496a9fee0..7099474eaf92 100644
--- a/usr.sbin/bhyve/pci_hostbridge.c
+++ b/usr.sbin/bhyve/pci_hostbridge.c
@@ -31,15 +31,30 @@
 #include <sys/cdefs.h>
 __FBSDID("$FreeBSD$");
 
+#include <stdlib.h>
+
+#include "config.h"
 #include "pci_emul.h"
 
 static int
-pci_hostbridge_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_hostbridge_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
+	const char *value;
+	u_int vendor, device;
+
+	vendor = 0x1275;	/* NetApp */
+	device = 0x1275;	/* NetApp */
+
+	value = get_config_value_node(nvl, "vendor");
+	if (value != NULL)
+		vendor = strtol(value, NULL, 0);
+	value = get_config_value_node(nvl, "device");
+	if (value != NULL)
+		device = strtol(value, NULL, 0);
 
 	/* config space */
-	pci_set_cfgdata16(pi, PCIR_VENDOR, 0x1275);	/* NetApp */
-	pci_set_cfgdata16(pi, PCIR_DEVICE, 0x1275);	/* NetApp */
+	pci_set_cfgdata16(pi, PCIR_VENDOR, vendor);
+	pci_set_cfgdata16(pi, PCIR_DEVICE, device);
 	pci_set_cfgdata8(pi, PCIR_HDRTYPE, PCIM_HDRTYPE_NORMAL);
 	pci_set_cfgdata8(pi, PCIR_CLASS, PCIC_BRIDGE);
 	pci_set_cfgdata8(pi, PCIR_SUBCLASS, PCIS_BRIDGE_HOST);
@@ -50,18 +65,19 @@ pci_hostbridge_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 }
 
 static int
-pci_amd_hostbridge_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_amd_hostbridge_legacy_config(nvlist_t *nvl, const char *opts)
 {
-	(void) pci_hostbridge_init(ctx, pi, opts);
-	pci_set_cfgdata16(pi, PCIR_VENDOR, 0x1022);	/* AMD */
-	pci_set_cfgdata16(pi, PCIR_DEVICE, 0x7432);	/* made up */
+
+	set_config_value_node(nvl, "vendor", "0x1022");	/* AMD */
+	set_config_value_node(nvl, "device", "0x7432");	/* made up */
 
 	return (0);
 }
 
 struct pci_devemu pci_de_amd_hostbridge = {
 	.pe_emu = "amd_hostbridge",
-	.pe_init = pci_amd_hostbridge_init,
+	.pe_legacy_config = pci_amd_hostbridge_legacy_config,
+	.pe_alias = "hostbridge",
 };
 PCI_EMUL_SET(pci_de_amd_hostbridge);
 
diff --git a/usr.sbin/bhyve/pci_lpc.c b/usr.sbin/bhyve/pci_lpc.c
index aa5997848f24..8cc536fef112 100644
--- a/usr.sbin/bhyve/pci_lpc.c
+++ b/usr.sbin/bhyve/pci_lpc.c
@@ -45,6 +45,7 @@ __FBSDID("$FreeBSD$");
 #include "acpi.h"
 #include "debug.h"
 #include "bootrom.h"
+#include "config.h"
 #include "inout.h"
 #include "pci_emul.h"
 #include "pci_irq.h"
@@ -68,20 +69,21 @@ SYSRES_IO(NMISC_PORT, 1);
 
 static struct pci_devinst *lpc_bridge;
 
-static const char *romfile;
-
 #define	LPC_UART_NUM	4
 static struct lpc_uart_softc {
 	struct uart_softc *uart_softc;
-	const char *opts;
 	int	iobase;
 	int	irq;
 	int	enabled;
 } lpc_uart_softc[LPC_UART_NUM];
 
-static const char *lpc_uart_names[LPC_UART_NUM] = { "COM1", "COM2", "COM3", "COM4" };
+static const char *lpc_uart_names[LPC_UART_NUM] = {
+	"com1", "com2", "com3", "com4"
+};
 
-static bool pctestdev_present;
+static const char *lpc_uart_acpi_names[LPC_UART_NUM] = {
+	"COM1", "COM2", "COM3", "COM4"
+};
 
 /*
  * LPC device configuration is in the following form:
@@ -92,41 +94,38 @@ int
 lpc_device_parse(const char *opts)
 {
 	int unit, error;
-	char *str, *cpy, *lpcdev;
+	char *str, *cpy, *lpcdev, *node_name;
 
 	error = -1;
 	str = cpy = strdup(opts);
 	lpcdev = strsep(&str, ",");
 	if (lpcdev != NULL) {
 		if (strcasecmp(lpcdev, "bootrom") == 0) {
-			romfile = str;
+			set_config_value("lpc.bootrom", str);
 			error = 0;
 			goto done;
 		}
 		for (unit = 0; unit < LPC_UART_NUM; unit++) {
 			if (strcasecmp(lpcdev, lpc_uart_names[unit]) == 0) {
-				lpc_uart_softc[unit].opts = str;
+				asprintf(&node_name, "lpc.%s.path",
+				    lpc_uart_names[unit]);
+				set_config_value(node_name, str);
+				free(node_name);
 				error = 0;
 				goto done;
 			}
 		}
 		if (strcasecmp(lpcdev, pctestdev_getname()) == 0) {
-			if (pctestdev_present) {
-				EPRINTLN("More than one %s device conf is "
-				    "specified; only one is allowed.",
-				    pctestdev_getname());
-			} else if (pctestdev_parse(str) == 0) {
-				pctestdev_present = true;
-				error = 0;
-				free(cpy);
-				goto done;
-			}
+			asprintf(&node_name, "lpc.%s", pctestdev_getname());
+			set_config_bool(node_name, true);
+			free(node_name);
+			error = 0;
+			goto done;
 		}
 	}
 
 done:
-	if (error)
-		free(cpy);
+	free(cpy);
 
 	return (error);
 }
@@ -146,7 +145,7 @@ const char *
 lpc_bootrom(void)
 {
 
-	return (romfile);
+	return (get_config_value("lpc.bootrom"));
 }
 
 static void
@@ -205,9 +204,11 @@ lpc_init(struct vmctx *ctx)
 {
 	struct lpc_uart_softc *sc;
 	struct inout_port iop;
-	const char *name;
+	const char *backend, *name, *romfile;
+	char *node_name;
 	int unit, error;
 
+	romfile = get_config_value("lpc.bootrom");
 	if (romfile != NULL) {
 		error = bootrom_loadrom(ctx, romfile);
 		if (error)
@@ -229,9 +230,12 @@ lpc_init(struct vmctx *ctx)
 		sc->uart_softc = uart_init(lpc_uart_intr_assert,
 				    lpc_uart_intr_deassert, sc);
 
-		if (uart_set_backend(sc->uart_softc, sc->opts) != 0) {
+		asprintf(&node_name, "lpc.%s.path", name);
+		backend = get_config_value(node_name);
+		free(node_name);
+		if (uart_set_backend(sc->uart_softc, backend) != 0) {
 			EPRINTLN("Unable to initialize backend '%s' "
-			    "for LPC device %s", sc->opts, name);
+			    "for LPC device %s", backend, name);
 			return (-1);
 		}
 
@@ -249,11 +253,13 @@ lpc_init(struct vmctx *ctx)
 	}
 
 	/* pc-testdev */
-	if (pctestdev_present) {
+	asprintf(&node_name, "lpc.%s", pctestdev_getname());
+	if (get_config_bool_default(node_name, false)) {
 		error = pctestdev_init(ctx);
 		if (error)
 			return (error);
 	}
+	free(node_name);
 
 	return (0);
 }
@@ -362,7 +368,7 @@ pci_lpc_uart_dsdt(void)
 		if (!sc->enabled)
 			continue;
 		dsdt_line("");
-		dsdt_line("Device (%s)", lpc_uart_names[unit]);
+		dsdt_line("Device (%s)", lpc_uart_acpi_names[unit]);
 		dsdt_line("{");
 		dsdt_line("  Name (_HID, EisaId (\"PNP0501\"))");
 		dsdt_line("  Name (_UID, %d)", unit + 1);
@@ -416,7 +422,7 @@ pci_lpc_read(struct vmctx *ctx, int vcpu, struct pci_devinst *pi,
 #define	LPC_VENDOR	0x8086
 
 static int
-pci_lpc_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_lpc_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 
 	/*
diff --git a/usr.sbin/bhyve/pci_nvme.c b/usr.sbin/bhyve/pci_nvme.c
index 24f401630d6d..0abc0415a1d8 100644
--- a/usr.sbin/bhyve/pci_nvme.c
+++ b/usr.sbin/bhyve/pci_nvme.c
@@ -81,6 +81,7 @@ __FBSDID("$FreeBSD$");
 
 #include "bhyverun.h"
 #include "block_if.h"
+#include "config.h"
 #include "debug.h"
 #include "pci_emul.h"
 
@@ -578,8 +579,9 @@ pci_nvme_init_nsdata(struct pci_nvme_softc *sc,
 		char *data = NULL;
 		uint64_t eui64 = nvstore->eui64;
 
-		asprintf(&data, "%s%u%u%u", vmname, sc->nsc_pi->pi_bus,
-		    sc->nsc_pi->pi_slot, sc->nsc_pi->pi_func);
+		asprintf(&data, "%s%u%u%u", get_config_value("name"),
+		    sc->nsc_pi->pi_bus, sc->nsc_pi->pi_slot,
+		    sc->nsc_pi->pi_func);
 
 		if (data != NULL) {
 			eui64 = OUI_FREEBSD_NVME_LOW | crc16(0, data, strlen(data));
@@ -2609,14 +2611,12 @@ pci_nvme_read(struct vmctx *ctx, int vcpu, struct pci_devinst *pi, int baridx,
 	return (0);
 }
 
-
 static int
-pci_nvme_parse_opts(struct pci_nvme_softc *sc, char *opts)
+pci_nvme_parse_config(struct pci_nvme_softc *sc, nvlist_t *nvl)
 {
 	char bident[sizeof("XX:X:X")];
-	char	*uopt, *xopts, *config;
+	const char *value;
 	uint32_t sectsz;
-	int optidx;
 
 	sc->max_queues = NVME_QUEUES;
 	sc->max_qentries = NVME_MAX_QENTRIES;
@@ -2625,81 +2625,81 @@ pci_nvme_parse_opts(struct pci_nvme_softc *sc, char *opts)
 	sc->num_cqueues = sc->max_queues;
 	sc->dataset_management = NVME_DATASET_MANAGEMENT_AUTO;
 	sectsz = 0;
-
-	uopt = strdup(opts);
-	optidx = 0;
 	snprintf(sc->ctrldata.sn, sizeof(sc->ctrldata.sn),
 	         "NVME-%d-%d", sc->nsc_pi->pi_slot, sc->nsc_pi->pi_func);
-	for (xopts = strtok(uopt, ",");
-	     xopts != NULL;
-	     xopts = strtok(NULL, ",")) {
-
-		if ((config = strchr(xopts, '=')) != NULL)
-			*config++ = '\0';
-
-		if (!strcmp("maxq", xopts)) {
-			sc->max_queues = atoi(config);
-		} else if (!strcmp("qsz", xopts)) {
-			sc->max_qentries = atoi(config);
-		} else if (!strcmp("ioslots", xopts)) {
-			sc->ioslots = atoi(config);
-		} else if (!strcmp("sectsz", xopts)) {
-			sectsz = atoi(config);
-		} else if (!strcmp("ser", xopts)) {
-			/*
-			 * This field indicates the Product Serial Number in
-			 * 7-bit ASCII, unused bytes should be space characters.
-			 * Ref: NVMe v1.3c.
-			 */
-			cpywithpad((char *)sc->ctrldata.sn,
-			           sizeof(sc->ctrldata.sn), config, ' ');
-		} else if (!strcmp("ram", xopts)) {
-			uint64_t sz = strtoull(&xopts[4], NULL, 10);
-
-			sc->nvstore.type = NVME_STOR_RAM;
-			sc->nvstore.size = sz * 1024 * 1024;
-			sc->nvstore.ctx = calloc(1, sc->nvstore.size);
-			sc->nvstore.sectsz = 4096;
-			sc->nvstore.sectsz_bits = 12;
-			if (sc->nvstore.ctx == NULL) {
-				perror("Unable to allocate RAM");
-				free(uopt);
-				return (-1);
-			}
-		} else if (!strcmp("eui64", xopts)) {
-			sc->nvstore.eui64 = htobe64(strtoull(config, NULL, 0));
-		} else if (!strcmp("dsm", xopts)) {
-			if (!strcmp("auto", config))
-				sc->dataset_management = NVME_DATASET_MANAGEMENT_AUTO;
-			else if (!strcmp("enable", config))
-				sc->dataset_management = NVME_DATASET_MANAGEMENT_ENABLE;
-			else if (!strcmp("disable", config))
-				sc->dataset_management = NVME_DATASET_MANAGEMENT_DISABLE;
-		} else if (optidx == 0) {
-			snprintf(bident, sizeof(bident), "%d:%d",
-			         sc->nsc_pi->pi_slot, sc->nsc_pi->pi_func);
-			sc->nvstore.ctx = blockif_open(xopts, bident);
-			if (sc->nvstore.ctx == NULL) {
-				perror("Could not open backing file");
-				free(uopt);
-				return (-1);
-			}
-			sc->nvstore.type = NVME_STOR_BLOCKIF;
-			sc->nvstore.size = blockif_size(sc->nvstore.ctx);
-		} else {
-			EPRINTLN("Invalid option %s", xopts);
-			free(uopt);
+
+	value = get_config_value_node(nvl, "maxq");
+	if (value != NULL)
+		sc->max_queues = atoi(value);
+	value = get_config_value_node(nvl, "qsz");
+	if (value != NULL) {
+		sc->max_qentries = atoi(value);
+		if (sc->max_qentries <= 0) {
+			EPRINTLN("nvme: Invalid qsz option %d",
+			    sc->max_qentries);
 			return (-1);
 		}
-
-		optidx++;
 	}
-	free(uopt);
-
-	if (sc->nvstore.ctx == NULL || sc->nvstore.size == 0) {
-		EPRINTLN("backing store not specified");
-		return (-1);
+	value = get_config_value_node(nvl, "ioslots");
+	if (value != NULL) {
+		sc->ioslots = atoi(value);
+		if (sc->ioslots <= 0) {
+			EPRINTLN("Invalid ioslots option %d", sc->ioslots);
+			return (-1);
+		}
+	}
+	value = get_config_value_node(nvl, "sectsz");
+	if (value != NULL)
+		sectsz = atoi(value);
+	value = get_config_value_node(nvl, "ser");
+	if (value != NULL) {
+		/*
+		 * This field indicates the Product Serial Number in
+		 * 7-bit ASCII, unused bytes should be space characters.
+		 * Ref: NVMe v1.3c.
+		 */
+		cpywithpad((char *)sc->ctrldata.sn,
+		    sizeof(sc->ctrldata.sn), value, ' ');
+	}
+	value = get_config_value_node(nvl, "eui64");
+	if (value != NULL)
+		sc->nvstore.eui64 = htobe64(strtoull(value, NULL, 0));
+	value = get_config_value_node(nvl, "dsm");
+	if (value != NULL) {
+		if (strcmp(value, "auto") == 0)
+			sc->dataset_management = NVME_DATASET_MANAGEMENT_AUTO;
+		else if (strcmp(value, "enable") == 0)
+			sc->dataset_management = NVME_DATASET_MANAGEMENT_ENABLE;
+		else if (strcmp(value, "disable") == 0)
+			sc->dataset_management = NVME_DATASET_MANAGEMENT_DISABLE;
+	}
+
+	value = get_config_value_node(nvl, "ram");
+	if (value != NULL) {
+		uint64_t sz = strtoull(value, NULL, 10);
+
+		sc->nvstore.type = NVME_STOR_RAM;
+		sc->nvstore.size = sz * 1024 * 1024;
+		sc->nvstore.ctx = calloc(1, sc->nvstore.size);
+		sc->nvstore.sectsz = 4096;
+		sc->nvstore.sectsz_bits = 12;
+		if (sc->nvstore.ctx == NULL) {
+			EPRINTLN("nvme: Unable to allocate RAM");
+			return (-1);
+		}
+	} else {
+		snprintf(bident, sizeof(bident), "%d:%d",
+		    sc->nsc_pi->pi_slot, sc->nsc_pi->pi_func);
+		sc->nvstore.ctx = blockif_open(nvl, bident);
+		if (sc->nvstore.ctx == NULL) {
+			EPRINTLN("nvme: Could not open backing file: %s",
+			    strerror(errno));
+			return (-1);
+		}
+		sc->nvstore.type = NVME_STOR_BLOCKIF;
+		sc->nvstore.size = blockif_size(sc->nvstore.ctx);
 	}
+
 	if (sectsz == 512 || sectsz == 4096 || sectsz == 8192)
 		sc->nvstore.sectsz = sectsz;
 	else if (sc->nvstore.type != NVME_STOR_RAM)
@@ -2711,20 +2711,11 @@ pci_nvme_parse_opts(struct pci_nvme_softc *sc, char *opts)
 	if (sc->max_queues <= 0 || sc->max_queues > NVME_QUEUES)
 		sc->max_queues = NVME_QUEUES;
 
-	if (sc->max_qentries <= 0) {
-		EPRINTLN("Invalid qsz option");
-		return (-1);
-	}
-	if (sc->ioslots <= 0) {
-		EPRINTLN("Invalid ioslots option");
-		return (-1);
-	}
-
 	return (0);
 }
 
 static int
-pci_nvme_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_nvme_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	struct pci_nvme_softc *sc;
 	uint32_t pci_membar_sz;
@@ -2736,7 +2727,7 @@ pci_nvme_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 	pi->pi_arg = sc;
 	sc->nsc_pi = pi;
 
-	error = pci_nvme_parse_opts(sc, opts);
+	error = pci_nvme_parse_config(sc, nvl);
 	if (error < 0)
 		goto done;
 	else
@@ -2813,6 +2804,7 @@ done:
 struct pci_devemu pci_de_nvme = {
 	.pe_emu =	"nvme",
 	.pe_init =	pci_nvme_init,
+	.pe_legacy_config = blockif_legacy_config,
 	.pe_barwrite =	pci_nvme_write,
 	.pe_barread =	pci_nvme_read
 };
diff --git a/usr.sbin/bhyve/pci_passthru.c b/usr.sbin/bhyve/pci_passthru.c
index 3305a4854812..f1d306b03959 100644
--- a/usr.sbin/bhyve/pci_passthru.c
+++ b/usr.sbin/bhyve/pci_passthru.c
@@ -59,6 +59,9 @@ __FBSDID("$FreeBSD$");
 
 #include <machine/vmm.h>
 #include <vmmapi.h>
+
+#include "config.h"
+#include "debug.h"
 #include "pci_emul.h"
 #include "mem.h"
 
@@ -648,10 +651,34 @@ done:
 }
 
 static int
-passthru_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+passthru_legacy_config(nvlist_t *nvl, const char *opts)
+{
+	char value[16];
+	int bus, slot, func;
+
+	if (opts == NULL)
+		return (0);
+
+	if (sscanf(opts, "%d/%d/%d", &bus, &slot, &func) != 3) {
+		EPRINTLN("passthru: invalid options \"%s\"", opts);
+		return (-1);
+	}
+
+	snprintf(value, sizeof(value), "%d", bus);
+	set_config_value_node(nvl, "bus", value);
+	snprintf(value, sizeof(value), "%d", slot);
+	set_config_value_node(nvl, "slot", value);
+	snprintf(value, sizeof(value), "%d", func);
+	set_config_value_node(nvl, "func", value);
+	return (0);
+}
+
+static int
+passthru_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	int bus, slot, func, error, memflags;
 	struct passthru_softc *sc;
+	const char *value;
 #ifndef WITHOUT_CAPSICUM
 	cap_rights_t rights;
 	cap_ioctl_t pci_ioctls[] = { PCIOCREAD, PCIOCWRITE, PCIOCGETBAR };
@@ -716,11 +743,18 @@ passthru_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 		errx(EX_OSERR, "Unable to apply rights for sandbox");
 #endif
 
-	if (opts == NULL ||
-	    sscanf(opts, "%d/%d/%d", &bus, &slot, &func) != 3) {
-		warnx("invalid passthru options");
-		return (error);
-	}
+#define GET_INT_CONFIG(var, name) do {					\
+	value = get_config_value_node(nvl, name);			\
+	if (value == NULL) {						\
+		EPRINTLN("passthru: missing required %s setting", name); \
+		return (error);						\
+	}								\
+	var = atoi(value);						\
+} while (0)
+
+	GET_INT_CONFIG(bus, "bus");
+	GET_INT_CONFIG(slot, "slot");
+	GET_INT_CONFIG(func, "func");
 
 	if (vm_assign_pptdev(ctx, bus, slot, func) != 0) {
 		warnx("PCI device at %d/%d/%d is not using the ppt(4) driver",
@@ -957,6 +991,7 @@ passthru_read(struct vmctx *ctx, int vcpu, struct pci_devinst *pi, int baridx,
 struct pci_devemu passthru = {
 	.pe_emu		= "passthru",
 	.pe_init	= passthru_init,
+	.pe_legacy_config = passthru_legacy_config,
 	.pe_cfgwrite	= passthru_cfgwrite,
 	.pe_cfgread	= passthru_cfgread,
 	.pe_barwrite 	= passthru_write,
diff --git a/usr.sbin/bhyve/pci_uart.c b/usr.sbin/bhyve/pci_uart.c
index 2e8177bafb2c..25ef1ed66217 100644
--- a/usr.sbin/bhyve/pci_uart.c
+++ b/usr.sbin/bhyve/pci_uart.c
@@ -36,6 +36,7 @@ __FBSDID("$FreeBSD$");
 #include <stdio.h>
 
 #include "bhyverun.h"
+#include "config.h"
 #include "debug.h"
 #include "pci_emul.h"
 #include "uart_emul.h"
@@ -89,9 +90,19 @@ pci_uart_read(struct vmctx *ctx, int vcpu, struct pci_devinst *pi,
 }
 
 static int
-pci_uart_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_uart_legacy_config(nvlist_t *nvl, const char *opts)
+{
+
+	if (opts != NULL)
+		set_config_value_node(nvl, "path", opts);
+	return (0);
+}
+
+static int
+pci_uart_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	struct uart_softc *sc;
+	const char *device;
 
 	pci_emul_alloc_bar(pi, 0, PCIBAR_IO, UART_IO_BAR_SIZE);
 	pci_lintr_request(pi);
@@ -104,9 +115,10 @@ pci_uart_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 	sc = uart_init(pci_uart_intr_assert, pci_uart_intr_deassert, pi);
 	pi->pi_arg = sc;
 
-	if (uart_set_backend(sc, opts) != 0) {
+	device = get_config_value_node(nvl, "path");
+	if (uart_set_backend(sc, device) != 0) {
 		EPRINTLN("Unable to initialize backend '%s' for "
-		    "pci uart at %d:%d", opts, pi->pi_slot, pi->pi_func);
+		    "pci uart at %d:%d", device, pi->pi_slot, pi->pi_func);
 		return (-1);
 	}
 
@@ -116,6 +128,7 @@ pci_uart_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 struct pci_devemu pci_de_com = {
 	.pe_emu =	"uart",
 	.pe_init =	pci_uart_init,
+	.pe_legacy_config = pci_uart_legacy_config,
 	.pe_barwrite =	pci_uart_write,
 	.pe_barread =	pci_uart_read
 };
diff --git a/usr.sbin/bhyve/pci_virtio_9p.c b/usr.sbin/bhyve/pci_virtio_9p.c
index 4296ccd11c7f..e27159eb22cb 100644
--- a/usr.sbin/bhyve/pci_virtio_9p.c
+++ b/usr.sbin/bhyve/pci_virtio_9p.c
@@ -51,6 +51,8 @@ __FBSDID("$FreeBSD$");
 #include <backend/fs.h>
 
 #include "bhyverun.h"
+#include "config.h"
+#include "debug.h"
 #include "pci_emul.h"
 #include "virtio.h"
 
@@ -227,54 +229,67 @@ pci_vt9p_notify(void *vsc, struct vqueue_info *vq)
 	}
 }
 
-
 static int
-pci_vt9p_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_vt9p_legacy_config(nvlist_t *nvl, const char *opts)
 {
-	struct pci_vt9p_softc *sc;
-	char *opt;
-	char *sharename = NULL;
-	char *rootpath = NULL;
-	int rootfd;
-	bool ro = false;
-	cap_rights_t rootcap;
+	char *sharename, *tofree, *token, *tokens;
 
-	if (opts == NULL) {
-		printf("virtio-9p: share name and path required\n");
-		return (1);
-	}
+	if (opts == NULL)
+		return (0);
 
-	while ((opt = strsep(&opts, ",")) != NULL) {
-		if (strchr(opt, '=') != NULL) {
+	tokens = tofree = strdup(opts);
+	while ((token = strsep(&tokens, ",")) != NULL) {
+		if (strchr(token, '=') != NULL) {
 			if (sharename != NULL) {
-				printf("virtio-9p: more than one share name given\n");
-				return (1);
+				EPRINTLN(
+			    "virtio-9p: more than one share name given");
+				return (-1);
 			}
 
-			sharename = strsep(&opt, "=");
-			rootpath = opt;
-			continue;
-		}
+			sharename = strsep(&token, "=");
+			set_config_value_node(nvl, "sharename", sharename);
+			set_config_value_node(nvl, "path", token);
+		} else
+			set_config_bool_node(nvl, token, true);
+	}
+	free(tofree);
+	return (0);
+}
 
-		if (strcmp(opt, "ro") == 0) {
-			DPRINTF(("read-only mount requested\r\n"));
-			ro = true;
-			continue;
-		}
+static int
+pci_vt9p_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
+{
+	struct pci_vt9p_softc *sc;
+	const char *value;
+	const char *sharename;
+	int rootfd;
+	bool ro;
+	cap_rights_t rootcap;
+
+	ro = get_config_bool_node_default(nvl, "ro", false);
 
-		printf("virtio-9p: invalid option '%s'\n", opt);
+	value = get_config_value_node(nvl, "path");
+	if (value == NULL) {
+		EPRINTLN("virtio-9p: path required");
 		return (1);
 	}
+	rootfd = open(value, O_DIRECTORY);
+	if (rootfd < 0) {
+		EPRINTLN("virtio-9p: failed to open '%s': %s", value,
+		    strerror(errno));
+		return (-1);
+	}
 
+	sharename = get_config_value_node(nvl, "sharename");
+	if (sharename == NULL) {
+		EPRINTLN("virtio-9p: share name required");
+		return (1);
+	}
 	if (strlen(sharename) > VT9P_MAXTAGSZ) {
-		printf("virtio-9p: share name too long\n");
+		EPRINTLN("virtio-9p: share name too long");
 		return (1);
 	}
 
-	rootfd = open(rootpath, O_DIRECTORY);
-	if (rootfd < 0)
-		return (-1);
-
 	sc = calloc(1, sizeof(struct pci_vt9p_softc));
 	sc->vsc_config = calloc(1, sizeof(struct pci_vt9p_config) +
 	    VT9P_MAXTAGSZ);
@@ -337,6 +352,7 @@ pci_vt9p_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 
 struct pci_devemu pci_de_v9p = {
 	.pe_emu =	"virtio-9p",
+	.pe_legacy_config = pci_vt9p_legacy_config,
 	.pe_init =	pci_vt9p_init,
 	.pe_barwrite =	vi_pci_write,
 	.pe_barread =	vi_pci_read
diff --git a/usr.sbin/bhyve/pci_virtio_block.c b/usr.sbin/bhyve/pci_virtio_block.c
index 00258d1993b0..0dc58e49594b 100644
--- a/usr.sbin/bhyve/pci_virtio_block.c
+++ b/usr.sbin/bhyve/pci_virtio_block.c
@@ -54,6 +54,7 @@ __FBSDID("$FreeBSD$");
 #include <md5.h>
 
 #include "bhyverun.h"
+#include "config.h"
 #include "debug.h"
 #include "pci_emul.h"
 #include "virtio.h"
@@ -435,26 +436,22 @@ pci_vtblk_notify(void *vsc, struct vqueue_info *vq)
 }
 
 static int
-pci_vtblk_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_vtblk_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	char bident[sizeof("XX:X:X")];
 	struct blockif_ctxt *bctxt;
+	const char *path;
 	MD5_CTX mdctx;
 	u_char digest[16];
 	struct pci_vtblk_softc *sc;
 	off_t size;
 	int i, sectsz, sts, sto;
 
-	if (opts == NULL) {
-		WPRINTF(("virtio-block: backing device required"));
-		return (1);
-	}
-
 	/*
 	 * The supplied backing file has to exist
 	 */
 	snprintf(bident, sizeof(bident), "%d:%d", pi->pi_slot, pi->pi_func);
-	bctxt = blockif_open(opts, bident);
+	bctxt = blockif_open(nvl, bident);
 	if (bctxt == NULL) {
 		perror("Could not open backing file");
 		return (1);
@@ -491,8 +488,9 @@ pci_vtblk_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 	 * Create an identifier for the backing file. Use parts of the
 	 * md5 sum of the filename
 	 */
+	path = get_config_value_node(nvl, "path");
 	MD5Init(&mdctx);
-	MD5Update(&mdctx, opts, strlen(opts));
+	MD5Update(&mdctx, path, strlen(path));
 	MD5Final(digest, &mdctx);
 	snprintf(sc->vbsc_ident, VTBLK_BLK_ID_BYTES,
 	    "BHYVE-%02X%02X-%02X%02X-%02X%02X",
@@ -568,6 +566,7 @@ pci_vtblk_cfgread(void *vsc, int offset, int size, uint32_t *retval)
 struct pci_devemu pci_de_vblk = {
 	.pe_emu =	"virtio-blk",
 	.pe_init =	pci_vtblk_init,
+	.pe_legacy_config = blockif_legacy_config,
 	.pe_barwrite =	vi_pci_write,
 	.pe_barread =	vi_pci_read,
 #ifdef BHYVE_SNAPSHOT
diff --git a/usr.sbin/bhyve/pci_virtio_console.c b/usr.sbin/bhyve/pci_virtio_console.c
index 1911210fa046..88d6c37f582e 100644
--- a/usr.sbin/bhyve/pci_virtio_console.c
+++ b/usr.sbin/bhyve/pci_virtio_console.c
@@ -60,6 +60,7 @@ __FBSDID("$FreeBSD$");
 #include <sysexits.h>
 
 #include "bhyverun.h"
+#include "config.h"
 #include "debug.h"
 #include "pci_emul.h"
 #include "virtio.h"
@@ -128,7 +129,6 @@ struct pci_vtcon_softc {
 	uint64_t                 vsc_features;
 	char *                   vsc_rootdir;
 	int                      vsc_kq;
-	int                      vsc_nports;
 	bool                     vsc_ready;
 	struct pci_vtcon_port    vsc_control_port;
  	struct pci_vtcon_port    vsc_ports[VTCON_MAXPORTS];
@@ -242,18 +242,17 @@ pci_vtcon_port_to_vq(struct pci_vtcon_port *port, bool tx_queue)
 }
 
 static struct pci_vtcon_port *
-pci_vtcon_port_add(struct pci_vtcon_softc *sc, const char *name,
+pci_vtcon_port_add(struct pci_vtcon_softc *sc, int port_id, const char *name,
     pci_vtcon_cb_t *cb, void *arg)
 {
 	struct pci_vtcon_port *port;
 
-	if (sc->vsc_nports == VTCON_MAXPORTS) {
+	port = &sc->vsc_ports[port_id];
+	if (port->vsp_enabled) {
 		errno = EBUSY;
 		return (NULL);
 	}
-
-	port = &sc->vsc_ports[sc->vsc_nports++];
-	port->vsp_id = sc->vsc_nports - 1;
+	port->vsp_id = port_id;
 	port->vsp_sc = sc;
 	port->vsp_name = name;
 	port->vsp_cb = cb;
@@ -264,7 +263,7 @@ pci_vtcon_port_add(struct pci_vtcon_softc *sc, const char *name,
 		port->vsp_txq = 0;
 		port->vsp_rxq = 1;
 	} else {
-		port->vsp_txq = sc->vsc_nports * 2;
+		port->vsp_txq = (port_id + 1) * 2;
 		port->vsp_rxq = port->vsp_txq + 1;
 	}
 
@@ -273,17 +272,33 @@ pci_vtcon_port_add(struct pci_vtcon_softc *sc, const char *name,
 }
 
 static int
-pci_vtcon_sock_add(struct pci_vtcon_softc *sc, const char *name,
-    const char *path)
+pci_vtcon_sock_add(struct pci_vtcon_softc *sc, const char *port_name,
+    const nvlist_t *nvl)
 {
 	struct pci_vtcon_sock *sock;
 	struct sockaddr_un sun;
-	char *pathcopy;
+	const char *name, *path;
+	char *cp, *pathcopy;
+	long port;
 	int s = -1, fd = -1, error = 0;
 #ifndef WITHOUT_CAPSICUM
 	cap_rights_t rights;
 #endif
 
+	port = strtol(port_name, &cp, 0);
+	if (*cp != '\0' || port < 0 || port >= VTCON_MAXPORTS) {
+		EPRINTLN("vtcon: Invalid port %s", port_name);
+		error = -1;
+		goto out;
+	}
+
+	path = get_config_value_node(nvl, "path");
+	if (path == NULL) {
+		EPRINTLN("vtcon: required path missing for port %ld", port);
+		error = -1;
+		goto out;
+	}
+
 	sock = calloc(1, sizeof(struct pci_vtcon_sock));
 	if (sock == NULL) {
 		error = -1;
@@ -336,7 +351,13 @@ pci_vtcon_sock_add(struct pci_vtcon_softc *sc, const char *name,
 		errx(EX_OSERR, "Unable to apply rights for sandbox");
 #endif
 
-	sock->vss_port = pci_vtcon_port_add(sc, name, pci_vtcon_sock_tx, sock);
+	name = get_config_value_node(nvl, "name");
+	if (name == NULL) {
+		EPRINTLN("vtcon: required name missing for port %ld", port);
+		error = -1;
+		goto out;
+	}
+	sock->vss_port = pci_vtcon_port_add(sc, port, name, pci_vtcon_sock_tx, sock);
 	if (sock->vss_port == NULL) {
 		error = -1;
 		goto out;
@@ -498,13 +519,13 @@ pci_vtcon_control_tx(struct pci_vtcon_port *port, void *arg, struct iovec *iov,
 		break;
 
 	case VTCON_PORT_READY:
-		if (ctrl->id >= sc->vsc_nports) {
+		tmp = &sc->vsc_ports[ctrl->id];
+		if (ctrl->id >= VTCON_MAXPORTS || !tmp->vsp_enabled) {
 			WPRINTF(("VTCON_PORT_READY event for unknown port %d",
 			    ctrl->id));
 			return;
 		}
 
-		tmp = &sc->vsc_ports[ctrl->id];
 		if (tmp->vsp_console) {
 			resp.event = VTCON_CONSOLE_PORT;
 			resp.id = ctrl->id;
@@ -615,14 +636,61 @@ pci_vtcon_notify_rx(void *vsc, struct vqueue_info *vq)
 	}
 }
 
+/*
+ * Each console device has a "port" node which contains nodes for
+ * each port.  Ports are numbered starting at 0.
+ */
 static int
-pci_vtcon_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_vtcon_legacy_config_port(nvlist_t *nvl, int port, char *opt)
+{
+	char *name, *path;
+	char node_name[sizeof("XX")];
+	nvlist_t *port_nvl;
+
+	name = strsep(&opt, "=");
+	path = opt;
+	if (path == NULL) {
+		EPRINTLN("vtcon: port %s requires a path", name);
+		return (-1);
+	}
+	if (port >= VTCON_MAXPORTS) {
+		EPRINTLN("vtcon: too many ports");
+		return (-1);
+	}
+	snprintf(node_name, sizeof(node_name), "%d", port);
+	port_nvl = create_relative_config_node(nvl, node_name);
+	set_config_value_node(port_nvl, "name", name);
+	set_config_value_node(port_nvl, "path", path);
+	return (0);
+}
+
+static int
+pci_vtcon_legacy_config(nvlist_t *nvl, const char *opts)
+{
+	char *opt, *str, *tofree;
+	nvlist_t *ports_nvl;
+	int error, port;
+
+	ports_nvl = create_relative_config_node(nvl, "port");
+	tofree = str = strdup(opts);
+	error = 0;
+	port = 0;
+	while ((opt = strsep(&str, ",")) != NULL) {
+		error = pci_vtcon_legacy_config_port(ports_nvl, port, opt);
+		if (error)
+			break;
+		port++;
+	}
+	free(tofree);
+	return (error);
+}
+
+static int
+pci_vtcon_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	struct pci_vtcon_softc *sc;
-	char *portname = NULL;
-	char *portpath = NULL;
-	char *opt;
-	int i;	
+	nvlist_t *ports_nvl;
+	int i;
 
 	sc = calloc(1, sizeof(struct pci_vtcon_softc));
 	sc->vsc_config = calloc(1, sizeof(struct pci_vtcon_config));
@@ -658,15 +726,24 @@ pci_vtcon_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 	sc->vsc_control_port.vsp_cb = pci_vtcon_control_tx;
 	sc->vsc_control_port.vsp_enabled = true;
 
-	while ((opt = strsep(&opts, ",")) != NULL) {
-		portname = strsep(&opt, "=");
-		portpath = opt;
-
-		/* create port */
-		if (pci_vtcon_sock_add(sc, portname, portpath) < 0) {
-			EPRINTLN("cannot create port %s: %s",
-			    portname, strerror(errno));
-			return (1);
+	ports_nvl = find_relative_config_node(nvl, "port");
+	if (ports_nvl != NULL) {
+		const char *name;
+		void *cookie;
+		int type;
+
+		cookie = NULL;
+		while ((name = nvlist_next(ports_nvl, &type, &cookie)) !=
+		    NULL) {
+			if (type != NV_TYPE_NVLIST)
+				continue;
+
+			if (pci_vtcon_sock_add(sc, name,
+			    nvlist_get_nvlist(ports_nvl, name)) < 0) {
+				EPRINTLN("cannot create port %s: %s",
+				    name, strerror(errno));
+				return (1);
+			}
 		}
 	}
 
diff --git a/usr.sbin/bhyve/pci_virtio_net.c b/usr.sbin/bhyve/pci_virtio_net.c
index 1c8af094b6dc..0ea470a71b56 100644
--- a/usr.sbin/bhyve/pci_virtio_net.c
+++ b/usr.sbin/bhyve/pci_virtio_net.c
@@ -54,6 +54,7 @@ __FBSDID("$FreeBSD$");
 #include <pthread_np.h>
 
 #include "bhyverun.h"
+#include "config.h"
 #include "debug.h"
 #include "pci_emul.h"
 #include "mevent.h"
@@ -558,13 +559,13 @@ pci_vtnet_ping_ctlq(void *vsc, struct vqueue_info *vq)
 #endif
 
 static int
-pci_vtnet_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_vtnet_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	struct pci_vtnet_softc *sc;
+	const char *value;
 	char tname[MAXCOMLEN + 1];
-	int mac_provided;
-	int mtu_provided;
 	unsigned long mtu = ETHERMTU;
+	int err;
 
 	/*
 	 * Allocate data structures for further virtio initializations.
@@ -584,81 +585,46 @@ pci_vtnet_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 	sc->vsc_queues[VTNET_CTLQ].vq_qsize = VTNET_RINGSZ;
         sc->vsc_queues[VTNET_CTLQ].vq_notify = pci_vtnet_ping_ctlq;
 #endif
- 
-	/*
-	 * Attempt to open the backend device and read the MAC address
-	 * if specified.
-	 */
-	mac_provided = 0;
-	mtu_provided = 0;
-	if (opts != NULL) {
-		char *optscopy;
-		char *vtopts;
-		int err = 0;
 
-		/* Get the device name. */
-		optscopy = vtopts = strdup(opts);
-		(void) strsep(&vtopts, ",");
-
-		/*
-		 * Parse the list of options in the form
-		 *     key1=value1,...,keyN=valueN.
-		 */
-		while (vtopts != NULL) {
-			char *value = vtopts;
-			char *key;
-
-			key = strsep(&value, "=");
-			if (value == NULL)
-				break;
-			vtopts = value;
-			(void) strsep(&vtopts, ",");
-
-			if (strcmp(key, "mac") == 0) {
-				err = net_parsemac(value, sc->vsc_config.mac);
-				if (err)
-					break;
-				mac_provided = 1;
-			} else if (strcmp(key, "mtu") == 0) {
-				err = net_parsemtu(value, &mtu);
-				if (err)
-					break;
-
-				if (mtu < VTNET_MIN_MTU || mtu > VTNET_MAX_MTU) {
-					err = EINVAL;
-					errno = EINVAL;
-					break;
-				}
-				mtu_provided = 1;
-			}
+	value = get_config_value_node(nvl, "mac");
+	if (value != NULL) {
+		err = net_parsemac(value, sc->vsc_config.mac);
+		if (err) {
+			free(sc);
+			return (err);
 		}
+	} else
+		net_genmac(pi, sc->vsc_config.mac);
 
-		free(optscopy);
-
+	value = get_config_value_node(nvl, "mtu");
+	if (value != NULL) {
+		err = net_parsemtu(value, &mtu);
 		if (err) {
 			free(sc);
 			return (err);
 		}
 
-		err = netbe_init(&sc->vsc_be, opts, pci_vtnet_rx_callback,
-		          sc);
-
-		if (err) {
+		if (mtu < VTNET_MIN_MTU || mtu > VTNET_MAX_MTU) {
+			err = EINVAL;
+			errno = EINVAL;
 			free(sc);
 			return (err);
 		}
-		sc->vsc_consts.vc_hv_caps |= VIRTIO_NET_F_MRG_RXBUF |
-		    netbe_get_cap(sc->vsc_be);
+		sc->vsc_consts.vc_hv_caps |= VIRTIO_NET_F_MTU;
 	}
+	sc->vsc_config.mtu = mtu;
 
-	if (!mac_provided) {
-		net_genmac(pi, sc->vsc_config.mac);
+	/* Permit interfaces without a configured backend. */
+	if (get_config_value_node(nvl, "backend") != NULL) {
+		err = netbe_init(&sc->vsc_be, nvl, pci_vtnet_rx_callback, sc);
+		if (err) {
+			free(sc);
+			return (err);
+		}
 	}
 
-	sc->vsc_config.mtu = mtu;
-	if (mtu_provided) {
-		sc->vsc_consts.vc_hv_caps |= VIRTIO_NET_F_MTU;
-	}
+	sc->vsc_consts.vc_hv_caps |= VIRTIO_NET_F_MRG_RXBUF |
+	    netbe_get_cap(sc->vsc_be);
 
 	/* 
 	 * Since we do not actually support multiqueue,
@@ -673,8 +639,8 @@ pci_vtnet_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 	pci_set_cfgdata16(pi, PCIR_SUBDEV_0, VIRTIO_ID_NETWORK);
 	pci_set_cfgdata16(pi, PCIR_SUBVEND_0, VIRTIO_VENDOR);
 
-	/* Link is up if we managed to open backend device. */
-	sc->vsc_config.status = (opts == NULL || sc->vsc_be);
+	/* Link is always up. */
+	sc->vsc_config.status = 1;
 	
 	vi_softc_linkup(&sc->vsc_vs, &sc->vsc_consts, sc, pi, sc->vsc_queues);
 	sc->vsc_vs.vs_mtx = &sc->vsc_mtx;
@@ -839,6 +805,7 @@ done:
 static struct pci_devemu pci_de_vnet = {
 	.pe_emu = 	"virtio-net",
 	.pe_init =	pci_vtnet_init,
+	.pe_legacy_config = netbe_legacy_config,
 	.pe_barwrite =	vi_pci_write,
 	.pe_barread =	vi_pci_read,
 #ifdef BHYVE_SNAPSHOT
diff --git a/usr.sbin/bhyve/pci_virtio_rnd.c b/usr.sbin/bhyve/pci_virtio_rnd.c
index 42feb7a2c4e0..d51301b32534 100644
--- a/usr.sbin/bhyve/pci_virtio_rnd.c
+++ b/usr.sbin/bhyve/pci_virtio_rnd.c
@@ -143,7 +143,7 @@ pci_vtrnd_notify(void *vsc, struct vqueue_info *vq)
 
 
 static int
-pci_vtrnd_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_vtrnd_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	struct pci_vtrnd_softc *sc;
 	int fd;
diff --git a/usr.sbin/bhyve/pci_virtio_scsi.c b/usr.sbin/bhyve/pci_virtio_scsi.c
index 1dd7b6ebcf0b..8edf64a11361 100644
--- a/usr.sbin/bhyve/pci_virtio_scsi.c
+++ b/usr.sbin/bhyve/pci_virtio_scsi.c
@@ -61,6 +61,7 @@ __FBSDID("$FreeBSD$");
 #include <camlib.h>
 
 #include "bhyverun.h"
+#include "config.h"
 #include "debug.h"
 #include "pci_emul.h"
 #include "virtio.h"
@@ -244,7 +245,7 @@ static void pci_vtscsi_eventq_notify(void *, struct vqueue_info *);
 static void pci_vtscsi_requestq_notify(void *, struct vqueue_info *);
 static int  pci_vtscsi_init_queue(struct pci_vtscsi_softc *,
     struct pci_vtscsi_queue *, int);
-static int pci_vtscsi_init(struct vmctx *, struct pci_devinst *, char *);
+static int pci_vtscsi_init(struct vmctx *, struct pci_devinst *, nvlist_t *);
 
 static struct virtio_consts vtscsi_vi_consts = {
 	"vtscsi",				/* our name */
@@ -665,32 +666,36 @@ pci_vtscsi_init_queue(struct pci_vtscsi_softc *sc,
 }
 
 static int
-pci_vtscsi_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_vtscsi_legacy_config(nvlist_t *nvl, const char *opts)
+{
+	char *cp, *devname;
+
+	cp = strchr(opts, ',');
+	if (cp == NULL) {
+		set_config_value_node(nvl, "dev", opts);
+		return (0);
+	}
+	devname = strndup(opts, cp - opts);
+	set_config_value_node(nvl, "dev", devname);
+	free(devname);
+	return (pci_parse_legacy_config(nvl, cp + 1));
+}
+
+static int
+pci_vtscsi_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	struct pci_vtscsi_softc *sc;
-	char *opt, *optname;
-	const char *devname;
-	int i, optidx = 0;
+	const char *devname, *value;;
+	int i;
 
 	sc = calloc(1, sizeof(struct pci_vtscsi_softc));
-	devname = "/dev/cam/ctl";
-	while ((opt = strsep(&opts, ",")) != NULL) {
-		optname = strsep(&opt, "=");
-		if (opt == NULL && optidx == 0) {
-			if (optname[0] != 0)
-				devname = optname;
-		} else if (strcmp(optname, "dev") == 0 && opt != NULL) {
-			devname = opt;
-		} else if (strcmp(optname, "iid") == 0 && opt != NULL) {
-			sc->vss_iid = strtoul(opt, NULL, 10);
-		} else {
-			EPRINTLN("Invalid option %s", optname);
-			free(sc);
-			return (1);
-		}
-		optidx++;
-	}
+	value = get_config_value_node(nvl, "iid");
+	if (value != NULL)
+		sc->vss_iid = strtoul(value, NULL, 10);
 
+	devname = get_config_value_node(nvl, "dev");
+	if (devname == NULL)
+		devname = "/dev/cam/ctl";
 	sc->vss_ctl_fd = open(devname, O_RDWR);
 	if (sc->vss_ctl_fd < 0) {
 		WPRINTF(("cannot open %s: %s", devname, strerror(errno)));
diff --git a/usr.sbin/bhyve/pci_xhci.c b/usr.sbin/bhyve/pci_xhci.c
index 9227b934f046..dd6dfc39d014 100644
--- a/usr.sbin/bhyve/pci_xhci.c
+++ b/usr.sbin/bhyve/pci_xhci.c
@@ -56,6 +56,7 @@ __FBSDID("$FreeBSD$");
 #include <xhcireg.h>
 
 #include "bhyverun.h"
+#include "config.h"
 #include "debug.h"
 #include "pci_emul.h"
 #include "pci_xhci.h"
@@ -276,7 +277,6 @@ struct pci_xhci_softc {
 	struct pci_xhci_portregs *portregs;
 	struct pci_xhci_dev_emu  **devices; /* XHCI[port] = device */
 	struct pci_xhci_dev_emu  **slots;   /* slots assigned from 1 */
-	int		ndevices;
 
 	int		usb2_port_start;
 	int		usb3_port_start;
@@ -570,7 +570,8 @@ pci_xhci_get_dev_ctx(struct pci_xhci_softc *sc, uint32_t slot)
 	uint64_t devctx_addr;
 	struct xhci_dev_ctx *devctx;
 
-	assert(slot > 0 && slot <= sc->ndevices);
+	assert(slot > 0 && slot <= XHCI_MAX_DEVS);
+	assert(XHCI_SLOTDEV_PTR(sc, slot) != NULL);
 	assert(sc->opregs.dcbaa_p != NULL);
 
 	devctx_addr = sc->opregs.dcbaa_p->dcba[slot];
@@ -843,7 +844,7 @@ pci_xhci_cmd_disable_slot(struct pci_xhci_softc *sc, uint32_t slot)
 	if (sc->portregs == NULL)
 		goto done;
 
-	if (slot > sc->ndevices) {
+	if (slot > XHCI_MAX_SLOTS) {
 		cmderr = XHCI_TRB_ERROR_SLOT_NOT_ON;
 		goto done;
 	}
@@ -857,7 +858,8 @@ pci_xhci_cmd_disable_slot(struct pci_xhci_softc *sc, uint32_t slot)
 			cmderr = XHCI_TRB_ERROR_SUCCESS;
 			/* TODO: reset events and endpoints */
 		}
-	}
+	} else
+		cmderr = XHCI_TRB_ERROR_SLOT_NOT_ON;
 
 done:
 	return (cmderr);
@@ -1906,7 +1908,7 @@ pci_xhci_device_doorbell(struct pci_xhci_softc *sc, uint32_t slot,
 	DPRINTF(("pci_xhci doorbell slot %u epid %u stream %u",
 	    slot, epid, streamid));
 
-	if (slot == 0 || slot > sc->ndevices) {
+	if (slot == 0 || slot > XHCI_MAX_SLOTS) {
 		DPRINTF(("pci_xhci: invalid doorbell slot %u", slot));
 		return;
 	}
@@ -2643,67 +2645,129 @@ pci_xhci_dev_event(struct usb_hci *hci, enum hci_usbev evid, void *param)
 	return (0);
 }
 
+/*
+ * Each controller contains a "slot" node which contains a list of
+ * child nodes each of which is a device.  Each slot node's name
+ * corresponds to a specific controller slot.  These nodes
+ * contain a "device" variable identifying the device model of the
+ * USB device.  For example:
+ *
+ * pci.0.1.0
+ *          .device="xhci"
+ *          .slot
+ *               .1
+ *                 .device="tablet"
+ */
+static int
+pci_xhci_legacy_config(nvlist_t *nvl, const char *opts)
+{
+	char node_name[16];
+	nvlist_t *slots_nvl, *slot_nvl;
+	char *cp, *opt, *str, *tofree;
+	int slot;
 
+	if (opts == NULL)
+		return (0);
 
-static void
-pci_xhci_device_usage(char *opt)
-{
+	slots_nvl = create_relative_config_node(nvl, "slot");
+	slot = 1;
+	tofree = str = strdup(opts);
+	while ((opt = strsep(&str, ",")) != NULL) {
+		/* device[=<config>] */
+		cp = strchr(opt, '=');
+		if (cp != NULL) {
+			*cp = '\0';
+			cp++;
+		}
+
+		snprintf(node_name, sizeof(node_name), "%d", slot);
+		slot++;
+		slot_nvl = create_relative_config_node(slots_nvl, node_name);
+		set_config_value_node(slot_nvl, "device", opt);
 
-	EPRINTLN("Invalid USB emulation \"%s\"", opt);
+		/*
+		 * NB: Given that we split on commas above, the legacy
+		 * format only supports a single option.
+		 */
+		if (cp != NULL && *cp != '\0')
+			pci_parse_legacy_config(slot_nvl, cp);
+	}
+	free(tofree);
+	return (0);
 }
 
 static int
-pci_xhci_parse_opts(struct pci_xhci_softc *sc, char *opts)
+pci_xhci_parse_devices(struct pci_xhci_softc *sc, nvlist_t *nvl)
 {
-	struct pci_xhci_dev_emu	**devices;
 	struct pci_xhci_dev_emu	*dev;
 	struct usb_devemu	*ue;
-	void	*devsc;
-	char	*uopt, *xopts, *config;
-	int	usb3_port, usb2_port, i;
+	const nvlist_t *slots_nvl, *slot_nvl;
+	const char *name, *device;
+	char	*cp;
+	void	*devsc, *cookie;
+	long	slot;
+	int	type, usb3_port, usb2_port, i, ndevices;
 
-	uopt = NULL;
-	usb3_port = sc->usb3_port_start - 1;
-	usb2_port = sc->usb2_port_start - 1;
-	devices = NULL;
+	usb3_port = sc->usb3_port_start;
+	usb2_port = sc->usb2_port_start;
 
-	if (opts == NULL)
-		goto portsfinal;
+	sc->devices = calloc(XHCI_MAX_DEVS, sizeof(struct pci_xhci_dev_emu *));
+	sc->slots = calloc(XHCI_MAX_SLOTS, sizeof(struct pci_xhci_dev_emu *));
 
-	devices = calloc(XHCI_MAX_DEVS, sizeof(struct pci_xhci_dev_emu *));
+	/* port and slot numbering start from 1 */
+	sc->devices--;
+	sc->slots--;
 
-	sc->slots = calloc(XHCI_MAX_SLOTS, sizeof(struct pci_xhci_dev_emu *));
-	sc->devices = devices;
-	sc->ndevices = 0;
-
-	uopt = strdup(opts);
-	for (xopts = strtok(uopt, ",");
-	     xopts != NULL;
-	     xopts = strtok(NULL, ",")) {
-		if (usb2_port == ((sc->usb2_port_start-1) + XHCI_MAX_DEVS/2) ||
-		    usb3_port == ((sc->usb3_port_start-1) + XHCI_MAX_DEVS/2)) {
+	ndevices = 0;
+
+	slots_nvl = find_relative_config_node(nvl, "slots");
+	if (slots_nvl == NULL)
+		goto portsfinal;
+
+	cookie = NULL;
+	while ((name = nvlist_next(slots_nvl, &type, &cookie)) != NULL) {
+		if (usb2_port == ((sc->usb2_port_start) + XHCI_MAX_DEVS/2) ||
+		    usb3_port == ((sc->usb3_port_start) + XHCI_MAX_DEVS/2)) {
 			WPRINTF(("pci_xhci max number of USB 2 or 3 "
 			     "devices reached, max %d", XHCI_MAX_DEVS/2));
-			usb2_port = usb3_port = -1;
-			goto done;
+			goto bad;
 		}
 
-		/* device[=<config>] */
-		if ((config = strchr(xopts, '=')) == NULL)
-			config = "";		/* no config */
-		else
-			*config++ = '\0';
+		if (type != NV_TYPE_NVLIST) {
+			EPRINTLN(
+			    "pci_xhci: config variable '%s' under slot node",
+			     name);
+			goto bad;
+		}
+
+		slot = strtol(name, &cp, 0);
+		if (*cp != '\0' || slot <= 0 || slot > XHCI_MAX_SLOTS) {
+			EPRINTLN("pci_xhci: invalid slot '%s'", name);
+			goto bad;
+		}
+
+		if (XHCI_SLOTDEV_PTR(sc, slot) != NULL) {
+			EPRINTLN("pci_xhci: duplicate slot '%s'", name);
+			goto bad;
+		}
 
-		ue = usb_emu_finddev(xopts);
+		slot_nvl = nvlist_get_nvlist(slots_nvl, name);
+		device = get_config_value_node(slot_nvl, "device");
+		if (device == NULL) {
+			EPRINTLN(
+			    "pci_xhci: missing \"device\" value for slot '%s'",
+				name);
+			goto bad;
+		}
+
+		ue = usb_emu_finddev(device);
 		if (ue == NULL) {
-			pci_xhci_device_usage(xopts);
-			DPRINTF(("pci_xhci device not found %s", xopts));
-			usb2_port = usb3_port = -1;
-			goto done;
+			EPRINTLN("pci_xhci: unknown device model \"%s\"",
+			    device);
+			goto bad;
 		}
 
-		DPRINTF(("pci_xhci adding device %s, opts \"%s\"",
-		        xopts, config));
+		DPRINTF(("pci_xhci adding device %s", device));
 
 		dev = calloc(1, sizeof(struct pci_xhci_dev_emu));
 		dev->xsc = sc;
@@ -2712,66 +2776,64 @@ pci_xhci_parse_opts(struct pci_xhci_softc *sc, char *opts)
 		dev->hci.hci_event = pci_xhci_dev_event;
 
 		if (ue->ue_usbver == 2) {
-			dev->hci.hci_port = usb2_port + 1;
-			devices[usb2_port] = dev;
+			if (usb2_port == sc->usb2_port_start +
+			    XHCI_MAX_DEVS / 2) {
+				WPRINTF(("pci_xhci max number of USB 2 devices "
+				     "reached, max %d", XHCI_MAX_DEVS / 2));
+				goto bad;
+			}
+			dev->hci.hci_port = usb2_port;
 			usb2_port++;
 		} else {
-			dev->hci.hci_port = usb3_port + 1;
-			devices[usb3_port] = dev;
+			if (usb3_port == sc->usb3_port_start +
+			    XHCI_MAX_DEVS / 2) {
+				WPRINTF(("pci_xhci max number of USB 3 devices "
+				     "reached, max %d", XHCI_MAX_DEVS / 2));
+				goto bad;
+			}
+			dev->hci.hci_port = usb3_port;
 			usb3_port++;
 		}
+		XHCI_DEVINST_PTR(sc, dev->hci.hci_port) = dev;
 
 		dev->hci.hci_address = 0;
-		devsc = ue->ue_init(&dev->hci, config);
+		devsc = ue->ue_init(&dev->hci, nvl);
 		if (devsc == NULL) {
-			pci_xhci_device_usage(xopts);
-			usb2_port = usb3_port = -1;
-			goto done;
+			goto bad;
 		}
 
 		dev->dev_ue = ue;
 		dev->dev_sc = devsc;
 
-		/* assign slot number to device */
-		sc->slots[sc->ndevices] = dev;
-
-		sc->ndevices++;
+		XHCI_SLOTDEV_PTR(sc, slot) = dev;
 	}
 
 portsfinal:
 	sc->portregs = calloc(XHCI_MAX_DEVS, sizeof(struct pci_xhci_portregs));
+	sc->portregs--;
 
-	if (sc->ndevices > 0) {
-		/* port and slot numbering start from 1 */
-		sc->devices--;
-		sc->portregs--;
-		sc->slots--;
-
+	if (ndevices > 0) {
 		for (i = 1; i <= XHCI_MAX_DEVS; i++) {
 			pci_xhci_init_port(sc, i);
 		}
 	} else {
 		WPRINTF(("pci_xhci no USB devices configured"));
-		sc->ndevices = 1;
 	}
+	return (0);
 
-done:
-	if (devices != NULL) {
-		if (usb2_port <= 0 && usb3_port <= 0) {
-			sc->devices = NULL;
-			for (i = 0; devices[i] != NULL; i++)
-				free(devices[i]);
-			sc->ndevices = -1;
-
-			free(devices);
-		}
+bad:
+	for (i = 1; i <= XHCI_MAX_DEVS; i++) {
+		free(XHCI_DEVINST_PTR(sc, i));
 	}
-	free(uopt);
-	return (sc->ndevices);
+
+	free(sc->devices + 1);
+	free(sc->slots + 1);
+
+	return (-1);
 }
 
 static int
-pci_xhci_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
+pci_xhci_init(struct vmctx *ctx, struct pci_devinst *pi, nvlist_t *nvl)
 {
 	struct pci_xhci_softc *sc;
 	int	error;
@@ -2790,7 +2852,7 @@ pci_xhci_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts)
 	sc->usb3_port_start = 1;
 
 	/* discover devices */
-	error = pci_xhci_parse_opts(sc, opts);
+	error = pci_xhci_parse_devices(sc, nvl);
 	if (error < 0)
 		goto done;
 	else
@@ -3109,7 +3171,6 @@ pci_xhci_snapshot(struct vm_snapshot_meta *meta)
 		SNAPSHOT_VAR_OR_LEAVE(dev->hci.hci_port, meta, ret, done);
 	}
 
-	SNAPSHOT_VAR_OR_LEAVE(sc->ndevices, meta, ret, done);
 	SNAPSHOT_VAR_OR_LEAVE(sc->usb2_port_start, meta, ret, done);
 	SNAPSHOT_VAR_OR_LEAVE(sc->usb3_port_start, meta, ret, done);
 
@@ -3121,6 +3182,7 @@ done:
 struct pci_devemu pci_de_xhci = {
 	.pe_emu =	"xhci",
 	.pe_init =	pci_xhci_init,
+	.pe_legacy_config = pci_xhci_legacy_config,
 	.pe_barwrite =	pci_xhci_write,
 	.pe_barread =	pci_xhci_read,
 #ifdef BHYVE_SNAPSHOT
diff --git a/usr.sbin/bhyve/pctestdev.c b/usr.sbin/bhyve/pctestdev.c
index be445e5c75ae..49e03b2a4343 100644
--- a/usr.sbin/bhyve/pctestdev.c
+++ b/usr.sbin/bhyve/pctestdev.c
@@ -89,15 +89,6 @@ pctestdev_getname(void)
 }
 
 int
-pctestdev_parse(const char *opts)
-{
-	if (opts != NULL && *opts != '\0')
-		return (-1);
-
-	return (0);
-}
-
-int
 pctestdev_init(struct vmctx *ctx)
 {
 	struct mem_range iomem;
diff --git a/usr.sbin/bhyve/pctestdev.h b/usr.sbin/bhyve/pctestdev.h
index c1c940146e05..5808abe6e356 100644
--- a/usr.sbin/bhyve/pctestdev.h
+++ b/usr.sbin/bhyve/pctestdev.h
@@ -38,6 +38,5 @@ struct vmctx;
 
 const char	*pctestdev_getname(void);
 int		 pctestdev_init(struct vmctx *ctx);
-int		 pctestdev_parse(const char *opts);
 
 #endif
diff --git a/usr.sbin/bhyve/rtc.c b/usr.sbin/bhyve/rtc.c
index 09ca3f61ae79..0f63156adb9b 100644
--- a/usr.sbin/bhyve/rtc.c
+++ b/usr.sbin/bhyve/rtc.c
@@ -40,6 +40,7 @@ __FBSDID("$FreeBSD$");
 #include <vmmapi.h>
 
 #include "acpi.h"
+#include "config.h"
 #include "pci_lpc.h"
 #include "rtc.h"
 
@@ -59,13 +60,13 @@ __FBSDID("$FreeBSD$");
  * Returns the current RTC time as number of seconds since 00:00:00 Jan 1, 1970
  */
 static time_t
-rtc_time(struct vmctx *ctx, int use_localtime)
+rtc_time(struct vmctx *ctx)
 {
 	struct tm tm;
 	time_t t;
 
 	time(&t);
-	if (use_localtime) {
+	if (get_config_bool_default("rtc.use_localtime", true)) {
 		localtime_r(&t, &tm);
 		t = timegm(&tm);
 	}
@@ -73,7 +74,7 @@ rtc_time(struct vmctx *ctx, int use_localtime)
 }
 
 void
-rtc_init(struct vmctx *ctx, int use_localtime)
+rtc_init(struct vmctx *ctx)
 {	
 	size_t himem;
 	size_t lomem;
@@ -101,7 +102,7 @@ rtc_init(struct vmctx *ctx, int use_localtime)
 	err = vm_rtc_write(ctx, RTC_HMEM_MSB, himem >> 16);
 	assert(err == 0);
 
-	err = vm_rtc_settime(ctx, rtc_time(ctx, use_localtime));
+	err = vm_rtc_settime(ctx, rtc_time(ctx));
 	assert(err == 0);
 }
 
diff --git a/usr.sbin/bhyve/rtc.h b/usr.sbin/bhyve/rtc.h
index 1c108eed9969..c8b3572e93c6 100644
--- a/usr.sbin/bhyve/rtc.h
+++ b/usr.sbin/bhyve/rtc.h
@@ -31,6 +31,6 @@
 #ifndef _RTC_H_
 #define _RTC_H_
 
-void	rtc_init(struct vmctx *ctx, int use_localtime);
+void	rtc_init(struct vmctx *ctx);
 
 #endif /* _RTC_H_ */
diff --git a/usr.sbin/bhyve/smbiostbl.c b/usr.sbin/bhyve/smbiostbl.c
index d9f8a649c710..716fb48178c1 100644
--- a/usr.sbin/bhyve/smbiostbl.c
+++ b/usr.sbin/bhyve/smbiostbl.c
@@ -43,6 +43,7 @@ __FBSDID("$FreeBSD$");
 #include <vmmapi.h>
 
 #include "bhyverun.h"
+#include "config.h"
 #include "debug.h"
 #include "smbiostbl.h"
 
@@ -588,11 +589,13 @@ smbios_type1_initializer(struct smbios_structure *template_entry,
     uint16_t *n, uint16_t *size)
 {
 	struct smbios_table_type1 *type1;
+	const char *guest_uuid_str;
 
 	smbios_generic_initializer(template_entry, template_strings,
 	    curaddr, endaddr, n, size);
 	type1 = (struct smbios_table_type1 *)curaddr;
 
+	guest_uuid_str = get_config_value("uuid");
 	if (guest_uuid_str != NULL) {
 		uuid_t		uuid;
 		uint32_t	status;
@@ -606,6 +609,7 @@ smbios_type1_initializer(struct smbios_structure *template_entry,
 		MD5_CTX		mdctx;
 		u_char		digest[16];
 		char		hostname[MAXHOSTNAMELEN];
+		const char	*vmname;
 
 		/*
 		 * Universally unique and yet reproducible are an
@@ -616,6 +620,7 @@ smbios_type1_initializer(struct smbios_structure *template_entry,
 			return (-1);
 
 		MD5Init(&mdctx);
+		vmname = get_config_value("name");
 		MD5Update(&mdctx, vmname, strlen(vmname));
 		MD5Update(&mdctx, hostname, sizeof(hostname));
 		MD5Final(digest, &mdctx);
diff --git a/usr.sbin/bhyve/uart_emul.c b/usr.sbin/bhyve/uart_emul.c
index 1b55231b2bf5..fd0ad2c846f4 100644
--- a/usr.sbin/bhyve/uart_emul.c
+++ b/usr.sbin/bhyve/uart_emul.c
@@ -679,7 +679,7 @@ uart_stdio_backend(struct uart_softc *sc)
 }
 
 static int
-uart_tty_backend(struct uart_softc *sc, const char *opts)
+uart_tty_backend(struct uart_softc *sc, const char *path)
 {
 #ifndef WITHOUT_CAPSICUM
 	cap_rights_t rights;
@@ -687,7 +687,7 @@ uart_tty_backend(struct uart_softc *sc, const char *opts)
 #endif
 	int fd;
 
-	fd = open(opts, O_RDWR | O_NONBLOCK);
+	fd = open(path, O_RDWR | O_NONBLOCK);
 	if (fd < 0)
 		return (-1);
 
@@ -711,17 +711,17 @@ uart_tty_backend(struct uart_softc *sc, const char *opts)
 }
 
 int
-uart_set_backend(struct uart_softc *sc, const char *opts)
+uart_set_backend(struct uart_softc *sc, const char *device)
 {
 	int retval;
 
-	if (opts == NULL)
+	if (device == NULL)
 		return (0);
 
-	if (strcmp("stdio", opts) == 0)
+	if (strcmp("stdio", device) == 0)
 		retval = uart_stdio_backend(sc);
 	else
-		retval = uart_tty_backend(sc, opts);
+		retval = uart_tty_backend(sc, device);
 	if (retval == 0)
 		uart_opentty(sc);
 
diff --git a/usr.sbin/bhyve/uart_emul.h b/usr.sbin/bhyve/uart_emul.h
index 5a53294da89e..a66701c18580 100644
--- a/usr.sbin/bhyve/uart_emul.h
+++ b/usr.sbin/bhyve/uart_emul.h
@@ -43,7 +43,7 @@ struct uart_softc *uart_init(uart_intr_func_t intr_assert,
 int	uart_legacy_alloc(int unit, int *ioaddr, int *irq);
 uint8_t	uart_read(struct uart_softc *sc, int offset);
 void	uart_write(struct uart_softc *sc, int offset, uint8_t value);
-int	uart_set_backend(struct uart_softc *sc, const char *opt);
+int	uart_set_backend(struct uart_softc *sc, const char *device);
 #ifdef BHYVE_SNAPSHOT
 int	uart_snapshot(struct uart_softc *sc, struct vm_snapshot_meta *meta);
 #endif
diff --git a/usr.sbin/bhyve/usb_emul.c b/usr.sbin/bhyve/usb_emul.c
index 6ecdd9530eb1..d97d7b38d2f8 100644
--- a/usr.sbin/bhyve/usb_emul.c
+++ b/usr.sbin/bhyve/usb_emul.c
@@ -43,7 +43,7 @@ __FBSDID("$FreeBSD$");
 SET_DECLARE(usb_emu_set, struct usb_devemu);
 
 struct usb_devemu *
-usb_emu_finddev(char *name)
+usb_emu_finddev(const char *name)
 {
 	struct usb_devemu **udpp, *udp;
 
diff --git a/usr.sbin/bhyve/usb_emul.h b/usr.sbin/bhyve/usb_emul.h
index d6e1e616cd82..67f4f3ff08f1 100644
--- a/usr.sbin/bhyve/usb_emul.h
+++ b/usr.sbin/bhyve/usb_emul.h
@@ -31,6 +31,7 @@
 #ifndef _USB_EMUL_H_
 #define _USB_EMUL_H_
 
+#include <sys/nv.h>
 #include <stdlib.h>
 #include <sys/linker_set.h>
 #include <pthread.h>
@@ -53,7 +54,7 @@ struct usb_devemu {
 	int	ue_usbspeed;	/* usb device speed */
 
 	/* instance creation */
-	void	*(*ue_init)(struct usb_hci *hci, char *opt);
+	void	*(*ue_init)(struct usb_hci *hci, nvlist_t *nvl);
 
 	/* handlers */
 	int	(*ue_request)(void *sc, struct usb_data_xfer *xfer);
@@ -149,7 +150,7 @@ enum USB_ERRCODE {
 			pthread_mutex_unlock(&((x)->mtx));		\
 		} while (0)
 
-struct usb_devemu *usb_emu_finddev(char *name);
+struct usb_devemu *usb_emu_finddev(const char *name);
 
 struct usb_data_xfer_block *usb_data_xfer_append(struct usb_data_xfer *xfer,
                           void *buf, int blen, void *hci_data, int ccs);
diff --git a/usr.sbin/bhyve/usb_mouse.c b/usr.sbin/bhyve/usb_mouse.c
index 3b7fc29ff249..eda3878ff5fb 100644
--- a/usr.sbin/bhyve/usb_mouse.c
+++ b/usr.sbin/bhyve/usb_mouse.c
@@ -241,8 +241,6 @@ struct umouse_bos_desc umouse_bosd = {
 struct umouse_softc {
 	struct usb_hci *hci;
 
-	char	*opt;
-
 	struct umouse_report um_report;
 	int	newdata;
 	struct {
@@ -299,7 +297,7 @@ umouse_event(uint8_t button, int x, int y, void *arg)
 }
 
 static void *
-umouse_init(struct usb_hci *hci, char *opt)
+umouse_init(struct usb_hci *hci, nvlist_t *nvl)
 {
 	struct umouse_softc *sc;
 
@@ -307,7 +305,6 @@ umouse_init(struct usb_hci *hci, char *opt)
 	sc->hci = hci;
 
 	sc->hid.protocol = 1;	/* REPORT protocol */
-	sc->opt = strdup(opt);
 	pthread_mutex_init(&sc->mtx, NULL);
 	pthread_mutex_init(&sc->ev_mtx, NULL);