\r360505-r360506

r360420:
lualoader config: don't call loader.getenv() as much

We don't actually need to fetch loader_conf_files as much as we do; we've
already fetched it once at the beginning, we only really need to fetch it
again after each file we've processed. If it changes, then we can stash that
off into our local prefiles.

While here, drop a note about the recursion so that I stop trying to
change it. It may very well make redundant some of the work we're doing, but
that's OK.

r360421:
lualoader: config: start exporting readConfFiles

In the process, change it slightly: readConfFiles will take a string like
loader_conf_files in addition to the loaded_files table that it normally
takes. This is to facilitate the addition of a read-conf CLI command, which
will just pass in the single file to read and an empty table.

r360422:
lualoader: cli: add read-conf

This is a straightforward match to the command used by many in forthloader;
it uses the newly-exported config.readConfFiles() to make sure that any
loader_conf_files gets done as appropriate.

r360423:
lualoader: cli: clobber loader_conf_files before proceeding

This makes sure that config.readConfFiles doesn't see a stale
loader_conf_files from before, in case the newly loaded file doesn't set it.

r360425:
config.lua(8): "may should" is not proper grammar

r360427:
config.lua(8): catch up to recently added hooks

While we're here, let's stylize these as functions instead of just raw text.
A future change may allow arbitrary data arguments to be passed some of
these, and the distinction is useful.

r360486:
loader.conf(5): document that loader_conf_files may be clobbered

A future change in lualoader may take some liberties with the
loader_conf_files in the name of efficiency; namely, it may start omitting
it from the loader environment entirely so that it doesn't need to worry
about maintaining any specific value.

This variable has historically been incredibly volatile anyways, as it may
get set to completely different values in any given configuration file to
trigger a load of more files.

Document now that we may not maintain it in the future, but perhaps we'll
reserve the right to change our minds and eventually formally export all of
the loader configuration files that were read using this variable.

r360505:
lualoader: config: add a table for restricted environment vars

This new table should be used for transient values that don't need to end up
in the loader environment. Generally, these will be things that are internal
details that really aren't needed or interesting outside of the config
module (e.g. if we changed how ${module}_* directives work, they might use
this instead).

To start, populate it with loader_conf_files. Any specific value of
loader_conf_files isn't all that interesting; if we're going to export it,
we should really instead export a loader_conf_files that indicates all of
the configuration files we processed. This will be used to reduce
bookkeeping overhead in a future commit that cleans up readConfFiles.

r360506:
lualoader: config: improve readConfFiles, rename to readConf

The previous interface was pretty bad, and required the caller to get some
implementation details correct that it really shouldn't need to (e.g.
loader_conf_files handling) and pass in an empty table for it to use.

The new and much improved interface, readConf, is much less of a hack;
hiding these implementation details and just doing the right thing.
config.lua will now use it to process /boot/defaults/loader.conf and the
subsequent loader_conf_files from there, and read-conf will also use it.

This improvement submitted by Olivier (cited below), loader_conf_files
handling from the original patch was changed to just clobber it before
processing and not bother restoring it after the fact following r360505
where it's now guaranteed to evade the loader environment.

PR:		244640

loader: Add a "kernel.loaded" hook

This hook can be useful, for example to run a local function to choose
different modules to load when a user has picked a different kernel
from the menu.

Reviewed by:    kevans
Approved by:    mav (mentor)
Sponsored by:   iXsystems, Inc.
Differential Revision:  https://reviews.freebsd.org/D24115

r357103:
loader.lua: re-arrange to load local.lua *after* config loading

The major problem with the current ordering is that loader.conf may contain
all of the magic we need to actually setup the console, so loading local.lua
prior to that can make it excessively difficult and annoying to debug
(whoops, sorry Ravi & Warner).

The new ordering has some implications, but I suspect they are a non-issue.
The first is that it's no longer possible for the local module to inject any
logic prior to loading config -- I suspect no one has relied on this. The
second implication is that the config.loaded hook is now useless, as the
local module will always be included after that hook would have fired.

For config.loaded, I will opt to leave it in, just in case we add an early
point for local lua to get injected or in case one wants to schedule some
deferred logic in a custom loader.lua. The overhead of having it if no hooks
will be invoked is relatively minimal.

r357104:
lua: add modules.loaded hook

This may be used for the local module to hook in and load any additional
modules that it wants, since it can't modify the modules table internal to
config. We may consider adding API to do so at a later time, but I suspect
it will be more complicated to use with little return.

status is captured but ignored for the purpose of loading the hook. status
will be false if *any* module failed to load, but we typically don't let
that halt the boot so there's no reason to let it halt hooks. Some vendors
or setups may have expected fails that would be actively thwarted by
checking it.

We may, at a later date, consider adding an API for letting non-config
modules check which modules have successfully (or not) loaded in case an
unexpected failure *should* halt whatever they are doing.

r344243:
lualoader: only clear the screen before first password prompt

This was previously an unconditional screen clear, regardless of whether or
not we would be prompting for any passwords. This is pointless, given that
the screen clear is only there to put our screen into a consistent state
before we draw the prompts and do cursor manipulation.

This is also the only screen clear besides that to draw the menu.  One can
now see early pre-loader and loader output with the menu disabled, which may
be useful for diagnostics.

r345517:
lualoader: Clear the screen before prompting for password

Assuming that the autoboot sequence was interrupted, we've done enough
cursor manipulation that the prompt for the password will be sufficiently
obscured a couple of lines up. Clear the screen and reset the cursor
position here, too.

r345518:
lualoader: Fix up some luacheck concerns

- Garbage collect an unused (removed because it was useless) constant
- Don't bother with vararg notation if args will not be used

r339677:
lualoader: unload upon kernel change if a kernel was previously loaded

In the majority of cases, a kernel is not loaded before we hit the menu.
However, if a password is set, we'll trigger autoboot and have loadelf'd
beforehand. We also need to take into account one dropping to the loader
prompt and twiddling with things manually; if they try to toggle through
kernels, we'll assume they mean it.

r339678:
menu.lua: Abort autoboot sequence on failed command

Currently, a timeout in the menu autoboot sequence would effectively do
nothing. We would return from the autoboot handling, then begin processing
the menu without redrawing it.

This change makes the behavior a little more friendly. Returning the user to
the menu can't have any good effects, so abort the autoboot sequence and
drop to the loader prompt.

r339702:
lualoader: Improve module loading diagnostics

Some fixes:

- Maintain historical behavior more accurately w.r.t verbose_loading;
  verbose_loading strictly prints "${module_name...}" and later "failed!"
  or "ok" based on load success
- With or without verbose_loading, dump command_errbuf on load failure.
  This usually happens prior to ok/failed if we're verbose_loading

r339805:
lualoader: Always return a proper dictionary for blacklist

If module_blacklist isn't specified, we have an empty blacklist; effectively
the same as if module_blacklist="" were specified in loader.conf(5).

This was reported when switching to a BE that predated the module_blacklist
introduction, but the problem is valid all the same and likely to be tripped
over in other scenarios.

Approved by:	re (kib)

This is a step in the process of easing migration into the new world order
of DRM drivers. Strongly encourage users towards loading DRM modules via
rc.conf(5) instead of loader.conf(5) by failing the load from loader(8).
Users so inclined may wipe out the blacklist via module_blacklist="" in
loader.conf(5), and it is expected that these modules will eventually be
removed from the blacklist. They may still be loaded as dependencies of
other modules or explicitly via the loader prompt, but this should not be a
major problem.

Approved by:	re (rgrimes)
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D16914

Earlier changes setup a config.module_path variable that was populated upon
reading of loader.conf(5) and used for restoring module_path to pristine
condition if multiple kernels are attempted. This broke the ability to
override module_path at the loader prompt in case of emergency.

Approved by:	re (rgrimes)

While loader.conf(5) suggests that all values should be quoted, reality is
that this was never strictly enforced and it is used. We already make some
concession to this in number values, which aren't strictly quoted either.

The compromise here is that multi-word values must be quoted. This lets
things like `foo_load=YES` work, while denying more complex expressions on
the right hand side. This likely catches the vast majority of current usage.

A bit of a kludge is needed to accomplish this since Lua regex doesn't
support branching. I had considered splitting up expressions and generating
the right-hand side of the expressions completely in config.parse, but
deemed this too large of an overhaul to take given the current timing. This
should be re-worked shortly after the thaw.

Reported by:	royger

lualoader was not respecting the 'xen_kernel' environment variable, which
hints to the interpreter that it should load a Xen kernel prior to loading
any other kernel that might be specified. If a Xen kernel is specified and
we fail to load it, we should not proceed to boot.

Reported by:	royger
Tested by:	royger

This includes some light rework to simplify the line parsing, as well.  If
we hit a line match, we'll always either use the line and move on to the
next line, or we'll spew out malformed line errors.

We had multiple spots to output the error and set the status based on
whether we had a non-nil first capture group or failed EOL validation, but
it was always the same error.  Light rework entails a small label jump to
skip error handling and elimination of 'found' local.