system(3): Fix null case

Our manual page states that if given a null pointer, system() returns
non-zero if the shell is available and zero if it is not.  This is
consistent with the C standard's description of system(), but it is not
what we actually do.  What we actually do is always return non-zero, as
required by POSIX.

As the POSIX rationale explains, implementing the logic required by the
C standard does not violate POSIX, since a conforming system always has
a shell, therefore the logic will always return non-zero.

Since our libc is commonly used in non-conforming situations such as
chroots or thin jails, we should implement the full logic required by
the C standard.

MFC after:	1 week
Sponsored by:	Klara, Inc.
Reviewed by:	obiwac, bnovkov, kevans
Differential Revision:	https://reviews.freebsd.org/D55484

2 files changed, 11 insertions, 1 deletions

diff --git a/lib/libc/stdlib/system.c b/lib/libc/stdlib/system.c
index 17dd63eb52f9..94f7460c9b68 100644
--- a/lib/libc/stdlib/system.c
+++ b/lib/libc/stdlib/system.c
@@ -64,7 +64,7 @@ __libc_system(const char *command)
 	pid_t pid;
 
 	if (command == NULL)			/* just checking... */
-		return (1);
+		return (eaccess(_PATH_BSHELL, X_OK) == 0);
 
 	/*
 	 * If we are the first concurrent instance, ignore SIGINT and
diff --git a/lib/libc/tests/stdlib/system_test.c b/lib/libc/tests/stdlib/system_test.c
index 713e4bb0f87a..b6a31583d52d 100644
--- a/lib/libc/tests/stdlib/system_test.c
+++ b/lib/libc/tests/stdlib/system_test.c
@@ -57,10 +57,20 @@ ATF_TC(system_null);
 ATF_TC_HEAD(system_null, tc)
 {
 	atf_tc_set_md_var(tc, "descr", "system(NULL)");
+	atf_tc_set_md_var(tc, "require.user", "root");
 }
 ATF_TC_BODY(system_null, tc)
 {
+	/* First, test in a normal environment */
 	ATF_REQUIRE_EQ(1, system(NULL));
+
+	/* Now enter an empty chroot */
+	ATF_REQUIRE_EQ(0, chroot("."));
+	ATF_REQUIRE_EQ(0, chdir("/"));
+
+	/* Test again with no shell available */
+	ATF_REQUIRE_EQ(0, system(NULL));
+	ATF_REQUIRE_EQ(W_EXITCODE(127, 0), system("true"));
 }
 
 /*