src/lib/libfetch/fetch.c, branch main

libfetch: don't include fragments in HTTP requests

2024-08-21T12:35:27Z

Summary: Fragments are reserved for client-side processing, see https://www.rfc-editor.org/rfc/rfc9110.html#section-7.1 Also, some servers don't like to receive HTTP requests with fragments. ``` $ fetch 'https://dropbox.com/a/b' fetch: https://dropbox.com/a/b: Not Found $ fetch 'https://dropbox.com/a/b#' fetch: https://dropbox.com/a/b#: Bad Request ``` This is a real-world scenario, where some download link from dropbox (eventually) redirects to an URL with a fragment: ``` $ fetch -v 'https://www.dropbox.com/sh//?dl=1' 2>&1 | grep requesting requesting https://www.dropbox.com/sh//?dl=1 requesting https://www.dropbox.com/scl/fo//?rlkey=&dl=1 requesting https://.dl.dropboxusercontent.com/zip_download_get/# ``` See how the last redirect ends with a `#`. Currently, libfetch includes the ending fragment and makes it impossible to download the file. Differential Revision: https://reviews.freebsd.org/D46318 MFC after: 2 weeks

libfetch: parse IPv4address in IPv6address properly

2024-07-11T19:48:28Z

This fix parsing problems of IPv6 addresses which contains IPv4 addresses. One example is ::192.168.0.1. Documents are in below: https://datatracker.ietf.org/doc/html/rfc2373#page-22 Sponsored by: Juniper Networks, Inc. MFC after: 1 week Reviewed by: jhibbits Differential Revision: https://reviews.freebsd.org/D45896

libfetch: parse scheme://domain:/ correctly

2024-03-25T20:11:42Z

This improves URL-parsing compability with cURL, and unbreaks parsing of similar kinds of URLs after commit 8d9de5b10a24. Sponsored by: Juniper Networks, Inc. Reviewed by: des MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D44493

lib: Automated cleanup of cdefs and other formatting

2023-11-27T05:23:59Z

Apply the following automated changes to try to eliminate no-longer-needed sys/cdefs.h includes as well as now-empty blank lines in a row. Remove /^#if.*\n#endif.*\n#include\s+.*\n/ Remove /\n+#include\s+.*\n+#if.*\n#endif.*\n+/ Remove /\n+#if.*\n#endif.*\n+/ Remove /^#if.*\n#endif.*\n/ Remove /\n+#include\s+\n#include\s+/ Remove /\n+#include\s+\n#include\s+/ Remove /\n+#include\s+\n#include\s+/ Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

2023-08-16T17:54:42Z

Remove /^[\s*]*__FBSDID$"\$FreeBSD\$"$;?\s*\n/

libfetch: disallow invalid escape sequences

2020-02-05T16:55:00Z

Per RFC1738 escape is "% hex hex"; other sequences do not form a valid URL. Suggested by: Matthew Dillon Reviewed by: Matthew Dillon MFC after: 1 week

Fix urldecode buffer overrun.

2020-01-28T18:37:18Z

Reported by: Duncan Overbruck Security: CVE-2020-7450

Improve URL parsing. In particular, convert scheme and host to lowercase.

2018-11-27T10:45:14Z

MFC after: 1 week

Fix an inverted conditional in the netrc code, which would ignore the

2018-05-29T13:07:36Z

value of $HOME and always use the home directory from the passwd database, unless $HOME was unset, in which case it would use (null). While there, clean up handling of netrcfd and add debugging aids. MFC after: 3 weeks

Use __VA_ARGS__ to simplify the DEBUG macro.

2018-05-29T10:28:20Z

MFC after: 3 weeks