src/lib/libfetch, branch releng/14.2 FreeBSD source tree libfetch: don't include fragments in HTTP requests 2024-09-05T14:05:15+00:00 Pietro Cerutti gahr@FreeBSD.org 2024-08-21T12:35:27+00:00 ab7a79806e3103accb1ba2d89ba51e977704a2e3 Fragments are reserved for client-side processing, see https://www.rfc-editor.org/rfc/rfc9110.html#section-7.1 Also, some servers don't like to receive HTTP requests with fragments. ``` $ fetch 'https://dropbox.com/a/b' fetch: https://dropbox.com/a/b: Not Found $ fetch 'https://dropbox.com/a/b#' fetch: https://dropbox.com/a/b#: Bad Request ``` This is a real-world scenario, where some download link from dropbox (eventually) redirects to an URL with a fragment: ``` $ fetch -v 'https://www.dropbox.com/sh/<some>/<thing>?dl=1' 2>&1 | grep requesting requesting https://www.dropbox.com/sh/<some>/<thing>?dl=1 requesting https://www.dropbox.com/scl/fo/<foo>/<bar>?rlkey=<baz>&dl=1 requesting https://<boo>.dl.dropboxusercontent.com/zip_download_get/<some-long-strig># ``` See how the last redirect ends with a `#`. Currently, libfetch includes the ending fragment and makes it impossible to download the file. Differential Revision: https://reviews.freebsd.org/D46318 MFC after: 2 weeks (cherry picked from commit 1af7d5f389536a2f391153513d95d92ffdf360e4) 
Fragments are reserved for client-side processing, see
https://www.rfc-editor.org/rfc/rfc9110.html#section-7.1

Also, some servers don't like to receive HTTP requests with fragments.

```
$ fetch 'https://dropbox.com/a/b'
fetch: https://dropbox.com/a/b: Not Found

$ fetch 'https://dropbox.com/a/b#'
fetch: https://dropbox.com/a/b#: Bad Request
```

This is a real-world scenario, where some download link from dropbox
(eventually) redirects to an URL with a fragment:

```
$ fetch -v 'https://www.dropbox.com/sh/<some>/<thing>?dl=1' 2>&1 | grep requesting
requesting https://www.dropbox.com/sh/<some>/<thing>?dl=1
requesting https://www.dropbox.com/scl/fo/<foo>/<bar>?rlkey=<baz>&dl=1
requesting https://<boo>.dl.dropboxusercontent.com/zip_download_get/<some-long-strig>#
```

See how the last redirect ends with a `#`.

Currently, libfetch includes the ending fragment and makes it impossible
to download the file.

Differential Revision:	https://reviews.freebsd.org/D46318
MFC after:		2 weeks

(cherry picked from commit 1af7d5f389536a2f391153513d95d92ffdf360e4)
libfetch: parse scheme://domain:/ correctly 2024-04-03T19:11:58+00:00 Ka Ho Ng khng@FreeBSD.org 2024-03-25T20:10:42+00:00 72c3d91294c4d9c46c8185f90cba522f539d1051 This improves URL-parsing compability with cURL, and unbreaks parsing of similar kinds of URLs after commit 8d9de5b10a24. Sponsored by: Juniper Networks, Inc. Reviewed by: des MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D44493 (cherry picked from commit fb860ed0c52c2c1e7792ef86718620a439663c7f) 
This improves URL-parsing compability with cURL, and unbreaks parsing of
similar kinds of URLs after commit 8d9de5b10a24.

Sponsored by:	Juniper Networks, Inc.
Reviewed by:	des
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D44493

(cherry picked from commit fb860ed0c52c2c1e7792ef86718620a439663c7f)
libfetch, fetch: Stop recommending the use of ca_root_nss. 2023-12-13T16:23:57+00:00 Dag-Erling Smørgrav des@FreeBSD.org 2023-10-08T04:35:15+00:00 6d0ea82cc2520726e49d9f03e1b67734608139e7 MFC after: 3 days Reviewed by: kevans, emaste Differential Revision: https://reviews.freebsd.org/D42119 (cherry picked from commit 2821a7498f65d357c68166e1978b491abef1ca4a) 
MFC after:	3 days
Reviewed by:	kevans, emaste
Differential Revision:	https://reviews.freebsd.org/D42119

(cherry picked from commit 2821a7498f65d357c68166e1978b491abef1ca4a)
libfetch: don't rely on ca_root_nss for certificate validation 2023-10-05T00:03:16+00:00 Michael Osipov michael.osipov@siemens.com 2023-10-03T05:53:20+00:00 fb058a9a40a5adc82721ed822fb4fba213446a7b Before certctl(8), there was no system trust store, and libfetch relied on the CA certificate bundle from the ca_root_nss port to verify peers. We now have a system trust store and a reliable mechanism for manipulating it (to explicitly add, remove, or revoke certificates), but if ca_root_nss is installed, libfetch will still prefer that to the system trust store. With this change, unless explicitly overridden, libfetch will rely on OpenSSL to pick up the default system trust store. PR: 256902 MFC after: 3 days Reviewed by: kevans Differential Revision: https://reviews.freebsd.org/D42059 (cherry picked from commit 09f5c1e118bb4eca77b83a0d08f559b20f60aa59) 
Before certctl(8), there was no system trust store, and libfetch
relied on the CA certificate bundle from the ca_root_nss port to
verify peers.

We now have a system trust store and a reliable mechanism for
manipulating it (to explicitly add, remove, or revoke certificates),
but if ca_root_nss is installed, libfetch will still prefer that to
the system trust store.

With this change, unless explicitly overridden, libfetch will rely on
OpenSSL to pick up the default system trust store.

PR:		256902
MFC after:	3 days
Reviewed by:	kevans
Differential Revision:	https://reviews.freebsd.org/D42059

(cherry picked from commit 09f5c1e118bb4eca77b83a0d08f559b20f60aa59)
Remove $FreeBSD$: two-line nroff pattern 2023-08-16T17:55:10+00:00 Warner Losh imp@FreeBSD.org 2023-08-16T17:55:10+00:00 fa9896e082a1046ff4fbc75fcba4d18d1f2efc19 Remove /^\.\\"\n\.\\"\s*\$FreeBSD\$$\n/ 
Remove /^\.\\"\n\.\\"\s*\$FreeBSD\$$\n/
Remove $FreeBSD$: one-line sh pattern 2023-08-16T17:55:03+00:00 Warner Losh imp@FreeBSD.org 2023-08-16T17:55:03+00:00 d0b2dbfa0ecf2bbc9709efc5e20baf8e4b44bbbf Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/ 
Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/
Remove $FreeBSD$: one-line .c pattern 2023-08-16T17:54:42+00:00 Warner Losh imp@FreeBSD.org 2023-08-16T17:54:42+00:00 1d386b48a555f61cb7325543adbbb5c3f3407a66 Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/ 
Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/
Remove $FreeBSD$: two-line .h pattern 2023-08-16T17:54:16+00:00 Warner Losh imp@FreeBSD.org 2023-08-16T17:54:16+00:00 b3e7694832e81d7a904a10f525f8797b753bf0d3 Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/ 
Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/
libfetch: remove all old OpenSSL support 2023-06-24T08:45:02+00:00 Enji Cooper ngie@FreeBSD.org 2023-06-22T03:53:54+00:00 bc1027a7785166fde9c2a3b48e6e70d198377d4b This change removes pre-OpenSSL 1.1 supporting code and removes/adjusted preprocessor conditionals which were tautilogically true as FreeBSD main has shipped with OpenSSL 1.1+ for some time. Reviewed by: emaste Differential Revision: https://reviews.freebsd.org/D40711 
This change removes pre-OpenSSL 1.1 supporting code and removes/adjusted
preprocessor conditionals which were tautilogically true as FreeBSD main
has shipped with OpenSSL 1.1+ for some time.

Reviewed by:	emaste
Differential Revision:	https://reviews.freebsd.org/D40711
libfetch: specify OpenSSL 1.1 APIs 2023-05-25T17:15:45+00:00 Pierre Pronchery pierre@freebsdfoundation.org 2023-05-25T06:46:02+00:00 77d788e23d0964053b81b5de307fa04bd1ccadc5 OPENSSL_API_COMPAT can be used to specify the OpenSSL API version in use for the purpose of hiding deprecated interfaces and enabling the appropriate deprecation notices. This change is a NFC while we're still using OpenSSL 1.1.1 but will avoid deprecation warnings upon the switch to OpenSSL 3.0. Future work should migrate to use the OpenSSL 3.0 APIs. PR: 271615 Reviewed by: emaste Event: Kitchener-Waterloo Hackathon 202305 Sponsored by: The FreeBSD Foundation 
OPENSSL_API_COMPAT can be used to specify the OpenSSL API version in
use for the purpose of hiding deprecated interfaces and enabling
the appropriate deprecation notices.

This change is a NFC while we're still using OpenSSL 1.1.1 but will
avoid deprecation warnings upon the switch to OpenSSL 3.0.

Future work should migrate to use the OpenSSL 3.0 APIs.

PR:		271615
Reviewed by:	emaste
Event:		Kitchener-Waterloo Hackathon 202305
Sponsored by:	The FreeBSD Foundation