From 0aa19eef44c7302ef3282592d3c5c3c86102c103 Mon Sep 17 00:00:00 2001 From: Francis Lavoie Date: Thu, 7 Apr 2022 13:43:39 -0400 Subject: [PATCH] Elaborate on log formats, proxy headers and examples (#225) * Elaborate on log formats, proxy headers and examples * Document HTTP transport resolvers --- src/docs/markdown/architecture.md | 4 +- src/docs/markdown/caddyfile/directives/log.md | 42 ++++++++++++++----- .../caddyfile/directives/reverse_proxy.md | 36 ++++++++++++++-- src/docs/markdown/caddyfile/options.md | 2 +- 4 files changed, 67 insertions(+), 17 deletions(-) diff --git a/src/docs/markdown/architecture.md b/src/docs/markdown/architecture.md index 7cfc316..b2cea99 100644 --- a/src/docs/markdown/architecture.md +++ b/src/docs/markdown/architecture.md @@ -53,7 +53,7 @@ The core of Caddy knows how to work with some of these fields natively: - [`admin`](/docs/json/admin/) so it can set up the [admin API](/docs/api) and manage the process - [`logging`](/docs/json/logging/) so it can [emit logs](/docs/logging) -But other top-level fields (like [`apps`](/docs/json/apps/)) are opaque to the core of Caddy. In fact, all Caddy knows how to do with the bytes in `apps` is deserialize them into an interface type that it can call two methods on: +But other top-level fields (like [`apps`](/docs/json/apps/)) are opaque to the core of Caddy. In fact, all Caddy knows to do with the bytes in `apps` is deserialize them into an interface type that it can call two methods on: 1. `Start()` 2. `Stop()` @@ -97,7 +97,7 @@ Since any properties from the JSON encoding will already have been decoded, only You can get a sense for this by [traversing Caddy's JSON structure in our docs](/docs/json/). Anywhere you see `{•••}` is where guest modules may be used; and as you click into one, you can continue exploring all the way down until there are no more guest modules. -Other common provisioning tasks are setting up internal values that will be used during the module's lifetime, or standardizing inputs. For example, the [http.matchers.remote_ip](/docs/modules/http.matchers.remote_ip) module uses the provisioning phase to parse CIDR values out of the string inputs it received from the JSON. That way, it doesn't have to do this during every HTTP request, and is more efficient as a result. +Other common provisioning tasks are setting up internal values that will be used during the module's lifetime, or standardizing inputs. For example, the [`http.matchers.remote_ip`](/docs/modules/http.matchers.remote_ip) module uses the provisioning phase to parse CIDR values out of the string inputs it received from the JSON. That way, it doesn't have to do this during every HTTP request, and is more efficient as a result. Validation also can take place in the provision phase. If a module's resulting config is invalid, an error can be returned here which aborts the entire config load process. diff --git a/src/docs/markdown/caddyfile/directives/log.md b/src/docs/markdown/caddyfile/directives/log.md index 1b54785..cd67725 100644 --- a/src/docs/markdown/caddyfile/directives/log.md +++ b/src/docs/markdown/caddyfile/directives/log.md @@ -137,15 +137,16 @@ In addition to the syntax for each individual encoder, these common properties c ```caddy-d format { - message_key - level_key - time_key - name_key - caller_key - stacktrace_key - line_ending - time_format - level_format + message_key + level_key + time_key + name_key + caller_key + stacktrace_key + line_ending + time_format + duration_format + level_format } ``` @@ -156,8 +157,27 @@ format { - **caller_key** The key for the caller field of the log entry. - **stacktrace_key** The key for the stacktrace field of the log entry. - **line_ending** The line endings to use. -- **time_format** The format for timestamps. -- **level_format** The format for levels. +- **time_format** The format for timestamps. May be one of: + - **unix_seconds_float** Floating-point number of seconds since the Unix epoch; this is the default. + - **unix_milli_float** Floating-point number of milliseconds since the Unix epoch. + - **unix_nano** Integer number of nanoseconds since the Unix epoch. + - **iso8601** Example: `2006-01-02T15:04:05.000Z0700` + - **rfc3339** Example: `2006-01-02T15:04:05Z07:00` + - **rfc3339_nano** Example: `2006-01-02T15:04:05.999999999Z07:00` + - **wall** Example: `2006/01/02 15:04:05` + - **wall_milli** Example: `2006/01/02 15:04:05.000` + - **wall_nano** Example: `2006/01/02 15:04:05.000000000` + - **common_log** Example: `02/Jan/2006:15:04:05 -0700` + - Or, any compatible time layout string; see the [Go documentation](https://pkg.go.dev/time#pkg-constants) for full details. +- **duration_format** The format for durations. May be one of: + - **seconds** Floating-point number of seconds elapsed; this is the default. + - **nano** Integer number of nanoseconds elapsed. + - **string** Using Go's built-in string format, for example `1m32.05s` or `6.31ms`. +- **level_format** The format for levels. May be one of: + - **lower** Lowercase; this is the default. + - **upper** Uppercase. + - **color** Uppercase, with console colors. + #### console diff --git a/src/docs/markdown/caddyfile/directives/reverse_proxy.md b/src/docs/markdown/caddyfile/directives/reverse_proxy.md index 5d4254c..d7c116e 100644 --- a/src/docs/markdown/caddyfile/directives/reverse_proxy.md +++ b/src/docs/markdown/caddyfile/directives/reverse_proxy.md @@ -262,14 +262,42 @@ The proxy **buffers responses** by default for wire efficiency: The proxy can **manipulate headers** between itself and the backend: -- **header_up** Sets, adds, removes, or performs a replacement in a request header going upstream to the backend. -- **header_down** Sets, adds, removes, or performs a replacement in a response header coming downstream from the backend. +- **header_up** Sets, adds (with the `+` prefix), removes (with the `-` prefix), or performs a replacement (by using two arguments, a search and replacement) in a request header going upstream to the backend. +- **header_down** Sets, adds (with the `+` prefix), removes (with the `-` prefix), or performs a replacement (by using two arguments, a search and replacement) in a response header coming downstream from the backend. + +For example, to set a request header, overwriting any existing values: + +```caddy-d +header_up Some-Header "the value" +``` + +To add a response header; note that there can be multiple values for a header field: + +```caddy-d +header_down +Some-Header "first value" +header_down +Some-Header "second value" +``` + +To remove a request header, preventing it from reaching the backend: + +```caddy-d +header_up -Some-Header +``` + +To perform a regular expression replacement on a request header: + +```caddy-d +header_up Some-Header "^prefix-([A-Za-z0-9]*)$" "replaced-$1-suffix" +``` + +The regular expression language used is RE2, included in Go. See the [RE2 syntax reference](https://github.com/google/re2/wiki/Syntax) and the [Go regexp syntax overview](https://pkg.go.dev/regexp/syntax). The replacement string is [expanded](https://pkg.go.dev/regexp#Regexp.Expand), allowing use of captured values, for example `$1` being the first capture group. + #### Defaults By default, Caddy passes thru incoming headers—including `Host`—to the backend without modifications, with three exceptions: -- It adds or augments the [X-Forwarded-For](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For) header field. +- It sets or augments the [X-Forwarded-For](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For) header field. - It sets the [X-Forwarded-Proto](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Proto) header field. - It sets the [X-Forwarded-Host](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Host) header field. @@ -308,6 +336,7 @@ transport http { dial_fallback_delay response_header_timeout expect_continue_timeout + resolvers tls tls_client_auth | tls_insecure_skip_verify @@ -331,6 +360,7 @@ transport http { - **dial_fallback_delay** is how long to wait before spawning an RFC 6555 Fast Fallback connection. A negative value disables this. Accepts [duration values](/docs/conventions#durations). Default: `300ms`. - **response_header_timeout** is how long to wait for reading response headers from the upstream. Accepts [duration values](/docs/conventions#durations). Default: No timeout. - **expect_continue_timeout** is how long to wait for the upstreams's first response headers after fully writing the request headers if the request has the header `Expect: 100-continue`. Accepts [duration values](/docs/conventions#durations). Default: No timeout. +- **resolvers** is a list of DNS resolvers to override system resolvers. - **tls** uses HTTPS with the backend. This will be enabled automatically if you specify backends using the `https://` scheme or port `:443`. - **tls_client_auth** enables TLS client authentication one of two ways: (1) by specifying a domain name for which Caddy should obtain a certificate and keep it renewed, or (2) by specifying a certificate and key file to present for TLS client authentication with the backend. - **tls_insecure_skip_verify** turns off security. _Do not use in production._ diff --git a/src/docs/markdown/caddyfile/options.md b/src/docs/markdown/caddyfile/options.md index 3f3606a..8ba17bd 100644 --- a/src/docs/markdown/caddyfile/options.md +++ b/src/docs/markdown/caddyfile/options.md @@ -373,7 +373,7 @@ If you wish to _not_ have these headers redacted, you may enable the `log_creden - **strict_sni_host** require that a request's `Host` header matches the value of the ServerName sent by the client's TLS ClientHello; often a necessary safeguard when using TLS client authentication. If there's a mismatch, an HTTP status `421 Misdirected Request` response is written to the client. - This option will be implicitly turned on if [client authentication](/docs/caddyfile/directives/tls#client_auth) is configured. This disallow TLS client auth bypass (domain fronting) which could otherwise be exploited by sending an unprotected SNI value during a TLS handshake, then putting a protected domain in the Host header after establishing connection. This is a safe default, but you may explicitly turn it off with `insecure_off`, for example in the case of running a proxy where domain fronting is desired and access is not restricted based on hostname. + This option will be implicitly turned on if [client authentication](/docs/caddyfile/directives/tls#client_auth) is configured. This disallows TLS client auth bypass (domain fronting) which could otherwise be exploited by sending an unprotected SNI value during a TLS handshake, then putting a protected domain in the Host header after establishing connection. This is a safe default, but you may explicitly turn it off with `insecure_off`, for example in the case of running a proxy where domain fronting is desired and access is not restricted based on hostname.