JSON Template Layout :: Apache Log4j (original) (raw)

Templates are JSON documents, where objects containing $resolver members, such as,

{
  "$resolver": "message", (1)
  "stringified": true (2)
}

are interpreted by the JSON Template Layout compiler, and replaced with the referenced event or stack trace template resolver rendering that particular item.

Templates are configured by means of the following JSON Template Layout plugin attributes:

• eventTemplate and eventTemplateUri (for encoding LogEvents)
• stackTraceElementTemplate and stackTraceElementTemplateUri (for encoding StackStraceElements)
• EventTemplateAdditionalField (for extending the event template)

Event templates

eventTemplate and eventTemplateUri describe the JSON structure JSON Template Layout uses to encode LogEvents. JSON Template Layout contains the following predefined event templates:

EcsLayout.json

The default event template modellingthe Elastic Common Schema (ECS) specification

LogstashJsonEventLayoutV1.json

Models Logstash json_event pattern for Log4j

GelfLayout.json

Models the Graylog Extended Log Format (GELF) payload specification with additional _thread and _logger fields.

| | If used, it is advised to override the obligatory host field with a user provided constant via additional event template fields to avoid hostName property lookup at runtime, which incurs an extra cost. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

GcpLayout.json

Models the structure described by Google Cloud Platform structured logging with additional _thread, _logger and _exception fields. The exception trace, if any, is written to the _exception field as well as the message field – the former is useful for explicitly searching/analyzing structured exception information, while the latter is Google’s expected place for the exception, and integrates with Google Error Reporting.

JsonLayout.json

Models the exact structure generated by the deprecated JsonLayout with the exception of thrown field.JsonLayout serializes the Throwable as is via Jackson ObjectMapper, whereas JsonLayout.json event template employs the StackTraceElementLayout.json stack trace template for stack traces to generate a document-store-friendly flat structure.

| | This event template is only meant for existing users of the deprecated JsonLayout to migrate to JSON Template Layout without much trouble, and other than that purpose, is not recommended to be used! | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

Event template resolvers

Event template resolvers consume a LogEvent and render a certain property of it at the point of the JSON where they are declared. For instance, marker resolver renders the marker of the event, level resolver renders the level, and so on. An event template resolver is denoted with a special object containing a $resolver key:

Example event template demonstrating the usage of level resolver

{
  "version": "1.0",
  "level": {
    "$resolver": "level",
    "field": "name"
  }
}

Here version field will be rendered as is, while level field will be populated by the level resolver. That is, this template will generate JSON similar to the following:

Example JSON generated from the demonstrated event template

{
  "version": "1.0",
  "level": "INFO"
}

The complete list of available event template resolvers are provided below in detail.

counter

Resolves a number from an internal counter

counter event template resolver grammar

config      = [ start ] , [ overflowing ] , [ stringified ]
start       = "start" -> number
overflowing = "overflowing" -> boolean
stringified = "stringified" -> boolean

Unless provided, start and overflowing are respectively set to 0 (zero) and true by default.

When overflowing is set to true, the internal counter is created using a long, which is subject to overflow while incrementing, though faster and garbage-free. Otherwise, a BigInteger is used, which does not overflow, but incurs allocation costs.

When stringified is enabled, which is set to false by default, the resolved number will be converted to a string.

See examples

Resolves a sequence of numbers starting from 0. Once Long.MAX_VALUE is reached, counter overflows to Long.MIN_VALUE.

{
  "$resolver": "counter"
}

Resolves a sequence of numbers starting from 1000. Once Long.MAX_VALUE is reached, counter overflows to Long.MIN_VALUE.

{
  "$resolver": "counter",
  "start": 1000
}

Resolves a sequence of numbers starting from 0 and keeps on doing as long as JVM heap allows.

{
  "$resolver": "counter",
  "overflowing": false
}

caseConverter

Converts the case of string values

caseConverter event template resolver grammar

config                = case , input , [ locale ] , [ errorHandlingStrategy ]
input                 = JSON
case                  = "case" -> ( "upper" | "lower" )
locale                = "locale" -> (
                            language                                   |
                          ( language , "_" , country )                 |
                          ( language , "_" , country , "_" , variant )
                        )
errorHandlingStrategy = "errorHandlingStrategy" -> (
                          "fail"    |
                          "pass"    |
                          "replace"
                        )
replacement           = "replacement" -> JSON

input can be any available template value; e.g., a JSON literal, a lookup string, an object pointing to another resolver.

Unless provided, locale points to the one returned by JsonTemplateLayoutDefaults.getLocale(), which is configured by the log4j.layout.jsonTemplate.locale system property and by default set to the default system locale.

errorHandlingStrategy determines the behavior when either the input doesn’t resolve to a string value or case conversion throws an exception:

fail

Propagates the failure

pass

Causes the resolved value to be passed as is

replace

Suppresses the failure and replaces it with the replacement, which is set to null by default

errorHandlingStrategy is set to replace by default.

Most of the time JSON logs are persisted to a storage solution (e.g., Elasticsearch) that keeps a statically-typed index on fields. Hence, if a field is always expected to be of type string, using non-string replacements or pass in errorHandlingStrategy might result in type incompatibility issues at the storage level.

| | Unless the input value is passed intact or replaced, case conversion is not garbage-free. | | -------------------------------------------------------------------------------------------- |

See examples

Convert the resolved log level strings to upper-case:

{
  "$resolver": "caseConverter",
  "case": "upper",
  "input": {
    "$resolver": "level",
    "field": "name"
  }
}

Convert the resolved USER environment variable to lower-case using nl_NL locale:

{
  "$resolver": "caseConverter",
  "case": "lower",
  "locale": "nl_NL",
  "input": "${env:USER}"
}

Convert the resolved sessionId thread context data (MDC) to lower-case:

{
  "$resolver": "caseConverter",
  "case": "lower",
  "input": {
    "$resolver": "mdc",
    "key": "sessionId"
  }
}

Above, if sessionId MDC resolves to a, say, number, case conversion will fail. Since errorHandlingStrategy is set to replace and replacement is set to null by default, the resolved value will be null. One can suppress this behavior and let the resolved sessionId number be left as is:

{
  "$resolver": "caseConverter",
  "case": "lower",
  "input": {
    "$resolver": "mdc",
    "key": "sessionId"
  },
  "errorHandlingStrategy": "pass"
}

or replace it with a custom string:

{
  "$resolver": "caseConverter",
  "case": "lower",
  "input": {
    "$resolver": "mdc",
    "key": "sessionId"
  },
  "errorHandlingStrategy": "replace",
  "replacement": "unknown"
}

endOfBatch

Resolves LogEvent#isEndOfBatch() boolean flag

exception

Resolves fields of the Throwable returned by LogEvent#getThrown()

exception event template resolver grammar

config              = field , [ stringified ] , [ stackTrace ]
field               = "field" -> ( "className" | "message" | "stackTrace" )

stackTrace          = "stackTrace" -> (
                        [ stringified ]
                      , [ elementTemplate ]
                      )

stringified         = "stringified" -> ( boolean | truncation )
truncation          = "truncation" -> (
                        [ suffix ]
                      , [ pointMatcherStrings ]
                      , [ pointMatcherRegexes ]
                      )
suffix              = "suffix" -> string
pointMatcherStrings = "pointMatcherStrings" -> string[]
pointMatcherRegexes = "pointMatcherRegexes" -> string[]

elementTemplate     = "elementTemplate" -> object

stringified is set to false by default.stringified at the root level is deprecated in favor of stackTrace.stringified, which has precedence if both are provided.

pointMatcherStrings and pointMatcherRegexes enable the truncation of stringified stack traces after the given matching point. If both parameters are provided, pointMatcherStrings will be checked first.

If a stringified stack trace truncation takes place, it will be indicated with a`suffix`, which by default is set to the configure truncatedStringSuffix in the layout, unless explicitly provided. Every truncation suffix is prefixed with a newline.

Stringified stack trace truncation operates in Caused by: and Suppressed: label blocks. That is, matchers are executed against each label in isolation.

elementTemplate is an object describing the template to be used while resolving the StackTraceElement array. If stringified is set to true, elementTemplate will be discarded. By default, elementTemplate is set to null and rather populated from the layout configuration. That is, the stack trace element template can also be provided using stackTraceElementTemplate and stackTraceElementTemplateUri layout configuration attributes. The template to be employed is determined in the following order:

1. elementTemplate provided in the resolver configuration
2. stackTraceElementTemplate layout configuration attribute (The default is populated from the log4j.layout.jsonTemplate.stackTraceElementTemplate system property.)
3. stackTraceElementTemplateUri layout configuration attribute (The default is populated from the log4j.layout.jsonTemplate.stackTraceElementTemplateUri system property.)

See Stack trace element templates for the list of available resolvers in a stack trace element template.

Note that this resolver is toggled by the stackTraceEnabled layout configuration attribute.

| | Since Throwable#getStackTrace() clones the original StackTraceElement[], access to (and hence rendering of) stack traces are not garbage-free. Each pointMatcherRegexes item triggers a Pattern#matcher() call, which is not garbage-free either. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |

See examples

Resolve LogEvent#getThrown().getClass().getCanonicalName():

{
  "$resolver": "exception",
  "field": "className"
}

Resolve the stack trace into a list of StackTraceElement objects:

{
  "$resolver": "exception",
  "field": "stackTrace"
}

Resolve the stack trace into a string field:

{
  "$resolver": "exception",
  "field": "stackTrace",
  "stackTrace": {
    "stringified": true
  }
}

Resolve the stack trace into a string field such that the content will be truncated after the given point matcher:

{
  "$resolver": "exception",
  "field": "stackTrace",
  "stackTrace": {
    "stringified": {
      "truncation": {
        "suffix": "... [truncated]",
        "pointMatcherStrings": ["at javax.servlet.http.HttpServlet.service"]
      }
    }
  }
}

Resolve the stack trace into an object described by the provided stack trace element template:

{
  "$resolver": "exception",
  "field": "stackTrace",
  "stackTrace": {
    "elementTemplate": {
      "class": {
       "$resolver": "stackTraceElement",
       "field": "className"
      },
      "method": {
       "$resolver": "stackTraceElement",
       "field": "methodName"
      },
      "file": {
       "$resolver": "stackTraceElement",
       "field": "fileName"
      },
      "line": {
       "$resolver": "stackTraceElement",
       "field": "lineNumber"
      }
    }
  }
}

See Stack trace element templates for further details on resolvers available for StackTraceElement templates.

exceptionRootCause

Resolves the fields of the innermost Throwable returned by LogEvent#getThrown(). Its syntax and garbage-footprint are identical to the exception resolver.

level

Resolves the fields of the LogEvent#getLevel()

level event template resolver grammar

config         = field , [ severity ]
field          = "field" -> ( "name" | "severity" )
severity       = severity-field
severity-field = "field" -> ( "keyword" | "code" )

See examples

Resolve the level name:

{
  "$resolver": "level",
  "field": "name"
}

{
  "$resolver": "level",
  "field": "severity",
  "severity": {
    "field": "keyword"
  }
}

{
  "$resolver": "level",
  "field": "severity",
  "severity": {
    "field": "code"
  }
}

logger

Resolves LogEvent#getLoggerFqcn() and LogEvent#getLoggerName().

logger event template grammar

config = "field" -> ( "name" | "fqcn" )

See examples

Resolve the logger name:

{
  "$resolver": "logger",
  "field": "name"
}

Resolve the logger’s fully qualified class name:

{
  "$resolver": "logger",
  "field": "fqcn"
}

main

main event template resolver grammar

config = ( index | key )
index  = "index" -> number
key    = "key" -> string

See examples

Resolve the 1st main() method argument:

{
  "$resolver": "main",
  "index": 0
}

Resolve the argument coming right after --userId:

{
  "$resolver": "main",
  "key": "--userId"
}

map

Resolves MapMessages. See Map resolver for details.

marker

Resolves the marker of the event

marker event template resolver grammar

config = "field" -> ( "name" | "parents" )

See examples

Resolve the marker name:

{
  "$resolver": "marker",
  "field": "name"
}

Resolve the names of the marker’s parents:

{
  "$resolver": "marker",
  "field": "parents"
}

mdc

Resolves the thread context map, aka. Mapped Diagnostic Context (MDC). See Map resolver for details.

| | log4j2.garbagefreeThreadContextMap flag needs to be turned on to iterate the map without allocations. | | -------------------------------------------------------------------------------------------------------- |

message

Resolves the message of the event

message event template resolver grammar

config      = [ stringified ] , [ fallbackKey ]
stringified = "stringified" -> boolean
fallbackKey = "fallbackKey" -> string

| | For simple string messages, the resolution is performed without allocations. For ObjectMessages and MultiformatMessages, it depends. | | --------------------------------------------------------------------------------------------------------------------------------------- |

See examples

Resolve the message into a string:

{
  "$resolver": "message",
  "stringified": true
}

Resolve the message such that if it is an ObjectMessage or a MultiformatMessage with JSON support, its type (string, list, object, etc.) will be retained:

{
  "$resolver": "message"
}

Given the above configuration, a SimpleMessage will generate a "sample log message", whereas a MapMessage will generate a {"action": "login", "sessionId": "87asd97a"}. Certain indexed log storage systems (e.g.,Elasticsearch) will not allow both values to coexist due to type mismatch: one is a string while the other is an object. Here one can use a fallbackKey to work around this problem:

{
  "$resolver": "message",
  "fallbackKey": "formattedMessage"
}

Using this configuration, a SimpleMessage will generate a {"formattedMessage": "sample log message"}, and a MapMessage will generate a {"action": "login", "sessionId": "87asd97a"}. Note that both emitted JSONs are of type object and have no type-conflicting fields.

messageParameter

Resolves LogEvent#getMessage().getParameters()

messageParameter event template resolver grammar

config      = [ stringified ] , [ index ]
stringified = "stringified" -> boolean
index       = "index" -> number

| | Regarding garbage footprint, stringified flag translates to String.valueOf(value), hence mind values that are not String-typed. Further, LogEvent#getMessage() is expected to implement ParameterVisitable interface, which is the case if log4j2.enableThreadlocals property set to true. | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

See examples

Resolve the message parameters into an array:

{
  "$resolver": "messageParameter"
}

Resolve the string representation of all message parameters into an array:

{
  "$resolver": "messageParameter",
  "stringified": true
}

Resolve the first message parameter:

{
  "$resolver": "messageParameter",
  "index": 0
}

Resolve the string representation of the first message parameter:

{
  "$resolver": "messageParameter",
  "index": 0,
  "stringified": true
}

ndc

Resolves the thread context stack – aka. Nested Diagnostic Context (NDC), aka – String[] returned by LogEvent#getContextStack()

ndc event template resolver grammar

config  = [ pattern ]
pattern = "pattern" -> string

See examples

Resolve all NDC values into a list:

Resolve all NDC values matching with the pattern regex:

{
  "$resolver": "ndc",
  "pattern": "user(Role|Rank):\\w+"
}

pattern

pattern event template resolver grammar

config            = pattern , [ stackTraceEnabled ]
pattern           = "pattern" -> string
stackTraceEnabled = "stackTraceEnabled" -> boolean

Unlike providing the pattern attribute to Pattern Layout in a configuration file, property substitutions found in the pattern will not be resolved.

The default value of stackTraceEnabled is inherited from the parent JSON Template Layout.

| | This resolver is mostly intended as an emergency lever when all other JSON Template Layout resolvers fall short of addressing your need. If you find yourself using this, it is highly likely you are either doing something wrong, or JSON Template Layout needs some improvement. In either case, you are advised to share your use case with maintainers in one of our support channels. | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

See examples

Resolve the string produced by %p %c{1.} [%t] %X{userId} %X %m%ex pattern:

{
  "$resolver": "pattern",
  "pattern": "%p %c{1.} [%t] %X{userId} %X %m%ex"
}

source

Resolves the fields of the StackTraceElement returned by LogEvent#getSource()

source event template resolver grammar

config = "field" -> (
           "className"  |
           "fileName"   |
           "methodName" |
           "lineNumber" )

Note that this resolver is toggled by the locationInfoEnabled layout configuration attribute.

| | Capturing the source location information is an expensive operation, and is not garbage-free.The logger resolver can generally be used as a zero-cost substitute for className. See this section of the layouts page for details. | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

See examples

Resolve the line number:

{
  "$resolver": "source",
  "field": "lineNumber"
}

thread

Resolves LogEvent#getThreadId(), LogEvent#getThreadName(), LogEvent#getThreadPriority()

thread event template resolver grammar

config = "field" -> ( "name" | "id" | "priority" )

See examples

Resolve the thread name:

{
  "$resolver": "thread",
  "field": "name"
}

timestamp

Resolves LogEvent#getInstant()

timestamp event template resolver grammar

config        = [ patternConfig | epochConfig ]

patternConfig = "pattern" -> ( [ format ] , [ timeZone ] , [ locale ] )
format        = "format" -> string
timeZone      = "timeZone" -> string
locale        = "locale" -> (
                   language                                   |
                 ( language , "_" , country )                 |
                 ( language , "_" , country , "_" , variant )
                )

epochConfig   = "epoch" -> ( unit , [ rounded ] )
unit          = "unit" -> (
                   "nanos"         |
                   "millis"        |
                   "secs"          |
                   "millis.nanos"  |
                   "secs.nanos"    |
                )
rounded       = "rounded" -> boolean

See examples

Table 1. timestamp template resolver examples

Map resolver

ReadOnlyStringMap is Log4j’s Map<String, Object> equivalent with garbage-free accessors, and is heavily employed throughout the code base. It is the data structure backing both thread context map (aka., Mapped Diagnostic Context (MDC)) and MapMessage implementations. Hence template resolvers for both of these are provided by a single backend: ReadOnlyStringMapResolver. Put another way, both mdc and map resolvers support identical configuration, behaviour, and garbage footprint, which are detailed below.

Map resolver grammar

config        = singleAccess | multiAccess

singleAccess  = key , [ stringified ]
key           = "key" -> string
stringified   = "stringified" -> boolean

multiAccess   = [ pattern ] , [ replacement ] , [ flatten ] , [ stringified ]
pattern       = "pattern" -> string
replacement   = "replacement" -> string
flatten       = "flatten" -> ( boolean | flattenConfig )
flattenConfig = [ flattenPrefix ]
flattenPrefix = "prefix" -> string

singleAccess resolves a single field, whilst multiAccess resolves a multitude of fields. If flatten is provided, multiAccess merges the fields with the parent, otherwise creates a new JSON object containing the values.

Enabling stringified flag converts each value to its string representation.

Regex provided in the pattern is used to match against the keys. If provided, replacement will be used to replace the matched keys. These two are analogous to Pattern.compile(pattern).matcher(key).matches() and Pattern.compile(pattern).matcher(key).replaceAll(replacement) calls.

| | Regarding garbage footprint, stringified flag translates to String.valueOf(value), hence mind values that are not String-typed. pattern and replacement incur pattern matcher allocation costs. Writing certain non-primitive values (e.g., BigDecimal, Set, etc.) to JSON generates garbage, though most (e.g., int, long, String, List, boolean[], etc.) don’t. | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

See examples

"$resolver" is left out in the following examples, since it is to be defined by the actual resolver, e.g., map, mdc.

Resolve the value of the field keyed with user:role:

{
  "$resolver": "…",
  "key": "user:role"
}

Resolve the string representation of the user:rank field value:

{
  "$resolver": "…",
  "key": "user:rank",
  "stringified": true
}

Resolve all fields into an object:

Resolve all fields into an object such that values are converted to string:

{
  "$resolver": "…",
  "stringified": true
}

Resolve all fields whose keys match with the user:(role|rank) regex into an object:

{
  "$resolver": "…",
  "pattern": "user:(role|rank)"
}

Resolve all fields whose keys match with the user:(role|rank) regex into an object after removing the user: prefix in the key:

{
  "$resolver": "…",
  "pattern": "user:(role|rank)",
  "replacement": "$1"
}

Merge all fields whose keys are matching with the user:(role|rank) regex into the parent:

{
  "$resolver": "…",
  "flatten": true,
  "pattern": "user:(role|rank)"
}

After converting the corresponding field values to string, merge all fields to parent such that keys are prefixed with _:

{
  "$resolver": "…",
  "stringified": true,
  "flatten": {
    "prefix": "_"
  }
}

Stack trace element templates

exception and exceptionRootCause event template resolvers can encode an exception stack trace (i.e., StackTraceElement returned by Throwable#getStackTrace()) into a JSON array. While doing so, they employ the underlying JSON templating infrastructure again. The stack trace template used by these event template resolvers to encode StackTraceElements can be configured in following ways:

1. elementTemplate provided in the resolver configuration
2. stackTraceElementTemplate layout configuration attribute (The default is populated from the log4j.layout.jsonTemplate.stackTraceElementTemplate system property.)
3. stackTraceElementTemplateUri layout configuration attribute (The default is populated from the log4j.layout.jsonTemplate.stackTraceElementTemplateUri system property.)

By default, stackTraceElementTemplateUri is set to classpath:StackTraceElementLayout.json, which references to a classpath resource bundled with JSON Template Layout:

StackTraceElementLayout.json bundled as a classpath resource

{
  "class": {
    "$resolver": "stackTraceElement",
    "field": "className"
  },
  "method": {
    "$resolver": "stackTraceElement",
    "field": "methodName"
  },
  "file": {
    "$resolver": "stackTraceElement",
    "field": "fileName"
  },
  "line": {
    "$resolver": "stackTraceElement",
    "field": "lineNumber"
  }
}

Stack trace element template resolvers

Similar to Event template resolvers consuming a LogEvent and rendering a certain property of it at the point of the JSON where they are declared, stack trace element template resolvers consume a StackTraceElement.

The complete list of available event template resolvers are provided below in detail.

stackTraceElement

Resolves certain fields of a StackTraceElement

Stack trace template resolver grammar

config = "field" -> (
           "className"  |
           "fileName"   |
           "methodName" |
           "lineNumber" )

All above accesses to StackTraceElement is garbage-free.