Flexible and efficient optionally-structured console logging out of the box (original) (raw)

Description

We want to add the capability to control the format of the logs produced by our console logger. Currently, there is an option to choose between Default (logging on multiple lines with colors) and Systemd (logs on a single line without color). We would like to be able to:

[a] add a readable and coloured compact format which will log using a single line
[b] add a JSON based log format.
[c] add an extension point so that users can specify their own custom log format.
[d] we would need to be able to control the formatter used as well as the formatters via Configuration.

Proposal:

The proposal below is aimed at satisfying all the above requirements.

Scope

We are considering limiting the scope of the formatter API to Logging.Console for now, instead of Logging.Abstractions, though this could potentially in the future be adapted to be used for Logging.Debug as well,

Usage and New APIs

Refer to gist: https://gist.github.com/maryamariyan/81f1526fe2156e95352e516f03a61724

we would need to be able to control the formatter used as well as the formatters via Configuration.

As seen in the sample appsettings.json below, the formatter would be selected via Console:Logging:FormatterName, and the formatter options would be set via Console:Logging:FomratterOptions:

{
  "Logging": {
    "LogLevel": {
      "Default": "None",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    },

    "Console": {
      "LogLevel": {
        "Default": "Information"
      },
      "FormatterName": "json",
      "FormatterOptions": {
        "JsonWriterOptions": {
          "Indented": true
        },
        "ExtraCustomProperty": "modified",
        "IncludeScopes": true,
        "TimestampFormat": "HH:mm:ss",
        "UseUtcTimestamp": true
      }
    }
  },
  "AllowedHosts": "*"
}

Screenshots

Screenshots for the different built-in selections (click to view)

default:

default:
(when DefaultConsoleLogFormatterOptions.SingleLine is true)
or we could have an additional formatter called compact rather than toggling by property.

systemd:

json:

Original proposal (click to view)

As a web app developer, I'm trying to write web apps (such as nodatime.org) that log efficiently and cleanly, with minimal effort.
As a Google Cloud Platform client library author, I'm trying to make it really easy for our users to integrate with Stackdriver logging and error reporting.
As a .NET community member, I'm trying to make 12-factor apps easier to write.

Currently, there are three reasonably obvious options for logging in my web application:

Option 1: Use the default console logger.

The "no code changed" option.

This has at least three problems:

Each log entry spans two lines, which makes it far less useful than it might be, and presumably means log entries on busy systems could be interleaved.
The log entries contain control characters for colour; I'm sure I can disable this, but it's annoying out-of-the-box behaviour
It's not structured in an easily machine-readable way.

Option 2: Use Google.Cloud.Diagnostics.AspNetCore

This library makes RPCs directly to the Stackdriver Logging API.

As one of the contributors to the library, I should be able to get this to work easily, right? Well, it does work, but not as conveniently as I'd like. I can improve things, but this is likely to always be a somewhat-heavyweight option. Useful in some cases, such as when there's nothing monitoring console output, but not necessary in modern container-based systems which almost always do monitor the console. It also requires appropriate credentials to be provided when not running on GCP, etc. This ties it to Stackdriver in a way that could be harmful to app portability. (It's fine if you don't need portability of course, but it's a heavyweight dependency otherwise.)

Option 3: Use Serilog

I did get this working, writing out structured JSON to the console, but it was relatively painful to do, and didn't write the JSON in a format that Stackdriver expects. In order to get a custom JSON format, I had to do far more work than I'd normally expect to. It worked, but there's no way that every app should have all of this code in it.

Describe the solution you'd like

Ideally I'd like three simple options:;

A more concise single-line-per-entry console logger, still just lines of plain text
A simple structured logger writing JSON in whatever format seems most reasonable after some research
A way of adapting the second option to write in a custom JSON format

The last is for integration with Stackdriver and presumably other cloud providers' monitoring offerings. With the right JSON, the Stackdriver user interface is clear and easy to read, but still provides structured data that can be used to trigger error reporting, create filters etc.

In an ideal world the CNCF would provide a common JSON format for everyone to use for maximum interoperability, but until that happens it would be useful for the ASP.NET Core team to handle the concerns of gathering the log data and writing the resulting to the console efficiently and atomically, leaving cloud providers to just provide the code to convert the raw log data into appropriate JSON. Even though an app using this would still have a dependency on Stackdriver, it's a much lighter-weight one than Google.Cloud.Diagnostics.AspNetCore - and it would be entirely reasonable to depend on multiple formatters and pick the right one at execution time, expecting each to be standalone and small.

Additional context

I have tried to find existing solutions to this, but I've failed so far.

For nodatime.org, I ended up writing my own console logger, which should only be used for the sake of seeing what I'm trying to achieve - it's not aimed at high-performance scenarios, for example. (If anyone wants to point and laugh, they're welcome - it does the job though :)
https://github.com/nodatime/nodatime.org/blob/master/src/NodaTime.Web/Logging/JsonConsoleLogger.cs

API Proposal

Assembly: Microsoft.Extensions.Logging.Abstractions.dll

Drop Log from names
Rename Default so something that doesn't imply a subset relationship, such as Simple

Assembly: Microsoft.Extensions.Logging.Abstractions.dll

namespace Microsoft.Extensions.Logging {

public readonly struct LogEntry
{
```
  public LogEntry(LogLevel logLevel,
```
```
                  string category,
```
```
                  EventId eventId,
```
```
                  TState state,
```
```
                  Exception exception,
```

                  Func<TState, Exception, string> formatter);

```
  public LogLevel LogLevel { get; }
```
```
  public string Category { get; }
```
```
  public EventId EventId { get; }
```
```
  public TState State { get; }
```
```
  public Exception Exception { get; }
```

  public Func<TState, Exception, string> Formatter { get; }

}

}

Assembly: Microsoft.Extensions.Logging.Console.dll

namespace Microsoft.Extensions.Logging { public static partial class ConsoleLoggerExtensions { public static ILoggingBuilder AddConsole(this ILoggingBuilder builder); public static ILoggingBuilder AddConsole(this ILoggingBuilder builder, Action configure);

   public static ILoggingBuilder AddConsoleFormatter<TFormatter, TOptions>(this ILoggingBuilder builder)

                                 where TFormatter : class, IConsoleFormatter where TOptions : ConsoleFormatterOptions;

   public static ILoggingBuilder AddConsoleFormatter<TFormatter, TOptions>(this ILoggingBuilder builder, Action<TOptions> configure)

                                 where TFormatter : class, IConsoleFormatter where TOptions : ConsoleFormatterOptions;

   public static ILoggingBuilder AddSimpleConsole(this ILoggingBuilder builder, Action<SimpleConsoleFormatterOptions> configure);

   public static ILoggingBuilder AddJsonConsole(this ILoggingBuilder builder, Action<JsonConsoleFormatterOptions> configure);

   public static ILoggingBuilder AddSystemdConsole(this ILoggingBuilder builder, Action<ConsoleFormatterOptions> configure);

} } namespace Microsoft.Extensions.Logging.Console {

public static partial class ConsoleFormatterNames
{

   public const string Simple = "simple";

```
   public const string Json = "json";
```

   public const string Systemd = "systemd";

}
[ObsoleteAttribute("ConsoleLoggerFormat has been deprecated.", false)] public enum ConsoleLoggerFormat { Default = 0, Systemd = 1, } public partial class ConsoleLoggerOptions { public ConsoleLoggerOptions();

   [ObsoleteAttribute("ConsoleLoggerOptions.DisableColors has been deprecated. Please use ColoredConsoleFormatterOptions.DisableColors instead.", false)]
   public bool DisableColors { get; set; }

   [ObsoleteAttribute("ConsoleLoggerOptions.Format has been deprecated. Please use ConsoleLoggerOptions.FormatterName instead.", false)]
   public ConsoleLoggerFormat Format { get; set; }
   public string FormatterName { get; set; }

   [ObsoleteAttribute("ConsoleLoggerOptions.IncludeScopes has been deprecated..", false)]
   public bool IncludeScopes { get; set; }
   public LogLevel LogToStandardErrorThreshold { get; set; }

   [ObsoleteAttribute("ConsoleLoggerOptions.TimestampFormat has been deprecated..", false)]
   public string TimestampFormat { get; set; }

```
   [ObsoleteAttribute("ConsoleLoggerOptions.UseUtcTimestamp has been deprecated..", false)]
   public bool UseUtcTimestamp { get; set; }
```
} [ProviderAliasAttribute("Console")] public partial class ConsoleLoggerProvider : ILoggerProvider, ISupportExternalScope, IDisposable { public ConsoleLoggerProvider(IOptionsMonitor options);

   public ConsoleLoggerProvider(IOptionsMonitor<ConsoleLoggerOptions> options, IEnumerable<IConsoleFormatter> formatters);
   public ILogger CreateLogger(string name);
   public void Dispose();
   public void SetScopeProvider(IExternalScopeProvider scopeProvider);

}

public partial class SimpleConsoleFormatterOptions : ConsoleFormatterOptions
{

   public SimpleConsoleFormatterOptions();

   public bool DisableColors { get; set; }

```
   public bool SingleLine { get; set; }
```
}
public partial abstract ConsoleFormatter
{

   protected ConsoleFormatter(string name);

```
   public string Name { get; }
```

   public abstract void Write<TState>(in LogEntry<TState> logEntry,

                                      IExternalScopeProvider scopeProvider,

                                      TextWriter textWriter);

}
public partial class JsonConsoleFormatterOptions : ConsoleFormatterOptions
{

   public JsonConsoleFormatterOptions();

   public JsonWriterOptions JsonWriterOptions { get; set; }

}
public partial class ConsoleFormatterOptions
{
```
   public ConsoleFormatterOptions();
```

   public bool IncludeScopes { get; set; }

   public string TimestampFormat { get; set; }

   public bool UseUtcTimestamp { get; set; }

}

}