GitHub - samber/slog-sampling: 🚨 slog sampling: drop repetitive log records (original) (raw)

Slog sampling policy

A middleware that samples incoming records which caps the CPU and I/O load of logging while attempting to preserve a representative subset of your logs.

Sampling fixes throughput by dropping repetitive log entries.

See also:

slog-multi: slog.Handler chaining, fanout, routing, failover, load balancing...
slog-formatter: slog attribute formatting
slog-sampling: slog sampling policy
slog-mock: slog.Handler for test purposes

HTTP middlewares:

slog-gin: Gin middleware for slog logger
slog-echo: Echo middleware for slog logger
slog-fiber: Fiber middleware for slog logger
slog-chi: Chi middleware for slog logger
slog-http: net/http middleware for slog logger

Loggers:

slog-zap: A slog handler for Zap
slog-zerolog: A slog handler for Zerolog
slog-logrus: A slog handler for Logrus

Log sinks:

slog-datadog: A slog handler for Datadog
slog-betterstack: A slog handler for Betterstack
slog-rollbar: A slog handler for Rollbar
slog-loki: A slog handler for Loki
slog-sentry: A slog handler for Sentry
slog-syslog: A slog handler for Syslog
slog-logstash: A slog handler for Logstash
slog-fluentd: A slog handler for Fluentd
slog-graylog: A slog handler for Graylog
slog-quickwit: A slog handler for Quickwit
slog-slack: A slog handler for Slack
slog-telegram: A slog handler for Telegram
slog-mattermost: A slog handler for Mattermost
slog-microsoft-teams: A slog handler for Microsoft Teams
slog-webhook: A slog handler for Webhook
slog-kafka: A slog handler for Kafka
slog-nats: A slog handler for NATS
slog-parquet: A slog handler for Parquet + Object Storage
slog-channel: A slog handler for Go channels

🚀 Install

go get github.com/samber/slog-sampling

Compatibility: go >= 1.21

No breaking changes will be made to exported APIs before v2.0.0.

💡 Usage

GoDoc: https://pkg.go.dev/github.com/samber/slog-sampling

Middlewares

3 strategies are available:

Uniform sampling: drop % of logs
Threshold sampling: drop % of logs after a threshold
Absolute sampling: limit logs throughput to a fixed number of records
Custom sampler

The sampling middleware can be used standalone or with the slog-multi helpers.

A combination of multiple sampling strategies can be chained. Eg:

drop when a single log message is produced more than 100 times per second
drop above 1000 log records per second (globally)

Matchers

Similar log records can be deduplicated and rate-limited using the Matcher API.

Available Matcher:

slogsampling.MatchByLevelAndMessage (default)
slogsampling.MatchAll
slogsampling.MatchByLevel
slogsampling.MatchByMessage
slogsampling.MatchBySource
slogsampling.MatchByAttribute
slogsampling.MatchByContextValue

Uniform sampling

type UniformSamplingOption struct { // The sample rate for sampling traces in the range [0.0, 1.0]. Rate float64

// Optional hooks
OnAccepted func(context.Context, slog.Record)
OnDropped  func(context.Context, slog.Record)

}

Example using slog-multi:

import ( slogmulti "github.com/samber/slog-multi" slogsampling "github.com/samber/slog-sampling" "log/slog" )

// Will print 33% of entries. option := slogsampling.UniformSamplingOption{ // The sample rate for sampling traces in the range [0.0, 1.0]. Rate: 0.33, }

logger := slog.New( slogmulti. Pipe(option.NewMiddleware()). Handler(slog.NewJSONHandler(os.Stdout, nil)), )

Threshold sampling

type ThresholdSamplingOption struct { // This will log the first Threshold log entries with the same hash, // in a Tick interval as-is. Following that, it will allow Rate in the range [0.0, 1.0]. Tick time.Duration Threshold uint64 Rate float64

// Group similar logs (default: by level and message)
Matcher func(ctx context.Context, record *slog.Record) string

// Optional hooks
OnAccepted func(context.Context, slog.Record)
OnDropped  func(context.Context, slog.Record)

}

If Rate is zero, the middleware will drop all log entries after the first Threshold records in that interval.

Example using slog-multi:

import ( slogmulti "github.com/samber/slog-multi" slogsampling "github.com/samber/slog-sampling" "log/slog" )

// Will print the first 10 entries having the same level+message, then every 10th messages until next interval. option := slogsampling.ThresholdSamplingOption{ Tick: 5 * time.Second, Threshold: 10, Rate: 0.1, }

logger := slog.New( slogmulti. Pipe(option.NewMiddleware()). Handler(slog.NewJSONHandler(os.Stdout, nil)), )

Absolute sampling

type AbsoluteSamplingOption struct { // This will log all entries with the same hash until max is reached, // in a Tick interval as-is. Following that, it will reduce log throughput // depending on previous interval. Tick time.Duration Max uint64

// Group similar logs (default: by level and message)
Matcher Matcher

// Optional hooks
OnAccepted func(context.Context, slog.Record)
OnDropped  func(context.Context, slog.Record)

}

Example using slog-multi:

import ( slogmulti "github.com/samber/slog-multi" slogsampling "github.com/samber/slog-sampling" "log/slog" )

// Will print the first 10 entries during the first 5s, then a fraction of messages during the following intervals. option := slogsampling.AbsoluteSamplingOption{ Tick: 5 * time.Second, Max: 10,

Matcher: slogsampling.MatchAll(),

}

logger := slog.New( slogmulti. Pipe(option.NewMiddleware()). Handler(slog.NewJSONHandler(os.Stdout, nil)), )

Custom sampler

type CustomSamplingOption struct { // The sample rate for sampling traces in the range [0.0, 1.0]. Sampler func(context.Context, slog.Record) float64

// Optional hooks
OnAccepted func(context.Context, slog.Record)
OnDropped  func(context.Context, slog.Record)

}

Example using slog-multi:

import ( slogmulti "github.com/samber/slog-multi" slogsampling "github.com/samber/slog-sampling" "log/slog" )

// Will print 100% of log entries during the night, or 50% of errors, 20% of warnings and 1% of lower levels. option := slogsampling.CustomSamplingOption{ Sampler: func(ctx context.Context, record slog.Record) float64 { if record.Time.Hour() < 6 || record.Time.Hour() > 22 { return 1 }

    switch record.Level {
    case slog.LevelError:
        return 0.5
    case slog.LevelWarn:
        return 0.2
    default:
        return 0.01
    }
},

}

logger := slog.New( slogmulti. Pipe(option.NewMiddleware()). Handler(slog.NewJSONHandler(os.Stdout, nil)), )

🤝 Contributing

Ping me on twitter @samuelberthe (DMs, mentions, whatever :))
Fork the project
Fix open issues or request new features

Don't hesitate ;)

Install some dev dependencies

make tools

Run tests

make test

or

make watch-test

👤 Contributors

💫 Show your support

Give a ⭐️ if this project helped you!

📝 License

This project is MIT licensed.