Fluent Bit Config Generator

Overview

This skill provides a comprehensive workflow for generating production-ready Fluent Bit configurations with best practices built-in. Generate complete pipelines or individual sections (SERVICE, INPUT, FILTER, OUTPUT, PARSER) with proper syntax, optimal performance settings, and automatic validation.

Fluent Bit is a fast and lightweight telemetry agent for logs, metrics, and traces. It's part of the CNCF (Cloud Native Computing Foundation) and is commonly used in Kubernetes environments for log aggregation, forwarding, and processing.

When to Use This Skill

Invoke this skill when:

Creating new Fluent Bit configurations from scratch
Implementing log collection pipelines (INPUT → FILTER → OUTPUT)
Configuring Kubernetes log collection with metadata enrichment
Setting up log forwarding to destinations (Elasticsearch, Loki, S3, Kafka, CloudWatch, etc.)
Building multi-line log parsing for stack traces
Converting existing logging configurations to Fluent Bit
Implementing custom parsers for structured logging
Working with Fluent Bit plugins that require documentation lookup
The user asks to "create", "generate", "build", or "configure" Fluent Bit configs
Setting up telemetry pipelines with filters and transformations

Configuration Generation Workflow

Follow this workflow when generating Fluent Bit configurations. Adapt based on user needs:

Stage 1: Understand Requirements

Gather information about the logging infrastructure needs:

Use case identification:
- Kubernetes log collection (DaemonSet deployment)
- Application log forwarding
- System log collection (syslog, systemd)
- Multi-line log parsing (stack traces, JSON logs)
- Log aggregation from multiple sources
- Metrics collection and forwarding
Input sources:
- tail (file tailing)
- systemd (systemd journal)
- tcp/udp (network input)
- forward (Fluent protocol)
- http (HTTP endpoint)
- kubernetes (K8s pod logs)
- docker (Docker container logs)
- syslog
- exec (command execution)
Processing requirements:
- Parsing (JSON, regex, logfmt)
- Multi-line handling (stack traces)
- Filtering (grep, modify, lua)
- Enrichment (Kubernetes metadata)
- Transformation (nest, rewrite_tag)
- Throttling (rate limiting)
Output destinations:
- Elasticsearch
- Grafana Loki
- AWS S3/CloudWatch
- Kafka
- HTTP endpoint
- File
- stdout (debugging)
- forward (Fluent protocol)
- Prometheus remote write
Performance and reliability:
- Buffer limits (memory constraints)
- Flush intervals
- Retry logic
- TLS/SSL requirements
- Worker threads (parallelism)

Use AskUserQuestion if information is missing or unclear.

Script vs Manual Generation

Step 1: Always verify script capabilities first:

bash

# REQUIRED: Run --help to check if your use case is supported
python3 scripts/generate_config.py --help

Step 2: For supported use cases, prefer using
generate_config.py
for consistency and tested templates:

bash

# Generate configuration for a supported use case
python3 scripts/generate_config.py --use-case kubernetes-elasticsearch --output fluent-bit.conf
python3 scripts/generate_config.py --use-case kubernetes-opentelemetry --cluster-name my-cluster --output fluent-bit.conf

Supported use cases: kubernetes-elasticsearch, kubernetes-loki, kubernetes-cloudwatch, kubernetes-opentelemetry, application-multiline, syslog-forward, file-tail-s3, http-kafka, multi-destination, prometheus-metrics, lua-filtering, stream-processor, custom.

Step 3: Use manual generation (Stages 3-8) when:

The use case is not supported by the script (verified via
```
--help
```
)
Custom plugins or complex filter chains are required (e.g., grep filtering for log levels)
Non-standard configurations are needed
The user explicitly requests manual configuration

Document your decision: When choosing manual generation, explicitly state why the script was not suitable (e.g., "Manual generation chosen because grep filter for log levels is not supported by the script").

Consulting Examples Before Manual Generation

REQUIRED before writing any manual configuration:

Identify the closest matching example from the
```
examples/
```
directory:
- For Kubernetes + Elasticsearch: Read
```
examples/kubernetes-elasticsearch.conf
```
- For Kubernetes + Loki: Read
```
examples/kubernetes-loki.conf
```
- For Kubernetes + OpenTelemetry: Read
```
examples/kubernetes-opentelemetry.conf
```
- For application logs with multiline: Read
```
examples/application-multiline.conf
```
- For syslog forwarding: Read
```
examples/syslog-forward.conf
```
- For S3 output: Read
```
examples/file-tail-s3.conf
```
- For Kafka output: Read
```
examples/http-input-kafka.conf
```
- For multi-destination: Read
```
examples/multi-destination.conf
```
- For Prometheus metrics: Read
```
examples/prometheus-metrics.conf
```
- For Lua filtering: Read
```
examples/lua-filtering.conf
```
- For stream processing: Read
```
examples/stream-processor.conf
```
- For production setup: Read
```
examples/full-production.conf
```
Read the example file using the Read tool to understand:
- Section structure and ordering
- Parameter values and best practices
- Comments and documentation style
Read
examples/parsers.conf
for parser definitions - reuse existing parsers rather than recreating them.
Use examples as templates - copy relevant sections and customize for the user's requirements.

Stage 2: Plugin Documentation Lookup (if applicable)

If the configuration requires specific plugins or custom output destinations:

Identify plugins needing documentation:
- Custom output plugins (proprietary systems)
- Less common input plugins
- Complex filter configurations
- Parser patterns for specific log formats
- Cloud provider integrations (AWS, GCP, Azure)

Try context7 MCP first (preferred):

Use mcp__context7__resolve-library-id with "fluent-bit" or "fluent/fluent-bit"
Then use mcp__context7__get-library-docs with:
- context7CompatibleLibraryID: /fluent/fluent-bit-docs (or /fluent/fluent-bit)
- topic: The plugin name and configuration (e.g., "elasticsearch output configuration")
- page: 1 (fetch additional pages if needed)

Fallback to WebSearch if context7 fails:

Search query patterns:
"fluent-bit" "<plugin-type>" "<plugin-name>" "configuration" "parameters" site:docs.fluentbit.io

Examples:
"fluent-bit" "output" "elasticsearch" "configuration" site:docs.fluentbit.io
"fluent-bit" "filter" "kubernetes" "configuration" site:docs.fluentbit.io
"fluent-bit" "parser" "multiline" "configuration" site:docs.fluentbit.io

Extract key information:
- Required parameters
- Optional parameters and defaults
- Configuration examples
- Performance tuning options
- Common pitfalls and best practices

Stage 3: SERVICE Section Configuration

ALWAYS start with the SERVICE section - this defines global behavior:

ini

[SERVICE]
    # Flush interval in seconds - how often to flush data to outputs
    # Lower values = lower latency, higher CPU usage
    # Recommended: 1-5 seconds for most use cases
    Flush        1

    # Daemon mode - run as background process (Off in containers)
    Daemon       Off

    # Log level: off, error, warn, info, debug, trace
    # Recommended: info for production, debug for troubleshooting
    Log_Level    info

    # Optional: Write Fluent Bit's own logs to file
    # Log_File     /var/log/fluent-bit.log

    # Parser configuration file (if using custom parsers)
    Parsers_File parsers.conf

    # Enable built-in HTTP server for metrics and health checks
    # Recommended for Kubernetes liveness/readiness probes
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

    # Enable storage metrics endpoint
    storage.metrics on

    # Number of worker threads (0 = auto-detect CPU cores)
    # Increase for high-volume environments
    # workers      0

Key SERVICE parameters:

Flush (1-5 sec): Lower for real-time, higher for batching efficiency
Log_Level: Use
```
info
```
in production,
```
debug
```
for troubleshooting
HTTP_Server: Enable for health checks and metrics
Parsers_File: Reference external parser definitions
storage.metrics: Enable for monitoring buffer/storage metrics

Stage 4: INPUT Section Configuration

Create INPUT sections for data sources. Common patterns:

Kubernetes Pod Logs (DaemonSet)

ini

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    # Exclude Fluent Bit's own logs to prevent loops
    Exclude_Path      /var/log/containers/*fluent-bit*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     50MB
    Skip_Long_Lines   On
    Refresh_Interval  10
    Read_from_Head    Off

Key INPUT patterns:

tail plugin (most common):
- ```
Path
```
  : File path or wildcard pattern
- ```
Tag
```
  : Routing tag for filters/outputs
- ```
Parser
```
  : Pre-parser for log format (docker, cri, json)
- ```
DB
```
  : Position database for crash recovery
- ```
Mem_Buf_Limit
```
  : Per-input memory limit (prevents OOM)
- ```
Skip_Long_Lines
```
  : Skip lines > 32KB (prevents hang)
- ```
Read_from_Head
```
  : Start from beginning (false for new logs only)
systemd plugin:

ini

[INPUT]
    Name              systemd
    Tag               host.*
    Systemd_Filter    _SYSTEMD_UNIT=kubelet.service
    Read_From_Tail    On

http plugin (webhook receiver):

ini

[INPUT]
    Name          http
    Tag           app.logs
    Listen        0.0.0.0
    Port          9880
    Buffer_Size   32KB

forward plugin (Fluent protocol):

ini

[INPUT]
    Name          forward
    Tag           forward.*
    Listen        0.0.0.0
    Port          24224

Best practices for INPUT:

Always set
```
Mem_Buf_Limit
```
to prevent memory issues
Use
```
DB
```
for tail inputs to track file positions
Set appropriate
```
Tag
```
patterns for routing
Use
```
Exclude_Path
```
to prevent log loops
Enable
```
Skip_Long_Lines
```
for robustness

Stage 5: FILTER Section Configuration

Create FILTER sections for log processing and enrichment:

Kubernetes Metadata Enrichment

ini

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix     kube.var.log.containers.
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On
    Labels              On
    Annotations         Off
    Buffer_Size         0

Key FILTER patterns:

kubernetes filter (metadata enrichment):
- ```
Merge_Log
```
  : Parse JSON logs into structured fields
- ```
Keep_Log
```
  : Keep original log field (Off saves space)
- ```
K8S-Logging.Parser
```
  : Honor pod parser annotations
- ```
K8S-Logging.Exclude
```
  : Honor pod exclude annotations
- ```
Labels
```
  : Include pod labels in output
- ```
Annotations
```
  : Include pod annotations (optional, increases size)
parser filter (structured parsing):

ini

[FILTER]
    Name          parser
    Match         *
    Key_Name      log
    Parser        json
    Reserve_Data  On
    Preserve_Key  Off

grep filter (include/exclude):

ini

[FILTER]
    Name          grep
    Match         *
    # Include only error logs
    Regex         level (error|fatal|critical)
    # Exclude health check logs
    Exclude       path /health

modify filter (add/remove fields):

ini

[FILTER]
    Name          modify
    Match         *
    Add           cluster_name production
    Add           environment prod
    Remove        _p

nest filter (restructure):

ini

[FILTER]
    Name          nest
    Match         *
    Operation     lift
    Nested_under  kubernetes
    Add_prefix    k8s_

multiline filter (stack traces):

ini

[FILTER]
    Name          multiline
    Match         *
    multiline.key_content log
    multiline.parser      java, python, go

throttle filter (rate limiting):

ini

[FILTER]
    Name          throttle
    Match         *
    Rate          1000
    Window        5
    Interval      1m

lua filter (custom scripting):

ini

[FILTER]
    Name    lua
    Match   *
    script  /fluent-bit/scripts/filter.lua
    call    process_record

Example Lua script (

/fluent-bit/scripts/filter.lua

lua

function process_record(tag, timestamp, record)
    -- Add custom field
    record["custom_field"] = "custom_value"

    -- Transform existing field
    if record["level"] then
        record["severity"] = string.upper(record["level"])
    end

    -- Filter out specific records (return -1 to drop)
    if record["message"] and string.match(record["message"], "DEBUG") then
        return -1, timestamp, record
    end

    -- Return modified record
    return 1, timestamp, record
end

Best practices for FILTER:

Order matters: parsers before modifiers
Use
```
Kubernetes
```
filter in K8s environments for enrichment
Parse JSON logs early to enable field-based filtering
Add cluster/environment identifiers for multi-cluster setups
Use
```
grep
```
to reduce data volume early in pipeline
Implement throttling to prevent downstream overload

Stage 6: OUTPUT Section Configuration

Create OUTPUT sections for destination systems:

Elasticsearch

ini

[OUTPUT]
    Name              es
    Match             *
    Host              elasticsearch.default.svc
    Port              9200
    # Index pattern with date
    Logstash_Format   On
    Logstash_Prefix   fluent-bit
    Retry_Limit       3
    # Buffer configuration
    storage.total_limit_size 5M
    # TLS configuration
    tls               On
    tls.verify        Off
    # Authentication
    HTTP_User         ${ES_USER}
    HTTP_Passwd       ${ES_PASSWORD}
    # Performance tuning
    Buffer_Size       False
    Type              _doc

Grafana Loki

ini

[OUTPUT]
    Name              loki
    Match             *
    Host              loki.default.svc
    Port              3100
    # Label extraction from metadata
    labels            job=fluent-bit, namespace=$kubernetes['namespace_name'], pod=$kubernetes['pod_name'], container=$kubernetes['container_name']
    label_keys        $stream
    # Remove Kubernetes metadata to reduce payload size
    remove_keys       kubernetes,stream
    # Auto Kubernetes labels
    auto_kubernetes_labels on
    # Line format
    line_format       json
    # Retry configuration
    Retry_Limit       3

AWS S3

ini

[OUTPUT]
    Name              s3
    Match             *
    bucket            my-logs-bucket
    region            us-east-1
    total_file_size   100M
    upload_timeout    10m
    use_put_object    Off
    # Compression
    compression       gzip
    # Path structure with time formatting
    s3_key_format     /fluent-bit-logs/%Y/%m/%d/$TAG[0]/%H-%M-%S-$UUID.gz
    # IAM role authentication (recommended)
    # Or use AWS credentials
    # AWS credentials loaded from environment or IAM role
    Retry_Limit       3

Kafka

ini

[OUTPUT]
    Name              kafka
    Match             *
    Brokers           kafka-broker-1:9092,kafka-broker-2:9092
    Topics            logs
    # Message format
    Format            json
    # Timestamp key
    Timestamp_Key     @timestamp
    # Retry configuration
    Retry_Limit       3
    # Queue configuration
    rdkafka.queue.buffering.max.messages     100000
    rdkafka.request.required.acks            1

AWS CloudWatch Logs

ini

[OUTPUT]
    Name              cloudwatch_logs
    Match             *
    region            us-east-1
    log_group_name    /aws/fluent-bit/logs
    log_stream_prefix from-fluent-bit-
    auto_create_group On
    Retry_Limit       3

OpenTelemetry (OTLP)

ini

[OUTPUT]
    Name                 opentelemetry
    Match                *
    Host                 opentelemetry-collector.observability.svc
    Port                 4318
    # Use HTTP protocol for OTLP
    logs_uri             /v1/logs
    # Add resource attributes
    add_label            cluster my-cluster
    add_label            environment production
    # TLS configuration
    tls                  On
    tls.verify           Off
    # Retry configuration
    Retry_Limit          3

Prometheus Remote Write

ini

[OUTPUT]
    Name              prometheus_remote_write
    Match             *
    Host              prometheus.monitoring.svc
    Port              9090
    Uri               /api/v1/write
    # Add labels to all metrics
    add_label         cluster my-cluster
    add_label         environment production
    # TLS configuration
    tls               On
    tls.verify        Off
    # Retry configuration
    Retry_Limit       3
    # Compression
    compression       snappy

HTTP Endpoint

ini

[OUTPUT]
    Name              http
    Match             *
    Host              logs.example.com
    Port              443
    URI               /api/logs
    Format            json
    # TLS
    tls               On
    tls.verify        On
    # Authentication
    Header            Authorization Bearer ${API_TOKEN}
    # Compression
    Compress          gzip
    # Retry configuration
    Retry_Limit       3

stdout (debugging)

ini

[OUTPUT]
    Name              stdout
    Match             *
    Format            json_lines

Key OUTPUT patterns:

Common parameters:
- ```
Name
```
  : Output plugin name
- ```
Match
```
  : Tag pattern to match (supports wildcards)
- ```
Retry_Limit
```
  : Number of retries (0 = infinite)
- ```
storage.total_limit_size
```
  : Disk buffer limit

Buffer and retry configuration:

ini

# Memory buffering (default)
storage.type      memory

# Filesystem buffering (for high reliability)
storage.type      filesystem
storage.path      /var/log/fluent-bit-buffer/
storage.total_limit_size 10G

# Retry configuration
Retry_Limit       5

TLS configuration:

ini

tls               On
tls.verify        On
tls.ca_file       /path/to/ca.crt
tls.crt_file      /path/to/client.crt
tls.key_file      /path/to/client.key

Best practices for OUTPUT:

Always set
```
Retry_Limit
```
(3-5 for most cases)
Use environment variables for credentials:
```
${ENV_VAR}
```
Enable TLS for production
Set
```
storage.total_limit_size
```
to prevent disk exhaustion
Use compression when available (gzip)
For Kubernetes: use service DNS names
Add multiple outputs for redundancy if needed

Stage 7: PARSER Section Configuration

IMPORTANT: Always check
examples/parsers.conf
first before creating custom parsers. The examples directory contains production-ready parser definitions for common use cases.

Step 1: Read the existing parsers file:

bash

# Read the examples/parsers.conf file to see available parsers
Read examples/parsers.conf

Step 2: Reuse existing parsers when possible. The

examples/parsers.conf

includes:

```
docker
```
- Docker JSON log format
```
json
```
- Generic JSON logs
```
cri
```
- CRI container runtime format
```
syslog-rfc3164
```
- Syslog RFC 3164
```
syslog-rfc5424
```
- Syslog RFC 5424
```
nginx
```
- Nginx access logs
```
apache
```
- Apache access logs
```
apache_error
```
- Apache error logs
```
mongodb
```
- MongoDB logs
```
multiline-java
```
- Java stack traces
```
multiline-python
```
- Python tracebacks
```
multiline-go
```
- Go panic traces
```
multiline-ruby
```
- Ruby exceptions

Step 3: Only create custom parsers when the existing ones don't match your log format.

Example custom parser definition (only if needed):

ini

# parsers.conf - Add custom parsers alongside existing ones

[PARSER]
    Name        custom-app
    Format      regex
    Regex       ^(?<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) \[(?<level>\w+)\] (?<message>.*)$
    Time_Key    timestamp
    Time_Format %Y-%m-%d %H:%M:%S

Parser types:

JSON: For JSON-formatted logs
Regex: For custom log formats
LTSV: For LTSV (Labeled Tab-Separated Values)
Logfmt: For logfmt format
MULTILINE_PARSER: For multi-line logs (stack traces)

Best practices for PARSER:

Reuse
examples/parsers.conf
- copy and extend rather than recreating from scratch
Use built-in parsers when possible (docker, cri, json)
Test regex patterns thoroughly
Set
```
Time_Key
```
and
```
Time_Format
```
for proper timestamps
Use
```
MULTILINE_PARSER
```
for stack traces
Reference parsers file in SERVICE section

Stage 8: Complete Configuration Structure

A production-ready Fluent Bit configuration follows this structure:

fluent-bit.conf          # Main configuration file
parsers.conf             # Custom parser definitions (optional)

Before writing a new configuration, consult the
examples/
directory for production-ready templates:

Review
```
examples/
```
files that match your use case
Use them as starting points and customize as needed
Reference
```
examples/parsers.conf
```
for parser definitions

Example complete configuration (Kubernetes to Elasticsearch):

ini

# fluent-bit.conf

[SERVICE]
    Flush         1
    Daemon        Off
    Log_Level     info
    Parsers_File  parsers.conf
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_Port     2020
    storage.metrics on

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Exclude_Path      /var/log/containers/*fluent-bit*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     50MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix     kube.var.log.containers.
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On
    Labels              On
    Annotations         Off

[FILTER]
    Name          modify
    Match         *
    Add           cluster_name my-cluster
    Add           environment production

[FILTER]
    Name          nest
    Match         *
    Operation     lift
    Nested_under  kubernetes

[OUTPUT]
    Name              es
    Match             *
    Host              elasticsearch.logging.svc
    Port              9200
    Logstash_Format   On
    Logstash_Prefix   k8s
    Retry_Limit       3
    storage.total_limit_size 5M
    tls               On
    tls.verify        Off

Stage 9: Best Practices and Optimization

Apply these best practices to all generated configurations:

Performance Optimization

Buffer management:
- Set
```
Mem_Buf_Limit
```
  on inputs (default 32MB can cause OOM)
- Use
```
storage.type filesystem
```
  for high-reliability scenarios
- Set
```
storage.total_limit_size
```
  to prevent disk exhaustion
- Recommended: 50-100MB per input, 5-10GB total disk buffer
Flush and batching:
- ```
Flush 1-5
```
  : Balance between latency and efficiency
- Lower flush = lower latency, higher CPU/network
- Higher flush = better batching, higher memory usage
Worker threads:
- Default (0) auto-detects CPU cores
- Increase for high-volume environments
- Monitor CPU usage before adjusting
Compression:
- Enable compression for network outputs (gzip)
- Reduces bandwidth by 70-90%
- Slight CPU overhead

Reliability

Retry logic:
- Set
```
Retry_Limit 3-5
```
  on all outputs
- Use filesystem buffering for critical logs
- Consider multiple outputs for redundancy
Health checks:
- Enable
```
HTTP_Server
```
  for liveness/readiness probes
- Expose port 2020 (standard)
- Kubernetes probes: GET http://localhost:2020/api/v1/health
Database files:
- Use
```
DB
```
  parameter for tail inputs
- Enables position tracking across restarts
- Store in persistent volume in Kubernetes

Security

TLS/SSL:
- Always enable TLS in production (
```
tls On
```
  )
- Default to
  tls.verify On
  for production deployments
- Use
```
tls.verify Off
```
  ONLY in these scenarios:
  - Internal Kubernetes cluster traffic with self-signed certificates
  - Development/testing environments
  - When proper CA certificates are not available (add comment explaining why)
- When using
```
tls.verify Off
```
  , always add a comment explaining the reason:
  ini
```
tls               On
tls.verify        Off  # Internal cluster with self-signed certs
```
- Use environment variables for credentials
Credentials:
- Never hardcode passwords
- Use environment variables:
```
${VAR_NAME}
```
- Or Kubernetes secrets mounted as env vars
RBAC (Kubernetes):
- Grant minimal permissions to ServiceAccount
- Only needs read access to pods/namespaces
- No write permissions required

Resource Limits

Memory:
- Set per-input limits:
```
Mem_Buf_Limit 50MB
```
- Kubernetes limits: 200-500MB for typical DaemonSet
- Monitor actual usage and adjust
CPU:
- Typically low CPU usage (5-50m per node)
- Spikes during log bursts
- Set requests/limits based on workload
Disk:
- For filesystem buffering only
- Recommended: 5-10GB per node
- Monitor with
```
storage.metrics on
```

Logging Best Practices

Structured logging:
- Prefer JSON logs in applications
- Easier parsing and querying
- Better performance than regex
Log levels:
- Use appropriate log levels in apps
- Filter noisy logs with grep filter
- Reduce volume = lower costs
Avoid log loops:
- Exclude Fluent Bit's own logs
- Use
```
Exclude_Path
```
  pattern
- Tag filtering if needed

Stage 10: Validate Generated Configuration

ALWAYS validate the generated configuration using the devops-skills:fluentbit-validator skill:

Invoke the devops-skills:fluentbit-validator skill to validate the config:
1. Syntax validation (section format, key-value pairs)
2. Required field checks
3. Plugin parameter validation
4. Tag consistency checks
5. Parser reference validation
6. Security checks (plaintext passwords)
7. Best practice recommendations
8. Dry-run testing (if fluent-bit binary available)

Follow the devops-skills:fluentbit-validator workflow to identify and fix any issues.

Validation checklist:

Configuration syntax is correct (INI format)
All required parameters are present
Plugin names are valid
Tags are consistent across sections
Parser files and references exist
Buffer limits are set
Retry limits are configured
TLS is enabled for production
No hardcoded credentials
Memory limits are reasonable

If validation fails, fix issues and re-validate until all checks pass.

Error Handling

Common Issues and Solutions

Configuration syntax errors:
- Check section headers:
```
[SECTION]
```
  format
- Verify key-value indentation (spaces, not tabs)
- Check for typos in plugin names
- Use validator for syntax checking
Memory issues (OOM):
- Set
```
Mem_Buf_Limit
```
  on all tail inputs
- Reduce buffer limits if memory constrained
- Enable filesystem buffering for overflow
- Check Kubernetes memory limits
Missing logs:
- Verify file paths exist
- Check file permissions (read access)
- Verify tag matching in filters/outputs
- Check
```
DB
```
  file for position tracking
- Review
```
Exclude_Path
```
  patterns
Parser failures:
- Test regex patterns with sample logs
- Verify parser file is referenced in SERVICE
- Check Time_Format matches log timestamps
- Enable debug logging to see parser errors
Kubernetes metadata missing:
- Verify RBAC permissions (ServiceAccount, ClusterRole)
- Check Kube_URL is correct (usually https://kubernetes.default.svc:443)
- Verify Kube_CA_File and Kube_Token_File paths
- Check Kube_Tag_Prefix matches input tags
Output connection failures:
- Verify host and port are correct
- Check network connectivity (DNS resolution)
- Verify TLS configuration if enabled
- Check authentication credentials
- Review retry_limit settings
High CPU usage:
- Reduce flush frequency
- Simplify regex parsers
- Reduce filter complexity
- Consider worker threads
Disk full (buffering):
- Set
```
storage.total_limit_size
```
- Monitor disk usage
- Clean old buffer files
- Adjust flush intervals

Communication Guidelines

When generating configurations:

Explain structure - Describe the configuration sections and their purpose
Document decisions - Explain why certain plugins or settings were chosen
Highlight customization - Point out parameters that should be customized
Provide examples - Show how to use the config with different scenarios
Reference documentation - Link to relevant Fluent Bit docs when helpful
Validate proactively - Always validate generated configs and fix issues
Security reminders - Highlight credential and TLS requirements
Performance notes - Explain buffer limits and flush intervals

Integration with devops-skills:fluentbit-validator

After generating any Fluent Bit configuration, automatically invoke the devops-skills:fluentbit-validator skill to ensure quality:

Steps:
1. Generate the Fluent Bit configuration
2. Invoke devops-skills:fluentbit-validator skill with the config file
3. Review validation results
4. Fix any issues identified
5. Re-validate until all checks pass
6. Provide summary of generated config and validation status

This ensures all generated configurations follow best practices and are production-ready.

Resources

scripts/

generate_config.py

Python script for generating Fluent Bit configurations
Template-based approach with common use cases
Supports 13 use cases:
- ```
kubernetes-elasticsearch
```
  - K8s logs to Elasticsearch
- ```
kubernetes-loki
```
  - K8s logs to Loki
- ```
kubernetes-cloudwatch
```
  - K8s logs to CloudWatch
- ```
kubernetes-opentelemetry
```
  - K8s logs to OpenTelemetry (NEW)
- ```
application-multiline
```
  - App logs with multiline parsing
- ```
syslog-forward
```
  - Syslog forwarding
- ```
file-tail-s3
```
  - File tailing to S3
- ```
http-kafka
```
  - HTTP webhook to Kafka
- ```
multi-destination
```
  - Multiple output destinations
- ```
prometheus-metrics
```
  - Prometheus metrics collection (NEW)
- ```
lua-filtering
```
  - Lua script filtering (NEW)
- ```
stream-processor
```
  - Stream processor for analytics (NEW)
- ```
custom
```
  - Minimal custom template

Usage:

python3 scripts/generate_config.py --use-case kubernetes-elasticsearch --output fluent-bit.conf

examples/

Contains production-ready example configurations:

```
kubernetes-elasticsearch.conf
```
- K8s logs to Elasticsearch with metadata enrichment
```
kubernetes-loki.conf
```
- K8s logs to Loki with labels
```
kubernetes-opentelemetry.conf
```
- K8s logs to OpenTelemetry Collector (OTLP/HTTP)
```
application-multiline.conf
```
- App logs with stack trace parsing
```
syslog-forward.conf
```
- Syslog collection and forwarding
```
file-tail-s3.conf
```
- File tailing to S3 with compression
```
http-input-kafka.conf
```
- HTTP webhook to Kafka
```
multi-destination.conf
```
- Logs to multiple outputs (Elasticsearch + S3)
```
prometheus-metrics.conf
```
- Metrics collection and Prometheus remote_write
```
lua-filtering.conf
```
- Custom Lua script filtering and transformation
```
stream-processor.conf
```
- SQL-like stream processing for analytics
```
parsers.conf
```
- Custom parser examples (JSON, regex, multiline)
```
full-production.conf
```
- Complete production setup
```
cloudwatch.conf
```
- AWS CloudWatch integration

Documentation Sources

Based on comprehensive research from:

Fluent Bit Official Documentation
Fluent Bit Operations and Best Practices
Kubernetes Metadata Enrichment Guide
Fluent Bit Tutorial - Coralogix
CNCF Parsing Guide
Context7 Fluent Bit documentation (/fluent/fluent-bit-docs)

fluentbit-generator

NPX Install

Tags

SKILL.md Content

Fluent Bit Config Generator

Overview

When to Use This Skill

Configuration Generation Workflow

Stage 1: Understand Requirements

Script vs Manual Generation

Consulting Examples Before Manual Generation

Stage 2: Plugin Documentation Lookup (if applicable)

Stage 3: SERVICE Section Configuration

Stage 4: INPUT Section Configuration

Kubernetes Pod Logs (DaemonSet)

Stage 5: FILTER Section Configuration

Kubernetes Metadata Enrichment

Stage 6: OUTPUT Section Configuration

Elasticsearch

Grafana Loki

AWS S3

Kafka

AWS CloudWatch Logs

OpenTelemetry (OTLP)

Prometheus Remote Write

HTTP Endpoint

stdout (debugging)

Stage 7: PARSER Section Configuration

Stage 8: Complete Configuration Structure

Stage 9: Best Practices and Optimization

Performance Optimization

Reliability

Security

Resource Limits

Logging Best Practices

Stage 10: Validate Generated Configuration

Error Handling

Common Issues and Solutions

Communication Guidelines

Integration with devops-skills:fluentbit-validator

Resources

scripts/

examples/

Documentation Sources