undefined

**Step 2: For supported use cases, prefer using `generate_config.py`** for consistency and tested templates:

```bash

**Supported use cases:** kubernetes-elasticsearch, kubernetes-loki, kubernetes-cloudwatch, kubernetes-opentelemetry, application-multiline, syslog-forward, file-tail-s3, http-kafka, multi-destination, prometheus-metrics, lua-filtering, stream-processor, custom.

**Step 3: Use manual generation (Stages 3-8)** when:
- The use case is not supported by the script (verified via `--help`)
- Custom plugins or complex filter chains are required (e.g., grep filtering for log levels)
- Non-standard configurations are needed
- The user explicitly requests manual configuration

**Document your decision:** When choosing manual generation, explicitly state why the script was not suitable (e.g., "Manual generation chosen because grep filter for log levels is not supported by the script").

[SERVICE]
    # Flush interval in seconds - how often to flush data to outputs
    # Lower values = lower latency, higher CPU usage
    # Recommended: 1-5 seconds for most use cases
    Flush        1

    # Daemon mode - run as background process (Off in containers)
    Daemon       Off

    # Log level: off, error, warn, info, debug, trace
    # Recommended: info for production, debug for troubleshooting
    Log_Level    info

    # Optional: Write Fluent Bit's own logs to file
    # Log_File     /var/log/fluent-bit.log

    # Parser configuration file (if using custom parsers)
    Parsers_File parsers.conf

    # Enable built-in HTTP server for metrics and health checks
    # Recommended for Kubernetes liveness/readiness probes
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

    # Enable storage metrics endpoint
    storage.metrics on

    # Number of worker threads (0 = auto-detect CPU cores)
    # Increase for high-volume environments
    # workers      0

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    # Exclude Fluent Bit's own logs to prevent loops
    Exclude_Path      /var/log/containers/*fluent-bit*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     50MB
    Skip_Long_Lines   On
    Refresh_Interval  10
    Read_from_Head    Off

[INPUT]
    Name              systemd
    Tag               host.*
    Systemd_Filter    _SYSTEMD_UNIT=kubelet.service
    Read_From_Tail    On

[INPUT]
    Name          http
    Tag           app.logs
    Listen        0.0.0.0
    Port          9880
    Buffer_Size   32KB

[INPUT]
    Name          forward
    Tag           forward.*
    Listen        0.0.0.0
    Port          24224

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix     kube.var.log.containers.
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On
    Labels              On
    Annotations         Off
    Buffer_Size         0

[FILTER]
    Name          parser
    Match         *
    Key_Name      log
    Parser        json
    Reserve_Data  On
    Preserve_Key  Off

[FILTER]
    Name          grep
    Match         *
    # Include only error logs
    Regex         level (error|fatal|critical)
    # Exclude health check logs
    Exclude       path /health

[FILTER]
    Name          modify
    Match         *
    Add           cluster_name production
    Add           environment prod
    Remove        _p

[FILTER]
    Name          nest
    Match         *
    Operation     lift
    Nested_under  kubernetes
    Add_prefix    k8s_

[FILTER]
    Name          multiline
    Match         *
    multiline.key_content log
    multiline.parser      java, python, go

[FILTER]
    Name          throttle
    Match         *
    Rate          1000
    Window        5
    Interval      1m

[FILTER]
    Name    lua
    Match   *
    script  /fluent-bit/scripts/filter.lua
    call    process_record

function process_record(tag, timestamp, record)
    -- Add custom field
    record["custom_field"] = "custom_value"

    -- Transform existing field
    if record["level"] then
        record["severity"] = string.upper(record["level"])
    end

    -- Filter out specific records (return -1 to drop)
    if record["message"] and string.match(record["message"], "DEBUG") then
        return -1, timestamp, record
    end

    -- Return modified record
    return 1, timestamp, record
end

[OUTPUT]
    Name              es
    Match             *
    Host              elasticsearch.default.svc
    Port              9200
    # Index pattern with date
    Logstash_Format   On
    Logstash_Prefix   fluent-bit
    Retry_Limit       3
    # Buffer configuration
    storage.total_limit_size 5M
    # TLS configuration
    tls               On
    tls.verify        Off
    # Authentication
    HTTP_User         ${ES_USER}
    HTTP_Passwd       ${ES_PASSWORD}
    # Performance tuning
    Buffer_Size       False
    Type              _doc

[OUTPUT]
    Name              loki
    Match             *
    Host              loki.default.svc
    Port              3100
    # Label extraction from metadata
    labels            job=fluent-bit, namespace=$kubernetes['namespace_name'], pod=$kubernetes['pod_name'], container=$kubernetes['container_name']
    label_keys        $stream
    # Remove Kubernetes metadata to reduce payload size
    remove_keys       kubernetes,stream
    # Auto Kubernetes labels
    auto_kubernetes_labels on
    # Line format
    line_format       json
    # Retry configuration
    Retry_Limit       3

[OUTPUT]
    Name              s3
    Match             *
    bucket            my-logs-bucket
    region            us-east-1
    total_file_size   100M
    upload_timeout    10m
    use_put_object    Off
    # Compression
    compression       gzip
    # Path structure with time formatting
    s3_key_format     /fluent-bit-logs/%Y/%m/%d/$TAG[0]/%H-%M-%S-$UUID.gz
    # IAM role authentication (recommended)
    # Or use AWS credentials
    # AWS credentials loaded from environment or IAM role
    Retry_Limit       3

[OUTPUT]
    Name              kafka
    Match             *
    Brokers           kafka-broker-1:9092,kafka-broker-2:9092
    Topics            logs
    # Message format
    Format            json
    # Timestamp key
    Timestamp_Key     @timestamp
    # Retry configuration
    Retry_Limit       3
    # Queue configuration
    rdkafka.queue.buffering.max.messages     100000
    rdkafka.request.required.acks            1

[OUTPUT]
    Name              cloudwatch_logs
    Match             *
    region            us-east-1
    log_group_name    /aws/fluent-bit/logs
    log_stream_prefix from-fluent-bit-
    auto_create_group On
    Retry_Limit       3

[OUTPUT]
    Name                 opentelemetry
    Match                *
    Host                 opentelemetry-collector.observability.svc
    Port                 4318
    # Use HTTP protocol for OTLP
    logs_uri             /v1/logs
    # Add resource attributes
    add_label            cluster my-cluster
    add_label            environment production
    # TLS configuration
    tls                  On
    tls.verify           Off
    # Retry configuration
    Retry_Limit          3

[OUTPUT]
    Name              prometheus_remote_write
    Match             *
    Host              prometheus.monitoring.svc
    Port              9090
    Uri               /api/v1/write
    # Add labels to all metrics
    add_label         cluster my-cluster
    add_label         environment production
    # TLS configuration
    tls               On
    tls.verify        Off
    # Retry configuration
    Retry_Limit       3
    # Compression
    compression       snappy

[OUTPUT]
    Name              http
    Match             *
    Host              logs.example.com
    Port              443
    URI               /api/logs
    Format            json
    # TLS
    tls               On
    tls.verify        On
    # Authentication
    Header            Authorization Bearer ${API_TOKEN}
    # Compression
    Compress          gzip
    # Retry configuration
    Retry_Limit       3

[OUTPUT]
    Name              stdout
    Match             *
    Format            json_lines

undefined

**Step 2: Reuse existing parsers when possible.** The `examples/parsers.conf` includes:
- `docker` - Docker JSON log format
- `json` - Generic JSON logs
- `cri` - CRI container runtime format
- `syslog-rfc3164` - Syslog RFC 3164
- `syslog-rfc5424` - Syslog RFC 5424
- `nginx` - Nginx access logs
- `apache` - Apache access logs
- `apache_error` - Apache error logs
- `mongodb` - MongoDB logs
- `multiline-java` - Java stack traces
- `multiline-python` - Python tracebacks
- `multiline-go` - Go panic traces
- `multiline-ruby` - Ruby exceptions

**Step 3: Only create custom parsers** when the existing ones don't match your log format.

**Example custom parser definition (only if needed):**

```ini

**Parser types:**

1. **JSON**: For JSON-formatted logs
2. **Regex**: For custom log formats
3. **LTSV**: For LTSV (Labeled Tab-Separated Values)
4. **Logfmt**: For logfmt format
5. **MULTILINE_PARSER**: For multi-line logs (stack traces)

**Best practices for PARSER:**
- **Reuse `examples/parsers.conf`** - copy and extend rather than recreating from scratch
- Use built-in parsers when possible (docker, cri, json)
- Test regex patterns thoroughly
- Set `Time_Key` and `Time_Format` for proper timestamps
- Use `MULTILINE_PARSER` for stack traces
- Reference parsers file in SERVICE section

fluent-bit.conf          # Main configuration file
parsers.conf             # Custom parser definitions (optional)

undefined

undefined

Invoke the devops-skills:fluentbit-validator skill to validate the config:
1. Syntax validation (section format, key-value pairs)
2. Required field checks
3. Plugin parameter validation
4. Tag consistency checks
5. Parser reference validation
6. Security checks (plaintext passwords)
7. Best practice recommendations
8. Dry-run testing (if fluent-bit binary available)

Follow the devops-skills:fluentbit-validator workflow to identify and fix any issues.

Steps:
1. Generate the Fluent Bit configuration
2. Invoke devops-skills:fluentbit-validator skill with the config file
3. Review validation results
4. Fix any issues identified
5. Re-validate until all checks pass
6. Provide summary of generated config and validation status