Fluent-bit Splitting Large JSON Log into Partial Messages

[gursharan-bagha]

Stack Details

• AWS Opensearch Service
• Micros running in AWS ECS Fargate
• Fluentbit running as a sidecar container
• Ingesting logs to Opensearch.

Using AWS Managed Fluent-bit image: public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

Problem Statement

All JSON-formatted logs are being parsed correctly, except for one particular log from an application that generates an unusually large log entry. The size details of this log are:

• Character Count: ~415,000
• Byte Size (UTF-8): ~415 KB
• Line Count (approx.): 41
• File Size on Disk: ~415 KB

Fluent Bit appears to split this single log into multiple partial_message chunks and stores them under the log field.

In the ECS task logs, I see the following warning, which seems related:

[ warn] [engine] failed to flush chunk '1-1747838836.435799559.flb', retry in 8 seconds: task_id=1, input=forward.1 > output=es.2 (out_id=2)

I’d like to understand:

How does Fluent Bit handle such large log entries?

What are the best practices or possible solutions to reliably process and forward large logs like this without data loss or chunking issues?

[patrick-stephens]

Note if it's for the AWS image then they are using an unsupported 1.9 version of OSS so I would raise on their repo directly. You can test with latest OSS image to confirm.

How does the actual log entry look on disk in the file? Is it a single line or separate lines there?
What is your actual config? There are various options there as well to control things so it is important.

[gursharan-bagha]

@patrick-stephens .. Thanks for your response.

Regarding log document.. Its consist of multipe json fields such as: (Problem with full_message field)

{
  "_index": "index_name-",
  "_id": "_9Cydfdfdfddfdfddffdum",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2025-05-22T11:13:30.527Z",
    "host": "ip-999-99-11-2.eu-west-1.compute.internal",
    "version": "1.1",
    "short_message": "There is a example of short message",
    "full_message": "There is a example of short message",  <<<< This fields contains large data.. Approx 415,000 characters. 
    "timestamp": 1747912410.527,
    "level": 6,
    "facility": "logstash-gelf",
    "some_field": "some-data",
    "some_field": 79dfddfff,
    "some_field": "00052661207820368341",
    "logLevel": "INFO",
    "logger": "m.dfdffdfddfd/.fddfdffr",
    "some_field": "dfdfddd",
    "some_field": "dfdfddfd",
    "source": "stdout",
    "container_id": "6565de65d6d8dfd8fd8ef8dfd",
    "container_name": "service-app",
    "ecs_cluster": "abcdemo",
    "ecs_task_arn": "arn:aws:ecs:eu-west-1:111111111:task/1111111/11106e7dc2d02111111111",
    "ecs_task_definition": "app-service:108"
  },
  "fields": {
    "@timestamp": [
      "2025-05-22T11:13:30.527Z"
    ]
  },
  "sort": [
    1747912410527
  ]
}

[patrick-stephens]

I mean how does it look on disk? The actual file you're reading.

Also, config please

[gursharan-bagha]

How does it look on disk?

it's a single-line JSON string, just very long...

Config details:

We’re using the official AWS Fluent Bit image public.ecr.aws/aws-observability/aws-for-fluent-bit:latest in a sidecar (fluentbit) via FireLens on ECS Fargate.

"containerDefinitions": [
        {
            "name": "fluentbit",
            "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:latest",
            "cpu": 0,
            "memoryReservation": 1024,
            "portMappings": [],
            "essential": true,
            "environment": [],
            "mountPoints": [],
            "volumesFrom": [],
            "user": "0",
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "firelens-container",
                    "awslogs-create-group": "true",
                    "max-buffer-size": "25m",
                    "awslogs-region": "eu-west-1",
                    "awslogs-stream-prefix": "firelens"
                }
            },
            "systemControls": [],
            "firelensConfiguration": {
                "type": "fluentbit",
                "options": {
                    "config-file-type": "file",
                    "config-file-value": "/fluent-bit/configs/parse-json.conf"
                }
            }
        },

The configuration is loaded from a static file /fluent-bit/configs/parse-json.conf, with no custom buffer or chunk size settings.

Application containers use the awsfirelens log driver to forward logs to OpenSearch using the es output plugin and parameters from AWS SSM.

"logConfiguration": {
                "logDriver": "awsfirelens",
                "options": {
                    "AWS_Region": "us-east-1",
                    "Logstash_Format": "On",
                    "log-driver-buffer-limit": "526870912",
                    "Suppress_Type_Name": "On",
                    "Port": "443",
                    "Logstash_Prefix": "abc-app",
                    "tls": "On",
                    "Name": "es"
                },
                "secretOptions": [
                    {
                        "name": "HTTP_Passwd",
                        "valueFrom": "arn:aws:ssm:us-east-1:323232323232323:parameter/OPEN_PASS"
                    },
                    {
                        "name": "HTTP_User",
                        "valueFrom": "arn:aws:ssm:us-east-1:323232323232323:parameter/OPEN_US"
                    },
                    {
                        "name": "Host",
                        "valueFrom": "arn:aws:ssm:us-east-1:323232323232323:parameter/OPEN_HOST"
                    }
                ]
            },

[gursharan-bagha]

@patrick-stephens Did you get a chance to check it.?
Thanks.