Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 17 additions & 8 deletions concepts/data-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,29 @@ The Fluent Bit data pipeline incorporates several specific concepts. Data proces
```mermaid
graph LR
accTitle: Fluent Bit data pipeline
accDescr: A diagram of the Fluent Bit data pipeline, which includes input, a parser, a filter, a buffer, routing, and various outputs.
A[Input] --> B[Parser]
B --> C[Filter]
C --> D[Buffer]
D --> E((Routing))
E --> F[Output 1]
E --> G[Output 2]
E --> H[Output 3]
accDescr: A diagram of the Fluent Bit data pipeline, which includes input, processors, a parser, a filter, a buffer, routing, and various outputs.
A[Input] --> B[Processors]
B --> C[Parser]
C --> D[Filter]
D --> E[Buffer]
E --> F((Routing))
F --> G[Output 1]
F --> H[Output 2]
F --> I[Output 3]
```

## Inputs

[Input plugins](../pipeline/inputs.md) gather information from different sources. Some plugins collect data from log files, and others gather metrics information from the operating system. There are many plugins to suit different needs.

## Processors

[Processors](../pipeline/processors.md) are components that modify, transform, or enhance data as it flows through the pipeline. Processors are attached directly to individual input or output plugins rather than defined globally, and they don't use tag matching.

Because processors run in the same thread as their associated plugin, they can reduce performance overhead compared to filters—especially when [multithreading](../administration/multithreading.md) is enabled.

Processors are configured in [YAML configuration files](../administration/configuring-fluent-bit/yaml.md) only.

## Parser

[Parsers](../pipeline/parsers.md) convert unstructured data to structured data. Use a parser to set a structure to the incoming data by using input plugins as data is collected.
Expand Down
13 changes: 11 additions & 2 deletions concepts/key-concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,15 @@ Before diving into [Fluent Bit](https://fluentbit.io) you might want to get acqu

- Event or Record
- Filtering
- Processor
- Tag
- Timestamp
- Match
- Structured Message

## Events or records

Every incoming piece of data that belongs to a log or a metric that's retrieved by Fluent Bit is considered an _Event_ or a _Record_.
Every incoming piece of data that belongs to a log, metric, trace, or profile that's retrieved by Fluent Bit is considered an _Event_ or a _Record_.

As an example, consider the following content of a Syslog file:

Expand Down Expand Up @@ -66,6 +67,14 @@ Use filtering to:
- Select a specific piece of the Event content.
- Drop Events that match a certain pattern.

## Processor

[Processors](../pipeline/processors.md) modify, transform, or enhance data as it moves through the pipeline. Unlike filters, processors are attached directly to individual input or output plugins and don't use tag matching. Each processor operates only on data from its associated plugin.

Processors run in the same thread as their associated plugin, which improves throughput compared to filters—especially when [multithreading](../administration/multithreading.md) is enabled.

Processors are supported in [YAML configuration files](../administration/configuring-fluent-bit/yaml.md) only, and can act on logs, metrics, traces, and profiles.

## Tag

Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string used in a later stage by the Router to decide which Filter or [Output](../pipeline/outputs.md) phase it must go through.
Expand All @@ -80,7 +89,7 @@ A tagged record must always have a Matching rule. To learn more about Tags and M

## Timestamp

The timestamp represents the time an Event was created. Every Event contains an associated timestamps. All events have timestamps, and they're set by the input plugin or discovered through a data parsing process.
The timestamp represents the time an Event was created. Every Event contains an associated timestamp. All events have timestamps, and they're set by the input plugin or discovered through a data parsing process.

The timestamp is a numeric fractional integer in the format:

Expand Down