Transforming Events in Vector

Sofar we've only been sending events from a source to a sink. But we can also take the effort of transforming the data before arrives at a sink.

Update the Config

To handle this, we'll add some more items to our configuration.

[sources.in]
type = "stdin"

[transforms.parse]
inputs = [ "in"]
type = "remap"
source = '''
  .message = parse_json(.message) ?? {}
'''

[transforms.filtered]
inputs=["parse"]
type="filter"
condition = 'length(.message) > 0 ?? true'

[sinks.file]
inputs = ["filtered"]
type = "file"
compression = "none"
encoding.codec = "ndjson"
path = "/tmp/vector-%Y-%m-%d.log"

[sinks.out]
inputs = ["filtered"]
type = "console"
encoding.codec = "text"

In this file the transforms.parse and the transforms.filtered steps are newly added. These two steps use a bit of code to describe what operation is happening.

Here's the "code" from the parse step. It turns the message into valid json. If the message cannot be parse it is turned into an empty json dictioary.

.message = parse_json(.message) ?? {}

The "code" from the filtered step is able to filter away the empty dictionaries.

length(.message) > 0 ?? true

Both of these snippets of code are written in a domain specific language called Vector Remap Language (VRL). The docs have a whole section on how to write it.