vector: Sofar we've only been sending events from a source to a sink. But we can also take the effort of transforming the data before arrives at a sink.
Update the Config
To handle this, we'll add some more items to our configuration.
[sources.in]
type = "stdin"
[transforms.parse]
inputs = [ "in"]
type = "remap"
source = '''
.message = parse_json(.message) ?? {}
'''
[transforms.filtered]
inputs=["parse"]
type="filter"
condition = 'length(.message) > 0 ?? true'
[sinks.file]
inputs = ["filtered"]
type = "file"
compression = "none"
encoding.codec = "ndjson"
path = "/tmp/vector-%Y-%m-%d.log"
[sinks.out]
inputs = ["filtered"]
type = "console"
encoding.codec = "text"
In this file the transforms.parse and the transforms.filtered steps
are newly added. These two steps use a bit of code to describe what operation
is happening.
Here's the "code" from the parse step. It turns the message into
valid json. If the message cannot be parse it is turned into an
empty json dictioary.
.message = parse_json(.message) ?? {}
The "code" from the filtered step is able to filter
away the empty dictionaries.
length(.message) > 0 ?? true
Both of these snippets of code are written in a domain specific language called Vector Remap Language (VRL). The docs have a whole section on how to write it.