Inspired by the Unix-tools design pattern, everything Brim does
can be run from the command line. Think of Brim's components like
Lego blocks that you can easily interconnect and assemble.
It’s all open source, so have a look.
Brim is packaged as a desktop app, built with Electron just like Slack.
Once installed, you can drag your data in or use the CLI tools to load
data into a Zed lake. The Zed language, embedded in Brim, has operators
to explore the shape of your data and develop low-code "shapers" to
transform messy data into clean models.
Brim provides a rich query language that allows you to easily
perform simple, intuitive keyword searches while running rich
analytics with sophisticated filters
and pipelines. Zed is available both in the Brim search bar as
well as in the
Zed CLI tools.
The ZNG Data Format
Does the world really need another data format? Yes indeed!
Brim stores all its data in the new data format
ZNG is richly typed and embeds its type schemas in the data stream.
It's inspired by Avro,
but without the hassle of out-of-band schema
definitions and schema registries.
Built on Go
The Brim backend is built from the ground up using Go, the system
language that powers massive infrastructure at the world's largest
Internet sites. In fact, when Brim runs on your desktop, there's
Zed server running in the background.
Zed and Python
Your data is everything. If you're running machine-learning experiments
or just exploring the schemas in your pipeline, Python is
your go-to environment. But when you spend all your time futzing with JSON,
waiting for it to load, or cramming messy data sets into a
relational database, you realize that you're not getting stuff done.
Zed turns this all around and makes managing data easy. With Zed,
you can spend your time doing real work instead of futzing with your
Brim was originally built as a security platform but we hit a wall
trying to proficiently support both search and analytics equally well
for eclectic security data.
After poring over all the existing tech stacks and data formats, we realized
there must be a better way and stumbled upon the concept of super-structured data.
Brim is still a great solution for security use cases, but we've been busy
generalizing the system to handle a broad range of thorny data engineering
The Zed Lake
The Zed Lake is hot off the presses. While very much a work in progress,
it's ready for you to kick the tires.
With a Zed Lake, your data can enjoy the structured-ness of relational
tables while simultaneously embracining the adapability and schemaless-ness
of JSON. No need to define tables or schemas. Just load data into
your Zed Lake using Git-like commits.
We're not trying to make data lakes run 10X faster.
We're trying to make them 10X easier!
Whether you explore a Zed Lake with the Brim app or with the
zed command-line client, we are maniacally focused on ease of use.
If you find anything confusing or difficult, let us know and we'll
work with you to fix it!