Zed is a new kind of data lake that provides lightweight search
and analytics for semi-structured data (like JSON) as well as
structured data (like relational tables) all in the same package.
You manage it like Git.
Brim is a desktop app to explore, query,
and shape the data in your Zed data lake.
Inspired by the Unix-tools design pattern, everything Brim does
can be run from the command line. Think of Brim's components like
Lego blocks that you can easily interconnect and assemble.
It’s all open source, so have a look.
Brim is packaged as a desktop app, built with Electron just like Slack.
Once installed, you can drag your data in or use the CLI tools to load
data into a Zed lake. The Zed language, embedded in Brim, has operators
to explore the shape of your data and develop low-code "shapers" to
transform messy data into clean models.
A Security Experience
Brim is built around a plugin model, where plugins implement
domain-specific UX. Currently, Brim supports just one domain, security,
but we are working on more.
While you're clicking around and drilling down into your data,
the security plugin runs
derivative searches in the background automatically, e.g., to join related log
events using the UID field, and presents these derived results
in user-friendly visualizations.
The ZNG Data Format
Does the world really need another data format? Yes indeed!
Brim stores all its data in the new data format
ZNG is richly typed and embeds its type schemas in the data stream.
It's inspired by Avro,
but without the hassle of out-of-band schema
definitions and schema registries.
Brim provides a rich query language that allows you to easily
perform simple, intuitive keyword searches while running rich
analytics with sophisticated filters
and pipelines. Zed is available both in the Brim search bar as
well as in the
Zed CLI tools.
Zed and Python
Your data is everything. If you're running machine-learning experiments
or just exploring the schemas in your pipeline, Python is
your go-to environment. But when you spend all your time futzing with JSON,
waiting for it to load, or cramming messy data sets into a
relational database, you realize that you're not getting stuff done.
Zed turns this all around and makes managing data easy. With Zed,
you can spend your time doing real work instead of futzing with your
Built on Go
The Brim backend is built from the ground up using Go, the system
language that powers massive infrastructure at the world's largest
Internet sites. In fact, when Brim runs on your desktop, there's
Zed server running in the background.
The Zed Lake
The Zed Lake is hot off the presses. While very much a work in progress,
it's ready for you to kick the tires.
We're not trying to make data lakes run 10X faster.
We're trying to make them 10X easier!
Whether you explore a Zed Lake with the Brim app or with the
zapi command-line client, we are maniacally focused on ease of use.
If you find anything confusing or difficult, let us know and we'll
work with you to fix it!