BRIM

A New Kind of Data Lake

It's free. It's open source.

Zed is a new kind of data lake that provides lightweight search and analytics for semi-structured data (like JSON) as well as structured data (like relational tables) all in the same package. You manage it like Git.

Brim is a desktop app to explore, query, and shape the data in your Zed data lake.

View Talk

A Tools-Based Approach

Inspired by the Unix-tools design pattern, everything Brim does can be run from the command line. Think of Brim's components like Lego blocks that you can easily interconnect and assemble. It’s all open source, so have a look.

Brim Desktop

Brim is packaged as a desktop app, built with Electron just like Slack. Once installed, you can drag your data in or use the CLI tools to load data into a Zed lake. The Zed language, embedded in Brim, has operators to explore the shape of your data and develop low-code "shapers" to transform messy data into clean models.

A Security Experience

Brim is built around a plugin model, where plugins implement domain-specific UX. Currently, Brim supports just one domain, security, but we are working on more. While you're clicking around and drilling down into your data, the security plugin runs derivative searches in the background automatically, e.g., to join related log events using the UID field, and presents these derived results in user-friendly visualizations.

The ZNG Data Format

Does the world really need another data format? Yes indeed! Brim stores all its data in the new data format ZNG. ZNG is richly typed and embeds its type schemas in the data stream. It's inspired by Avro, but without the hassle of out-of-band schema definitions and schema registries.

Zed Language

Brim provides a rich query language that allows you to easily perform simple, intuitive keyword searches while running rich analytics with sophisticated filters and pipelines. Zed is available both in the Brim search bar as well as in the Zed CLI tools.

Zed and Python

Your data is everything. If you're running machine-learning experiments or just exploring the schemas in your pipeline, Python is your go-to environment. But when you spend all your time futzing with JSON, waiting for it to load, or cramming messy data sets into a relational database, you realize that you're not getting stuff done. Zed turns this all around and makes managing data easy. With Zed, you can spend your time doing real work instead of futzing with your eclectic data.

Built on Go

The Brim backend is built from the ground up using Go, the system language that powers massive infrastructure at the world's largest Internet sites. In fact, when Brim runs on your desktop, there's a miniature Zed server running in the background.

The Zed Lake

The Zed Lake is hot off the presses. While very much a work in progress, it's ready for you to kick the tires. We're not trying to make data lakes run 10X faster. We're trying to make them 10X easier! Whether you explore a Zed Lake with the Brim app or with the zapi command-line client, we are maniacally focused on ease of use. If you find anything confusing or difficult, let us know and we'll work with you to fix it!

Please check out our Zed lake design and give us feedback.

Try Brim

Whether you love our ideas or are skeptical, Brim and Zed are easy to try. One of our users told us: "Once I get my data into Zed, everything is easy." Seeing is believing, so give it a try.

And, please reach out to us. We'd love your feedback and ideas.