Query 💖 OCSF: how it began
Query set out to build a new federated search platform in late 2022. We had conviction that the market needed:
- One search to rule them all. Security Operators should be able to use the same search terms and operators regardless of the data platform.
- The ability to search disparate data in place. Stop taking your data to your search – take your search to your data. Put down those ETL tools!
- Enrichments and pivots in one query. No SOAR required.
- Graphs of security data. Not pretty charts. Security teams need to understand the relationships between data that span sources and vendors.
But providing all of this is a challenge when there are so many ways data is represented. We needed a unified data model. Should we use CIM? ECS? UDM?!? And what about MITRE? STIX?
Luckily for us, a new schema had just been announced at Black Hat: the Open Cybersecurity Schema Framework (OCSF). Right out of the gate there was a lot we liked about it:
- Hierarchical design. OCSF’s use of objects (complex types) lets Query represent relationships in data well, and graft data from disparate sources together.
- Security domain coverage. Even in the earliest days, we saw that OCSF aimed to be a broad model.
- Industry participation. OCSF came off the blocks with strong backing from Splunk and AWS, and others were quick to get on board.
- Open community. We wanted our voice to matter, even as a small company, and we didn’t want any one vendor to own the roadmap.
- Extensibility. We assumed we’d need to extend the schema, and OCSF extensions offer that out of the box.
- Machine readable. Because OCSF is defined in JSON, Query uses it in code that writes more code. Our data classes, API contracts, documentation, and more are all generated using the schema as input.
These features aligned well with our vision for Query, and so our relationship began!
Because we adopted OCSF early, we’ve got to grow along with the schema. In the early days prior to the release of OCSF 1.0, the rate of change was very high. I’d be lying if I said that didn’t give us cause for frustration at times.
In Query, customers can map their data in flexible stores like AWS S3 or Splunk to our data model, and changing our data model meant customers had to change their configurations. But this frustration led to higher involvement from me and ultimately positive changes for both OCSF and Query. Along the way, Query built tooling to help manage this change for customers, and we’re about to open source a derivative of it to automatically protect future versions of the schema from breaking changes.
One side effect of the fast pace of iteration in pre 1.0 OCSF was human error. The OCSF schema is defined by a large collection of JSON files that are hand-edited by humans. Sometimes changes snuck in that broke Query’s tooling and didn’t have the result the original author intended, so I wrote the ocsf-validator to enforce compliance with the metaschema. This has successfully prevented a lot of errors, and has been made better by contributors from several other organizations.
And that leads to one of my most important lessons about OCSF: the community is responsive, grateful, and helpful. Query was a quiet user of OCSF for our first nine months or so, and I regret that. When we started speaking up and contributing, we found that the community was happy to not only include our suggestions, but also to make them better.
Even though OCSF is young, many of the world’s best security companies are contributing. See for yourself at the list of contributing organizations!
Thinking about jumping in?
I meet people who are curious about OCSF all the time. They usually ask me questions about whether or not the schema can model certain data or how responsive the community is.
My answers are always positive. Nothing is perfect, but OCSF has a strong foundation and a promising future. The schema is broad and growing all the time (1.3.0 will be able to model threat intelligence, courtesy of Query’s own Jonathan Rau), but the changes are backwards-compatible. The tooling is getting better all the time (also in part due to Query’s contributions) and we’re adding support for more encodings.
But most importantly: the community around OCSF has been great to work with. Everyone’s been thoughtful and responsive. The list of contributing organizations keeps growing, and OCSF’s adoption is growing along with it.
If you’ve been following along from home, now’s the time to jump in!
Check out our OCSF absolute beginner’s guide for the fundamentals or our introduction to OCSF mapping to get started.
