9/18/2023 0 Comments Moxtra observability![]() ![]() ![]() Data Observability: The Next Frontier of Data EngineeringĪt its simplest, data observability means maintaining a constant pulse of the health of your data systems by monitoring, tracking, and troubleshooting incidents to reduce - and eventually prevent - downtime.ĭata observability shares the same core objective as application observability: minimal disruption to service. This will be accomplished as more companies prioritize data observability. These teams are investing in tooling, developing data-specific SLAs, and delivering on the high level of data health their companies need to flourish.Īs we see more data teams across more companies make similar strides, we’ll enter the new frontier of data engineering: preventing data downtime from happening in the first place. And we’ll get there by following in the footsteps of application engineers to develop our own specialty of DataOps.Īlready, innovative data teams at leading companies like GitLab are applying best practices of DevOps to their data pipelines and processes. Over the next decade, we’ll reach a new level of data reliability. Executives won’t have their data team leads on speed-dial for troubleshooting when an important dashboard looks out of whack - because it won’t happen. We predict that as companies increasingly rely on and invest in data as a key business driver, instances of broken pipelines, missing fields, null values, and the like will go from a tolerated inconvenience to inconceivably rare. And that’s precisely where data engineering needs to go. That’s how far application engineering has come in the last several years. Online applications have become mission-critical to almost every business, and companies invest appropriately in avoiding service interruptions. Today, if Slack, Twitter, or LinkedIn suffers an outage, it makes national headlines. Twenty years ago, when a company’s website or application went down or became unavailable, it was considered par for the course. What Data Engineering Teams Can Learn from DevOps And it’s hard to let something you don’t trust drive your company’s decision-making, let alone your product. It’s a pain, sure, but it’s also a problem for almost every company on a regular basis. What happened to our data?Īt this point, unreliable data is an expected inconvenience. And too many data engineers know the pain of that panicked phone call - the numbers in my dashboard are all wrong. With every additional layer of complexity, opportunities for data downtime - moments when data is partial, erroneous, missing, or otherwise inaccurate - multiply.įorrester estimates that data teams spend upwards of 40% of their time on data quality issues instead of working on revenue-generating activities for the business. We’ve found this is often due to a complete, and not unwarranted, lack of trust in the data itself.Īs companies gather seemingly endless streams of data from a growing number of sources, they begin to amass an ecosystem of data storage, pipelines, and would-be end users. Well-meaning companies are collecting and storing data, but don’t allow it to actually drive their operations. Well, beyond an awful lot of unused data. Behind this desirable label, you’ll usually find an admirable goal-and not much else. In 2021, most businesses’ websites, pitch decks, and job descriptions lay claim to their identities as data-driven companies. Lior holds an MBA from Stanford and an MSC in Computer Science from Tel-Aviv University. At Barracuda, Lior was SVP of Engineering, launching award-winning ML products for fraud prevention. Prior to Monte Carlo, Lior co-founded cybersecurity startup Sookasa, which was acquired by Barracuda in 2016. Lior Gavish is CTO and co-founder of Monte Carlo, a data observability company backed by Accel and other top Silicon Valley investors. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |