Governing the Kafka Firehose

In an oft-memed scene from the cult film UHF, the goofy host of a TV variety show shouts at his guest:

“You get to drink from the firehose!”

When it comes to managing and governing Kafka data, data teams can easily identify with that guest.

It’s as if they’re tasked, hopelessly, with sipping from a firehose of streaming data.

And it isn’t just the sheer volume of event data, nor the incredible velocity at which it gets vectored into Kafka. Nope, the biggest issues stem from frequent and unexpected schema changes.

Kafka’s schema registry and data portal are great, but without a way to actually enforce schema standards across all your upstream apps and services, data breakages are still going to happen. Just as important, without insight into who or what depends on this data, you can’t contain the damage.

And, as data teams know, Kafka data breakages almost always cascade far and wide downstream—wrecking not just data pipelines, and not just business-critical products and services, but also any reports, dashboards, or operational analytics that depend on upstream Kafka data.

Breaking Free from Breaking Bad

To tackle this and other challenges, Saxo Bank, fintech innovator Chime, and dozens of other companies rely on Acryl Cloud, the SaaS data catalog and metadata platform based on DataHub.

With Acryl Cloud, data teams not only get alerted to Kafka data breakages as soon as they occur, but also enjoy one-click access to all of the tools they need to triage and resolve them—in a single UI.

Plus, by integrating checks for ownership and data contracts into their deployment workflows, data teams can practice shift-left governance. This improves the quality and availability of production Kafka data, along with that of the critical reports, charts, and dashboards depending on this data. Best of all, shift-left governance cuts down on post-deployment rework, making teams even more productive.

The Challenge of Governing Kafka Data

Data teams face three primary operational problems with their Kafka data:

1. Unpredictable schema changes.
2. Rapidly proliferating Kafka topics.
3. Redundancy and duplication within topics, usually as a result of partitioning and replication.

But by focusing just on the operational dimension, we’re missing the forest for the trees.

Because above all, teams need a way to manage and govern how their Kafka data is used.

They need to understand what this data is, who owns it, where it came from, what’s been done to it, and what it’s used for. Is it important? Why? Who or what depends on it? Which downstream deliverables or assets? Associated with which processes, services, or products? They need to know if certain Kafka topics contain sensitive data, and, if so, sensitive how? (Can it be used? Under what circumstances? How long can it be retained? What are the procedures for destroying it?)

And they need to be able to monitor, maintain, and improve the quality of their Kafka data. Say a schema change breaks one or more data pipelines, starving downstream KPIs, metrics, and measures of time-sensitive data. The problem is that these analytics themselves don’t always break—they might just be “off.” Ideally, they’re “off” enough that at least one downstream consumer notices; sometimes, however, no one notices, and incorrect data is used to inform decision-making.

Teams need to get notified when breakages occur, and they need to ensure downstream consumers are informed, too.

The Acryl Cloud Difference

This is why customers like Saxo Bank and Chime rely on Acryl Cloud to keep their Kafka-powered analytics, along with other critical business processes, services, and products, reliable and available.

First, Acryl Cloud is based on DataHub, which—like Kafka—was first developed at LinkedIn. In fact, DataHub was designed from the ground up to integrate with Kafka. This tight integration enables you to detect Kafka schema changes in near-real-time. On top of this, Acryl Cloud’s built in alerting and notification capabilities—which connect to Slack and Teams, as well as PagerDuty, OpsGenie, and similar incident-management tools—ensure data teams get alerted almost as soon as problems occur. These same features automatically notify stakeholders, and keep them updated in real time.

Second, Acryl Cloud bundles all of the tools and features data teams need to triage and work around Kafka-related outages or data quality issues. It enables one-click access to rich metadata and documentation—including detailed lineage metadata, ownership information, and compliance labels and documents. Its automated impact analysis tools equip teams to understand and work around breakages. And its discovery and data profiling tools help teams quickly get to the root of breakages.

Third, its integrated data observability solution, Acryl Observe, enables teams to understand how, if at all, breakages affect critical warehouse data—and the KPIs, metrics, and measures that depend on it.

The Takeaway

Ready to govern the Kafka firehose? Interested in learning more about Acryl Cloud and Kafka?

Download and check out this free information sheet, then speak with an Acryl Cloud expert about how to get a customized demo for you and your team!