OpenTelemetry (or OTel) is in no way an existential threat to DataDog. Primarily because OTel is simply the substrate/protocol by which data is collected from your apps/systems. DD does _a lot_ more than what OTel provides (RUM, SIEM, synthetics, on-call, dashboarding, anomaly detection, and much much more). If anything it's an existential threat to the more legacy vendors that aren't equipped to provide an OTel ingest layer.

OTel _does_ prevent lock-in on the agent side (making it easier to switch vendors) with open source components and consistent schemas, but OTel doesn't enable you to do anything that you couldn't do before with a specific vendor. It just empowers you to take ownership of your observability data, should you want to. Many don't, though. They want to throw money at someone else who can do it relatively well, hence the ridiculous DD pricing.

Regarding:

> Not sure how companies rationalize these types of services at scale when there are so many open source options to run for a fraction of the cost

At Coinbase's scale (which I assume is a lot due to the DD bill, but I haven't looked closely at it) the open source options simply won't cut it. Plus there are no scalable open source options for many of the things that DD does (synthetics, SIEM come to mind - not to mention onerous regulatory requirements). 65M seems like a lot, but it also means their cloud costs are insane - so maybe it lets them put focus elsewhere?

I don't know the exact history of OTel, but I agree with you. A few years ago, open source applications basically included one type of telemetry: Prometheus metrics. (Maybe Jaeger for tracing if you're really, really lucky.) What OTel is is a way to write an open-source library that emits telemetry, but without dictating that the end user use a particular storage system. Basically, you can use Datadog instead of the open-source stuff, even if the library authors have never heard of Datadog.

Datadog specifically, I don't know if they care. They had an army of junior engineers that existed to hack Datadog into every open source project imaginable. If something had monitoring, they would just add their vendor-specific stuff and upstream it. That was probably expensive but probably accounts for a vast amount of their early marketshare. The other vendors wanted in on that racket without having to do too much work; OTel was born.

> OTel _does_ prevent lock-in on the agent side (making it easier to switch vendors) with open source components and consistent schemas, but OTel doesn't enable you to do anything that you couldn't do before with a specific vendor

This is exactly the reason why we are moving away from NewRelic's SDK to an OpenTelemetry SDK (even though we are still using NewRelic to ingest everything). If (more like when) we decide to switch vendors, it will be much easier to do so.

> OpenTelemetry (or OTel) is in no way an existential threat to DataDog. Primarily because OTel is simply the substrate/protocol by which data is collected from your apps/systems.

As a vendor building in this space [1] - it definitely is. We're able to onboard teams faster to do side-by-side comparisons because they can simply point their existing Otel telemetry to both us and their existing provider with just a few lines of config. That wasn't possible before otel, and levels the playing field more than before. As otel matures, it'll continue to erode against DD's position.

It also allows us as a company to focus on what users care about (as you mention that's things like dashboarding, search performance, RUM, etc.) as opposed to spending all our time building basic integrations into every platform (though we still do plenty of work to polish places where Otel hasn't). Again, levels the playing field.

[1] https://www.hyperdx.io/

So your claim is that Datadogs pricing is ridiculous and that Otel allows you to not be locked in, but somehow Otel isn't a thread to Datadog? I don't follow.

The only part of overlapping functionality between DataDog and Otel is the agent.

In theory you could use the Otel Collector (or any other Otel-compatible agent) instead of the DD Agent to collect metrics/logs/traces. This would then make it easier for you to switch from DD to another Otel-compatible provider (Grafana, for example)... but 99.9% of what DD provides is _not_ the agent, it's dashboarding, alerting, RUM, synthetics, etc.

Basically Otel has made _agent_ switching costs effectively drop to zero, but that is a very small part of the whole picture. Like I said above, this primarily hurts vendors with proprietary agents that can't/won't adopt Otel for ingesting data.

We are building SigNoz (https://github.com/SigNoz/signoz) - an open source alternative to DataDog. We are natively based on opentelemetry and see lots of our users very interested in that.

As mentioned in some other places in the thread, DataDog pricing is very unpredictable and high - and I think more open standards based solutions are the way forward which provides users more predictability and flexibility

I’ve seen an attempted move from DD to OT and it was a nightmare of undocumented features and little compounded issues. Tracing was non functional. It doesn’t seem mature enough yet.

Were you watching me?

Because that's what happened.

I am a user of New Relic. Not because I'm happy. But because OpenTelemetry doesn't come close to the same features. Fortunately, at least OT is about 10x harder to set up with worse documentation.

Wait a minute...

yup at my last company CTO wanted to switch away from New Relic to save money for Open Telemetry and I can tell you we wasted months of work because OpenTelemetry is dogshit.

New Relic is really really good too, so it was even more painful.

Yeah, New Relic is just an excellent product. No way around it.

Apparently OT is a threat to DD to the point of them asking contributors to not add support for their agents… https://github.com/open-telemetry/opentelemetry-collector-co...

I am not sure when you tried OpenTelemetry, but it is decently mature now, esp. for tracing. I am a maintainer at SigNoz (https://github.com/signoz/signoz) and we have good support for tracing using Otel for most of the common frameworks.

I agree it was a bit rapidly evolving in early days, but now its much more mature.

You can check out our docs for distributed tracing here - https://signoz.io/docs/instrumentation/

We had to use otel with Datadog because Datadog did not support Elixir with an official SDK.

There are a lot of subtle issues, but we've been able to work through them to get usable traces. (Metrics and log ingestion already have pretty good existing open-source tooling, like statsd).

There's some purposeful friction with DD when it comes to OTel.

Incumbents in this space are in for a rough time as more applications provide meaningful telemetry, beyond just logs. Fortunately for them that timeline is 'fuzzy' at best.

> You could hire 100+ engineers and still save money compared to a 65M bill

I see cloud costs like this a lot and it really puzzles me. It seems like people would rather pay 10X+ more to just not have to think about it than even to hire other people to think about it, because then you have to think about hiring and HR.

"Here's a blank check. Just make it go away."

Of course corporate consultants run on that, so I guess it's not without abundant precedent elsewhere. I guess if you work for a big company with budget and it's not your money you really have little incentive not to take the easy path.

The real issue is finding the right 100+ engineers and then managing them.

That takes calendar time.

There's a multiple to what people are willing to pay for SaaS solutions precisely because HR for knowledge work is such a pain.

This is indeed the case and one can argue that it makes sense for a small company to focus on the MVP and initial growth. But every such decision needs to be reexamined from time to time as the company scales up. That is unpleasant, requires an expert, so many businesses procrastinate on this and make expensive mistakes. My 2c.

Operational costs can be billed to a project. It's not really that the business doesn't want to save money on this stuff, but it's much easier to wind up in this situation when the loops to hire some engineers are 4x as complicated as just adding another sub-org to the data dog billing..

Does seem like a pretty wild bill, thou.

FWIW, trying to hire 100+ engineers now is probably a lot easier than it would have been in early/mid 2021....

I would argue it's a much easier blank check to write than for consultants. Software complexity over time is a major headache and new initiatives happen all the time.

Take for instance GDPR - in AWS it was a company wide effort to get all the services GDPR compliant and that was basically a non-existent pricing change to consumers.

Also the fact that I can call up AWS support and have them look into a bug immediately with real devs on the other end is invaluable when my business needs rely on a certain feature working.

Presumably these companies have higher ROI projects to do with 100 engineers than reimplementing and maintaining an in-house datadog alternative.