Love the landing page design! I'm excited to give it a whirl this evening.

Initially, I started to compare PicoMQ to https://github.com/s2-streamstore/s2. If you're familiar with them, how would you compare PicoMQ to S2?

Thank you! I'd say pretty close. But S2 is not open source, and the OSS version (https://github.com/s2-streamstore/s2) is S2-Lite, which is single-node only and uses SlateDB.

SlateDB is great, but it's not purpose-built for streaming workloads. With PicoMQ, the goal is to really do one thing well. It would have been far easier to just extend SlateDB, but instead, the S3Stream engine is built from the ground up with AutoMQ's core primitives.

Not to mention, Pico supports multi-node/cluster deployments and a bunch of other features (no feature gating).

Can you help an old man understand?

This sounds like a kafka-like streaming system, but backed onto s3-objects?

Doesn't this mean that write performance is going to be bad?

I'm glad you asked, and yes, PicoMQ does have some Kafka-like semantics. However, Kafka is great at being a huge pipe, so you'd create topics like tables. PicoMQ, on the other hand, recommends creating granular streams that make the most sense, say, by user, session, or vehicle (still bottomless).

And it's also fair to question write performance, since it's backed by object storage. The optimization is primarily from the shared WAL across streams, server-side batching, and client-side in-memory pipelining, especially with HTTP/2, without as much connection pool overhead.

In practice, you can go to the extent of achieving up to 100 MiB/s throughput per stream. Considering how granular streams can be, you'd rarely need as much. The latency for a durability ACK is, however, the price to pay, which is going to be ~250 ms, or lower with S3 Express, which I'd say covers most real-time use-cases. The design itself is easy enough to extend to a disk-staged WAL for single-digit durability ACK latency.

Have you look at Google's new Rapid Bucket offering? Google Cloud Storage is arguably as good as S3, and is protocol compatible; Rapid Buckets are a type of bucket which supports appendable objects and low latency I/O. The downside is they can only be zonal, and they're a bit more expensive.

PicoMQ works with any S3-compatible object store. But I wasn't aware of GCS Rapid Bucket, it sounds a lot like AWS S3 Express, which is also zonal. And it does help with durability ACK latency quite a bit, keeping it closer to ~50ms.

I'll be setting up a GCP deployment example similar to AWS soon. I'll be sure to try Rapid Bucket as well, thanks for sharing!

Ah, S3 Express does look like the same thing! Looks like directory buckets S3 Express also allows appending to any object, while I believe GCS only allows you to append to a new object and then "finalize" it.

I would also check out Tigris [1], which has an S3-compatible API. Their main claim to fame is that buckets are low-latency, multi-region and replicated by default, so supposedly you get region-local latency no matter where you are reading or writing from. I have not done any rigorous performance comparisons, though. What's amazing, if it does perform well, is that egress is free, and the pricing is otherwise the same as GCS/S3.

[1] https://www.tigrisdata.com/

Totally agree! Tigris is awesome, although I haven't used it at scale. PicoMQ (even cluster mode) is a one click deployment on Fly.io and they use Tigris as the Object Store. https://picomq.com/docs/operations/deployment/fly

Fwiw the only two times I've used kafka in my career have been for traffic on the order of GBs/sec. And the folks I know who have relied on streaming pipes for genuinely realtime stuff built bespoke systems with RTT on the order of 10s of micros.

There are plenty of usecases for lower scale or higher latency (my examples are somewhat unique), and owning the opinionated middle instead of claiming to cover everything is a really useful thing, but acknowledging that the system is opinionated such that it covers a specific set of things well is generally a better argument than 'this basically does everything that people need'.

Precisely this! I might even add a section in the docs, “Not a replacement for Kafka,” under the FAQ.

Where Kafka starts to fall short is routing. If you want to access the data of one user from user-events-topic, that’s expensive to do. Most other streaming technologies are built around the same design, such as Kinesis.

There are other implementations that support the Kafka wire protocol and are cheaper in exchange for latency, e.g., AutoMQ and WarpStream.

That said, I’ll release Disk/EBS-staged WAL soon enough: https://github.com/PicoMQ/picomq/issues/13 as an add-on to cover low-latency needs.

Also (in case this isn't obvious) I'm 100% a fan of the decision to _not_ tightly couple to the kafka wire protocol. Kafka's apis are full of landmines and gotchas, and you can do much better from a UX perspective if you're not married to their quirky semantics.

I had heard people talk about the operation pain involved in keeping Kafka alive (which is a thing for sure), but what I was surprised by was how many things behaved in a slightly unobvious manner that wasn't loudly-documented (e.g. if you're using transactions for RWP loops the default rebalance protocol is unsound and transaction markers take up an index in the log so you no longer have contiguous indices in your message stream etc).

Haha, it was evident. I'd echo the complexities of managing self-hosted Apache Kafka at scale.

But because the Pico's semantics are close to Kafka, but not tightly coupled, it's quite feasible to support the Kafka protocol, * with some restrictions *, such as a topic can only ever have one partition, no support for transactions (since they would only really apply when producers are publishing to multiple streams), and more along those lines. As a result, you'd get Kafka with no topic tax and a lot less operational complexity.

Not familiar with this particular library, but similar libraries use S3 Express One Zone which has write latency <10ms, so you can use that for the WAL and compaction can move data onto other storage classes in the background.

Regular S3 has write latency 100-150ms, which might be fine depending on your workload anyway.

You nailed it! With some of the similar products I've seen, they either inherit the Kafka protocol and hence the KRaft and other complexities, or go the other way with single-node only deployments, commonly just using SlateDB's single-writer model for durability.

I’ve been obsessed with everything durable lately and this is good timing as electric just got acquired so the state of their durable streams project is unknown.

Have you looked into the semantics of their like StreamDB stuff ? Pretty interesting

Also there is this that was built on top of it. Sounds somewhat similar goal as yours but I could be off. Need to explore more https://ursula.tonbo.io/

On the same boat, and honestly, it’s not getting the attention it deserves. Given Databricks acquired ElectricSQL primarily for PGlite, the Durable Streams project is likely going to be abandoned unless new maintainers step in.

I did, in fact, initially write PicoMQ with OpenRaft, similar to Ursala, but I really wanted the operational complexity to be minimal and the nodes to be stateless (at least for most use cases), Pico uses SQL database as a metadata command log, inspired by RisingWave.

But Ursala would, without a doubt, have better durability ACK latency, as it wouldn’t have to wait for an ACK from S3. That said, I do plan on extending disk/EBS-staged WAL to hit similar low-latency durability ACK numbers. But again, most use cases don’t need single-digit-millisecond latency for ACKs.

I was planning to use electric cloud for durable streams recently and found out they are winding it down post acquisition.

Unfortunate timing and self hosting isn't complex or anything but still as you said, durable streams as a project is most likely going to be abandoned.

True, there are also a lot of open PRs and issues. I'm surprised no other developers or orgs are stepping in to keep the project afloat. No wonder the Apache Way goes a long way! Either way, their approach of building around an open-protocol was a good decision.

If you'd like to consider using PicoMQ, I have examples to deploy on Fly.io and AWS. ursula.tonbo.io is also a good option to try.

Thanks for the reply, sounds great, going to dive in!

Your welcome! And feel free to ask any questions as try out PicoMQ or raise on Github issues. The disk-staged WAL is currently tracked here: https://github.com/PicoMQ/picomq/issues/13, I'm planning to ship it as an optional add-on.

Awesome work on this, The "one binary" approach with an embedded admin dashboard makes the barrier to entry delightfully low.

Since you mentioned nodes are stateless and you have examples for Fly.io, how does the system handle rapid node scaling (scaling to zero and back up) in response to bursty traffic? Does the WAL recovery add significant cold-start latency for consumers?

I've been recently exploring the object store based data stores such as turbopuffer, slateDB. have tons of questions revolving in my mind about latency and relaibility. I'll try out this to understand low level details.

The team at opendata.dev/Responsive is killing it, with a strong, opinionated stance on what works well and what doesn't. I did consider building on SlateDB, but it was forcing an unnecessary primitive on streams. I have documented the design choices here: https://picomq.com/docs/design/overview

Sounds great, and documentation is very clear.

So - in theory - something like a massive chat client, discord like, can be implemented via this solution? And what would be the pricing of such a solution. Cheap-serverless-discord

Thank you! I spent a decent chunk of time designing PicoMQ, so the documentation had a natural progression.

Exactly, streams can essentially be rooms, and since the ordering is preserved, a Discord-like application is a strong use case. I’m even considering building one using PicoMQ as an example showcase.

The pricing is going to be dirt cheap, and the best part is how easy it is to scale up vertically or add nodes. For some raw numbers, assuming 1M messages/day, 200B per message, ~6GB/month, all-inclusive, it would be $30 to $150 a month, and storage would be the cheapest part.

Yes, discord as an example would be awesome, exactly what I thought.

Is this back of envelope pricing include the traffic/bandwidth of the readers?

That could easily be 10-100x of number of messages.

This solution together with a cheap/free caching layer (especially for non members/writers) could be amazing.

BTW, a classic example would be an hn mirror. ;)

Yeah, much of the base cost I mentioned was from the compute, networking and S3 write costs, which could sustain a lot more messages for sure. It also depends on the number of active streams/rooms, the throughput bursts, as opposed to a steady state.

But as we speak, I am in the process of running open-benchmarks on AWS with cost attribution. I'll be posting them in the docs with transparency soon enough.

Read and write through cache already exists today, which would work in favour of both cost, latency, and consumer reads fanout.

I like the HN Mirror over a Discord-like app for simplicity. No doubt that's where my weekend is going!

PicoMQ looks worth experimenting with during your prototype phase, especially because it is designed to run as a single binary and supports a relatively simple HTTP-based model

Indeed! The binary even embeds the admin dashboard. A good starting point to get the hang of PicoMQ: https://picomq.com/docs/playground

only if it could be run serverless.

but nice product overall - just like s2 lite.

I'd say PicoMQ is close to being fully serverless, since nodes are stateless. But would miss out benefits of batching, caching and low-latency.

S2-lite is pretty cool, but it's single-node only and built on SlateDB. So, PicoMQ is closer S2 Cloud.