I read it a little differently. The whole article is about how Erlang/Elixir fails at its core reason for existence (fast message passing between distributed processes) and all the complicated work-arounds they had to implement to avoid actually using this core feature of Erlang.
People tend to forget that scalability is not a binary property. You always scale up to some users, up to some architecture, up to some amount of nodes. There is no system that will scale to infinity without requiring developer intervention once business needs and application patterns start to settle in.
Distributed Erlang/Elixir has known limitations. For example, the network is fully meshed, which gives you about 60 to 200 nodes in a cluster. Or don't send large data over distribution, as that delays the other messages, etc. Some of those are easily solvable. For example, you can rely on your orchestration tools to break your clusters in groups. Or you can setup an out of band tcp/udp socket for large data. Others may be more complex.
The important question, however, is how far you can go without having to tweak, and, once you reach those roadblocks, how well you can address them. In many platforms, writing a distributed system is a no-no or, at best, they require you to assemble and tweak from day one. In this case, the ability to start with Erlang/Elixir and tweak as you grow is a feature.
And if you never run into those roadblocks, then you can happily continue running on the default stack. Just look at the many companies using Phoenix PubSub and Phoenix Presence, both distributed, without having to worry about fine-tuning the distribution.
Thanks a lot for the very insightful reply. I'm learning a lot about Erlang from the responses, and a lot of respect for the community as well. Not bad for an on reflection somewhat inflammatory comment :)
What difference does it make? Erlang/OTP distribution doesn't have pluggable architecture. Sooner or later you will reach a point that you have to modify it. Then you are diverging from the original branch which makes it even more difficult to maintain it. You have to merge your additions into every release (minor or major) and test it thoroughly.
A better architecture for a distributed system has a strong composability property. It should be possible to modify every possible aspect of it on a running cluster without introducing downtime.
Write your own standard well documented distribution layer and become independent of underlying technologies.
What do you mean by architecture here? If you mean the roles different nodes take and their topology, I actually doubt you can be decoupled from your architecture in a distributed system because they directly affect your design and capabilities.
You can move away from fully meshed for topologies but how does this choice affect node ups and node downs? Rebalancing can affect how you store data in the cluster.
How many connections should you have between nodes? A single connection makes the ordering guarantee straight-forward. Multiple connections is more performance but requires care if you need ordering and is more efficiently done along side your application.
And what about process placement? On which side of CAP do you want your registries to sit?
If you and your team is capable of "writing your own standard well documented distributed layer" upfront, then you are in a better position than most to take those decisions. But writing a distributed system is hard, so I will gladly start with a well-designed system, especially at the beginning of the project, when it may be unclear which patterns I will need as my application and business grow.
And most times, it will be good enough.
As far as OTP goes, you can plug your own discovery/topology mechanism as well as your own module for handling the connection between nodes. But, as mentioned in my previous reply, some of those issues may be better solved on the side, e.g. a different tcp/udp connection for data transmitting.
I did also mention that for fast time to market it is indeed a good idea to use already available tools. By the way writing a distribution layer is not that difficult. Many companies/startups with scalable back-ends that are not using Erlang/OTP have already done it. The point is scalable software requires knowledge and Erlang/OTP is not going to magically solve it. But it seems many fans are trying to promote it such that it is a magic tool that is going to make a shit software become a hyper scalable one. Just look at the comments below that how people have gone crazy.
Erlang (and by extension Elixir) definitely provides a set of tools which are good for building highly concurrent distributed systems. And your systems are likely to have few errors as well as being resilient, if you know what you are doing.
But indeed, you need to know your tools like every other part of computer science. Storing 9 billion elements in an array, doing linear search and then complaining your linear search is too slow and needs to run faster will be disastrous to your architecture. Likewise, assuming communication is free is equally disastrous.
The problem with building a distribution layer in another language is the effort involved. Most of the companies which pull that off and get stability in addition are usually large, multinational, and has ample amounts of engineering resources to throw at the effort required.
Consider for some reason (either technical or political), this company decides to migrate their web socket servers to Akka/Rust/Go/NodeJS. Integrating these new servers into their core cluster is going to be deadly painful. They rely heavily on Erlang/OTP internal clustering. This is not even considered as a challenge in distributed systems but still really painful to implement because of their design decision. This is what they did wrong. Clustering and message routing part of the application should be technology agnostic.
It isn't that hard. I have code which lets an OCaml program speak the Erlang distribution protocol. It can more or less stand-in for an Erlang node in the distribution layer. Writing the same for another language shouldn't be that hard: the protocol spoken is a simple prefix-protocol which can be parsed with LL(1) from the head plus an atom cache that is easy to maintain.
That said, good Erlang architectures definitely knows when you should fan out and use a common layer for distribution. At $work we have a Kafka cluster for some event handling simply because it is the right tool for the job, and because it makes it easier to interface other programs on top of it.
In my experience, the distribution features are best used as orchestration however. Your distribution layer is Erlang, and then you use other languages for "leaves" in the architecture. If the right library is present in e.g., Go, then by all means use it. Erlang easily speaks protobuf for instance.
That depends what you're going for. Moving web sockets elsewhere are pretty simple regardless of whether you're using Erlang/OTP or not because it's a frontal layer. If you do that all you are doing is creating another layer to hold connections that needs to relay to the rest of the application. The application behind it still has the responsibility to receive and relay everything from those web sockets, determine what message goes to who, etc.
You can move it out just the same as you can elsewhere, you just add an unnecessary layer of complexity and lose a lot of the capabilities already built into the language. It boils down to a simple question of, what would be the purpose behind that decision? What improvement would you get from deciding to move any of that elsewhere? Do you gain speed at the cost of complexity?
Clustering is a non-trivial problem. It can't happen naturally in any language that includes mutable data without explicit oversight and thus most of the needed tooling can't be built into the system and guaranteed to work everywhere. As soon as some part of the system goes from "send message, get response" to "send message, change variable referenced in memory on this machine, get response" you break your ability to naturally cluster.
That leads to a dependency in central relay points for things like setting up web sockets and then having code behind them communicate. The code behind them in whatever language is usually going to be talking to other central relay points like load balancers or pub sub databases rather than directly to other specific servers.
As soon as you have central relay points, they become something else that you have to monitor and scale even if they are very high volume.
There's nothing wrong with this approach and it will certainly work but if you want to avoid that and build a distributed system in another language, you've got to invest a lot in other areas.
> Moving web sockets elsewhere are pretty simple regardless of whether you're using Erlang/OTP or not because it's a frontal layer.
It is supposed to be easy. But they are not using language agnostic messaging. Instead they rely on Erlang distribution protocol. This is what makes it difficult to integrate.
> what would be the purpose behind that decision?
1. To achieve composability. I can replace one web socket server with a C++ implementation to see if I can handle 10 million connections per server. If I fucked it up return back to Erlang implementation otherwise fuck BEAM. I'm moving to native code. The beauty of it is that I can do it one step at a time. Even I can implement web socket layer in multiple languages and experiment all possible options in production and no one even notices it.
2. I don't have to pay for 20 Erlang guys who are mostly senior devs. Get 5 of them to write the core and the rest of the team NodeJS guys.
you're talking about undoing an architecture that lifts the benefits of an entire platform having done the work for you, in order to pick one that is probably easier to migrate to in the future.
You prefer doing all the work mostly from scratch, upfront and straight away, instead of having to maybe adapt a library later if you actually need to migrate.
> 1) Better to have primitives that scale you to X rather than having to get to X on your own.
For a fast time to market, I do agree with you. But if you are well established company in the market, then in most cases "do it yourself" is the best idea. Unless you have the money to call for experts to fix the problem for you.
> 2) The grand parent wrote the Elixir language.
I didn't know him, nor I care.
> 3) Making sweeping like "A better architecture for a distributed system has a strong composability property" is very easy.
I'm considering this as an excuse, than a technical argument.
> But if you are well established company in the market, then in most cases "do it yourself" is the best idea. Unless you have the money to call for experts to fix the problem for you.
This requires having distributed system experts on your team or having the money to call them from day one, before you even reach the market and before you are even sure you will have custom needs. If you have the need, the expertize and the time, then surely. But writing a distributed system from scratch should certainly not be the first choice (IMO).
Erlang's core reason for existence is to control telephone switches, which had two independent general purpose computers connected to the physical switch. So reliability, redundancy, recovery, and fault isolation were the core needs; that drove the design for isolated processes with message passing between them. Because Erlang was in the control plane, and only managing the signal path, not passing the signals itself, there wasn't a big need for speed, as long as it wasn't too slow
Fast forward several years, and isolated processes turns out to be a great fit for large SMP systems, and Erlang/beam is now doing signal path work in a lot of places. Erlang tends not to put explicit limits, but some techniques are going to fail at large scale; ex: if you have 50,000 processes across many nodes, sending the same message to each of those processes is going to be slow; sending one message to each node and fanning out from there is going to be faster; in no small part because you've reduced the network bandwidth you're using.
The nice thing when you hit Erlang scaling limits is that almost everything you need to fix is going to be in a pretty simple state. You're not going to find many things that are layers of optimizations on top of hacks on top of optimizations --- they do a good job of keeping things simple, and not optimizing until it's needed (and even then, they usually pick simple optimizations). Keeping things simple goes a really long way (especially with today's enormous servers).
Edited to add: I don't think they've even needed to tweak the vm yet either, just their user space code. That's pretty huge too.
That, indeed. When I compare Elixir/Erlang to some other systems I worked on, "shallow" is the word that pops up. You hit a limitation, you dig into some source code, and you find out that it's pretty simple to understand and to fix it. It feels manageable, I've yet have to run into frustrating roadblocks, and that all gives me the confidence that when I do need to scale up, I have a system I will understand and will be able to adapt. It looks like Discord's story confirms that.
It sounds like the main benefit to Elixir is that message handling is built into the language. How does that compare to using a message queue service like zeromq?
As macintux, said they don't really compare. Messaging is everywhere in Erlang, in a way that nobody would do with a message queue. For example, you don't read or write to a tcp socket; you receive and send messages to a 'port'. The same is true for file i/o. Rather than calling a method on a shared object, you generally would send a message to a process that owns the state (or a process that manages the state in a database).
Sending messages to processes on other nodes has the same syntax as sending to a process on your node, which makes it easy to run a distributed system. (Ports are different, you'd have to setup a proxy process on the remote node in order to send/receive from that).
Of course, with the base of process to process messaging you can build a higher level messaging queue (see RabbitMQ for a popular message queue built in erlang).
Messaging is implicit in everything Erlang & Elixir do. Bolting on a message queue to software written in another language isn't really comparable (not a value judgement, it's just not really useful to compare them).
I have never seen a language or framework used at a large scale that did not require digging, understanding, and tweaking of the underlying machinery for its specific use case.
The true failure of the platform happens when you cannot do these tweaks and adaptation, or that their cost is shadowed by having written it in a more appropriate technology at a lower time/cost/effort budget.
That's a good point. The article is certainly a good summary of what's needed to make Erlang/Elixir scale and a reminder that there are no "magic bullets".
The solutions they presented were all Elixir + Earlang. It just took some rewrites to get there.
In truly amazing open source fashion, they also made some libraries for other companies to leverage!!! Super big props to Discord for that one. Seriously can't thank their team enough for going above and beyond.
I'm an erlang fan but I wouldn't claim message passing is "fast" in the sense that you might be thinking of (though copying data does have some GC and CPU cache performance benefits that are harder to reason about.) There's no magic in Erlang; copying data is copying data. The benefits of Erlang lie elsewhere, and I've heard Joe Armstrong, when asked about BEAM performance, say something along the lines of "why do you care about performance [in this day and age]?"
Copying data is usually necessary in OTP 20 as well I'm afraid. That new optimization doesn't trigger for most of this. But binary data is not copied and hasn't been since at least OTP11 :)
The release says "Erlang literals". Wouldn't that be things like atoms, integers, booleans, tuples, and the like? That plus binary data should cover a good deal, unless I'm reading too much into the blurb on the release notes.
Under what circumstances does it get triggered (since you seem to be more knowledgable about this than I am!)? I expect records would not fall under this.
A "literal" in this case is a constant value defined in a module. Those live in a separate space in the VM and are referenced directly because they are immutable and can be shared. If you sent such a literal before OTP20, it would be copied into the heap of the target process. Not anymore.
I read it as "Elixir was a good choice, but it isn't a silver bullet." The fact that they spend most of the article talking about the gotcha's makes their statement about going with Elixir given the opportunity to repeat more favorable in my mind.
For a scalable 5M connected users system no language is a silver bullet. Some tools are just better suites for some jobs. Elixir/Erlang OTP just happen to be suited very well for these kind of jobs.
It may well be a misconception on my part - I assumed that as such message passing is such a core feature of using Erlang for distributed systems one should be able to treat it as a low-cost operation.
You can generally, just Discord is getting to a level of scale where the issue has more to do with architecture than language. All of their solutions in this post were Elixir-based solutions, and have very clean, easy APIs(and so far not very much code- all those libraries that solved their problems came out to ~400 LOC).
I guess the message here is, "There's never a magic bullet, but you've got everything you need to make yours."
To put this amount in perspective, you get 5 million connections after receiving 3000 connections per second, which are never dropped and remain connected, for ~28 minutes. The majority of websites do not get even 300 requests per second.
