Id love an ELI5 for PLE. Im trying to work it into my back of the napikin math for compute vs memory bandwidth limitations on tok/s in PP vs TG work.
My attempt at a simplification of this article on it https://sebastianraschka.com/llm-architecture-gallery/per-la... into a couple of sentences is that they are linear embeddings of the input token space projected per layer, which are then gated by the transformer outputs per layer.
This would mean that the only one set of weights for the ple path needs to be pumped across the memory bandwidth as they are the same linear weights for all layers?
Sheit, maybe im trying to simplify something that i need to look at in detail. but id love to leverage others understanding if possible
> It also includes additional 196B Engram memory which you can put on an SSD. I think
You can put Qwen 3.8 Flash Next engram on SSD, but prompt processing takes a good hit. On my mac studio, I get 300 pp and 33 tg with SSD offload, versus 550/40 with everything in RAM.
I will be very happy if 300 pp is achievable with this model though.
You can warm cache regularly used engram/n-gram if you're willing to merge PRs into a personal branch and build it yourself. I was trying this with qwen 3.8 flash next and the n-gram to get it to fit on my very average gaming desktop (it worked)
Hopefully it will be open weights and have the same architecture and size as the current v4 flash vision, which is probably the best LLM that can be run on 128G devices.
IQ3_XXS (~3.2 BPW). For me this is an option because my Mac studio is only used for serving LLMs, so I can afford to dedicate most of its RAM to this. I can run with 256k context and only uses ~117G, with the remaining (up to 125G which I can allocate to VRAM) being used for prompt caching and context checkpoints.
I'm making my own quants, though the Vision-Exp version is outdated and won't work on llama.cpp master branch (I built it before llama added support):
Don't use my Vision-Exp GGUF though. As I said I built those GGUFs before llama.cpp supported, and they can't be loaded on current master (require my own branch).
I already have new GGUFs but haven't uploaded yet. If you want Vision-Exp, maybe use bartowski or unsloth's GGUFs.
Side note:
As an alternative to deepseek v4, you might want to give it a shot at qwen 3.8 flash next. I have IQ4_NL GGUFs that can be loaded fully into 128G, or Q5_K GGUFs that can offload the PLE to disk (use --load-mode none --lazy-mode on for that): https://huggingface.co/tarruda/Qwen3.8-Flash-Next-GGUF.
llama.cpp master is still somewhat bad in Qwen 3.8 next performance, but I was able to achieve 40tps tg and 600 tps pp on my private branch.
I'm curious if you've tried dwarfstar and decided to move to llama.cpp and 3 bit quants or what made you go that route instead? I've been using ds4 for months now and it's already got support for the new vision model, haven't tried it yet, still on 0731 but it's been very solid for me.
I tried dwarfstar when llama.cpp DSV4 support was still very weak, and while it worked, I didn't see anything that would make me want to stick with it vs llama.cpp. llama.cpp is simply better with its awesome built-in webui, router and server APIs and certainly support much more models and quantizations than dwarfstar.
Since then, I started maintaining my own vibe coded dsv4 branch with metal optimizations, so I actually get much better metal performance on my llama.cpp branch than on dwarfstar (plus all the extra llama.cpp features). Here it is in case you want to give it a shot: https://github.com/tarruda/llama.cpp/tree/qwen4exp-dsv4-opti...
My only blocker for installing Asahi on my Mac Studio (M1 ultra) is that llama.cpp performance is very bad when compared to using the Metal backend on the same hardware.
I use my Mac as a headless server to run LLMs, and never even start a desktop session so it always stays at pre-login state without any applications running. Even so, I never saw it using less than 2G in htop when logged via SSH.
I have similar issues with my ultra. Have put it on a laptop, but my desk Setup with my multiple desktop computers is usb-c video based ( kind of a poor man’s kvm) , and asahi doesn’t do usb-c video yet
I haven't tried any of these demos, but I'm not surprised they stop impressing once you go deep.
What got be mostly impressed were the demos of Astra doing computer use. At my job I do some RPA and can appreciate how challenging it can be. Yet they make it look extremely easy to operate a tool like Blender at super human speeds.
I'd be surprised if any of the impressive Blender demos doing the rounds at the moment were built by having an agent control the mouse and keyboard against the Blender application.
A scripting API makes the problem much more approachable, but what about those videos where Astra is drawing people from a photo? Here's one using canva: https://x.com/iam_zachi/status/2095992132620136677
Is that also using scripting to batch updates? It does look as if the mouse is moving.
Could be hallucination, but I gave this to an LLM and this is what it suggested:
"
The workflow shown in the video—processing an image and then controlling a computer interface to draw it—is a combination of two well-established fields: Computer Vision and UI Automation.
You do not necessarily need a Large Language Model to perform the underlying image processing; standard algorithms can do this deterministically.
Step A: Image Processing (The "Brain")
You can write a script (using Python libraries like OpenCV or Pillow) to process the reference photo:
- Edge Detection: Use filters (like Canny or Sobel) to find the "high spatial frequencies" (outlines).
- Color Quantization: Use clustering algorithms (like K-Means) in the HSL space to group millions of pixels into a small palette of distinct colors.
- Vectorization: Convert these processed shapes into a set of coordinates (SVG paths) that represent exactly where the mouse needs to move.
Step B: UI Automation (The "Hand")
Once the image is converted into a set of instructions (coordinates and color codes), you can use automation tools to physically control the computer and draw on Canva.
- Browser Automation: Developers have already created projects that use Selenium (a web automation tool) combined with edge detection algorithms to draw images onto HTML canvases.
- The script reads the pixel data, calculates the mouse coordinates, and executes the "click-and-drag" actions in the browser.
- Computer Use APIs: In the case of GPT-6 Astra, the model uses a "Computer Use" interface. It effectively takes the processed image data (or generates it internally) and outputs high-level commands (e.g., "Move mouse to X,Y," "Click," "Select Hex Color #FF5733"), which the system then executes on the screen.
"
Seems plausible and easier to believe. Also, feels like a "magic trick" designed to fool the user into believing that the agent is drawing interactively by using its vision, since it could just have written a python script that takes the input image, and produces the exact same result without automating the screen.
I don't agree, although it is likely the case. But even if you don't teach an agent about a sandbox bypass, it doesn't matter. Does it know curl? Does it know DNS? Does it know proxying? Then it knows how to pull this off, and it doesn't even need to understand that it's "bypassing" because it thinks it's just iterating towards its goal.
In fact, I wonder if teaching it "this is a bypass" would help it to model when it's doing its job vs working around the job.
> not that Apple prices weren’t insane before the ram/ssd shortages
Funny that after the price started to increase last year (I think October/November), there was a window of a few months where Apple prices stayed the same as they were before, thus making apple prices actually good when compared to the rest of the market.
It was a unique opportunity to have acquired a 512G M3 ultra for $10k.
Also proud of having spent $2.5k on a used 128G M1 Ultra back in September 2024.
Despite being outdated in terms of compute, it stills let me run very good recent models locally, with Deepseek V4 Flash 0731 being the greatest one right now, and hopefully Qwen 3.8 Flash will also fit well when it is released tomorrow!
The M1 ultra definitely leaves to be desired in terms of its token speeds, but I think 20 tps generation and ~200 tps prompt processing (which is what I get with DSv4 flash), is already enough to do a lot of serious work when you combine with the decent prompt caching provided by llama.cpp.
This is what I copied from the en version of the modelscope page, right when they published it:
> Redisgned Multimodal MoE Model: 125B main model parameters, supplemented by an additional 51B N-gram embeddings,and 6B parameters activated per token.
> Efficient Training and Inference: Significantly reduces training and inference costs. At ~1/9th the training cost，Qwen3.8-Flash-Next achieves comparable capability against Qwen3.7-Plus, while being more capable in areas of coding and cowork.
There was another paragraph about a new attention, but I didn't copy that.
reply