I think it's a genuinely good implementation of the concept. It looks good and plays reasonably well and captures what it needs to capture from the original.
(Update: OK the sword fighting isn't good. As I got deeper into the game my positive first impressions wore off.)
I haven't tried any of these demos, but I'm not surprised they stop impressing once you go deep.
What got be mostly impressed were the demos of Astra doing computer use. At my job I do some RPA and can appreciate how challenging it can be. Yet they make it look extremely easy to operate a tool like Blender at super human speeds.
I'd be surprised if any of the impressive Blender demos doing the rounds at the moment were built by having an agent control the mouse and keyboard against the Blender application.
A scripting API makes the problem much more approachable, but what about those videos where Astra is drawing people from a photo? Here's one using canva: https://x.com/iam_zachi/status/2095992132620136677
Is that also using scripting to batch updates? It does look as if the mouse is moving.
Could be hallucination, but I gave this to an LLM and this is what it suggested:
"
The workflow shown in the video—processing an image and then controlling a computer interface to draw it—is a combination of two well-established fields: Computer Vision and UI Automation.
You do not necessarily need a Large Language Model to perform the underlying image processing; standard algorithms can do this deterministically.
Step A: Image Processing (The "Brain")
You can write a script (using Python libraries like OpenCV or Pillow) to process the reference photo:
- Edge Detection: Use filters (like Canny or Sobel) to find the "high spatial frequencies" (outlines).
- Color Quantization: Use clustering algorithms (like K-Means) in the HSL space to group millions of pixels into a small palette of distinct colors.
- Vectorization: Convert these processed shapes into a set of coordinates (SVG paths) that represent exactly where the mouse needs to move.
Step B: UI Automation (The "Hand")
Once the image is converted into a set of instructions (coordinates and color codes), you can use automation tools to physically control the computer and draw on Canva.
- Browser Automation: Developers have already created projects that use Selenium (a web automation tool) combined with edge detection algorithms to draw images onto HTML canvases.
- The script reads the pixel data, calculates the mouse coordinates, and executes the "click-and-drag" actions in the browser.
- Computer Use APIs: In the case of GPT-6 Astra, the model uses a "Computer Use" interface. It effectively takes the processed image data (or generates it internally) and outputs high-level commands (e.g., "Move mouse to X,Y," "Click," "Select Hex Color #FF5733"), which the system then executes on the screen.
"
Seems plausible and easier to believe. Also, feels like a "magic trick" designed to fool the user into believing that the agent is drawing interactively by using its vision, since it could just have written a python script that takes the input image, and produces the exact same result without automating the screen.
(Update: OK the sword fighting isn't good. As I got deeper into the game my positive first impressions wore off.)