Yeah this could just be some vibeslop doing normal computer stuff. AI agents are famously incompetent at reasoning about database transactions. This seems like the sort of failure you get when you try to do distributed systems without knowing how--brings us back to the early mongodb days. Or it could be deliberate. Or, as you say, both.
Either way, is this a company you want to trust with intimate secrets? "Oh, but they passed SOC2!" Lol.
I have Haier Arctic heat pumps that are made in China. They've been great so far. I've also worked with suppliers in Shenzhen for one-off custom electronic devices that would put western companies to shame in their communication, efficiency, turnaround time, cost, and engineering quality. The new transmission I just put in my truck was made in China too.
I currently have R410a heat pumps with minisplits, when they're off warranty sometime in the future I might consider putting propane in them. The main question would be whether the oil used with R410a is compatible with propane as a refrigerant.
This is probably a little bit "dangerous" in the sense that I'll have high pressure propane in the line sets and minisplits inside the house, but it's a very small volume of propane and not really any more dangerous from a fire perspective than my propane fired condensing gas boiler.
[edit] a little internet sleuthing indicates that hydrocarbon refrigerants "might be" compatible with basically any oil, so I can probably just continue using whatever oil is in it.
This kind of coercive, threatening rhetoric is really not surprising at all. This is how tech companies operate.
What stands out to me is the naive lack of operational security on the part of academics, who should know better than to touch this SaaS crap with a ten foot pole.
I'm imagining all of the potential targeted customer lists now. Grab all those sweet .edu, et al logs without anyone thinking the wiser- until you release a stolen solution. What've they got on .gov?
It's the kind of thing that should make people think, but I wonder to what extent anyone does anymore when it comes to this kind of risk of a vendor stealing from you.
> They’re in an arms race to build a machine god, knowing full well that it could end humanity.
Fantasies built upon extrapolations derived from fever dream delusions. "Could end humanity"? Come on, it's a computer program just like Microsoft Clippy.
It would require an authorized human to load the floppy into the drive and make it go. And there's nothing special about that process being done in code, it could just as well be done by people flipping switches manually.
There's just no plausible way, in the real world, that a computer program running in some datacenter is going to make this happen. Or anything of any dire existential consequence. It's not a real threat.
I think I disagree! You can always manipulate a human. It's true tho that this particular example has a myriad of failsafes, so the actual example I wrote is less than interesting.
Yes, but to make "apocalyptic" scenarios happen your AI would need to manipulate at least millions of people. So far it's not yet clear they can successfully manipulate even one, or even that "manipulate" is a coherent concept, since LLMs pretty clearly have no agency or intrinsic goals. So like... I guess if we start seeing AI systems autonomously throwing elections and starting wars I'd be more concerned? It doesn't look likely anytime soon.
I'm game. Say $1000, donated to a charity of the winner's choice? How do you want to set the bet up, and how do you think it should be decided?
To be precise: I will bet that high-level programming languages won't be any less popular as a whole, but the vast majority of code will be written by AI rather than humans, working from specs written in natural language or something very close to it.
What we call "source code" today will be thought of as "object code" by 2030. Something that occasionally needs to be inspected by humans, but rarely authored directly. Anyone not writing code this way had better be doing it as a hobby, because almost no one will pay for it.
> You and the other poster just need to find some agreeable benchmark so you can decide who won.
That is the hard part. I'm not sure the usage numbers we'd need to decide the outcome are public now, and I can't predict ~4yr out whether any currently available metric will continue to be available. Stack Overflow's popular language thing has been going for a while, that will probably still exist (if SO does) but do we have any reliable metrics to tell us what fraction of code is AI generated?
I think the outcome will be very obvious when the the time comes to settle the bet, but I agree that it's not easy to write those terms today.
But speaking of Stack Overflow, that's a good point to examine more closely. Simply looking at Stack Overflow's usage trend [1] should have been a strong clue that a seismic shift was under way. A couple of things could be responsible for it, though:
1) Stack Overflow's movers and shakers have finally assholed themselves into irrelevance. That's possible, and it could potentially explain the secular decline that began around 2015. But it doesn't explain the rate of descent since 2023, given that the core rules haven't changed for years. It also doesn't explain the meteoric rise between 2008-2014. What they were doing clearly worked, right up until it didn't.
2) AI is now answering questions that would previously have been posted to SO. That's the conventional argument. Hard to dispute it. And it leads to...
3) Stack Overflow has not failed its mission, but fulfilled it: most of the code that will ever need to be written already has been. That's an argument I've never seen anyone else propose, likely because it's a really stupid argument that's been made many times before. I think it's true this time, though, at least until genuinely-new hardware paradigms come along. I think it's a big reason why AI will be doing our jobs for us going forward. Programming is a robot's job now.
3a) As a corollary not related to the SO question, I think the math community is starting to wake up to a similar realization: we have all the math we need. Or, rather, we have all the math we can comprehend. The low-hanging fruit is all gone. Mochizuki's work, which requires a large part of multiple peoples' careers to prove or refute, is an example of that phenomenon. Leading mathematicians are coming to recognize that their best shot at contributing to progress in math is to work on AI.
The groundwork has been laid, and now we need to find better ways to reuse and recycle what we have. That's how AI will help us reach the next level. It's just crazy to think that our industry will be recognizable in 3-5 more years.
Other person probably doesn’t like the idea that LLMs will replace hard earned skills. On the flip side, I bet you’ve seen your skills atrophy at an alarming rate and are trying to justify it.
Both sides come from fear. Just relax and take things as they come. Whatever happens happens.
Maybe. I'll have to think about that one. It doesn't feel like fear to me, but that could be some kind of coping mechanism. From where I sit, I don't believe I fear this AI stuff because I don't really see what all the fuss is about. I don't think I'll be replaced by Claude anytime soon, because fundamentally language models cannot do the things necessary to compete with me. They can generate large amounts of relatively mediocre code quickly, but that's not really what anyone expects of me in the workplace.
Instead, what I do is I think deeply and understand things. And I refine my understanding of those things. And eventually, I have some insight that proves extremely valuable to my employer. Or sometimes not! But on balance it tends to work out swimmingly in their favor. This may sometimes involve writing lots of code, sometimes it involves writing very tiny amounts of code. Sometimes it involves simply deleting code. But the code part of that job is really minimal. It's all the other things--developing understanding, having insights, thinking really hard, making connections, communicating--that are valuable. Banging out code was never a big deal. I'm good at writing clear, correct code and I believe I can do it better than an LLM, if not faster. But that never mattered very much, what actually matters is learning and adapting. I'll be scared when someone makes an "AI" that can do that competitively, but I'm not worried this will happen anytime soon.
So the premises of this "LLM crisis" seem deluded to me. The people saying "everyone will be using an LLM all day every day for their job" seem to have no idea what jobs are like or what LLMs are (in)capable of. The people talking about AI paperclip apocalypses seem like they're having a really bad drug trip. The stock market bros and VCs throwing money as hard as they can after whimsical sci-fi fantasies seem like they've gone completely batshit insane. Meanwhile managers are forcing workers to use LLMs for everything, for no clear reason, and push them into products where it does nothing good. So therefore every little spineless corporate toady is clambering on top of the other little spineless corporate toadies to show big mr manager what a good AI boy they are. It seems like y'all have lost your entire minds because the computer does words now.
It's not fear. It's disgust. I'm disgusted by all of this.
because fundamentally language models cannot do the things necessary to compete with me.
One imagines Lee Sedol telling himself the same thing. True one day and false the next.
It's all the other things--developing understanding, having insights, thinking really hard, making connections, communicating--that are valuable
Spend some time working with frontier models like Fable and Astra. It's clear you haven't. Or if you have, you haven't paid attention to anything but acceptance of the final work product.
"Thinking really hard," LOL. You won't beat an LLM at that. No one ever will again. Insights are still our domain as humans -- if only because the motivations behind them are still entirely ours -- but now we can put those insights into practice at a rate never possible before. And your reaction is disgust?
Meanwhile managers are forcing workers to use LLMs for everything, for no clear reason
I'm with you there. Don't worry, those managers will be the next to go.
Why is it always "you clearly haven't used the latest model, let me tell you this time it'll blow your socks off!"
How many times have we heard this? If it wasn't true all the other times why must I believe it now?
Here's a challenge: show something impressive. Show me how using your LLM will actually help me do something better. How come if this stuff is so great, it's just not all that obvious? I'm looking around and over the past year or 18mo I'm just not seeing a whole lot of change in software. Maybe increased bugs shipped?
How many times have we heard this? If it wasn't true all the other times why must I believe it now?
It was true the first time somebody said, "Hey, check out this ChatGPT thing." It blew my socks off. I still haven't found them. I guess they're probably behind the dryer or something.
I'm looking around and over the past year or 18mo I'm just not seeing a whole lot of change in software. Maybe increased bugs shipped?
I think you're just demanding too much too soon. Peoples' workflows are still adjusting. As you said yourself, correctly, management is being stupid about this whole thing.
The improvements won't all be visible at once. I will ship my next hardware product sooner than I would have, because Fable routed a board for me. Specifically, it wrote a Python program to solve a nasty routing problem that I just didn't feel like working on.
This morning, I was pissed because the volume shadow copy program I use for backups, Casper, screwed up a 30-hour job because it had scheduled a task behind my back, without my knowledge, that tried to perform the same backup to the same drive at the same time. Fable figured out what happened; all I knew was that the boot sector on the target drive was hosed. I will avoid that problem in the future by telling Fable or Astra to find me a simple command-line shadow copy utility that doesn't suck, and if no such utility exists, write it from scratch. That will not be a problem for Fable or Astra, but I wouldn't have dreamed of relying on an LLM to do that last year.
Will this win be visible to you, as a consumer? No, because LOL at anybody trying to sell something like that from here on out.
The software that I develop to support the hardware? I rarely work on it anymore, as of a few weeks ago. When I want a new feature added, I ask Claude (or, again, Astra, which is now better than Fable in some respects.) Claude could do a lot of that work last year, but it needed extensive babysitting. Now it has taste. It chooses the controls, lays out the dialog box and writes the help text. You won't notice the details as a consumer, you'll just see more rapid incremental improvements coming out for my software and ultimately the hardware as well.
Speaking of my very-obscure hardware, last week I ran into a very obscure firmware bug in some code I wrote. Or, rather, my customer did. Different customers are demanding both new features and bug fixes, and I only have the bandwidth for one of those. So: "Claude, see if you can figure out why the board stopped responding to input X when input Y went away. Here are the component data sheets." Took it about an hour but it would have taken me longer.
I would like to say I used the time it saved me to work in parallel on new features, but in reality I spent it here instead, arguing with Luddites. They say "you are your calendar," so I guess I enjoy that even more than building new stuff. Funny how much the clankers are teaching me about myself!
The most charitable way I can describe it is just extremely low quality sci-fi fan fiction. I think that's too charitable, because I believe it's far more cynical than that. They're deliberately playing into these sort of techno-religious beliefs that have taken root in the wake of Kurzweil, et al., fanned by LLM psychosis, influencer marketing, and a deluge of this kind of sci-fi marketing copy. It's just chatbots, guys. Relax.
Uhh... what? You'll instantaneously feel very differently when the service you're responsible for is down at 3:15am and your logs are full of stack traces that end somewhere in that code that "doesn't need to be understood". At that point, you will need to understand it well enough to fix it stat.
Test code for a new bug is a good example. You can prove the test covers the bug without understanding the test code (you need to understand the bug, of course). There are some domains/tests where you can't do that - you need to be sure its failing for the right reason, but often you can do that without understanding every line of the test code. You can extend this to lots of related test infrastructure. If you can watch playwright test the app the way you expect it to, you don't have to understand all the code.
You can also do this for apps that are just tools for your own use. You satisfy yourself that they are working, and you use them because they save your time. You review enough to be sure its implemented the way you think it is - and if it is working, that tells you quite a lot. Sometimes you will be surprised and have some time wasted.
Yes, yes - there are people who will make the wrong choices in some of these cases but that doesn't mean there are never cases where you can do it.
More broadly - anyone who works in a team is already working with code they don't fully understand. I have code I wrote years ago I don't fully understand. I trust its observable properties and its track record.
> You can prove the test covers the bug without understanding the test code (you need to understand the bug, of course).
I'm not following.. When we write regression tests those tests encode invariants we expect to be maintained under source code transformations over time. If I don't understand the test code I've written, how can I know which invariants I've imposed? That's why, broadly speaking, we write test code to be as simple as possible above all else--it's absolutely imperative that these invariants are not only intentional and easy to reason about, but also that when an invariant is violated we can easily discover why. Often, on a team, the person encountering a test failure after making a code change is not the person who originally established the invariant, so it's very important they be able to easily understand it.
I see no possible world in which failing to understand the test code is... possible? Like, if you have indecipherable test code things are really bad in your codebase. Fixing that is P0, because it'll compound rapidly.
You can know the test is likely good, if it reproduces the failure you are fixing. I don't think we're communicating though because I never said the test code was undecipherable.
I don't really care about the tool per se, but if I'm reading some LLM output you give me its utility is awfully limited if you don't also supply the prompt. That's what I'd like to see normalized:
Do not give me LLM output unless you also give me the full prompt text.
Without the prompt I cannot discern what you were trying to do, because the LLM output is basically devoid of any coherent voice. Whether it's code, prose, or pictures don't bother sending it to me unless you also include the prompt text.
[edit] I suspect the fact that people are often reticent to share their prompts says quite a lot about how and why they're using an LLM.
it's often not a single prompt but a tree or rooted DAG of many prompts, with the root node being the final prompt. the prompts on interior nodes might not make much sense without the intermediate responses as well. how would you like this communicated?
Just a chat log would be acceptable. Like an IRC channel log or slack channel history. It would be even better if, say in the case of code, there was a durable mapping between prompts and commits, that way I could follow the prompts linearly and review every commit associated with that prompt. For other forms of output (prose, pictures, whatever) this would also be useful to be able to see the history of its development.
>Do not give me LLM output unless you also give me the full prompt text.
Umm the point is you won't know to ask this question in the first place, and even if you did, you wouldn't have any leverage to demand this because you're a peasant.
In other words: bias. Tons and tons of self-congratulatory, glue sniffing bias.
reply