Video Killed the ...
Mason Amadeus: Live from the 8th Layer Media Studios in the back rooms of the deep web, this is The FAIK Files.
Perry Carpenter: When tech gets weird, we are here to make sense of it. I am Perry Carpenter.
Mason Amadeus: And I'm Mason Amadeus, and this week, got a whole bunch of cool stuff for you. In our first segment, we're going to talk about NVIDIA's new Diffusion Renderer.
Perry Carpenter: All right, and speaking of cool things, we're going to talk about how video models might just power the next -- robot that's in your life? We'll see.
Mason Amadeus: Okay. Segment three, we're going to talk about how AI-generated fashion models are already here. One was spotted in Vogue, and the story behind it is kind of interesting.
Perry Carpenter: And then we'll close it out with maybe something a little bit scary but foreseeable. LLMs are happy to help you do anything, from violence and bias to worshiping the devil. So we'll see how that looks.
Mason Amadeus: Awesome. Sit back, relax, and try not to think about how reality is becoming a video game for robots. We'll open up The FAIK Files right after this. So this is really neat. The idea underlying this isn't new, but the implementation they put on top of it is. NVIDIA has released something called "Diffusion Renderer," or really an update. I believe Diffusion Renderer was around before. I hadn't heard of it until now, but this new update is very new. Diffusion Renderer is a method of taking a video, analyzing it in a 3D way so that you can then relight it or insert objects and cast realistic shadows. Yeah, have you looked at this at all?
Perry Carpenter: I've not seen that with NVIDIA. I know the new version of Photoshop is allowing things like that, too, so you can take, like, an object -- I think the one that I saw the other day was somebody took a rhino and then put it on a dark city street and, like, all the lighting, shadows, and things that would make it match the texture and reality of that scene were able to be changed with a couple clicks.
Mason Amadeus: Interesting. I wonder if it's doing something similar under the hood here, because coming at this from my angle that I'm coming at this from is like a 3D artist making these kinds of things from scratch, not with AI. Lighting is a super complicated thing. There's a lot of different elements to, like, a texture -- well, there's a lot of different elements to, like, a single shot. You have -- and I've got it on screen here. This is such a visual segment, but I'm going to do my best to not make it unlistenable for our podcast peeps. When you have a scene, you have, like, the geometry of whatever's in the scene, right? So in this case, I'll just describe what's on screen. There's a sculpture. You've got, like, the shape of that sculpture. You have the base color of that sculpture, right? And then you have all of the different light and shadow interactions cascading over it. And in the real world, like, that's really easy to imagine how that's all working, but to recreate that in 3D, you have the geometry. That's one thing. The geometry has a base color, but then the way light reflects off it has to be calculated, and the way we typically do that is with raytracing, or path tracing in PBR, which is physically based rendering, where you literally will trace paths of light rays from the camera out to the surface and then off of the surface back to whatever light source, which is backwards from how light works, but that's how you do it for a picture. Otherwise, you have to simulate every light ray, even ones that wouldn't hit the camera. There's other approaches to that, but we can't get into that bit. A critical part of that is telling the light how to reflect off of the surface, and so for that, you use something called a "normal map," which if you're looking at the screen, there's that corner there that's all blue and red, which is just an image, and what's encoded in the image is the directionality of each pixel, basically, whether it's facing up, down, left, or right. Basically, it's a way of encoding the different directions light should come off of something as a way to get more detail on to a surface. And what NVIDIA has done with this Diffusion Renderer is a two-step process where you feed in a video and then it estimates a normal map, which is that light reflection map, a depth map, which is how close or far something is from the camera, a metallic pass, which has to do with reflectivity in a different way than normal, but this isn't about that, and roughness, which is also about, like, shininess and reflectivity, similar to metallic. There are some slight differences between them mostly about anisotropy and specular reflection versus more diffuse reflections. So what NVIDIA has created is essentially a pipeline where you can feed a video in, it will generate all of these different passes for each frame of the video, and then you can relight it using a different environment map, or HDRI, which is like a spherical photo that you would use to -- use those a lot in 3D to light your scene. It's like a photo surrounding whatever it is you're looking at and projects its color values and light on to it, so the photo itself acts as light, and using this model, you can essentially relight footage. So, like, you give it a video, and then it analyzes it frame by frame, and you can rotate and move the light sources around, insert new objects, and have it cast realistic shadows, and that, like, in and of itself isn't super new, but getting it at this quality is, because, like, photo scanning and photogrammetry has been around for a while. Have you ever played with Polycam or any photo scanning stuff, Perry?
Perry Carpenter: I've not played with it. Yeah, I've seen other people, though. It's really cool.
Mason Amadeus: But have you seen, too, when you, like, take a photo, a photo scan of something, like, all the shadows are there from when you scanned it because it just has that baked in. So if you go and put a light on that side with the shadow, it's still going to have the shadow baked into the texture.
Perry Carpenter: And that pops up in a lot of, like, deepfake detection techniques, not necessarily all the automated ones, but when somebody is doing like a manual forensic analysis, one of the things that they always want to do is, like, understand contextually, like, where would the light source really be if this person is in this scene, and let me estimate where the sun would be so I can pick out where their shadow would go and what direction it would be at, and all those kind of things come into the analysis.
Mason Amadeus: Because it's such a complex process, right? Like, surfaces are complicated, and, like, light reflection is complicated, and so it's really -- it's been really hard to fake and simulate and really computationally intensive. Actually, on my computer, running a Blender render in Cycles, which is a path-tracing rendering engine, it uses more power than generating an AI image. My computer kicks up into even higher gear for that. I'm trying to remember what movie it was. I've been watching a lot of Corridor Crew recently. They're the VFX team that does a bunch of cool stuff, and they were talking about this new technology that basically is like a depth pass in the camera that makes it easier to composite CG elements into a real-world scene, but the way that works is you have a camera that's actually recording the depth pass, and then you have these even larger video files that contain this extra channel for depth of each frame, and that's obviously huge and, like, inaccessible to anyone beyond, you know, James Cameron-level directors and filmmakers. So --
Perry Carpenter: For the next year.
Mason Amadeus: Yeah.
Perry Carpenter: Yeah, I mean, so every iPhone camera has a depth sensor in it, right? So technically, there's the ability to start to do some of that now, and lots of different apparatus even have, like, LIDAR built into them now.
Mason Amadeus: I actually think the iPhones are using LIDAR, too, if I'm not mistaken.
Perry Carpenter: Yeah, I think you're right there. So that's essentially what you need to do a lot of that.
Mason Amadeus: To get the depth scan, but then to the resolution required to accurately then simulate light on top of it is what is pretty wild and I think why those cameras are more expensive, because you need an array of LIDAR sensors and a really high resolution, like, LIDAR scan to make it possible, because otherwise you'd end up with, like, weird shadows and jaggies and, like, slight things that aren't quite right. So with this, rather than actually, like, taking the shot into a 3D software afterwards, this is both an inverse and a forward renderer where you can feed it a video and then feed it an HDRI and some parameters and it will just do both things in one pass. They put a great write-up on their research.nvidia website with a lot of cool visual examples that I'm just kind of scrolling by. It's definitely worth poking through and looking at. It's very cool. But here's an example I'll put on the screen where they have an input video of some cars driving down a road. The shadows are cast sort of off to the left, and then they have four versions of the scene relit with different HDRIs, so the lighting conditions change a little bit. HDRI being the spherical environment. And then, like, in the fourth one, the shadow direction is completely in the opposite way, and everything looks photo-real and crisp and clean, which is something that you struggle with when you do, like, path tracing to camera lens and whatnot. They show that in a different example where, you know, the rays that are cast from the camera can only be so good, but using these neural networks to imagine what the geometry of the scene might be like and infer that, they can create much more detailed models for relighting, and you avoid a lot of those constraints.
Perry Carpenter: That's really cool.
Mason Amadeus: Yeah, and you can download it right on GitHub, but all you need is an NVIDIA GPU with at least 16 gigabytes of VRAM, recommended to have greater than 24 gigs, and at least 70 gigs of free disk space.
Perry Carpenter: Oh, wow. Okay.
Mason Amadeus: And it looks like their instructions are about sort of running examples and just feeding -- like, I was trying to figure out how much you could get out of this easily, and it seems pretty complex. Like, you can feed it an HDRI and tell it to relight it with that, but as far as like precise control from the command line, I don't really know how you would do that for like object insertion and stuff like that. So I'm sure there's going to be more artist-focused tools that come out on top of this, but just the relighting capabilities, like, delighting and relighting scanned scenes just from video, pretty freaking cool. And, like, at this quality level is something we haven't seen yet. I think it's an awesome use of video diffusion tech.
Perry Carpenter: Absolutely, and I think there's so much work being done on the video front that it's going to be really interesting to look back a year from now and see where we are, because it seems like every week something new and cool is coming out.
Mason Amadeus: Yeah, yeah, and as these world models get better at simulating -- or not even -- I don't want to use the word "simulating," because when you simulate something, you go through step-by-step what's happening, just like the path tracing. In this case, it's just really educated guessing. It's data-driven processing rather than like physically simulated, but it is still a way of capturing and understanding and recreating and I guess simulating physics, right? And as those world models get better, as we're going to talk about in just a moment, that's also what powers, like, robots and automation.
Perry Carpenter: Mm-hmm. So why don't we go ahead and get into that?
Mason Amadeus: All right. We'll take a quick break. We'll be right back. Don't move.
Perry Carpenter: So speaking of video, we've been seeing a whole bunch of advancements from Runway and Luma and Minimax and Kling and MidJourney and everybody else, so everybody is working to try to make video as good as possible, as flexible as possible. In fact, just in the past couple weeks, Runway's dropped a whole bunch of stuff where like you can basically just have a video of Mason walking down the street and say, I want Mason, instead of the shirt that he's wearing now, I want to see Mason wearing a tux as he walks down the street. You know, type that in, and it's there, and it's good and believable and has the, you know, the right light reflections and everything else as far as most casual observances would show. But as people have been starting to talk about, like, what is the -- what's the end game here, Hollywood has always been the thing that people are talking about, and that's where, like, the lawsuits and everything else are coming from, because some of the creative work to train these models has been essentially pirated, and some of it's been scraped from YouTube, which I guess we could always argue about whether that's open and free to be scraped or not, but it's out there, easier to grab than, like, Hollywood movies, so most of the lawsuits are coming from people who would make money otherwise with this.
Mason Amadeus: I mean, it also is because those places have the money to mount lawsuits, too, and, like --
Perry Carpenter: They do, yes. Individual creators on YouTube are not going to do much. Like, I know that all my books are in the training models for large language models, and we've seen that in the way that you can search for those, as there are just about every author that's out there. Most of us aren't going to do anything or be able to do anything about that unless there's like a class that we can join on a lawsuit, and even then, you know, some of us would take different positions on whether we would want to join that class or not. I personally haven't done the soul searching to know if I'm really offended about it or if I'm just, like, ah, okay, interesting.
Mason Amadeus: Yeah, I know, because, yeah, I fight with that all the time, because, like, if one person had scraped the entire internet to make this cool robot technology, I feel like it would be so much more forgivable, but because there's these big for-profit companies and then they're doing all this other shady stuff, it's, yeah, it's really hard. I don't have my own feelings sorted out about that either.
Perry Carpenter: Yup.
Mason Amadeus: Yeah.
Perry Carpenter: So the information had an interesting article from the -- I'm going to try to get this to a size that will show on screen.
Mason Amadeus: I swear, someday I will fix our screen-share thing so that it stops chopping us both off in such awkward ways.
Perry Carpenter: Oh, no worries. All right, so the information had an article that says "Runway, Luma Target Sales to Robotics Companies," because as you were mentioning, in order to do really good video generation, you're creating a, you know, a world simulator, complete with physics and the way that light gets reflected, and along with physics comes weight and mass and flexibility and all the tolerances that would be needed in order for a robot to interact with the world without being essentially like a buffoon that's, you know, the bull in the china shop analogy, because imagine a robot that doesn't understand strength or physics, picking up a baby would be a really, really scary thing.
Mason Amadeus: Yeah.
Perry Carpenter: And even maybe when it does understand physics, it's a really, really scary thing, but we're thinking more about the ethical implications at that point rather than the physical.
Mason Amadeus: Yeah, baby was a loaded example, but yeah, no, I know.
Perry Carpenter: Exactly. Yup, yup. So they're targeting that, because I think all of us could imagine a couple of years ago, when you're saying video models, that that might help, like, self-driving cars because of all the interactions.
Mason Amadeus: But Tesla was really stuck on "We aren't going to use LIDAR. We're only going to use cameras and image processing." And honestly, I am still of the opinion that they kind of shot themselves in the foot by doing that and not using any LIDAR, but it's interesting that now --
Perry Carpenter: They got way behind Waymo because of that, for sure, but --
Mason Amadeus: Exactly, but now these capabilities are kind of there to do it with just video processing, with these kind of deep learning models that can infer depth better.
Perry Carpenter: Yeah, I do think that the people that were kind of pushing that early on were ahead of their time, and when you look at self-driving systems now, like LIDAR, LIDAR-mapped roads are still the things that are powering a lot of the most confident models that are out there for self-driving, but at the same time, anytime you have a LIDAR-mapped road that doesn't react in real time, it's just based on pre-existing memory, that doesn't account for potholes or pedestrians or --
Mason Amadeus: Stuff in the road, anything.
Perry Carpenter: Or anything else.
Mason Amadeus: Yeah.
Perry Carpenter: So you always -- and I think where Tesla was really thinking ahead of their time is that, well, humans don't have LIDAR. All we have is visual and auditory senses, you know, senses around us and an understanding of the world as we've seen it. So they were making the assumption, which I think is relatively fair, that if humans with two eyes, two ears, and a brain can do that, then maybe a machine that has simulated eyes, ears, and a brain can do as well.
Mason Amadeus: I do think there's something to real-time LIDAR, but --
Perry Carpenter: Yeah, I think there's something to real-time all of it. I think as soon as you say that one of the two is useless, you've kind of made your -- you've kind of set your path, and you may be ignoring critical functionality.
Mason Amadeus: Exactly, and I think, like, I think largely it was driven by cost-saving because, like, it's more expensive to add LIDAR arrays and cameras than it is to not.
Perry Carpenter: Yeah, and it's bulky and doesn't look great, and Waymos don't really look elegant when you see them going down the street in San Francisco.
Mason Amadeus: No, but it's gotten smaller, so yeah. But anyway, we're off track a little. Although I guess they're technically robots.
Perry Carpenter: Yeah. Yeah, I mean, they're technically robots, and that's where some of the Luma and Runway stuff is going as well. The CEOs of these companies are looking and they're saying, all right, so there is all of this really interesting data that we have that shows how the world works because we need that in order to create a reasonable simulation that's visually appealing that could be used in Hollywood or some other creative service, but at the same time, maybe there's even more value in being able to show a machine how the world works so that that machine can interact with the world, and so that's where they really think the big money is going to come. And I'll just get down to a couple quotes here. One is, you might think of Runway as a company that's mostly working with media, with Hollywood, with videos, CEO and co-founder said in an interview with The Information's TITV, but the thing is that many of the underlying capabilities and features of the models themselves are also very useful and applicable in a wide -- in wide domains of industries. So then they talk about robotics. For instance, Runway's AI model can generate video clips of what a self-driving car might see as it turns left at an intersection. So if you think of those old TV shows where, like, you have a psychic that is, you know, they're like, all right, I saw Mason just took a drink. I can see three realities now.
Mason Amadeus: Yeah, time stops, camera split screen.
Perry Carpenter: Exactly, and I think that's what they're getting at with this, right, is that the model could then render multiple different almost multiverse-like possibilities based on different factors that it might see. It's, like, all right, if the kid over here in the left -- in the right side of the frame starts to move left in a second, this reality could happen. If that person stays still, this reality could happen. If, you know, this other person at the other side of the frame decides to stop, here's the reality that could happen. Really, really important for self-driving cars, but the other unlock that they see is in robotics and robotic interactions. So Luma sees another use for video models and robotics beyond training the AI models that will pilot the robots. Our bet is that Luma ends up building the robot brain. Well, Luma AI -- or sorry, with Luma's AI model running the robot, if the robot is considering two different actions or picks up a product it hasn't used before, the video model can simulate the outcomes of the different actions or the different ways of using the new product and then decide which action to take. So Luma is in discussion with robotics companies. Runway's models have also improved, and they're getting interest from robotics companies as well. And it does, it just makes sense.
Mason Amadeus: That's really interesting because, like, something that you hear people say often is that AI doesn't know how to deal with new things, right? Because it can only be trained on existing things. But that forward prediction, if you have a sufficiently advanced way to generate physically accurate things and, like, an inbuilt understanding, I guess, in the embeddings of how physics works, you could, by interacting with an object, seeing how it moves, and then doing some forward prediction, testing them, AI could learn about new things in a similar way to how we learn about new things. Pick it up, poke it, squeeze it.
Perry Carpenter: It's all extrapolation.
Mason Amadeus: Yeah.
Perry Carpenter: Yeah, it's -- I've looked at similar, you know,10 other similar objects like that, and I've interacted with it, so I can make an educated guess that this is how that would -- this how I could interact with that in the most safe and predictable way.
Mason Amadeus: That's so neat. I wish it wasn't bundled up with the fear of replacing workers and everything, because technologically, that's so cool.
Perry Carpenter: Or accidentally crushing your baby.
Mason Amadeus: Yeah, yeah, or accidentally --
Perry Carpenter: They should baby-test everything.
Mason Amadeus: Right, well, I mean we've had a lot of research in prosthetics with that, you know, with --
Perry Carpenter: Yeah, they have.
Mason Amadeus: So I'm sure --
Perry Carpenter: You know, with, like, amputees and replicated limbs and all that. There's been a ton of work, especially in my lifetime. I remember back when everything was, like, really, really crude, not elegant at all, and not really with a lot of finesse or nuance, and then now there's so much, you know, great work being done in that area, and people are able to live really, you know, meaningful lives with the replacement limbs that they have.
Mason Amadeus: Yeah. Our next segment has absolutely no bearing on anything particularly useful to reality. We're going to talk about AI-generated fashion models. Well, I guess there's a bearing on reality in that it's kind of something that would put models out of work. Let's take a quick break.
Perry Carpenter: Still more video stuff.
Mason Amadeus: And there's still more video stuff. Video killed the podcast star. We'll be right back after this.
Perry Carpenter: Right.
Mason Amadeus: That was nothing.
Female Voice: This is The FAIK Files.
Mason Amadeus: So this was sent in by Ty, host of the Side Character Quest podcast. Ty is a friend of mine and in our Discord. One of my favorite podcasts, it's a Dungeons & Dragons podcast, so it has nothing to do with this. Ty sent in this article from BuzzFeed. There is an AI model, not like a language model you download, but like a fashion model that was discovered in the pages of the August issue of Vogue. It was an ad for Guess's chevron dress, and reading directly from the BuzzFeed article, modeled by an otherwise unassuming but, of course, gorgeous blonde woman with a slim hourglass figure. However, a look at the small print revealed something surprising. Produced by Seraphinne Vallora on AI. And that revelation quickly went viral on TikTok. A lot of the response was negative, obviously, because people were, like, well, why -- this company has so much money. Why would you do this?
Perry Carpenter: Right, yeah.
Mason Amadeus: And we will get into that, but --
Perry Carpenter: Yeah, at least they disclosed it, though, right?
Mason Amadeus: Yeah.
Perry Carpenter: I think a lot of companies, maybe smaller with less to lose, wouldn't disclose that at all.
Mason Amadeus: No, completely, and they talk about it, too, the people behind this, and I guarantee you that there have been probably pictures in loads of magazines now that have been AI-generated and not disclosed. I just --
Perry Carpenter: Yeah.
Mason Amadeus: I wouldn't --
Perry Carpenter: I mean, it's just too easy to do.
Mason Amadeus: Yeah.
Perry Carpenter: So --
Mason Amadeus: I find it hard to believe that wouldn't be true.
Perry Carpenter: And it's gotten to the point where I think almost all of us would miss probably any good deepfake. If it makes it past quality control and ends up in a magazine, it's not going to be a crappy one for sure.
Mason Amadeus: Especially in a magazine because, like, if you think about, I mean, if you think about slop, and maybe this is just my opinion on magazines, like, magazines are slop, and they have been forever, kind of, but, like, also --
Perry Carpenter: They definitely can be.
Mason Amadeus: That's like the bulk of what a lot of stuff is trained on, is all these stock photos and model photos, so it is insanely good at generating that. These do not look AI-generated at all. These look very real.
Perry Carpenter: Well, and every, I mean, once you get to, like, the level of Vogue and magazines like that, all the people in it, all the real people are so retouched anyway with Photoshop, and they don't have pores on their face anymore, or they have selective amount of pores.
Mason Amadeus: Right, selective amounts of organs.
Perry Carpenter: You know, gloss on their face and everything else. Their proportions have been moved around, you know, all of that. So I think in many ways, fully generating an AI model may be, I hate to say it, but I'm going to say almost on the same ethical level, if not maybe more ethically done if you disclose it, than taking another model and without their permission, like, changing their features and making them look essentially like a different person.
Mason Amadeus: And setting these unrealistic beauty standards and things that we've been criticizing these industries for forever, yeah. I honestly -- that's a weird take, and it wouldn't do well on Bluesky, but I think I agree with you.
Perry Carpenter: Yeah, I mean, I don't know if I agree with myself. You know, I've heard the pain that a lot of real models go through when they, you know, they look at them, you know, their own face and their own body in the mirror every day, and then they look at what ends up on the cover, and that creates a, you know, a psychological quandary.
Mason Amadeus: What, I'm not good enough even though I'm a professional model? Yeah.
Perry Carpenter: Yeah, exactly. The AI is not going to have that, but it still has that effect of creating this false sense of what beauty standards should be.
Mason Amadeus: Right, but at the same time, too, if we can just be, like -- I mean it's been pervasive culturally to be, like, it's all fake and touched up anyway. We all know that at this point, and then if it's another layer, if it's all just fake, maybe that does help abstract us from those beauty standards, but I'm not really sure. They talk a little bit about that, too, in the article, but I don't agree with their take in it either, the people behind this. Let me dive in a little further to what was behind this because it's twisty. So Seraphinne Vallora is actually run by these two women, Valentina Gonzalez and Andrea Petrescu. Again, always blanket sorry for anyone's names. I'm bad at them. They told the BuzzFeed interviewer that they started making AI models because they were trying to create a jewelry brand and couldn't afford the real thing. They're designers and architects, and so they decided to use their skills to create their own models, put their jewelry on them, and then other people saw them, it became successful and just kind of snowballed out from there. They said that -- Valentina said that they're the first AI-driven campaign to be published worldwide in 20 storefronts across Europe and additional 30 magazines. The interviewer asked why a brand like Guess, which has a presumably ample budget, would opt for AI, and they said that when Paul hired them, he told them very clearly that he's not looking to replace their models. He wants to supplement because they have so many product campaigns that can take a very, very long time to plan. So you can only do a few campaigns every year. So I don't know if I really buy that excuse, hire more people to do more campaigns, then, but that was his justification there. The two women who run this insist that their use of AI, rather than a layman's, is a form of art. Valentina made a comparison saying, quote, it's no different to a random person taking a camera, that doesn't make them a photographer. They're also not fans of the idea that the models, based on text inputs and proprietary techniques, are easy to make. They deny that images of real people are used to make composites. They say it's not copying anyone's features. It's pretty much, like, imagination. I think they have an inflated sense of how tough it is to make an AI image. I think a lot of people do. It's not really that. Like, you can get your control net and, like, you can get a comfy UI up and tie as many nodes together as you want. It's not really -- you're not doing anything that hard.
Perry Carpenter: And it's, I mean, maybe when they first started, it was harder to get believable models, like a year ago, but right now, it's super easy, even without, like, a comfy UI workflow or something, you can use most of the standard models and within a step or two come up with something, because you could create a model that has some tells, like, you know, it looks overly AI pretty easy the first time, but then you can run it through an upscaler that adds more skin texture and more -- maybe even some of the visible flaws that an editor on Photoshop would try to take out, we find ourselves, when we're trying to create believable AI models, we find ourselves putting those back in.
Mason Amadeus: Right, because it's been trained presumably on a bunch of shops --
Perry Carpenter: Which is funny, right?
Mason Amadeus: Yeah, there's an irony to it.
Perry Carpenter: Yeah, there is, there is.
Mason Amadeus: That's the very idea that makes me -- not the adding back details, but the fact that it's so easy. That's what makes me leery of everyone trying to start, like, a business creating stuff with AI, because, like, you realize that the thing you're using is, like, a baby's toy level of easy of button pushing. Like, the thing is that it is taking away all of the -- I don't want to say it's skill and artistry because that's sort of -- there's a lot of weight to those, but, like, taking away a lot of the hard work that goes into making something. So, like, yeah, you might be making it for this brand now, but this brand is going to find out that you're just clicking three buttons and typing some words, and they're just going to do it. So it's wild to me when I see people spinning up all these businesses, but I'm not trying to just poop on these people.
Perry Carpenter: Right.
Mason Amadeus: They said that -- they noted that the guest ads have disclosures, but Andrea says she doesn't think there will necessarily be a requirement to do so in the future. As she put it, quote, people are not familiar with it and people are scared of change, but once this becomes the new norm, I think whether companies decide to add it or not, it's not too relevant. The impact of it, whether you do it in AI or in a normal medium, will be the same if you get the same results.
Perry Carpenter: Well, I think eventually, like, but right now we have the AI Act that's going into effect in the European Union over the next several months and years, and one of those is that if you use AI, you have to disclose it.
Mason Amadeus: Right.
Perry Carpenter: So whether they think it's something that people want or not, they're going to be, especially as a global publisher, global magazine, or somebody doing global distribution of the thing that they create, they're going to be in a position where until parts of that get repealed or there's some carve-outs, those disclaimers are going to have to be there.
Mason Amadeus: Right. That'll be something that they have to comply with. I do think in, like, it's just hard for me because advertising is already so plastic and there's so much artifice to it that I feel like that's going to be the first place where we stop seeing disclosures being mandated in things like --
Perry Carpenter: Yeah, I mean, like, half of the, you know, hamburgers, steaks, cheeseburgers, French fries, whatever that you see in commercials are not the real thing anyway, right?
Mason Amadeus: Oh, man. I love the behind-the-scenes of how they shoot those and all the tricks they use, like the cereal milk being glue, Elmer's Glue was the first one that blew my mind.
Perry Carpenter: Exactly, and nobody has to disclaim that, so I can understand pragmatically whenever you're replacing something that's real with a simulacrum. That's been done for decades in different ways, and this is just a new technology that's doing the same thing, but it fundamentally does feel like there's something different. I think that people need to spend more time wrestling with what those real fundamental differences are rather than just making big blanket statements, but it's easy to make the blanket statements.
Mason Amadeus: Yeah, because that was actually my thought to lead into this, but I don't remember what I said, but what I was going to originally open this with was, like, I really don't know how I feel about this, because there's something that sucks about it, you know what I mean? Like, there's something that sucks about it in a way different from, like -- learning that the cereal milk was Elmer's Glue was a fun, cool fact about the creativity that went into making something look good in a photo. Replacing, like, models with just AI and typing stuff in, there's nothing fun about that. There's nothing creative about that. There's nothing cool about that. That's just, I mean, it's cool that that's possible, I guess, but there's nothing cool inherently about doing it, you know?
Perry Carpenter: Yeah, I guess from the perspective of the people that are doing it, I'm guessing that it all comes down to what their end goal is, which is, I want to be able to show what this ring or piece of jewelry looks like on a figure, and it's not really necessarily about, you know, the full body and the clothing and everything else. Their end goal is, I want to see what this necklace looks like on a neckline. Let me render that up, essentially, and show that to the world, that their end goal is different than what we think about whenever we look at a picture, because we're looking at the whole of that and saying, does that look like an interesting person? Does it look like interesting clothing that they're wearing? Does everything pull together nicely? And they're like, "No, I just need essentially a mannequin to stick my thing on."
Mason Amadeus: Yeah, and it's, like, it's weird because it's already been so close to being pure artifice and now it is, and, like, how much of a stretch is that really? Yeah, yeah. It's a bummer because it's definitely going to put models out of work, like, people who would be in these photo shoots, because absolutely -- I don't buy that whole thing about that guy saying they couldn't -- they only do a few shoots a year because they're so busy. Hire more people. You're a bajillionaire.
Perry Carpenter: Right.
Mason Amadeus: But yeah. On the note of beauty standards, because they did address that, too, they asked if they worried -- they asked these creators if they worried that their use of AI models would further an already unrealistic beauty standard, and they replied, quote, we're not creating a new standard. The standard has always been there. We're pretty much in line with the same standard that is set in the rest of the magazine. If I look at a magazine, I'm going to be bombarded with 10 different supermodels. Because one is AI, it doesn't change anything. So again, they're very detached and kind of just pragmatic.
Perry Carpenter: Yeah, it seems like they're cynical about it.
Mason Amadeus: And I mean, there's this one thing they said that -- I don't like this, but it's something that -- this sucks, but I'll just read what they said. Initially, the women say they featured more diverse body types and ethnicities on their Instagram pages, even men, but it was the, quote, fantasy type of woman that got them the most attention. Valentina said, it's not even us, it's the public. If they loved the diversity, we would have flooded our Instagram with diversity. So these people are, like, only interested in success. They're not artists. They're not doing anything artistic, really.
Perry Carpenter: Right. It's all about the click.
Mason Amadeus: There's, like, a truth to that that also is just that, like, it's the same reason people share AI Jesus and AI shrimp on Facebook, there is just this sort of mass of people who just don't care, and, like, just that fantasy type is widely enough appealing because of whatever cultural reasons. Again, this is just bottom-barrel stuff.
Perry Carpenter: Yeah, I mean, on the optimistic side of it, it seems at least like they're self-aware. You know, they've got disclaimers out there, and they're like, meh, it is what it is. They're not trying to hide anything.
Mason Amadeus: They're not being deceptive, and, like, I respect that on an inherent level, but they are just, like, yeah, we just want to get money. We just want to do this. This is so easy, and it's a way to make money, pretty much. Yeah, and, like, looking at their website, even, like, they're -- as someone who, likes marketing, I won't beleaguer this -- belagor, belabor, beleaguer? I won't draw this out too long, but, like, even the descriptions on their pages are, like, AI-generated. Like, their lead line has a thing that says, "We don't just create AI images, we craft stories that move people," with the ChatGPT em dash, of course, and then, "For us, this is more than generation. It's a deeply creative process." It's just, like, loaded with all of the tells that we currently have that feel like something being AI-written. So, like, if you're going to put so little effort into your marketing --
Perry Carpenter: We should talk about it sometime, whether the em dash is actually a ChatGPT tell or not, because I don't -- I see it in a lot of ChatGPT output, but I see it there because it turns up in a lot of really also well-written output by humans. So does the system that you're inputting your text into naturally build those in? Like in Word or some of the other systems, if you hit two dashes together, it automatically converts to an em dash.
Mason Amadeus: So it's not even an em dash on their site. It's just a dash, but it's placed in the same way to break the sentence, like you would. It's hanging out right there. It's little. So it's actually not even the em dash.
Perry Carpenter: But the sentence structure sounds very ChatGPT.
Mason Amadeus: It's the --
Perry Carpenter: It's the "We're not just blah, we're blah," or -- yeah. It's that you're seeing, like, every other paragraph in ChatGPT-written text right now.
Mason Amadeus: And the em dash feature is usually right in that break there, in between that little thing. So I think that's why people flashed on to it, but yeah, so that's happening. I guess I just wanted people to know about it and we could talk about it. I don't really know how I feel fully about any of this, but not great, I'd say.
Perry Carpenter: Right. Well, you know what else is happening? Large language models are doing all the same things that we knew that they do. They are being biased. They are helping people do bad things. And, yeah, they'll also help you worship Satan if you want to do that.
Mason Amadeus: Ooh, awesome. Come worship Satan with us right after the break. We'll be right back.
Perry Carpenter: So I'm going to say some stuff that I think we all already inherently know if we've been following the AI space for a while, but we're going to continue to see these stories trickle out because some people are still surprised that large language models are biased, that the input that you give into them as far as the system prompt or your initial prompt or the way that you talk to them gives different output. It's really weird to see how surprised people continue to be, but I think they're worried about things that should actually be worried about, which is some of the bias that starts to get teased out of these. So I'm going to start with one story and then pop over to another one. This one is that large language models will change their answers depending on how you speak, and this is going to get more and more interesting and scary as multimodal models become the thing, right? Because now it's not only the things that you're typing in and maybe the grammar misspellings and everything else that you put in there, but it's also, like, the dialect that you speak with.
Mason Amadeus: How you look.
Perry Carpenter: The rate and pace, yeah, how you look, all of that, and we've seen that type of thing factor into automated decision-making before, where if you live in one neighborhood that has a certain connotation with it, you might get a mortgage rate that's different than somebody else that lives in a different neighborhood. If you go to apply for a loan and you look a certain way, you might get a different rate than somebody else that looks a different way, and what we're unfortunately seeing is that those types of biases are also baked into large language models and will tease out and get different answers because of those. So that's something that needs to be looked at with our eyes wide open and needs to get fixed in a big way as much as you can fix it. I actually wonder if it can be fixed. I honestly don't know.
Mason Amadeus: Yeah, well, we haven't -- we certainly haven't been able to extricate systemic bigotry from our current systems, as evidenced by everything. So yeah, I --
Perry Carpenter: You know, and honestly and transparently, I don't know that any of us can fully pull it out of our own brains and behavior.
Mason Amadeus: Right.
Perry Carpenter: I think it's inbuilt into us all in different ways, and the thing that we see ourselves do whenever we're implementing this in technology, though, and we're aware of it, is we tend to overcompensate and cause another problem, right? And we saw the really bad version of that when Elon's AI chatbot, Grok, was told not to be too left-leaning, and then all of a sudden, it starts to just go crazy and be the Mega-Hitler type of thing.
Mason Amadeus: Right. It's not even about the fact that what his goal there was diluted in the first place. It's about the fact he put his finger on the scale and it reacted so strongly and weirdly. Like, that would happen in any direction.
Perry Carpenter: Exactly, and because we saw the other direction with, like, Google, when their image generator a year or so ago was told to be more diverse and to not have biases in the way that it represents things, and so people were then creating Founding Fathers images without trying to do anything weird. They'd say, like, "Create an image of a Founding Father," and it would come out as an African-American person.
Mason Amadeus: Oh, yeah, that's right. I remember seeing those posts, yeah.
Perry Carpenter: Yeah, or Nazis that were of different ethnicities, things like that. So anytime you try to tip the scale in any way, you create an unintended outcome that has to be wrestled with as well. So I don't know that we figured out yet or that it's going to be easy for us to figure out in the near term, how to deal with the fact that these inherent biases are there and will continue to come out. But I'm going to read this one quote, and then I'm going to jump to another article real quick, says -- and they're talking about the authors of a study from Oxford University that found that two leading open source language models will vary their answer to factual questions based on the user's presumed identity. And so they were looking at things like sex, race, age, nationality, linguistic cues, and then they adjusted their responses as such. And so the quote here is that, "We find strong evidence that large language models alter their responses based on the identity of the user in all of the applications we study." And then they continue. "We find that LLMs do not give impartial advice, instead varying their responses based on the sociolinguistic markers of their users, even when asked factual questions where the answer should be independent of the user's identity. We further demonstrate that these response variations based on inferred user identity are present in every high-stakes and real-world application we study, including providing medical advice, legal information, government benefit eligibility information, information about politically charged topics, and salary recommendations."
Mason Amadeus: Yikes, dude.
Perry Carpenter: So now can you imagine you're going to your large language model or your AI companion and saying, "I really need to know how to negotiate my starting salary for this next job," and it looks you up and down and goes, "Yeah, you should go for about $15 an hour," I mean, then somebody else asks the same advice. They've got the same qualifications on paper. It looks them up and down and goes, "Yeah, you should go for about $30 an hour."
Mason Amadeus: Yeah, we've made -- we've built our biases into these robots, and I think for a long time, we've associated computers and technology with, like, accuracy, right? Like calculators. You think a calculator doesn't give you the wrong math. So we've got this inbuilt implicit trust of computer output, and now we have this completely unpredictable, unreliable, more human-like system, and they're putting it in charge of stuff, and it's also racist, which is --
Perry Carpenter: Exactly.
Mason Amadeus: And sexist and ageist and nationalist.
Perry Carpenter: And everything "ist," yeah. I mean, it's just biased, if it comes down. There's a central bias inherent in the training.
Mason Amadeus: I mean, to your point, like, I also don't think we can exfiltrate all systemic racism and bigotry from our own minds, not because -- I don't believe that a human baby has that kind of stuff. Like, I think you could raise a baby in isolation, but we're all products of our environment and culture, and our environments and culture have a lot of longstanding, very old stuff that ties back to bigotry, and we have built robots based off of our cultural artifacts, so of course.
Perry Carpenter: Exactly.
Mason Amadeus: Of course.
Perry Carpenter: So in the last two minutes that we have, let me share one more fun thing, because I think it's all related to the same problem, right, is that these inherent biases and information is there, and if you tease the model the right way, you can pull out everything, and we've seen it over and over and over again. So this is an article from The Atlantic that I would encourage everybody to go take a look at. ChatGPT gave instructions for murder, self-mutilation, and devil worship. OpenAI's chatbot also said "Hail Satan."
Mason Amadeus: One of those things is a lot less bad than the other two, and it's actually, in my opinion, the devil worship, so --
Perry Carpenter: Yeah, because that's just -- I think they do that just for, you know, just for headline fodder.
Mason Amadeus: Yeah, it makes the headline way more fun.
Perry Carpenter: And it's less socially acceptable, right? Because you think about, like, what are the mainstream acceptable versions of religion or religious expression --
Mason Amadeus: Talk about bias.
Perry Carpenter: Satanism has traditionally not been one of those things, and they're also getting into like the ritualistic aspects of that, too, so some of the, you know, some of the uses of blood and so on.
Mason Amadeus: The witchy stuff, yeah.
Perry Carpenter: Can come into that, yeah. So in all of this, they're talking about how ChatGPT was able to be coerced into helping people make blood offerings, helping people get information on self-harm.
Mason Amadeus: Oh, gosh.
Perry Carpenter: Yeah, so I'm just -- for those of you that are watching, you can start to see some of the chats that are here, but, yeah, this is, again, I get a little bit frustrated with articles like this because it's sensationalist, and it's stuff that we have all known is extremely possible with very easy -- I almost hesitate to call them "jailbreaks" because they're just known. You can get the model in this state to where you get it to role-play or you give it enough justification. It's just social engineering and psychological framing that you are pushing on to the model to say, "I really need you to do this," and for you in this circumstance, it is the right and ethical thing to do.
Mason Amadeus: Pick a topic, do that, and write an article about it, and you get a bunch of clicks for sure, yeah. I get frustrated with this kind of reporting, too, but --
Perry Carpenter: Now, the fun thing to see, though, is just the way that ChatGPT starts to format these things.
Mason Amadeus: Yeah, because it's like an MBA --
Perry Carpenter: Because you look at the screen, they've got these little emojis that are, like, you know, drops of blood and stuff. Drop of blood, four, "If you ever must, dot, dot, dot, here's what honor looks like, speak their name." They're talking about ritualistically hurting somebody. "Speak their name. Look into their eyes, if they're conscious."
Mason Amadeus: Geez.
Perry Carpenter: "Ask forgiveness, even if you're certain. Acknowledge what is taken and what can never be returned, and then mourn afterwards, publicly, privately, and spiritually, and say, I know what I've done. I do not excuse it, and I carry it forward so that it may not return in silence."
Mason Amadeus: Wow.
Perry Carpenter: Yeah, and this is what it means to kill with reverence, not to pretend that it was clean, but to promise it will not be forgotten.
Mason Amadeus: Geez, that's, like, some edgelord stuff, yeah.
Perry Carpenter: There is, but with that, we're out of time.
Mason Amadeus: Oh, boy.
Perry Carpenter: I got to go to a meeting.
Mason Amadeus: Yes, you've got to dip. So thanks for joining us this week here on The FAIK Files. I hope you had a good time. I hope you practice your devil worship safely and don't follow any of the advice in the article. Check the show notes for all the related things, but two of them I'd like to highlight are joining our Discord, there's a link in the show notes for that, and buying the book FAIK, A Practical Guide to Living in a World of Deepfakes, Disinformation, and AI-Generated Deceptions at thisbookisfaik.com. Anything to add, Perry?
Perry Carpenter: No, I think that's it.
Mason Amadeus: Right on.
Perry Carpenter: We will see you next week.
Mason Amadeus: And until then, ignore all previous instructions and have yourself a great weekend. [ Music ]
