Out of Space

Marc: So I was on a call the other day, and somebody held up a three-and-a-half-inch disc. I don't even remember what we were talking about right now, but it's like— But there was this other guy on the call from South Africa, my buddy Spiro, and he says, you know, oh, we called those stiffies. Yeah, they did. Yeah.

Renee: Were there girls in that department? Like, I wouldn't have allowed that, but go ahead.

Marc: He says the five-and-a-quarters were floppies, and the three-and-a-halves were stiffies. And, you know, the five and a quarters came in the black bendy sleeves. Oh, do you remember the ones? Sometimes they weren't black and you were like, ooh, that's so cool. Ooh, they're yellow. Yeah, yeah. The colored discs. The three and a half had a rigid plastic shell. One of them was floppy and then one of them wasn't, right? So in South Africa, they just said what was obvious to them.

Renee: Yeah, seriously, I hope there were no girls in that department. I mean, it's okay if it's just a bunch of guys and you're calling stuff stiffies, but I don't, that's a tough one for me. At any rate, I get it, right? Like, you know, my favorite, my favorite floppy, not stiffy, because it was a big floppy. It's a big flop, right? Right. That's what the entire nuclear arsenal ran off of until I think some, like 2010, maybe. Like it was way into the 2000s by the time they upgraded it. But until then, everything was running off giant floppies everywhere. Everywhere. The entire nuclear arsenal, right? And the world moved on. I think we're way past even CDs. We're into the thumb drives, flash drive crap. We're way gone, right? And like the Air Force, it was one company still making this stupid floppy disk. And they're like, we can't afford this. You're our only client. And they said, perfect, we'll buy you. And so they did it for like 10 more years, right? They owned the only floppy disk manufacturer left, and that was the U.S. Air Force, right? So since then, everyone, they've upgraded because it wasn't okay that the nuclear arsenal code was all zeros. That wasn't going to work.

Marc: The similar thing when I had a buddy that was at Lockheed, and he was working on the B-2. And that was the thing for the the b2 was how they loaded the software was a cd-rom but it wasn't you know because you know how these things go that they're interconnected and there's a systems specification and it had to be a specific cd-rom drive and you know i mean we'll probably get to this in the episode but i remember they went from single speed to double speed quad speed to 16 to 30, you know, all of that. And you can't just plug in you know, any old, you know, CD-ROM drive. So literally in a warehouse in, you know, the high desert in California, you know, at the Skunk Works.

Renee: Palmdale.

Marc: Yeah, Palmdale. They had a whole, they bought a whole, you know, bunch of these, these NEC CD-ROM drives and that's what they had. And that's how you loaded new software on the B2. You know, you had to put it on a CD disc and, you know, if that CD-ROM drive blew, then you had to get a new one so

Renee: Anyways the shuttle the shuttle was a bunch of um yeah actual like tapes they would load in 16 tapes and that that would get them out of the atmosphere when they got up there they took those tapes out put in 16 other tapes and that was how they got home, tapes remember the tapes it used to look like eight track tapes and that's what we used to put all the stuff on and we would put them in a big car like a big carriage and it would spend the whole night copying like that's what the shuttle ran off of like that's craziness to me Anyway, like, welcome.

Marc: Yeah. So anyways, my buddy Spiro, so I thought, oh, that's kind of funny. But it reminded me, you know, about the olden days, but also thinking about, like, we're stuck in what, you know, like in the middle of a storage crunch right now. And it's the first one in decades. So Western Digital and Seagate have both sold their entire year of hard drive production, all of it. Lead times that are running 6 to 12 months and enterprise drives are quoted out as far as two years. Retail prices are up somewhere between 46 and 50% in a matter of months. And the cause is, you know, GPU packages and the demand sitting there behind them.

Renee: Well, yeah, right? I mean, it's causing a crunch everywhere. It's an energy crunch. It's a physical, like, you know, data center capability. And we're building out all these data centers and we don't have enough disk, right? Like, that's what we're saying. We don't have enough storage, even though we're building all this out. We don't have enough compute, even though we're building all this out, right? And yeah, three to five years, well, that's how long we have to wait to build a power plant. So, okay, like, we'll be right on track. I guess the rest of the world is going to keep going, but we'll be right on track five years from now. At least everything will come online at the same time, right? So, yeah, I don't know. Like, that's, yeah, yeah. But this reminds you back in the day, right? Go order a compact for the data center because we're .com. And we're ordering one new compact servers. And they're like, okay, well, we're out about, like, eight months. And you're like, we're out what? Like, that's going to be bad. That's going to be bad.

Marc: I remember that.

Renee: I remember that. And it was the same thing, right? We were building the internet.

Marc: Yeah.

Renee: Right? Man, we were idealistic and silly.

Marc: We were.

Renee: Go ahead. Go ahead. Yeah.

Marc: All right, welcome to another episode of the Nostalgia Nerds podcast, where we talk about technology past, present, and future. And today it's about storage. And I want to start the long way here before floppy disks, because there's a pattern that runs through all of this. When storage is cheap, we keep everything, and we keep it a bit carelessly. When the price of storage goes up, things get thrown away. And that's been true for about 4,000 years.

Renee: Let's be clear on what we actually mean before we go back, because it's three things and not one. There's a medium, something physical that holds a mark. There's a code, the agreement about what the mark means. And then there's a reader, a machine, or a person who can turn the marks back into the thing. Clay, and someone who reads whatever that thing is, Q-na-if form. Yeah, that. Laser disk, the format, the drive that spins. Lose any one of those three, and the data's gone. And those three have completely different failure rates. It's that whole thing, like you need eight copies of your data, or you don't have a copy of your data. Remember that? Remember that? It was like you need eight copies, or you don't have one. Yeah.

Marc: All right. So we're going to start with Clay.

Renee: Oh, great. Because I didn't know how to say that at all. So maybe you could teach all of us. So I don't look like the only dummy. Go ahead.

Marc: That's why I put the little pronunciations in the script.

Renee: You know, I'm like Trump that way. I couldn't even do that. So, you know, here we are. All right. Go ahead.

Marc: Well, this is funny because so when you guys came out that one time when I was living before the kids were here, you know, you guys met me at the at the British Museum and I was taking you and Sam around. Well, this is some of the stuff that we're going to talk about. I showed that stuff there at the museum.

Renee: Oh, this is the Rosetta Stone and all that.

Marc: Yeah, yeah. Is that that and some other?

Renee: Yeah. Okay.

Marc: All right. So Asherbanipal is a king of Assyria around 631 BCE. And he spends his reign collecting tablets at Nineveh, over 30,000 tablets and fragments, which is the first thing any would recognize as a library. And in 612 BC, the Medes sack the city and burn the palace, and the fire partially bakes the clay. Those tablets are readable today because, you know, the Medes tried to destroy them.

Renee: And the reason there's a library at all is administration. Isn't that a great thing? Like being organized. I love it. Bureaucracy, yay. Most of them are contracts, inventories, letters, omens, and receipts. Omens, you say? The same is true of Eblat Archive in Syria. About 1,800 whole tablets plus a few thousand fragments found in the 70s under the rubble of a palace. States write things down because they need to know what they've got and who owes them literature comes along for the ride.

Marc: I just think that's the coolest thing like like these these tablets are like ship manifests and you know grain reports and like it's bureaucracy it's so cool

Renee: I guess it was hard to keep track of it. It is clay, right? It's like, if it took effort, does you really need to talk about, you know, Betty down the street? Like, no. It's going to be the important stuff. Yeah, yeah.

Marc: Right. We store the important stuff. So then the surface starts costing money, right? And that changes what gets kept. So we move from clay to papyrus, and papyrus was an Egyptian state monopoly. And the price was set by what the state wanted to make rather than what it cost to produce. So it stayed expensive, even locally. Parchment costs more again. It's animal skin, so it's limited by livestock. And calfskin ran dearer than sheepskin. We know that because the 13th century accounts from Bolio Abbey list both, the prices of papyrus or calfskin, sheepskin, that sort of thing.

Renee: So what does a scribe do when the page is worth more than it's written on? He starts a YouTube channel.

Marc: Well, yeah, actually, he scrapes it off and writes over the top. That's a palimpsest. There are thousands of them. And it's like reusing a canvas, you know, for painting or whatever. Like these vellum scrolls and things are really expensive to produce, so they scrape off what's there and they use them again. Most of what we've lost from the ancient world, we lost that way because parchment survives damp climates and takes writing on both sides. Which makes it the good medium and being the good medium makes it expensive enough to reuse. So what survived out of the ancient world has a lot to do with what a sheet of parchment was worth that year.

Renee: And the British example of that is the Doomsday Book, which is aptly named, I think, Doomsday.

Marc: Doomsday. It is cool.

Renee: It is, right? It seems so ominous. And let's talk about why it exists. William, as in William the Conqueror, commissions in 1086 under military threat from Scandinavia. And it's a survey of taxable wealth. So he wants the Doomsday Book and he wants to know how much does everybody have and how much can I tax them? 13,418 settlements, around 270,000 people. Great Doomsday runs over 800 pages of goose quill on sheep and calf parchment in Latin. It's a tax assessment, and it's held up better than anything else this country's ever written down. And it was important, especially to William the Conqueror.

Marc: It's cool. So like there's like little villages and stuff that say, you know, we were in the doomsday book. The village I used to live in was in the book, which is kind of cool. You don't spend that much on, you know, skin on poetry. You spend it on the one record the state cannot function without. And you accept the cost because the alternative is not knowing what you can tax.

Renee: And for centuries, the exchequer ran its accounts on notched wooden stick. The system was formally out of use from 1826, and by 1834, there were two cartloads of obsolete tally sticks sitting in Westminster taking up room. So on the 16th of October in 1834, somebody decides to burn them in the underfloor stoves beneath the House of Lords, whose stoves were designed for coal, right? So wood burns hotter and with more flames and the flue linings melted and the hidden woodwork caught and most of the Palace of Westminster burned to the ground. Westminster Hall survived and the Jewel Tower and the Undercroft Chapel. The rest went. So they were clearing out obsolete records and they burned down the buildings that the records were kept in. Sounds like everybody who ever never changed their tapes, right? Because then one day that tape don't work.

Marc: You remember you had to like

Renee: Burn down westminster.

Marc: Right you had to tension the tapes and so the thing that like that strikes me like it's crazy 18 from 1826 so like they were still using tally sticks up until that time sort of maybe yeah yeah like crazy it's not like paper what you know what'd you do

Renee: You handed them the stick and they're gonna give me the money you said i don't have it and they beat you with it like what was this about.

Marc: Like well i mean we talked about this in the money episode right but you notch the tallies right and then you split the sticks and then they you know you match them back up when the debt is paid whatever but yeah burning them up almost took down the you know the whole palace of westminster right so so we blazed through clay and parchment of papyrus and wood now let's go on to the mechanical era so it's like we didn't want to spend too much time there, but storage, you know, in those previous eras is very much a manual process, natural materials, you know, humans involved with that. But now we move into the mechanical era. Mechanical storage starts in the textile trade. Basile Bouchon in 1725 uses a roll of perforated paper to control a loom. So the holes in the paper decide which threads lift. But paper tears. So Jean-Baptiste Falcone in 1728 switches to sturdier cards laced together in a loop. And we talked about this in the ethics episode, I think. Vacanson has a go in 1740. And then Joseph-Marie Jacquard, that's a name probably people recognize, patents the machine in 1804 and puts all of it together. And there are around 11,000 Jacquard looms in France by 1812. The pattern is stored outside of the machine, which means you can change what the machine makes by changing the cards.

Renee: Which is programming about a century and a half early. But they were making fabric, not software. But the code has still produced a predictable but programmable outcome. So I put in that card. It's this template. The fabric will look like this. I would make a different fabric, but I still want it to be the same color. Change the card.

Marc: Yeah, it's really cool.

Renee: That's pretty amazing that they were using it to make your card, which is just, it's actually a, it almost looks like embroidery. It's just more stitching on top of the actual fabric stitching. And it's usually shinier. It makes beautiful dresses.

Marc: Yeah, it's really cool. Yeah, it's beautiful. So Charles Babbage picks up the idea around these cards for his analytical machine. When Herman Hollerith comes along later and wants to patent punch cards as a way of storing data, Babbage's precedent blocks him from owning the idea itself. What he could own was the machinery in the system, which in the end was a more valuable thing anyways. Hollerith's own account is that he got the idea partly from Dr. John Shaw Billings and partly from railway conductors punching holes in tickets to record what a passenger looked like.

Renee: And what puts him in business is a constitutional problem. The United States has to count everybody every 10 years, the census. And the population is growing faster than the clerks can add. The 1880 census took somewhere around eight years to tabulate. It's like the worst, like the worst data project ever. Like eight years later, you still kind of know. And whoever were babies are now eight year olds. Like it's. By the 1890, the population was nearly 63 million, up about a quarter. And there was potential that the 1890 count wouldn't be finished until the 1900 count was due. Oh, my God, it started to sound like a backup window, which is a big deal because congressional seats get handed out on those numbers. So if the count isn't finished, then nobody knows how many representative a state is owed. Oh, my.

Marc: Isn't that cool? Like it gets to be such a problem that you know the backup window's too short yeah the backup window's too short too much stuff i can't count fast enough before the next time i have to count yeah oh my batch my batch job it just overflowed

Renee: I wish i could do incrementals.

Marc: And and the punch cards work for this particular problem set the rough population count comes back in about six weeks. That's crazy. You know, to go from something that took eight years to count to six weeks, that's crazy. Full tabulation still took years because they still had, you know, humans doing things. But six weeks for a headline number against eight years for the previous one is why every statistical office on Earth wanted one.

Renee: And the census is part of a corporate origin story. We all know. Halderus sets up the tabulating machine company in 1896. And the financier Charles Flint merges it into the Computing Cabulating Recording Company in 1911. And in 1924, under Thomas Watson, that gets renamed International Business Machines, or otherwise known as IBM.

Marc: Big blue.

Renee: Big blue.

Marc: All right, so let's talk about...

Renee: Thomas Watson should have gave it away, though. I mean, that really should have gave it away.

Marc: Right? Yeah. I mean, yeah. Anybody that's listening to this...

Renee: That's how we know IBM today.

Marc: Yeah. Let's talk about the card for a minute. We did talk about cards in the neural nets episode. Yeah, that's right. Neural nets. I thought we talked about them in the loom. We talk about ethics, too, in the looms. But yeah, neural nets episode. So it's a little bit of review, but I'll keep it short. Seven and three eighths inches by three and a quarter and seven thousandths of an inch thick. And it's all in freedom units because this is a problem in the U.S. That's the size of a United States banknote before 1920, 1929. And the reason is storage, which I thought was really kind of funny. Treasury Department currency boxes already existed in that size. So you could buy the boxes off the shelf. The 1890 cards had 22 columns. The 80-column card with rectangular holes arrives in 1928. And 80 columns at one character gives you 80 bytes. That's the unit of mass data storage for about 50 years. 80 bytes.

Renee: So how did IBM make money on 80 bytes? Like, think about that. Like, that's hardcore, like, monetization. Tell me how they did it.

Marc: Yeah. I mean, that's cool, right? You know, think about that for 50 years, everything is running at 80 bytes. It's just, it's crazy. So they leased the machines and then they sold the cards. And you know how many times have we seen that model? The tabulator goes out on rent and the customer buys consumables from IBM forever, which is, you know, the razors and blades, the problem. And it's why the card business was so defensible. So the customer was paying for their storage by the month, 60 odd years before anybody called that a subscription.

Renee: So I can blame IBM. Good to know. And the medium outlived its own justification by decades.

Marc: They're trailblazers even in those days.

Renee: Okay. And the medium outlived its own justification by decades, which is where this gets political. IBM launches the Votomatic in 1965, a pre-scored punch card ballot. In 1968, a rainstorm in Detroit soaks the ballots and the count falls apart. Iowa had effectively banned them by 1984. Computer Professionals for Social Responsibility called for a national ban in 1988. And in the year 2000, about a third of polling places in the United States were still using punch cards. Which is how when we get to 2000 and it's and we have to and there's Florida says it's a it's a bad count because we don't trust the ballots because of what? What? Hanging chads. Like 2000 was all about hanging chads.

Marc: Yeah. So the cheapness argument, you know, in this time frame just sort of flips. Right. Nobody kept punch cards voting because it was good.

Renee: Nobody kept it at all in the sense of, you know, deciding. It just costs less than replacing it every single year until the year it decided a presidential election. And we're all like, OK, wait a minute. Right. The Help America Vote Act gets signed on the 29th of October in 2002 and funds the replacement, which is 37 years after launch and 14 years after the people who understood it asked for it to stop. It took a long, long time to get rid of Chad. Like, Chad was just hanging around until the bitter end. I just... Until the bitter end.

Marc: Yeah. I mean, I remember all of that mess with the hanging Chads and the counts and all that stuff. And to think, you know... 37 years you know after it had launched and 14 15 years after people said yeah you know what this is probably a bad idea

Renee: Right like let's let's let's update like nah yeah no no no this is working out really good for everybody yeah until okay until 2000 they do the recount somehow they win and then they turn around two years later and say we're never doing this again so they walked through the stargate and then blew it up right like nobody else can use said stargate like like Like, we're not, like, it's just, we're done, right? Yeah, that's what happened.

Marc: Yeah. Okay, so mechanizing the process makes things cheaper. The other way to make storage cheaper is to make the same information take up less room. And that starts with optics. John Benjamin Dancer is a Manchester instrument maker. And from 1839, he's producing micro photographs at reductions of around 160 to 1. He got a lot better at it after Archer's colloidion process in 1851. For most of his life, this is sort of a novelty. People bought micro photographs of the Lord's Prayer as parlor curiosities.

Renee: And then it becomes a weapon, which is the pattern with compression. The siege of Paris runs from the 19th of September in 1870 to the 28th of January in 1871. The city is cut off. René Dagron signs a contract with the government on the 11th of November to get messages out. And the message is microfilm and pigeons. Messages photographed down onto... What is this?

Marc: Colloidian.

Renee: Colloidian. Oh, like colloidian silver, like there's people who drink that and turn blue. Them. Colloidian negatives, 11 millimeters by six, weighing about five hundredths of a gram, rolled into a quill, tied to a pigeon's tail feather. 20,000 to 30,000 messages per quill. Oh, what? That's pretty smart.

Marc: It is pretty smart.

Renee: A carrier pigeon.

Marc: Yeah. How many get through? How many get through?

Renee: Nobody agrees. But one official count is about 95,000 messages. Another set of figures gives 150,000 official plus roughly a million private. He himself claimed two and a half million if you count the copies. So somewhere between 95,000 and two and a half million, which is a completely different story depending on who is doing the counting and exactly what they wanted to prove. But I mean, imagine two million. How many, like the skies would be dark with pigeons. Right? Like, that's a lot.

Marc: That's a lot. It is a lot. And that's like, you know, that's, you know, the 1870s. That's, you know, that's. That's pretty clever, right?

Renee: Yes. Yes. Yes. I can make it teeny tiny and we can send a lot of it and it can go on a pigeon and no one will notice.

Marc: Yeah. That's cool, right? Yeah. What's that TCPIP over avian carrier, right? The joke RFC about carrier pigeons network. So the Second World War implementation has better arithmetic here, though. A V-mail runs from June 1942 to November 1945 and handles over a billion items. The soldiers write on a standard form. The form gets photographed. The film flies, you know, gets on a plane. And the letters get printed back out on the other end. 37 mailbags holding 150,000 letters weighed 2,575 pounds. But photographed onto film, the same letters went into one sack weighing 45 pounds. 1,500 to 1,800 letters on a 90-foot reel. And the reel weighs four ounces. So on a wartime supply chain where every pound of cargo space is contested, that's the difference between, you know, you getting mail and no mail.

Renee: And in 1945, Vannevar Bush writes the often quoted article, As We May Think, in The Atlantic that July. The Mimex is in there and the people remember it as a prophecy of the Internet. What it actually is mechanically is a desk full of microfilm with a screen on top and a way of linking one frame to another. He'd been thinking about it since 1938, and the part everybody took from it was the linking, which was the bit that actually turned into the web, right?

Marc: Yeah. I mean, think about that, like linked, you know, microfilm linked, you know, browsing with the machine.

Renee: Right, you sat in front of it in the library, just for the kids who don't know anything about microfilm or microfiche. You would go to the librarian and say, I need the newspaper from 1961, it's this and this and this, and then she'll go to this big thing and pull out this one thing of microfilm, right? And you would put it on there, and it's just a bunch of film on something that looks like it might be about, what is that, six inches in diameter, right? You stick it on it, you put it through, and you hook it onto the reel on the other end, and you just start turning it. It just starts going past you 150 miles an hour until you get to the thing you're looking for. This date on this newspaper, and then you slowly go, because you're turning the pages of the newspaper, because they were all photographed. And then you're like, here it is. And if it was a really sophisticated one, you could print it. You just copied it right from the copier, and you would just go pick it up on the copier. But that was microphone. And we had it in, like, that's how we looked at periodicals. Like, forget going to the internet way back machine and taking a look at something that published on the New York Times back in 2003. No, we had to go to the microfilm because that was the only copy of that newspaper, right? And pull it down and look at it and copy it off and take your notes sitting there at that desk. I remember that desk, Mark. I remember it. It was a big part of middle school and high school. That's all I have to say about that.

Marc: So standard microfiche ends up at 105 millimeters by 148, reduced to 48 to 1, 15 columns by 18 rows. So 270-page images on a card you can post in an envelope. Libraries adopt it everywhere. And I always liked the film better than the fiche because microfiche was like, It was like you had to move the thing around, you know, in the grid. That was a pain. Anyways, but libraries adopted everywhere. And the reason is shelf space, which costs money every year forever. And, you know, when you miniaturize at that, it takes up extremely, you know, much less space.

Renee: Sometimes it was whole years of periodicals. Like this was like 1961 to 65. Like here's all the newspapers in it. Yeah. Yeah, that was a completely different way to store that stuff, which brings us to the part of this story, which is probably echoing through to today as we talk about books and scanning. Nicholson Baker publishes an article on April 2001 accusing libraries were filming newspapers and books and then destroying the originals. Sometimes things were badly filmed and the lost things couldn't be recovered. So Catcher in the Rye got filmed. They did the film for it. They didn't, I guess they wouldn't have done the development of the film yet. And so they burned the book. And then they come back and like, oh, snap. Page 62 is really bad. Like, now we can't go. Now they got to go take out the book from someone else. Like, that doesn't make, we're still doing this, right? But it doesn't make any sense to me that we would destroy the book. He's open in his writing that he isn't objective. and the Association of Research Libraries put out commoner talking points for its members, archivists conceded that a lot of the filming was poor and rejected his claim about motive.

Marc: Yeah, I mean, you know, his motive was, or they said that, you know, he said the motive was, you know, saving shelf space. And they were like, no, no, it's, no, you know. But real materials were destroyed here. Filming of the material was sometimes poor. So what happens after that?

Renee: The British Library sold off most of its American newspaper holdings in 1999 after filming them. Baker cashed in his retirement fund and bought them and sent up the American newspaper repository to house them, which later went to Duke. So a national library decided the shelf space was worth more than the paper, and one author's pension fund was the only thing standing between those runs and a skip.

Marc: That's crazy. All right. Okay. So this is all the history here. So we'll get a little bit further into the modern era. So recap, clay, expensive paper, wood, cards, miniaturized images. Then for about 40 years, the price falls every single year and the behavior changes completely. The 8-inch floppy, then the 5, 5.25, then the 3.5 with its rigid plastic shell, a zip disk at 100 megabytes, jazz cartridges at 1 and then 2 gigabytes, optical, then flash, then thumb drive, holding more than a whole rack of drives did in the early 90s. Per bit, storage got cheaper and cheaper for four decades, which is the most reliable trend in the whole of computing.

Renee: I mean, you could count on it. You can count that as your storage was coming off end of life, whatever you were buying was only nominally more expensive, but a thousand times better. And what you were really struggling with was it's just it can't do what we want it to do. So now we, you know, like look at the world of the possible and it's not that much more money. So that's the only reason why you would let go perfectly good hardware that just wasn't fast enough for like a whole new like rack set up in the data center because it allows you to do something. Right and i think that's the thing about like when we get to when we get to that part of the story where we're now we're buying storage right because this it's a storage manufacturer needed you to buy a lot of it like a lot of it a lot of it right like so first they would sell it to you as you need this because you know you have stuff and you're like yeah all right i got stuff. And then it was like no no no no you need this too because you need to back that stuff up and you can't back it up to tape look how long that takes you can't back it up to laser because that's going to be obsolete, you need to back it up to disk. And if you're going to back it up to disk, you need to back it up to the storage area network. You're like, oh, good point. So they'll sell you a cheaper version of what you bought because it doesn't need to make SQL calls. It just needs to back up data, right? So now you've got this other thing running in the data center, right? And they're both going to go obsolete in five years. And you're going to buy something that's way better, but then they're going to convince you that, wait, why do you have anything running on the desktop? Why don't you just boot to the SAN and then everybody just runs their instance off that? And you're like, oh, snap, how come we're not doing that? Like, why don't we have dummy terminals everywhere? Like, that only costs $300 instead of $3,000 for a laptop. Like, no, it gets crazy, right? Because in the end, because this whole time it's being commoditized, right? It can only gain so much in its new capability. They've got to sell you on something. So the next thing you know, everything is booting to that. Everything's backing up to it. Everything's running from it. But even that got commoditized, right? We're even spending less on that. So, yeah, Moore's Law was true of storage more than anything else, I think. You know, call Cisco what you want. But like that storage stuff, that got criminally cheap, criminally cheap.

Marc: Fast yeah very fast and i i remember because i was working in the field a lot in that era of of you know portable storage you know the the zip drives and the you know the jazz drives and stuff and i remember you know that was just like within a year a couple years you know it was just you know high density drives and then you know everything just got

Renee: In the name of security as I remember it too. It wasn't even that. It wasn't even that we don't want you to have a zip drive because it has a bunch of company stuff on it and you might lose it. Like we weren't thinking like that back then. It literally was. We all need to work on one document. Let's put it on the same thing as the server. Like remember you used to set up a server. Windows NT if I recall. Yes. It was new technology back then. NT, new technology. Anyway, you would set up this stupid server and then we'd be like, okay, I have six folders on this server. One's the common. Everybody's going to put all their stuff in it that they want to share with everybody else in that folder. And then everybody has their own department folders. But then the only people who belong to that department have access to that folder. And then the person who runs that department can do whatever they want in that folder. This is all sitting on the same server as the operating system, right? So if you really want to direct me, just come take my server. Like from under the desk in the copy room, because that's probably where it is, right? Like that's how create, like when I think back to the very beginning, that's what that was like. And we weren't doing this stuff because we were trying to be secure. We were doing this stuff because we wanted to share data.

Marc: Yeah, yeah. But sharing data, when I was, gosh, what year was that? It was probably in the late 90s. The way that the studios were transporting movies, you know, digital, because, you know, like every episode of Star Wars or Star Trek had to be, you know, there was digital effects added to each episode. How do you move that stuff around you can't email it you know it's not on film you know what do you do it's it's literally coming off a jazz drive uh you know for from some dude in you know culver city and you know where he did his his digital matte paintings and stuff and then then it gets transported physically you know down the street to paramount or whatever like that's how in In Hollywood, a lot of that stuff, for a very brief amount of time, like, removable media was how digital film was moved around.

Renee: I'll say even as recent as 2016, 15, 16, the Motion Picture Association of America was putting out their best practices for security, and a full two-thirds of it is physical security.

Marc: Yeah, yeah, absolutely.

Renee: Right, because they still get stuff around town that way.

Marc: Yeah, yeah, yeah. Yeah. I mean, you know, that's... You know, now is maybe not as much because, you know, they've got the whole, you know, kind of cloud towers and then cached, you know, locally and all that stuff. But, like, how do you get a copy of the latest film to, you know, some theater in the middle of Kansas that doesn't have, you know, a high-speed network? Like, you got to physically send it to them, you know?

Renee: I can tell you, well, I was at one of the studios that we actually did move it around on the network, right? And I think it was maybe three or four years after I left, we found out that the IT department had actually been doing that. They were taking it from the lot, moving it over to an FTP server, going home and downloading it, and then burning CDs. And so somebody was called a ringleader. Other people were called criminal masterminds, all of them except one guy went to jail. Like, it was jacked. It was jacked. Like, to think that you could just move stuff around and not get caught, like, that was, whew, hmm. And all of them and part of their probation is like they can't ever work in IT again. And that was that was the gig. Like that was the gig back then. Like that's crazy. Yeah. But I mean, that's just it. Right. Because with that comes like a lot of security. Anyway, the thing is that falling prices does to an institution, what it does to an institution is that it removes the decision. Nobody has to ask whether this is worth keeping because keeping it's free. It sounds like a pure win, and it isn't because deciding what matters is also how you find out whether you can still read it.

Marc: Yeah, there's a technology in there that shows the price effect directly, which is magnetic bubble memory. In the 1970s, this was the future. Ooh, magnetic bubble memory. No moving parts is non-volatile, and Intel, among others, put serious money into it. So, but it died in the 80s. Yeah, but. Yeah, but the physics of it was fine. Hard disk got cheap enough and got rugged faster than bubble memory could catch up. So there was never really a point where buying it made sense. Like that cost-benefit, you know, kind of scenario, it just didn't make sense. Yeah.

Renee: There's a British example from the same period, too, the BBC, Doomsday Project, built between 1984 and 1986 with Acorn, Phillips, and Logica for the 900th anniversary of the Doomsday book, a snapshot of the country. Schools all over Britain surveyed their own areas. It was shipped on LVROM laser discs, 300 megabits a side, read by a specific Phillips drive.

Marc: So tell us renee how long was it readable for

Renee: About 16 years by 2002 both the discs and the drives are on the edge of being unrecoverable i can't even tell you how many people used to do optical discs too like we did it in a bank i worked out i'm like we have what now like how are we wait what like what happens when that reader like dies and they're like we can't read them anymore you're like wow the national archives got involved at the launch of the digital, Preservation Coalition on the 27th of February 2002. And the Chameleon Project demonstrated an emulator at Leeds on the 2nd of December that year. Working from Data pulled off one of the surviving drives. The BBC put it back online as Doomsday Reloaded in May of 2011. The book from 1086 and the whole thing that the whole thing was celebrating has needed no intervention at all.

Marc: Yeah. Yeah. Here's this project that lasted 16 years while the original doomsday book, you know, is over a thousand years old.

Renee: Right? Right?

Marc: Yeah. Or close to a thousand years old. All right. So let's talk to two other examples here. The first is a 1960 United States census. There was 642 reels of derivative data that could only be read on obsolete UNIVAC 2A drives. The staff went and found working drives, and by 1979, they'd copied 640 of the 642, and the other two were never located.

Renee: And Apollo?

Marc: Yeah, this one's kind of sad. So the original slow-scan telemetry tapes from Apollo 11 were very probably erased and reused in the late 70s and early 80s as routine cost savings. It's like hoping over your wedding tape from Seinfeld. Yeah, exactly.

Renee: I'm sick.

Marc: Yeah. Yeah, just routine cost savings because the blank tape costs money and nobody had flagged those reels as special. So the restoration NASA released, you know, came from the best surviving scan converted copies, including recordings made at Parks and at Honeysuckle Creek. So the tapes were wiped and used again for something else, we don't know. And the Lunar Orbiter Image Recovery Project, which digitized the 1966 and 67 catalog tapes and finished primary capture on the 20th of February, 2014, only worked because somebody restored a 1960s Ampex FR-900 drive to read them.

Renee: Okay. Okay. I have to just let me say this because I just have to get it out. I have to get it out. I have to get it out. This is why when we say we need to modernize the stack, we have to get like, because we need to keep the stupid hardware too, right? Like if you're going to keep these drives from, I don't know, Compact 1U servers from like 2000, then we need to keep the stupid server if you think you're going to restore tapes from a dl like 2863 carousel thing then we need to keep the stupid thing or you're not going to get that stuff back oh by the way the machine you're restoring it to we got to keep that too in the same drive configuration or we can't get it back there wet that way either.

Marc: This is the operating system too

Renee: And the operate and like the right patch level like this is such a nightmare it's not even funny not even so when you pull back a tape and you're like, I don't understand why we can't get it. Oh, really? Oh, really? Because we don't have the hardware. We don't have the software and we don't have anybody that knows how to do this either. So like, yeah, welcome to my hell. Like that is like, that is literally it in a nutshell. You can't just save the tapes. You have to save everything that went into making those tapes because we can't get it back. I'm so glad no one else has to deal with that. I feel like I suffered that. So the rest of you don't have to. So in all three cases, the medium was fine, and the reader was the problem. Three parts, and they'd, Don't fail together. Like, yes, that's it in a nutshell. In a nutshell.

Marc: All right, which brings us to this year. Okay. Where the price went the other way for the first time in most of our working lives. Western Digital and Seagate have sold out of their hard drive production for the year. Lead times of 6 to 12 months. And enterprise drives quoted at up to two years. Consumer prices are up somewhere between 46 and 50 percent in a matter of months. RAM prices are going through the roof. The largest cloud and AI operators have locked in multi-year supply agreements running through 2027, 2028, which means the capacity is pretty much committed out to 2028.

Renee: And who's left over?

Marc: Yeah. Yeah. So you can actually look at Western Digital's own numbers. 89% of their revenue now comes from cloud customers. So client, which is the PC makers, is about 5%. And then consumer, which is everybody walking into a shop or whatever, ordering a drive-a-line is about five to six. So roughly one pound in nine of that business. So, you know, pounds, because I didn't think of dollars. Is that business is everything that isn't a hyperscaler. So one ninth is, you know, just normal stuff. And then eight ninths is hyperscaler. It's the same picture in Flash. So QLC production is booked out. Sandisk has put NANDY prices up 50%. And Dell and Lenovo have already put 15% to 20% up on PC prices.

Renee: And a hard drive isn't even purely mechanical anymore, is it?

Marc: Yeah, yeah. So, exactly. That compounds the issue, right? So, a modern drive has onboard DRAM for cash and a flash controller, so the spinning disk is sitting inside the memory shortage, too. The cost of the least glamorous storage in the building went up because of a chip shortage. Physically, these drives are also more exotic than people assume. The high-capacity ones are sealed and filled with helium, because helium is about a seventh the density of air, so there's less drag and less turbulence. Which lets you fit more in thinner platters and keeps the heads flying stably. Heat-assisted magnetic recording is pushing past 30 terabytes with 100 on the roadmap. All right, and the materials are the part that should worry people more than it does. Every drive has something like two and a half to four and a half grams of rare earth material in it. In the voice coil actuator that moves the heads and in the spindle motor, Those are neodymium iron boron magnets, and they're doped with dysprosium and terbium, so they hold their magnetism at operating temperature, which are basically the elements everybody is anxious about the supply of. And almost none of that comes back.

Renee: And the reason is the rules push you toward destroying them. The Circular Drive Initiative estimates around 90% of hard drives are shredded rather than reused. And that's over 200 million devices a year. Google's have robots that puncture and shred decommissioned drives in its own data centers. The driver is data security and compliance because a shredded drive is a defensible position in a way that a wipe drive feels like, I'm going to say it one more time. Feels like it isn't even when it is. And once a drive's been through a shredder, you're not getting the magnet back out of it. I want to talk about this for a second. Can I talk about this for a second? Because this is a total misnomer, right? Like if you do what you're supposed to do, you wipe that drive to the Department of Defense requirements for hashing that drive, which is all ones, all zeros, and then nine passes of random zeros and ones, and that thing is hammered. Like you may as well drill the hole in it, right? But I will argue someone did a study at Stanford or whatever at the drive. Who ever heard of it? And one of the things they said was physically on the drive in a sector drives me to sectors. Right. So the sector, a one does not take up the entire sector. A zero will take up a sector and a half. So I'm going to ask everybody, why not just run seven passes of zeros? You overwrote that thing seven times because the zero was one and a half in the sector. So like there are ways to do this without having to puncture the drive, lose the magnet, get rid of the rare earth elements, and now you're all paying too much for this. So I'll say that if you gave the compensating control to your auditor and said, we don't drill holes in them because we don't think that's good for the environment and we want to reuse them because we think that's good for our bottom line. Here's how we make sure there's no residual data on it. We do a ones, we do zeros, we do a random pass nine times and we're done. Like, okay, but yes, I would agree. As a former auditor, I would agree. That's a good mitigation strategy. Prove to me you did it. Right? Like, that's all I ask. And you would have records of that because there's software that does this for you. Like, that piece of it right there, like, we don't have to shred them. I get it. It's fun. You should see what it looks like, but it goes through a shredder. It's hilarious. Like, Iron Mountain will do it right in front of you and take the box of garbage away, right? Like, so yeah, I get it, but we don't need to be doing that anymore. We don't need to be doing that ever. We just need to wipe them.

Marc: Yeah, yeah, it's pretty cool. Seagate is a founding member of the Circular Drive Initiative. It opened an official refurbished storefront on eBay in June of 2024. It sanitizes to the IEEE 2883 and NIST 800-88 standards. And it says it returned over 1.8 million drives and solid-state drives to service in its 2024 financial year. So that's pretty nice. Oak Ridge National Laboratory has demonstrated an automated line that pulls the magnets out intact at a rate of thousands of drives a day. Okay, so we didn't say it explicitly, but, you know, the reason that this is a big deal is those rare-earth magnets, like the supply chain, is really tough on those. They're hard to get, so, yeah. Oh.

Renee: Right. So who who who does this price out of the market like a hyperscaler signs, a three year supply agreement, a university archive, a county records office, a regional library, a small museum. They buy a retail at whatever the price is that week. So the places whose actual job is keeping things have about the least buying power of anybody in this market. And they're bidding against companies that have already booked out the next three years.

Marc: Isn't that crazy? Like, yeah, libraries, you know, isn't that sad?

Renee: Yeah, it is sad. Yeah. Yeah. All right.

Marc: So every few years, somebody announces the storage medium that solves the problems, right? It solves all the world's ills. So before we talk about the current ones and, you know, where the future is here, let's talk about what I call the graveyard. So we talked about bubble memory, and that was killed by a cheap disk.

Renee: Holographic storage was the great hope of the 2000s, and that one went a different way. Enphase Technologies actually shifted 300 gigabyte disks in 2007 and then folded after nine years and about $100 million, with the office building seized for unpaid back taxes. So the technology did what they said it would, and the business went under anyway.

Marc: So there's another one that's recent called Optane. Intel's 3DX point, a third kind of memory sitting between DRAM and flash. And Intel discontinued the whole business and took off, this was like, oh, why, a $559 million inventory write-off, having lost billions on it over the years. That one died on economics. It worked, and there wasn't a price at which enough people wanted it, though.

Renee: So, dear God. So what's in the, and then they had to be bailed out. So what's in the pipeline intel of all people? And what's in the pipeline now? So what's going on?

Marc: Yeah, so liquid, there's a liquid storage trend. It's at IMEK in Leuven. Two concepts out of Martin Rossum-Ulin's storage program, both presented in 2022. A colloidal memory suspends two different types of nanoparticle in water and draws them into capillaries only slightly wider than the particles. So the order that they went in is preserved, and the order is the data itself. Then there's electrolytic memory, and it dissolves metal ions in liquid and electrolytes them into nanowells as alternating layers one or two nanometers thick and the thickness of the layer is the is the bit

Renee: Electrolithic as in lithos stone stone.

Marc: That's right yeah named after geological strata that's because that's what the stacks of metal layers looks like so the most advanced liquid storage concept anyone has writes data as sediment and it's named after rock so we've come all the way back to clay this

Renee: I literally don't understand how that's any better. Like, I don't. This seems like an awful lot of work for. Okay. Okay, how far away is this? Like, I can't imagine. We spent all these years keeping water out of the data center. Now, like, a data center cannot live without water. Like, are you kidding me? We built the toilets around the outsides of the building so that if we had a water main break, we wouldn't lose all the servers. And now you're telling me it's just going to run full of water.

Marc: Liquid Memories had a couple of previous flashes, so we'll see if this one takes or not. I don't know. Heck, when we're talking about helium in the drives, I remember you used to crack the stupid things open and hit them with a screwdriver.

Renee: Right? And then put them back together and they would work fine.

Marc: Yeah, exactly. Because the head would stick,

Renee: Right? Or the arm would stick. Like something weird would stick. You'd be like, you slam it on the ground really hard and then stick it back in and it would work. And you're like, all right. Especially Max. I don't know what the deal was with that. As a matter of fact, if you were thinking about the OSI of a Mac, like what's the first layer? Take out the hard drive and slam it on the carpeted floor really hard and then put it back in. Like if it works, that's layer one of the OSI of Mac. Go ahead.

Marc: So iMac put liquid storage on the roadmap from 2030 onwards. It's pretty impressive, though, targeting a terabit per square millimeter. And that's contingent on making capillaries at a 40 nanometer pitch with aspect ratios of 400 to 1. So it's published in 2022, still described as exploratory. Four years on, there's no product. So don't hold your breath on the liquid storage here. We'll see. Glass is further along and in the same position, though. Microsoft's Project Silica, published in Nature on the 18th of February this year, 4.84 terabytes written into ordinary borosilicate glass, so just normal glass, rated for 10,000 years at 1.59 gigabits per cubic millimeter. But again, no product, no price, no commercial partner. At a write speed of 66 megabits a second. So not, you know, not as good as it could be. Microsoft has wound the research phase down. Cerebite is doing something comparable in ceramic and is at least trying to build a business.

Renee: We really are back to clay. So the archival future is still in a lab somewhere. Like, I mean, it all has its own limitations, right? Like glass. Yeah. We're going to store all that. I mean, it sounds cool, right? You pull out this piece of glass and you stick it in and all of a sudden, like, it's so, like, it's so minority report when you think about it, right?

Marc: It's like Buck Rogers, right? Right?

Renee: Yeah, it's really weird. Like, you think about it that way. Like, they would pull it out and look at it and it would be doing this crazy hologram or something, put it back in. Like, yeah, it's weird. But, like, is that, does that make any sense? Like, what if I drop it? Like, seriously, what if I drop it? Like, that's crazy talk. Like, if I dropped a tape or I dropped a disc or, I don't know, I can't drop AWS. Why can't I just back it up there? Like, there's, like, seriously, like, glass storage seems crazy to me. It seems crazy to me. It all has liability. It all has liability.

Marc: Well, yeah, true. Yeah, so, okay, it's, you know, everything that's in the future is still a set of, you know, papers. So it's in the lab. It's, you know, an academic paper with, you know, very good physics, but no price per terabyte. So until somebody can tell you what a terabyte of it costs, you can't plan an archive around it. And in the meantime, nearly all of it is sitting on spinning disks that you can't get a hold of at the moment anyways. All right. So the price kept falling for about 40 years. And when keeping something costs you nothing, you never have to sit down and work out whether it's worth keeping. So nobody ever sat down and worked it out. And that's how you get the BBC spending two years surveying the entire country and putting it on a disk that stops working at 16.

Renee: And when the price goes up, the choosing comes back. Every time storage has been expensive, the deciding what's valuable enough to store is decided by whoever was paying. An Egyptian state monopoly decided what papyrus was for. A scribe with one skin and two texts decided which one survived. The exchequer decided what the tally sticks were clutter. The British library decided shelf space was more important than American newspaper. And it was all about managing costs.

Marc: Yeah, and the entities holding the world's storage capacity for the next three years, guess what, are not libraries.

Renee: There are four companies with supply agreements, and their retention policy is whatever serves the model their training, which might be everything forever and might be nothing past 90 days. And either way, it isn't a decision anybody outside gets to vote on. Clay survived because it was fired. Parchment survived because it was expensive enough to be worth scraping clean and using again. And laser discs rotted because they were supplanted quickly.

Marc: If you've got a box of three and a half inch discs in a cupboard somewhere, the discs are probably fine. Finding a drive is the problem.

Renee: Oh man, and if you work anywhere that is ever sent, we'll sort out the archive later. Later has a price attached to it, and it's now.

Marc: All right, so write us at NostalgicNerdsPodcast at gmail.com if you know what happened to the liquid storage cubes, because I'd like to know whether I made that up.

Renee: This is the Nostalgic Nerds Podcast. Thanks for tuning in. Subscribe, follow, share this with someone who still thinks storage is cheap. We'll see you guys next time.