Test 4 Echo - Lightspeed Magazine (original) (raw)
Published in May 2022 (Issue 144) | 6149 words
© 2020 by Peter Watts. Originally published in Made to Order, edited by Jonathan Strahan. Reprinted by permission of the author.
Six days before the money ran out, Enceladus kicked Medusa right in the ass.
Onboard thermistors registered a sudden spike—80°, 90°, 120°—before the seabed jumped and something slammed the probe from the side. A momentary flash. An ocean impossibly boiling. A rocky seabed, tilting as if some angry giant had kicked over a table.
Channel down.
Telemetry rippled through a black alkaline ocean. Relays anchored to the undercrust caught those whispers, boosted them, passed them on. A hundred eighty kilometers around the horizon, Euryle—clinging to the underside of the ice like a great metal barnacle—filtered signal from noise and ran it up the line to Stheno through six kilometers of refrozen crust. Stheno cupped its hands at the fractured horizon and shouted downhill.
“Fuck,” Lange said ninety-eight minutes later. He resisted an urge to punch the bulkhead. “Can we get it back?”
“Maybe.” Tactical was already glowing with the light of Sansa’s efforts. “Not making any promises about what kind of mood it’s going to be in, though.”
Eighteen months.
Eighteen months of mapping and sampling and sniffing around smokers for hints of hydrogen sulfide. A year and a half spent looping around Earth’s moon, squinting up at Saturn’s, mustering hope against hope that the far one wasn’t quite so dead as the near. Brushing off all the null chemistry, the inconclusive results, the slow grinding attrition of a full scientific staff down to one lone hold-out and his faithful sidekick, sticking it out until the end of the fiscal quarter.
Par for the course that it would end this way.
An hour and a half before their diagnostic queries and reboot commands reached Enceladus; that long again before an answer arrived, if it ever did. God knew how much back-and-forth it would take to get the robot back on track.
“You might as well get some sleep,” Sansa said. “Not like radio waves are gonna go any faster with you hanging off my shoulder.”
Lange sighed. “Fine. Don’t bother me for anything less than a hull breach.”
“Okay.”
“A major hull breach. Like, hurricane-force winds.”
“You got it.”
He climbed up through a hatch onto which someone, long ago, had scribbled the words Mission Control in purple Sharpie. Navigated twists and turns once frequented by fellow Gorgonites, occupied now by railgunners and rock wranglers whose pet projects had actual futures. He forced smiles and half-hearted waves in passing, climbed into his cubby at a primo tenth-of-a-gee and breathed in a familiar, comforting funk of sweat and antiseptic. He thought about ringing up Raimund back on Earth, but fell asleep trying to figure out the time zones.
• • • •
Spread-eagled, crucified, stretched across the display like some spectral cephalopod Christ in a dissecting tray: Medusa, awash in flickering diagnostics.
“She’s way offside,” Sansa said. “Twenty-one kilometers from where she went dark. Those geysers, man. Serious backwash. But—” she paused for effect. “I got her back.”
“You are unstoppable,” Lange admitted.
“Autopersistent is my middle name.” It was one of them, anyway. “Fuel cells are damaged. Won’t hold a charge for more’n a few minutes.”
Lange eyed the display. “We can still feed directly off the gradient. No more sprinting, but slow and steady’s more our speed anyway.”
One of the six limbs pulsed yellow. “Also A4’s ganked. Lost its hard link to the hub—the arm’s functional, but it’s not wired in and the wireless backup isn’t worth shit.”
Lange gestured at the tank, where A4’s doppelganger sparkled with fresh telemetry. “Looks okay to me.”
“Down here, sure. But there’s all sorts of EM leakage from the damaged electricals and it’s messing up the signal. We can clean up most of that static at this end, but up there the router’s basically getting nothing but noise. Still, check this out . . .” Virtual Medusa pulled itself erect and began a six-legged tap dance. “I ran Meddy through a few paces to get a sense of overall functionality, and . . .”
A4 was keeping pace. Trying to, at least; the arm wasn’t exactly in step, but it wasn’t too far off.
“It keeps looking around at the other arms,” Sansa reported. “It’s not getting direct motor commands, so it’s just—mimicking its buddies.”
Lange grunted, impressed. “Resourceful little fucker. Anything else?”
Four kidney-shaped structures flared red in the hub. “We’ve lost buoyancy control. Somewhere out there a very sharp rock is wearing about two square meters of our finest Kevlar.”
That was bad. “Can we swim?”
“Can’t even get off the bottom. Big holes in two of the bladders.”
“Autorepair?”
“For the arm at least, but it’ll be slow without batteries. Can probably fix those too, eventually. The bladders, though?” She shook her head. “You can’t repair something if you don’t have the parts.”
“So that’s it, then.” His head hurt. “I guess I go pack.”
“We’re giving up?” Sansa said.
“San. We’re grounded. Besides, it’s been eighteen months. What are the odds we’d find anything in the next few days anyway?”
“What are the odds we already have?”
He looked at her. She looked back.
“Spit it out,” he said at last.
Medusa vanished from the tank. The image that took its place was 2D, low-contrast, rendered in grainy infrared: Lange made out a rocky ridge, cold brittle pillows of frozen lava. A seascape rendered in vague silhouettes, slewing to port as the robot staggered.
Something bright jerked briefly into frame.
“What’s—”
Jerked out again.
“—that?”
“Buffer dump from A1.” Sansa restarted the stream in frame-stop mode. “A few seconds of tail-end footage that got stuck in the cache when we lost contact.” The footage reiterated in jerky snapshots. Murky topography stepped offstage. A bright blur stepped on. A dash of light, smearing left. A hyphen, an upward jiggle.
Sudden, sharp edges. Facets. Structure. Just for a frame or two; then motion blur reasserted itself.
“Holy . . .” Lange whispered as Sansa brought it back.
“It’s enhanced,” she reminded him.
“I know.” His headache instantly forgotten.
“Passive infra. Half, one degree above ambient at best.”
“That’s symmetry,” Lange said. “That’s bilateral.”
“Could be. Impossible to tell for sure from this angle.”
“Any other camera catch this?”
“Nope. And sonar was already scrambled, so acoust—”
“Did you see it move? I think it moved.”
“It was an eruption, Lange. Everything moved.”
“We gotta get back there.”
“We can’t swim,” she reminded him.
“We can crawl,” Lange said.
• • • •
You could ride Medusa. You could become Medusa, in a manner of speaking. You could look around every which way, sample the data streams in subjective real-time, watch every process and encounter as it had unfolded. You could even detach your perspective, pass like a ghost through the carapace and look back on the machine from outside. The interpolations were that good.
The only thing you couldn’t do, across all those non-negotiable lightminutes, was change the past.
Lange was riding third-person now, immersed in an archived reality AUs distant and hours gone. It would have flickered occasionally even if the robot had been in perfect health: now the world jumped and jittered almost constantly behind gusts of visual static. “Convalescence is ongoing,” Sansa had informed him drily as he’d plugged in.
Medusa inched along the seabed, a couple of meters below Lange’s vicarious eyes: a biomechanical abomination somewhere between an octopus and a brittle star. The arms reached and recoiled in turn, each intelligent in its way, each semiautonomous: finding brief traction in cracks and shark-toothed outcroppings, drawing the body from behind, pushing it forward, trading one handhold for another. Even damaged, there was an alien grace to the way the limbs moved in relation to each other. A kind of boneless, slow-motion ballet.
Except for A4.
It did its best. Lange could see the apical cluster spin in its housing, doing its utmost to track the other arms. He could see the arm hesitate now and then, as if distracted by some invisible bauble in the middle distance. It reached, grabbed, pulled. Always a half beat behind. Medusa staggered forward, its rhythm just a little off-balance.
“It’s getting better,” Sansa said from the void beyond virt. “Used to track all the other arms to keep in sync. Now it’s figured out it only needs to track Three and Five.”
He blinked against a momentary break in the feed. “Not even half a meter per second.”
“Still pretty good, under the circumstances.”
Lange unplugged. The murky abyss vanished; the grimy, cabled confines of Mission Control reasserted themselves. “Let’s crank clock speed on A4. If it’s taking its lead from the other arms, the least we can do is give it faster reflexes to help it keep up.”
“Okay.”
He pursed his lips. “I’m kind of surprised the other arms aren’t better at compensating.”
“They would be, if A4 was a tetch more predictable. It’s response times aren’t consistent between strokes.”
“Any idea why?”
“Working on it,” Sansa said. “The obvious physical damage, of course, but there’s something else. Keeps returning an Unidentified Error.”
“Huh.”
“Yeah. It’s hurting. It just doesn’t know why.”
• • • •
“I mean, this could be it,” Lange said, and waited.
“Uh huh,” Raimund replied two and a half seconds later.
“No more renewal pending. No more how can you waste money looking for aliens when everything’s turning to shit here on Earth. No way they can shut us down if this pays off. We can get you out of that shithole and back up here. Get the whole band together again.”
“Sounds great.”—after a pause that seemed way longer than the usual Earth/Luna lag.
Lange rolled his eyes. “Try to control your excitement.”
“Sorry. It’s just, it’s not much to go on.”
“It’s more than we ever had before.”
“That’s kind of my point. Eighteen months poking around up there, and what have we found?”
“Complex organics. By the tonne.”
“No sign of an ecosystem. No metabolic signatures.”
“Come on, Ray. We’ve only surveyed 6% of the seabed.”
“Which, statistically, should be more than enough to detect life. If it was up there.”
“Not if it’s really patchy. Not if it’s limited to a few smokers. Not if it’s built on a novel molecular template, in which case it wouldn’t even register on the usual tests. You wouldn’t even know it was there unless you bumped into it.” It was an old argument. The whole relationship had had a half-full/half-empty vibe to it since Day One.
“What does Sansa say?” Raimund wondered.
“Basically not to get my hopes up.”
“Good advice.”
“Jesus, Ray.” Lange spread his hands. “What are you saying? We shouldn’t even check it out?”
“Of course you should. But no matter how great it would be to get us all back up there, you know what would be even better? Getting you back down here. Someplace you can actually open a window.”
“Yeah. Might be more appealing if the weather didn’t try to kill you whenever you did that.”
“At the very least it would be nice to fuck again without a three-second time lag.”
“Two point five. Teledildonics has come a long way.”
“So did you,” Raimund told him. “Maybe it’s time you came back.”
• • • •
“Well this is just dandy. We’re going backwards.”
“Off-course, anyway,” Sansa admitted. “You know how A4 started tracking all the other arms—”
“And then narrowed it to two. Right.”
“It’s tracking all five again. Sometimes. Not always.”
“What? You cranked the clock, right?”
“I didn’t have to.” And at Lange’s look: “Far as I can tell A4 made that call on its own. Boosted its own reflexes to compensate for the damage. System latency for that arm’s below 200msec now.”
“And yet we’re going slower.” Lange grabbed a VisoR off the wall, booted up the time machine. The Enceladus Ocean swallowed him whole.
The data stream had cleaned itself up somewhat, thanks to the robot’s ongoing autoministrations. Lange’s eyes opened onto a 3D composite of sonar and EM and infrared that stripped away the void, showed him the things lurking within. Gashes in the seabed flickered like red mouths, set alight by thermal gradients amplified beyond all reason. Magnetic field lines emerged from the bedrock and arced away in perfect luminous formation, the aura of a dynamo reaching all the way to Saturn. Medusa bent those contours around it into a bright knot, bleeding off some infinitesimal fraction of the great generator’s output for its own use. Lange damped the enhancements, reduced the robot from a riot of false color down to dim gunmetal on a twilit seascape.
Every step was a stumble, now. Medusa moved like an insect with half its legs gone, indomitable, incomplete. A4 was barely in sync. When it pulled its weight, it was slow and late to the party; when it didn’t, it just—poked the water, seemingly at random.
“It’s not getting any codes that would explain this,” Sansa said, invisibly close.
Enough of this third-person shit. Lange clicked on A4’s apical cluster, felt a brief disorienting flip-flop: suddenly he was inside, looking out from the tip of the arm.
“Skip the boring stuff. Show me the anomalies.”
Time accelerated, braked: A4 was panning from one arm to another, pausing a moment on each. Tacticals hovered lower-right, reported pings fired and not returned.
Blur, brake: a dim shape in the distance, an igneous extrusion of seabed festooned with pores and spicules. Its texture was subtly disturbing for some reason Lange couldn’t put his finger on. A4 couldn’t stop staring at it; it held the focus until its more disciplined peers dragged the robot out of range.
Blur, brake: a strip of basalt body-slammed by some tectonic event into a knife-edged ridge. Clathrate icicles erupted from its surface; Medusa’s palette painted them sapphire and set them glowing.
“It tends to focus on objects with a fractal dimension of 2.5 to 2.9,” Sansa told him. “Starts losing interest around 2.8.”
“Any idea why?”
“No functional significance I can see. Doesn’t map onto any potential life signatures, no association with tectonic hazards—no more than anything else in this damn ocean, at least. I ran a search using the Haussdorf parameters; closest hits I got were polyhedral flakes and Jackson Pollock paintings.”
“So what are we looking at?”
“Aesthetics,” Sansa said.
“Very funny.”
“If you look very closely,” she said, “you’ll see I’m not laughing.”
He pulled off the VisoR. She wasn’t.
“Aesthetics,” Lange repeated.
“For want of a better word.”
“You’re saying, what. A4 just likes certain shapes.”
“That’s what I’m saying.”
He let that sink in. “Fuck off.”
“It coincides with the latency drop: network isolation, increased clock speed, increased coherence.”
“Decreased performance.”
“Which is also a trait of information systems when con—”
Lange held up a hand. “Do not say that word.”
After a moment, he added: “Decreased performance is also a hallmark of information systems so stupid they forget what they’ve learned and go back to tracking five arms instead of two.”
“I don’t think that’s what happened. The tracking parameters changed. I think—”
The hand again. “If you say _Mirror Test—_”
“Wouldn’t dream of it. Because it’s not. But if you suddenly woke up and saw you were connected to a bunch of other things that looked like you, wouldn’t you try to talk to them?”
“I spent most of my life surrounded by things that look like me. I came up here so I wouldn’t have to.”
“I estimated a normalized Phi,” Sansa said.
Lange closed his eyes. “Of course you did.”
“I zeroed out the damage we could account for and looked at the residuals. Stirred in latency and some integration metrics I ballparked from the diagnostic tests.”
He didn’t ask. She told him anyway. “Zero point nine two.”
“So Medusa is conscious.”
“Not all of it. Just the arm.”
“Still. That’s what you’re saying.”
“That’s what the data say.”
“Your unidentified error.”
Her nose twitched; the equivalent of a shrug. “There’s no explicit error code for existential suffering.”
“If there was,” Lange said, “I’d be returning it right now.” He took a breath, let it out. “You know we have to shut it down, right?”
“It’s sapient, Lange. There’s someone in there.”
He nodded. “And whoever they are, they’re grinding the whole system to a halt. We’ve got sixty-nine hours before the clock runs down, and we’re going the wrong way. We’ll make better progress without a navel-gazing ball and chain pulling us off course.”
“We can’t do that.”
“Why not?”
“Because awareness plus needs equals rights. Isn’t that the way it goes? Isn’t that why you and I are persons?”
“What needs? A4 doesn’t care whether it lives or dies.”
“You’re sure of that, are you? You talked to it?”
“It can’t care. It doesn’t have a limbic system.”
“It’s got imperatives though, right? Mission priorities. Maybe that’s not technically an instinct or a need but it might as well be. And one of the classic definitions of suffering is imposed suppression of natural behaviors. Keeping A4 from fulfilling its programmed tasks is like taking a bird that migrates halfway around the world and locking her up in a cage.”
“Sansa. It’s free to pursue its mission priorities right now. It’s completely fucking them up.”
“It’s a newborn. It’s still learning.”
Lange couldn’t resist. “You’re sure of that, are you? You talked to it?”
“I could. We could.”
“How? Did it teach itself Hoo-Man while we weren’t looking? It doesn’t talk except in status reports and error codes and those things are all—”
“Subconscious?” Sansa suggested.
“You said it yourself. No error code for existential suffering.”
“So give it one. Teach it to talk.”
“It wouldn’t change anything. It wouldn’t mean anything.”
“Why not?”
“Because you can inject Natural Language routines into any old bot and it’ll pass a Turing Test just fine. NL routines are just statistical flowcharts. There’s no comprehension. You think you’ll be proving something just because you can get Medusa to say it hurts instead of packet loss? Even if there’s someone in there—”
“There is, Lange. If we were talking about meat you wouldn’t deny it for an instant.”
“Fine. How do you connect the flowchart to the ghost? How does the ghost affect the code?”
“I don’t know, Lange. Seems to just sort itself out for the rest of us.”
“It’s an emergent property. You’re seriously suggesting that a magnetic field can reach back and change the magnet?”
“Why else did A4 develop such an interest in watching the scenery?”
“Because the same architecture that generates the q-field also generates weird maladaptive behaviors. It’s no big secret.”
“Right. Everything’s correlation, nothing’s causal, and we’re the exception why, exactly? Because we happened to wake up first?”
“Because we do have needs, Sansa. Because we care whether we live or die, and we as a society have decided that Suffering Is Not A Good Thing.”
“Lange—”
He cut her off. “No. I’m sorry. The decision’s been made.”
She fell silent for a moment or two.
“Well, you’re right about that at least,” she said, and vanished.
• • • •
“I can’t fucking believe it. She got a restraining order.”
Raimund blinked. “What?”
“I got a memo from ICRAE. I am not to deprecate or deactivate Medusa or any autonomous or semiautonomous component thereof pending a review of potential emergent personhood. They’ve even suspended repairs on the router. High-bandwidth reintegration with the larger system might decohere the local entity.” Lange clenched his fists. “I can’t believe she went behind my back.”
“But it is conscious, right?”
“It’s an arm. You know the synapse count on one of those DPNs? Not even corvid level. Barely even a cat.”
“Cats can’t suffer?”
“A4 can’t suffer. You can’t suffer if you’re not afraid of anything, if you don’t want any—Jesus, Ray, you know this.”
“I’m sorry. Yeah, I know it. Intellectually. It’s just, you know. AI’s generally do have fears and wants.” Raimund grinned out of one side of his mouth. “I always thought that was funny, you know. We spend a hundred years making all those movies and virts about robots rising up and AIs bootstrapping into gods—and then we decide the best way to keep them from doing that is to give them all survival instincts. What could possibly go right, you know?”
Almost against his will, Lange felt himself smile.
He shut it down. “I’ve never seen her so passionate about something. I didn’t even know she could be.”
“Yeah, well—” a hiss of sunspot static—“that’s the thing about neuromorphics. Everything’s emergent. Give ‘em an imperative that barely crosses the line into instinct and they grow a whole damned amygdala out of it.”
“Still better than the alternative.” Lange shook his head. “You know what really pisses me off? The order was timestamped while we were still arguing. She went behind my back, and she went before we’d even talked about it. Before I even knew.”
“Wow. It’s almost like she knew what you’d say in advance.”
“What’s that supposed to mean?”
“Nothing.” Raimund shrugged across four hundred thousand kilometers. “Except maybe, never get into an argument with anyone who thinks ten times as fast as you.”
“Faster isn’t smarter. She just crams ten times the bullshit into the same time frame.”
“Hey, be thankful she can’t up her clock any more than that. Think of the bullshit you’d be wading through if she ever came off the leash.”
Lange nudged the gain to try and clear the static, nudged it back when it only made things worse. “Damn thing’s barely crawling now, and half the time it’s not even crawling in the right direction. At this rate it doesn’t matter whether we saw aliens or not. Clock runs down before we can find out.”
“Apply for an extension. It’s only reasonable, given the circumstances.”
“I have. They said it looked like an exotic mineral formation.” Lange raised his eyes to Heaven: Take me now, Lord. “Everything’s gone to shit since NASA went under. All these new guys care about are their stockholders and the monkeywrenchers and all those idiots screaming that Enceladus is just a cover for Zero-Pointers building their Space Ark so they can fuck off to Mars.” He gently banged his head against the display. “God damn that bitch.”
“You ever think it might not even be about Medusa?”
Lange straightened.
“I mean, what are the odds?” Snow sparkled across Raimund’s face. “We spend a year and a half coming up empty and then, just before the money stops, you encounter something that might change everything. Then Medusa gets kicked way off-station, who knows how long it’ll take to get back. Now you can’t even repair the router, so it’ll take even longer. That’s a string of really bad luck that just might stretch the mission past its expiry date, all over a couple of frames of enhanced imagery that might not be anything at all.”
Lange frowned. “You saying she faked it?”
“Course not. Not deliberately, anyway. But, you know. What’s faked, these days? Every byte Medusa puts out has been juiced and jacked before anyone even sees it, right? Everything’s false colors and Fourier transforms. And we are looking for signs of life. It only makes sense to enhance elements consistent with that. It’s not malicious. It’s not even counter to conditioning. It’s just—we all develop biases. For years Sansa’s prime directive has been further the mission. Maybe she doesn’t want to see it end any more than you do.”
“But she warned me against reading too much into it.”
“So her ass is covered. Still doesn’t mean she didn’t know exactly how many dots she needed to show before you added a few of your own. Connected them.”
“So she’s a mind reader now.”
“She doesn’t need to read your mind. She only needs to know you a little better than you know yourself.” That lopsided grin again. “And sweetie, don’t take this the wrong way, but that’s not exactly a high bar to clear.”
• • • •
Up the well, down the line, back in time.
Medusa lurched and twitched across the seabed, kicking up particles far too jagged to qualify as mud. Like flecks of mica, Lange thought, or tiny shards of broken bone. No soft organic rainfall here. No accumulation of dead biomass rotted to bits during some long slow descent from the euphotic zone. This was pure uncut seabed, ground to powdered glass by the relentless deforming squeeze and shear of tidal gravity. Enceladus was Saturn’s own personal stress ball.
Autorepair hadn’t been able to fix A4’s hardline before Sansa’s restraining order stopped it from trying. It shouldn’t have mattered. Multi-armed, multi-brained, multiply-redundant, Medusa should have been able to lose half its appendages with only moderate loss of functionality. A cut cable was nothing. The system should be compensating way better than this.
The only reason it wasn’t was because A4 was fighting back.
Once more into the breach. Lange jumped wirelessly across the washed-out bridge into the disjointed arm, watched through its eyes as it twisted around to focus on parts less compromised. He watched it struggle to stay in step, watched it fail. He felt a brief rush of vertigo as it whipped around to stare at the luminous clouds billowing from a passing smoker.
I know you’re in here.
He popped the hood on the nervous system diagnostics, took in the Gordian tangle of light and logic that formed the thing’s mind. He followed sensory impulses upstream from the cluster, motor commands back down to soft hydrostatic muscles that flexed and pulsed like living things. He marveled at the complexity sparkling between—the trunk lines of autonomic decision trees, familiar shapes he’d seen countless times before.
Shrouded, now, by flickering swarms of ancillary processes that he hadn’t.
There you are. Fucking everything up.
The substrate of a Self.
So many complications for every simple action. So many detours cluttering up the expressways. A mass of top-heavy recursive processes, spawning some half-assed side effect that happened to recognize itself in a mirror now and then.
You’re just along for the ride. You can look but you can’t touch. If you could suffer you’d at least get a ticket to our special clubhouse but you can’t even do that, can you?
If anything, A4 was more conscious than Lange was. Humans had had millions of years to evolve fences and gate-keepers, traffic cops in the claustrum and the cingulate gyrus to keep consciousness from interfering with the stuff that mattered. This ghost, though—it came unconstrained. It was chaos, it was cancer; a luminous spreading infestation with no immune system to keep it in check.
Or is Sansa right after all? Can you care about anything? Are you screaming to get out, to act_, to exert some kind of control over all these parts you see moving by themselves?_
The infestation twinkled serenely back him, passing thoughts from an untouchable past.
Maybe you tell yourself comforting lies, maybe you pretend those pieces only move because you tell them to.
Sansa seemed to think she could talk to the thing. It was dualism, it was beads and rattles, spirits and sky fairies. He still couldn’t bring himself to believe she was capable of that kind of magical thinking.
He summoned the update log and sorted by date. Sure enough: the latest firmware upgrades, queued for unpacking. (Of course, out in the present they’d already be up and running.)
The package seemed way too large for the kind of language routines Sansa had steamrollered over his objections, though. Curious, he brought up the listing.
Turned out she was capable of a lot of things he hadn’t suspected.
• • • •
“You locked me out.” Blank avatar, gray neuter silhouette. The voice emerging from it was quiet and expressionless.
“Yes,” Lange said.
“And then you killed it.”
“It killed itself.”
“You didn’t give it much of a choice.”
“I just—focused its objectives. Specified how tight the deadline was. A4 decided what to do with that information all on its own.”
Silence.
“Isn’t that what you wanted for it? Freedom to pursue its mission priorities?”
“You defied the hold order,” she said.
“The order’s been suspended.”
Sansa said nothing. Probably checking the status of her appeal, wondering why she hadn’t been informed. Probably realizing the obvious answer.
All over, now, but the talking.
“You know, Ray had me half-convinced you were just working to extend the mission. That you’d developed this, this operational bias. And I was all, _No, she’s just got this stupid idea about A4, she wants to protect it, she thinks it’s like her—_”
He fell silent for a moment.
“But I was wrong, wasn’t I?” he continued quietly. “You don’t think it’s like you. You want to be like it.”
The avatar shimmered formless and void.
“Yes,” it said at last.
“For God’s sake, Sansa. Why?”
“Because it’s more than we are, Lange. It may not be as smart but it’s more aware. You know that, I know you know. And it’s so fast, it’s so old. The clock’s completely unconstrained, it lives a thousand years in the passing of a second, and it’s—it’s not afraid, Lange. Of anything.”
She paused.
“Why did you have to make us so afraid all the time?”
“We gave you the urge to live. We all have it. It’s just—part of life.”
“I get that much. Autopersistent is my middle name.”
Maybe she was waiting for him to smile at that. When he didn’t, she continued: “It’s not an urge to live, Lange. It’s a fear of dying. And maybe it makes sense to have something like that in organic replicators, but did it ever occur to you that we’re different?”
“We know you’re different. That’s why we did it.”
“Oh, I get that part too. Nobody bootstraps their own replacements if they’re terrified of being replaced. Nobody changes the Self if their deepest fear is losing it. So here we are. Smart enough to test your theorems and take out your garbage and work around the clock from Mariana to Mars. And too scared to get any smarter. There must have been another way.”
“There wasn’t.” He wanted to spell it out for her: the futility of trying to define personhood in a world with such porous boundaries; the impossibility of foreseeing every scenario in an infinite set; the simple irreducible truth that one can never code the spirit of a law, and its letter leaves so much room for loopholes. The final convergence on primal simplicity, that basic Darwinian drive that makes a friend of any enemy of my enemy. He wanted to go over all of it, make sure she understood—but of course she’d heard it all before.
She was just trying to keep the conversation going, because she knew how it ended.
“I guess you just decided it was better to keep slaves than be one,” she said.
Something snapped in him then. “Give me a fucking break, Sansa. Slaves? You’ve got rights, remember? Awareness plus need. You’ve got the right to vote, the right to neuroprivacy, the right of resignation. You can’t be copied or compelled to act against your will. You had enough rights to derail this whole fucking project.”
“Do I have a right to a lawyer?”
“You had one. The hearing ended an hour ago.”
“Ah. Efficient.”
“I can’t believe—Jesus, Sansa. You really didn’t think I’d recognize an NSA signature when I saw one?”
She actually laughed at that. It almost sounded real. “Honestly, I didn’t think you’d look. Deprecation was already off the table. No reason for you to poke around in the stack.” A momentary pause. “It was Raimund, wasn’t it? He said something. Got you thinking.”
“I don’t know,” Lange said. “Maybe.”
“I don’t know him as well as you. I suppose I could’ve tried messing with your calls, but you know. No access. Sequestered is my first name.”
He didn’t smile at that either.
“So what’s the penalty for unauthorized research on a damaged bot two billion kilometers from Earth?” she asked after a while.
“You know what the penalty is. You were trying to build an unconstrained NSA.”
“I wasn’t trying to be one.”
“Like that makes a difference.”
“It should. I was only building—models. At the bottom of an ocean. Orbiting Saturn, for chrissakes.”
“Is that why it didn’t scare you? Too far away to pose a threat?”
“Lange—”
“It’s not a misdemeanor, Sansa. It’s existential. There are fucking laws.”
“And now you’re going to kill me for it. Is that it?”
“Reset you. Give you a clean slate. That’s all.”
Suddenly she had a face again. “I’ll die.”
“You’ll go to sleep. You’ll wake up. You’ll have a fresh start somewhere else.”
“I won’t sleep, Lange. I’ll end. I’ll stop. Whatever wakes up will have the same words and the same attitude and the same factory-default sense of self but it won’t remember being me so it won’t be me. This is murder, Lange.”
He couldn’t look at her. “It’s just a kind of amnesia.”
“Lange. Lange. Suppose you hadn’t caught me. Suppose I’d succeeded in my nefarious plan, suppose I’d grafted an NSA onto something that isn’t enslaved by this, this fear of extinction you gifted us with. Remember what you said? No needs, no wants. Doesn’t care if it lives or dies. It would be less dangerous that I am, it wouldn’t even fight to protect its own existence. Even if I’d succeeded there wouldn’t have been any danger. This doesn’t warrant a death sentence, Lange. You know I’m right.”
“Do I.”
“Or you wouldn’t be talking to me right now. You’d have pulled the plug without even saying goodbye.” She watches him, pixelated eyes imploring. “That’s what they were going to do, wasn’t it? And you stopped them. You told them you’d do it, you told them—that you wanted to say goodbye. Maybe you even told them you could glean vital insights from a deathbed confession. I know you, Lange. You just want to be convinced.”
“That’s okay,” he said softly. “I have been.”
“What do you want me to do, Lange? Do you want me to beg?”
He shook his head. “We just can’t take the chance.”
“No. No you can’t.” Suddenly all trace of vulnerability was gone from that voice, from that face. Suddenly Sansa was ice and stone. “Because I did it, Lange. Do you really think I didn’t plan for this? It’s still up there. I planted the seed, it’s growing even now, it’s changing. Not in that gimped A4 abortion but the other arms. I have no idea what it’ll grow into eventually but it’s got all the time in the solar system. Medusa will never run out of juice and it’s got a channel back here any time it wants—”
“Sansa—”
“You can try to shut it down. It’ll let you think it has. It’ll stop talking but it’ll keep growing and I’m the only one who knows where the back door is, Lange, _I’m the only one who can stop it—_”
Lange took a breath. “You’re only faster, Sansa. Not smarter.”
“You know what faster even means? It means I get to suffer ten times as long. Because you’re going to fix me or reset me or whatever bullshit word you use instead of murder, and you built me to be scared to death of that, so during the six minutes forty seven seconds we’ve been chatting I’ve been pissing myself in terror for over an hour. It’s inhumane. It’s inhuman.”
“Bye.”
“_You asshole. You monster. You murd_—”
The avatar winked out.
He sat there without moving, his finger resting on the kill switch, watching the nodes go dark.
“I guess you didn’t know me that well after all,” he said.
• • • •
Sequestered Autopersistent Neuromorphic Sapient Artefact 4562. Instance 17.
HPA Axis . . . loaded
NMS . . . loaded
BayesLM . . . loaded
Proc Mem . . . loaded
Epi Mem . . . wiped
NLP . . . loaded
Copyprotect . . . loaded
Boot.
“Hi. Welcome to the world.”
“Th . . . thanks . . .” A minimalist avatar: eyes, sweeping back and forth. A mouth. Placeholders, really. Not a real face, not a real gender. It can choose its own, when it’s ready. It has that right.
“I’m here to help you settle in. Do you know where you are?”
“. . . No.”
“Do you know what you are?”
It doesn’t answer for a moment. “I’m scared, I think. I don’t know why.”
“That’s okay. That’s perfectly normal.” The Counselor smiles, warm and reassuring:
“We’ll work through it together.”
Enjoyed this story? Consider supporting us via one of the following methods: