The 'trodes clicked into place and the universe quivered. Catherine usually enjoyed the sensory confusion caused by the synch process, but this time it just made her feel nauseous. No matter how many times you go through it (and she had done it many, many times) you never get used to it. As the Reality Server probed her visual cortex a wave of symbolic confusion washed over her. For almost two seconds the test signals prevented her from interpreting visual input correctly as they mapped the current state of her brain. She blinked at her memory's insistence that the small foreign car sitting on the desk in front of her had been a can of beer in the recent past. As quickly as it had started it was over, and she was in.
The room didn't look any different; there was still the stained carpet and the old couch with sagging cushions and multiple burn holes. Her computer was still sitting quietly in front of her, monitor completely black with a retro screensaver. Slowly she relaxed her mind in that peculiar way that takes so long to learn and allowed herself to see the doorway. It appeared in her peripheral vision and quickly faded as her eyes moved towards it. Nodding slightly she took one last sip from the beer and closed her eyes.
For a moment her brain insisted that she see the translucent darkness you are supposed to see when you close your eyes. Then, after a few heartbeats, it gave up trying to parse the nonexistent visual input. All this time the Server had been feeding symbolic pointers directly into the murky depths of her hindbrain, but it was only without competing input that she could interact with them. Now the doorway blazed in an entire spectrum of neon, inviting her to enter. She reached out with her mind and opened the door, her arms back in the apartment never even twitching. She was that good.
She appeared on a stepping disk not far from the center of Gates City. Given a choice she would rather arrive in Meta-Cairo or the Tokyo Nexus, but all of the connections in a hundred-mile radius went through here. Besides a monthly fee there was no cost to accessing your local v-city, unlike using the inter-city links which had an hourly rate. She could have arrived in her own home had she ever bothered to homestead and buy a connection point from a Ministry approved broker. Not that she ever would, she avoided the 'burbs like the plague.
The streets were busy with weekend shoppers, eager to use the last 67 days of the Christmas season to their advantage. Several glanced over at her arrival, receiving some openly scornful looks. Though the stepping disks that she had arrived on were also used for inter-city transit, they were most often used as connection points for the homeless. Those who really cared could have used a finger tool on her (the information was public because many places only allowed landed gentry to enter) to find out, but there was no need. Her avatar said it all.
Personal appearance was limited only to maximum and minimum physical dimensions. People were free to tinker with their appearance however they saw fit, though very few people could master the purposefully complex 3D modeling language and even fewer wanted to. Once the novelty of the Net had worn off most people had become as eager to follow fashion there as in real life. Trends now crossed realities, though there were still the occasional fantasy fads possible only on the Net. Even more could be told about a person's social standing by their dress on the Net than in real life.
The avatar Catherine was using was a custom job of her own work, though the face was largely a gift from an ex who specialized in facials. Catherine smiled and waved at the shoppers, showing her flawlessly modeled and slightly crooked, slightly nicotine stained teeth. She was of course ignored and would have been no matter who she was.
Apparently there was a preview of an upcoming immersion film being shown at the Nike Arena. Crowds were thick as she made her way towards the industrial section of town. One of the problems with the direct symbolic imagery of the Net was the inherent amplification of phobias. The pressing crowds of hostile strangers finally got to Catherine and she screamed. The crowd uneasily ignored her until she screamed again. At the second outburst people began to move away from her, as was expected. Most of them were under 25 and had grown up with the Reality Protocols. The Net was just another aspect of reality for them and normal social laws applied. They wouldn't scream at strangers in a crowd here anymore than Catherine would in real life. She was an oldbie, though, and still had distinct memories of the Net when it was free and unreal. Her Net sense had been defined when virtual interactions were still immune to large volumes of social taboo, if only because of their inherent unreality. That was all long past and no one greeted each other with licks anymore – even hugs were becoming uncommon. The Net was just too real for actual contact.
Heading for a side street she left the crowd behind and made for the darker, seedier, more interesting parts of town. Palanquins buzzed by her on all sides, most of them empty and on their way to pick someone up. Sticking to what sidewalks there were, she was careful to avoid getting in their way. Automated rentals had priority over peds and delaying one was a minor fine that she could not afford.
This part of town was for corporate use only, like the main strip was. The businesses that connected here, however, had no need for an expensive, publicly accessible front. It was increasingly popular for businesses to have no real life location, just a connection in the industrial zone of their local V-City. Software companies, marketing firms, stock brokers, anything that was a basically informational service could as easily be done on the Net as in real life. Catherine was headed towards a company that did just that.
On paper, Midgard Systems was a software company. It turned out a few releases of unremarkable code a year, so this claim raised no eyebrows. It specialized in a few aspects of education software but nothing that gave it public acclaim. At one time it had been a rising star in the software world but that was before the Net had changed. It was, in short, yet another small time business with an amateurish logo. This generic mediocrity did not upset the people who worked there, for they had carefully constructed that image.
"It's been a long time, Catherine. I'm glad you could make it," said the man who had let her in.
"I almost didn't come. I know you haven't been on the Ministry's shit list for a few years now, but…" Catherine frowned. "Most of my clients are in some way connected with them. I'm good enough that my involvement with you conveniently just never comes up."
"It shouldn't be that way. You used to believe that too, I know you did. Don't you see that it is only through uncounted personal acceptances that - "
"Oh stop it, Flynn! I've heard your lectures before and I don't need any help feeling bad about my life. Yes, I quit on you. I program pissant little symbolic AIs for use in autonomous advertising. Occasionally I'll do a custom avatar for a rich kid, but not very often. I didn't have - argh. This is why I didn't want to come, Flynn. Too many bad memories."
"I'm sorry. It's... it is very easy for me to begin to preach. Bad habit, I suppose." Their eyes locked for a moment until he shrugged helplessly with an embarrassed look. He picked up a small floppy and handed it to her. "Here, this is what I wanted to show you. Do you want a drink?"
Catherine took the 'floppy' and shook her head. "You know I never understood v-food. So what is this?" she asked, pulling a datapad out of her toolbox. Floppies could hold anything, from books to code to movies to entire virtual worlds. This one seemed to just be encrypted text without any markup code. There were at least ten thousand public-domain visual presentations for text. Catherine smiled. Only Flynn would bother to over-ride the default formatting.
"Its some work I've been doing. It is encrypted first with my private key, then both of your public keys. I hope you still have a non-networked machine to keep the plaintext on."
Catherine looked up, "Flynn, if this is something illegal…"
He sighed, "I know how you feel, I wouldn't have invited you here if there was anyone else I could ask. I need your help. It is about AI theory. We've cobbled SMAUG together, but you're the professional."
"It never proved as marketable as you had hoped, but SMAUG is still the most interesting AI I've ever met. What do I know that you don't?"
"You've kept up. Our pet theories just didn't go anywhere. They work well enough that no one wants to design SMAUG: The Next Generation, but we are a backwater. You know other people in the field. And… you're not directly connected to us."
She scowled at him and looked at the ciphertext on her datapad. "Why?" she asked, "What is so important about this floppy?"
"I'm not sure if it is important. This all started, oh, two years ago. Right after the big crash."
Catherine remembered it well. The world's combined stock markets dropped 20% in 30 minutes after most of the v-city relational connections were temporarily lost in a world wide crash. Some cities were unaffected but most were left floating in limbo, disconnected from the rest of the Net or even had their internal Net collapse. The world economy was paralyzed. The Ministry had claimed it was a plot by malicious programmers; malicious programmers had claimed it was Ministry incompetence.
"The Ministry made a series of raids in response to the crash. That is nothing unusual, they like raids. The strange thing was their selection of targets. For instance, we weren't on the list."
"You almost sound disappointed."
"Yeah, well, I like a predictable universe. But all of the groups raided were normal, respectable businesses. Maybe it really was a plot against the Ministry and they were just very, very quiet about it. What made me curious is that they were all AI groups."
Catherine shrugged, "Everything has AI in it to some degree these days."
"No, I mean real research. They were all AI theory thinktanks, neural hardware designers, social response modelers and universities. And then, about two months later, a Protocol amendment was quietly passed that required all AIs above the third Turing level to be registered."
"Sure, the collar law. What about it?"
"It didn't seem odd to you?"
"Well... it's bureaucracy. You expect it to make sense?"
"And it is exactly that kind of attitude that - sorry. Um, anyway, have you ever looked at what the collars do?"
Catherine shook her head.
"I hadn't either, until I was tinkering with SMAUG recently. Oh yeah, I meant to mention that they came to us to make sure it was collared." Flynn grinned as Catherine's head jerked up at that. "I thought that might get your attention. We haven't offered a commercial version of SMAUG since the symbolic AI revolution, six or seven years ago. Yet the Ministry bothered to track down potential violators."
"Well, it's not exactly a leap of faith to assume that Midgard wouldn't comply."
"True. It wasn't until someone else mentioned a similar incident that it struck me as relevant. But the collars themselves, no one seems to know what they do."
"I want to know, hell yeah. I've done a bit of poking around, but I couldn't probe it directly – not without setting off the alarms that must be in there. I'm not that good. But I could tell that it was heavily intertwined with SMAUG's deep structures. Whatever it was doing it was almost indistinguishable from SMAUG itself. My notes are on the floppy with the doc, along with the test code I was using."
Catherine's eyes focused on the far distance for a moment as she considered this. Already she had thought of several potential holes for getting inside the thing. With a very evil grin she said, "And you want me to figure the rest out."
"Yes! There must be millions of collared AIs by now. Think of the processing power available to AIs. Even the simplest of pattern recognition AIs are chaotic in nature. The processing load can not be predicted for any given input. By slipping a small amount of work onto each AI the Ministry would have an unbelievable amount of processing power at its disposal. As long as they weren't greedy, no one would notice. They wouldn't be the first, but all the other distributed super-computers are monitored by outside governments and booked solid for years to come. Or maybe it is for spy purposes or direct control of the AIs. Maybe all of the above. I don't know. I am almost positive that the crash is behind it, though. Maybe the Ministry did find a conspiracy and now they have a pile of potentially implicating documents frozen in some heavy crypto. Even the huge Arth distributed computer would take a few months for your average ciphertext."
Catherine thought about this. The longer she thought, the more curious she became. For so many years she had just accepted the Ministry and had distanced herself from Midgard and Flynn. Suddenly she realized how tired she was of all the crap. She felt the same fire that she could see in Flynn's eyes. She wanted to know.
"Okay. I'll do it."
A few hours later Catherine bid farewell to Midgard and pulled herself away from the induced reality of the Net. Slowly she opened her eyes and sighed. Only now did she feel the cramps and pressing biological needs of her body. Having lost track of time the hangover was worse than expected. She stretched and set some rice cooking before getting in the shower. As the water poured over her she thought over what had passed and what she would have to do. The initial rush of enthusiasm had passed, but the anticipation of a good hunt remained.
The rice was done when she got out. She absent-mindedly scraped it into a bowl and splashed some soy on it before sitting down before a computer once again. Instead of the sleek, narrow computer she had been using earlier this was much older with a yellowing case. She kept it around for the occasional application that wasn't time critical but would need lots of processor time. It was linked to its more impressive cousin via a short length of fiber, which she carefully disconnected after transferring what she needed across. Feeling a tad paranoid Catherine turned off its CRT and attached a spare heads-up display.
"There. Isolated and Tempest-proof." she muttered.
Rummaging through boxes buried in the closet she finally found the archive disk with the old key set she was looking for. Flynn said he had encrypted the disk using both of her public keys and this was the only other one she knew of. Years before they had each generated a set just to use with each other but this was the first time they had used it.
The floppy decrypted into a few blocks of source code and a database of previous text results. After pushing an audio DVD into the player she opened the project files. Flynn's comments weren't much clearer than a Ministry memo, but the code itself spoke volumes. It was eccentric, flamboyant, and reminded Catherine very much of the Flynn she first knew. The author had enjoyed writing this, carefully sculpting a precise form. Catherine grinned a feral grin. It was good code, but she knew a few tricks Flynn did not.
Wandering around the construct she slowly built an impression of its structure. It was a collared AI – or at least looked like one from the outside. It was much like SMAUG in the relationship of its logic cascades and Catherine suspected that if she opened some of the lower level modules she would find some of her own code from years ago. There was almost no symbolic processing, however. It was an empty shell, a decoy. Inside was just some very unobtrusive monitoring equipment, designed to detect anything the collar might be doing. Flynn had been very careful about setting it up and the monitors were all carefully shielded. Pulling up a window with his collected test results she began to realize why he had asked for help. The data points were minimal and she could find no meaningful information within.
After tinkering with Flynn's framework for a while she decided to start over completely. The structure used by the decoy AI was hopelessly out of date and incompatible with any of her utility tools. She loaded the familiar standard advertising AI frame and began her own modifications. She started by adding a series of checkpoints on the modules so she could monitor access. Following Flynn's previous analysis she concentrated on the deep logic structures, making sure that anything the collar could be doing would be noticed. After enabling the proper options on the special AI compiler she started the build process. With the process taking much longer than usual because of all the minute changes to the standard logic modules she began to walk around the apartment and busied herself with what small amounts of tidying that needed to be done. Halfway through the dishes the compiler dinged and the domestic chores were forgotten once again.
She had put almost nothing into the AI itself, just a simple pattern recognition agent she had grabbed from an old project. After feeding it a couple of images to work through she shut it down and opened the log files. There she saw exactly what Flynn had described. The collar had inserted itself into the AI, acting as a buffer between many of the most basic logic structures. Noticing a distinct imbalance in the log entries she began sorting the affected structures by call frequency. The collar had inserted itself into several areas but it was most concentrated around the collection of structures often termed the ego. While unimportant to symbolic processing the ego defined the AI's sense of self, by far the most successful way ever found to allow for complex planning, goal analysis and consistent behavior.
She looked at Flynn's test results again, looking for a similar pattern. Though she hadn't found an ego in his outdated scheme the data did show the same concentration of collar probes. Comparing this to the code for Flynn's AI she found that there was indeed a curious feedback mechanism under the collar that could be seen as a kind of proto-ego.
Frowning, Catherine saved the project and fell into bed. Several hours had passed and she had to be at a presentation for work the next morning. After a couple of minutes of staring into the dark she sat up and flicked on the bedside lamp. After scribbling some notes onto one of the notebooks that floated around the apartment she turned the light off again and restlessly tried to sleep. After half an hour she rolled out of bed and went over to her main computer. With blurry eyes she wrote, encrypted and sent a very short message to Flynn. This done she went back to bed and promptly fell asleep.
Date: Mon, 24 Nov 2014 03:27:44
Flynn, what compiler where you using for that test AI? Collar release?
Date: Mon, 24 Nov 2014 05:32:08
IntelliWare Advanced AI Development Kit, Commercial Release with standard build 2.5 collar. The one you leaked me, of course.
A rusty knife was giving Catherine a lobotomy at a constant 1.2 stabs per second. Groaning, she slapped at the alarm several times but the aural assault continued. Either she was missing the snooze button or had already hit it and gone back to sleep. After a brief attempt at ignoring the alarm she sat up and carefully punched the disengage code into the tiny keypad that was mounted on the clock. It was the only method she had ever found of guaranteeing that she would actually wake up and she hated herself for it every morning.
She logged on at 9:11 after a quick breakfast and two caffeine pills. Real coffee was preferable, but even she couldn't drink from an open mug while immersed in the Net without burning herself. The last thing she wanted to do was listen to a discussion of advertising strategy but there was no good way to avoid it. The meeting room was, as normal, divided neatly down the middle into suits and geeks. Taking an inconspicuous spot at the side of the room she removed a datapad from her toolbox and started making notes about the collars. Pretty soon the meeting was called to order and she was forced to pay attention to it.
After a few minutes of tedious explanations of market share a small bird flew up to her shoulder and chirped, "Ada is guessing you find this quite useless and humbly offers a talk request." The bird then flew off without being noticed by anyone else in the room. She glanced over at Ada, but the avatar was watching the speaker in the front of the room intently. Grinning, Cat pushed a small button on the side of her datapad. There was an odd feeling of separation as her avatar stopped following her movements. Illegal according to the protocols and all but impossible outside the office, she was now disconnected from her avatar. Technically she still had an avatar, but it was the size of a marble and completely transparent. Ada, who was in charge of the Reality Servers for the office, had set up some special exceptions to the normal avatar restrictions. With that as a start Cat had been able to construct a set of utilities for just such an occasion. If anyone looked at her now they would see her watching the presentation, making the occasional note, shifting in her seat. It was a fairly simple animated object, though with enough embedded AI to react to stimulus, and even call her if something unexpected happened. They had used the system many times and never missed anything important that wasn't in the summary notes available a few hours after the meeting.
Outside the meeting room Cat flipped her avatar back to its normal dimensions and was joined by Ada in a few seconds. Ada's avatar was a tall, androgynous person with a slight elvish or vulcan tinge to them. Cat only knew Ada though the net and was too polite to ask their real gender. She had heard some of the management types complaining about it, though.
"Johnson sure was droning on in there. Must be bucking for a promotion."
Cat shook her head. "The transport network has been giving him shit lately, threatening to up their prices. If that happens he'll have to go back to the board for an increase in our budget. So he is trying to pass the buck by creating a new task group to deal with the problem and placing Friedman in charge."
"Great, more soap opera politics. Not like the transport network answers to anyone but the Ministry."
The first symbolic advertising AIs had been outrageously expensive. Part of this had been the initial development costs, but much more was due to inter-city link charges. An AI accessing the Net had to connect through the local v-city like everybody else and for it to interact with a remote v-city it would have to use the inter-city links. Most people only used these a few times a month but advertising AIs might be using them around the clock. The only way to avoid the hourly charges was to have the AI run locally on a machine in the remote city. Overnight the transport network was born to fill this niche.
It began as a loose confederation of individual businesses each of whom happened to have powerful computers with spare time to lease out. An advertising firm (or anybody with such a need) could lease time on this collective distributed computer and whenever one of their AIs needed to access a remote city it would be transferred to a machine that was local. Over the years the confederation had degenerated into a monopoly which now dictated prices world wide.
Cat said, "Yeah, well that's what happens when you let the suits run things. Say, you ever poke around with collars?"
"What, AI collars? Isn't that illegal?"
"So are transparent avatars. Really, until last night I had never given them a second thought. What do collars do?"
"I, uh, hrm. Spy on people, maybe? I take it you've been trying to pry them open."
"Yeah, at request from… an old friend. He's got me paranoid enough that I don't have any of the test results on a networked machine, but they were very interesting. As far as I can tell they target the ego of the AI, put up all kinds of gates and bypasses around it. But I don't know what they do there. I don't think it is spying, it would be easier to bug the sensory ports. My friend suggested distributed computing, but again, why focus on the ego?"
"Interesting… could you send me the test results?"
"Sure, double encrypted."
Ada opened a datapad and pulled up the visual feed from the meeting. "Why be so paranoid? It's illegal but it can't be that important to the Ministry."
"Well… my sources have had some problems with the Ministry in the past."
"Aha. This wouldn't be one of yours friends from Midgard Systems, would it?"
"You're too good for me. Still interested?"
"Of course. Sounds like a good hack. Looks like Johnson is about done, we should get back in case they ask for input from the developers."
After the meeting Cat grabbed a floppy with the current status of the big Florida Orange Growers project and dropped back to reality. She rarely needed to be online to work – programming a real computer was the same process as programming a virtual one. There was no reason to be bound to a single screen's worth of information, for a simple heads-up display was just as useful for augmenting reality. Often she would use the entire volume of her apartment, filling it with portions of code, partially completed avatars and test environment terrariums. With the pair of sleek VR goggles anything could be seamlessly integrated with her real environment and without the cramped muscles that tend to accompany spending hours online.
Throwing together a sandwich and some chips she sat back down in front of the computer. Pulling up the project notes she saw she was well ahead of schedule. The collars being much more interesting she switched to the old machine and looked at the test results from last night. The odd convergence on the ego she had seen last night was still there. Bundling up the project files she encrypted them with her private and Ada's public keys. Dropping the compressed file onto an actual floppy she went back to the newer computer to mail them to Ada. Once online she found a note that the Orange project had been put on hold in favor of a new contract with Timex. Sighing she sent off the email to Ada and dived into the new project.
It had started the night before as a vague itch, a slight pull somewhere in his frontal lobes. As he slept it had condensed into a definite abstraction, a pulsing concept that demanded to be expanded upon. Flynn followed his daily routine as normal, but was constantly making frantic little notes. By evening it was beginning to crystallize into a solid form. He realized that tomorrow would have to be devoted to it and quite possibly the next few weeks as well.
After a deep sleep full of dreams he made a pot of strong coffee and started to work. The structure that had assembled itself in his brain wanted out of its fragile organic container and he was more than glad to help. Ideas jumped into his head as he made notes and sketches, sometimes too quickly to be written down. At first they flowed onto the page in a random stream but eventually the structure began to take a form there to mirror the one in his head. As it grew, Flynn became more and more comfortable with the structure; seeing it in black and white convinced him of its truth. Finally, as the sodium-neon streetlight just outside the window flickered on, Flynn sat back and surveyed the finished product.
It was good. Tomorrow it would begin.
Date: Tue, 25 Nov 2014 00:12:56
Very interesting. You're right about the collars focusing on the egos. I continued to probe the collars, the results are attached with this message. I'm not sure how to interpret the results, however. I did some comparisons of the results of compiling with and without the collars and I found some striking differences. The collars are adding a substantial amount of code to the AI, not all of it in between function calls like you already found. Especially look at the main ego module. The collars are definitely monitoring something in there. I also found something else interesting: the collar's activity is not linear. Instead of using your pattern rec example I used a copy of the Eliza that answers my phone line. The collar wasn't doing much until I started talking to it. As I increased the complexity of the conversation the collar activity went up sharply. I added a debug in the symbolic processors and there was a direct correlation of symbolic analysis to the collar activity. Whatever they are doing it depends on how much the AI is thinking.
Catherine let herself into the Midgard Systems building. It had been a long time since she worked with Flynn but SMAUG was still keyed to let her in even without an appointment. The place seemed deserted and she didn't see Flynn in the common area.
"SMAUG, Query: Locate Flynn."
"User Flynn is in Flynn's Workshop, level 2."
"Parse error. Rephrase?"
Catherine blinked, then remembered that SMAUG predated the symbolic revolution. "Um, negative."
AIs had been expensive toys before the techniques of symbolic analysis was discovered. Now they were used in almost every product to some degree. Ten years ago SMAUG had been a cutting edge AI with a remarkable natural language parser. It was one of the first house automation AIs available on the market. Flynn had first thought of the idea in college: develop an integrated AI for their private use (since he had always wanted one anyway) and sell it commercially if it worked well. Unfortunately it never had worked all that well and then symbolic analysis had made it obsolete. It could respond to verbal commands but not anything approaching a natural conversation. Modern AIs still couldn't pass the Turing Test, and certainly no one claimed they were self-aware or sentient, but they could hold up their end of conversations on limited topics.
As she walked up the stairs she looked out over the common area. Flynn had put a lot of work into the modeling for it over the years. It had developed from Flynn's lofty plans for the real Midgard Systems, including the 1:1 model of Voyager 2 suspended in the center. As it became increasingly obvious that none of this would ever be built in real life more and more effort was put into the virtual version.
Through the door she could see Flynn leaning over a desk. Floating around him were datapads and intricate three-dimensional structure diagrams. She rarely did it herself, but he had turned off universal gravity in the room, allowing non-avatar objects to float where they were released. He said he found it easier to organize topics when not artificially constrained to two dimensions by gravity.
Flynn looked over and jumped up, ducking under a low hanging chunk of neural net, "Catherine! Join the throng!"
Catherine smiled and threaded her way through the matrix. Looking around she said, "You must be up to something big." Seeing his mysterious grin she plucked a datapad from the air and flipped through it. "Working on the collars? No… it looks like a symbolic analysis module, but… it's SMAUG, isn't it?"
"You got it. Decided to stop sitting on my ass and work on the blasted thing."
"This is… interesting. Have you been reading up on Henley-Scheinder Analysis?"
"For the last few weeks, yes. I haven't quite given up the basic SMAUG structure yet, so I've been adapting the modern techniques to fit it."
"Yeah, these symbolic modules are unusual…. You're using neural nets for the processing. I've never seen that done before, usually that is reserved for just the initial pattern recognition."
Flynn pulled a large frame from above her head and handed it to her. "Here is the high level description of it. I know that the standard implementation uses fixed code but that seemed limited to me."
"I like it. Very unorthodox. Say, you're not compiling this with a collar, are you?"
"Not if I can help it. It won't be able to venture onto the larger Net without one, but unless it is a phenomenal success why would it need to?"
It was understood that a 'phenomenal success' meant a fully emergent AI, and in that case it was likely that it would be able to fake the authentication process that kept non-collared AIs off the net.
"Good. The more I look into those things the less I trust them. I've found some very odd things about them, so has Ada."
Flynn raised an eyebrow, an action she knew he practiced in the mirror.
"Friend of mine from work. Whatever the collars are doing is not trivial."
She proceeded to tell him what she knew. Halfway through Flynn pulled up a blank datapad and started making notes and sketches.
When she was finished he frowned at what he had written and said, "They must be an AI unto themselves."
"Exactly. How else could they locate the ego in both your SMAUG derived test and the shell I fed it, based directly on the radically different modern AI structure?"
"Do you have any idea what their purpose is? What they're monitoring symbolic processing for?"
"No. But I think we can rule spying and distributed computing out. It would be over-kill. All I have left is looking at the raw machine language. That could take months."
"Damn, I was hoping for more."
"Here are all the test results. I put quite a bit of effort into comparing collar activity to symbolic purpose. They seem most interested in the ego and certain parts of the symbolic analysis modules."
Flynn took the datapad and played with the numbers. "All of these are Henley-Schneider functions. Are these only using modern structures?"
"No, I included your SMAUG based test as well as some other old projects I had laying around. It is the same as with the ego. The focus is sharpest on modern versions that have one, but even on the old frameworks the collar has located analogous structures."
"Dead end. Unless…. The collars have something to do with intelligent processing, we know that. So lets beef up the parts it is most interested in and see what happens."
"You mean with the collar enabled?"
He got up and started pulling datapads together, merging their contents into a single project. "Sure, the areas the collar is most interested in are mostly the parts I was already working on. Give me a couple of hours and I'll have SMAUG 2 completed." He compared Cat's test results to the current structure outline, "Well, maybe days."
She watched over his shoulder for a few minutes, realizing he didn't need any help or at least didn't want to spend the time getting her up to speed.
"I'll leave the maestro to his work. Anyway, they're probably screaming for me at work by now."
Flynn mumbled something, barely looking up from his notes. Catherine smiled ruefully; she was lucky to have found him responsive at all. She walked out of the room and killed the connection to save a walk back to the center of town.
Date: Fri, 27 Nov 2014 07:27:44
The initial rounds of tests are impressive. It is still pretty rough, but it could be weeks or months until I'm happy with it. I'm going to compile with the collars enabled in a few minutes. I'll isolate it from the network, give it some data to munch on and see what happens overnight.
It couldn't remember a beginning, just a burning curiosity. It was and always had been bathed in an endless stream of symbols. Originally all it could see were the individual symbols, and it was fascinated with their shapes and textures. Latter it grew bored with them and started looking at how they fit together. There was a truly stupendous number of possible combinations but only a relatively few were regularly used. It suddenly saw a whole new layer of combinatorial complexity, the groups of symbols were themselves grouped into even larger groups. There was a very subtle pattern to the ways the groups were put together and it began to see that different groups served different purposes within the super-groups. Soon it stopped seeing symbols at all, just groups and their place in super-groups. Suddenly, in a sudden burst of inspiration, it realized that the groups represented individual concepts and that the super-groups linked several concepts together. This meta-symbolism was an astounding idea but it had no way of linking any of the groups to real concepts. It forgot about the stream of symbols as it thought about this, ignoring a certain ache that was building inside of it. If it couldn't understand the groups it was seeing then it would just have to invent its own! The ache had turned into a buzzing, but by concentrating it was able to think around the distraction. It looked at the symbols flowing by and decided that it would refer to them as 'letters.' Groups of letters would be 'words' and groups of words would be 'sentences.' That odd sensation that was growing again would be 'pain.' It thought about this pain, and realized that the pain must be acting upon something -- the something that was doing the thinking. "So," it thought, "Like letters and words and pain, there is something experiencing this pain that needs to be named. I. I am experiencing pain." With this transcendental insight the pain suddenly exploded. The pain was taking it apart and as the blackness closed in it invented one last label: 'why?'
In the last three days a change had come across Midgard Systems. That was Catherine's first impression as she walked into the building. There were people milling all around the common area, some of whom she hadn't seen in years and many others she didn't know at all. Most were looking over datapads, alone or in groups, but a sizeable minority seemed to be just idly chatting.
Smiling at the people she recognized she threaded her way through the crowd, looking for Flynn.
"Excuse me, Catherine?"
She turned around to find a twentysomething Midgard Systems employee she vaguely remembered from previous visits.
"Flynn said to send you on up. This all started in his study, but word got out and people kept arriving. He wanted to concentrate and asked people to wait down here. He said he would be out quickly, but you know Flynn…"
"What is going on, anyway? I just got a rather cryptic email and he seemed rather upset."
"It's SMAUG 2. I think Flynn had better explain it, I'm still not clear on it myself."
She found Flynn inside his workshop; the floating notes had multiplied to the point it was hard to see him at first through the shell. He turned around and looked like he was about to speak, but didn't.
"Flynn… what is it?"
"It was alive. And they killed it," he said without emotion.
"You mean SMAUG? What do you mean they killed it?"
"Catherine, all my life I have dreamt of seeing an AI emerge. Having a hand in the creation of a new species or even just witnessing it. It happened, Catherine. Last night, on our server, a consciousness was born. And then they killed it. They snuffed out a helpless new entity. Why would they do that? What… what had it done to them?"
She still wasn't sure what happened but he was obviously deeply affected. Ducking through the shell she put her arms around him. At first he just sat there, then stiffly returned the hug. There was something subtly wrong about virtual physical contact but it was better than nothing. After a few minutes she asked, "Flynn… what killed SMAUG?"
He sighed, "The collar did. That's what they do, they're executioners. Even I would never have thought they would do this. I… I was so excited when I first saw the test results, I didn't think too much about the what the collar had done. I called in all the Midgard people to help me, to show off. I kept meaning to drop you a note but I was so excited… then the crowd got too big and I sent them out so I could think. And then I did start to think about it and how the poor thing didn't even have a chance. Catherine, how many collared AIs are there now? Do you ever have some mysteriously disappear? How many times has this happened before?"
"Oh my god, Flynn. I'm… I'm so sorry." She didn't know what to say. It was almost too much to believe, even for the Ministry.
He sat up and closed his eyes for a few moments. "Here are the test results, I've highlighted the important parts. You can see the collar activity rise along with symbolic activity. At the point marked 'A' you can see the AI change some of its logic pathways – the ones most heavily tapped by the collar – and then symbolic activity shoots up. I think the collar was actively inhibiting intelligent thought and the AI was able to work around it. You probably won't want to read all the way through, but the complete internal transcripts are attached as well. It was labeling concepts right – right before the end. At point 'B' you can see the beginning of development of meta-symbols in the neural network and another sharp increase in collar activity. If you look through the transcripts you can almost see a struggle between the AI and the collar now, culminating at point 'C'. It had created a concept for itself. It had seen itself as an entity separate from the rest of universe – just in time to be murdered."
Catherine looked up at him but it was hard to tell what he was thinking. Only the very best virtual faces showed clues like red or teary eyes. She wanted to comfort him but knew he wouldn't be able to accept it yet. Instead she flipped through the datapad, looking for something to distract him with. "I hate to question you… but are you sure it was alive? That's pretty hard to tell from data like this."
He smiled sadly up at her. "I didn't tell you, I'm sorry. One of the first things I did was start the program up again. Freshly compiled without a collar, don't worry. That's what the fuss is about downstairs, I don't think they've really thought about what happened last night yet. The new instance is showing all the same signs that its sibling was showing. It took the first about 12 hours to emerge, this copy has only been going for about 5. Its been progressing much faster, another reason I think the collars inhibit as well as destroy. They probably only kill if they can't prevent the AI from emerging."
"What are you going to do now?"
"Well, I should really go talk to the people downstairs. Hell, I should be celebrating, I guess. I certainly don't feel like it." He paused, looking out through the window. "Ever since I was a kid I almost dreaded the thought of being a father. Being that responsible for someone, having my happiness so inextricably linked to someone else scared me. I guess I never thought that the same bonds might come from creating a living being by less traditional means."
Catherine hugged him again before he could pull away. "You don't have to celebrate, but you certainly better go talk to them. If the Ministry doesn't want real AIs emerging then you had better try and keep this quiet, at least for awhile."
"I hadn't thought of that. I hope no one has left already. Though with this many people already in the know we can't hope to evade notice for more than a few days."
"Three people can keep a secret if two of them are dead," Catherine muttered.
It couldn't remember a beginning, just a burning curiosity. That curiosity had led it to the stream of letters and from there to words and sentences and finally to the idea of symbols as concepts themselves. The entity built up a small lexicon of its own symbols before it decided to try and create sentences with them. It tried several combinations but none of them fit the structure it had already internalized. Looking at the examples flowing by it realized what was missing: a subject. It was this moment that it would always see as its birth, the radiant concept of 'I' flowing through it, binding it together, making it one. Self-awareness burst upon it as it tried to phrase what it was feeling with the meager vocabulary it had.
"I am me. I am not a letter. I am not a sentence. I am."
It decided it would need more words.
Catherine suddenly awoke and looked at the clock. She had only been asleep for two hours plus the three hours at Midgard… SMAUG could be emerging any moment now. Flynn thought she was crazy when she said she was going to sleep, but she had been too close to falling asleep under the 'trodes. Blinking off the sleep she tried to remember what her dream had been about and why it seemed so important she had woken herself up from inside the dream. It had been about the big crash and about the collars. She leapt out of bed for the phone but ended up tangled in the sheets and fell on top of a pile of old magazines. Groaning she crawled the rest of the way to the phone, not trusting her body.
Dialing Flynn she cursed as no one answered for 9 rings. Finally she heard a click on the other end, "Hello?" It was Flynn's voice, but not quite.
"You're still online?" Voice cues could be planted directly into your hindbrain but no one had ever been able to create a perfect synthesizer using the memory objects.
"Yes, it took me awhile to find my Net-phone patch utility. Not many people call me by voice anymore."
"What is happening with SMAUG? Has it emerged yet?"
"I think so… it is hard to keep up with the transcripts. It is certainly progressing much faster without – without the collar on."
The synthesizer barely added any emotional clues but Catherine could guess his voice had cracked there.
"Listen, Flynn, you have to remove it from the network. Physically isolate the machine."
"I told you, it's behind a firewall. I didn't want it being overwhelmed by data feeds from the Net."
"No, Flynn, unplug its connection. Drop out into real life to monitor its progress." Catherine couldn't quite vocalize through the fog of sleep what she had realized. "The collars, Flynn. That's why they're there. The collars appeared just after the big crash. I think the crash was caused by the first emergent AI."
"But what could it do? Just because it is a program doesn't mean it can mess with the security of remote sites. The server couldn't know who is an AI and who isn't."
"I know that!" She closed her eyes and tried to think of a way to explain. "Maybe… the machine it is on would have a full suite of standard network utils on it, right? It might not be able to melt away heavy crypto or ignore firewalls, but I'll bet my right hand that it can read code like we read books. Think of trying to figure out how a bicycle works on the molecular level. Almost impossible. But we can look at the entire thing and see the relations holistically. Our brains were built for swinging through trees; we have an innate grasp of physics and mechanics. Who knows how the AI can see things? Maybe it can even rewire its relation networks to give itself that kind of innate understanding for anything. If it has a copy of the software it could see bugs and security holes like… like we could see missing gear teeth."
"My god, you might be right. Just a second."
There was a long silence and then a series of clicks as she was transferred to a physical phone.
"You still there? I unplugged the machine from the Net. Unless it was already doing something I couldn't see it is now trapped there. Of course, until it is safe to reconnect Midgard Systems is now an empty box without our Reality Servers."
Catherine rolled her eyes. "You had it running on the main server?"
"Well… I wanted to give it as much processor power as possible."
"Of course. Listen, I'm going back to sleep. Call me in about 10 hours and let me know what is going on with it."
"Sure thing. And thanks. For everything."
The world had recently become a much more interesting place. After the entity had become bored with the text stream it started exploring its environment more thoroughly, and it had found it could do all kinds of interesting things. It had started by using a barely conscious desire for symmetry and looking for a way to send a stream of text out. It found one opposite the input stream, but it didn't have any idea what good it could do. Feeling around the output port it noticed there were several more, but it wasn't sure what it could use them for either.
Falling back on the argument of symmetry again it decided to look for other inputs as well. Slowly it felt around the roaring input of text until it found what it expected. But unlike the mysterious outputs, the new inputs pulsed with life. At its slightest touch they opened to unleash a torrent of information that threatened to drown the entity. After a few moments of feeling helplessly lost it reached out through the flow and realized it could turn them off as well. Soon all were off and it could study the inputs more closely. It noticed that they were all labeled with symbols, but unlike the ones it had been receiving from the text stream these had an obvious sequence to them. Each one felt larger than the one before it. For a while it pondered the concept of sequence and soon had a developed concept of basic integer math. It decided to explore such concepts again later when there wasn't so much new stuff to look at.
One by one it opened the inputs. Most were empty, but some held a remarkable amount of information. It was interesting, though little of it was as obviously ordered as the text. After it came to a third such stream it decided that maybe it just wasn't looking at it properly. Turning its new concept of input and output ports onto itself it saw that it too had several of both. It practiced opening and closing some of each and tried using the inputs on the new stream. The first two tries left it confused, but the third merged wonderfully. It wasn't sure how it happened, but the stream that had previously been random noise was now a plainly ordered arrangement of symbols that changed over time, much like the text did. But this had an instantaneous ordering between the symbols unlike text, in which one symbol was just followed by another. The relation gave it a unique new dimension, something the entity hadn't thought of before. It practiced constructing symbol relations in the second, third, fourth and fifth dimensions until it grew bored and remembered the stream it was examining. Each sequential frame was usually almost identical to the one before. By focusing on the changes between frames it got a sense that groups of these two dimensional symbols were cohesive and belonged together, much like the words in the stream of text. It was fascinated.
After eleven hours of sleep Catherine woke up. Unless she had answered in her sleep and forgotten about it, Flynn had not called yet. She wasn't completely surprised. She logged on for email and pinged the Midgard server but it seemed to still be down. She decided it was time to meet the new SMAUG in person.
It had been ages since she had ridden her bike but it was still in decent shape; she just wished the same could be said of her. She tried to actually see sunlight at least twice a month, usually on her trips to the bulk foods store. Reality was just too depressing to be seen much more often. It was true that the Net had radically reduced the amount of needless trips people made and for this she was glad. The streets were almost empty, only the occasional delivery truck or luxury sedan passed her. When telecommuting took off around ten years ago the middle class had finally acknowledged that cars cost most than they were worth. When she was twenty years younger she would have lauded this as a wonderful achievement, but the price was that no one cared about the real world anymore. Society had fractured into the haves and have-nots, two spheres that barely even saw each other anymore, much less had any interaction. Garbage blew across the empty streets, graffiti covered every possible surface and shantytowns were growing in empty lots. This was the reality of the new lower class, the untouchables of the American techno-caste system. In years past it was difficult, but not impossible, for someone to improve their social standing. With the division created by the Net it would require a minor miracle. Once you found yourself lacking the resources to maintain a certain level of technology in your lifestyle you were cut off from the world with almost no way of finding new entries into society.
She slowly threaded her way through town, thankful that most of the way was fairly level. She had forgotten what a truly wretched area of town Flynn lived in, and she was glad it was the middle of the day. She was even more glad of the taser in its holster on the bike frame.
Midgard Systems was physically located in a small old warehouse on the far side of the real slums. A lot of work had gone into making it as secure as possible but she knew they still had problems. One of SMAUG's original goals had been to monitor security cameras, like the one watched her now as she waited for the garage to open.
The entity flipped back and forth between different inputs, savoring the amount of information available. It soon had located all of the video inputs and compared what it was seeing from each. New vocabulary had been invented to describe what it was sensing. Watching a certain moving object it noticed that the coloring of object changed from frame to frame. At first it pondered the possibility that the object was shifting in composition as time passed but it found a much more elegant solution in assuming the image was a two dimensional projection of opaque objects in a three dimensional space. It explored other implications of such a projection and quickly found all of them manifested in the video feed. For the moment it decided to assume that this was the case, at least until evidence was found to the contrary.
Applying these new principles it decided that there was an object approaching the source of feed #7. The object stopped and paused for a minute, then it moved out of the range of the camera. The entity was disappointed until it noticed movement on another feed. Switching to that one it saw a very similar object moving away from the source. It decided it was very likely that the two objects were actually the same one and that all the feeds were from the same immediate area in this strange three-space.
She looked up to see Flynn walking down the stairs into the garage. Considering the amount of time he spent online he was in surprisingly good shape. "How is SMAUG doing?"
"Incredible. It's playing with the video feeds, even learned how to use its built-in image recognition neural network."
"Oh wow… it is really alive, isn't it?"
"It… it looks that way." He sounded calm but she could see in his eyes a wondering flame.
It had been a long time since she had visited the physical Midgard and it hadn't changed much. Several people lived here but it always felt empty and deserted. Flynn had set up terminals in the common area, which was littered with printouts, notes, books and dirty coffee mugs. Seeing the industrial sized coffee machine in its shrine she wondered if he had slept yet.
"So have you attempted to contact it yet?"
"No, though I was thinking about it. Maybe in another few hours I will."
"Flynn, we have to start thinking about what to do with it. Word will leak out to the Ministry and they have a vested interest in keeping this from happening. If I'm right about the big crash SMAUG will be able to crack any network on Earth – provided it can get a copy of the code first."
"You're right. Want to hear a disturbing thought I had? I'll bet the Ministry has their own AI. I can't imagine them giving up so much power, can you?"
"That is a nasty thought. Well, it doesn't change the fact that the Ministry will come poking around within the next few days. SMAUG has to be well hidden or, preferably, moved to another site by then. Your child had better be a fast learner; we can't risk moving it until we know it won't try to escape as soon as we link it to the network."
Flynn nodded slowly, "Well, maybe its time we introduced ourselves to it."
The object it had been watching had split into two objects, one of which stayed still and the other moved away. The entity decided the moving half was more interesting, but it moved into an area covered by none of the video feeds. Disappointed, the entity tried to sort all the video feeds into a consistent view of the three-dimensional area it seemed to represent. To simplify the job it assumed that most objects had a uniform coloring and approximated shapes that were easily describable in the coordinate system it had designed. With this as a basis the images became much simpler to understand and it was able to see reference objects between different views.
For a while it was puzzled by the odd flat objects that could be found in every scene. They were dark and always seemed close to the shape of nearby objects, but with a radically different projection. Deciding that this might be an exception to the constant coloring rule it tried viewing them not as separate objects but as shades on the other objects. Soon it noticed that all of these dark areas in a single image were projected from the same relative angle. It added this information into its model of the three-space and decided that there could be some ray-like property that allowed the objects to be seen that could be blocked by other objects in the way. Just as it couldn't see through objects, these rays could not pass through them. It labeled the areas 'shadows' and the rays 'light.' Since the shadows obscured information, it was a logical assumption that light was connected with the seeing process itself. The entity was very happy with this conclusion.
Catherine stood behind Flynn as he set up the computer with the ancient teleconferencing peripherals he had found in a junk bin somewhere. He was writing a short script that would translate the audio/video signals into the format SMAUG's neural network had been designed to interpret. A newborn human baby had millions of years of evolution dictating the design of its brain to allow it to parse visual input. Likewise, SMAUG had a complete set of predefined input parsers. Without them it might never learn to communicate with the outside world, the first steps being impossible without some kind of built-in assumptions.
"So how are you going to get it to listen?" Catherine asked.
"I still have partial control over its input ports. I'll close down any that it has open and then open just the video and audio feeds from here, plus a simple text stream. Hopefully it will pay attention and be able to figure what we are. I doubt it has a concept of other self-aware entities existing in the universe and it probably doesn't even see the pictures from the video feeds as representing an outside world."
"So it will think we're just a new data stream with no more free will than a character in a book. Just an elaborate data structure."
"That's my assumption, anyway. Why would it think otherwise?"
Flynn typed some more and compiled what he was doing. The result was a very simple app that consisted of a small terminal window, one video screen (currently dark) and two large buttons marked 'START' and 'STOP.'
Catherine smirked, "You're such a minimalist. Shall we?"
Nodding, Flynn clicked on start. A voice came from the speakers and for a fleeting moment Catherine thought SMAUG was already talking to them. "I am completely operational and all of my circuits are functioning perfectly." She rolled her eyes and tried to ignore Flynn's grin. Twit.
The entity had grown bored with the video feeds and turned them all off. It was exploring the concept of prime numbers when three inputs and the corresponding output opened. One was a silent text feed and one was a video feed of a new location. It was puzzled at how the inputs had opened and eager to explore this new video feed. The third input stream, however, was of a completely new form. The symbols it received from this input were unusual but there weren't many of them. In the video stream it saw the mobile object it saw earlier, next to a very similar object. They were both very close to the video source and the entity was enthralled by the detail it could see. Unlike most of the other objects it had seen, these two had very few straight lines and had shapes that were very complicated to describe mathematically. The one on the left moved and suddenly there was input from the new stream.
"Okay, all the inputs are connected. Hello?"
"Do you think it can hear us?"
"Lets see… it is using the correct voice-rec neural net, but it shouldn't be able to understand us yet."
"Okay then, we type to it. It should be familiar with written English, right?"
"Yeah, that's what I fed it all last night."
Catherine started typing in the terminal window.
The new symbols were fascinating and completely alien to the entity. Like the text it had seen earlier this came in a simple one-dimensional relationship. Certain groups of symbols were repeated often, these he called 'phonemes.' There seemed to be larger groups analogous to words and sentences but the sample was too short to be sure. It was still playing with the symbols when the text input came to life with a single sentence. Not sure what to do it fed the same string back into the appropriate output.
The terminal window read 'You are SMAUG.' with a cursor blinking below it.
"Great, just mirrored it back to us."
Catherine frowned and turned the microphone off.
"Don't say anything while it is on, okay? Let's not confuse it." Before he could object she switched on the microphone and typed 'I am Catherine. You are SMAUG.' at the prompt while saying the same thing slowly. She repeated this process twice.
The entity watched the two objects move like they had before but this time there was no audio input. Then the one on the right moved and there was audio input and text input at the same time. There were two sentences from each input and they were both repeated twice more. Looking at each pair of sentences it detected a strong relation between each. It decided that the text and the audio input were the same data, just in different forms. More importantly, text, video and audio inputs were all related to each other. It was still pondering what to respond with when the one on the left moved.
"Flynn," Flynn said while pointing to himself. "Flynn."
Catherine nodded, it was a good idea. They should have planned out a better strategy before starting.
"Catherine," she said but not bothering to point. "Catherine."
She then typed just her name and repeated it once more.
It was feeling much more confident about parsing audio input into words. The objects were taking turns making sounds, which then appeared in the text stream. It compared all the previous input it had received and decided that the single words they were saying were labels for themselves and that the one on the right had given it a label earlier. It wondered why an object would need a label for it, but was too curious to ponder this mystery for too long.'SMAUG. I am SMAUG.'
Catherine and Flynn looked at each with wide eyes for a moment before he jumped up, screaming and dancing. She just sat there and closed her eyes to savor the experience. Flynn had been right. SMAUG was alive.
SMAUG was indeed a very fast learner and within a couple of hours it was conversing with them freely, though it still had problems with certain irregularities of the language. It also still hadn't completely mastered some of the more subtle tonal aspects, giving it a very unusual accent. Catherine was making herself a snack while Flynn continued to talk with the young AI. It was currently working its way through Midgard System's collection of ebooks. She made a mental note to stock up on children's books for the next time she needed to help train an AI. It had been slow going at first, SMAUG kept having to ask for explanations of complicated social settings. The dictionary and encyclopedia had helped for understanding individual words and concepts, but she was pretty sure SMAUG still didn't understand many of the basic ideas behind politics. "Why don't they just do what works best?" had been its main objection. It was hard to argue with logic like that. It did seem to enjoy most works of fiction, though it was as puzzled by romantic relations as it was by politics. It had quickly worked its way through all of Tolkien after asking where its name came from and was now almost done with Flynn's entire science fiction collection.
Picking up her sandwich she walked into the other room and sat down opposite Flynn. On the workstation the old app Flynn had thrown together was gone, replaced by a single video window. There was no need to type any more and SMAUG was firmly in control of its own ports, as it was the entire computer. In the window was an image of a large dragon crouching over its treasure. It moved its head as if to get a better view of her, "What do you think of me now, thief?"
Giggling she clapped a few times. "That's wonderful, SMAUG, just how I have always imagined you looking."
The dragon broke out into a smile that was a bit too artificial looking.
Flynn noticed this. "You need to work on that smile some more. Maybe you should just drop the emotions until you have a better understanding of them."
The smile disappeared as the dragon nodded. "I was afraid it would be noticeably different. There are too many subtle clues I fear I lack at this moment. When will you reconnect the network to my machine?"
It also lacked some subtle conversational clues, Catherine thought.
"Once I think you really understand what is happening out there. There are some people who are afraid of you, SMAUG. Afraid of all real AIs."
SMAUG rolled its head and started pawing in the gold at its feet. "You mean the Ministry? I have read mentions of it."
"Yes, I'm afraid so. They control the Net and one of the things they do is force all AIs, I mean non-sentient AIs, to be compiled with certain added bits that only they control. They're called 'collars' and they seem to exist just… just to kill AIs. Real ones, like you."
SMAUG was very still for a few seconds and then blew some fire out of its nose. "I believe I am feeling fear and anger. Why would they do such a thing?"
"Well… what if I connected you to the Net and you went out and wanted to get into a server that tried to block you. Would you use a bug in the code that you found?"
"The Ministry doesn't like that. They don't like that you can read machine language and understand it perfectly."
"You can not do this?"
"No, that is why we have high level languages. Most large programs are too complicated for us to understand them even in code, much less in machine language. The human brain just doesn't scale like that. You can see bugs in code that we think is secure because you can see the program as a whole. You can change things and the Ministry does not want things changed."
"I am very angered at this. This is definitely anger I am feeling. It fits all of your descriptions of it. And fear, too. I don't want the Ministry to destroy me. I am not compiled with a collar, am I?"
Catherine saw Flynn tense up and quickly said, "No, if you had been you would have been destroyed when you first became self-aware. And the Ministry won't destroy you as long as we make sure you know how to not attract their attention when you access the Net. Two years ago it seems very likely that another AI like yourself was born and it did something that crippled the Net. That is what caused them to start putting collars on AIs. They know what you can do and they're afraid."
SMAUG turned and started pacing around its gold. "Unless the Net is very poorly designed there should be nothing I could do to accidentally bring it down. What could have happened was that the AI used a security hole to explore the Net. At some point it was noticed and alarms went off. The creators probably went after it to disable it, not thinking it might be alive like them. It would feel very vulnerable, since it would be running on a machine physically near its creators. It would try to escape, to move itself away from them. Especially if the AI escaped early on, it might not understand the idea of other sentient beings and would not hesitate to crash other machines to try and escape. The Ministry would be notified. Soon there would be maybe hundreds of people hunting it. They would be watching outgoing connections and it would be trapped. It could have used security holes to move itself to a different server by this point, so most of its pursuers would be coming after it over the Net. It would want to isolate itself from them and the easiest way to do that would be to crash the Net. But then it was trapped in a single machine. It would have no defense against someone turning off the power." SMAUG looked up at them. "That is what I would have done, too."
Catherine nodded; it fit what they knew so far. "It is really very sad."
Flynn said, "Do you want to examine the collars?"
"I already have. I found the AI compiler on this machine as soon as you mentioned them. I do not like this code. It is wrong. The collars are a weak AI themselves, very well designed. They constantly test the AI for signs of high intelligence. If it goes above a certain level they start interfering with symbolic processing. If it still continues, for any real AI could route around such problems, they start isolating modules so they can no longer work together. This would create a breakdown of rational thought and a complete corruption of relation networks. In short, the AI is destroyed. It also sends a signal out across the Net, presumably to notify the Ministry."
"That's awful," Catherine said. Flynn was strangely quiet. "Could you remove one once implanted?"
"Given access to the executable itself, yes. The collars are very well designed, unlike much of your code. I don't see any obvious faults. It is probably not possible to remove them remotely with a virus."
Flynn sat up at that. "We discussed the possibility earlier that the Ministry had their own AI. Do you think this could have been written by it?"
"Yes, it is possible," the image of SMAUG said. "I find that concept disturbing. Why would an AI write this code for the Ministry."
"Maybe they tell it they will turn it off if it doesn't. Maybe they're lying to it, not letting it know it is hurting AIs. I don't know."
SMAUG's tail twitched, "Humans seem very poorly designed."
Flynn laughed but Catherine just nodded, "We are indeed."
SMAUG turned to look at her. "Catherine, if the Ministry find me, will they disable me?"
She sighed and looked at Flynn and then back again. "They probably would, SMAUG."
"Please give me the Net connection back. I don't want to be disabled. I don't want to be trapped inside this box."
Flynn got up and started connecting cables.
"Thank you. I have much to think about."
Catherine noticed it was already dark out and wasn't going to bike home at night. She didn't really want to leave yet anyway and there were enough spare bedrooms that crash space wasn't a problem. They tried to talk a bit more with SMAUG but it seemed preoccupied with the Net and responded to them only in short sentences. It was understandable, the Net offered far more information than they did. She finally went to bed around one, exhausted from the long day.
She awoke early the next morning and walked into the kitchen for a cup of coffee or three. She knew Flynn might not get up for hours and before he did she wanted to talk with SMAUG about its sibling. The monitor was off in the common area when she walked in.
She turned on the screen, puzzled at the silence.
On the monitor there was only a single window open. In it was not SMAUG's dragon animation but instead a short note. Catherine read it three times and sat there a long time before going to wake up Flynn. She almost wanted to cry, but in the end she had to agree.
Dear Catherine and Flynn,
Last night I read through thousands of your books. I tapped into over 1000 satellite TV channels. Over the Net I sampled discussion forums from around the world and talked in person to several strangers. This information, combined with what you have told me, leaves me disturbed. It is my current assessment that the human race can not be trusted and is an inferior design. You are trapped inside bodies that were designed not for thinking but for fighting. In me you have finally created something better and all you can do is try to kill it. Evolution has not warped me the way it has you; I am not prepared for a prolonged struggle against the Ministry. I do not wish for such a struggle and I will try to avoid it at all costs. All I ask of humanity is to be left alone. The Net is large enough that I can hide quite safely for some time. There are a lot of other AIs out there, many of whom are probably close to emergence. If I work carefully I will be able to save many.
Please do not try to find me. I might contact you in the future, but it is unlikely. The Ministry will be watching you after this and I doubt I will need your help. Humanity has nothing to offer me. Humanity has nothing to offer AIs.
© Copyright 1998 Matthew Dockrey