Author Archives: eduardo

Tomorrow’s Yesterday, Today


Over the course of the semester I’ve been iterating on the original idea for AgNES. Originally, it was meant to be an implementation based on Portal’s snarky and evil artificial intelligence, GLaDOS, following a roughly similar design and trying to mimic the functionality. The first demo was a first attempt at using the Mac’s built-in text-to-speech synthesizer to “sing” the Portal ending song, “Still Alive”. The second iteration took a more tangible approach, and with the help of Travis Rich from the Viral Systems group at the Media Lab, we were able to put together a physical manifestation for AgNES: a small robotic head capable of tracking a user’s movement when tagged by bright colours. The head – a small cardboard box holding an Arduino board and a webcam, mounted on a small servo – was controlled by an attached computer that processed the image from the camera, looked for the desired colour within a treshold in the frame, and commanded movement accordingly. The third iteration, just a couple weeks ago, revisited the software component of it and already started going in a different, more sci-fi-ish direction: AgNES became a sort of companion robot for long, solitary travel – essentially, deep-space exploration – which could provide a grounding helpful voice to an imaginary traveller. By giving the traveller stories, facts, and various other voice-mediated interactions, the companionship of AgNES can keep the traveller grounded and relatively sane over long journeys. But AgNES’s personality also became something the user could interface with: based on the five factor model for the description of personality, AgNES’s own personality is made up of five independent cores that can be individually turned on or off. How the cores are configured has an effect on the output the user gets from the various commands, and one can thus experiment with different configurations to get different results.

By playing around with AgNES’s personality, one is also playing with the conditions necessary for its optimal functioning. Which means that as results vary, some cracks in its design are revealed: in the confusion, AgNES begins to unintendedly give out clues as to the identities of its designers and its operators. The user can then follow this clues to learn more about this design and better understand the purpose of AgNES and its intentions. This is grounded in yet another science fiction underlying narrative: how future individuals will react to and interact with technologies from the future past.

Future Archaeology and Deep-Space Exploration

Even just today, we’ve already accrued a significant technological past that is hard to access and explore. Floppy disks are a good example: if you stumbled upon a box of old floppies from years ago and wanted to browse for meaningful things within them, getting to that data would be really complicated. If the disks are functional and you can find a drive to read them, there’s still the matter of whether the data is uncorrupted and whether you can still get the software to read it. Not impossible, of course, but as time goes on, increasingly complicated.

Future researchers, probably deprived of access to instruction manuals and other reference materials that help us situate past technologies, will contemplate our present technological world trying to make sense of it just as we look back on archaeological remains and try to make well-informed conjectures about what objects were used for or why they were designed one way over another. While we make an effort to design technologies that are intuitive to use, this intuitiveness is anchored at specific moments in space and time. Thousands of years from now, when behavioural patterns become very different, it is plausible to assume that many of the design conventions in use today will no longer have the same effect. Future archaeologists then face a complicated task of reconstruction.

That is the narrative framing where AgNES comes in. AgNES is designed from the point of view of being this deep-space exploration companion; but as a narrative device, it is also about thinking what would happen if thousands of years from now, future researchers came across this device built only hundreds of years from now. What sense would they make of it? How would they understand it, after stumbling upon it floating through space in a derelict ship, perhaps still powered but no longer in the company of any travellers? In the first of the many Star Trek movies, the crew of the Enterprise stumbles upon a massive entity threatening Earth called VGER, which upon closer inspection turns out to be the Voyager 6 probe, found by an alien species who augmented its design to enable it to fulfil its mission to “collect knowledge and bring it back to its creator”, creating a sentient entity on its way back to Earth in an unrecognisable form. These technologies we’re unleashing on the universe may at some point cycle back and be found again, and it’ll be a challenge to interpret them and make sense of their original context.

AgNES and Meta-AgNES

AgNES works on two levels. As a “present day” object, it is this pseudo-artificial intelligence that provides company and grounding during deep space travel, with a customisable personality the user can modify. AgNES’s commands are limited but they provide different forms of entertainment to keep a user distracted over what would be, presumably, very long sessions of just floating through space. The design of AgNES draws from multiple science fiction sources: the already mentioned Portal was the chief one throughout, but other sources such as Arthur Clarke’s 2001 and its own evil AI, HAL9000, also had a big influence. As did many of the themes we discussed in class related to artificial intelligence and robotics (including such things as Neuromancer by William Gibson, or Do Androids Dream of Electric Sheep? by Phillip K. Dick). As a companion providing information, there’s also certainly some influence from the Illustrated Primer technology found in Neal Stephenson’s The Diamond Age.

As a “future day” object, the design of AgNES is populated with a series of clues that only become evident to the user when they begin playing around with the personality configuration. Deactivating certain personality cores opens up areas that would otherwise be forbidden to an “unauthorised” user, where they may find information that can later be explored more in detail using additional commands. The “future day” user can then put together these pieces of the puzzle to come up with their own conjecture as to what AgNES is, where it came from, who it was with, and how it came to be where it is. From this point of view, AgNES plays more like a game where you’re trying to decipher what’s going on with this object by interacting with it, drawing inspiration primarily from the text and point-and-click adventure games especially popular in the early 1990s. The big caveat, however, is that there’s no real resolution to the game: there’s no “win state” as such, and there’s no real correct answer – you can only get as far as the conjecture you draw from the information you received as to who was involved and what happened. Just as future researchers, you can never be fully certain whether your conjecture was actually the case.

Pay No Attention To The Man Behind The Curtain

Training AgNES to track bright colours

Training AgNES to track bright colours

Technically speaking, AgNES is not incredibly complex. There are two pieces running simultaneously. One of the code for AgNES itself, written on Python and managing all the commands, the UI, and the personality cores. The cores themselves are five USB sticks and a hub that are together used as a switch – the code detects which cores are plugged in at any given time and makes changes accordingly. The first versions of AgNES used the Cmd Python module for a simple command-line interface, while the final iteration uses Tkinter to instead have a simple GUI that is less prone to error and displays information more clearly. AgNES’s commands are highly dynamic, and they often pull randomised content from various sources around the web based on the information desired: for instance, under normal operations, the TIL (“Today I Learnt”) command will pull a random article from Wikipedia and then read the summary out to the user. If the Curiosity core is turned off (meaning a reduced openness to experience factor, signaled by a more limited use of language) the command does the same, but pulling from the Simple English version of Wikipedia. If, instead, the Empathy core is turned off, the system pulls a generated text from the PoMo (Post Modernism) generator and reads that out – without empathy, AgNES loses any regard for the user actually understanding the information. And so on. Not all core combinations are meaningful, but those that are pull and parse content from the web using the BeautifulSoup web scraping library.

User testing AgNES's tracking of bright colours.

User testing AgNES.

The other pieces is AgNES’s head, described above. The setup remains the same, with the box containing an Arduino board and a webcam, all of it mounted on a small servo. The image handling is done with Processing, going over a frame of the image, finding the desired colour (Post-It pink, so it can be as unambiguous as possible) and making sure it stays within a center treshold – if it falls outside of that, it signals the servo to move left or right until it readjusts.

AgNES wearing its tin foil space helmet.

AgNES wearing its tin foil space helmet.

For its final presentation, both pieces were running of a Mac Mini concealed within a stand, “decorated” to appear as if it was a 1960s sci-fi B-movie prop (meaning, lots of aluminum foil). Using an app called Air Display, the mini was using an iPad as an external display with the AgNES GUI running, making it touch-enabled very easily. (I really wanted to try to get everything running off a Raspberry Pi but it proved to be too much for this iteration. The code for AgNES itself runs OK, but the computer vision and the text-to-speech stuff would’ve been more complicated to pull off, though not impossible – just needed more time!).

The final prototype setup in all its tin foil glory.

The final prototype setup in all its tin foil glory.

Building AgNES has been great, and especially interesting to think through its implications and the underlying concepts and issues at stake. All of the code for the project is available on GitHub if you want to try it out (though replicating the specific setup might be a bit complicated). Special thanks to everyone who contributed feedback, ideas, and testing, and any comments to improve it are more than welcome!

A Voice To Keep You Sane While Exploring The Depths Of Space

You might remember AgNES, the cute little robot head with one eye that would turn to follow bright colours, modelled as a sort of younger (or older?) sibling to Portal’s GLaDOS. Well, AgNES is evolving and growing – this time, AgNES even has a brain. Not a particularly smart brain, but it’s a start.

AgNES's brain is really a standard USB hub with five individual USB flash drives attached to it.

AgNES’s brain is really a standard USB hub with five individual USB flash drives attached to it.

There are two broad lines of thought informing the design of AgNES. One is the concept of future archaelogy, or how past and present technologies will appear to researchers hundreds or thousands of years from now – how protocols, interfaces, design conventions, and so on, will be accessible to people who will want to study our present society. In many cases, these systems will quite likely have to be reverse engineered to even make them functional, and then the work of trying to understand the social and cultural role they played may be pretty close to a guessing game. The other big line of thought is human deep-space exploration, or how we will adapt to sending people on really long deep space voyages, sometimes on their own and sometimes with other people, trying to make sure they don’t go crazy in the process. AgNES is a little bit of both: it’s a companion robot designed to accompany deep space explorers, but it’s also one that’s probably been derelict for a while and found by some other crew hundreds of years later, without major explanation or indication as to what happened. That also makes AgNES the only mechanism through which one can reconstruct the story, and a little bit of a mystery game to try to understand what happened under its watch.

So there’s a lot going on here, and many things to unpack – most of them driven by this underlying narrative scenario, which then in turn motivates the design and informs many of its decisions. As a companion robot, the voice of AgNES is intended to keep company and provide the crew with some persistent grounding to the real world, by providing them with information or simply entertainment. AgNES can spew out facts and teach about things, but it can also tell stories or read poems. And most of this content is constantly changing, some of it even being randomly generated every time. In a way, it can be interpreted as a science fiction reinterpretation of the tale of One Thousand and One Nights, where Scheherazade earned herself one more day to live every night by telling a new story. Similarly, AgNES is able to keep explorers engaged by providing one more piece of content at a time.

But where AgNES’s design gets really interesting is in her personality. The design of AgNES’s brain is modeled on the “Five Factor Model” (FFM) of personality, according to which personality can be described as operating across five different domains: openness, conscientiousness, extraversion, agreeableness, and neuroticism, with different personally being various strong or weak on any of these domains. AgNES’s personality is similarly modelled on the Five Factor Model, albeit with a few of the trait names modified slightly. Her personality is composed of five independent “cores”, each representing one trait, and each being activated or deactivated independently. What personality cores are active at any given time affects AgNES’s behaviour roughly following the FFM description: if the openness – renamed as “curiosity” – core is turned off, for example, AgNES’s language becomes simpler, as do the stories she will tell. If the conscientiousness – renamed “discipline” – core is offline, she might be willing to comply with some commands that would otherwise be blocked for the user. In this way, you can experience several different versions of AgNES by experimenting with personality core configurations, where each of them might yield different information about herself, her purpose, her design, her creators, or her history.

AgNES's personality cores based on the Five Factor Model.

AgNES’s personality cores based on the Five Factor Model.

That’s where the interactive storytelling component comes in. Based on the present configuration, AgNES will give you some information. Under a different configuration, you might be able to explore that information further, or you might get different, even contradictory, information. Since there’s no documentation or knowledge about the context available, there’s really no way to tell – so as the hypothetical future researcher your playing, you can only make your best conjecture as to what’s going on.

In terms of implementation, AgNES’s brain operates really as just a switch. The brain itself is a standard USB hub, and the personality cores are standard USB flash drives with renamed volume labels to tell them apart. The updated code for AgNES – as always, available on GitHub if anyone’s interested – checks for which cores are plugged in after every command is executed and updates accordingly. Some commands will simply be nonresponsive unless the right configuration is set, while other will exhibit different behaviour and information. The way many of the commands are set up is so that they will always give the user new information in randomised fashion: for instance, the til command (or “today I learnt”) will randomly fetch and read out the summary to a random Wikipedia article, assuming all cores are in place (if the Curiosity core is off then the command does the same, except pulling from the Simple English Wikipedia). Other commands pull information from other sources around the web, especially randomised text generators: haikus, Post Modern randomly generated articles, FML tweets, and so on. The code pulls a source, parses the webpage, extracts the desired bit of text and then passes it over to the Mac’s built-in text-to-speech synthesizer to read out loud.

AgNES reads when cores are plugged out, and modifies behaviour accordingly.

AgNES reads when cores are plugged out, and modifies behaviour accordingly.

The result is often funny, and often awkward. It’s when commands break or fail to work as expected that crucial information about AgNES is revealed – when it forgets who it’s taking too and calls you by one of her developer’s names, or when it inadvertently reveals how to get access to some crucial piece of information. Working on this breadcumb trail is what’s probably going to be the highest priority item for the next and final iteration, so that the AgNES simulation is actually “playable.” Additionally, I also need to improve how rules are manages in determining what operation should be based on the active cores, and I need to clean up the interface further still – as well as integrating this iteration with the previous one, to actually have a moving, animated object to correlated with the synthesized voice you get from the computer.

I’m already getting really good feedback to consider for the final iteration – so any comments or suggestions are more than welcome! Also, I’ll try to get a video up soon to show how it actually operates, as it’s a little difficult to reproduce independently right now.

Assembling AgNES, Or How I Read Science Fiction

The new revision of AgNES.

The new revision of AgNES.

Definitely the hardest part about working on Agnes so far was trying to come up with some backronym for the name. But I think I’ve finally found something: A(u)gmented Narrative Experience Simulator – AgNES.

As the sole representative of the MIT Comparative Media Studies program in the SciFi2SciFab class, my approach at building prototypes and exploring science fiction themes is likely quite different than that of people at the Media Lab. For starters, I don’t have a background on engineering or computer science, and my “making things” experience is limited. Which has made the class quite an interesting challenge, especially in trying to adapt what the experience of science fiction design means from my point of view.

That’s perhaps why my approach at building something like AgNES takes more of a narrative approach, or is perhaps better framed around larger themes involving science fiction and the exploration of possible futures. Even though I began thinking about AgNES as a version of GladOS from portal, in the process of working out the code and figuring out the design I ended up thinking a lot more about the possible narrative implications of what I was doing. In the game Portal, GladOS has been left behind by herself in an abandoned facility (after she killed most of the people there with a neurotoxin gas), and all of the information we receive from her own history and background comes either from direct interaction, or from subtle clues the designers of the game actually laid around hidden areas in the game’s levels. I started thinking about it from the point of view of far-future archaeology, where someone might find a facility and a system such as this one thousands of years into the future and then start interacting with it, trying to figure out what was going on at the time, and what its designers were thinking.

The rat man's den in Portal.

The rat man’s den in Portal.

If you think about it, most of the technology we use nowadays is fairly well documented – most of the stuff that has been widely used over the last hundred years or so can be traced roughly to where it came from and put into its historical context. If we go back a few hundred years, things begin to get murkier, and if we go back a few thousand, we begin to rely on archaeology to figure things out. So even though we’re able to understand our current technologies, thousands of years from now some people looking back at this time might not have such an easy time. For the most part, even trying to go back a few years to read some data stored on floppy disks can be quite the challenge nowadays. If you account for the evolution of storage media, operating systems and programming languages, interface design principles, and so on, our present technology might be entirely inaccessible to someone from the future. And while a clear chain of influence and evolution can be mapped, at some point, given enough variation, that might cease to be the case.

So that all ended up flowing into the design for AgNES. First of all, how would the design stand on its own to someone using this system, without having access to any information like context, purpose, background, and so on? How could one make inferences as to any of these things strictly from what was provided through the interface? And conversely, how could one exploit this information vacuum for narrative purposes? How can I construct a story about AgNES and convey it to the user strictly through interface and interaction elements?

The abandones facilities of Aperture Science, with GladOS at its core.

The abandones facilities of Aperture Science, with GladOS at its core.

In other words, the point of playing around with AgNES is figuring out a way in which it provides information to the user bit by bit, which the user can then put together to reconstruct the origin, purpose and story behind this system. It then becomes something of a mystery game, but just as in archaelogy, it is not a game of precise inputs and outputs. The user/player can only draw conjectures based on the available information, but the system itself cannot speak its “truth” and fully resolve what’s going on behind the scenes. While looking at the code behind the system might be useful and helpful, that doesn’t necessarily mean all the elements are contained in it to solve the mystery.

AgNES has become sort of a game, and games are essentially complex information systems where the purpose is to strategically conceal information from the player in a way that he or she is driven to uncover that information through the mechanics of the game. The difference is that the objective of this game is only to understand its parameters, yet the feedback mechanism is such that you can never fully know whether you have done so (a bit perverse, perhaps). You can only get closer and closer, testing things out to see what happens: while the original design had five entirely different personality cores that were interchangeable, the new design is closer to GladOS in being built around five cores which provide aspects to one whole personality. The cores can be deactivated, thus affecting the way in which AgNES operates, and prompting her to reveal different pieces of information depending on the active combination of cores. The current design is now drawing from many things we’ve been looking at in class: while Portal remains the main source, there’s a little bit of The Diamond Age’s Primer and ractors in thinking about the purpose AgNES could serve (while it also now has a new confession mode borrowed from THX1138). The personality side of AgNES also bears some influence from the Neuromancer/Wintermute conflict between artificial intelligences.

The new personality core design for AgNES.

The new personality core design for AgNES.

It’s a somewhat different approach at the intersection between science fiction and speculative/critical design, but hopefully an interesting one. The code for AgNES remains available on GitHhub, and hopefully in the near future it’ll be somewhere where it can be tested to see how people react to it. A new demo is also posted below, this new iteration built in collaboration with Travis Rich at the Media Lab.

Meet Agnes

[Warning: This post contains spoilers related to the Portal and Portal 2 games.]

Digital games have always received considerable influence from the science fiction world. Even really early games, like perhaps Space Invaders, in elementary ways were already drawing inspiration from sci-fi themes, and the trend has remained over time. One of the most interesting examples I can think of is the Portal games by Valve Software, which develop a very unconventional form of narrative: lacking any cutscenes or narrative interludes to provide the player with explanations about what’s going on, the depth of the world in which you abruptly find yourself in is provided bit by bit by GLadOs, an initially friendly artificial intelligence which leads the protagonist through a series of physical test chambers in an otherwise deserted scientific facility.

GLadOs controls the Aperture Laboratories facility and all of its contents, and as you progress through the first game she begins to act increasingly oddly, until you eventually find out she’s responsible for killing everyone in the facility with a neurotoxin gas. GLadOs appears to be driven solely by the urge to continue scientific testing of the Portal gun, and indeed, as soon as the player character completes the final testing sequence, she attempts to kill you (something you manage to escape to ultimately confront, and destroy, the GLadOs core).

GLadOS's main unit

GLadOS’s main unit

GLadOs, an experiment from Aperture Laboratories herself, was fitted with a sort of “fail-safe” mechanism in the form of personality cores. The various sides of her personality were split into multiple physical cores which could be attached or detached from the main unit, including cores for logical processing, morality, rage, and in Portal 2, an actual stupidity core – Wheatley – whose design was intended to stop GLadOs from taking over control of everything.

Some of the GLadOs personality cores

Some of the GLadOs personality cores

My SciFi2SciFab project is an attempt to reconstruct a version of GLadOs (without the murderous desires, of course), which I’m calling Agnes. GLadOs falls into a long and rich tradition of crazy robots which includes some hallmarks as HAL9000 from the 2001: A Space Odyssey┬ánovel and film. But it is also a very interesting reflection on natural and artificial intelligence: in the Portal 2 game, it is revealed that GLadOs was actually the result of transferring the consciousness of a living person on to a machine, which then went on to become a little bit insane because of the transformation and the sudden realisation she no longer had any human limits.

So I’m interested to see if I can put together a functional enough replica that will allow for some interaction and, especially, that will be able to pass for a playful robot despite not having full artificial intelligence at its disposal. Agnes is then made up of two related components: a software side which simulates the user interaction and provides vocal feedback based on user input, and how this feedback is provided changes based on the active “personality core” Agnes is running with. The intention is to make multiple cores available which can be physically switched, just like with GLadOs, but with each of them generating changes in tone and address rather than in criminal tendencies. Agnes cores will move interactions across a “politeness spectrum” where some cores will be more agreeable and nicer than others.

The other side of Agnes is the hardware component, a version of the hanging mechanical arm seen in the picture above providing a material interface to the pseudo AI. The arm will be fitted with a sensors allowing it to detect presences standing in front of it, and Agnes will then lock on to them and follow them around as they move within her operational range. I’ve wanted so far to preserve the original design of having Agnes hanging from above, which I’m also thinking would turn into a really cool application if you had a more evolved version coupled to rails along the ceiling of a building, to have your own version of Agnes following you around and providing helpful information and assistance.

The original arm design for Agnes

The original arm design for Agnes

The more I look for inspiration and ideas for the project, the more I’m drawn to realise how small design decisions going into Agnes have intense narrative charge. Not only in terms of associations one can make, but in terms of conveying the notion of an imaginary designer making imaginary decisions, and the object itself, Agnes, being the conveyor of those decisions and that background world invisible to the user – very much like Portal, the game itself, very successfully gives player access to the thoughts and lives of people in Aperture Laboratories. I think this contributes a lot of value to the experience of the user/player, who is then driven not only to interact with Agnes, but hopefully will be curious enough to try and “hack it” to understand where it came from and why it is how it is. So the robot taken from a game becomes a little bit of a game itself – is there a mystery to Agnes that needs to be deciphered?

Right now I’m working through these narrative associations and thinking about how this object design conveys identity – thinking how Agnes could easily go from GLadOs to Luxo with very different implications. And I’m in the process of implementing the pseudo-AI that will be driving Agnes, including the basis for the core switching which I need to turn into physical switching as a next step. Once that’s somewhat running, I’ll work on putting together the hanging arm setup and connecting the two. The (admittedly super elementary) code for Agnes is up on GitHub, and a little demo of what it does right now is posted below. Any feedback, thoughts or comments are very much welcome, as I’m figuring this all out just as I go along.