Flong

Printed from www.flong.com/texts/essays/essay_creative_code/
Contents © 2020 Golan Levin and Collaborators

Golan Levin and Collaborators

Essays and Statements

Essay for Creative Code, edited by John Maeda

For his book "Creative Code", John Maeda sent interview questions to a number of computational artists and designers. Below are my own responses to his questions. The final article included in Maeda's book is a small excerpt from my responses.
Golan Levin, August 2003.

[The new medium is:] Dynamic, manipulable graphics with complex behavior.
— Bill Verplank, 1981

It has now been than twenty years since Bill Verplank wrote these words, and, with the closing of John Maeda's Aesthetics and Computation Group (and its replacement by his new Physical Language Workshop), it may finally be kosher to risk the suggestion that there is a new new medium. But before we do, it's worth taking stock of what we accomplished with the old one. And so I am tremendously honored to contribute to the present volume, both as a former member of Maeda's ACG, and also in my present capacity as an outside writer doing some of the stock-taking.

Maeda's first act as a newly appointed professor, in 1996, was to invite Paul Rand to lecture at the Media Laboratory. Maeda's second act was the creation of course MAS964 — Fundamentals of Computational Visual Form — which formed "the initial template for a studio art course in computer science." With an aesthetic reductiveness and a technical rigor almost unseen since the Weimar Bauhaus, Maeda's class addressed the ways in which computation could intersect with the most elementary concepts of abstract form, color, movement, and interaction. The students' weekly Java assignments were instantaneously posted to the nascent World Wide Web, and, as there was nothing else like it online at the time, the world quickly took notice. Tom White's ColorCube, Reed Kram's Kite and Circle, Matt Gorbet's Balance: the purity and insight of these simple experiments from MAS964 contrasted so starkly with the ham-fisted aesthetics of the prevailing CD-ROM multimedia that, outside MIT, designers and artists on at least three continents tuned in to the weekly updates, writing emails and arguing amongst each other as to which student had produced the best results. It was through one of these forwarded emails, in fact, that I first became aware of the ACG. Two years later, after learning how to write C++, I was in its walls.

Dynamic abstraction was an essential component of our artistic and educational practice in the ACG, and although only a few students made it their explicit specialty, it was implicitly acknowledged as a core component of everything we produced. And so, while my own work at the ACG focussed on transmediality in dynamic abstraction, and Casey Reas directed his attention to the processuality of abstract forms, dynamic and interactive abstractions were explored and created by everyone in the group. In my own private Hall of Fame, I continue to hold special places for Ben Fry's graceful Disgrand video processor, Jared Schiffman's exquisite Honey, and Tom White's astounding but little-known Implicit Venn Diagrams. Most of these works were hurriedly-created and poorly-documented experiments. It warms me to think that they might attain some additional longevity in this current book.

What makes something abstract versus concrete?

To provide a definition for abstraction is one of the most difficult questions I have ever been asked, and I continue to doubt that I am qualified to answer it, though naturally I will try. Delving into its use, for the moment, I observe that the term "abstract art" is nowadays applied to a very broad range of activities:

  • art whose subject matter is one or more of the intrinsic formal elements of visual design (such as line, shape, value, color, texture, etc.);
  • art whose subject matter is the structure it employs to pattern and organize the classic formal elements of visual design;
  • art whose subject matter is the materials from which it is created;
  • art whose subject matter is the process by which it was created;
  • art whose subject matter is the human psyche's iconic representations of archetypic concepts (such as "Man," "Woman," "House," or "Tree");
  • art which is non-objective and/or non-narrative (non-representational).

It is straightforward to imagine how each of these practices would have analogies in the realm of visual art made on the computer. To the traditional formal elements of visual design (which Jacques Bertin calls ocular variables), the computer adds the additional elements of temporality (through iteration), and contingency (through conditional testing). And so, it becomes possible for abstract computational art-systems to address, as their subject matter, dynamism (the way things change over time), interactivity (the character of the feedback loop established with a user), and processuality (the character of algorithmic processes). Furthermore, with the aid of various hardware extensions such as sound processors, haptic actuators, and network interfaces, the computer allows artists to concern themselves with transmediality (the way the senses are addressed in simultaneity), and connectivity (the character of what is expressible when the computer is used as a communications medium). For the identification of these themes I am indebted to Gerfried Stocker.

So the above is a brief enumeration of the topics about which abstract artworks are most often concerned. But what is abstraction itself? Abstraction, I believe, is the act of ignoring or suppressing certain details, in order to bring certain basic concepts or structures (gestalts) into relief. And the fundamental subject of abstract art, I therefore claim, is pattern.

The logic of abstraction, according to this definition, works something like this: by omitting whole categories of information from a communication, the abstract artist forces the mind of the observer to attend to what remains, and fill in the details on its own. When the observer's mind fills in these details, it commits an act of what Scott McCloud calls closure — the completion of some visual or conceptual structure. By completing this structure, the viewer's mind becomes freshly aware of this structure's presence and force in a way that would ordinarily be obscured by the details of everyday experience. And so, in this way, the abstract artist communicates a general truth (or perception) about the way in which the world (or some part of it) is structured.

While abstraction deals with the realm of the pure pattern or idea, divorced from the specifics of worldly instances, Concrete, by contrast, is very precisely a hard, strong construction material consisting of sand, gravel, pebbles, stone, or slag in a mortar or cement matrix....

In your mind, is the computer just a tool? How is it a tool? How is it not?

I like this question, "In your mind, is the computer just a tool?", because it actually embeds my answer: in fact, I believe that mind and computer are co-extensive. Of course, it's difficult to propose something like this without seeming overly addled by the trendy Gnostic philosophy of the Matrix, or caught in a swirly miasma of McLuhanisms about the "extensions of Man." But there is no question that software is more than just some externalized record of thought; for that, we already have writing. It seems to me that software is a living record of a thought one has had, or is having, about how the world ought to be: a brittle but determined little piece of mind which not only contains a model of one's point of view, but actively works to impose one's point of view onto the environment. If writing is a medium of thought, then software is yet additionally an agent of will. When it executes my will, my software and I form a single, coextensive unit of both thought and purpose. The computer, accordingly, is merely the software's mortal coil.

Our intellects grow together with our tools. Eventually, they become inseparable. Although it may be hard to see how this could happen with a clunky item like a desktop PC, consider a much older tool like language: after a quarter-million years of using language, our brains have actually evolved to accommodate it, and now have several substantial regions which are solely dedicated to processing it. Radical augmentations of human intellect like writing and computation, though much newer than spoken language, stand no less a chance of becoming integrated into who we are, if our race can survive long enough.

To answer the second and third questions, then: I believe computers represent a hybrid form of tool, in that they combine the abilities of conventional tools to act-on-the-world, with the special attribute of tools like language and writing to serve as enabling substrates for thought. This combination makes them optimally suited for a continually deepening integration with both our minds and bodies, to the point that it may someday sound as odd to ask your question, "is the computer a tool?" as it is to ask the same question about language today. Language is a tool, and yet also a part of who we are; computation will be the same.

Who is the audience for your work?

The answer to this depends on whether you are interested in knowing for whom I make my work, or identifying who consumes it (and understanding why they do). I make my work entirely for myself, in order to answer various questions I have knocking around inside my head. Fortunately, these questions seem to have interested enough other people besides myself that I've just about managed to cover my rent for the last couple of years. Naturally I'm very grateful that I do have an audience, not least because a few of the people who express interest in my work are often, themselves, thinking about questions related to mine, and sometimes with very different and intriguing solutions. It's reciprocal: I'm the audience for their work. But identifying "who my audience is" in the general sense is quite elusive to me, as it seems to span a broad range of people.

The audience for my work certainly varies considerably with the different contexts in which I present my projects. Like many people who create things, for example, I enjoy publishing what I do on the Web; my hope is that each of my Internet projects is somehow seen as a small gift to whomever might happen to stumble across it. As could be expected, I have only a faint idea who my surfers might be. Their traces through my sites tell me very little about them, except, perhaps, statistically. For the most part, my best guess is that many of the people who encounter my work on the Web are students or practitioners of new-media art, which is to say, somewhat like myself. But judging from the quantity of bizarre emails in my inbox, I'd also hesitate to generalize.

What is definitely quite clear is that a visual culture has arisen, let's say since 1995 or so, in which artists and designers offer interactive abstractions in the online realm, and — perhaps for the same core reason that anybody looks attends to art at all — people really do consume them. The artists' justifications for these offerings are as varied as the projects themselves: some are presented as modest formal experiments, others as technical demonstrations, and yet others as finished artworks or even calling-cards for their makers' design businesses. Whatever the reason, there seems to be a widespread interest in such abstract work, and a number of splendid designers and artists have been able to find food for themselves as a consequence of sharing their experiments in this peculiar barter economy of abstract visual forms. I'm thinking of artists like Lia, Danny Brown, James Tindall, Marius Watz, Manny Tan, and of course Martin, Yugo and Joshua. Whether by email or by word of mouth, these artists, and of course many others as well, are engaged in a discourse with each other, and share what they do in the hope that someone else finds it interesting. They make new works as critical responses to previous ones. So when I imagine someone else experiencing one of my online works, it's usually one or more of these colleagues who I imagine, and whose critique I attempt to anticipate. These artists are exceptionally educated and sensitive observers, and also merciless critics. They're the audience I'd like to have.

I don't present my work exclusively on the Web because, in the final accounting, I really enjoy sharing my work in person with other people. It was Maeda who first pointed this out to me explicitly, when sometime in 1999 he suggested that I ought to consider performance as a vehicle for my ideas about audiovisual abstraction. I'm not certain that I would have realized my interest in new-media performance had he not prompted me in this way. My first performance came at the invitation of Gerfried Stocker for the 2000 Ars Electronica festival, shortly after I graduated from the ACG, and I was immediately hooked by the intensity of the communication I could have with live observers. Since then I've performed interactive projects at dozens of venues and festivals. It's an irregular living, but it has become my chief artistic outlet and my main way of getting by.

The audiences at electronic arts festivals can be quite mixed, and comprise many sectors of individuals interested in new music, cinema, interactivity, audiovisuality, connectivity, processuality, commerce, and so forth. Compared with the much more numerous consumers of American popular culture, rave music, or the traditional Art World, however, the audiences of the festivals I where I present are unquestionably a tiny, electronic-arts "ghetto." Assuming some fraction of this ghetto is the portion of people who enjoy my work, I'm sorry to say that I'm not yet clear as to what distinguishes this fraction from the people who don't!

Where do you see roadblocks for new abstraction systems?

In the Master's thesis I created at the ACG, I identified five or so pitfalls which seemed to consistently plague prior implementations of audiovisual performance systems [see sidebar]. Three years later, I now believe that progress for abstraction systems is chiefly stymied by the same factors which hold back nearly any field of creative inquiry: researchers' premature satisfaction with conventional design solutions, willing acceptance of convenient but flawed tools, fear of failure, technical laziness, ignorance of history, unresearched assumptions, intellectual territoriality, commercial greed, etcetera, etcetera. These are the dark and tempting forces I've encountered in the world after graduate school, anyway. Don't get me started! I'd rather look on the bright side, and answer this question in terms of the opportunities that I believe exist for abstraction systems.

The last time I approached this topic in writing, it was with the tunnel-vision of my Master's project, and the opportunities I identified for abstraction systems comprised a rather narrow bunch of interaction-design patterns: gesture augmentation through signal processing techniques, gesture capture and playback, functionally overloaded gestural inputs, statistical control.... Probably four people in the world have actually managed to read this unnecessarily abstruse chapter, and three of them were required to. For the present article, I thought it would be some relief to pop up, instead, to a discussion of contextual opportunities for making and deploying abstraction systems: where can dynamic abstraction systems belong in culture? And what other trends provide new tools for their making?

In the course of the research work for my Master's thesis, I compiled lists of all of the interactive audiovisual performance systems I could find, and attempted to sort their design logics into broad but meaningful categories. In the end, I reckoned that three basic paradigms — "interactive widgets" (e.g. object simulations), "control panels", and "scores" — covered nearly all of the examples I could collect, with only one or two isolated exceptions. To these I added a "painterly metaphor", structured around the idea of an inexhaustible, infinitely variable, audiovisual substance. I thought I had mapped the territory of possibilities.

Wow, was I wrong! In the years since my last writing, the design of audiovisual abstraction systems has been taken up by a new generation of talented practitioners, and numerous new metaphors for audiovisual performance have proliferated. To take a few examples, Masaya Matsuura's Vib Ribbon represents the vanguard of a new metaphor of navigable audiovisual terrains; the folks at the Laboratory for Architecture and Urbanism (Lab/AU) in Belgium have made notable developments in navigable audiovisual spaces; and Tom Betts from Nullpointer has developed cellular automata-based performance systems to a completely new level of expressivity. But one particular concept, which I would like to elaborate on here, was first demonstrated to me in a tremendously compelling audiovisual performance by the Barcelona artist Joan Leandre (also known as Retroyou). Leandre, like many media artists working today, modifies the instruction codes and resource files of commercial computer games in order to produce new interactive artworks. In Leandre's case, he then employs these modified games as audiovisual performance instruments.

A typical performance of Retroyou's R/C project — based on a modification to a popular racing car game — begins with Leandre's selection of his performance vehicle. Grasping his standard two-handed game controller, what follows is a sonically and visually stunning mélange of shapes and vibrations. Visually, his world presents a rapid evolution of shattered textures and colored fragments, rendering superimposed real-time perspectives into a Cubist space which is at once two-dimensional and three-dimensional. Sonically, we hear the sounds of Leandre's revving racecar engine, heavily layered with itself and expressively manipulated into a highly articulate, swooping drone. It is lovely, surging, abstract. The piece lulls, waxes, wanes, and intensifies until, after a few minutes, there is some kind of rumbling catastrophe. The field rotates wildly, we see shards of yellow (fire?) and gray (smoke?) and suddenly the words "GAME OVER." Joan is asked by the software to enter his initials into the high-score list.

What makes Leandre's work an important contribution to the field is its explicit recognition of its own cultural context. Rather than forge the pretense of a hermetic, Modern visual space — predicated on the utopian illusion of having no referents other than a few idealist Platonic forms — Leandre's work aggressively admits to its participation in our contemporary visual culture, and, by abstracting its deep structure, even provokes us into re-considering it. If my own Audiovisual Environment Suite, like most previous interactive abstraction systems, implicitly upheld a purist design stance of Modernist (dynamic) abstraction, then Joan Leandre's R/C is legitimately and successfully Postmodern dynamic abstraction. It points to an opportunity which I feel we cannot ignore, and suggests some of the directions to which we might turn, when we have had our fill of circles and squares.

A second area of opportunities for dynamic abstraction lies off the screen, in new, physical means and materials used for interactive control and display. Of course, it would be superfluous in 2003 to make an argument about the necessity or advantages of high-bandwidth, multidimensional physical interfaces — I'll take for granted that dozens of excellent university courses in so-called "ubiquitous computing", "physical computing" and "tangible media" now exist around the world to rightly remind us that the mouse is just about the narrowest straw through which we could possibly suck all of human expression. Instead, I would like to bring attention to the new idea of granular robotics — a physical, particulate analogy to the audiovisual substance metaphor of my own Master's project.

Granular robotics portends the still as-yet-unrealized vision of "digital clay," whose basic concept, as always, is that the interactive audiovisual medium should be its own transparent user interface. The twist is that, in digital clay, the audiovisual medium has a physical (as opposed to virtual or screen-based) manifestation. Were digital clay to exist, what properties would it have? In addition to all of the properties of normal clay, as one might expect, it would also have the ability to store and recall previous configurations on demand, and also to assume new configurations interactively or automatically according to programmed logics and in response to environmental stimuli. Sound like fantasy? Professor Kris Pister, who directs the "Smart Dust" project at UC Berkeley, doesn't think so. By combining digital circuitry, laser-driven wireless communications, and MEMS (Micro Electro-Mechanical Systems) techniques, Pister is close to realizing his goal of creating autonomous sensing, actuation and communication in a cubic millimeter. By itself, a one-millimeter robot is probably something to sneeze at. Put a million of them together, however, and one has a programmable substance suitable for three generations of artists to create interactive, audiovisual, dynamic, sculptural forms.

Finally, I would like to mention a third area of opportunity for dynamic abstraction, which follows from the recognition that interactive instruments are cybernetic systems, whose models of human-machine coupling are infinitely improvable. When sophisticated instruments are performed expertly, the boundary between human and machine dissolves, and we perceive only a single expressive system. How can the instrument level itself to the user naturally and gracefully, continuously but unobtrusively suggesting what can be done next? Can the parts of the interaction language that users do know intrinsically suggest the parts of the language they don't? As users engage in a dialogue with a medium, how can both user and software grow and change together as a coupled system? Kevin McGee suggested these questions to me in 1998, but these questions seem no less vital to me after more than six years and four generations of computers: today's machines continue to lack substantial models of their users, sufficient depth to the knowledge representations they use, and adequate flexibility in the ways they acquire and manipulate these knowledge units. It will be quite some time before machine intelligence of this sort becomes a standard element in the toolchest of the abstract visual designer, but the payoff in deep experiences will be immense.

SIDEBAR: Five pitfalls of abstraction systems.
The following are adapted from Chapter 4 of my ACG Master's thesis, "Painterly Interfaces for Audiovisual Performance."

Randomness
The appeal of randomness, in theory, is that it could be used to introduce new “information” into interactions order to keep experiences fresh and ever-changing. Unfortunately, the problem with randomness is that it contains no information at all—and can therefore be misinterpreted as actual information or teleology by an unsuspecting human participant. Randomness finds an optimal employment below the threshold of perception, as noise in somatic textures, without which computational designs can often seem lifeless, overly-regular, and dull. Maeda has written about this.

The taste of mathematics
Interactive artworks whose design philosophies are predicated on the “intrinsic beauty of mathematics” are only rarely more interesting, personal, or provocative than the equations which generated them. At the same time, nearly all computational artwork inescapably involves, at some level, the implementation of mathematical relationships and equations; the medium itself is so deeply structured by these relationships that the entire field of computer art is often (and justly) regarded as cold or impersonal. The challenge in designing an interactive abstraction system is to overcome the mathematical materials which one must necessarily use, and surpass them in the service of some greater expression.

Cartesian mappings
The commonplace decision to map x or y input coordinates to parameters of an interactive experience is one which relies on an abstract and artificial convention, rather than on geometric intuitions that are more analogous to the natural world. Although Cartesian coordinates may be computationally and electromechanically convenient for input and output on raster-based systems, the reactions and processes of nature obey an organic logic which is often poorly described by relationships to a Cartesian grid. I owe this observation to Scott Snibbe.

Modal interactions
It is difficult to create an interface which can be entirely used by continuous interactions in a single state. For this reason, many designers have become fond of modal interactions, which present a large payoff in terms of the number of possible operations and states that a system can offer. Unfortunately, although modal interactions are learnable, they are not necessarily easily to intuit, and they therefore frequently necessitate considerable user instruction. If it is absolutely necessary for an interface to have different modes, it should be as easy to discover and navigate those modes as it is for a musician to switch from bowed to pizzicato violin, or for an artist to turn a pencil upside-down to use its eraser.

ROM-based solutions
Pre-rendered audio, images, and video clips may sound and look great, but they wear thin when used repetitively, and afford few handles for expressively-controlled variation or evolution. In situations which call for the organic qualities of sampled signals, consider the malleability of granular techniques, or accept the challenge (and burden) of parametric, multidimensional synthesis.