Some Ingredients for Einstein's Dreams

Just scattered notes from the plane (notes scattered from the plane?), for an Einsteins Dreams chat with Lina Thursday :)

Artistic Research  R&D
Alan Lightman novel 
Time consciousness (Sha + David Morris Memory)

TML / Lenka Novakova / Laura Emelianoff, Bain St Michel residency, 
public responsive space exhibition, 2-3 months April - May 2011
Michael M, Xin Wei
teams from FG and Ozone
TML /Hexagram: Temporal textures (Harry, Pat, Ozone)
Installation+Movement (summer 2011?)
Satinder Gill, Anglia-Ruskin University, movement + sound lab
(Tim, Richard composers; Jane choreographer)
Prosody (Gill Cambridge )

Production Venues
TML, Montreal
RedCat, LA
Eyebeam, NYC

Symposium Venues
Banff Math Institute  (R Mazzeo), New Media Institute, Banff Performing Arts
UC Berkeley, Dance and Tech May 2011, Wymore
classes: Wymore, de Kosnik
AI Society, July / August 2012: Cambridge University, session on time consciousness (not time)

Exhibit Venues
Dana Centre, Sep 2012
Berkeley / SF 
Tate Modern, June 2013 
Eyebeam, NYC

Foundation support
Image Text Sound ?
CCA / NSERC New Media April 2011?
study of time consciousness?

Prior Research & Art
Frankensteins Ghosts
Montanaro portfolio
Temporal textures (FQRSC)
Memory-Identity-Place (with David Morris, philosophy)

Collaborators, Advisors
Linnaea Tillett Lighting Design
Prof. Peter Galison, Harvard, History of Science

Einsteins Dreams: primordial readings on time consciousness ?

Dear Einsteins Dreamers,

As background for the Einsteins Dreams project, first off, let's replace "perception of time" by time consciousness. There is an enormous literature on time of course, but let me suggest some passages to orient a starting place that's suitably elastic and avoids the rampant cognitivism:

Heidegger, Metaphysical Foundations of Logic
In the second part, there is a particularly clear exposition of the triadic structure of time consciousness, which is also discussed by Husserl.

Husserl, The Pheomenology of Time Consciousness.
I am not sufficiently familiar with this text to suggest any one section, so feel that I should read the entire book. But Id want company for that :) The one paragraph from the appendix to Maturana and Varela, Autopoeisis and Cognition:
"Time as a dimension" I'd like to understand this incredibly lucid passage clearly with you, because I think it clears an enormous ground for understanding time(s) as an effect. That's one of my pet hypotheses :)

This all could be discussed quite efficiently by hanging out with some of our friendly neighborhood philosophers -- David Morris and the Memory Place Identity group. Perhaps ED could meet with them?

Hab keine angst, very rapidly we will need to generate concrete funding proposals, so very very rapidly we will need to create some sketches or at least some scenarios. Cheers,
Xin Wei

Einsteins Dreams: raw work plan

5 The integrated work plan for the research and/or production activities
5.1 studies -- specific techniques
person scale
site scale
5.2 rehearsals -- putting it together
blackbox theatre
mock up on-site
rehearsals on-site
5.3 checkpoints and R&D
initial blackbox scratch, bring creators together
bench studies
mock-up on site
large feature redesign
bench studies
sub-contracted work
rehearsal 1.1 in blackbox and rehearsal 1.2 on-site
fine feature redesign
rehearsal 2
•  production
WHERE will it be done? ?
Berkeley, Pacific Film Archive 2011 ?
Montreal Usine-C
Tate Modern 2011
Harvard Biochem building (Galison)
Under Brooklyn Bridge
San Francisco Presidio

fresh start, raw outline

This is a raw outline for a fresh start, based on Michael's notes for the CCA-NSERC  new media program last year.

    1     WHAT is it?

    1.1    Einstein's Dreams is an environment in which visitors encounter performers in responsive fields of video, light, and spatialized sound, in a set of tableaus     Each tableau is inspired by a vignette  from Alan Lightman's novel, Einstein's Dreams, set in Berne Switzerland, in 1904, the year that Albert Einstein received the Nobel prize.   Or rather, a set of parallel 1904's, each of which is a different kind of time: in one, time slows to a halt as you approach a particular place; in another there is no future;  in third, time sticks and slips; in a fourth age reverses and what is rotten becomes fresh as time passes.

    1.2    In one concept, a large theatrical space (24 x 20 x 8m) will contain multiple tableaux, each with room for 6-12 people in its own pool of light and sound.  Visitors and perhaps performers can move from tableaux to tableaux.   The performers' actions will evolve in concert with the dynamics of lighting, sound, and visitors' expectations in order to create different kinds of time, inspired by the novel's vignettes.   Sometimes a performer will walk from a location, dragging the pool of conditioning light and sound.   The pool mutates or merges into another pool with a different type of time.


    2     A literature review and/or historical contemporary understanding of the context for the research

2.1    One hundred years after the two epochal advents in physics of relativity theory and quantum mechanics, we are still reverberating with the consequences.   ED is not a biography or a didactic allegory, but a poetic exploration of our consciousness in time.

2.2    There is huge volume of elaborations and didactic works about the theory of relativity and quantum mechanics, including Einstein's own popular essays, and the canonical scientific / philosophical works by Hermann Weyl, Alfred N. Whitehead Principle of Relativity, Henri Bergson Duration and Simultaneity.   In the arts, there have been some distinguished works that treat the theories and the theorists in an externalist way, as icons or as social phenomena.   But what we propose is to work directly with the spectators' felt experiences of time.

2.3    What Alan Lightman evoked in his novel was a poetic variation around the felt experience of time, not the "actual" physics of time, but alternatives of time, those hypothesized modes of living in time that could have been imagined, or that never were.   By foregrounding how time works in these worlds, the vignettes foreground movement, which is the temporalization of the body.   These movements are embedded in everyday life, made marvelous by poetic conceits of time.  This fits perfectly with both Sha and Montanaro's own artistic research into the charging of movement and gesture, of finding or evoking marvelous configurations of movement in the everyday.

2.4    Scientifically ED will create an experimental apparatus for inducing perceptibly different senses of temporal passage, chance and order, mortality and anisotropy -- the arrow of time.  Our goal is to use the poetic techniques of theater and dance and responsive media to evoke these sharply different ways of temporal experience so that the visitors will feel this for themselves.   Our experimental goal will be to discover ways to not just depict different kinds of temporal processes, but to condition a physical setting to yield first-person experiences of these different kinds of being in time.

Re-interpreting C. Geertz's term "thick," the Topological Media Lab designs thick experimental events environments sensitive to the effect of the whole ensemble, rather than neutralizing all perceptual features but one measurable parameter such as the end-to-end latency of a force-feedback controller.

One scientific and phenomenological application of TML's holistic -- thick -- design approach to responsive environments is a collaboration with Dr. Tore Nielsen's Dream and Nightmare Laboratory (U. de Montréal) investigating the phenomenon of felt presence.   In that alternative to a minimalist psychological experiment, we built a whole room furnished like a small parlor, but whose furniture, lighting, and sound fields were subtly "animated" according to precisely fashioned scripts.   Another is a project with Prof. David Morris (Philosophy, Concordia) about the relation between memory, identity, and how a body comports in a physical place.

The Topological Media Lab was founded in 2001 on the premise that the disciplines with the most experience in the design of responsive environments, media, and "interaction" are in fact the performing arts.

As an artist, Montanaro has been concerned with shaping whole events, not as the sum of parts like special effects,  or technical action and reaction.  This meshes very well with the Sha's guiding concern with creating "thick" events that are "rich, but not complicated."   In fact the TML's methodological research has been to discover ways to understand how to condition a physical space using "thick" design techniques: "media choreography."  [EXPLAIN]  

2.6    Montonaro's aesthetic values  _____________ (symbolic power, joy, magic, ...),  and prefers not to fetishize the computational technology, which accords perfectly with Sha's aesthetic as well.   Following analogous design strategies in the engineering community like Mark Weiser's calm technology, or the European Union's ___-th Framework: "Disappearing Computer," we intend to create events in which the computational technology works as unobstrusively as well-designed lighting.

2.7    The core concerns of ED circle around the relationships set up in a performance, or an event in which at least momentarily the intents of creator (author, composer, choreographer) entangle with the intents of the performer and observer.    For Montanaro, performance at its heart is a relational triangle: performer - choreographer/director - observer.   For Sha this suggests fresh insights into questions raised in relativity and quantum mechanics about the how the observer is implicated in the construction of phenomena.  In relativity, the fundamental relation is between two frames of reference, one at rest with respect to the observer, and another in motion.    In quantum mechanics, the observed parameter is replaced by the notion of an expectation value which is the symmetrical inner product of the state of the observer with the Hermitian acting on the state of the observed.    We can never directly measure the state of the observer or observed.

[FIGURE 1. Performance is relational triangle: performer - choreographer/director - observer]

[FIGURE 2.  compare with relativity: observer - rest-frame - moving-frame;  or quantum mechanics:  <observer state, apparatus (hermitian operator), observed state>]


2.8    Remove if-then logic at macro (human) scale as well as CPU scale  =>

realtime concurrent programming vs "object-oriented programming" (OOP)

texture programming vs. OOP.


2.9    Performer  <-- act-on -->  environment;  but also self-act-on as well.  concurrency and distributed agency =>

need new movement structures <----> continuous state-based media choreography.  the experiment is to test the reach and adequacy of the Ozone as a compositional technique

2.10    Superposing, overlaying, imposing ritual/ formalized / artificially enhanced action on top of pedestrian, everyday actions  generates interesting tensions of intent ===> Einsteins Dreams: apply temporal operators  as a suite of estranging , magicking "devices"  using temporal textures (parallel , prior research at TML)

2.11    For performance, we can always "cheat" and depict. - i.e. allegory.  but instead of thinking of that as cheating, think of it as "Turk"-- ie theater:

2.12    The question is how can we bring the observer inside the event?  how can we induce altered, or alter ... temporal consciousness, according to Einsteins Dreams?

2.13    The role of the observer

In an interactive performing arts event a large part of the observer’s perception, understanding and appreciation is third person reliant, resting heavily on the performer’s interpretation of the creative artists dialogue as well as the performer’s physical and emotional relationship to the environment. Even though the observer and performer have access to the same audio and visual information the observer’s physical connection with the material is, experientially speaking, vicarious. Though this separation exists in traditional live performance the difference is amplified many times over with the introduction of the technologically enhanced or intelligent space. It is here, at the point of divergence between performance and non-performance based forms of interactive artistic expression, that my interest and current research lie. Performer or voyeur, the separation runs deep

2.14    Parallel with relativity and modern physics:

    The idea of the observer that both relativity and QM brought explicitly into the frame : see next sections

    3     The underlying technical/scientific issues or complexities, along with an explanation of how they will be resolved

    3.1    Temporal textures

    3.2    Think of filters rather than projections of image or sound-field.  We create realtime systems that sample the existing sound or lightfields (eg from arrays of microphones and cameras), and re-synthesize them and send them with meaningful spatial or temporal transforms back into the given tableaus.  Simple techniques, such as temporal delays that we can regionalize down to individual pixels or fixtures with resolution from milleseconds to hours, based on high resolution cameras with high frame rates (120 to 600 FPS) already can provide rich effects.  

But additionally we propose to leverage the TML's 5 years of experience adapting computational physics techniques to create temporal textures that have the qualities of physical materials -- smoke, water, syrup, ice, etc.   We are investigating how to leverage the person's tacit knowledge of physical materials to give a sense of tangibility that cuts across sense modalities.   This is an example of user interaction with "interfaces" so dense (like cloth or elastic controllers) as to require continuous models and a phenomenological rather than cognitive approach.

    3.3    Continuous State-based Media Choreography

    4     The names and titles of the members of the project team, their roles in the collaboration and how the project will be managed

    4.1    Michael Montanaro
    4.2    Sha Xin Wei

    5     The integrated work plan for the research and/or production activities

    5.1    WHERE will it be done?
    Montreal Usine-C
    Tate Modern 2011
    Harvard Biochem building (Galison)
    Under Brooklyn Bridge
    San Francisco Presidio

    6    The nature of the industrial participation, if any, and its contribution to the project (including a letter of support and an NSERC Form 183A for each industry partner)

    7    How participation in this project will enrich the professional experience of the individual artist, the scientist/engineer and other participants

    7.1    Montanaro has worked for years with dance and

    fresh ways to choreograph, other than usual action (dancer) reaction (system)

    7.2    Xin Wei: new ways to understand time and sense of time, ways to create temporal texture : field-based ways to

    7.3    Opportunity to working together experimentally,.   Have done it voluntarily on our own time for 5 years, now wish to do it funded.

    8    Plans to manage intellectual property and copyright issues related to the proposed project


    9    How the collaborators will evaluate their collaboration and research outcomes

    9.1    production

    9.2    publication in relevant journals: ACM Multimedia, Smart Graphics, Ubicomp, TDR,


    10    How the project results will be disseminated

    10.1    performance

    10.2    workshops with students in dance and media systems

    10.3    in publications

    11    The project’s impact on the broader artistic and scientific communities, and the social, cultural, economic or environmental benefits to Canada.

    11.1    lighting and movement

restart Einstein's Dreams conversation

Hi Michael, Lina,

OK, I've started Einsteins Dreams on a new blog service, where we're now posting other TML blogs too.

It's private, with just us three for now.  You can post via email:

I've put up a new first draft, using the CCA-NSERC outline, even though we're not making this year's round.

Scientific question :  how do temporal textures  -- in light and sound and most importantly of activities of other bodies around us -- induce different senses of temporality?

Artistic - conceptual question: moving from from Husserl's phenomenology of time consciousness, but taken out of its cartesian frame, made into a field effect using our realtime media apparatus)

Practical (engineering) support:
We may have an opportunity to do this with allies from London and/or Berkeley (EE CITRIS, CNMAT); TML's temporal textures research; the GRAND NCE I'm in with Jeremy Cooperstock et al....

Xin Wei

Lina (16 April 2009):

“In a world where time is a sense, like sight or like taste, a sequence of episodes may be quick or may be slow, dim or intense, salty or sweet, causal or without cause, orderly or random […]” (Lightman 1993: 89)

In the real world memory and the sensory register are affected by fully embodied experiences that take place within a very specific time and space. What if these fixed variables of time and space were to be disturbed? In a series of short poetic vignettes, Alan Lightman’s novel, Einstein’s Dreams, presents such worlds where time and space behave differently. In one, time is an infinite loop, in another, the higher you go the slower time is, in a third, there is one place where time comes to a halt, and in another everyone knows the world will come to an end at a particular date and moment.