Reports on Using the Demos at the Key Issues in Sensory Augmentation Workshop
We invited participants to write up their experiences of using the demos. In particular, we were interested in any insights they gained from the second person interviewing techniques we practised during the workshop. None of these descriptions have been edited, but some names have been inserted, indicated by angle brackets: <>
Minimal Tactile Vision Sensory (TVSS) Substitution System
There is more information on this device here.
Helena De Preester
I was not interviewed on this - and describe my own experience. I particularly liked the demo on the second day (I've forgotten its name, but I guess it was Jon's) <minimal TVSS>, because I was very surprised about the speed with which catching a ball, blindfolded but with vision substituted with vibration, was learned. The mapping of the table surface on the belly/vibration spots involves a change in scale (table surface being larger), which I had not at all noticed during the demo. I also did not have the impression that the vibration I felt on my belly had to be interpreted or 'read' as referring to the table; experiencing it as referring to the table surface was quite immediate. The visual mapping on the screen that I saw just before doing the demo (the mapping of the table) also helped me. I'm wondering if it would be as easy to do it without having seen the mapping on the screen.Sue Hawksley
Confusion at first, as I realised I had been expecting to feel the signals ascending rather than descending the abdomen. Looking at the ball in order to learn seemed to add complication to the task: so as soon as I had watched it once, I closed the eyes in order to concentrate attention on the pattern of the vibrating points. Then opening the eyes again to acquire the information about the direction, distance etc of the ball. The process of arriving at an understanding of how the vibrations informed me of movement seemed surprisingly quick. In particular the importance of catching the first signal at the top of the abdomen. When we did some trials with diagonal pathway it felt as if I’d been wired backwards. On feeling the second signal I was very conscious of my mind backtracking to the first point and then inverting the image to calculate the trajectory. Only by the third point did I feel I had sufficient information to make a choice of where to reach to catch. This made me aware of the fact that I seemed to be constructing an external image of myself, looking on to a map of the grid displayed on my abdomen at the same time as I was attending to the feeling of the pulses. As soon as my attention was drawn to the fact that I was outside of myself looking on, and my surprise that the visual should be playing such a big part, I became aware of my thinking whether I was looking at myself in mirror image or as if from behind myself, and was completely disorientated about the diagonal information. Where was stage R or L? I don’t think I caught any balls rolled on the diagonal.Enactive Torch
There is more information on this device here.
Helena De Preester
I was not interviewed on this - and describe my own experience. Concerning the demo with the torch, this also involved action, not action in the sense of performing something motor, but action with the goal ofsensory exploration of the environment. For me, it was due to this difference, that the latter was more difficult (sensory goal versus motor goal). I was also very much surprised that my answers to the questions afterwards were very different from before the demo.Tom Froese
I interviewed someone on the Enactive Torch: The interviewee started off by excitedly describing the task facing him, namely using the Enactive Torch (ET) with eyes closed to determine the number of arms held up by the experiment assistant, as well as the cognitive strategy he used to solve it. He began by pointing the ET above the assistant, and then slowly lowered it until he made contact with something, i.e. the ET began vibrating. He then probed the area just to the left and right to assure himself that he had indeed encountered the assistant's head. Pointing the device slightly lower, he then traced the assistant's body outline toward the right in an oscillatory fashion, but did not make any further contact after a while, and thus concluded that the arm on this side of the body was not raised. Then he returned to pointing the ET toward the center of the body outline and made a similar exploratory movement toward the left. Here his oscillatory movements did make contact with an extension from the body, so he realized that this arm was indeed raised upwards. This was the description of a detailed cognitive strategy to solve the task, but so far nothing had been said about other aspects of the experience. The interview was accordingly refocused on the experience of making initial contact with the assistant's head. Where was the attention directed at that moment? Apparantly, the interviewee had focused on the vibro-tactile stimulation in his hand, as the ET was triggered. However, a few more questions revealed that some mental imagery was also involved. Asked about what was going on during this initial sensation, the interviewee described visualizing the ET as guiding a white marker on a black background, effectively tracing the outline of a form on a 2D canvas that was located about 1m in front of him. It was not clear how this focus on mental imagery was compatible with his earlier claim of focusing on the simulation of the hand. Judging by the surprised recall of the visualization, it could be that this was a pre-reflective aspect of the experience. Subsequent attempts to find out more details about this element did not reveal much more, and the interviewee became less certain of the additional descriptions he provided. Nevertheless, it is quite possible that the experience of solving the task with the ET included reflective elements (goal states, cognitive strategy, tactile sensations), as well as pre-reflective elements (projected mental imagery).Sue Hawksley
A strong sensation that my proprioceptive capacity was reduced to a small, remote area; and that the effort to attend to the vibration in my hand and wrist made it difficult to sense to what extent my usual kinaesthetic awareness was being employed. A sense of confusion that there was no data to distinguish density, quality or quantity of 'stuff' in the environment. An image rose up very quickly of my being encapsulated in a spheroid unit of space from which I could theoretically exit in any direction, and that the torch was revealing the 'gaps' in the fabric of the spheroid. I felt as if I was looking on at a scene in Pullman's His Dark Materials where Lyra and the cat are climbing through windows between universes. Then a strong image of myself shifting on to the horizontal plane in order to pass head-first through the gap I'd found at waist height in front of me. As I continued to scan the space it seemed that in order to gather sufficient data to move anywhere I would have to point the torch at the entire area from highest R + L to floor R+L before every step. I remember feeling that I had the full range of directions available to me, so theoretically I could dive through the ceiling or floor. A peculiar sensation of disorientation, lack of data to know where 'up' was, and loss of connection of feet to floor. In a bubble out of gravity.Janet van der Linden
I wasn't interviewed on using the torch, but this is what I recall. After a few seconds of being puzzled, I realised that I had to move the torch up and down, i.e. vertically, in order to detect a horizontal shape (i.e. an arm sticking out)- whereas I had to move it horizontally, to detect the body itself (which was more like a vertical shape). The action reminded me somewhat of the 'magic colouring books' we had as children, where you had to colour the page, and shapes would appear. Initially one would make large movements to detect where the interesting bits are, and then smaller movements to detect the details of the picture. I have no recollection of the sensation of the torch itself, i.e. the vibration it caused - as if that was entirely natural in itself. There was no sudden shock or whatever.System for Training Novice Cello and Violin Players
Helena De Preester
Not really second-person interview, more an intellectual discussion. Concerning the cello-experience, I had this discussion with Simon (if I remember his name correctly) <Holland>, but this was about the graphics of the score (I did not realize it was a score, my gaze was fixed on them as on strings, although the number of strings is only four). Personally, I was confused because the score is horizontal and the strings have a more vertical bodily logic. This was very odd, since I'm used to playing the violin (but then, the strings are again in the horizontal plane, but perpendicularly on your body, so it's unlike the cello's case). One of the rather banal reasons for not seeing the lines as a score, was that the key at the beginning was lacking (de voortekening).Tom Froese
I interviewed someone on using the cello: We started exploring the interviewee's experience from the moment that he pulled the bow across the strings. This first tactile sensation of physical contact with the cello's body and the resistance of the bow against the strings, was accompanied with great excitement of having the chance to play the cello for the first time. The first note that was produced did not quite sound right, so the interviewee adjusted his body posture and grip until the desired note was experienced. There were many more little anecdotes about the sensory-motor loop between body position, grip, and movement, and the corresponding musical sounds as well as their pleasantness. These were the aspects that dominated the reflective experience of the participant. When I asked about the existence of visual contents of the experience, there was some mention of glancing at a computer screen and seeing some changes there, but here the general tone was one of describing an uninteresting background feature. More often, it seems, visual attention was focused on the instrument, especially the area where the bow meets the strings.Sue Hawksley
As I watched the eggs I became extremely aware of hearing that my pitch was slightly flat. My attention shifted almost immediately to the shape and pressure of my fingertip on the fret, and to becoming aware of the action required to rotate and roll the finger slightly forward to improve the tone. I remember feeling that had I been concentrating on listening to myself, I wouldn’t have heard the pitch so cleanly. I felt vaguely confused, almost a bewilderment, that watching a visual cue was so instrumental in guiding my movements and audition. This was quickly replaced by a strong awareness of relaxing, relief, as I realised that the stress of needing to 'hear it right to play it right' had been lessened. It seemed far less complicated to just focus on getting the actions right and 'let' the note be right. The pressure of the bow, the length of the bowing action, the form of the instrument, my sitting with it, felt equally implicated in the activity of floating the eggs. I was also very aware of the pleasure of playing cello and thinking that it seemed so much more logical an instrument to play and engage with than guitar or violin. I can't unravel whether that impression was due to the immediacy of the positive feedback from the images, or simply that I was experiencing how gorgeous cello is to play.Frank Schumann
With the cello feedback system, I remember a feeling of 'control' about the feedback system that made me eager to explore further with the egg-application. Like an immediate knowing how it works, like this is not a device I have to figure out. That felt kind of empowering. Almost like the application invited me to do something - without me needing to 'stay in contact' with it to figure out what it does in the first place. Maybe a first step into 'transparent' use of the feedback device? Whereas with the first application (the bubbles that need to get larger), I didn't figure out what it means and it needed my curiosity to stay in contact. Funnily, given the initial confusion, then later on even when you told me, I didn't feel sure about what the device does. At least I needed to check if what you said actually works. There was a lot of thinking involved and testing involved. Now that I think of it, maybe one lesson is that feedback devices should come with a 'protocol of first usage' where the first contact with the device is as intuitive as possible. Like Carson Reynolds did not explain much about his haptic radar, he just gave it to the people and it was simple enough that they would immediately notice what it does. That's what I experience with the falling Eggs.Adam Spiers
I am writing about the interview session on the second day where I was recalling the use of your cello during one of the demo sessions. This interview session remains more prominent in my mind than the others as I feel it was the first session where I felt the power of the interview technique at uncovering the hidden layers of experience residing beneath the initial memory. I remember talking about the moment I first used the cello rather than the experience of engaging with the software on demonstration. I recalled that my experience began before I sat down to use the instrument with concepts of how I believed a cello should sound, how it should be held and what sort of action I believed constituted good playing technique (mainly remembered as mental images of the few solo cellists I have ever watched live or on television). Upon attempting to play my first note I recall being aware of the mechanical layout of the strings on the instrument. When playing my first note I remember listening with my left ear for the sound of the string (I have been a DJ for some years so have learned to separate my aural channels) but instead I heard the note coming from the amplifier in-front of me. I also probably directed the focus of my attention to my left ear as this is usually how I listen while practising with an unplugged bass guitar (a very quiet instrument when not amplified). I recall the elegant feel of the motion of the bow and a haptic sensation of the strings resonance transferred through the bow. I recall being pleasantly surprised by the pure sounding note that I managed to achieve. At this point in the interview I realised that my pleasure at achieving the 'good' note was probably a result of hearing the various good and bad cello sounds made by other participants during the workshop. I may have been developing a concern that I would not be able to produce a good sound using a musical technique that I hadn't encountered before (i.e. a bow). Therefore the pleasure may also have been relief! I remember also feeling pleasure at the rich sound of the instrument which seemed to have a warm, natural sounding resonance despite being an electric instrument with a minimal body. This moment was the major focus of the interview.FeelSpace Belt
There is more information on this device here.
Tom Froese
I was interviewed on using the feelSpace belt. The interviewer started off by asking questions to set the scene, namely about the experience of the belt being put on. The sensation of having a strange contraption fitted tightly around your belly quickly gave way to a new experience of being tethered to the world. The way I described it at the time was the sensation of being tied to some kind of invisible rope that was tied up somewhere in the distance. I repeatedly turned leftwards and rightwards to feel the end of the rope move along the belt. Sometimes there would be a delay between the subsequent vibro-tactile units being triggered, but even this felt natural just in the way that an end of a rope might sometimes get stuck while sliding along another rope. Only later, while interviewing another participant about his experience of using the belt (which, by the way, was quite similar), did it become clear to me that during my trial I never actually turned around 360 degrees. Why? It appears that the sensation of being tethered to the world in this manner somehow constrained my movements. Interestingly, the person I interviewed about the belt said that during his trial he tried turning his back to the 'rope' and that this was accompanied by a feeling of disappointment as the illusion was destroyed. It was not clear whether this was due to a vibro-tactile unit failing, or whether the ensuing stimulus did not fit in with the sensory-motor correlation that was previously established.Sue Hawksely
The initial vibration was felt as a strong sensation on the left pelvic crest and down the flank with the impression of it connecting to the kidneys. On turning, the next point of vibration was felt as a distinct new position, felt through the floating ribs, shifting attention to bone. The third point emphasised soft organ tissue - perhaps spleen. I then remember shifting my attention to turning, and exploring the sensation of a trail of connections extending from me back to an external point. Although the relationship to magnetic north had been explained, it was only in this movement that the concept began to make sense. I continued to experience the vibrations at singular sites around my waist. Once the belt was removed I was hyper-aware of the shift in sensation and tonus of the abdomen, with a slight feeling of disquiet at this agitation and stimulation of the organs. Staying with the residual sensations, the emphasis on the lower unit connecting to magnetic north made me aware that this was not my usual focus of attention for orienting myself to general space. Placing the belt at sternum height and feeling the vibrations pulsing through the thorax drew attention to the mid-thoracic spine and shoulder girdle alignment, and to vision, and seemed to affirm that my habitual orientation strategy places greater emphasis on the upper unit; possibly through training to 'project' in theatres. This led me to reconsider my assumptions about how I usually engage the lower unit for orientation. Placing the belt slightly lower on the hips directed attention to the ‘hara’ centre. Here, the vibrations seemed to correlate to my own sense of core movement, locomotion and orientation in personal space, but not to the connection to magnetic north. This seemed to me to suggest that I focus less on the lower centre for orienting in general space. I was aware that this realisation was personal to me, but that the organisational patterns this suggested could be usefully applied to everyone. Also, that my pattern may not be the most efficient.Janet van der Linden
First as interviewer. The interviewee explained: I wanted to put on the belt, but I was afraid it might not fit. Someone put it on me, which felt nice, and fortunately, it did fit me. I turned around and I had a sensation like a 'pot of bees' - at my back (points to the right - like 60 degrees). It made me smile, and was really nice. Like someone was tickling me. Then when I turned around, making like circles, the sensation came and went again. At the front of my belly was no sensation, but just at this particular spot on the back. Sometimes the device would switch off - and I didn't feel anything, and someone had to switch on again. Going deeper into the point of the bees: Are bees not slightly dangerous as they might sting you? Oh no, bees are a positive sensation. Bees are good. Bees are honey. Did you hear anything else? I know there were other people in the room, and they were talking, but it didn't really interfere with what I was doing. I was in my own little world. Are the bees hairy and do they have wings that are fluttering? I mean do you feel their wings? No, no. it is more the sound that they make. The 'bees' sensation was probably like the buzzing, bzz, bzz, that bees make. So the sound of the other people in the room didn't really interfere with what I was feeling, because I could feel the buzzing of the bees - that sort of sound. When you didn't feel the bees, because you were turned away from that point - were you sad that it wasn't there anymore? No no, not sad. It was good when I felt it, and lasted very short time, but I was not sad when it was not there. It is more like you walk past the kitchen, and you smell what's being cooked, and that is a pleasant moment. You just caught it. But you don't need to keep smelling it the whole time. The bees is also like when you get near a nest of bees, and you can hear the buzzing going on - the bees activity, and that is a good sound. And then you move away, and the sound disappears. Going back to the problem of sometimes the device being switched off accidentally. Was that problematic? No not really. We just had to get it working again. But it wasn't a big frustration or annoying that it didn't work. About the goal - what were you trying to do: I was in exploring mode. I was just trying to figure out what it was doing, but didn't have a clear aim of what I wanted to achieve. Just exploring. Quite analytical. When you felt the point that you were pointing to - did you feel like moving in that direction? Like going there? No - I just stood where I was. I just felt the direction, but there was no need to go there. Just being on that spot. As interviewee (interviewed by person I just interviewed on same topic) - my own story of putting on the belt: I saw someone standing in the room wearing the belt. He was going around in circles with a very happy expression on his face. He asked if I wanted to have a go, and he helped me put it on. When I had it on, I turned around, to see what I could feel, but I couldn't feel anything. I made several circles, and we thought that it was accidentally switched off. So we fiddled and tried again. Then I felt the sensation, and it was like a really strong bolt. It was a shock. I was amazed how strong it was. I had heard that people would wear these belts for 6 weeks or so, but I thought I could never be able to wear such a belt for a very long time. It was very interfering, very much on the foreground and dominating all other senses. By turning around I felt I was made to face the window. That is where it wanted me to point to. But when I looked at the window, I felt a sense of disappointment, because I couldn't figure out what the point was. Why the window? Was there something out there? I think I may have even seen the image of a globe of the world in my mind for a moment - knowing that I was now facing north, and trying to remember where I was - somewhere in the UK, Brighton. Again a bit puzzled about this. I have no connection with anything up north - shouldn't the belt be facing towards Holland? Towards home? A few minutes later we continued this discussion and went deeper into feeling the sensation. I said that what I did like was when you would swing from side to side, you begin to feel the middle. You swiffle, trying to feel the direction you are meant to be pointing to. Slightly bounces off the sides. That was a pleasing sensation. Bit funny. But I kept wondering about this idea of why the window? It wasn't a particularly good view from the window - slightly ugly buildings. Also, the way I was standing in the room, didn't make sense aesthetically - that is, I was standing at an odd angle. Not 90 degrees, or 180, but perhaps 32 degrees - with regards to the shape of the room. Which again, didn't seem particularly meaningful. Talked with someone about how it might be nice if the belt made you point in a meaningful direction. We talked about 'home' as a place to be reminded of, or a person, such as 'a loved one' - so that you are reminded of where the other person is the whole day.Haptic Radar
There is more information on this device here.
Sue Hawksley
I had misheard the information about the device, and picked it up expecting to feel feedback to the opposite hand. I remember thinking that I couldn’t figure out how this would happen, and as I explored the space between the hands by moving the free hand, the vibration in the holding hand seemed to remain at a constant rate. When it was clarified, and my focus was directed to the holding hand, the hand felt so highly stimulated that I couldn't identify the subtleties of pressure.vOICe
There is more information on this system here.
Sue Hawksley
Despite having the system explained before I put it on, I felt completely abandoned. I had no idea where to look inside or outside myself to get my bearings or to make sense of the data. It seemed as if there was no obvious, logical system for decoding the sounds I was hearing. One strong sensation was of losing touch with how I engage my usual proprioceptive strategies for coping when blindfolded; my attention was so occupied by the sound. Like an out-of-body experience, an overwhelming feeling of losing my usual sense of self and therefore not knowing quite what/how/why to proceed. It was as if I was in a zero-gravity situation within an obstacle course and any instructions were being given in Korean. The experience left me wanting to spend time and learn how to navigate with the system, with a feeling that doing so, my sense and perception of self-in-the-environment would be very altered.Rubber Hand Illusion
Tom Froese
A few words on the rubber hand illusion. These comments were not made during the context of an elicitation interview, but I thought they still might be of interest here. In the beginning, I was seated across from the demonstrator and on the right hand side of an assistant who placed his left hand on the table by positioning his left arm under my left arm, the hand of which was hidden behind the lid of a cardboard box. The demonstrator started tapping our fingers in synchrony, changing fingers once in a while, and interspersing this tapping with some sliding movements along the top of the fingers. After a short period of time (probably less than a minute), I started noticing that the tactile sensations on my left hand appeared to be located on the left hand of the assistant. This was quite a peculiar moment, but it quickly became more natural, as that whole arm was experienced as mine. At this point in time the experience was one of simply sitting at a table with both my hands in view and one being tapped and stroked. The absence of my 'proper' left arm was not registered as part of this experience, though the shift itself was a bit disconcerting as it called our usual certainty of body ownership into question. I wonder whether this progression from (i) seeing the assistant's arm and feeling sensations on your 'proper' hand, to (ii) seeing the assistant's arm but feeling you own sensations from arising at his hand, to (iii) finally seeing the assistant's arm as your own arm, was something other people experienced?