Shadows of an Icebound Dream

Summary

Shadows of an Icebound Dream is a video performance created as a visual part of the concert Frozen Flowers, performed at Cornerteateret in Bergen, December 2023.

Shadows of an Icebound Dream relates to the five parts of Frozen Flowers: wind, love, horror, laughter and frozen flowers. The video visualization can at the same time be considered an independent, visual work and is therefore given its own title. 

There is considerable development behind the visualisations, the various components, testing of the technique and the performance itself. This is particularly about exploring the relationships between video recordings and digital 3D models. The videos included in the work are created through a process where ordinary, two-dimensional recordings are used to create three-dimensional models, which form the basis for generating video loops. The 3D models are imperfect, and the visual expression thus acquires the character of "glitch". 

The videos are of different lengths, designed to be seamlessly played repeatedly, creating a loop aesthetic. This makes it possible to combine the videos, as part of a live performance, together with an improvised musical expression. A number of videos are mixed together, and the visual expression is thus unique to the concert, even though most of the material is planned and produced in advance.

The majority of the visual expression is based on site-specific video recordings, partly taken with a spherical (360) camera, partly ordinary video recordings. In the work with Shadows of an Icebound Dream, the aim is to represent place- and time-specific qualities, conveyed through the video images, and to give the audience an extended experience, partly in extension, partly in contrast to the music.

The recordings are special in the sense that they focus on capturing all visible surfaces of selected objects in an environment. This is because the video recordings are not used directly, but form the basis for creating 3D models using a technique called Neural Radiance Fields (NeRF). Neural Radiance Fields (NeRF) are artificial neural networks trained to model volumetric representations of 3D scenes. This technique makes it possible to store complex information about light and spatial conditions in a scene, and uses this to create realistic 3D models of the objects filmed. 

The level of detail in the 3D models will vary, depending on how much visual information is contained in the raw recording. Based on the generated 3D model, different camera paths are set, which also enables viewing angles and movements that would not have been possible to film in a traditional way. The technique makes it possible to create visual effects, where one alternates between the model's outside and inside, with associated narrative possibilities.

VJ-ing with OBS

VJ-ing is then understood as the combination of different video sources in real time, while video mapping is about how the different video sources are projected onto different surfaces. 

During the performance of Shadows of an Icebound Dream, a large number of video files are handled, which are mixed live, but where the overall video image is projected onto one surface. Open Broadcaster Software is used for this. OBS is a powerful and flexible software for video mixing, streaming and recording, and it can be adapted to handle VJ-ing. 

The interface, the main screen, in OBS. Here it is suggested how the scenes (bottom left) can be built up from a number of sources, a structure that can be changed live. The transition from one scene to another is controlled manually, and can thus be adapted to the progress of the music and respond directly to changes.

In the production of Shadows of an Icebound Dream, OBS software was used to create a series of scenes and add different sources to each scene. The sources were videos, images, animations, live camera feeds and screen recordings. During the performance, these scenes alternate in real time. OBS also has built-in effects and transitions. A selection of these effects were used in the transitions between scenes. In this way, it becomes easier to control and synchronize the visual changes, so that this fits with the dynamics of the music. Given that the musicians improvise to a large extent, this places corresponding demands on rapid adaptations and control of the visuals. The visualizations can be planned at an overall level, but the concrete details are adapted to what is happening in the soundscape and on stage.

Neural Radiance Fields (NeRFs)

As an example of the application of NeRFs, a model made based on a copy of the sculpture Laocoon and his sons, a Greek sculpture in late Hellenistic style from around 50 BC. The statue is preserved in a marble copy which is in the Vatican Museums. There are also several other copies. In this case, a plaster copy of Laocoon's head, on display at the Ashmolean Museum in Oxford, has been used. The expression is very dramatic, and was planned as a possible element for part 3 of Frozen Flowers - "Terror" (see below).

The video of Laocoon's head expresses pain and suffering, both through the face as it is reproduced in the sculpture, but here also metaphorically – in that the video takes us into a red chaos. 

Because this is a 3D model, you can freely place any virtual camera path through the model. This can be used to create spectacular graphic effects that also makes the glitches in the models become a visual feature.

The virtual camera path, set in the model, takes the viewer into the 3D model of Laokoon's head.

The illustrations above, and the video examples, show how the 3D model reproduces the sculpture with a high degree of spatial precision, texture, light and shadow effects. However, this does not apply at the outer edge of the model, i.e. in areas that have not been filmed with the same high precision as when the original recording was made. The model thus becomes progressively less detailed, coinciding with missing data in the periphery. At the same time, I feel that this creates an effect that serves the overall visual expression well, while at the same time it gives flexibility to create an expression that can be adapted to different scenes and areas of use.

In the specific performance during the concert, this particular video never appears in detail. This is due to the real performative nature of the performance, and the fact that in meeting the music I chose to keep the iconic Laocoon in the background. Here, however, we see how this video becomes like a flaming sea, which gives depth to the representation of the character Zosimos as old, in part 3. - "Terror". 

Loop aesthetics

Both animations and video loops are short, most only a few seconds. This is partly a personal choice, and a look back at my own work with video loops more than twenty years ago (Hoem, 2004), at a time when online video required that one consciously take file size into account. At the time, these limitations created a dawning awareness of the loop as a form of visual expression, also with clear parallels to sampling and looping of sounds, as this has been used within various musical genres. The loop is thus a meeting place between music and video.

Loop aesthetics, understood as repetitive sequences, utilizes repetition to create visual and narrative effects. The technique links modern video expression to earlier avant-garde and modernist art (Hatton, 2017). By using loops, you can enhance certain moments, and highlight visual patterns and rhythms. The video loop is closely related to repetitive GIF animations, for example as this is expressed in cinemagraphs. The loop also breaks with linear narrative structures, by focusing on details and brief moments, rather than a sequence of events that follow each other in time.

Loops have some specific advantages when VJ-ing. Looped videos can both create a continuous visual experience, without a fixed beginning or end, which one can pull forward and leave three back without breaking the visual connection.

 

Scenes

Below are some still images from individual scenes. However, these do not do the visual expression justice, as the video provides a passage in time and a visual dynamic that the still images cannot represent.

In addition to the various video tracks, which are prepared in advance and combined live, combined with two video sources that film the musicians on stage. One source is a large section that includes the projection behind the musicians. The second source is used for various sections that are closer to the musicians. Together, this creates several visual levels, where recordings and live video are mixed together. 

I use this visual interplay to break down the visual divide between the images on the screen and the musicians in the foreground. The projector is located upstairs, at the back of the room, and the projection therefore does not hit the musicians.  

All the scenes are based on video recordings and animations created with this performance in mind.

Intro 

Ulriken in fog, filmed as a timelapse where an hour is shortened to a couple of minutes. This is one of the few video clips that is not a seamless loop, and it is also one of the few examples that is not generated from a NeRF model.

1. Wind

The first image shows rotating figures, a form of zoetrope, where a series of figures are mounted along a circular plate. The figures represent a short action or movement. When the plate is rotated, an illusion of movement is created. This movement is shown here together with the flames from a bonfire.

The next sequence from this part shows three different levels: the face of an elderly man, who steps forward between flowers in a raging sea. A hint towards the last scene.

2. Love

Here, the animation of a younger woman is combined with a video loop representing parts of the Reactor Hall, at KTH in Stockholm. This is a room with special features, which are used here so that it is mainly visible through the woman's face.

The next example shows a video loop made from a copy of the statue Pia de' Tolomei and Nello della Pietra. The statue refers to Gaetano Donizetti's tragic opera 'Pia de' Tolomei', a tale originally mentioned by Dante's Divine Comedy. The sculpture depicts Pia dei Tolomei and her jealous husband, who believes the false accusations against her of having committed adultery.

This image also shows an example of how live images of the musicians are combined with the video loops designed in advance. In this way, the planned sequences are linked directly to what is happening there and then, and the audience gets several visual levels in their encounter with the musicians.

3. Terror

Here an example of how a loop is picked up again and recontextualized. The scene with the zoetrope and the people in front of the flames are combined here with a model of a crowned skull. The model is made based on a burial site at Christchurch in Oxford.

The combination of a crown (symbolizing power) with a skull (symbolizing death) represents the idea that even the most powerful do not escape death – Memento mori.

The skull builds a visual bridge when the sculpture of Pia dei Tolomei and her husband is brought up again, here together with a video loop of wilting flowers. There is perhaps a bit too much emphasis here, as withered flowers are also a clear symbol of the transience of life. However, this double visual link was not fully planned, but something that arose as a consequence of choices, made directly, during the performance. 

4. Laughter 

In this sequence, a loop created with a starting point in the forest close to the college's premises at Kronstad in Bergen dominates. Here we see how live video of two of the musicians is included in the video image, so that they almost step forward between the trees.

5. Frozen flowers

In the last part, links are created between the different characters and the flowers are drawn out more clearly as a symbol. The old man may be looking back, and the musical and visual narrative is drawing to a kind of conclusion.

References

________________________________________


Video documentation

Video from performance, Frozen Flowers – Zosimos at Cornerteateret 17.12.2023

https://youtu.be/9SaSPySx_eE