A Year of Sunflower

overview

A Year of Sunflower is an interactive multimedia installation, in which users engage with a combination of soft electronics and computer vision to navigate an experimental storybook. The project itself is comprised of two halves: a digital collage and the handknitted sunflowers used to operate it. Within the collage are layered images, shifting snippets of writing, various symbols, and letter fragments; the intention is to create an impressionistic sense of unfamiliarity, while emphasizing themes of language as a lens, hope rooted in agency, and the trappings of individualism. Using soft materials to control the installation is about enchantment: the possibility for technology to progress beyond the rigid and mechanical, while also fostering an appreciation for material, and returning a sense of innate wonder to the world through physicality.

A Year of Sunflower tells a story of people, not necessary as protagonists, but as actors within a larger tapestry of intermingled living systems. The letter fragments, and their illegibility, illustrate how language and, by extension, human intention and action, isn't enough to encompass the living world. Simultaneously, individuals cannot hope to fathom the complexities required to address the many ecological crises we face, at least not alone. The collage then, is a literal representation of the patchwork of lives and livelihoods which contribute to the story of people and how, from that perspective, we can hope to find change in small places.

ideation

The first stage of the project centered around fleshing out its two main features. Assembling the collage meant developing a visual style and piecing together assets, while the physical component required a form and means of being tracked by the computer. To address the latter, considered options like RFID tags and more advanced forms of computer vision, eventually selecting color tracking as the most reliable method.

As for the controls, originally, I wanted to experiment with lights or color films as a non-conventional kind of interaction, before settling on something more tangible, as the boundaries of the color tracking became more apparent. Because of its intuitive computer vision capabilities and the fact that I was already well versed in the language, I chose to code the project using p5.js

process

Color Tracking

Generating the initial color tracking code was fairly straightforward, which iterated through every pixel in the video input and noted the average position of designated colors. Following that, I underwent a much longer process of refinement. My testing setup involved a webcam pointed at colored marker caps, which affected the display based on the distance between them. That distance-based logic would remain a throughline in determining the mechanics of the final design, where the position of each object controls some portion of the display.

Difficulties emerged with the first algorithm as testing progressed. It was highly sensitive to changes in lighting and failed to differentiate between similar colors (red and yellow for example) in most kinds of light. Beyond that, the averaging logic often meant the positions being displayed weren’t well aligned with the actual positions of the objects being tracked. Meaning it was hard to understand how physical movements related to onscreen changes and the interaction as a whole didn’t feel very fluid.

To account for these limitations, I tried first experimenting with different lighting setups and altering sensitivity values, to see if they could be corrected in ideal circumstances. These efforts fell short in the end and I needed to rework the underlying color tracking formula. Instead of averaging the color of pixels on screen, the final iteration applies a grouping technique to similar color pixels, before taking the average position of the largest group of each color. This method allows for greater consistency in positioning, as stray bits of discoloration due to atmospheric lighting would be much smaller in comparison to the objects being tracked. Lighting still plays a role in how the colors appear to the computer; so, for further tests I used a white cloth as a canvas and artificial lighting in dark room, opposed to natural light, which is too variable.

Knitting Sunflowers

During the development of the color tracking code, it became clear that lights or film wouldn’t suffice as control devices, given the general sensitivity to lighting changes and so sunflowers felt like a natural choice of object. I chose to knit them for two reasons: one, it felt more organic and interesting to handle the soft material, opposed to the kind of plastics, metals, or wood that might typically be involved in projects like this.

Second, because the color tracking camera needed a top-down view, users couldn’t move the sunflowers without obscuring the camera’s view. This felt clunky, so I decided to embed a magnet in each sunflower and make a fifth flower to manipulate them from underneath. The yarn was lightweight enough to allow for this kind of interaction and reintroduced some of the novelty I was looking for during ideation.

Without knowing how to knit, my partner helped show me the basics and create the underlying structure for most of the petals. We went through one iteration with larger petals, before settling on the final design. In addition to the magnet, each flower has a faint light in its center for visual effect.

Animating the Collage

On the digital side, the collage consists of 8 sections split between 4 chapters. Each section contains four images paired with animated text. The types of animation applied to each line are generally meant to align with the words being animated. Only portions of text within each image can be viewed at once, so it can be read differently as users move through the page. In terms of navigation, the colors of each sunflower are related in a set order, such that, the farthest left determines the chapter and the relative positions of the remaining three are used to scroll across the page, change the section, and animate the text.

A digitally operated version of the collage can be found here.

The controls are as follows:

Left & right arrow keys - cycle between chapters
Up arrow key - switch between sections within a chapter
Shift - enter and exit fullscreen
Mouse movement - scroll and animate text
Left click - hold in place

User Testing

After the major components were complete, I conducted several full-scale user tests which assured the intended functionality of the project. After a few iterations, the tests revealed that a projection was a better method of displaying the collage vs a monitor, which felt less immersive. Additionally, the overhead camera was somewhat distracting and in an ideal setup, would be much higher above the surface where the sunflowers rest.

Controlling the sunflowers using the magnets was the most satisfying part of the interaction. That said, there’s also a slight tension between wanting a large amount of space to move the sunflowers around, without making it difficult to reach each flower from underneath the table. An interesting future direction might involve scaling certain elements to further enhance emersion and allow for greater freedom of movement, guiding the sunflowers across a lengthy surface.