Split/Merge AudioVisual 視聴覚の分離と融合

index@kenjikojima.com

Technical Note of "Split/Merge AudioVisual" using LiveCode programming 1, 2, 3
When we treat different sences of computer files in binary, there is no boundaries between them.




Please consider making a donation
to support Kenji Kojima's works

Software Art: Split/Merge AudioVisual
Artist: Kenji Kojima

The project "Split/Merge AudioVisual" split a photograph to a random color pixels file and a music file, and merged them to the original photograph. It was an experimentation that different senses "Audio and Visual" erased their boundaries. They might be fundamentally a same perception. It was also a transformation, and an encryption and a decryption of visual data by an audio key. Actually the programming method was one time pad that was a type of encryption scheme. One time pad was used quantum cryptography which was impossible to crack. The project handled a perception of visible(color) and invisible(music), and audible(music) and inaudible(color). It was the related things what was form that was emptiness, or what was emptiness that was form.




Download Full Screen Player "MergeAudioVisual" MacOS X Only
This application has only player function, does not split audio and visual of an image.
The performance of merging audio and visual files are "Bee Series" by Kenji Kojima.
When you like to stop it, please command-key down. If you are a Windows user, please see the videos below.

MergeAudioVisual_Mac05.zip
If an alert asked “MergeAudioVisual.app can’t be opened because it is from an unidentified developer". Please right click or mouse-click with control-key on the app icon, then select "Open" from the contextual menu.

If your MacOS cannot play MIDI files, download QuickTime 7.


Left: Opening the player.
Right: Select split audio and visual files.


Left: Set visual on full screen.
Right: Processing audio and visual. It takes 3 steps, collecting musical notes, converting notes to binary, and merging binaries of audio and visual.





Merge AudioVisual Performance Videos:
The Player "Merge AudilVisual" performs these files. The original art is not a video.


Bee_1380418994
Watch Video (original work is not a video)


Bee_1380587313
Watch Video (original work is not a video)


Bee_1381080098
Watch Video (original work is not a video)


Bee_1381333259
Watch Video (original work is not a video)



Bee_1381521759
Watch Video (original work is not a video)


Bee_1381523174
Watch Video (original work is not a video)



Bee_1382369078
Watch Video (original work is not a video)



Bee_1382730493
Watch Video (original work is not a video)



Bee_1384707307
Watch Video (original work is not a video)





Split:
The software splits a digital image to audio and visual files by a bitwise XOR operation (logical exclusive OR). The bitwise operation takes two bit patterns of equal length and performs exclusive OR on each pair of corresponding bits. The software makes random numbers. It is same numbers of pixels of the original image. A binary of the random number performs on a binary of the image by bitwise XOR. Bitwise XOR makes 0 and 0 to 0, 0 and 1 to 1, 1 and 1 to 0. But the position of the decimal does not change. If a binary of one pixel element is 10010011 and random number is 00100001, the result will be 10110010. See the example of 8 bits binary. It is same length of a color data element.

10010011 (original image)
00100001 (random number)


10110010 (the result)




PLAY MIDI File | STOP

Audio and Visual:
The software converted the random number data to musical notes by Kenji Kojima's RGB Music technology. RGB value 120 is converted to middle C. Each two step values increase or decrease a half tone. One pixel make three notes of harmony. 0,0,0 is no sounds. The note length is determined by the color value. The music is a twelve-tone music. The left sample MIDI file was created from the musical notes of the random numbers. You can listen it. The result of the image would be a new image which processed by bitwise XOR. The perception of human being could not recognize what the image was. We have 2 digital files as audio and visual. You can think that the audio is a key and the visual is a cipher image.




Merge:
The software converts an audio key (music file) to musical notes then makes binary. The binary performs on the binary of cipher image (visual) by bitwise XOR. See the example.

10110010 (split image)
00100001 (split audio key)


10010011 (same binary of the original)

The audio and visual files merge into an original image.




Project File Creation Software: Splitting Audio and Visual Files (left), Merging Files (right)
This application is not distributed.



Extended Development: studies began in the late February 2014.

Two Musics Merge into an Image Study 001 (02/2014) Watch Video
Two Musics Merge into an Image Study 002 (02/2014) Watch Video
Two Musics Merge into an Image Study 003 (02/2014) Watch Video
Two Musics Merge into an Image Study 006 (03/2014) Watch Video
Two Musics Merge into an Image: Bees and Birds (03/2014) Watch Video





Video, midi and photograph files are under Creative Commons License BY-NC-SA 3.0
Key words: Audio, Visual, Perception, Cognition, Technology, RGB Music, Cipher, Bitwise XOR, Transformation
Bee in Central Park Slideshow The Bee Series Photographs are used for this Project