Overview:
For our Halloween project, Chunhan Chen and I collaborated with NYU’s Graduate Musical Theatre Writing Program to create an interactive crystal ball for their fortune teller room. The idea is that 2 – 3 guests would enter the room and have their tarot cards read by the NPC (non-player character). In the middle of the reading, the alien ghost that haunts the town of Twyp would interrupt the scene and reveal clues about the death of the professor. We achieved this crystal ball using a touch sensor that would trigger this premonition animation. In the crystal ball’s non-triggered state, it would loop through various tv show clips from the 1950’s, which is the setting the town takes place in.
Process:
There were many parts to our 2 week long process, so this documentation is broken down into the chronological steps that were taken: a.) brainstorm, b.) idea + technique, c.) triggered state animation, d.) normal state animation, e.) fabrication, f.) coding, g.) testing, h.) putting it all together, i.) in action (to be added in after the musical show) , f.) reflection (also to be adding after the musical).
a.) Brainstorm
Before our first get together, I sketched up 4-5 ideas. Only 2 of them I was actually interested in pursuing further. After discussing, we agreed we were more excited to create a project for a group with a practical need.
Chunhan shared with me some details of the musical meeting and described to me the room that she was most excited to work on: the gourd museum/ blue room. This whole room seemed fabulous, so we got straight into emailing with Danny, the one coordinating the musical, and Briana, who was in charge of the Blue Room. However, we soon found out that 2 other ITP groups were already working on that room. We decided to look at the GMTWP’s Wish List and choose from there. The interactive on that list that we were most excited for was the Crystal Ball for the Fortune Teller Room.
Luckily, the interactive for this room was not taken. The very next day we were able to meet up and discuss their thoughts.
b.) Idea + Process
Our first meeting with Professor Robert and Wilson was extremely helpful as we went over more specifics beyond just having a crystal ball (i.e. what is the message the crystal ball is trying to tell, how does it relate to the whole musical storyline). Robert also had a crystal ball handy with a hole cut out ready to be used!
The idea is that Wilson will be reading tarot cards to 2-3 of the guests on the night of the musical. And in the middle of the reading, an alien ghost spirit would interrupt with a premonition of sorts; this premonition would reveal some details about the murder that will occur in the story’s overall plot.
Since the small town of Twyp is set in 1950’s, they had an idea of playing classic tv shows of that time when it was in it’s normal state. In this way the crystal ball would also act as symbol of a tv set.
We also talked about having an interaction that Wilson would be able to perform in order to unlock the triggered state. Some ideas that came up was trying to trigger the crystal ball with 3 claps or a specific melody. This eventually changed to be just touch, because we were afraid that there would be too many other sounds in the room and the sensor wouldn’t be able to read the sounds properly.
For creating the crystal ball effect, Professor Tom mentioned a technique called the Pepper’s ghost. This technique creates a projection of the graphic using the reflection created from a plexiglass/clear surface tilted at 45 degree angles. Chunhan and I both researched online about how to create this effect; here is a google sheet with some links we compiled.
The diagrams below explains how the Pepper’s Ghost effect works. Source: comsol.com
See a schematic and quote below regarding the pepper’s ghost. Here is the link of the source by Jason Poel Smith.
After researching, we sketched out how it would work in the fortune teller room. See diagram below.
Chunhan also diagrammed out how the Arduino would be connected to the serial and the p5.js sketch that would be loaded on a browser which the iPad would be loading as well. See sketch below.
For the tasks we agreed to divide up the work (since we are limited in time) and do a skill share session when we are further along with the project.
Tasks:
1.) Creating a web page that that the animations would be displayed in. It would need to be in the same 5.2 circle shape that the diameter of the crystal ball is. – Chunhan
2.) Gathering the tv clips. Animating the “triggered” state with all the hints. – Emily
3.) The interaction of finger hitting on the force sensing resistor or touch sensor so that it would start the “triggered state.” – Chunhan
4.) Fabrication of the base so that the crystal ball will not move around/ change positions when it is on top of the iPad. – Emily
5.) Prototyping in space and email communications with Robert, Wilson and Danny – Chunhan and Emily
c.) Triggered State Animation
For the animation, I tried to incorporate all the elements and ideas that Professor Robert Lee, Wilson Weber, Chunhan and I had discussed in the prior meeting. The main points after our talk was that the triggered animation should: 1.) allude to the murder of the professor (potentially by acid), 2.) include the Cthulhu alien monster symbol, 3.) have a Twilight Zone, intergalactic, sci-fi vibe, 4.) potentially match the gypsy fabrics that would be used in the fortune teller room. From this very helpful information, I sketched up a quick storyboard:
The next step was drawing the scenes in photoshop. Below are some screenshots of the Photoshop files with all the assets separated into different layers ( to make it easy to animate in After Effects).
Then I brought in these into After effects and started animating! Love having the chance to animate for a project! Below are some screenshots of the process of working in After Effects.
Side note & shout out: the cool galaxy animation I used for the background was initially taken from this clip. I downloaded it and was able to manipulate it using After Effects’ Kaleidoscope and Flomotion effect to create some animated gypsy like patterns!
Here was the first version of the animation.
We wanted to share this with Professor Robert and Wilson because we were not sure if this animation was giving away too many clues about the murder. It is also good to just check in and make sure we are on the right track. So, we emailed them this animation and some updates on our progress last Thursday.
Almost immediately they responded and let us know that the animation wasn’t too explicit and they loved where it was going! Big yay! They were also kind enough to give us some helpful direction and great ideas. Robert mentioned that he thought we could drop even more hints and incorporate the idea that the professor in the story is a woman. He was also hoping for it to evoke some H.P. Lovecraft feel and suggested adding in tentacles and eyes (as a lot of H.P. Lovecraft artwork incorporates those elements). Wilson agreed and also suggested possibly incorporating the Cthulhu symbol at the end.
This was such helpful feedback and I got to work on incorporating those changes by including the tentacles and eyes, making it more sepia-toned, adding a women’s voice screaming at the end. Chunhan also mentioned that the animation gif had to be under 5MB in order for it to be loaded in p5.js. Therefore, I sped up the animation so it would be able to export under 5 MB. Below is the newest version with all the edits mentioned above.
Then the mp4 was exported into Photoshop and exported into a low-res gif. So the version that will be seen via the reflection in the crystal ball, won’t as crystal clear as these mp4 videos seen here, but I’m hoping the general vibe and storyline will still show through.
d.) “Normal State” Animation
The idea that Robert and Wilson came up with is that in the crystal ball’s “normal state” there would be classic 1950’s television clips (i.e. ” Twilight Zone”, “Bewitched”, “I Love Lucy”).
Chunhan mentioned that it would take the browser longer to load if there were too many clips, which is why we decided on 5 tv clips. I searched through Youtube and made a playlist with some episodes, and tried to find a good variety between the “everyday” life tv shows like “I Love Lucy” and the more absurd horror like shows, such as “The Twilight Zone.”
I then took these clips and edited it down to 10 seconds. Then these clips had to be turned 180 degrees because the screen will be reflected on the transparent sheet inside the globe. Below is a screenshot of the process.
Chunhan was thankfully careful enough to notice that one of the clips was not reverted. So, she helped turn it upside down herself. She was then able to test it out in the code.
e.) Fabrication
Initially we had just talked about putting a black fabric over the iPad and cutting a hole out. However this would both not provide a sturdy connection between the iPad and the crystal globe and it would not look great. We needed a way to position the iPad in relation to the cutout hole from which the iPad will project it’s circular image. First step was sketching different designs out. See sketch below.
After sharing the designs with Chunhan, we decided on which one would work best for us. This box design works like a drawer so that there is an insert that the iPad would be placed on and be able to slide into the base of the box. This would do 3 things: 1.) place the iPad with the circle animation in the exact location in relation to the crystal ball, 2.) have a nice opening for the iPad charger, 3.) look more structurally sound and considered.
Next was getting precise measurements for everything, including the exact placement that the circle on the iPad would be with the cutout of the hole for the top layer of the box. The insert for this iPad was a fun challenge: it required consideration of the diameter of the inner rim of the box, and to keep in mind the depth of the wood that will be used. Below are some notes, measurements and sketches that were taken to design the box and get it ready for laser cutting.
I got the measurements for the depth of the iPad Pro and inner diameter of the crystal ball using the caliper.
Next came laying out the individual parts in illustrator to get it ready to be laser cut. I used an online box template and adjusted it to be the dimensions we had measured. After that was done it was ready for laser cutting on the cardboard prototype. See images below of the illustrator files. Then I assembled the cardboard prototypes together.
The issue that we ran into after putting this prototype together was the circle cutout on the top layer. We had hoped that the bottom of crystal ball would be able to fit snugly in the hole so it would be a good support for it. Unfortunately, it would be able to work. I tried cutting out a bigger hole, but the bottom of the circle was still not able to properly fit in the hole (see images below for reference). The solution was to just make the hole smaller and just glue the crystal ball’s bottom rim to the surface of the base.
After the design and measurements were adjusted in Illustrator, the wood version was laser cut and assembled!
The insert worked! See video below.
One big mistake was the choice of wood. This 3/16″ plywood was cheap but it was not great wood for laser cutting. It was not cutting the wood cleanly and it was also taking the laser cutter at least 6 runs before the cuts would go through the wood. Many times I had to use force the cutouts from the wood by hand and chisel them out as well. This was unfortunate because it would ruin the wood and create these tears in the wood. It also created a dark and ashy burn mark on the edges. See the image below. Lesson learned: research different types of woods and try not to laser cut anything over 1/8″ thickness.
Next is to sand and spray paint! This was a straightforward and fun process. See images below. I ultimately decided to not spray the support stands and the insert inside the box because I was afraid that the rougher texture of the spray paint would not allow the sanded down wood to slide into the box as nicely.
Then we put it all together with the work Chunhan has been doing with the touch sensor and p5.js!
f.) Arduino and p5.js (See Chunhan’s blog for detailed documentation)
Chunhan coded it so that when Wilson hits the sensor 3 times it will set the “triggered” animation. When he hits it 2 times it returns to the “normal” state.
I set the “trigger cipher” to 2, which means that when Wilson hit the sensor for twice, it will change the triggerState to “1” and then reset the switchCount value to 0. Similarly, a “normal cipher” of 3 means that when he hit the sensor for another three times, it will change the triggerState to “0” and then reset the switchCount value to 0. – Chunhan Chen
Here is the final code that Chunhan wrote the crystal ball:
” (1) Arduino part
const int switchPin = 2;
boolean triggerState = false;
const int triggerCipher = 2; //touch twice to toggle the state to “triggered”
const int normalCipher = 3; //touch three times to toggle the state to “normal”
int switchCount = 0; //recording the number of times people touch the switch
int prevSwitchValue = 0;
void setup() {
pinMode(switchPin, INPUT);
Serial.begin(9600);
// Starting the “hand-shaking”
while(Serial.available() <= 0){
Serial.println(“Waiting Data”);
delay(300); //wait 1/3 second
}
}
void loop() {
checkButton();
checkTriggerState();
if (Serial.available()>0){
int inByte = Serial.read();
//send a new message to p5.js if it has been triggered
Serial.println(triggerState);
}
}
void checkButton(){
//read the touch sensor value
int switchValue = digitalRead(2);
//if the switch has changed,
if (switchValue != prevSwitchValue){
//debounce the switch;
delay(5);
//and that the switch is touched
if (switchValue == HIGH){
switchCount++;
}
}
prevSwitchValue = switchValue;
}
void checkTriggerState(){
if (triggerState==false){
if (switchCount >=triggerCipher){
triggerState = true;
switchCount = 0; //reset the swithCount to 0.
}
}
if (triggerState==true){
if (switchCount >=normalCipher){
triggerState = false;
switchCount = 0; //reset the switchCount to 0.
}
}
}
(2) p5.js part
/*
Holloween CrystalBall
P5.js side for Physical Computing mid-term
Made by Emily Lin and Chunhan Chen
Oct 22 2018
NOTE: You have to use “python -m SimpleHTTPServer” in the terminal.
Then open a tab and type “localhost:8000” for the address to get access to the music file.
If you are viewing from another device, use “ID address:8000” instead. Make sure the device share the same local internet with the laptop.
Also you should keep p5.serialcontrol application open while running the program.
Reference: https://codepen.io/dclappert/pen/mJeYye?editors=1100
*/
const normalJpegNum = 0;
const normalGifNum = 5;
const triggeredJpegNum = 0;
const triggeredGifNum = 1;
var normalAsset = [];
var triggeredAsset = [];
var triggeredSound;
var normalInterval = 10; // Interval to switch images of normal conditions
var triggeredInterval = 7.2; // Interval to switch images of triggered conditions
var triggerState = 0; //KEY VALUE! To record the triggered (1) or normal (0) state.
var prevTriggerState = 0;
var triggerFrame; // to record the frame that triggering happens
var nIndex = 0;
var tIndex = 0;
var serial; // instance of the serial port library.
var portName;
function preload() {
if (normalJpegNum>0) {
for (leti=1; i<=normalJpegNum; i++) {
letimg=createImg(‘normal_state_assets/normal_’+i+”.jpeg”);
img.position(windowWidth/2-0.415*windowWidth, windowHeight/2-0.415*windowWidth);
img.size(0.83*windowWidth, 0.83*windowWidth);
img.class(“round”);
img.style(“visibility”, “hidden”);
}
}
if (normalGifNum>0) {
for (leti=1; i<=normalGifNum; i++) {
img=createImg(‘normal_state_assets/normal_’+i+”.gif”);
img.position(windowWidth/2-0.415*windowWidth, windowHeight/2-0.415*windowWidth);
img.size(0.83*windowWidth, 0.83*windowWidth);
img.class(“round”);
img.style(“visibility”, “hidden”);
normalAsset.push(img);
}
}
if (triggeredJpegNum>0) {
for (leti=1; i<=triggeredJpegNum; i++) {
img=createImg(‘triggered_state_assets/triggered_’+i+”.jpeg”);
img.position(windowWidth/2-0.415*windowWidth, windowHeight/2-0.415*windowWidth);
img.size(0.83*windowWidth, 0.83*windowWidth);
img.class(“round”);
img.style(“visibility”, “hidden”);
triggeredAsset.push(img);
}
}
if (triggeredGifNum>0) {
for (leti=1; i<=triggeredGifNum; i++) {
img=createImg(‘triggered_state_assets/triggered_’+i+”.gif”);
img.position(windowWidth/2-0.415*windowWidth, windowHeight/2-0.415*windowWidth);
img.size(0.83*windowWidth, 0.83*windowWidth);
img.class(“round”);
img.style(“visibility”, “hidden”);
triggeredAsset.push(img);
}
}
//mySound = new sound(‘triggered_state_assets/triggered_1.mp3’);
triggeredSound=loadSound(‘triggered_state_assets/triggered_1.mp3’);
}
function setup() {
triggeredSound.loop(0,1,1,0,7.68);
serial=newp5.SerialPort(‘10.17.35.252’);
serial.on(‘list’, printList); // ‘list’ is an event, printList is a call back function
serial.on(‘connected’, serverConnected);
serial.on(‘open’, portOpen);
serial.on(‘data’, serialEvent);
serial.on(‘error’, serialError);
serial.on(‘close’, portClose);
}
function windowResized() {
//To automatically adjust the size of images, based on physical mesasurements of the crystal globe and the iPad pro screen.
for (letthisImgofnormalAsset){
thisImg.size(0.83*windowWidth, 0.83*windowWidth);
thisImg.position(windowWidth/2-0.415*windowWidth, windowHeight/2-0.415*windowWidth);
}
for (letthisImgoftriggeredAsset){
thisImg.size(0.83*windowWidth, 0.83*windowWidth);
thisImg.position(windowWidth/2-0.415*windowWidth, windowHeight/2-0.415*windowWidth);
}
}
function draw() {
console.log(triggerState);
letnIntervalCount=normalInterval*60;
lettIntervalCount=triggeredInterval*60;
if (triggerState==0){
triggeredSound.amp(0); // set the volume of the sound to 0.
for (thisImgoftriggeredAsset){
thisImg.style(“visibility”, “hidden”); // hide all of the triggered state images
}
if (nIndex>=normalAsset.length){
normalAsset[nIndex-1].style(“visibility”, “hidden”);
nIndex=0;
}
normalAsset[nIndex].style(“visibility”, “visible”);
if (frameCount%floor(nIntervalCount) ==0){
if(nIndex<normalAsset.length){
normalAsset[nIndex].style(“visibility”, “hidden”);
nIndex++;
}
}
}
if (triggerState==1){
if(prevTriggerState==0){
triggerFrame=frameCount;
}
for (thisImgofnormalAsset){
thisImg.style(“visibility”, “hidden”);// hide all of the normal state images
}
if((frameCount-triggerFrame) %round(tIntervalCount) ==0){
triggeredSound.amp(1);// set the volume of the sound to normal.
// triggeredSound.playMode(‘restart’);
// triggeredSound.play();
triggeredAsset[0].style(“visibility”, “visible”);
}
//console.log(triggeredSound.frames());
}
prevTriggerState=triggerState;
}
//Below are serial communication parts.
function printList(portList) {
// portList is an array of serial port names
for (vari=0; i<portList.length; i++) {
// Display the list the console:
console.log(i+” “+portList[i]);
//automatic choose port as Arduino port:
if (portList[i].indexOf(‘usbmodem’) >=0) {
portName=portList[i];
console.log(‘–Using ‘+portName+’ as serial port, probs Arduino’);
serial.open(portName, {
baudrate:9600
});
}
}
}
function serverConnected() {
console.log(‘connected to server.’);
}
function portOpen() {
console.log(‘the serial port opened.’)
}
function serialEvent() {
varinString=serial.readStringUntil(‘\r\n’);
if (inString.length>0) {
if (inString!=”Waiting Data”) {
triggerState=Number(inString);
}
}
serial.write(‘x’);
}
function serialError(err) {
console.log(‘Something went wrong with the serial port. ‘+err);
}
function portClose() {
console.log(‘The serial port closed.’);
}
(3) CSS part
html, body {
margin: 0;
padding: 0;
background-color: black;
}
canvas {
display: block;
}
.round{
overflow:hidden;
border-radius:50%;
}”
Note from Chunhan:
” NOTE: You have to use “python -m SimpleHTTPServer” in the terminal. Then open a tab and type “localhost:8000” for the address to get access to the music file. If you are viewing from another device, use “ID address:8000” instead. Make sure the device share the same local internet with the laptop. Also you should keep p5.serialcontrol application open while running the program.
Reference: https://codepen.io/dclappert/pen/mJeYye?editors=1100″
g.) Prototyping in Space
We met up with Wilson and Robert in the fortune teller room in order to test out our crystal ball in the space. Some issues that came up was that the crystal ball is not at eye level. We decided that it would have to be elevated using books to get it to be eye level.
Another issue that came up was that there is a time lag between what the HTTP site on the laptop and the iPad’s browser. This seems to be an issue with the wifi connection. The FSR also was not the most reliable source for reading the values, so it was sometimes hard to tell if the issue was the sensor or if there was just a lag between the server on the iPad and the laptop. See image of the prototyping session below.
Thankfully, Chunhan’s touch sensor had just arrived! So she adjusted the code and attached the new touch sensor to the Arduino yesterday (10/23/18). She also covered the touch sensor and wires with Gaff tape so that it both looks more hidden and secures the sensor to the board!
h) Putting it All Together
Today (10/24/18) we went to the fortune teller room to prototype again with the touch sensor instead of the FSR. We also wanted to get a nice shot of everything put together in the actual setting that it will be in.
Emily Lin 