Escher's Reflecting Sphere

The Dutch artist M.C. Escher is one of my favourite artists. His surreal depictions of recursion, self-reference, and infinity strike a strong chord with me as a programmer.

This is an attempt at creating artwork inspired by his famous self-portrait, using Javascript and ThreeJs, Pixel Shaders, and a phone camera.

First, the result: A sphere that reflects the image from your webcam.

Of course, this has some deficiencies - because a phone camera is flat, you do not see a true 3D view, so the hand holding the phone is never visible. However, the phone sensors do allow us some other liberties, such as adding a shadow beneath the sphere.

This is how the effect above is produced.

Step 1: Create a renderer and scene using ThreeJS

This will be used to render the sphere.

const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(45, width / height, 0.01, 100);
const renderer = new THREE.WebGLRenderer({ alpha: true, antialias: true });
        

Step 2: Add lighting

ThreeJs has several types of lights (and cameras) each with their own features. A simple AmbientLight would suffice for a basic scene, but this does not cast shadows. For this reason I went with a SpotLight, which does.

const spotLight = new THREE.SpotLight(0xffffff, 2 );
spotLight.position.set( -10, 150, -10 );
scene.add( spotLight );
        

Using a camera helper and the OrbitControls helper is invaluable in testing and debugging anything to do with ThreeJs.

The camera helper helps trace how the light affects the screen and the orbit controls helps adjust the camera for best effect.

Setting up the lights and camera just right so they focus on the right part of the scene at the right distance and angle can be tedious without these tools.

Step 3: Add the sphere and a floor below it to receive shadows

const sphere = new THREE.Mesh(
    new THREE.SphereGeometry(sphereRadius, 64, 64),
    new THREE.MeshPhongMaterial({ color: 0xffffff })
);
sphere.castShadow = true;
sphere.receiveShadow = true;
scene.add(sphere);

const floorGeometry = 
    new THREE.PlaneBufferGeometry(1000, 1000);
const floor = new THREE.Mesh(floorGeometry,
    new THREE.MeshPhongMaterial({ color: 0xffffff })
);
// Move floor into position
floor.position.y = -sphereRadius;
floor.rotation.x = Math.PI / - 2;
floor.castShadow = false;
floor.receiveShadow = true;
scene.add(floor);
        

This adds the sphere and a floor, and then translates and rotates the floor into position beneath the sphere, ready to catch the sphere's shadows.

Both objects consist of a geometry, specifying the number of sides and faces; and a material which describes its appearance. This is a solid colour for now.

Step 4: Replace the solid colour material with a webcam feed

var videoImage = document.getElementById('videoImage');
videoImageContext = videoImage.getContext('2d');
videoImageContext.fillRect(0, 0, 
    videoImage.width, videoImage.height);

var videoTextureM = new THREE.Texture(videoImage);
        

This gets the webcam feed, renders it to a canvas, and then creates a ThreeJS texture from the canvas, ready to apply onto the sphere replacing the solid colour texture.

Step 5: Use Shaders to create a pencil sketch effect

A fragment shader (aka Pixel shader) takes an input pixel (represented by an RGBA colour value), applies some processing on it, and outputs another RGBA value which may be used by the renderer to represent the final pixel on the screen.

This is based on an awesome shader from user Starea on Shadertoy: Sketchy Stippling Stylization

// It takes each pixel
vec3 col = texture2D(webcam, vUv).rgb;

// Applies a random blur to it
vec2 r = random(vUv);
vec3 blurred = 
    texture2D(webcam, vUv + cr * (vec2(44.0) / 1000.0) ).rgb;

// Does a colour dodge over the original image, 
// merging the blur over the original drawImage
// This retains the sharp edges of the original 
// image along with the blurred edges of the blur layer
vec3 inv = vec3(1.0) - blurred;
vec3 lighten = colorDodge(col, inv); 

// Converts to greyscale
vec3 gcol = vec3(greyScale(lighten));

// And outputs to screen
gl_FragColor = vec4(gcol, 1.0);
        

Fragment shaders are a fascinating subject, and too complicated to go into detail here.

Pixelshaders.com is an incredible resource to get started with shader programming, explaining the basics for beginners.

Shadertoy provides many shaders to play with, you can edit them right in the browser.

Step 6: Use the phone gyroscope to move the lights as you move your phone

The shadow below looks more realistic if it moves along with your phone. Making this happen is as simple as getting the angle of rotation from the phone gyroscope and moving the light appropriately.

window.addEventListener("deviceorientation", function (event) {
    // Move the light to match the position of gravity
    spotLight.position.x = event.gamma / 3;               
}, true);

And this creates the final effect of the reflecting sphere.

This post covers the important bits of code, but a lot of what is needed to get this project to run has been left out.

View the source of this webpage for the full working code.