Time for a change of pace.
Last time I finished up promising to look at a non-invasive way to render wireframes in WebGL, but that’s really a bit boring. Instead, I’m going to focus on updating a texture after it’s been created. See, a texture isn’t fixed once uploaded to the GPU (via texImage2D); provided that you don’t change the dimensions, it can be updated with the appropriately-named texSubImage2D.
The Difference Between texImage2D and texSubImage2D
You use texImage2D to initialise (or reinitialise) storage for a texture. Bearing in mind that createTexture just reserves an identifier, texImage2D can be thought of as allocation plus a copy of texture data. Compared to that, texSubImage2D is just the copy, making it significantly faster. As the name implies, you can choose to overwrite part of the associated texture, but you can also replace the whole image.
Sourcing a Moving Image
For this demo to be convincing, I need multiple images of the same dimensions. In order to show off the impressive bandwidth between your CPU and graphics card, this image should change frequently. We could some kind of page-stack animation in a loop, but—for suitably-equipped hardware—HTML 5 provides access to a webcam via getUserMedia. Binding to a <video> element is fairly simple, which is great, because that element can be the destination parameter for texImage2D and texSubImage2D.
Internet Explorer and Safari users: sorry, but your browser of choice is lagging behind Mozilla and Google. getUserMedia was unavailable to you when I wrote this post.
Making Things a Bit More Interesting
Stitching video onto a flat surface is a good way to create a highly expensive, power-hungry mirror, so I’m going to give the vertex shader something to do: time-based mesh deformation. This will give the video a ripple effect. Hopefully one smoother and more colourful than this animated gif:
The version of Firefox that I’m using lets me run this straight off the disk. Chrome insists that it be provided by a web-server (python -m SimpleHTTPServer to the rescue).
The Code
<!doctype html>
<html>
<head>
<title>Hacking WebGL</title>
<script type="text/javascript" src="gl-matrix.js"></script>
<script id="vertex-shader" type="x-shader/x-vertex">
precision mediump float;
uniform mat4 modelMatrix, viewMatrix, projectionMatrix;
uniform float time;
attribute vec3 pos;
attribute vec2 texCoords;
varying vec2 tc;
void main() {
float d = -length(pos);
float z = 0.05 * sin(3.141592653589793 * d * 5.0 + time * 3.0);
tc = texCoords;
gl_Position =
projectionMatrix * viewMatrix *
modelMatrix * vec4(pos.xy, z, 1.0);
}
</script>
<script id="fragment-shader" type="x-shader/x-fragment">
precision mediump float;
uniform sampler2D image;
varying vec2 tc;
void main() {
gl_FragColor = texture2D(image, tc.st);
}
</script>
<script type="text/javascript" src="gl-matrix.js"></script>
<script type="text/javascript">
function createCanvas() {
var canvas = document.createElement('canvas');
document.getElementById('content').appendChild(canvas);
return canvas;
}
function createProgram(gl, shaderSpecs) {
var program = gl.createProgram();
for ( var i = 0 ; i < shaderSpecs.length ; i++ ) {
var spec = shaderSpecs[i];
var shader = gl.createShader(spec.type);
gl.shaderSource(
shader, document.getElementById(spec.container).text
);
gl.compileShader(shader);
if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) {
throw gl.getShaderInfoLog(shader);
}
gl.attachShader(program, shader);
gl.deleteShader(shader);
}
gl.linkProgram(program);
if (!gl.getProgramParameter(program, gl.LINK_STATUS)) {
throw gl.getProgramInfoLog(program);
}
return program;
}
var frameCount = 0, fpsTimer = null;
function resetFpsCounter() {
fpsTimer = setTimeout(function() {
fpsTimer = null;
}, 1000);
}
function render(gl, scene, time) {
if ( fpsTimer == null ) {
console.log(frameCount);
frameCount = 0;
resetFpsCounter();
}
gl.clear(gl.COLOR_BUFFER_BIT);
gl.useProgram(scene.program);
gl.uniform1f(scene.program.timeUniform, time / 1000);
gl.uniformMatrix4fv(
scene.program.modelMatrixUniform, gl.FALSE,
scene.object.modelMatrix);
gl.bindBuffer(gl.ARRAY_BUFFER, scene.object.buffer);
gl.bindTexture(gl.TEXTURE_2D, scene.object.texture);
var video = scene.object.textureSourceElement;
gl.texSubImage2D(
gl.TEXTURE_2D, 0, 0, 0, gl.RGBA,
gl.UNSIGNED_BYTE, video);
gl.drawArrays(
scene.object.primitiveType, 0,
scene.object.vertexCount);
gl.bindTexture(gl.TEXTURE_2D, null);
gl.bindBuffer(gl.ARRAY_BUFFER, null);
gl.useProgram(null);
frameCount++;
requestAnimationFrame(function(timestamp) {
render(gl, scene, timestamp);
});
}
function createFlatMesh(gl) {
var MAX_ROWS=64, MAX_COLS=64;
var points = [];
for ( var r = 0 ; r <= MAX_ROWS ; r++ ) {
for ( var c = 0 ; c <= MAX_COLS ; c++ ) {
points.push({
location: [-0.75 + (1.5 / MAX_COLS) * c,
0.75 - (1.5 / MAX_ROWS) * r,
0.0],
texture: [1.0 / MAX_COLS * c,
1.0 / MAX_ROWS * r]
});
}
}
var OFFSET = function(R,C) {
return ((R) * ((MAX_COLS)+1) + (C));
};
var
vertices = [],
rotations = [-1,-1,-1,0,1,1,1,0,-1,-1,-1,0,1,1,1,0];
for ( var r = 1 ; r <= MAX_ROWS ; r += 2 ) {
for ( var c = 1 ; c <= MAX_COLS ; c += 2 ) {
for ( var i = 0 ; i < 8 ; i++ ) {
var off1 = OFFSET(r, c);
var off2 = OFFSET(r + rotations[i], c + rotations[i+6]);
var off3 = OFFSET(r + rotations[i+1], c + rotations[i+7]);
Array.prototype.push.apply(
vertices, points[off1].location);
Array.prototype.push.apply(
vertices, points[off1].texture);
Array.prototype.push.apply(
vertices, points[off2].location);
Array.prototype.push.apply(
vertices, points[off2].texture);
Array.prototype.push.apply(
vertices, points[off3].location);
Array.prototype.push.apply(
vertices, points[off3].texture);
}
}
}
var buffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, buffer);
gl.bufferData(
gl.ARRAY_BUFFER, new Float32Array(vertices),
gl.STATIC_DRAW);
gl.bindBuffer(gl.ARRAY_BUFFER, null);
return {
buffer: buffer,
primitiveType: gl.TRIANGLES,
vertexCount: vertices.length / 5
}
}
function loadTexture(gl, mesh, andThenFn) {
var texture = gl.createTexture();
navigator.getUserMedia = navigator.getUserMedia // WC3
|| navigator.mozGetUserMedia // Mozilla
|| navigator.webkitGetUserMedia; // Chrome
navigator.getUserMedia(
{video: true, audio:false},
function(stream) {
var video = document.getElementById('video');
video.src = URL.createObjectURL(stream);
video.onplaying = function() {
gl.bindTexture(gl.TEXTURE_2D, texture);
gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, false);
gl.texImage2D(
gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, video);
gl.texParameteri(
gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
gl.texParameteri(
gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(
gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texParameteri(
gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.bindTexture(gl.TEXTURE_2D, null);
mesh.texture = texture;
mesh.textureSourceElement = video;
andThenFn();
};
video.play();
}, function(e) {
alert(e);
});
}
function init() {
var canvas = createCanvas();
var gl = canvas.getContext('experimental-webgl');
var resize = function() {
canvas.width = window.innerWidth;
canvas.height = window.innerHeight;
gl.viewport(0,0,canvas.width,canvas.height);
};
window.addEventListener('resize', resize);
resize();
gl.enable(gl.DEPTH_TEST);
gl.clearColor(0.0, 0.0, 0.0, 0.0);
var mesh = createFlatMesh(gl);
var program = createProgram(
gl,
[{container: 'vertex-shader', type: gl.VERTEX_SHADER},
{container: 'fragment-shader', type: gl.FRAGMENT_SHADER}]);
var projectionMatrix = mat4.create();
mat4.perspective(
projectionMatrix, 0.75, canvas.width/canvas.height,
0.1, 100);
var viewMatrix = mat4.create();
var modelMatrix = mat4.create();
mat4.translate(modelMatrix, modelMatrix, [0,0,-2]);
mat4.rotate(modelMatrix, modelMatrix, -1, [1,0,0]);
mesh.modelMatrix = modelMatrix;
gl.useProgram(program);
program.modelMatrixUniform =
gl.getUniformLocation(program, 'modelMatrix');
program.viewMatrixUniform =
gl.getUniformLocation(program, 'viewMatrix');
program.projectionMatrixUniform =
gl.getUniformLocation(program, 'projectionMatrix');
program.timeUniform =
gl.getUniformLocation(program, 'time');
gl.uniformMatrix4fv(
program.projectionMatrixUniform, gl.FALSE,
projectionMatrix);
gl.uniformMatrix4fv(
program.viewMatrixUniform, gl.FALSE, viewMatrix);
gl.uniform1f(
program.timeUniform, gl.FALSE, 0.0);
gl.bindBuffer(gl.ARRAY_BUFFER, mesh.buffer);
program.positionAttribute =
gl.getAttribLocation(program, 'pos');
program.textureCoordsAttribute =
gl.getAttribLocation(program, 'texCoords');
gl.enableVertexAttribArray(program.positionAttribute);
gl.enableVertexAttribArray(program.textureCoordsAttribute);
gl.vertexAttribPointer(
program.positionAttribute, 3, gl.FLOAT, false,
5 * Float32Array.BYTES_PER_ELEMENT,
0);
gl.vertexAttribPointer(
program.textureCoordsAttribute, 2, gl.FLOAT, false,
5 * Float32Array.BYTES_PER_ELEMENT,
3 * Float32Array.BYTES_PER_ELEMENT);
gl.bindBuffer(gl.ARRAY_BUFFER, null);
gl.useProgram(null);
loadTexture(gl, mesh,
function() {
requestAnimationFrame(function(timestamp) {
render(gl, {
program: program,
object: mesh
}, timestamp);
})
});
}
</script>
</head>
<body onLoad="init()">
<video
id="video"
width="640" height="480"
style="display:none">
</video>
<div id="content"></div>
</body>
</html>
What’s New
The loadTexture function now refers to a <video> element, and uses the getUserMedia API to bind it to the input from a camera. The onplaying callback of the video element actually initialises the texture with the texImage2D call (allocating storage on the GPU and coyping pixel data from the surface), and then sets up the call to render. The play() method is immediately called on the element in order to kick the whole process off.
The render() method itself is much like it’s always been, with the exception of the call to texSubImage2D, using the video element as a pixel store. When the underlying mesh is drawn with drawArrays, the image is stitched over the surface just like any other.
Finally, the vertex shader does a little more than just translating between coordinate systems. It’s now generating a replacement z-index for each vertex that goes in, displacing it by a function of its distance from the centre of the view and the current time. There are some not-so-magic numbers in there: 0.05 is the amplitude, 5.0 is the frequency, and 3.0 is the speed. What’s worth noting here is that the GPU is doing the deforming. As far as the CPU is concerned, a flat mesh was uploaded to graphics memory once on initialisation and never updated. The only data being sent to the GPU thereafer are the new uniform values and the updated texture for each frame.
[side-note: It’s nice that all this work is being done by the GPU, but the trade-off is that the CPU can’t just look at its copy of the mesh to find out what you’ve selected if you click on something. It also complicates lighting, as the adjustments made by the vertex shader affec9t what the normals should be; these would have to be recalculated in shader, but the vertex shader doesn’t have visibility of the other vertices in its triangle. A geometry shader—which can see a whole polygon at a time—could calculate and feed appropriate values to the fragment shader, but WebGL doesn’t support those.]
Where did I get the time from? Well, the callback that you provide to requestAnimationFrame gets a parameter when it’s actually called: a timestamp in milliseconds. This timestamp doesn’t correspond to the actual time in any way, it’s just the time since the rendering cycle started for your page. In any case, it’s usable as something that varies relatively smoothly and which can be fed into the vertex shader to indicate change on a frame-by-frame basis.
Window Refresh vs. Camera Capture Rates
The browser refresh rate is going to be synchronised with your display—typically about 60fps—but unless you’re working with a good camera, the video data itself is only going to be updated at 30fps or less. This means that the code above pushes twice as much texture data to the GPU as it has to. If you’ve been tracking performance, you might be thinking that this is why the numbers say that the CPU is still quite busy. I thought it was worth trying out by passing along a step counter to my render function. If step % 2 == 0, upload the texture (i.e. only update the texture on every second frame). Any impact on performance wasn’t noticable. Trying mod-5 didn’t make a difference, either, so it doesn’t appear to be the OpenGL layer that’s chewing up time. Performance figures were similar between Firefox 40 and Chrome 43 on a 2011 Macbook Pro. I wonder what’s hogging the time?
Follow-Up Practice
If you don’t have access to a webcam, try replacing the getUserMedia code in loadTexture with something that plays a video in a loop.
Stereoscopic animated GIFs—a web search will yield many—give an impressive sense of 3D using just two images. Using an appropriate graphics package (I recommend GIMP), save their constituent images and replicate the effect by flipping between their images on the fly. [if you do this, it’s probably better not to update a single texture, but rather to initialise and upload two textures, flipping between which one you bind/render on render calls].
Create some HTML controls (e.g. <range> elements) for each of the amplitude, frequency, and speed. Add some uniforms to the vertex shader that allow these parameters to be manipulated on-the-fly.
The continuous ripple effect is pretty, but there’s fun in playing with it. How about a ripple that diminishes with the distance from the origin? What about a flag effect instead?
What’s Next
This series of blog posts was called ‘WebGL from Scratch’, and we’ve gone from drawing a red background through 2D shapes, 3D shapes, lighting, mesh loading, animation, and textures. That’s about as much as I want to accomplish in an introductory tutorial.
So is that all there is? Absolutely not. Graphics is an endlessly deep subject. Just scratching the surface, I haven’t talked about shadows, environment mapping, bump mapping, multi-texturing, or object picking. I haven’t played with any of the more difficult visual effects to model, such as fire, smoke, or glare. Lighting has been deliberately simple, and is usually extended with some kind of material parameters for specularity and reflection. Descriptions of multiple approaches to handle any one of these subjects are a web search away, in almost all cases trading off accuracy and efficiency.
The thread through these posts has been very much nuts n’ bolts. I don’t apologise for that. Higher-level APIs to 3D functionality is available via something like three.js, which masks complexity and gets results faster. My goal here though has been to get as close to the GPU as JavaScript and web browsers permit.
I hope you’ve enjoyed it.
If you would like something explained in more depth, attach a note and I’ll put together a post that builds on this material. Or posts, if need be.
Bits in Pieces 
