Time for a change of pace.
Last time I finished up promising to look at a non-invasive way to render wireframes in WebGL, but that’s really a bit boring. Instead, I’m going to focus on updating a texture after it’s been created. See, a texture isn’t fixed once uploaded to the GPU (via
texImage2D); provided that you don’t change the dimensions, it can be updated with the appropriately-named
The Difference Between
texImage2D to initialise (or reinitialise) storage for a texture. Bearing in mind that
createTexture just reserves an identifier,
texImage2D can be thought of as allocation plus a copy of texture data. Compared to that,
texSubImage2D is just the copy, making it significantly faster. As the name implies, you can choose to overwrite part of the associated texture, but you can also replace the whole image.
Sourcing a Moving Image
For this demo to be convincing, I need multiple images of the same dimensions. In order to show off the impressive bandwidth between your CPU and graphics card, this image should change frequently. We could some kind of page-stack animation in a loop, but—for suitably-equipped hardware—HTML 5 provides access to a webcam via
getUserMedia. Binding to a <video> element is fairly simple, which is great, because that element can be the destination parameter for
Internet Explorer and Safari users: sorry, but your browser of choice is lagging behind Mozilla and Google.
getUserMedia was unavailable to you when I wrote this post.
Making Things a Bit More Interesting
Stitching video onto a flat surface is a good way to create a highly expensive, power-hungry mirror, so I’m going to give the vertex shader something to do: time-based mesh deformation. This will give the video a ripple effect. Hopefully one smoother and more colourful than this animated gif:
The version of Firefox that I’m using lets me run this straight off the disk. Chrome insists that it be provided by a web-server (
python -m SimpleHTTPServer to the rescue).
loadTexture function now refers to a <video> element, and uses the
getUserMedia API to bind it to the input from a camera. The
onplaying callback of the video element actually initialises the texture with the
texImage2D call (allocating storage on the GPU and coyping pixel data from the surface), and then sets up the call to
play() method is immediately called on the element in order to kick the whole process off.
render() method itself is much like it’s always been, with the exception of the call to
texSubImage2D, using the video element as a pixel store. When the underlying mesh is drawn with
drawArrays, the image is stitched over the surface just like any other.
Finally, the vertex shader does a little more than just translating between coordinate systems. It’s now generating a replacement z-index for each vertex that goes in, displacing it by a function of its distance from the centre of the view and the current time. There are some not-so-magic numbers in there: 0.05 is the amplitude, 5.0 is the frequency, and 3.0 is the speed. What’s worth noting here is that the GPU is doing the deforming. As far as the CPU is concerned, a flat mesh was uploaded to graphics memory once on initialisation and never updated. The only data being sent to the GPU thereafer are the new uniform values and the updated texture for each frame.
[side-note: It’s nice that all this work is being done by the GPU, but the trade-off is that the CPU can’t just look at its copy of the mesh to find out what you’ve selected if you click on something. It also complicates lighting, as the adjustments made by the vertex shader affec9t what the normals should be; these would have to be recalculated in shader, but the vertex shader doesn’t have visibility of the other vertices in its triangle. A geometry shader—which can see a whole polygon at a time—could calculate and feed appropriate values to the fragment shader, but WebGL doesn’t support those.]
Where did I get the time from? Well, the callback that you provide to
requestAnimationFrame gets a parameter when it’s actually called: a timestamp in milliseconds. This timestamp doesn’t correspond to the actual time in any way, it’s just the time since the rendering cycle started for your page. In any case, it’s usable as something that varies relatively smoothly and which can be fed into the vertex shader to indicate change on a frame-by-frame basis.
Window Refresh vs. Camera Capture Rates
The browser refresh rate is going to be synchronised with your display—typically about 60fps—but unless you’re working with a good camera, the video data itself is only going to be updated at 30fps or less. This means that the code above pushes twice as much texture data to the GPU as it has to. If you’ve been tracking performance, you might be thinking that this is why the numbers say that the CPU is still quite busy. I thought it was worth trying out by passing along a step counter to my render function. If
step % 2 == 0, upload the texture (i.e. only update the texture on every second frame). Any impact on performance wasn’t noticable. Trying mod-5 didn’t make a difference, either, so it doesn’t appear to be the OpenGL layer that’s chewing up time. Performance figures were similar between Firefox 40 and Chrome 43 on a 2011 Macbook Pro. I wonder what’s hogging the time?
If you don’t have access to a webcam, try replacing the
getUserMedia code in
loadTexture with something that plays a video in a loop.
Stereoscopic animated GIFs—a web search will yield many—give an impressive sense of 3D using just two images. Using an appropriate graphics package (I recommend GIMP), save their constituent images and replicate the effect by flipping between their images on the fly. [if you do this, it’s probably better not to update a single texture, but rather to initialise and upload two textures, flipping between which one you bind/render on render calls].
Create some HTML controls (e.g. <range> elements) for each of the amplitude, frequency, and speed. Add some uniforms to the vertex shader that allow these parameters to be manipulated on-the-fly.
The continuous ripple effect is pretty, but there’s fun in playing with it. How about a ripple that diminishes with the distance from the origin? What about a flag effect instead?
This series of blog posts was called ‘WebGL from Scratch’, and we’ve gone from drawing a red background through 2D shapes, 3D shapes, lighting, mesh loading, animation, and textures. That’s about as much as I want to accomplish in an introductory tutorial.
So is that all there is? Absolutely not. Graphics is an endlessly deep subject. Just scratching the surface, I haven’t talked about shadows, environment mapping, bump mapping, multi-texturing, or object picking. I haven’t played with any of the more difficult visual effects to model, such as fire, smoke, or glare. Lighting has been deliberately simple, and is usually extended with some kind of material parameters for specularity and reflection. Descriptions of multiple approaches to handle any one of these subjects are a web search away, in almost all cases trading off accuracy and efficiency.
I hope you’ve enjoyed it.
If you would like something explained in more depth, attach a note and I’ll put together a post that builds on this material. Or posts, if need be.