INB382 Real Time Rendering Techniques
Lecture 1: Introduction and Hardware
Background
Ross Brown
CRICOS No. 000213J
Queensland University of Technology
Lecture Contents
•
•
•
•
•
Subject Preamble
Hardware Rendering History
Present Hardware Pipeline
Geometric Primitives
My First Unity Project
a university for the
real world
R
CRICOS No. 000213J
2
Suggested Books
• No prescribed textbook
• Suggestions:
– Unity 4.x Cookbook - Matt Smith, Chico Queiroz
– Learning c# by Developing Games with Unity 3D,
Terry Norton
• Hardcore for Ninja Enthusiasts
– Real-time Rendering, Tomas Akenine-Moeller, Tim
Haines, Naty Hoffman, 3rd ed.
a university for the
real world
R
CRICOS No. 000213J
3
The Beginning of the Rendering Journey
• “…drawings are graphic accounts of essentially
verbal processes…” (Karl Buhler)
a university for the
real world
R
CRICOS No. 000213J
4
Who am I?
•
•
•
•
•
•
•
13 years at QUT teaching and
research in graphics, games and
virtual worlds
PhD in visual attention applied to
ray tracing
~21 years of computer graphics
teaching – undergraduate to PhD
level
First teaching system was my own
version of OpenGL 1.0 in 1995
(right bottom) on SGI Irix!
First taught this topic at QUT in
2003
Research use of virtual reality for
environmental knowledge
elicitation (right top – ABC)
Blog: www.bpmve.org
a university for the
real world
R
CRICOS No. 000213J
5
Goal – Total World Graphics Engine
Domination!
https://www.youtube.com/watch?v=pXWAsayTFTo
a university for the
real world
R
CRICOS No. 000213J
6
Subject Preamble - Context
• This unit is about shader-based rendering in
games engines
• Previously you had to do INB381 – I still
strongly recommend it
• We concentrate on surface detail by rasterising
this geometry and illuminating and shading the
visible pixels to give them colour
• So scenes are almost static, with minimal
animation, modelling and no game mechanics
a university for the
real world
R
CRICOS No. 000213J
7
Subject Preamble - Context
• MZB151 is assumed knowledge
• You have covered:
–
–
–
–
–
Transforms
Projections
Vector Algebra
Calculus
Matrix Operations
• MZB151 THEORY IS NOT REVISED
• You should know C#, I will not be teaching it at all
• While INB381 is not required, I recommend it to fill out other
areas in geometric modeling
a university for the
real world
R
CRICOS No. 000213J
8
Preamble - Purpose
• To teach advanced methods which are applicable to real
time rendering using state of the art software and
hardware
• To teach advanced rasterisation techniques to simulate
various observed visual phenomena which use a
programmable shading interface
• To teach you how to integrate high-end rendering
solutions into a modern games engine
• To make INB380 games look less similar to each other

a university for the
real world
R
CRICOS No. 000213J
9
Overview of Advanced Rendering
Component
• Theory and practice about various advanced
real-time rendering techniques will be delivered
• Techniques will be implemented in Unity 3D
environment with CG shaders
• You will use your OWN shaders, not the
included ones
• Models are up to you, assessment is general
• You may use content from INB379 and INB380
a university for the
real world
R
CRICOS No. 000213J
10
Assessment and Tutorials – INB382
• Two Assignments – General Virtual Worlds
– Assignment 1 – 20% - directional effects – lights
textures etc.
– Assignment 2 – 30% - global effects – shadows etc.
– Exam – 50% – programming of shaders and
descriptive theory
• Tutorial solutions appear after tutorials
• SVN Server – www.theblacksphere.com/INB382
a university for the
real world
R
CRICOS No. 000213J
11
Why Unity3D and CG?
• Because a large majority of people use it for Indie Games
• Good abstraction of shaders - less programming of supporting
code
• Works on DX11 and OpenGL
• Still allows for theoretical and practical implementation of all
the effects
• Good for your CV
• However, loss of lower level knowledge – cannot win them all!
• I am keen to run a Direct3D course for Honours students as a
reading unit – like the old form of INB382
a university for the
real world
R
CRICOS No. 000213J
12
Overview of Advanced Rendering
Component
• CG is a version of C++ modified for Shading purposes
• I will point out the differences, and guide you in the
usage of the shader languages
• Will introduce you to Unity 3D, but many of you will be
adept at using it
• All demonstrations will be in Unity 3D / CG
• You will in the large will have to pick the usage of Unity
3D for yourself. NB: I do not expect deep usage of Unity
for this unit, so it should not be a problem
• THIS IS A HARD UNIT – be prepared to work
a university for the
real world
R
CRICOS No. 000213J
13
Broad PC Architecture wrt Graphics
• Most application programming so far on CPU
• PCs are now a multiprocessor architectures:
– CPU – multicore (6 physical cores for i7)
– Cell – 1 x Power PC core, 7 x Synergistic Processing
Units (SPU)s
– GPU – Graphical Processing Units (Multi GPU)
– Haswell – CPU/GPU hybrid – Intel HD
• GPUs are now General Purpose GPUs
(GPGPU)
a university for the
real world
R
CRICOS No. 000213J
14
GPU Card Hardware Outline
http://www.techpowerup.com/?85999
a university for the
real world
R
CRICOS No. 000213J
15
Typical GPU Card Layout
a university for the
real world
R
CRICOS No. 000213J
16
PC/GPU Relationship – GeForce 6 [1]
CPU
6.4 GB/s
6.4 GB/s or
More
System DRAM
Up to 8 GB/s
North Bridge
GPU
To Display
Up to 35 GB/s
South Bridge
Graphics DRAM
Other Peripherals
a university for the
real world
R
CRICOS No. 000213J
17
PC/GPU Relationship – GeForce 6 [1]
• PCI Express bus (x16) - 8 GB/sec
• GPU memory interface - 35 GB/sec
• CPU memory interface (800 MHz) Front-Side
Bus – 6.4 GB/sec
• Note the differing rates – more on this later
• In general, must keep processing on GPU
a university for the
real world
R
CRICOS No. 000213J
18
Early PC GPUs
• 1st Gen (up to 1998) NVIDIA TNT2 & ATI Rage
– Triangle rasterisers, no geometry stage – done on CPU
• 2nd Gen (1999-2000) NVIDIA GeForce256 &
GeForce2
– Full geometry transform, lighting and rasterisation on the
card
• 3rd Gen (2001) NVIDIA GeForce3 and 4Ti
– Vertex shaders and register combiner fragment shaders
• 4th Gen NVIDIA Cine FX 5900 (2002)
– Fully programmable vertex and fragment shaders
a university for the
real world
R
CRICOS No. 000213J
19
Present GeForce - Maxwell
http://www.hardwarezone.com.sg/review-geforce-gtx-960-performance-review-nvidia-targets-sweet-spot-gfx-market-updated
a university for the
real world
R
CRICOS No. 000213J
20
Present GeForce
– Zoomed!
http://www.hardwarezone.com.sg/review-geforce-gtx-960performance-review-nvidia-targets-sweet-spot-gfx-marketupdated
a university for the
real world
R
CRICOS No. 000213J
21
Intel HD Devices (SOC)
http://wccftech.com/review/intel-haswell-core-i7-4770k-review-dz87kl75k-motherboard/
a university for the
real world
R
CRICOS No. 000213J
22
Present Generation – GeForce GTX Titan X
• Unified Shaders : Texture mapping unit : Render
Output unit – 3072:192:96
• 12288 Gig of texture memory
• Is a highly superscalar general purpose processor –
GPGPU
• Supports Direct X 12, OpenCL, CUDA, Shader 5.0
• ATI – same, same but different feel free to read in
own time
• See –
en.wikipedia.org/wiki/List_of_Nvidia_graphics_proce
ssing_units – valid ref as list
a university for the
real world
R
CRICOS No. 000213J
23
GPU Speed Increases Exceed Moore’s
Law – CPU vs GPU
http://docs.nvidia.com/cuda/cuda-c-programming-guide/
a university for the
real world
R
CRICOS No. 000213J
24
Graphics Pipeline – Programmatically
• While using Unity, still have to understand what is
happening underneath the games engine
• Present hardware provides access to internal hardware
rendering information to modify algorithms outside of
normal API calls from libraries such as Direct3D
• Programs are written in a shading language, compiled
and loaded onto the card
• Will base this component on the DirectX pipeline
• Problem is that you switch depending on the hardware
being used – Direct X PC, OpenGL everywhere else
• You will get the DX11 pipeline in GPS515
a university for the
real world
R
CRICOS No. 000213J
25
Rendering Pipeline – Unified Shader
Model
From: David Luebke and
Greg Humphreys. 2007.
How GPUs Work.
Computer 40, 2, 96-100.
•
•
•
Modern scanline rendering systems have a unified rendering shader
pipeline with shared resources for stages
Programmable Vertex, Geometry and Pixel shaders
Polygons are transformed, lit, textured, shaded and rasterised into the
framebuffer via these programs
a university for the
real world
R
CRICOS No. 000213J
26
Graphics Pipeline – DirectX
• These programs are
executed when a related
primitive is being rendered
• This process is derived
from the Renderman
model of pixel shading –
www.pixar.com
a university for the
real world
http://msdn.microsoft.com/enus/library/windows/desktop/bb205123(v=vs
.85).aspx
R
CRICOS No. 000213J
27
Graphics Pipeline Stages – Direct X
• Input-Assembler Stage - responsible for supplying data
(triangles, lines and points) to the pipeline
• Vertex-Shader Stage - processes vertices, typically
performing operations such as transformations, skinning, and
lighting
• Geometry-Shader Stage - input is a full primitive and
supports limited geometry amplification and de-amplification.
Given an input primitive, the Geometry Shader can discard
the primitive, or emit one or more new primitives
• Stream-Output Stage - designed for streaming primitive data
from the pipeline to memory on its way to the rasterizer
a university for the
real world
R
CRICOS No. 000213J
28
Graphics Pipeline Stages – Direct X
• Rasterizer Stage - responsible for clipping
primitives, preparing primitives for the pixel shader
and determining how to invoke pixel shaders
• Pixel-Shader Stage - receives interpolated data for
a primitive and generates per-pixel data such as
color
• Output-Merger Stage – responsible for combining
various types of output with the contents of the
render target and depth/stencil buffers to generate
the final pipeline result
a university for the
real world
R
CRICOS No. 000213J
29
How do we render 3D imagery?
Primitives are converted to pixels via
process of Rasterisation in GPU
pipeline.
3D objects in games are
constructed of polygon meshes,
formed from 3D geometric
primitives.
So far you have let game engines like
Unity control this process in abstract
fashion.
Primitives are atomic
geometric constructs combined
to form complex shapes.
a university for the
real world
R
CRICOS No. 000213J
30
Graphics Primitives - Points
• Modelled as a 3D coordinate in world
coordinates (x, y, z)
• Transformed and projected as with other
primitives
• Colour pixel at position x, y in the final viewport
(x, y, z) – object coords
a university for the
real world
(x,y) – screen coords
R
CRICOS No. 000213J
31
Graphics Primitives - Lines
• Lines should
– appear straight
– terminate accurately
– have constant density
– have density which is independent of
line length
– be drawn rapidly
What the line should look like
Failure to join correctly
Termination incorrect
Uneven line density
How a poor line generating
algorithm would look
a university for the
real world
R
CRICOS No. 000213J
32
Triangles
• Process can be simplified for a triangle
• Triangles are the primitive used in GPU surface
reps.
• Set number of vertices: 3
• Set number of edges: 3
• Same basic process as with other fill area
objects
• Use edge coherence to generate edge pixels
• Then fill the scanline with your colour(s)
a university for the
real world
R
CRICOS No. 000213J
33
Triangle Scan Line Conversion
(Rasterisation)
Void ScanConvTri(Triangle T, Color rgba)
{
for each edge pair
{
initialize XL, XR;
compute dxL / dyR and dxR / dyR;
for each scanline at y
for (int x = xL; x <= xR; x++)
SetPixel(x, y, rgba);
xL += dxL / dyL;
xR += dxR / dyR;
}
}
http://www.cs.princeton.edu/courses/archive/fall99/cs426/lectures/scan/sld022.htm
a university for the
real world
R
CRICOS No. 000213J
34
Polygon Mesh
• Set of connected planar polygons
– true polygons
V5
V6
V3
V2
V4
V1
• the figure surfaces are actually polygons
– approximations of curved surfaces
• Consists of a collection of vertices,
edges and polygons
E9
E7
E8
– vertex is shared by at least 2 edges
E2
E1
E3
E4
– each edge connects 2 vertices
– an edge is shared by (at most) two
polygons
E6
E5
P4
P3
P5
P2
P1
a university for the
real world
R
CRICOS No. 000213J
35
Polygon Mesh Polygon/Edge/Vertex List
• Vertex list is constructed as previously
described
• Edge list
• each edge contains
– pointers to 2 vertices (edge endpoints)
– pointers to 2 polygons (that share the edge)
•
•
•
•
•
•
E1 = (V1,V2,P1,..)
E2 = (V2,V3,P1,..)
E3 = (V3,V1,P1,P2)
E4 = (V4,V3,P1,..)
E5 = (V4,V1,P1,..)
E6 = (V4,V2,P3,P4)
a university for the
real world
V2
V3
P3
E2
E1
E6
E4
P1
E3
P2 E V4
5
P4
V1
R
CRICOS No. 000213J
36
Polygon Mesh Polygon/Edge/Vertex List
• Polygon is represented as a set of pointers
into edge list
•
•
•
•
P1 = (E1,E2,E3)
P2 = (E3,E4,E5)
P3 = (E2,E4,E6)
P4 = (E1,E5,E6)
V3
P3
V2
E2
E6
E4
P1
E1
E3
P2
E5
V4
P4
V1
a university for the
real world
R
CRICOS No. 000213J
37
Polygon/Edge/Vertex List
• Used to display all edges
– each edge is only drawn once
• Transformations performed on vertex list
• Information on shared edges is readily
available
a university for the
real world
R
CRICOS No. 000213J
38
Example-Unit Cube
V1 - (0,0,0)
V2 - (1,0,0)
V3 - (1,0,1)
V4 - (0,0,1)
V5 - (0,1,0)
V6 - (1,1,0)
V7 - (1,1,1)
V8 - (0,1,1)
Polyhedron
V8
V7
V4
V3
V5
V6
V1
EDGE LIST
V2
Back in/Front out
Front in/Back in
E11
E6
E7
E3
E2
E10
E5
E12
E4
E9
E8
E1
Wire Frame
a university for the
VERTEX LIST
E1 - V1,V2
E2 - V4,V1
E3 - V3,V4
E4 - V2,V3
E5 - V1,V5
E6 - V4,V8
E7 - V3,V7
E8 - V2,V6
E9 - V6,V5
E10 - V5,V8
E11 - V8,V7
E12 - V7,V6
Back out/Front out
real world
R
CRICOS No. 000213J
39
Polygon Lists For Unit Cube
List 1
Polyhedron
V8
V7
V4
V3
V5
V6
V1
P1 - E1,E2,E3,E4
P2 - E9,E12,E11,E10
P3 - E2,E5,E10,E9
P4 - E6,E3,E7,E11
P5 - E7,E4,E8,E12
P6 - E8,E1,E5,E9
List 3
V2
List 2
P2 - E9,E12,E11,E10
P3 - E2,E5,E10,E9
P4 - E6,E3,E7,E11
P5 - E7,E4,E8,E12
P6 - E8,E1,E5,E9
Empty list
List 4
Front in/Back in
E11
E6
E7
E3
E2
E12
E10
E5
E4
E9
P3 - E2,E5,E10,E9
P4 - E6,E3,E7,E11
P5 - E7,E4,E8,E12
P6 - E8,E1,E5,E9
E8
E1
Back out/Front out
Wire Frame
a university for the
Back in/Front out
real world
R
CRICOS No. 000213J
40
Mesh as List of Triangles
Each of the following steps occurs for each mesh triangle
(example triangle highlighted in red)
a university for the
real world
R
CRICOS No. 000213J
41
Pipeline Shader Example – Step 1
Application
Space
PCIe Bus
V1
V3
GPU Vertex
Stage
V1
V3
V2
V2
DrawSubset(…)
• Commands are issued from the CPU and stored in a shared location
• Vertices are fetched from parsed CPU command stream using the
Input-Assembler Stage
a university for the
real world
R
CRICOS No. 000213J
42
Pipeline Shader Example – Step 2
World
Coordinates
Homogeneous
Clip Space
V1
Vertex
Stage
V3
V1
V3
V2
V2
Vertex Shader
OUT.position = mul(worldViewProj, v );
OUT.color0 = float4(1.0, 0.0, 0.0);
• Vertices transformed, lit and projected by shader
• Emerge in Clip Space
a university for the
real world
R
CRICOS No. 000213J
43
Pipeline Shader Example – Step 3
Homogeneous
Clip Space
Homogeneous
Clip Space
V1
V3
Geometry
Stage
V1
V3
V2
V2
Geometry Shader
triStream.Append(gOut);
triStream.RestartStrip();
• Processes entire primitives, emitting new primitives if needed into
clip space
a university for the
real world
R
CRICOS No. 000213J
44
Pipeline Shader Example – Step 4
Projection
Space
Screen
Coordinates
V1
V3
Pixel
Stage
V2
Pixel Shader
OUT.color = IN.color;
•
•
•
Triangle rasterised into fragments (possible pixels), modified by shader
Colours and texture values are interpolated automatically
Fragments are in screen coordinates
a university for the
real world
R
CRICOS No. 000213J
45
Pipeline Shader Example – Step 5
Screen
Coordinates
Framebuffer
Output-Merger
• Pixels (fragments) are integrated into framebuffer via raster
operations
a university for the
real world
R
CRICOS No. 000213J
46
Shader Programming - An Introduction
• CG is a high level version of C for implementing the
previous vertex and fragment shaders
• CG is data flow based – computation occurs in
response to data that flows through a sequence of
processing steps [3] – not imperative as you have so far
learned
• Also known as a kernel approach - kernel function is
applied to data stream
• Shaders are applied to vertices and fragments
• Each time a vertex or a fragment (potential pixel) is
processed by the card, then the programs are executed
a university for the
real world
R
CRICOS No. 000213J
47
Unity Preliminaries – Unity Technologies
• Unity 3D version 4
• Built in C++ but exposes
functionality in C# via the
Mono environment
• Allows for programmable
shaders using CG
• Lots of other features for game
dev.
• Is cross platform, and allows
for desktop, console and
mobile targets
• Fully functional free licence
a university for the
real world
R
CRICOS No. 000213J
48
First Unity Project Demo
•
•
•
•
•
•
•
Project Creation
Object Importation
Material Creation
Shader Creation
Script Creation
Built-in Shaders
Video link below
https://www.youtube.com/watch?v=bDx81zT36e8&list=UUkXgxotjrq5E4twVyZNMPxg
a university for the
real world
R
CRICOS No. 000213J
49
C# Script Data Flow
• Variable are set to appropriate values for shader
• Then passed via the renderer object to the shader
• This is how data is sent to rendering shaders
Texture2D tex1 = (Texture2D)Resources.Load(
"oildrum_col", typeof(Texture2D));
renderer.material.SetTexture("_Tex1", tex1);
a university for the
real world
R
CRICOS No. 000213J
50
Shader ”ShaderName"
{
Properties // Variables from scripts and IDE interface
{…}
Unity Shader
Structure
SubShader // List of shaders for hardware profiles
{
Pass // Each shader divides into passes
{
CGPROGRAM // Shader proper begins here
#pragma // Assorted preprocessing commands
• Is a modified version
of a CG shader
• Unity performs preprocessing to align
with hardware
profiles
• So this differs to
standard CG and
HLSL shader files
// Declare uniform variables and structs here
VSOutput VERTEXSHADER(…) // Vertex Shader
{…}
float4 PIXELSHADER (…) : COLOR // Pixel Shader
{…}
ENDCG
}
}
}
a university for the
real world
R
CRICOS No. 000213J
51
Shader – Data Flow
Properties
{
_Tex1("Base (RGB)", 2D) = "white" {}
_Tex2("Base (RGB)", 2D) = "white" {}
}
…
sampler2D _Tex1; // Aligned
variable
a university for the
real world
• Remember the _Tex1 in the
C# script?
• Properties section signifies
data to be passed to the
shader
• In this case texture
information
• Which is passed through
the shader file from the
script
• Must be an aligned variable
in the file
R
CRICOS No. 000213J
52
Shader Data Flow
struct VSInput
{
float4 pos: POSITION;
float4 tang: TANGENT;
float3 nor: NORMAL;
float2 tex: TEXCOORD0;
};
struct VSOutput
{
float4 pos : SV_POSITION;
float2 tex : TEXCOORD0;
float3 lightDir : TEXCOORD1;
float3 posWorld : TEXCOORD2;
};
a university for the
real world
• The Semantic terms
POSITION (amongst others)
refer to registers that receive
the data from the vertex/edge
stream generated by UNITY
3D
• Eg. POSITION is vertex
register zero
• Important – this is varying
data, passed in as a stream
• Most of this is handled by
Unity, but should be aware of
caveats
R
CRICOS No. 000213J
53
Vertex Shader
// normal mapping vertex shader
•
VSOutput VS_NormalMapping(
VSInput a_Input)
•
{
…
// calculate homogenous position
output.pos = mul(UNITY_MATRIX_MVP,
a_Input.pos);
…
return output;
}
a university for the
real world
Multiplies the position by World
to Projection matrix
Copies the vertex position
through to the pixel shader
R
CRICOS No. 000213J
54
Pixel Shader
• Pixel shader is executed
on the fragments
generated by the
rasterisation stage
• It applies a function to the
fragment
• Then outputs a value into
the framebuffer as a pixel
• NB Input is output of
vertex shader
float4 PS_NormalMapping(
VSOutput a_Input) : COLOR
{
// index into textures
float4 colour = tex2D(_Tex1,
a_Input.tex);
…
return colour;
}
a university for the
real world
R
CRICOS No. 000213J
55
Tutorials
• Let’s talk about tutorial allocations
–
–
–
–
Tuesday 4pm-6pm – limit of 23 – ~size of enrolment
Wed – 12pm-2pm
Thurs – 9am-11am
need at least ten in others to make viable
• Start in Week 2 in GPS515 – two hours – one
theory and one practice
a university for the
real world
R
CRICOS No. 000213J
56
References
1. Pharr (ed.) GPU Gems 2, Addison Wesley,
New York, 2005
2. Foley et al. Computer Graphics: Principles and
Practice, Addison Wesley, 1992
3. Fernando, R and Kilgard, M. The CG Tutorial,
Addison Wesley, New York, 2003
a university for the
real world
R
CRICOS No. 000213J
57