Tác giả giữ toàn bộ bản quyền (All Rights Reserved) — không phải giấy phép mở. Bạn vẫn xem và học miễn phí trên OpenER; muốn tái sử dụng cần xin phép tác giả.
PROFESSOR 1: And action. PROFESSOR 2: We can see James. Yeah. Yeah, James is dark.
PROFESSOR 1: Oh, he's a little better. He's a little better. It'll get better when we get — PROFESSOR 2: --looking better. Yeah.
PROFESSOR 1: Okay. All right. All right, ladies and gentlemen. Welcome to the penultimate lecture, a guest lecture by our awesome host James Percy from Apple and GPU Architecture.
Woo! [LAUGHTER] So happy! All right, I'm going to stop my camera and make sure to add James's camera here. Add his spotlight.
Boom. PROFESSOR 2: Add James's spotlight. Yeah. PROFESSOR 1: Take mine out?
All right, James. You're up. Thank you for that R&M comment. James, you're up.
Welcome. Thank you so much for coming to share us how GPUs, work what they are, and actually how to program them.
That's the really exciting thing I think students are waiting to see, so good stuff. Let's rock and roll.
JAMES PERCY
Great! Thanks for that intro, Dan. So, yeah, we'll be talking about GPU architecture today. This is the presentation that myself and two of my colleagues, John Coers and Harald Obermaier, have put together.
Before we get started, I do have one background thing to cover.
I know we are recording this for the purposes of Berkeley, but for the class I just want to remind everybody else please refrain from recording, posting, or streaming any slides or taking pictures due to confidentiality purposes.
So now we've got that out of the way, and I should be better lighting now. So, what are we going to talk about today?
So I wanted to give a little bit of an overview of what is a GPU and what do they do for us.
We'll get into a little bit of details about the graphics pipeline at a high level, and we'll cover some of the major stages and talk a little bit about the programmability inside a GPU and why that programmability is powerful.
And then we'll walk through a programming example, and hopefully I'll try to run a live demo on my Mac and show you guys the power of a GPU when it comes to parallel programming and what we can do so.
So ask questions along the way. Write into the chat box. Dan is going to help monitor that for me, and we'll also try to leave some time at the end to answer questions.
First, what is a GPU? Well, a good way to answer that question is actually to compare it to a CPU.
Many of you, I believe, are familiar with what CPU architectures look like, and, frankly, CPU architectures are a little bit easier to think about.
I mean, they are very complicated in their own right for very good reasons, and GPUs are complicated as well but for somewhat different reasons.
And so what I have here is obviously a very stylized version of a CPU on the left and a GPU on the right with very similar components that are color coded.
At a really high level, you can think about — and hopefully you can see my cursor here.
The way the CPU works is there's a whole bunch of control logic to decode and execute instructions.
There's execution units that actually do the math and run those instruction, and it's backed by memory, usually a hierarchical level of caches and then a memory system.
The general idea is to get better performance out of a CPU you want to minimize your latency through all of these different steps.
In fact, modern CPUs — the number of cycles of latency that you measure from these execution units out to your memory is a critical piece of your performance.
It's not the only thing that determines your performance, but it is a very important piece.
GPUs have similar concerns, and you can kind of see the way that these boxes are stylized is you have many, many execution units.
You have smaller control units that are perhaps simpler, but you have more of them controlling all of these execution units, and then you also have these little caches — that's what these kind of yellowish things are — associated with each control unit.
And they're all backed by the same DRAM. The key point here — and we'll get into some of the details of what this actually means.
But the key point here is that these blue squares are multiplied many, many times.
So when you look at a CPU complex, you might have two, three, or four, maybe even up to eight CPU cores on a given chip or SSD.
On a modern large GPU, you could easily go up to well over 100, and that's the key difference, and that's what gives us the big parallel programming boost we get from a GPU.
So in order to get the maximum performance from a GPU, what your goal really is is to maximize your throughput through all of these blue execution units, or, in other words, you've got all these parallel processing units.
You want to optimize for using as many of them as you can, and what we'll talk about as we go through this presentation is that maps very well to moving pixels on the screen.
You can kind of imagine if you've got a whole bunch of groups of pixels, you can distribute them across this array of execution units, and they can somewhat get executed in parallel.
So that's kind of what we're going to talk about today.
So this slide is this kind of a breakdown of what I just talked about and some of the care-abouts as we're trying to optimize for CPU and GPU performance.
I talked about the number of cores on a CPU. We typically have more on a GPU.
Frequency we typically care more about on the CPU, not so much these days, but there is the frequency wars back in the late '90s and early 2000s where all we care about for CPU performance was just pushing max frequency.
GPUs are much less about pushing maximum frequency and just pushing maximum throughput, which is what we talked about on the previous slide I'm not going to go through every line on this slide, but, to hit on a couple of points, speculation and execution order are important, particularly if you've covered CPU architecture.
The speculation and execution order on the GPU tends to be simpler than on a CPU.
On a CPU there's an awful lot of logic and area spent around how do I do branch prediction, how do I reorder instructions to maximize performance.
GPUs typically don't do that simply because doing those kind of optimizations across the number of execution units that we showed in the previous diagram is very complex and doesn't scale very well.
And so that's not something that we try to optimize on optimize for on GPUs, and what that nets out to is execution control tends to be simpler.
And then the last thing I'll mention here is coherency.
Kind of likewise on CPUs, especially when you have a multi-core CPU complex, there's a lot of complexity and effort spent around managing coherency across multiple CPU threads, and for good reasons.
You want to have relatively simple programming model.
For the GPU, where we have a more complicated software programming model — and I'll show some examples of that in the bit — coherency tends to be software-managed.
Meaning if you have two different threads that need to communicate with each other, it's up to the programmer or the developer to manage that synchronization to ensure a data consistency.
To summarize — PROFESSOR 1: James, there's a really funny — a really nice analogy that Brian brings up, which is he hears analogy for a CPU is like a math professor and a GPU is like a classroom of elementary school students.
JAMES PERCY
[LAUGHS] PROFESSOR 1: And you're giving them both math problems, but you want to give the complex problems to the professor and many easy problems to the students. What do you think about that analogy?
I like it, Brian.
JAMES PERCY
It's an interesting analogy.
I think if your goal at the end of the day was to get as many simple math problems complete in a shortest time as possible, then that's an excellent analogy because what you really want is a bunch of elementary students that can just crank away in parallel at a bunch of simple problems.
PROFESSOR 1: Simple problems. Exactly. Yeah.
JAMES PERCY
Even the smartest math professor is not going to be able to keep up with 100 smart students if you're just cranking through basic math problems.
From that perspective, I think that's a good way of thinking about it. PROFESSOR 1: Love it. Love it. Great!
Thanks.
JAMES PERCY
So maximizing throughput and hiding latency. We've talked about throughput. We want to use these execution units that we have as much as possible.
And so many of you may have heard about something called SIMD. SIMD is a very important concept in parallel programming. It stands for Single Instruction Multiple Destination, or Multiple Data.
What that means is that you have one set of instructions that you're going to execute across a large data set.
So every element — for example, let's say you had a screen with a bunch of pixels on it.
Every pixel on your screen is going to execute the same instructions, and what that gives you is what we call thread-level parallelism.
Typically, you'll hear about threads, work groups, warps, or things like that, and generally the work that you put into a thread is all going to be executing the same instruction.
And the work of the GPU and the complexity of the GPU is managing that.
So how do I schedule work across all these different execution units such that I get the maximum throughput across all of the data that I have and all the different sets of instructions that I need to schedule?
So that's really what this is about, and there's a lot of complexity about how do these things access memory, how do you — and this kind of goes back coherency.
How do you schedule them, how do they talk to each other, and some of that complexity falls on the programmer and the driver, and some of it falls on the hardware.
One other thing to mention here that's important is this memory bandwidth aspect of it, and memory bandwidth we'll talk about in the context of pixels, games, and images.
But typically GPUs have access to very high memory bandwidth, more memory bandwidth that you would have on the GPU. Sorry. On the CPU.
And the reason for that is if you just imagine a game — if you're playing a game and you need to refresh your screen 30 times a second for your game to look good, just right there you need to refresh your screen.
You need to touch theoretically every pixel on the screen, but on top of that you need to render a bunch of stuff behind it, and I'll show you what that rendering looks like.
You're moving data back and forth to generate these images multiple times a second, and so you need to have high memory bandwidth to do that efficiently.
And so a lot of the GPU challenges and optimizations are around reducing that memory bandwidth — how do we move less data, how do we have cache hierarchies to reduce the memory that's moving from on-chip to off-chip, or the data I should say moving from on-chip to off-chip.
So those are kind of the basic concepts. Let's talk a little bit about what the pipeline looks like and how we apply some of these concepts.
At a very high level, a GPU consists of really four or five major sections.
There's the vertex processing section, where we are processing vertices and we're doing some computation on them, and I'll explain each of these in a little bit more detail.
There's rasterization, where we're converting triangles into individual pixels that show up on the screen.
There's fragment processing, where we can actually run programs on individual pixels to generate their colors, and then there's the frame buffer section where we actually output the pixels to a buffer in memory so they can be displayed on screen.
One detail I glossed over — I mentioned it in the context of fragment processing, but for vertex process and you can also run individual programs on each vertex.
The modern GPUs actually use the same hardware — that's where you see this unified shader core on the left here in green — to run these programs.
And that really comes back to the parallel programming model that I talked about earlier.
In order to process millions of pixels a second and potentially millions of vertices per second, you need to have a large computation engine that can run these programs.
And so what you see kind of in our stylized example over here on the right — these individual pixels, these red guys, blue guys, and green guys, for example.
Each of these could be running a program to generate these final pixels and write them out to memory.
So one way to think about that is if you're playing a game and you're seeing pixels on your screen, every pixel in your screen is running a program.
Now, they might be running the same program, what's called a pixel shader, but they're all running a program. And so that's kind of the beauty of GPU parallel processing.
And at the end of the day — and we'll go through the kind of examples of what builds up this frame — you get an image that looks something like this.
You have objects, which is this kind of shiny unicorn thing in the middle, you have a background with the fireworks on it, you have these objects in the background with the Ms on them, which stands for "Metal."
These are textures. I'll explain what textures are. You've got this surface that's kind of reflecting.
Those are based off of textures and lighting models, and so we'll talk about some of the details of what goes into making a frame like this.
So let's get into the pipeline in a little bit more depth.
Typically, if you're making a game, you've got a model of your world, or you at least have objects that you have models that are made up of triangles.
And so somebody is going to program their game in a graphics API such as Metal.
That's what we use on Apple devices, and they're basically going to input their models or where they're vertices are into the game and give them some sort of description of how they connect, how they turn into triangles.
So we're going to run our vertex processing algorithm that we talked about to turn these individual vertices into triangles. That's called assembly.
You'll notice that we have this kind of world cube or shape here, where we've actually clipped off a piece of this triangle that does not get rendered.
So we go through a process of clipping pieces of the model that we don't see.
We rasterize or scan convert them, and you can imagine a process where if you have a given triangle you literally kind of walk left to right on each pixel on the screen and say is this inside my triangle or not.
And if it is, then you render it. If not, you throw it away.
That's not a great optimized algorithm for doing it, but it's an algorithm that will work, and you can kind of conceptualize how you might turn a triangle into individual pixels.
Then we get into the fragment processing, where we do lighting algorithms. For every pixel inside a triangle, we're going to compute a lighting equation.
We may texture it, and I'll explain that is meaning pulling textures from memory. You can compute effects.
You can do basically whatever you want to do to generate the final pixels, and then we send them off to our screen. So let's dive into vertex processing a little bit.
So here we have the same scene that we looked at a minute ago, and this has been drawn in something that's called wireframe in the bottom right here.
You can see the unicorn, you can see the shapes, and if you look closely, you can see that everything is broken into triangles, even these kind of square-like shapes in the background here.
If you look closely, you can see the diagonal, and it's been broken into a triangle. The reason for that is triangles are planar.
They're a lot easier for the hardware to work with rather than a true 3D shape.
And so the first thing we need to do is we need to get our unicorn into the right space in the world.
And so you can do that with the vertex shader by doing a number of transforms on the unicorn. You can rotate it. You can translate it, meaning move it.
You can stretch it in the x, the y, or the z dimension. That's basically what the vertex shader is going to do. It's also going to place a camera in the world.
So you can imagine if we were rendering this in a game, this camera might be like moving around relative to where the unicorn is, or the unicorn might be moving around relative to the camera.
It doesn't really matter. The vertex shader is going to get all the objects in the world by doing these transforms into a specific place so that we can go render it from the screen.
PROFESSOR 1: And James, just to connect this with what the students are doing now, they're working on a faster matrix multiply. That's exactly what's happening here.
The transformation matrix is a 4-by-4 matrix, and the pixel matrix is a xyzw by however many vertices you've got. So basically it's that hugely tall matrix times a 4 by 4.
That's all this is doing. It's exactly what the GPU was meant for. Before we had general purpose GPUs, we had GPUs that just knew how to do this really fast.
JAMES PERCY
Yeah. PROFESSOR 1: Take those points and then transform it into the right space. Really, that's exactly what they were optimized for.
JAMES PERCY
Exactly, and all — PROFESSOR 2: And — JAMES PERCY: --transforms that you guys probably know can be represented as a 4-by-4 matrix, and you can actually pre multiply these matrices together, which is what you would do in a real vertex shader to do these computations.
Sorry, [INAUDIBLE]. I think you had a comment. PROFESSOR 2: No problem. Our students, fortunately or unfortunately, do not get to program a GPU.
This time around, they just got access to Intel ADX extensions.
JAMES PERCY
Got it. Back in the good old days when I learned this stuff, GPUs actually had what we called fixed-function hardware to do these transforms.
So what that would mean is you would program more or less a matrix.
It wasn't quite described that way, but you're basically programmed what is my rotation transformation, what is my translation transformation into the GPU, and you would send it in vertices, and it would do the transforms for you, and spit out triangles out the backside.
That's basically what would happen. These days, instead of having that fixed-function hardware to do these specific multiplications, these things are expressed in what we call the vertex shader.
Meaning you can program in whatever transforms that you want and whatever makes them more or less whatever matrices that you want, as long as they're not nonsensical to do these transforms.
All right. So let's talk a little bit about geometry, and I'm going to try to pick up the pace a little bit here. Geometry processing. So we've taken on triangles.
We've now assembled them. We do primitive assembly into triangles, so we're not really dealing with individual vertices anymore. We're dealing with triangles, and this is a little bit of a detail.
I'm going to kind of go over it somewhat quickly, but one thing that many — not all, but many modern GPUs will do is they will actually take the triangles in screen space — that's what's shown on the left here — and then actually bin them.
And that's kind of what this is intended to represent on the right. And so what this picture on the right represents is the lighter colors, or the whiter colors represent triangle complexity.
So what you can see here is where we mapped to the unicorns hair. There's a lot of little triangles in here to do the hair.
Same thing with the tail, and so you can get kind of an approximation for how expensive this piece of the scene is to render just based purely on the number of triangles that are there, whereas if you go into the background there are zero triangles, so it's black, or on these two surfaces that we show here there aren't very many triangles.
So they're gray or close to black. That's important for a number of reasons.
It lets you estimate how expensive it is to render that piece of the scene, and that can help you with your work distribution, going back to the topic we had beforehand where it's all about optimizing the throughput of the machine.
We can use that to estimate this piece of the scene is going to be expensive, so we want to be able to distribute this to the scene to a lot of our execution units.
So that's something that not all GPUs operate this way, but many modern GPUs do. Okay. Rasterization.
We're at the step now where we've got our triangles, and we're going to go do a process called scan conversion. Scan conversion is kind of — I mentioned this earlier.
It's converting the triangles into individual pixels, and so you can imagine an algorithm where perhaps I start at this vertex.
I walk along an edge until I hit another edge, and then I walk over a little bit, and I walk back.
That's basically scan conversion, where I've taken this triangle and I've turned it into individual pixels.
I'm going to skip over anti-aliasing for the purposes of this presentation, but you can imagine actually along these edges you could actually create even more samples to make higher quality images if you wanted to.
So once I got my pixels figured out, now I need to go figure out how am I going to color them, and that's a step called fragment processing.
We've now got these triangles in the screen space. We also carry depth information along with each pixel, and we can do something called a depth test.
The depth test basically tells you is my triangle visible or not. So if you've got two triangles like this that's shown on a screen, they intersect.
Let's say this one to left is in the back.
You don't need to render the part of that triangle mostly if it doesn't show up on the screen, so we could just throw it away and only render the triangle on top.
So that's what the depth test is, and then the other piece of the scene may just be unwritten.
So we might just store a bit for each tile, or really it just means each pixel on the screen that says this is the clear color. We're not going to touch it.
We don't have to do anything with that particular pixel. So now comes the really interesting and exciting part, which is the shading.
So we talked about this common unified shader core that executes vertex fragment, and we didn't talk about compute, but we will in a little bit shaders on the hardware.
And the shader is what lets you specify that per-pixel algorithm that you're going to run to do a lighting calculation.
Just to kind of give some insight on what that might look like is there's a teapot example here. On the left, you see it's just very simply black or white.
There's no interesting shading to it. Basically, it looks pretty simple. On the right-hand side is the same teapot, but it's been shaded with some specular lighting.
What that means is you can kind of see the light reflecting off the top surface of the teapot, depending on where the angle of the light is coming from.
That gives it some interesting shading effects, and you can kind of see it on the handle, on the top, and then over here on the spout a little bit.
So this is a relatively simple lighting algorithm, but it does require a fair amount of computation. You have to compute some angles.
You need to do a texture look-up, and I'll explain what a texture look-up is in second to go compute this.
And you need to do this for every pixel on the screen, so it's not a cheap calculation, and that comes back to the fact that the SIMD algorithm we talked about beforehand — for every pixel on this teapot, we're going to go run this program down here, and we can do that in parallel so that it happens very quickly
as opposed to if you were doing just a pixel at a time on a CPU would take much longer. And that's where the beauty of having dedicated GPU hardware comes in.
So this particular shading program that I've got as an example down here on the right is written in Metal, which is again a programming language for Apple hardware.
You can see it's computing a light angle based off of a normal. It's doing a dot product.
It's doing a texture lookup to figure out the diffuse color, and then it's multiplying the diffuse color by the computed angle of the light, and that's what it's outputting.
That's basically how we're going to shade every pixel of this teapot.
Modern training languages are much more complex. Originally, we could just do kind of simple math operations and texture look-ups.
They've gotten more complicated over the years to get better compute operations that matrix multiplies, special instructions for doing neural nets.
Anything kind of related to parallel processing is kind of now a part of modern GPU shading languages, and I mentioned kind of coherency earlier.
There's also instructions for doing synchronization, so you can wait for other threads to finish so you can synchronize data. There's some limited amount of predication.
You don't want to be putting a ton of if/then else statements inside your shader program. That's not going to be good for performance, but there is some support for predicated instructions.
So this is what we're going to come back to at the end, and I'll hopefully show you a live demo of a program running both on the CPU and on the GPU.
To talk a little bit more about the execution model — so I talked about how we're going to execute this fragment shader on every touched pixel, so every pixel on this teapot.
And we're going to do that in a way that takes advantage of thread-level parallelism, meaning for every pixel we're going to run the same destruction on multiple data.
So we come back to our concept of SIMD. So if we're going to do this, imagine that this kind of square that we have is a zoomed-in piece of this teapot.
You could imagine an algorithm where we just walked one at a time left to right for each pixel inside this square we run the program.
And that would work, but it's obviously very slow because it does everything one at a time. And so the idea of a SIMD machine is that you have groups that work in lockstep.
Excuse me.
So you might just as a simple example take groups of 4, a 2-by-2 sub-square inside this big square, and you're going to execute the same destruction on all four of those pixels at the same time.
And then in a simple case where you're just executing simple instructions, they'll just kind of all execute 1, 2, 3, 4 in lockstep and finish at the same time.
Now, you can get into more complicated examples — and I'm going to gloss over this a little bit — where you might have one of those pixels has to run a high-latency texture effects or latency operation.
And what that means is you have to stall all four of those pixels. So you kind of take them off.
You stall them — that's what's intended to show in this diagram — and you go find a different set of work to go run.
So you might find a different thread group that you can go execute can make progress while this high-latency operation is running.
That's where a lot of the complexity and, frankly, the interesting parts of the GPU comes in because you're managing these thousands of threads that are running within the machine and trying to optimally execute them in a way that fills the machine in the best manner.
And that's a really hard problem.
I'm going to talk about texture mapping briefly so I have time to get to the demo. I kind of mentioned at the beginning in the background of our pony we have these fireworks.
We also have these Ms that were showing up on these surfaces. Well, this background is not generated programmatically or on the fly as part of the rendering algorithm.
These are pre-stored images or textures, and what you can do is when you have pre-stored images like this like these fireworks you can map them into your rendered image.
And so you can imagine there's just a big triangle or — sorry.
A big square back here, and what you would do is map each set of pixels in your world into this texture and go fetch them.
That's basically what the texture unit or what texture mapping does — it maps on-screen pixels to pixels that are inside a texture. Now, that can actually create some very interesting and challenging sample problems.
You could imagine if this image here were very small compared to the size that you wanted to render in on screen, you've got a problem where how do I sample the right number of pixels for this to show up.
Likewise, you can have the opposite problem, where you have a very large image that you're sampling into a very small area on screen.
So there's a lot of hardware built into texture units to try to do a good job of sampling. And this is the kind of example that's showing that.
You're taking a square kind of checkerboard pattern, and then you're mapping it to something that is tall and skinny, and you can see the results of doing that mapping is your nice square image is now a number of rectangles.
That's basically what the texture hardware is going to do. This is kind of more advanced. It's called anisotropic filtering filter mode that's shown on the right-hand side.
That kind of avoids some of these weird artifacts that you see on the top screen, but that's probably a bit beyond today's presentation. PROFESSOR 1: That's the anti-aliasing we mentioned before.
JAMES PERCY
Yes, that is a version of anti-aliasing.
Okay, so our last stage. We need to actually get all these pixels that we worked on and get them written out to memory. So there's usually an instruction, not necessarily exposed to the developer.
But at some point you've got all these pixels usually in a cache on your chip.
You need to actually write them out to the framebuffer, and then tell the framebuffer or tell the display usually my rendering is done.
Go pick up all this data and go put it on display. That's kind of what the output stage is. There's some other steps in there. There's blend tests.
There's some stencil tests that I kind of glossed over a little bit, but that's really the gist of it.
What you can see on the bottom right-hand side of this picture is this is kind of a framebuffer that's been captured like mid-render.
So some of the tiles or some of the pixels have completed, and you can see them on the screen. Others have not.
So you basically need to wait until all of the work is completed before you say that you're ready for a display.
All right. So let's get to our example. I'm trying to leave about 10 minutes for questions at the end.
What I want to show you guys is a SAXPY algorithm run on a CPU and the GPU, and actually it's even simpler than that.
We're simply going to implement a function in C that takes an array x of floats, multiplies each element in that array by a float a, and outputs it to an output array y.
And so we're just going to loop through the entire array and do that. Super simple, and we're not even doing the plus part. So we're just doing the multiply part.
We're going to do that in C on a single-threaded CPU, and we're going to do a similar thing on the GPU in Metal.
This kind of code snippet is maybe less familiar to you guys than the C program, but it essentially does the same thing.
I've got an array x, an output buffer y, and I'm going to multiply them by a. So it does the same function, just a different language. So what does that look like?
And I'm going to try to do this interactively in a second.
Basically, the results I'm showing you here are going through a number of different buffer sizes, and so it's really i squared. So you can see the buffer sizes here.
In the next column is the time it took for the GPU to finish the computation, and then the last column is the time that it takes the CPU to complete the computation.
So there's a few things that you'll notice as you look at this data.
As you get to the bottom, once your buffer sizes are getting very large, the GPU is clearly beating the CPU in terms of performance.
It's taking less time to complete these operations, and the crossover point is — where is it?
It's kind of right around this area where we see that the GPU starts to be beat the CPU.
You'll notice that the CPU does much, much better on the smaller buffer sizes than the GPU, and really what that tells you is that the GPU is not great at doing kind of small chunks of work.
And the reason for that is there's a lot of overhead in getting the GPU set up and started, particularly on this platform to setting it up to run the computation.
And so you can see when I go through the first kind of five or six runs of this the time it takes is more or less the same.
In fact, it's oscillating up and down a bit due to other random stuff going on in the system. But once we get down to about here, the time starts increasing.
So what that tells you is there's some fixed GPU overhead that's going on here, and the CPU is way better. So the reason for that is pretty simple.
We're not getting any advantage of parallel processing for these very small buffers, and so for very small, simple jobs your CPU is going to do just fine.
Once you get up to jobs that are millions or hundreds of millions of elements in size, the GPU is going to do better. And so why is that?
Well, that goes back to what we've been talking about throughout this talk, SIMD processing.
If you go back — and I will go back to our example — the CPU is basically doing what we talked about on the left-hand side.
Here it's going linearly through the array, running this very simple calculation one at a time.
So when the number of entries that it needs to do the calculation gets very large, that's going to take a long time.
The GPU is actually executing elements of that buffer in groups of actually more than 4, probably 32, 64, or even larger such that we're executing it in — let's call it 64, 64-sized chunks.
And so what that means is that the GPU is going to execute those instructions much, much faster, and we start to see that as the buffer size gets very, very large.
I'm going to attempt to show this live, and I'm also running Zoom and sharing Zoom.
So we'll see how this goes, but what you'll be able to see, assuming my computer doesn't crash, is this program — let me make it larger — is executing this live during the talk.
You can see how hopefully the GPU performance starts to outstrip the CPU as we go, and there it is. GPU is starting to beat the CPU.
CPU times getting longer and longer. We're about 2x the GPU right now.
About 3x. I can hear my fan going up. PROFESSOR 1: There's a question whether this is being done on the new M1 or whether that would change things.
JAMES PERCY
I wish it was being done on the new M1. PROFESSOR 1: [LAUGHS] JAMES PERCY: No.
Oddly, my wife has one of those because I bought one for her, but the Mac I'm running this on for work I believe is a 2018 Intel-based system.
I don't know the exact number, but it's an Intel integrated GPU. PROFESSOR 1: Mhm.
JAMES PERCY
And then you can see by the time we got done here that the GPU was almost seven times faster than the CPU for the largest buffer size.
Now, this was a very simple program, and, in fact, an astute engineer might notice that this thing's actually not doing that much calculation.
So it actually is at least on the GPU memory limit, and we talked about memory bandwidth earlier. The GPU is much better at moving large chunks of memory around than the CPU.
So we are also benefiting from not just the number of execution units but the amount of memory bandwidth that we have.
Now, it turns out I think on this particular CPU that I'm running on — because we've explicitly set it to operate in a single-core or single-thread manner the CPU core is likely mass limited, though I have not actually tested that.
All right. So here's a graph of what we just did. This is obviously not the experiment I just ran, but you could see the results were pretty similar.
So I ran this over the weekend without Zoom and other stuff running in the background.
This is a logarithmic scale — so notice that carefully — of our array size on the bottom with the runtime in milliseconds on the y-axis.
You can kind of see the crossover point here, and, again, remembering that this is a logarithmic scale, as we get up to these very large buffer sizes the GPU is beating the CPU performance handily.
All right. So let's wrap it up and then leave some time for questions. So if you get nothing else from this presentation, please take away these three points.
Getting efficiency and performance out of a GPU is all about maximizing parallelism, maximizing throughput via parallelism. You want to run parallel programs where you're doing SIMD operations.
You're running the same instruction across multiple data. That's what GPU performance is about.
There's some fixed-function stuff on the side to do specialized math like texturing, but really what we optimize around is maximizing performance through parallelism.
That makes image processing for cameras, for doing computational photography, for running algorithms to do image detection, so neural networks.
PROFESSOR 1: Neural nets, yeah. JAMES PERCY: --to be done on the GPU. PROFESSOR 1: Just ask Tesla. [LAUGHS] JAMES PERCY: Yes, exactly.
And modern GPUs are very easily expanded to do general purpose compute.
So companies like Nvidia and AMD have been very successful selling GPUs not just for games but for building compute farms of GPUs for doing machine learning algorithms and things like that.
So that's a big growth factor for GPUs in recent years. With that, I'll stop there. I hope you guys found this interesting. Let's — PROFESSOR 1: All right.
JAMES PERCY: --questions in the last few minutes. PROFESSOR 1: That was great, James. PROFESSOR 2: Thank you. Thank you.
PROFESSOR 1: That was wonderful. We've got a couple of questions that are open. The first is what's the software that's actually managing the coherency in the GPU.
Would that be something built into the OS, or how do you manage the coherency making sure you worry about threads stepping on each other?
JAMES PERCY
That's an excellent question. The answer is it's really a combination of the application and the driver that is programming and managing the lower-level details of the GPU.
So if you went and looked at a modern GPU API such as Metal, for example, there are facilities inside the Metal programming language that give you mechanisms for thread synchronization and memory synchronization.
And those kind of high-level facilities get mapped to low-level hardware commands like a cache flush, a fence, or a barrier that the driver will convert into something that the hardware can actually understand.
To kind of perhaps bring that back to the question, the programmer does have to be aware of some of these synchronization requirements and how to manage parallel programming in combination with the driver because if you're not aware of those things you're not going to get good performance.
PROFESSOR 1: Makes sense. On kind of the same topic — do you need to worry about deadlock in GPUs?
JAMES PERCY
The hardware team, meaning my team, certainly does.
At a programming level, you need to worry about it kind of from the similar perspective as you would worry about hanging a CPU by writing an infinite loop or two loops, especially if you're doing parallel programming, that kind of depend on each other in ways that could create deadlocks.
If you get down to the hardware implementation details, there are actually all sorts of hazards that can cause deadlocks that we spend a lot of design, verification, time trying to find and either just avoid architecturally or put in deadlock breakers, you may call them, and it's the hardware so
that the hardware can lock itself up. PROFESSOR 1: Mhm.
JAMES PERCY
So, yes, it's something that we are very worried about. PROFESSOR 1: Boy, these are coming! As we answered one, four more questions showed up.
JAMES PERCY
All right. Hit me. PROFESSOR 1: Students are doing great. We got 70 students here.
What's the ideal relationship or is there a relationship between the number of pixels and the number of execution units in a GPU?
Maybe as you go to 6k, 8K, how does that — there's some limits where GPUs can't drive an 8k screen. That's just talking about the screen.
But in terms of the computation, is there mapping at all or maybe a good ratio?
JAMES PERCY
That's a good question. The nice thing about GPU processing is that it scales very, very well.
In particular, if you're doing super high-resolution 8k displays, then you're going to want — especially if you're running games and you want high frame rates, you're going to want a big GPU to drive that.
Now, there's no kind of fixed relationship. You could have a small modern GPU drive a 4K or an 8k display.
You're just going to get less performance out of your game if you were playing a game. And you may not get 30 frames per second, so your games — PROFESSOR 1: Right.
JAMES PERCY: --look choppy and not play well. So there's no fixed requirement, if you will.
But certainly if you're going to run a complex modern game on a very high-resolution display, you're going to want a bigger GPU with more cores to get a good gaming experience.
Hopefully that kind of — PROFESSOR 2: Yeah, and I don't get this user perception of 120 frames per second. I don't think my eyes can do it.
[LAUGHTER] JAMES PERCY: Yes, there's a lot of debates about 30, 60, or is 120 Hertz even useful. PROFESSOR 1: Well, it's in film too.
I mean, was it Lord of the Rings had like 60 or 120? And it was like, wait. This looks like TV. Something looks a little different here.
Exactly.
JAMES PERCY
Yeah. PROFESSOR 1: This is a great question from Adrian. What are some surprising applications where GPUs are used in places we might not expect?
JAMES PERCY
Oh, wow! That's a fantastic question. PROFESSOR 1: Yeah, I love it.
JAMES PERCY
I'm not sure if this would be surprising or not these days, but if you drive a Tesla or have been in a Tesla — and I'm not revealing anything secret about Tesla.
I don't know anything secret about Tesla — PROFESSOR 1: Right. JAMES PERCY: --just to put that out there.
But my understanding is they use GPU hardware to do a lot of their image processing algorithms for self-driving. PROFESSOR 1: OK.
JAMES PERCY
So you'll find GPUs in automobiles. You'll find GPUs a lot of places you wouldn't necessarily expect them a few years ago.
Similarly, on any modern phone — and I'm not talking specifically about Apple, but any modern phone is going to do a fair amount of computational photography these days.
PROFESSOR 1: Ooh! That's a good point. JAMES PERCY: --run through the GPU to generate or to blend images together to generate an image. PROFESSOR 1: Right.
JAMES PERCY
So photography is not simply about how many megapixels are in my sensor.
It's how good are the algorithms that you are running and how much GPU and other horsepower do you have to run those algorithms such that the person taking the picture can see it in less than a second so it's interactive.
Imagine if you were taking a picture and it took 10 seconds for the final picture to show up. That's not very fun for a phone-type use.
So GPUs matter a lot in that space as well. PROFESSOR 1: Especially with HDR imagery.
I mean, you can compress that tone mapping, where you have a bright part of the screen and a dark part of the screen, but the GPU can take many pictures simultaneously or in sequence and then compress them down into one that just captures at the highs and lows to produce the dynamic range.
I love it. I knew this is — PROFESSOR 2: And by the way, I got one. There's one surprising thing. It's a Bitcoin mining.
Bitcoin mining basically — PROFESSOR 1: Oh! PROFESSOR 2: Because of that a few years ago, it was all low-end GPUs have been sold out.
I mean, it was not possible to buy a low-end GPU — PROFESSOR 1: [LAUGHS] PROFESSOR 2: --because they're all. PROFESSOR 1: That's fascinating! I didn't know that! PROFESSOR 2: Yeah.
Now you can buy cheap dedicated hardware that does these hash functions more efficiently.
JAMES PERCY
That's an excellent example. Three years ago, I think it was December when Bitcoin was taking off.
You could not find a high-end GPU, and people or were selling them for thousands of dollars — PROFESSOR 1: Wow! JAMES PERCY: — Bitcoin mining. So that's an excellent example.
PROFESSOR 1: And by the way, I'm just looking at the latest Mac Pro.
The high-end GPUs are like $2,000 just by themselves before the price increased because people want to buy them to do these things. The high-end GPUs can be very expensive.
This is Adam. A great question. I knew this was going to come up. This is so cool!
How can we students play with some graphics image processing code? How do we code to use the GPU ourselves? First answer is get a Mac. [LAUGHS] JAMES PERCY: Yep!
Get a Mac. We have Metal. Apple has Metal programming guides and very simple examples of just how to get started.
If you just Google — I should put a link in here, but if you just Google "Metal programming introduction," there is some online tutorials that you can go through and go like, hey, how do I write like Hello, world of GPUs using Metal.
And I'm sure Nvidia and AMD have the same thing for their hardware using other languages. So there's definitely tutorials out there. PROFESSOR 1: And there's also I think CUDA and OpenCL.
Is that right? Those two libraries.
JAMES PERCY
Those are computer programming APIs. CUDA is specific to Nvidia, and OpenCL — I'm not sure how much that's used as much these days. That was more of a OpenAPI cross-platform.
I think it's been replaced by CUDA and others. PROFESSOR 1: Right.
And I'll just add to the question, which is the code that you showed in Metal you could be running on a built-in CPU on a laptop or a high-end Mac Pro with a dedicated two boxes of a CPU.
Do you need to worry about that at that level at the Metal level? It just is handled by the abstraction layer, right?
JAMES PERCY
As long as you write it in a way that's parallel programming-friendly, it doesn't matter if you're targeting something built into a phone or if you're targeting a giant GPU system.
The API takes care of that for you. PROFESSOR 1: That's wonderful. Now, unfortunately I've got I think 14 questions left, but we promised chemistry, which actually needs this webinar like now.
We're now two minutes into their webinar. So folks, let's give James a hand. Thank you so much for coming and sharing those thoughts about the GPU. PROFESSOR 2: Thank you, James.
JAMES PERCY
Woo! PROFESSOR 2: Thank you, James.
JAMES PERCY
You're welcome. PROFESSOR 1: Wonderful, wonderful! So grateful to have you here. Thanks again.
Outstanding lecture. There was a meta-question about, are these slides confidential. Well, we don't have the copies of the slides, but we have the webinar.
So we'll just keep that, and they'll just be able to watch the slides through the webinar, I'm guessing.
JAMES PERCY
Yeah. PROFESSOR 1: Okay. Perfect. Thanks so much.
All right. Take care, everybody. Stay here if you want to watch — PROFESSOR 2: Thank you. PROFESSOR 1: --lecture.
[LAUGHS] Okay! Take care, everybody. Thanks again, James.
JAMES PERCY
All right. PROFESSOR 1: Stay safe, folks. Wonderful! All right.
Bye-bye.
Đăng nhập để đánh giá học liệu này.
Đăng nhập để bình luận — chưa ai bình luận về học liệu này.
Không có khóa học liên quan.