Tác giả giữ toàn bộ bản quyền (All Rights Reserved) — không phải giấy phép mở. Bạn vẫn xem và học miễn phí trên OpenER; muốn tái sử dụng cần xin phép tác giả.
[MUSIC PLAYING] BORA NIKOLIC: Hey, hello. Long time no see.
Welcome back to 61C. We are getting into our brand-new module. This time, we are talking about parallelism.
So we are still continuing our quest to make a better computer and to run our applications faster.
So we have learned some of the software techniques that are helping us with that, but we spend most of the time trying to build a better computer and understand the principles how these better computers are built.
One of those principles is the use of parallelism, and we have encountered parallelism already. We have encountered multiprogramming as a way to better utilize a processor.
While it's waiting for data to come from DRAM, we can perform some other task.
We have seen pipelining as a way to increase the performance by working on multiple instructions at the same time in the pipeline.
And that was the primary way how people tried to increase performance or processors in, say, the '90s up to early 2000s.
We kept building deeper and deeper pipelines and simultaneously used that to crank up the clock frequencies.
But as a result, the power kept shooting up in every processor generation, and that was not sustainable. We had to do something different.
In order to stay under reasonable power limits, people explored the dimension of using parallelism. And in this case, we are going to take a look at how do we work with parallel data.
What does that mean?
Simply, we, often in our applications, work with some sorts of vectors, and we can perform vector additions, in this case, in parallel if we have multiple units that are capable of adding multiple elements
of a vector at the same time. There are many opportunities for data parallelism out there, one of them we have already encountered.
It is one of the most popular workloads nowadays, which is machine learning inference.
In machine learning inference, we often start with something, and we would like to perform classification.
In this case, we can start with an image of a cat and would like to say that's a cat and not a dog.
The way how we do that is by going through multiple layers of classification, feature extraction here, that in their essence are fairly parallel.
They could be convolutions, matrix-to-vector multiplications or matrix-to-matrix multiplications. And after a sequence of those classification steps, we can say that there is a cat in this image and not a dog.
And in turn, if we feed to our neural network an image of a dog, it'll tell us it's a dog.
If we feed handwritten numbers, it'll be able to tell us whether that number is an 8 or 9.
There is parallelism here, both within each layer, but also we can process multiple layers concurrently. We have introduced matrix multiplication.
Matrix multiplication is not just common in neural nets.
It's one of the most common operations that we'll encounter in engineering, data science, and many other application domains.
Generally, we'll find it in image filtering or blurring or noise reduction, and many of them are seen in learning kernels.
But it's not a modern type of workload. I mean, it has been so prevalent in engineering that it has been supported in languages since long time ago.
For example, there was a language, one of the first languages that I've learned as a student was something I just called Fortran. You guys might not have even heard of a Fortran.
Tells something a little bit about my age.
Fortran was a common language for engineering applications in the past, up to probably early 2000s, when it was basically replaced in most of the domains by C.
It took C a while to displace Fortran, legacy Fortran from everywhere in the world. So in Fortran, there was this instruction, which was dgemm.
And dgemm is a double-precision floating-point matrix multiplication. So it was as prevalent or as important that it needed its own instruction.
Just to quickly recap our matrices because it will be helpful in understanding how we work on them a bit later, here is an example of a square matrix with i rows and j columns.
Important thing to keep in mind is we generally can't store this matrix as a square matrix in a memory or in a disk. It is generally stored sequentially.
So we have a choice here, whether we would like to store it serially by rows or columns.
Generally, it is serialized by columns, so in this case, the first element in the matrix, A(0,0,), will be followed by the second element in that column, A(1,0), and then we will store them all up to A(N-1,0).
And then we would continue with the second column, A(0,1), store all of them up to A(N-1,1), and so on.
All right, and what does the matrix multiplication do?
Well, if we say that we would like to multiply two matrices, A and B, and store the result in a C, what we are essentially doing, we are performing an iterative process that computes elements of a resulting matrix Cij and sums of pairwise products Aik
and Bkj.
The inner coefficient, k, is the same, and that basically means that the number of columns in matrix A has to be the same as the number of rows in matrix B. That's the only way how the matrix multiplication is valid.
So we would like to find this element Cij.
We would perform pairwise multiplication of all the elements in a row i with all the elements in column B and sum them together to get the element Cij.
Notice that all these multiplications of elements Aik and Bkj can be performed concurrently, and then we would just add them together.
So we're going to look a little bit more into that after a quick break.
Đăng nhập để đánh giá học liệu này.
Đăng nhập để bình luận — chưa ai bình luận về học liệu này.
Không có khóa học liên quan.