Tác giả giữ toàn bộ bản quyền (All Rights Reserved) — không phải giấy phép mở. Bạn vẫn xem và học miễn phí trên OpenER; muốn tái sử dụng cần xin phép tác giả.
[MUSIC PLAYING] BORA NIKOLIC: Welcome back to our parallelism module.
So far, we have explored the performance of our baseline application, which was the matrix multiplication.
We have compared the throughput that we get for matrix multiplication when it's written in Python versus C, and we have seen that C is significantly faster than Python.
The reason for that is Python is a productivity language, and it is interpreted, so it is easier to write code in Python but at the cost of performance.
"Interpreted" means that it is not directly compiled.
There is something that comes in the way of executing every single one of these instructions and adds a bunch of other assembly instructions that need to be executed.
On the other hand, C is compiled and compiles directly into the assembly and is much more efficient.
Can we be more efficient in writing sequential C code on a processor on a serial processor like the ones that we have examined so far?
Yeah, perhaps, and we can write assembly code and do better. But in general, compilers are pretty good nowadays. It's not that easy to beat them, but still, there is a chance.
The other way how we can get speedups is by trying to address that problem, slowdown when we are working with larger matrices.
But we can do that either in C or in assembly by using so-called blocking techniques, and we might see them a bit later on in this module.
Now, how can we do actually faster? What we can do things faster by using parallelism.
And one of the goals of parallelism is to essentially do the entire or most of that inner loop of matrix multiplication concurrently by having hardware that can do multiple multiplications at the same time.
But before we get into that, let's take a look at different types of parallel hardware that we can encounter. So first, there is an important thing to distinguish in parallelism between software and hardware.
They're generally orthogonal to each other, so this table is fixed straight from the book that compares an older processor that is strictly serial, which was Intel Pentium 4.
And then there is a modern processor, more modern processor, Intel Core i7, that has parallel features in it.
So the same type of code can be run on a serial processor or on a power processor, and the speedup that we are going to do to get is perhaps from using some of these, doing some of the inner loop concurrently.
Now, software can be running sequentially, like our matrix multiplication, regardless of what type of hardware we are running on, or concurrently, like our operating system. Now, this is an operating system.
This example is Vista. You might not even know what is Vista.
It's an operating system that predates Windows 7, or actually, Windows 10, Windows 7, and somewhere around there, around Windows XP, there was Vista as well.
What we have seen, we use multiprogramming to run multiple processes, what it looks like concurrently, even though we are running it on serial hardware.
So choice of hardware and software parallelism are generally independent. Concurrent software can run on serial hardware. Sequential software can run on parallel hardware.
Flynn's taxonomy, which is what we'll address next, is essentially a classification of parallel hardware. So let's take a look at that.
This is named after Professor Michael Flynn, who is a professor emeritus at Stanford University.
So he has classified different types of parallelism between the parallelism in data streams and instruction streams.
So there are four entries in this table that correspond to single instruction, single data; single instruction, multiple data; multiple instructions, single data; and multiple instructions, multiple data.
Let's take a quick look at those. What we have encountered so far, everything was single instruction, single data. That's how our single-core sequential processors operated.
Most of what we are going to be talking about in this module are SIMD and MIMD architectures. SIMD is Single Instruction, Multiple Data. MIMD is Multiple Instructions, Multiple Data.
However, most of the programs that we'll find nowadays are kind of a combination of that, and they can be called single program, multiple data.
And the idea there is that if you're going to run a single program, it is going to try to use multiple degrees of parallelism.
So it will be running off all the processors that have MIMD, multiple instructions and multiple data, and there will be some cross-processor coordination that we are going to see a bit later in this module.
SIMD is a particular type of processors that has specialized hardware that can run multiple — it can operate on multiple data at the same time in so-called lockstep fashion.
And that is very good for running these arrays, so it can basically crank through one dimension of the array in one instruction.
And that's very useful, not just for neural nets, imaging, but many of the scientific applications. Okay, so let's quickly walk through all four entries that we have in this table.
First is the one that we have seen already seen so far. This is the Single Instruction, Single Data, or SISD.
So there is a sequential processor that basically sequences through the instruction pool and matches it with the data that is in the memory and processes it one at a time.
So it's our traditional unit processor. The second in class is SIMD. SIMD is a type of computer that has multiple processing units.
These PUs are the Processing Units. And it will essentially issue one instruction, say, add, and it will operate on multiple data pairs at the same time.
In this case, there will be four processing units that would be capable of doing four additions for each add instruction.
So it would be essentially adding four-element vector to another four-element vector.
Then there is the third type that we see here, which is MIMD, which are Multiple Instructions, Multiple Data streams.
So we would be running multiple — issuing multiple instructions at the same time, that each one would be operating on multiple data. This is generally not just one processor that does that.
This is generally a concept of multiple processors, where each one of them is a SIMD processor that are operating concurrently. So each processor issues its own instruction that runs on multiple data.
So these MIMD architectures are going to be, again, covered a little bit later in this module that involve multicore processors and data centers, or what we are going to call a warehouse-scale computers, where there are many computers that are housed in the same building.
And finally, the fourth entry in that table is Multiple Instruction, Single Data stream, MISD. Well, that is not something that we really encounter nowadays, and it's not something that we really need.
This is a concept of having the same set of data and then doing multiple things to that.
So it's like, would you like your eggs scrambled or sunny side up? Well, you say both.
So it does not always — it is not something that we'll frequently encounter, as you want to do two different operations on the same data.
That's it for now. We are going to dive next through the rest of this segment into SIMD architectures, but just after a quick break.
Đăng nhập để đánh giá học liệu này.
Đăng nhập để bình luận — chưa ai bình luận về học liệu này.
Không có khóa học liên quan.