Video
Đăng bởi Quản trị viên OpenER ·
Tác giả giữ toàn bộ bản quyền (All Rights Reserved) — không phải giấy phép mở. Bạn vẫn xem và học miễn phí trên OpenER; muốn tái sử dụng cần xin phép tác giả.
[MUSIC PLAYING] BORA NIKOLIC: Hello, and welcome back to our parallelism module.
We've introduced Flynn's taxonomy of parallel architectures, and we said that we are interested in SIMD and MIMD architectures. SIMD stands for Single Instruction, Multiple Data.
MIMD is Multiple Instructions, Multiple Data.
What we are going to do now, we will take a bit of a dive into some of the SIMD architectures.
We're not going to actually build a SIMD architecture, but we'll see how does a programmer see it. So let's get into that.
So our SIMD architectures are there to exploit data-level parallelism.
So we would like to fetch one instruction and apply it to multiple sets of data.
Typically, if it is an addition, we would just fetch one add — that would be some sort of a vector add — and apply it to all of the elements of two vectors.
So instead of a scalar add, where we would be just adding two operands, two scalar operands, this time, we would be doing vector addition of element by element for both of the vectors.
Same thing applies to, say, multiplication.
And in this example here, we can simply multiply coefficient vector by data vector, such that we can perhaps perform some kind of filtering.
In the next step, we would just add those results. So why is this faster? Well, because we just need to fetch one instruction instead of, every time, fetching the same instruction.
So if we're operating on a vector, we would fetch one instruction and apply it to multiple data pairs.
Now, you may say, well, but I'm going to be [INAUDIBLE] by the data.
No, if the bandwidth for getting all the data is wide enough so that we can get chunks of data, say, from the memory or, more likely, from the cache.
People have noticed this, and there has been always — or for a long, long time, there has been interest in vector architectures or these kinds of SIMD extensions.
The first noted SIMD machine was implemented by MIT Lincoln Labs. That was the TX-2 in 1957.
And if you look at it, they had an ability to run their full 36-bit data, or they could split it into two 17-bit operands or split it further into 9-bit operands.
They were not quite working with standardized bytes and words back then. Or actually, the word here was 35 bits.
These architectures were around for a while but were seen in a wide commercial use when they were introduced by Intel in the late '90s, around 1997.
It was not as that people at that time on PCs were running more and more multimedia applications.
And multimedia was mostly, at that time, audio and some video, and that often involves some kind of filtering.
And when we are filtering, in media applications, we are operating typically on vectors, vectors in one or two dimensions.
So the way how this is implemented is essentially there are two source vectors placed in relatively wide registers.
And then on some part of those very relatively wide words, we could apply the same operation and get the result in the destination register.
So we would fetch one instruction and do the work that corresponds to multiple instructions by applying it to this kind of vectorized data.
This was called MMX, MultiMedia eXtension, appeared in Intel's Pentium II processor. And then later generations were named SSE, Streaming SIMD Extension.
That appeared in Pentium III and Pentium 4 and beyond. And then later on, we got AVX, Advanced Vector eXtensions. Here is a quick evolution of what has happened there.
So it started with MMX around 1997. And remember, all Intel processors have to be backwards compatible. So MMX is still around with us.
Every single newer processor is still implementing MMX. But what has been happening, the SIMD width has increased.
So the first, we went from a 64-bit SIMD to 128-bit SIMD by adding different versions of SSEs.
And then with AVX, we got to 256-bit SIMD. And finally, the most recent processors have 512-bit SIMD.
There is an expectation there'll be relatively soon 1,024-bit-wide SIMD.
So in order to support this, Intel had to add new instructions to their already fairly complex instruction set.
So all these instructions have been added one after another one in each generation.
At the same time, they also have to add new registers, such that they can operate out of these registers, and they get more registers as they also get wider.
Now, when you run something like this on AVX, what do we get? Do we get any speedup? Do we get expected speedups compared to our C code? And the answer is yes.
Look at this. Now, if you use AVX extensions on an Intel processor, you will get some speedups compared to a plain C.
The speedups are around 4x. And no, you're not going to get around cache size limitations, even with AVX. That doesn't help.
But the code generally just gets faster.
Is this as fast as you can go? Well, you can check on this, a few-years-old Intel Processor i7 from a few years ago. Peak performance is 25 gigaflops per second.
That's processing rate is running a 3.1 gigahertz with two instructions per cycle and capable of doing four multiplications per instruction.
So we are still at, like, a quarter of the maximum throughput that we can get through that.
But we are getting nearly theoretically speedup of 4x over a single scalar operation.
A few other things about the architecture — so when we look at SSE or MMX added these XMM registers.
These are the extended registers. In MMX, they were 64 bits wide. In SSE, they were 128 bits wide.
So there is a separate set of registers that exist in the processor. These are not the general-purpose registers and not the floating-point registers.
They are additional vector registers there.
So they also can be split.
So one can work on a whole 128-bit-wide data or split it into two 64-bit words or four 32-bit words or break it down even further.
By the way, MMX — often, usage model was to take these 64-bit-wide words and break them down to 8-bit chunks and work with that generally as integers for all the applications.
Generally, you'll hear that Intel implements something that is called the packed SIMD type.
What does that mean is that in these 128 bits in SSE and SSE2, we can pack 16 bytes that are 8 bits wide, or we can pack Intel's words, which are 16, but Intel called the 16-bit operand a word,
so we can pack eight of those. Or we can pack four double words. Double words are 32 bits. Or we can pack two quad words, and quad words are 64 bits.
Single-precision floating-point takes 32 bits. Double-precision floating-point is 64 bits.
Now, it really depends what kind of an Intel processor you have is what you get. Some of these things may appear in laptops. Some of them may appear in desktops.
Some of them may appear in Xeon server-class processors.
This is showing that different generations will support different kind of vector extensions.
So if you have a bit of an older one, you may just have SSE.
Then there was an SSE IA-64, then that got upgraded with AVX, and then finally we got AVX-512.
The number of registers is also growing. Early on, there were 8 vector registers, then that increased to 16. Now there are 32 of them.
So both the width of the registers is growing and the number of registers is growing.
So if you have something that is really, really highly parallel — and a typical lingo that is used for that, even in scientific community — this is embarrassingly parallel.
So if you're working with a lot of matrices, then this is a great architecture for that.
It is great for video processing, image processing, and processing neural nets. So what do we have?
Well, we do run our standard command in Linux OS CPU, and then we find out what our processor supports.
In order to record this class, I needed to get a little better processor, so I find out that I have a floating point here.
Then I have SSE, then SSE2, SSE3, and various AVX extensions. This is useful to know when we try to write assembly code for that.
But also, this is something that our GCC C compiler generally is able to use.
We are going to see a bit of examples of how we use them, these extensions, in the next modules — in the next sections, actually. See you after a quick break.
Đăng nhập để đánh giá học liệu này.
Đăng nhập để bình luận — chưa ai bình luận về học liệu này.
Không có khóa học liên quan.