Tác giả giữ toàn bộ bản quyền (All Rights Reserved) — không phải giấy phép mở. Bạn vẫn xem và học miễn phí trên OpenER; muốn tái sử dụng cần xin phép tác giả.
PROFESSOR
Welcome back.
Now we're going to teach you Spark, which is another way to think of the MapReduce paradigm with a little bit lighter weight touch in terms of how much programming you have to do for it.
So it's a fast and general engine for large-scale data processing. It came out of UC Berkeley. Very excited about that.
And this is a performance bar that says running time on a particular file, and Spark is blowing it out of the water at a factor of more than 100 times faster.
And part of the reason for that is that Spark does the work in memory. MapReduce is always about disk, disk in, disk out.
Basically, MapReduce says, we're dealing with such big data sets that I can't do it in memory. I'm going to load it in memory and do it, but where do you put it.
Basically, the kind of input output characteristic is always based on disk, file in, file out. That's the idea. You write a file and then you done.
If it crashes halfway through, at least your file is written and you can rest.
Spark says, well, that's nice, but wouldn't it be nice if I could work just in memory and just live in memory and not have to pay the 1,000 time or more performance penalty to go to disk.
That's painful. So Spark has an advanced execution engine that figures out what's needed when, and it gives you just in time computation. It's quite clever. It's lazy.
It's a lazy evaluation model, which is really nice, and as much as you can in-memory computing. Easy to write applications in Java, in Scala, or Python.
We're going to see the Python example, and it offers quite a few high level operations to make it very easy, and you can even interact with it interactively.
You can interface with it interactively, as well as the batch processing way, which is how MapReduce is done. But the fact that you can play this interactively is really very fun and very exciting.
So, ladies and gentlemen, let me show you. This is the Java code I showed you last lecture. This is the same word count in Spark to Python. File.
I've loaded the file. There's another line above it, which says what file it is. I've loaded file in, but the actual processing is first I flatMap by splitting my document into words.
Now I have a list of words. I then map across that, word and 1, and I have returned, now, a list of word and 1. And I reduce by key.
Remember the reduction by key.
Rather than having a shuffling phase, this is reduced by key, and what that says is, each of these worker elements is going to get, again, just the values of the same key.
Reduce by key means each of those worker bees only get the values for a particular key, and what does it do with those two values? It adds them up.
Ladies and gentleman, mic drop. So not only are you 100 times faster, but you can do in three lines what Java had to do in 45 lines. Pretty impressive.
So let's actually play with it a little bit. So you can load this up. I encourage you to do this as well. I'm going to show you what flatMap does.
If you didn't know what flatMap does, I'm going to explain what that does and why we need that. So here's an example called neighbor.
Neighbor takes a number and returns a list of the number to the left, my number, and then to my right. Pretty easy. We start this by saying sc, Spark context, .parallelize(range(5)).
That says I'm going to take the number 0 through 4 inclusive, and in some sense parallelize it, load it into its already D set, into the way it stores data, into a parallel way it stores data.
You call it R. Now, nothing's happened yet, by the way.
The difference between range in Python 2 versus Python 3 is Python 2 range, if you say range of a million, it will actually make a million list and return it.
Python 3 range says, yeah, but what if you never use it. That's silly to make that million list range but never use it. So Python 3 takes the lazy approach and returns a promise.
Here's range. Every time there's a new type called range of 5, range 051 one maybe, and it says, well, that didn't do any work. It returns instantly.
Python 2, call range of a million and see how long it takes. Call range of a million on Python 3, instantaneously. Here's a promise for range of a million if you ever wanted them.
And then, if you say, well, let me iterate. Well, it's going to, one-by-one, grab a value from range. Hey, range, I need your next value. Okay, here it is, 3.
Range, I need the next value. Here's 4. That's how that works. That's how lazy evaluation works.
Same idea here in Spark. That range didn't do any work. So range returns a promise. That parallelize didn't do any work, because you never asked for it.
Only when you say collect are you saying, Okay, now you've got to pay the piper. Where's your data? Parallelize says, well, where's the data? Who do we call it on?
Oh, range of 5. Quick, range of 5, give us a value. Range says, Okay, how much do you need? All of them, great.
Range will give you those values. Only at that moment does range do any work, and only that moment does parallelize do any work. That's the idea.
It's a very lazy system, very clever that way. It also means you might have delays. There's 20 commands. Why are they so fast?
Yeah, because nothing's happened yet. Only when you say collect does it say, Okay, now I've got to do the work.
So only on a collect will it actually gather this up and try to spit it back out to you. It's a really important idea that you understand that.
So now, if I were to say R.map of neighbor, it will return a list of five of these triplets, the n minus 1, n, and n plus 1, but it wouldn't do it until I say .collect.
If I just say R.map on neighbor, that's fine. Range could have been over 10 billion, 100 billion, 1 trillion.
That R.map of neighbor would have instantly returned doing no work, because it's not asked to produce anything yet.
If you say .collect, it's going to try to give it all and actually try to do its work. So if I say R.map(neighbor) .collect, it will give you that.
It'll give you a list of five sublists. But I'm saying to myself, I don't want five sublists. I want one flat map. I want one flat list.
Well, that's what flatMap does. It's exactly the same ideas map, except it flattens whatever sublists came out of what map it would have given you. It flattens those.
It's like it appends them all together. So flatMap returns the same thing as map does, except they're all flattened. Does that make sense?
I teach you that because we're going to see this as we look at what word count does when we play with word count.
We're going to actually run a slow motion version of word count in Spark. So let's do this now. So here is word count.
We saw this part already on one line where Stephen Colbert dropped his mic, but let's do it a little bit slower.
This is exactly the same data I showed you before, literally to the letter, the same data.
These are the files, ah, ah, er in the first one, and then there's an empty file in the second one. All these are the same data before in that picture you saw earlier.
So the first thing I say is W, for word count, equals sc.textFile, and you give it a text file. That's like it's parallelized, but it's reading text files rather than parallelizing a Python object.
First thing I say is flatMap. Again, lambda line line.split, and only if I say .collect will it print something out. If I just did that, it wouldn't normally give you any output.
But I say this, and it gives me, essentially — What this flat map does, again, it says, I don't care what file you came from.
I want to just flatten you into one big pool. We call it a pool of words, one big pool of words. So here we go. There's all the words I have.
Then I say flatMap. And by the way, the orange is the only thing that's new. All the white is the same. I'm adding .map lambda word, word of 1.
And what that's going to do is, for each of these words, I tag it 1. None of that's new. You've seen that before now three times here. And now, here's the key.
I add reduceByKey lambda a,b a plus b, and this is literally — I just copied and pasted this yesterday — what you get from Spark, and that's what you get.
By the way, these are in no particular order, obviously. They're not alphabetized or anything, but er is 1, 4 ahs, 2 ifs, 3 ors, and an uh. That's my output.
Word count. I broke it into pieces. I probably could've done this in one line.
This sc.textFile is W. I could have said that .flatMap, and then that .map, and then that .reduceByKey. You get the same again, one line like I showed you before.
Pretty neat. Isn't that beautiful? So that's an example. Now you say, well, Dan, is it faster?
Are you sure it did it in parallel? Well, I did this little baby thing to sanity check it. Let's run this thing.
Let's say I'm going to crunch some numbers, simulate crunching a big number. I take a number, n, and I'm going to be chugging, and chugging, and chugging, and doing lots of testing in here.
Something is going to take an hour. I mean, five seconds, I'm saying. So I'm going to actually say I'm going to sleep for five seconds and then return the square of that number.
So if I say crunch of 10, it returns 100 five seconds later. By the way, try this yourself, trust me. If I then say map(crunch,range(4)).
So I'm going to square all the elements from 0 through 3. Remember, if I just say that, what will happen is it'll say, oh, yeah, return a map type.
No, I want you to actually give me the value. So I have to say list that to get the actual values, but it takes 20 seconds later. There's four elements there.
4 times 5 is 20. That's 20 seconds later, and that does take 20 seconds if you time it. Then, just for fun, I say, well, let's try it in Spark's world.
sc.parallelize(range(4)), fine. R.map(crunch).collect. Five seconds later. Five seconds later it came up with that.
So this is like a little silly test. But now, what's interesting is I ran this on an eight core machine.
I would love to retest this, and I want you to play with this on your own machines. Please play with this.
We know a way of hyper-threading. So maybe I have eight cores. That's 16 threads I can work with. How big is it before this stops taking five seconds?
So continue to change the size of this to see what happens, and that's kind of a way to poke into your machine.
And by the way, when you instantiate Spark, you can tell it how many cores you want to have, how many parallel threads you want to allow. So play with that.
Play with that number and see what happens as you change this number to see if it's always five seconds, or when it gets above five seconds.
But it's kind of neat that this whole thing happened in five seconds. Just as a sanity check, play with that. So let's end this lecture.
Fourth big idea that you've seen in this course is parallelism. We are hitting it hard from many different ways.
In fact, we saw two of those ways today, data level parallelism and request level parallelism. Amdahl's law is heartbreaking and unfortunate, but it's part of our reality.
By the way, part of the reality of, not computer science. This is everything. Anything that working in parallel, Amdahl's law will affect that. Again, with infinite parallelism, speed up is 1 over s.
That's the maximum. MapReduce is a wonderful abstraction. Google uses it, many, many people. Yahoo uses it for their big data processing.
It is file based. Spark does it even better, and it's very fast growing. The number of developers involved in Spark is growing, and by the way, this makes you very marketable.
Nudge, nudge, wink, wink. Say no more. I encourage all of you to learn about Spark and be fluent in spark. Play with it over the break.
You add Spark to this, you say I can work with big data on to your resume, you're going to look like a pretty good candidate. That's it.
We'll see you at the next segment when we talk about cloud computing. Pretty exciting. See you there.
Đăng nhập để đánh giá học liệu này.
Đăng nhập để bình luận — chưa ai bình luận về học liệu này.
Không có khóa học liên quan.