NASA ARSET Visualizing Land Cover Change P 2

0:03 Speaker 1: Hello everyone, and welcome back for Part two of the

0:06 Speaker 1: r SET training on visualizing land cover and land use

0:09 Speaker 1: change with NASA Satellite Imagery. My name is Justin Faine

0:13 Speaker 1: and I'm part of the r SET Ecological Conservation Team

0:16 Speaker 1: based out of the NASA AMES Research Center at the

0:18 Speaker 1: South end of the San Francisco Bay. As before, I'd

0:23 Speaker 1: like to take a moment to go over some details

0:25 Speaker 1: of the r SET program. Our SET is the Applied

0:29 Speaker 1: Remote Sensing Training program. We provide free trainings on a

0:32 Speaker 1: variety of topics which are divided into thematic areas, which

0:35 Speaker 1: is what you see listed here on the right. We

0:38 Speaker 1: do our best to listen to feedback from our audience

0:40 Speaker 1: to make sure we're doing trainings on the topics you

0:43 Speaker 1: are most interested in. We cover remote sensing satellites, sensors, methods,

0:48 Speaker 1: and tools at various levels of complexity, from the fundamentals

0:51 Speaker 1: of remote sensing to advanced trainings. Our SET trainings come

0:57 Speaker 1: in a few different formats. They can be online or

1:00 Speaker 1: in person, live an instructor led or asynchronis and self pace,

1:04 Speaker 1: but they are always provided.

1:05 Speaker 2: Free of charge.

1:07 Speaker 1: Many of our trainings are available with bilingual or multi

1:09 Speaker 1: lingual options. We only use open source software and data,

1:14 Speaker 1: and the trainings are targeted at a variety of levels

1:16 Speaker 1: of expertise. This training is called visualizing land cover and

1:22 Speaker 1: land use change with NASA Satellite Imagery. These are the

1:27 Speaker 1: learning objectives for the entire two part training, of which

1:30 Speaker 1: this is part two. By the end of this training,

1:33 Speaker 1: you should be able to access NASA Earth observation data

1:37 Speaker 1: relevant to land cover and land use change mapping, convert

1:41 Speaker 1: NASA Earth observation data into distinct land cover land use

1:44 Speaker 1: classes using supervised and unsupervised machine learning classification methods in

1:48 Speaker 1: the R programming language. Recognize the role of classification methods

1:53 Speaker 1: as one part of a change monitoring strategy. Computer change

1:57 Speaker 1: matrix representing the change in land covered land use between

2:00 Speaker 1: two dates, and create a map in our studio.

2:02 Speaker 2: Visualizing the differences in land cover and land use between

2:05 Speaker 2: two dates. The prerequisites to get.

2:10 Speaker 1: The most out of this training are the fundamentals of

2:12 Speaker 1: remote sensing course and some experience working with our in

2:15 Speaker 1: our studio. Note that I'm not asking you to have

2:19 Speaker 1: downloaded any of the code or data for this training.

2:22 Speaker 1: I'm going to be jumping back and forth between slides

2:24 Speaker 1: and code. Later in a way that will probably be

2:26 Speaker 1: hard to follow, and I've also gone ahead and prepared

2:29 Speaker 1: some of the parts which take a while to run,

2:31 Speaker 1: so that we can keep training moving along. I suggest

2:34 Speaker 1: that you follow along with this presentation and then explore

2:37 Speaker 1: the code afterwards. The code and data I'll be using

2:41 Speaker 1: in both sessions of this training will be available on

2:43 Speaker 1: the our set GitHub, which you can find on the

2:46 Speaker 1: training web page, along with these slides, a recording of

2:48 Speaker 1: this presentation and all of the other materials. If you're

2:52 Speaker 1: not familiar with cloning data from GitHub repository, we've got

2:55 Speaker 1: you covered on that topic as well. There's a short

2:58 Speaker 1: video linked on the training web page which covers the

3:00 Speaker 1: cloning operation. This is the second part of a two

3:06 Speaker 1: part training. The homework for this training will open following

3:09 Speaker 1: this session and will be due on March twelfth, which

3:12 Speaker 1: is two weeks from now. You can find the homework

3:15 Speaker 1: on the training web page, and we'll send out a

3:17 Speaker 1: certificate of completion to all who attend both parts of

3:21 Speaker 1: this training and complete the homework on time. With the

3:26 Speaker 1: logistics covered, we can begin Part two visualizing land cover change.

3:32 Speaker 1: Once again, I'm just in fame and I remain your

3:35 Speaker 1: sole instructor for this training. I'm a research scientist with

3:38 Speaker 1: NASA AMES and the Bay Area Environmental Research Institute in

3:42 Speaker 1: Mountain View, California. Here are the objectives specific to the

3:47 Speaker 1: second part of this training.

3:49 Speaker 2: By the end of.

3:49 Speaker 1: Part two, you'll be able to apply a random forest

3:52 Speaker 1: model to NASA mmetry to create land cover and land

3:55 Speaker 1: use maps for two different time periods and analyze both

3:59 Speaker 1: numerically and usually the change in land cover and land

4:02 Speaker 1: use over time. Please put any questions into the Q

4:08 Speaker 1: and A box in this training. There are a bunch

4:10 Speaker 1: of our set people watching the Q and A who

4:12 Speaker 1: are working to answer your questions while I'm presenting. Well,

4:15 Speaker 1: try to answer any questions you submit at the end

4:17 Speaker 1: of this training, but any questions we don't get to

4:20 Speaker 1: because of time or because they require some extra time

4:23 Speaker 1: on our end answer fully, We'll be answered in that

4:26 Speaker 1: Q and A document and uploaded to the training website

4:28 Speaker 1: in a week or so. The questions you ask and

4:31 Speaker 1: the responses we get from you and the post training

4:33 Speaker 1: surveys help us to sign on topics for future our

4:36 Speaker 1: set training, So your contributions are appreciated. Now we can

4:42 Speaker 1: get started. Just to review, here's some important concepts which

4:47 Speaker 1: we went over in the first part of this training

4:50 Speaker 1: and the image on the right is where we're headed

4:52 Speaker 1: today as we learn how to implement a random forest

4:54 Speaker 1: classifier and R and use it to show land covered

4:57 Speaker 1: land use change over time. In the previous session, we

5:03 Speaker 1: looked at one unsupervised classification method K means in detail.

5:08 Speaker 1: Today we're going to focus on supervised classification and ultimately

5:12 Speaker 1: build that random forest classifier for land cover. Recall this

5:18 Speaker 1: table from part one. With supervised classification were limited by

5:22 Speaker 1: the quality and specificity of the training data. Classes are

5:25 Speaker 1: targets which we establish ahead of time when we collect

5:28 Speaker 1: and label the training data. We'll focus primarily on random

5:32 Speaker 1: forest today since it's a great introduction to some other

5:34 Speaker 1: important concepts. However, since we discussed K means and detail

5:39 Speaker 1: in part one, let's briefly discuss the related K nearest neighbor.

5:45 Speaker 1: K Nearest neighbor uses the class labels of the training

5:48 Speaker 1: data to decide how to classify new data. K means

5:52 Speaker 1: being unsupervised, has no predetermined classes and only relies on

5:56 Speaker 1: similarities and brief this is how CA nearest neighbor works.

6:01 Speaker 1: Give it a new sample. Check the K nearest points

6:04 Speaker 1: or neighbors, and assign the new point the class which

6:07 Speaker 1: is most frequent among those neighbors. K can be any

6:10 Speaker 1: integer greater than one, but it's usually good to choose

6:13 Speaker 1: an odd number to help avoid ties. The points on

6:16 Speaker 1: the right are examples of known samples of different classes,

6:20 Speaker 1: where each class is represented by a different color. We

6:25 Speaker 1: can go through an example step by step step to

6:27 Speaker 1: see how this works in practice. Watch how K nearest

6:31 Speaker 1: neighbor classifies a new point. We'll set K equal to three,

6:36 Speaker 1: find the closest neighboring points, tally the votes, and the

6:47 Speaker 1: class that's most frequent in the list of neighbors is

6:49 Speaker 1: the class of the new point. Will now shift our

6:56 Speaker 1: focus and explore random forest models and lay the ground

7:00 Speaker 1: work for making our own random forest model. In R

7:04 Speaker 1: random forest is a tree based model, which means that

7:07 Speaker 1: it works by creating decision trees. The decision trees explain

7:11 Speaker 1: possible ways to divide the data based on binary true

7:15 Speaker 1: false filters. New data is classified by applying the decision trees,

7:20 Speaker 1: which work to split the training data most accurately. To

7:25 Speaker 1: see how this works, let's assume we have a training

7:28 Speaker 1: data set which consists of aliens and acorns. Random Forest

7:33 Speaker 1: knows that they are different classes because the training data

7:37 Speaker 1: is labeled with class information.

7:40 Speaker 2: There are some.

7:40 Speaker 1: Features which separate the aliens and acorns and our training data.

7:45 Speaker 1: Aliens can be blue or green, and acorns are smaller

7:52 Speaker 1: than aliens. Each split in the tree represents a binary discision.

8:05 Speaker 1: The first decision the first question might ask is it

8:09 Speaker 1: larger than some size? The next might ask is the

8:14 Speaker 1: object green? And finally it might ask.

8:17 Speaker 2: Is the object blue.

8:20 Speaker 1: Note that we didn't tell the model to make these splits.

8:23 Speaker 1: The random forest model actually makes hundreds of decision trees,

8:26 Speaker 1: each of which is the result of leaving out a

8:28 Speaker 1: random set of variables and observations. But for demonstration purposes,

8:32 Speaker 1: we're only going to consider the single tree where we're

8:35 Speaker 1: using all of the variables and all of the observations.

8:40 Speaker 1: So we've built a decision tree that describes a series

8:44 Speaker 1: of questions which we can answer to sort objects. Any

8:49 Speaker 1: object sorted into the left hand side is more likely

8:52 Speaker 1: to be an alien, and any objects sorted into the

8:56 Speaker 1: right hand side are more likely to be acorns. Now

9:01 Speaker 1: we can apply this to some data that the model

9:04 Speaker 1: hasn't seen before and observe what happens. The decision tree

9:16 Speaker 1: correctly sorted all of the objects, but notice that there's

9:19 Speaker 1: an unused path. This would represent acorns, which are bigger

9:25 Speaker 1: than aliens, but neither blue nor green. In this case,

9:31 Speaker 1: we got lucky and the big acorns still being correctly classified.

9:36 Speaker 1: But we might also ask what about small aliens, And

9:45 Speaker 1: now we see that we have a problem. There were

9:48 Speaker 1: no small aliens in the training data, so our model

9:51 Speaker 1: couldn't classify it correctly. This highlights the importance of having

9:59 Speaker 1: high quality training data that's representative of variations in the

10:03 Speaker 1: real world. In Part one, I mentioned that it was

10:05 Speaker 1: a good idea to have multiple samples for each class

10:08 Speaker 1: in your training data. As you can see in the

10:11 Speaker 1: false color image of our study area, the water has

10:13 Speaker 1: a high degree of variation due to suspended sediments. If

10:18 Speaker 1: we only took one or two measurements for training data,

10:21 Speaker 1: the model might not correctly classify clear and turbid water

10:25 Speaker 1: as both belonging to the water class. Taking many training

10:29 Speaker 1: data measurements ensures that we're providing the model with a

10:31 Speaker 1: full picture of the variability of spectral reflectants within a

10:35 Speaker 1: single landcover class. In this example, I've pulled the data

10:41 Speaker 1: for two of our water training points. I've indicated the

10:44 Speaker 1: actual points these correspond to in the map on the right,

10:47 Speaker 1: where you can see that one sample is very turbid

10:50 Speaker 1: and the other is relatively clear. Providing both samples to

10:54 Speaker 1: the model helps it learn that water can have a

10:55 Speaker 1: wide range of characteristics so that it can correctly classify

10:59 Speaker 1: picksel of water that lie between these extremes. To return

11:03 Speaker 1: to our previous analogy, we're accounting for both big acorns

11:07 Speaker 1: and small aliens. Here, I've laid out a high level

11:14 Speaker 1: overview of the workflow. We'll be covering an r as

11:17 Speaker 1: we build a random forest model for classification, So let's

11:22 Speaker 1: head to our studio to see how it's done. Just

11:28 Speaker 1: as a reminder, you'll have access to the code, data, slides,

11:32 Speaker 1: and training materials via the training page and the rset GitHub.

11:36 Speaker 1: I highly recommend that you watch the demonstration and then

11:39 Speaker 1: explore the code after the training is concluded. I've also

11:42 Speaker 1: precomputed some of the results for the sake of time,

11:45 Speaker 1: so if you try to follow along, you'll probably fall behind.

11:50 Speaker 2: With that, let's build.

11:52 Speaker 1: Our own random forest model. We'll be building a random

11:55 Speaker 1: forest model for supervised classification, and we'll begin by loading

11:59 Speaker 1: all of the live libraries we're using this training. This

12:02 Speaker 1: is the same set we used in the code for

12:04 Speaker 1: part one, and I'm also setting the same random seed

12:07 Speaker 1: so that your outputs will look like mine. Just as

12:13 Speaker 1: in part one, we need to load the raster data

12:15 Speaker 1: for twenty seventeen and twenty twenty four. I'm also going

12:18 Speaker 1: to go ahead and make the same RGB plots for

12:21 Speaker 1: both years as we did in part one, so we

12:23 Speaker 1: have them ready if we need to use them as

12:25 Speaker 1: base maps later. I won't go over this since you've

12:28 Speaker 1: seen the process before, but you can refer to the

12:30 Speaker 1: comments and the code block for a quick refresher. Now

12:36 Speaker 1: we can also load the training data points that I

12:38 Speaker 1: prepared for this training. The points are associated with the

12:41 Speaker 1: image from twenty seventeen, since we are going to build

12:44 Speaker 1: our model on the twenty seventeen data before applying it

12:48 Speaker 1: to the twenty twenty four data. Notice that I also

12:51 Speaker 1: apply the naomit function to remove any points that are

12:54 Speaker 1: missing a class label. The training sets small enough that

12:58 Speaker 1: we could do a manual check for missing gas labels,

13:00 Speaker 1: but this is just good practice.

13:02 Speaker 2: For cleaning any training data. Set.

13:09 Speaker 1: It's also useful to take a look at the training

13:11 Speaker 1: points in the context of the imagery, so let's plot

13:14 Speaker 1: them on top of the RGB plot for twenty seventeen

13:17 Speaker 1: we stored earlier. The points should be colored according to

13:20 Speaker 1: their class. We can confirm that they're representing the land

13:23 Speaker 1: cover types we expect. Here, I'm using a variant of

13:27 Speaker 1: the okapia ETO colors for the points and setting the

13:30 Speaker 1: philosthetic to match the LC class variable. We find ourselves

13:44 Speaker 1: in a situation where the training points and the underlying

13:48 Speaker 1: raster are spatially associated, but the training points don't have

13:51 Speaker 1: any information about the reflectance values stored in the raster.

13:56 Speaker 1: We can extract the reflectance values of the raster at

13:59 Speaker 1: each of the training points using the extract function with

14:02 Speaker 1: the bind options set to true, so that the results

14:06 Speaker 1: include the reflectance values as well as the land cover

14:08 Speaker 1: class labels. We also need to ensure that the LC

14:12 Speaker 1: class variable is stored as a factor, which is the

14:15 Speaker 1: art data type for categorical values or random. Forest will

14:19 Speaker 1: assume we're trying to do regression and throw an error

14:22 Speaker 1: because regression on character.

14:24 Speaker 2: Vectors is undefined.

14:30 Speaker 1: Here's a quick check of our combined training data to

14:32 Speaker 1: make sure everything worked as we expected, and we can

14:35 Speaker 1: see here that we have the land cover class and

14:38 Speaker 1: the reflectance values for each of the bands and our raster.

14:44 Speaker 1: We can also see that lank cover class LC class

14:46 Speaker 1: variable is listed as factor, which indicates that it's been

14:51 Speaker 1: converted to the r factor data type, just like we

14:53 Speaker 1: want it. The reflectance values are still listed as double,

14:57 Speaker 1: which is double precision floating point data. That's a numeric

15:01 Speaker 1: data type. And with that sort of we're free to

15:04 Speaker 1: move on. There are a few important elements to the

15:10 Speaker 1: random forest function call. We'll go ahead and run the

15:13 Speaker 1: function here and then explain what each part's doing. This

15:20 Speaker 1: is a breakdown of the arguments to the random forest

15:22 Speaker 1: function call. These aren't all of the possible arguments, but

15:26 Speaker 1: they're the ones that I think are core to understanding

15:28 Speaker 1: random forest or just practical to know. The formula is

15:35 Speaker 1: a formula describing the relationship between the objective and the predictors.

15:40 Speaker 1: When we type ELS class tilda period, the variable is

15:46 Speaker 1: predicted by all of the variables in data. ELC class

15:51 Speaker 1: is excluded since it's the target of the predictors for classification.

15:57 Speaker 1: This always takes the form class tilda predictors, where the

16:01 Speaker 1: period just stands in for everything that's not.

16:04 Speaker 2: The class.

16:08 Speaker 1: Data is the data from which to draw all of

16:10 Speaker 1: the class and value information. This must contain labels as

16:15 Speaker 1: factors as that factor data type for classification to take place.

16:22 Speaker 1: Proximity equals T. Proximity equals true gives you an output

16:30 Speaker 1: where the values are closer to one when the pairer

16:32 Speaker 1: of observations are more similar. Well, look at how to

16:36 Speaker 1: visualize these later. You can set this to f to

16:40 Speaker 1: false to turn it off if it's not.

16:41 Speaker 2: Important to you.

16:43 Speaker 1: Speaking of importance, importance equals T returns information about how

16:48 Speaker 1: much each of the predictors contributed to the final model.

16:53 Speaker 1: In tree equals one hundred, adjust how many random trees

16:57 Speaker 1: to build. This defaults to five hundred, but we're only

17:00 Speaker 1: using one hundred. Our example is very limited, so in

17:05 Speaker 1: my testing, building more trees didn't really seem to improve

17:08 Speaker 1: the results, and it just takes longer to run. Do

17:13 Speaker 1: trace false. If you set this to one or twue,

17:17 Speaker 1: then it returns verbose information for each tree built. For

17:21 Speaker 1: any number greater than one, it will give the verbose

17:24 Speaker 1: information for every n trees. So if you're doing five

17:29 Speaker 1: hundred trees and you set into fifty or set do

17:32 Speaker 1: trace to fifty, it will give you ten trees every

17:36 Speaker 1: fifty trees, it will give you an output. But now

17:43 Speaker 1: we have a trained random forest model which describes the

17:46 Speaker 1: classification process to return to the example and the slides.

17:50 Speaker 2: You can think of this as being.

17:51 Speaker 1: Equivalent to the decision tree with all of the yes

17:54 Speaker 1: no questions filled out. Importantly, the model object is not

17:59 Speaker 1: a raster we can plot. In order to get from

18:02 Speaker 1: the model object to a raster output, we have to

18:05 Speaker 1: apply the trained model using the predict function. Since the

18:09 Speaker 1: model's already trained, we can apply it to both the

18:11 Speaker 1: twenty seventeen and twenty twenty four imagery. So now let's

18:18 Speaker 1: jump back to the slides to look at how we

18:20 Speaker 1: can quantify the land cover change. Here I've laid out

18:30 Speaker 1: another high level overview of the workflow we'll be covering

18:33 Speaker 1: in r As we look at how to quantify land

18:36 Speaker 1: cover change with a change matrix. We'll apply the model

18:40 Speaker 1: which you just saw at the end of the last segment,

18:43 Speaker 1: compare the model results for two dates, compute the change matrix,

18:47 Speaker 1: and then talk about how to interpret changes overall. So

18:55 Speaker 1: now that we have the two maps of land cover

18:57 Speaker 1: change in twenty seventeen and twenty four, we can create

19:01 Speaker 1: a change matrix which shows how many pixels of each

19:04 Speaker 1: class in twenty seventeen became pixels of the same or

19:07 Speaker 1: a different class in twenty twenty four. To do this,

19:10 Speaker 1: we'll join the single band rasters for each year into

19:13 Speaker 1: a multiband raster and use the crosstab function. I'm actually

19:17 Speaker 1: going to do this in two slightly different ways. The

19:20 Speaker 1: first will return a matrix style table, whereas the second

19:23 Speaker 1: will return the same information in a long format, which

19:27 Speaker 1: is conducive to sorting and a bit easier to read,

19:29 Speaker 1: in my opinion. First, let's look at the change matrix

19:36 Speaker 1: in the typical form. In this case, the rows correspond

19:40 Speaker 1: to the predicted landcover class in twenty seventeen, and the

19:43 Speaker 1: columns correspond to the predicted landcover class in twenty twenty four.

19:48 Speaker 1: It may look a bit complicated at first, but let's

19:52 Speaker 1: just consider the changes in pixels that were classified as

19:55 Speaker 1: water in twenty seventeen. Here we see that of the

20:01 Speaker 1: pixels classified as water in twenty seventeen, about one and

20:05 Speaker 1: a half million were classified as barren ninety thousand were

20:08 Speaker 1: classified as forests, one hundred and sixty eight thousand were

20:12 Speaker 1: classified as vegetation, and one point two million remained classified

20:17 Speaker 1: as water. Notice that when something's classified as a change

20:21 Speaker 1: from water to water, we can instead say that those

20:24 Speaker 1: pixels did not change, and all of these A to

20:28 Speaker 1: A pairs lie along the diagonal of our matrix. Since

20:34 Speaker 1: the diagonal of the matrix represents the number of pixels

20:37 Speaker 1: which didn't change or rather were a change from the

20:41 Speaker 1: class to the same class, and the sum across the

20:45 Speaker 1: rows represents the total number of pixels for that class

20:48 Speaker 1: in twenty seventeen, we can very easily calculate the percentage

20:52 Speaker 1: of each class that remains unchanged. As I've done here,

20:59 Speaker 1: we immediately see that the barren class changed the least,

21:02 Speaker 1: at ninety two point twenty five percent of the pixels

21:06 Speaker 1: classified as baron in twenty seventeen still being classified as

21:10 Speaker 1: baron in.

21:10 Speaker 2: Twenty twenty four.

21:12 Speaker 1: Vegetation saw the most change, as only thirteen point four

21:16 Speaker 1: to four percent of the pixels which were classified as

21:20 Speaker 1: vegetation in twenty seventeen were still classified as vegetation in

21:24 Speaker 1: twenty twenty four. Referring back to our change matrix. You

21:28 Speaker 1: may also notice that the twenty seventeen vegetation pixels most

21:32 Speaker 1: commonly changed to baron in twenty twenty four, and with

21:36 Speaker 1: only a bit more math, we can calculate that sixty

21:39 Speaker 1: six point three five percent of the vegetation pixels became

21:42 Speaker 1: barren pixels in twenty twenty four. Notice that this matrix

21:48 Speaker 1: format gives us a handy from class to class syntax

21:53 Speaker 1: for isolating specific changes between any pairs of classes. The

21:58 Speaker 1: long version I mentioned earlier simply takes all of the

22:01 Speaker 1: information and the change matrix and stretches it out so

22:04 Speaker 1: that each of the possible from class and two class

22:07 Speaker 1: pairs are listed individually. This is particularly useful if you

22:11 Speaker 1: want to filter and sort your data to get the

22:13 Speaker 1: magnitude of change. You could summarize the data further, such

22:21 Speaker 1: as when you're looking for the total loss gain of

22:23 Speaker 1: a particular LC class.

22:26 Speaker 2: In this case, I'm doing vegetation.

22:32 Speaker 1: Now you can see that we've isolated the overall changes

22:35 Speaker 1: in vegetation measured by the magnitude of both gain and loss,

22:39 Speaker 1: and determine that the vegetation class only lost sixty seven

22:42 Speaker 1: point one percent of the pixels it gained To put

22:46 Speaker 1: that another way, even though vegetation experienced the greatest overall change,

22:50 Speaker 1: it still had a gain of about one and a

22:52 Speaker 1: half pixels for each pixel at lost. We can therefore

22:56 Speaker 1: say that the vegetation class expanded between twenty seventeen and

23:00 Speaker 1: twenty twenty four overall, but areas which were previously vegetation

23:05 Speaker 1: didn't tend to remain vegetation. Change matrices give us a

23:10 Speaker 1: really good quantitative insight into the magnitude and direction of

23:14 Speaker 1: change over time, but it's also important to understand where

23:18 Speaker 1: these changes occur in space. We'll return to the slides

23:22 Speaker 1: while I will lay out the process for turning all

23:25 Speaker 1: of this information into a map of land cover and

23:28 Speaker 1: land use change. So we've applied classification to the imagery

23:36 Speaker 1: for both of our dates, and the change matrix gave

23:39 Speaker 1: us some information about the direction and magnitude of land

23:42 Speaker 1: cover change. But now that we've answered the question of

23:45 Speaker 1: how land cover is changing over time, we want to

23:48 Speaker 1: map the changes to figure out where the changes occur.

23:53 Speaker 1: Instead of an overview as I provided for the last

23:55 Speaker 1: two sections, let's look at a simplified example. We use

24:01 Speaker 1: simple mathematical and logical operations to compare land cover maps

24:04 Speaker 1: and create maps of land cover change. A simple approach

24:09 Speaker 1: to finding which cells have changed is to use not

24:13 Speaker 1: equal operation. Going cell by cell, the operation checks to

24:19 Speaker 1: see if the earlier and later land cover classes are different.

24:25 Speaker 1: Where there has been a change, the output returns true

24:29 Speaker 1: or one. Where there's not been a change, the output

24:33 Speaker 1: returns false or zero. The result is a binary map

24:37 Speaker 1: with the same dimensions as the input, which answers the

24:40 Speaker 1: question where change has occurred. You might want to apply

24:47 Speaker 1: this binary map as a mask over the land cover

24:50 Speaker 1: maps to figure out what types of change occurred in

24:53 Speaker 1: the areas we identified with binary mask. In practice, this

24:57 Speaker 1: is really as simple as multiplying the mask by the

25:00 Speaker 1: land cover map.

25:01 Speaker 2: The result is a.

25:02 Speaker 1: Change map which shows only the pixels where we detected

25:05 Speaker 1: land cover change and shows what the land cover class changed.

25:09 Speaker 2: Two.

25:11 Speaker 1: This is very simple to do programmatically, so let's go

25:14 Speaker 1: back to our to see how it's done. Just as

25:20 Speaker 1: we did in part one of this training, I'm gonna

25:23 Speaker 1: use a plotting helper function so that all of our

25:25 Speaker 1: plots use a consistent layout and legend. This is very

25:28 Speaker 1: similar to the plotting helper function in part one, so

25:31 Speaker 1: I won't go over it in detail. Now I can

25:34 Speaker 1: use the plotting helper function to create plots of each

25:37 Speaker 1: of the classified rasters and arrange them side by side,

25:40 Speaker 1: which is what you see here. In the twenty seventeen image,

25:47 Speaker 1: the one on the left, we see a large extent

25:50 Speaker 1: of flooded areas outside of the main course of the river,

25:53 Speaker 1: which are replaced by the barren class. In the twenty

25:55 Speaker 1: twenty four image on the right, we also see some

26:01 Speaker 1: significant expansion of vegetation in parts of the forested area

26:05 Speaker 1: to the south.

26:06 Speaker 2: Of the river.

26:07 Speaker 1: This is giving us visual clues about the strange metrics

26:11 Speaker 1: regarding vegetation we were seeing in the change matrix. Although

26:15 Speaker 1: very few pixels which were vegetation in twenty seventeen remained

26:19 Speaker 1: vegetation in twenty twenty four, the expansion of vegetation in

26:23 Speaker 1: the forested areas explains why there was still an overall

26:27 Speaker 1: gain in the vegetation class. In the slides, I mentioned

26:33 Speaker 1: that all we need to do in order to create

26:35 Speaker 1: a binary change mask is to use the not equal

26:38 Speaker 1: logical inequality operator. So let's do that now and then

26:42 Speaker 1: plot the results, and here we have our binary change mask.

26:54 Speaker 1: That's exactly what he demonstrated in the slides with the

26:57 Speaker 1: small toy model follows exactly the same procedure, and following

27:03 Speaker 1: on from the slides, we might be interested in applying

27:06 Speaker 1: that mask back to one or both of the previously

27:09 Speaker 1: classified link cover maps, which is as symbol as multiplying

27:12 Speaker 1: the original map with this new change mask.

27:15 Speaker 2: However, the TEA package in our gives a.

27:18 Speaker 1: Much cleaner way of achieving the same result, using the

27:21 Speaker 1: mask function, which is much easier to remember, and specifying

27:25 Speaker 1: that we want our mask value to be zero or false.

27:34 Speaker 1: Now we can plot both maps using the same helper

27:37 Speaker 1: function we used earlier. By only highlighting the pixels that

27:47 Speaker 1: experience change between twenty seventeen and twenty twenty four, we

27:51 Speaker 1: can get a better idea of the locations where certain

27:54 Speaker 1: changes took place.

27:56 Speaker 2: If we think of.

27:56 Speaker 1: These maps in terms of the change major, we've essentially

28:01 Speaker 1: visualized everything except for the diagonal of the matrix, which

28:05 Speaker 1: is now ana. We can see in stark contrast, the

28:10 Speaker 1: loss of forest to vegetation and the loss of water

28:13 Speaker 1: to barren, as well as where in the scene these

28:18 Speaker 1: changes are occurring. That map on the right is the

28:21 Speaker 1: same one I had promised to show you how to

28:23 Speaker 1: make at the beginning of this presentation. So for one

28:26 Speaker 1: final time, we're going to return to the slides for

28:29 Speaker 1: a summarine conclusion, and then we'll open up the Q

28:31 Speaker 1: and A session. I will say at the end of

28:34 Speaker 1: this document, which will be on the training web page,

28:37 Speaker 1: I've provided some additional details which I don't have time

28:40 Speaker 1: to get into now, about other ways that you can

28:43 Speaker 1: visualize the results and assess the appropriateness of your remote

28:48 Speaker 1: sensing random forest model. So let's review what we've learned today.

28:56 Speaker 1: In part two, today learned how to recognize the importance

29:01 Speaker 1: of training data for supervised classification. Compare K meines and

29:06 Speaker 1: K and N methods for clustering and classification. Construct decision

29:11 Speaker 1: trees as used in random forest models. Apply a random

29:15 Speaker 1: forest model to NASA imagery to create land cover land

29:18 Speaker 1: use maps at two different time periods, and analyze both

29:23 Speaker 1: numerically and visually the change in land cover and land

29:26 Speaker 1: use over time. So now let's review what we've seen

29:32 Speaker 1: in both parts of this training. Over the course of

29:36 Speaker 1: this two part training, we've hopefully learned how to access

29:40 Speaker 1: NASA Earth observation data relevant to land cover and land

29:43 Speaker 1: use change mapping, convert NASA Earth observation data into distinct

29:47 Speaker 1: land cover land use classes using supervised and unsupervised machine

29:52 Speaker 1: learning classification methods in the our programming language. Recognize the

29:56 Speaker 1: role of classification methods as one part of a change model.

30:00 Speaker 1: During strategy computer a change matrix representing the change in

30:04 Speaker 1: land covern land use between two dates. And finally, create

30:08 Speaker 1: a map in our studio visualizing the differences in lank

30:11 Speaker 1: cavern land use between two dates. A reminder about the

30:17 Speaker 1: homework it will open today following this training and what

30:20 Speaker 1: will be due in two weeks on March twelfth. You

30:24 Speaker 1: can find the homework on the training web page and

30:26 Speaker 1: will submit your answers.

30:28 Speaker 2: Via Google Forms.

30:30 Speaker 1: You will qualify for the Certificate of Completion if you've

30:33 Speaker 1: attended both live webinars and complete the homework assignment by

30:37 Speaker 1: the deadline. Of course, I've been your trainer, but I

30:42 Speaker 1: want to thank the rest of the r SET team

30:44 Speaker 1: as well. You've been hearing me talk for a long time,

30:47 Speaker 1: but there are many other people who putting countless hours

30:49 Speaker 1: to bring these trainings to you. They're the ones who

30:52 Speaker 1: handle the logistics, the review of the materials, and all

30:55 Speaker 1: the other behind the scenes work to make our set

30:57 Speaker 1: trainings possible. You can contact me via email or direct

31:04 Speaker 1: any questions and comments to our set to the r

31:06 Speaker 1: st Gmail address. You can sign up for the rset

31:09 Speaker 1: newsletter to stay up to date on future trainings following

31:12 Speaker 1: these instructions, and here are a couple of links to

31:17 Speaker 1: other resources I personally recommend Geeks for Geeks has a

31:22 Speaker 1: good selection of articles about data science and the random

31:25 Speaker 1: forest algorithm, which covers some of the other details I

31:28 Speaker 1: had to skip over for the sake of time. IBM

31:32 Speaker 1: has a fantastic guide to machine learning which has subsections

31:35 Speaker 1: for supervised and unsupervised models, but also some other types

31:39 Speaker 1: of machine learning that I didn't cover in this training.

31:43 Speaker 1: The r Terra package we've been using is an incredibly

31:46 Speaker 1: powerful set of tools for spatial data analysis and has.

31:50 Speaker 2: A really good set of tutorials.

31:53 Speaker 1: And finally, I want to give a shout out to

31:55 Speaker 1: Josh Starmer's stack quest videos, which are fun and easy

31:58 Speaker 1: to understand videos on statistic machine learning. Whenever somebody contacts

32:02 Speaker 1: me with a question about stats or mL, I checked

32:05 Speaker 1: to see if there's already a stat quest playlist on.

32:08 Speaker 2: The subject and send that to them.

32:10 Speaker 1: First, and for one last time, I want to thank

32:14 Speaker 1: you all for attending the SARSET training. In just a moment,

32:17 Speaker 1: we'll jump over to the Q and A document and

32:19 Speaker 1: address some of the questions you've submitted. Okay, let's get

32:31 Speaker 1: into the Q and A for part two. So question

32:37 Speaker 1: one was actually a question that we had in the

32:40 Speaker 1: earlier session Session A this morning, but I wanted to

32:44 Speaker 1: bring it up again here just in case anybody else

32:47 Speaker 1: ran into the same.

32:49 Speaker 2: The same issue.

32:51 Speaker 1: I was storing the multi band HLS data with the

32:56 Speaker 1: get large file storage. It seems that we had so

32:59 Speaker 1: many people wanting to take this training and trying to

33:03 Speaker 1: download the data at once that we had a rate limit,

33:05 Speaker 1: So I went ahead and put the data up on

33:07 Speaker 1: Google Drive. You're welcome to download it from there instead

33:11 Speaker 1: if we continue to run into issues with the get LFS.

33:19 Speaker 1: Question two, what's the difference between this method and Google

33:23 Speaker 1: Earth Engine GEE? Also, is land cover and land use

33:27 Speaker 1: data for Iraq available only for the year twenty seventeen,

33:31 Speaker 1: so this should be equivalent to the ge.

33:36 Speaker 2: Implementation of random forest.

33:39 Speaker 1: And there may be an already existing land cover and

33:44 Speaker 1: land use data set for Iraq, or you can make

33:49 Speaker 1: your own using hyperspectral or multi spectral imagery using the

33:54 Speaker 1: same process we should today in our or find an

33:58 Speaker 1: example in Google Earth Engine. I know there are some

34:01 Speaker 1: examples out there, one of which I reference in question.

34:04 Speaker 1: For question three with regard to forrest, what classification method

34:11 Speaker 1: should I use in addition to reflectance values if I

34:14 Speaker 1: want to add elevation, aspect, slope, or any other values

34:17 Speaker 1: such as tree type, evergreen, deciduous, Will there be too

34:20 Speaker 1: many variables to consider? That doesn't seem like an unreasonable

34:25 Speaker 1: number of variables to consider if you think that those

34:28 Speaker 1: might be important for the landcover type in your study area.

34:33 Speaker 1: For example, if one of the target classes that you're

34:36 Speaker 1: interested in is a plant that only grows on a

34:40 Speaker 1: south facing slope, adding the slope and aspect, or if

34:46 Speaker 1: you have like a highly varied topographic area, I could

34:51 Speaker 1: see elevation coming into play there. You simply would need

34:55 Speaker 1: to stack all of that extra data with your multi

34:58 Speaker 1: spectral bands the RAST and SPRC functions, just like we

35:04 Speaker 1: did in the first part of both sections to bring

35:08 Speaker 1: our data in before you extract the band information for

35:15 Speaker 1: your training data point locations. As to which classification method

35:22 Speaker 1: you should use, that's going to be highly dependent on

35:26 Speaker 1: the constraints and the particulars of the study you're looking

35:32 Speaker 1: to do. Could be done with random forest, but there

35:36 Speaker 1: might be better options out there, just depending on exactly

35:39 Speaker 1: what it is you're trying to answer. As I mentioned

35:46 Speaker 1: in the answer to question two. Question four, can we

35:49 Speaker 1: do random forest classification in Google Earth Engine? Yes, and

35:54 Speaker 1: there is an example like linked here from the folks

35:59 Speaker 1: at NASA Goddard who have made a Google Earth Engine

36:04 Speaker 1: app for random forest methods to detect mangrove maps to

36:11 Speaker 1: detect mangroves. So it's a very cool tool. You can

36:14 Speaker 1: also find other implementations of random forest and Google Earth

36:19 Speaker 1: Engine for examples.

36:25 Speaker 2: Question five.

36:26 Speaker 1: How were the locations for the training data set generated?

36:29 Speaker 2: Was it done in points on qgis?

36:31 Speaker 1: Yes. I used the twenty seventeen imagery as a base

36:35 Speaker 1: map and drop the points on the base map using

36:40 Speaker 1: qgis so that I could see what label I needed

36:42 Speaker 1: to add. That makes it very easy and quick, and

36:47 Speaker 1: then I could zoom in and out as I needed

36:50 Speaker 1: to make sure that my training data points were accurate

36:53 Speaker 1: and that I was capturing a good range of.

36:56 Speaker 2: The variability within each land cover.

36:59 Speaker 1: Class, which is when I took points for both the

37:02 Speaker 1: clear and turbid water to include in my water class.

37:11 Speaker 1: Question six, what is the most efficient way to obtain

37:14 Speaker 1: a training data set for a random forest model? The

37:18 Speaker 1: gold standard is going to be getting data on the

37:22 Speaker 1: ground through field work and having that tied to a

37:26 Speaker 1: date for which there is a cloud free satellite image.

37:31 Speaker 1: For practical reasons, sometimes that's harder to do than others,

37:36 Speaker 1: so it's perfectly perfectly acceptable to do what I've done

37:40 Speaker 1: here and use high resolution imagery and drop your points

37:45 Speaker 1: on the imagery, just as I described. As far as

37:54 Speaker 1: the efficiency of using points versus lines or polygons for

38:00 Speaker 1: homogeneous areas, it's more efficient to use polygons because you're

38:06 Speaker 1: sampling many pixels all of the pixels that are covered

38:10 Speaker 1: by your polygon at once, rather than taking individual samples

38:15 Speaker 1: of individual pixels at a point.

38:22 Speaker 2: Question seven. The label classes used in the.

38:26 Speaker 1: Training They use the twenty seventeen imagery, but note that

38:32 Speaker 1: the twenty seventeen and twenty twenty four imatry were taken

38:36 Speaker 1: roughly the same time of year, so we can assume

38:40 Speaker 1: that the spectral profiles for each land cover class are

38:43 Speaker 1: similar between the years. That is to say that the

38:46 Speaker 1: vegetation in twenty seventeen should be analogous to the vegetation

38:51 Speaker 1: spectral profile in twenty twenty four, which is why we're

38:54 Speaker 1: able to get away with training the model on data

38:57 Speaker 1: from one year and applying it to two different years.

39:03 Speaker 1: What about metrics, How can we evaluate weight classification performance? Again,

39:09 Speaker 1: this is fantastic. I wanted to get into this, but

39:15 Speaker 1: really wanted this training to focus on giving you an

39:17 Speaker 1: intuitive sense of the how and why of classification. There

39:23 Speaker 1: is an additional information section at the end of the

39:26 Speaker 1: part two document where I cover some of the topics

39:29 Speaker 1: and explore some of the available functions in the random

39:33 Speaker 1: Forest explainer package, which I very cleverly had added to

39:39 Speaker 1: the package. Imports at the top of both of the sections,

39:43 Speaker 1: so you should already have it download if you're running

39:45 Speaker 1: the code. Question nine, when using the r random forest tool,

39:56 Speaker 1: does our randomly create the training and validation data from

39:59 Speaker 1: the entire day set or is this.

40:01 Speaker 2: A manual process? There are evaluation.

40:07 Speaker 1: Metrics that are inherent to the way that the random

40:12 Speaker 1: forest model builds many trees using subsets of the training

40:18 Speaker 1: data points and subsets of the variables for each of

40:22 Speaker 1: those training data points.

40:25 Speaker 2: However, if you wanted.

40:26 Speaker 1: To do a post classification assessment, you could make say

40:32 Speaker 1: a seventy thirty split on your training data, but you

40:35 Speaker 1: would have.

40:36 Speaker 2: To do that manually.

40:37 Speaker 1: Some of the evaluation metrics that are already available to

40:40 Speaker 1: us are things like the out of bag error if

40:44 Speaker 1: that makes sense. If not, there is more information in

40:48 Speaker 1: the Random Forest Explainer package help and.

40:52 Speaker 2: In the additional information section Question ten.

41:05 Speaker 1: What would be the best way to do a multi

41:07 Speaker 1: year analysis of land cover in the same area in

41:10 Speaker 1: several different times months or years. As I mentioned briefly,

41:18 Speaker 1: when doing direct comparison, it's important to make sure that

41:22 Speaker 1: the images you're comparing are in the same part of

41:25 Speaker 1: the phenological cycle. As you can imagine, green vegetation has

41:30 Speaker 1: a significantly different.

41:31 Speaker 2: Spectral profile than vegetation that.

41:35 Speaker 1: Is sinescent or you know, in the autumn as it were.

41:42 Speaker 1: But you can train a model for each each pair

41:47 Speaker 1: of months if there is significant difference throughout the year,

41:52 Speaker 1: throughout the phenological cycle. Otherwise, what you've seen today works

41:56 Speaker 1: year to year as long as as long as you're

41:58 Speaker 1: keeping track of the time of year and the part of.

42:01 Speaker 2: The phenological cycle. You're looking at.

42:15 Speaker 1: Reading question eleven. I'm not sure if this is within

42:21 Speaker 1: the scope of this webinar, but can you talk about

42:23 Speaker 1: the relationship between study scale and resolution of images? For example,

42:28 Speaker 1: if one wants to examine the changes in vegetation in

42:31 Speaker 1: the home range of an animal species that lives in

42:33 Speaker 1: a forest in the DRC in the last twenty five years,

42:36 Speaker 1: pared with changes in the vegetation and the African continent

42:39 Speaker 1: during the same period, require different resolutions.

42:43 Speaker 2: Yeah.

42:44 Speaker 1: So the issue you're going to run into here is

42:49 Speaker 1: that as you move to lower resolution lower spatial resolution

42:54 Speaker 1: satellite imagery to deal with larger study areas, such as

42:59 Speaker 1: a continent scale, you're going to get more mixed pixel effects,

43:05 Speaker 1: which is to say that a single pixel might contain

43:10 Speaker 1: multiple target land cover classes.

43:13 Speaker 2: There are some.

43:15 Speaker 1: Spectral unmixing techniques that can help get around this, but

43:20 Speaker 1: in general you would prefer to be able to use

43:25 Speaker 1: the same resolution at both scales, and even better if

43:30 Speaker 1: they're on a commonly aligned grid. Is it possible to

43:40 Speaker 1: have the R script used for the demo? Yes, the

43:42 Speaker 1: R script, the R markdown files and the rendered HTML versions,

43:49 Speaker 1: which are just a little nicer to look at. All

43:52 Speaker 1: of those for parts one and two are available in

43:54 Speaker 1: the GitHub which is linked there and also on the

43:57 Speaker 1: training web page. Okay, and it seems that questions are

44:12 Speaker 1: already slowing down. I'm actually going to scroll back to

44:16 Speaker 1: the questions from part one and see if there's anything

44:19 Speaker 1: that I think might be important to.

44:24 Speaker 2: Bring up.

44:31 Speaker 1: All of these questions will be posted, so don't worry.

44:36 Speaker 1: Don't worry if I bring something up that you haven't

44:38 Speaker 1: seen yet. Oh yes, the other thing that I would

44:50 Speaker 1: like to bring up, which was question thirty eight in

44:55 Speaker 1: the first session, which satellite data set is best for

45:00 Speaker 1: land use land cover assessment from nineteen eighty to twenty fifteen.

45:04 Speaker 1: What do you recommend and why? This is going to

45:07 Speaker 1: be dependent on your study area. But I do want

45:10 Speaker 1: to point out that it is best practice to ensure

45:16 Speaker 1: that you're using the same sensor for both your previous

45:21 Speaker 1: and current or each of the time steps in between,

45:25 Speaker 1: for your land cover and land used analysis. Random forest

45:30 Speaker 1: will only work if the bands are exactly as they

45:33 Speaker 1: were in the training data. So if you were to say,

45:39 Speaker 1: train your model on landsat and then try to apply

45:44 Speaker 1: it to Tanager data, which has a different set of bands,

45:49 Speaker 1: the random forest model would fail, So that is a

45:52 Speaker 1: consideration when working with these longer term projects. Given the

46:05 Speaker 1: working that spatial data in r can be computationally intensive,

46:09 Speaker 1: what you recommend for this type of analysis, specifically, what

46:11 Speaker 1: spatial resolution and approximate raster size and number of pixels

46:15 Speaker 1: would be reasonable to handle on a standard personal computer.

46:19 Speaker 1: This is something that has been greatly improved in just

46:23 Speaker 1: the past few years with the Terra package which we

46:28 Speaker 1: used in the demonstration today and on Tuesday. There is

46:36 Speaker 1: the capability of doing on disc processing, which means that

46:40 Speaker 1: you're less limited by the RAM that your personal computer has.

46:48 Speaker 1: I have done all of this on a standard consumer

46:52 Speaker 1: grade laptop and these images are roughly on the order

47:01 Speaker 1: of half a gigabyte, but you could reasonably do images

47:07 Speaker 1: up to say five gigabytes. It's really just depending on

47:13 Speaker 1: what makes sense for your study area, and there are

47:16 Speaker 1: always ways that you can go about letting the process

47:21 Speaker 1: take longer or spreading it across multiple cores to conserve

47:26 Speaker 1: your computational resources.

47:37 Speaker 2: Question fourteen.

47:38 Speaker 1: My supervisors suggested to use the landsat and dem is

47:42 Speaker 1: it possible to lay them out together for LULC analysis. Yes,

47:49 Speaker 1: I don't recall which question that was, but the process

47:52 Speaker 1: would just be to stack your rasters before.

47:58 Speaker 2: You do the training data extraction.

48:00 Speaker 1: So you would take your multi band raster and combine

48:04 Speaker 1: it with the DEM data before you do the extraction,

48:08 Speaker 1: so that you're labeled. Extracted training data points contain all

48:13 Speaker 1: of the bands as well as the DEM value. Question fifteen,

48:28 Speaker 1: how does the spatial and temporal resolution of NASA satellite

48:31 Speaker 1: imagery affect the accuracy of land use change detection? I

48:37 Speaker 1: will take this in two parts because the answer for

48:39 Speaker 1: spatial and temporal resolution are slightly different. The spatial resolution,

48:46 Speaker 1: particularly when dealing with legacy products that have lower spatial resolution,

48:51 Speaker 1: you do run into the issues of mixed pixels, where

48:55 Speaker 1: a single pixel might actually represent a portion of one

49:00 Speaker 1: land cover class and some some portion of another. So

49:04 Speaker 1: it's always best to understand the scale at which your

49:09 Speaker 1: your landscape is self organized. That is, the minimum pixel

49:14 Speaker 1: size you need to be able to distinguish one land

49:18 Speaker 1: cover from another.

49:23 Speaker 2: In terms of the temporal.

49:25 Speaker 1: Resolution, You're not only dealing with the temporal resolution in

49:30 Speaker 1: terms of revisit time, which, depending on factors like cloud cover,

49:36 Speaker 1: might limit your ability to get accurate imagery at exactly

49:43 Speaker 1: one year or two years on, which is necessary for

49:49 Speaker 1: addressing the seasonality of things like vegetation, but also the

49:55 Speaker 1: period of record. So if you're looking to go back

49:58 Speaker 1: a very long time, your or limited by what products

50:02 Speaker 1: have been available for that long And the further back

50:07 Speaker 1: you go in time, the lower the spatial resolution tends

50:11 Speaker 1: to be, so there's sort of a trade off there.

50:36 Speaker 1: And just a reminder, all of these questions will be

50:39 Speaker 1: up on the training web page within a week after

50:41 Speaker 1: I've had time to go through and clean up and

50:44 Speaker 1: answer everything more thoroughly.

51:07 Speaker 2: And yes, thank you.

51:08 Speaker 1: I see in the chat Elizabeth, the link on slide

51:12 Speaker 1: thirty nine does appear to be incorrect, and that is

51:15 Speaker 1: the correct link. We'll have that fixed on the upload

51:19 Speaker 1: to the training web page. But yes, thank you for

51:21 Speaker 1: catching that. It seems I copied the same link twice.

51:30 Speaker 2: And since it.

51:31 Speaker 1: Looks like there are no more questions coming in, I

51:34 Speaker 1: think we can go ahead and call it a bit early.

51:37 Speaker 1: I do want to say one more time, I really

51:39 Speaker 1: appreciate everybody attending both these sessions. Don't forget to do

51:42 Speaker 1: the homework, and we'll send out that certificate of completion.

51:47 Speaker 2: Thanks again, everyone,