NASA ARSET_ Estimation of PM2.5 from AOD – Methodologies and Available Datasets

0:02 Speaker 1: Welcome to Part two of our aur set training on estimating surface PM two point five using satellite data and other information sources. In this part, we'll be learning about the estimation of PM two point five from AOD and discussing the methodologies and available data sets using these methodologies to create PM two point five estimates. The trainers for this part will be doctor Aaron von Donkolar, a research associate at the Washington University in Saint Louis, doctor Paulan Gupta, a research scientist in Godard Space Flight Center, and they'll be joined by doctor Jin Juncio, a assistant research scientist at Morgan State University.

0:45 Speaker 1: Our objectives for this training will be by the end of part two that participants will be able to explain the geophysical, hybrid and machine learning methodologies used to infer surface PM two point five from satellite, AOD information and other data sources. Second, we hope you'll be able to differentiate between these available data products for surface PM two point five from NASA and the Washington University in Saint Louis based on their methodologies and the strengths and weaknesses of the two data products.

1:17 Speaker 1: And finally, you'll be able to use NASA tools and the SATPM website to access these surface PM two point five data products for a region of interest and a time period of interest to you. Recall from Part one last week that PM two point five is a measure of the in situ mass concentrations of particles smaller than two point five microns and aerodynamic diameter, and that aerosol optical depth or AOD is an optical measurement of the atmospheric column aerosol loading, that is, the total presence of aerosols in the atmosphere from the surface up to the top of the atmosphere.

1:53 Speaker 1: AOD can be well related to PM two point five under certain conditions, but that relationship will do grade or break down if, for example, the aerosols are prevalent in the atmosphere well above the surface level, if coarser particles that has particles larger than two point five microns are dominant among the aerosols, if the aerosols are absorbing water, and if the humidity is variable and the particles absorb water under high humidity conditions. If the atmospheric or the surface conditions prevent reliable retrieval of AOD, for example, over highly reflective surfaces or under dense cloud cover, if the spatial resolution of the AOD does not capture the local PM two point five variability as measured by ground based PM two point five monitors, or if the timing of the PM two point five measurements does not well coincide with the satellite overpasses.

2:50 Speaker 1: As a reminder, if you have any questions during this part, please put them into the questions box and we'll address them all at the end of the webinar. You can put the questions in as you go. We'll collect those questions and those questions, along with the answers, will all be posted in a document which will be available on the training website about a week following today's training. So with that, I'll hand things over to doctor Aaron von Dongler to talk about geophysical and hybrid PM two point five estimation methods.

3:24 Speaker 2: Thanks Carl So. Last week we touched on using a simple linear regression between ground based observations of PN point five and satellite retrievals of aerostoptical depth as it means to use AOD to locally estimate P and point five concentrations. We saw that in certain cases this simple approach works really quite well, but in others there can be some severe limitations. For this part of the workshop, I want to focus on two other methods of using AOD to estimate P and T point five that specifically seek to either avoid the use of ground based observation altogether or at least ensure the information they provide me applied over a larger area.

4:01 Speaker 2: I wanted to start by showing the density of publicly available ground based P and twenty five observations around the world. Here of updated figure from Martin at Ull twenty nineteen showed the average population weighted distance to ground based monitors for countries around the world. You'll note that this average proximity can vary quite significantly, with populations in large regions in North America, Europe, and China is well a few others, typically residing within less than fifty kilometers in the nearest P and twenty five monitor.

4:32 Speaker 2: Populations within South Asia as a whole typically reside within seventy kilometers, Central Latin America within one hundred and thirty klomebers, and North Africa at least typically within a few hundred clombers. Even over a monitor dense region such as North America, the typical distance can vary significantly, from less than fifteen kilometers over western states to twenty five to fifty over more central ones. The net impact of all this with respect to estimates of P and twenty five from satellite is that any method that seeks to be globally applicable will need to either minimize its direct use of ground based observations or find an effective way to account for how the relationship used between AOD and P five changes with distance from the monitor locations used to constrain that relationship.

5:18 Speaker 2: One way to deal with this challenge is to remove the need for P and twoentty five bonders altogether. To do this, we need to revisit a relationship that was introduced to you last week that calculates AOD given P and twenty five concentration or set of assumptions, namely that skies are cloud free, the aerosols are well mixed, with none above the boundary layer, and last so that the aerosols present are optically homogeneous. That is similar, given these conditions, you can see that AOD is proportional to P and point five concentration itself as well as the boundary layer height and the extinction coefficient, which relates to how much the aerosol in question attenuates that it scatters or absorbs any passing light particle.

5:56 Speaker 2: Effective radius and density are inversely related with the potential for an additional effect from relative humidity if the aerosols hydrophilic. In order to broaden the applicability of this equation, we can relax a few of the initial assumptions and rearrange to solve for p and twoin five given the retrieval of AOD. Here, rather than assuming the aerosols are well mixed throughout the boundary layer, we're going to assume that we know what the fraction of AOD that occurs near the surface up to a heighth delta z and is associated with PAN twoin five.

6:28 Speaker 2: We also remove the assumption that there is no aerosol above the boundary layer and that all the aerosols are similar, instead just focusing on the qualities of those near surface aerosol. With the exception of this fractional term, the formula itself is quite similar, except that now the variables related to aerosol properties are no longer representative of the entire boundary layer, but again just to those aerosols that are near the surface within that delta z height. This equation shown again here forms the basis of the geophysical method to estimate PN twoiny five from ALI.

7:00 Speaker 2: It is theory based, physically driven, and completely independent of ground based P and twoin five observations. The challenge, however, is that it requires some assumptions or knowledge about the aerosols that are producing the retrieved AOD. This includes information on aerosol type which impacts a density, effective radius and extinction coefficient. It also needs an understanding as to how those aerosols respond to changes in relt of humidity. And lastly, it requires some knowledge with the vertical profile, that is, how much of the local airsol are located near the surface or again within that delta zt height.

7:37 Speaker 2: For simplicity, this large massive terms relating AODP and twenty five is typically reduced to a single scalar term called ATA. The challenge of the geophyscal method is how to best represent ata. This is where chemical transfer models play an essential role in most applications of the geophysical method. As a reminder, a CTM is a computer model that simulates atmosphere composition using the physical and chemical equations that govern the state of the atmosphere. It does this by inputting assimilated meteorology and natural and anthrogenic emissions into the model and allowing those equations to represent the atmosphere's chemical and dynamic response, as well as the transport and deposition of these individual chemical constituents or tracers.

8:23 Speaker 2: A CTM provides a complete representation of the atmosphere and in the context of geophyscal p point five, provides all the information we need to calculate ATA. Here we have an example of a CTM predicted ATA, in this case for the month of July twenty twenty three. The different colors are associated with the contributions of different aerosol types to ATA, with yellow fermeneral dust, blue for sea salt, green for organic aerosol, black for carbonaceous aerosol, and red for inorganics being sulfate, ammonium, and nitrate.

8:55 Speaker 2: The ATA values plotted here represents the relationship to instantaneous rather than twenty four hour Pene point five, and you can very clearly see a pulsing and intensity that circles the globe with a diurnal rhythm. This corresponds to the impact of solar heating on bound do lair heights, essentially diluting concentrations at the surface as the bound dolair increases with rising temperatures associated daylight hours, which in turn alters the vertical structure to decrease the value of ATA. Also visible are the regional differences in the particular aerosol sources and types driving the AOD to Pine point five relationship, such as dust emissions over the Sahara organic carbon VIEO biomass burning were parts of South America and Central Africa, or anthropogenic inergantic aerosol over China, Europe and North America.

9:40 Speaker 2: You can even see the impact of the long range transport of dust and biomass burning plumes, where ATA essentially drops to zero, implying that the aerosols almost exclusively aloft and therefore the relationship between AOD and P twointy five is effectively removed. As is pretty clear from this example, using a CTM to represent ATA has some definis advantages. You can see how the spatial and vertical variability impacting ATA is captured via the model's response to emissions. In meteorology, temporal variability is also captured which can have advantages when compensating for sellid overpast times and sampling.

10:16 Speaker 2: And also quite clear is the global applicability of this approach when compared to limiting yourself to justice locations that have dense ground based observations. Counter to these advantages, of course, CTM inherently has limitations to its resolution, whether by code, structure or resource limitations, and most global models are typically about it a resolution on the order of one hundred klometers or so. Any model is also, of course, only as good as the inputs that go into it, and regional uncertainty in emission, inventories or meteorology will impact the quality of the simulation.

10:50 Speaker 2: It should also be noted that while these simulations can be quite effective at representing general features, individual events are much more of a challenge and often limit it to the most effective applications of a simulated eight to two longer term means. And lastly, even at one hundred klumeter resolution, these simulations are quite computationally intensive to run, which can limit an individual's ability to run these simulations themselves and require a dependency on whatever simulation output is already available via other sources.

11:22 Speaker 2: All that said, a CTM based simulation of ATA is often an effective means to quantify how AOD relates to PM two point five, especially as the distance to the nearest ground based monitor increases. Here you can see an example over Eastern Asia, primarily China, for annual mean values during twenty twenty one. The left hand column shows AOD, whereas the right hand column shows the corresponding geophysical PM two point five. Black dots show the locations of the ground based monitors used for the comparisons in the lower row.

11:54 Speaker 2: In this example, you can see quite clearly that AOD is well related directly to P and TOY five concentrations over this region with an R squared of a round point four. Converting this AOD to PM two point five using a simulated ATA not only allows a direct interpretation of this AOD as PM twoint five, but additionally improves the R squared to approximately point five. This example nicely shows some of the strengths as well as from the challenges of this geophysical approach. You can see how major spatial features in PM twoint five are captured and even improved upon compared to AOD alone, and unlike AOD, the air quality implications can be more directly interpreted.

12:36 Speaker 2: On the side of challenges, there is potential for bias and scatter, which is difficult to evaluate without some level of ground based observations. As a result, purely geophysical estimates are typically best used for large spatial and temporal scales. Of course, ground based observations do not strictly have to be used for only evaluation, and can also be to understand any bias is present in the geophysical values. This idea forms the basis of what is known as hybrid PM tw point five or geophysical hybrid PM two point five.

13:12 Speaker 2: Hybrid PM two point five essentially builds on geophysical PM two point five using ground based observations. To give an example, here, i've shown the difference between ground based observations of PM two point five and coincidently sampled geophysical estimates. As you look at this figure, you'll begin to notice regional structures and features within these differences, such as a large scale underestimate of a few microgrants per cubic meter on the part of the geophysical values in the southeastern US, and a large scale overestimate of the same magnitude over the northeastern Great Lakes region.

13:49 Speaker 2: We don't see these large scale features over western regions, but rather we see a high degree of variability between seemingly neighboring monitor locations. Hybrid methods therefore, look at these features using statistical models and training to predict biases in geophysical PM two point five. In other words, these hybrid methods predict limitations associated with AOD retrievals and ATA simulations and their geophysical implications. So how do we do that well. Multiple statistical techniques can be used to make these associations, but perhaps the most intuitive approach uses multiple linear aggression.

14:31 Speaker 2: From last week's session and from other work, most of you will be quite familiar with single linear aggression, such as the example shown in the upper right. Here, a line of best fit is created between a single independent variable X one and some other desired dependant variable, why with a slope of beta one. One way to think of this result is that beta one shows the response of Y to changes in X one. Multiple linear aggression is an extension of single linear aggression, where as the name implies, multiple dependent variables are used.

15:04 Speaker 2: Rather than fitting a line of best fit, you're fitting a plane. But the concept is very similar and the result produces a series of predictor coefficients denoted by betas in this case, that can be interpreted as the linear response of your dependent variable to each independent predictor variable. For some systems, these beta predictor coefficients are a universal truth, that is, they don't change in space or time, but for our particular application estimating the bias in geophysical PM twoenty five, this is unlikely to be the case.

15:38 Speaker 2: As a result, an extension of multiple linear aggression called geographically weighted regression or GWR, is recommended. In effect, GWR allows the predictor coefficients from MLR to vary in space. In the equation shown here, this is denoted by the addition of the I and J subscripts, which are simply identifying different locations on a grid such as the one shown on the right here. In practice, the spatially unique coefficients are calculated by running a series of MLR regressions, one for each location on the grid, where the data points used for each regression are modified based on their local significance.

16:17 Speaker 2: This can either be through some sort of waiting scheme associated with the data points during regression, or their removal entirely if they are not relevant to the location at hand. It's worth noting also that while grids can be regularly spaced, such as I've shown here, they can also be irregular in nature. An irregular grid can be advantageous to capture changes in these relationships in regions where we either expect rapid relational changes in beta or have additional training capacity due to more ground based monitors, or alternatively reducing the computational costs associated with running on a fine grid over regions where neither of these conditions are expected.

17:00 Speaker 2: Okay, so what variables are going to be most effective at predicting the geophysical bias? Well, the ants would really tie back into one of two sources, either uncertainties in the retrieval of AOD itself or uncertainties in the relationship that's being used to relate AOD to PM two point five. Between this week and last, we've covered a lot of the sources of this potential uncertainty, which typically falls into four broad categories. Firstly, composition, Different aerosol types can be associated with different artical properties that may challenge AD retrievals, but probably more impactful in this context is the associations they can have with different emission sources, vertical profiles, and growth factors, all of which impact the AOD to PM two point five relationship.

17:49 Speaker 2: In this way, the presence of a particular aerosol component, such as mineral dust, for example, could be insightful due to uncertainties in the simulated representation of its emission transport, which in turn have the potential to bias ATA. Similarly, meteorological inputs also feedback on ATA within a chemical transport model, with local winds and temperature impacting several natural emission sources for example, or atmospheric conditions impacting the chemical production of aerosol. Land type information can also be related to emission type uncertainties, but primarily I tend to think of this sort of predictor as being associated with uncertainties in the AOD retrieval itself.

18:37 Speaker 2: The dominant assumptions in satellite aidor retrievals typically relate to the amount of light that is reflected off the Earth's surface, which naturally is strongly connected to the type of surface that is doing the reflecting. Urban and desert biases, for example, have both been documented and improved upon in AD retrievals over the years, but surface type remains a solid predictor of some of the challenge is facing an AOD retrieval, and any bias in AOD will directly carry forward into a bias in geophysical PM two point five.

19:09 Speaker 2: Lastly, elevation or more specifically, changes in elevation, can be an important predictor in geophysical bias. In this case, what is being represented is not so much of an error on the part of a model to simulate ATA, but rather changes in the AOD to PM point five relationship that are occurring at a resolution below that of our simulation. You'll recall that simulations are often run at fifty or one hundred klomebers, a mountain or valley can often occur at scales well below this, with obvious implications for how AOD relates to PM two point five.

19:44 Speaker 2: In essence, a predictor such as elevation changes is not so much accounting for errors in a model which could be correctly represent representing the average conditions at the simulated resolution, but still need adjustments to account for the impactive features that occur below the model's resolution. Bringing this all together, we end up with a GWR based equation such as you see here, where the difference between the ground based observations of P and point five and their coinstantly sampled geophysical estimates are locally regressed against a series of predicted variables associated with simulated composition neear logical parameters, elevation parameters, and land type parameters, and below you can see an example of the results that can be obtained.

20:30 Speaker 2: On the left we have the original biases that were observed against the ground based observations, and on the right we see the total GWR predicted bias in the geophysical PN point five. You can see other This approach captures the major features we discussed earlier, such as the overestimate in the Great Lakes region, but beyond that, it predicts that this feature extends further north and west into parts of the Canadian Prairies, a region with far less ground based constraints. One of the real strengths this GWR technique for predicting biases compared to some recent and arguably more advanced machine learning techniques, is a relative ease of understanding the impact of individual variables and how they shape the overall bias prediction.

21:14 Speaker 2: For example, I've plotted here both the predictor coefficients and that impact of the nitrate and percent urban variables on the overall predicted bias. As you can see in this case, nitrate large explains give shape to that Great Lakes Region over estimate, as well as its extension to under monitored regions. This opens up a question about what it is about nitrate that produces this bias, in this case potentially related to uncertainties in either the chemical reactions of all the local emissions.

21:45 Speaker 2: But knowing these connections can in turn direct developments and improvements within the CTM, which can then feed back into the original geophysical p in twenty five with potentially global benefits. You can also see the advantage of allowing predictor coefficients to vary in space. Again shown the left hand panels. Here is a very clear East West divide in the strength of association between urban content and predicted geophysical bias, with the coefficient approximately tripling over western North America compared to the east.

22:17 Speaker 2: Given the fine spatial scale at which present urban impacts hybrid PM twoint five, this may suggest something regionally unique in the way urban surfaces impact the AOD retrieval, or it could be a resolution impact on ATA where the urban environment impacts its relationship on a scale below that of a simulation more strongly in the west. In either event. Putting the impact of all these predicted variables together, we can demonstrate how hybrid PM two point five offers some significant gains compared to geophysical PM two point five alone.

22:53 Speaker 2: Here I've shown an overall global map of PM two point five provided by the hybrid methodology as as well as the improved agreement it offers, with an R squared increasing to zero point nine from approximate point seven, and that root means score difference decreasing by about a factor of two. One of the big strengths of this approach is that it builds on the fundamental strength of the geophysical method while simultaneously taking advantage of the constraint provided by ground based observations.

23:22 Speaker 2: There are, of course, limitations, and first and foremost, unlike purely geophysical estimates, we now require ground based observations, and the challenge becomes how to incorporate those measurements in a way that is most representative of your particular region of interest. Related to this, setting up a model such as GWR or alternatively, machine learning based algorithms like a convolutional know network or Ingredient boosting can be quite complicated, including but not limited to the selection of appropriate predictor variables.

23:54 Speaker 2: Lastly, evaluation of these models, like any statistical models, can be quite challenging, and to that point, appropriate cross validation is key. It's relatively easy to create a model which is very high agreement against the data points and locations that we're used to train it. It can be much more challenging to determine how well your model performs at other locations and times. With this in mind, I'm going to spend a few minutes touching on a few different methods of cross validation. Cross validation is a technique that's used to understand the uncertainty of statistical models.

24:29 Speaker 2: It involves the creation of a subset of data points that can be used for independent evaluation of that model. The idea being that they provide insight into how well a model will perform a part or away from the data that has been used to train it. Cross validation can be time consuming and realistically often has to balance its complexity, accuracy and representation. Above all, the cross validation data set must maintain what I would call true independence. The nature of which may depend on the questions being asked.

25:03 Speaker 2: For applications making use of the resulting model, The most basic form of cross validation is what is referred to as random cafold cross validation. Here, we randomly withhold the set fraction often ten percent of the data set from model training and use the remaining ninety percent for training purposes. We do this multiple times until we've withheld a sufficient amount of the original data set from at least one of the cross validation models, and use the cross validation models predictions of the withheld data points to evaluate the model quality.

25:36 Speaker 2: An advantage of this approach is that it's fairly simple to understand and implement. Is also perfectly suitable to many situations where the points in the data set being modeled are themselves fully independent of one another. For our particular application, those last point is often not the case, with ground based monitors often being clustered around urban centers and the concentration of PM two point five present at one station being very much connected to the concentration that we might expect at another.

26:07 Speaker 2: Buffered cross validation methods have been developed specifically to address this last point. The concept is that when a particular data point where station is withheld for cross validation. Not only is that point withheld for model training, but also any sites located within a given spatial distance. As with kfold cross validation, multiple models are trained and used to evaluate uncertainty, with the distinction being that the sites excluded from each cross validation model are not only the cross validation evaluation sites, but also those within the spatial buffer.

26:42 Speaker 2: The initial implementation of this approach was called buffered leave one out or blue cross validation. It essentially withheld one side at a time pluss buffer points running through each data point in the overall training data set. This is effective, but as you can imagine, is also computationally very expensive, as you need to train a unique cross validation model for each point in your data set. Buffer leave cluster out, and buffer leave isolated sites and cluster out methodologies were developed to combine the improved independence of the blue methodology with improved computational efficiency.

27:22 Speaker 2: Specifically, these methods withhold groups of nearly located sites plus or overlapping buffers out of each cross validation fold, in essence running multiple Blue like models simultaneously, thereby providing similar information. Is Blue both far less training of the individual models. The last type of cross validation that I wanted to highlight is temporal holdback, which is used to address a slightly different question. In this case, the focus of the model being developed is not to represent missing values at different locations, but rather at the same locations as the training data set, just for different time periods.

28:03 Speaker 2: As before, the goal here is to separate a set of data from the training of the model that is independent in the specific way that the model is to be used. In the case of a temporarily focused model, this requires the removal of specific blocks of time from the training process, which can then be used for subsequent validation. This approach does not inform both spatial uncertainties and should not be used in that way, but is quite insightful for specific temporal applications and can be combined with a spacely buffer analysis to provide a complete representation of model uncertainties.

28:40 Speaker 2: Okay, So, in summary, geophysical methods to estimate PM two point five avoid the use of ground based monitors by using the theory based physically driven relationship between total column aerosoloptical depth and PM two point five. CTMs are invaluable for this approach as they are able to represent all the necessary parameters and provide global coverage. Hybrid methods build on the geophysical methods using a statistical framework to predict the bias in geophysical PM two point five, and Lastly, effective cross validation of these or any models must consider how to ensure strong independence of the validation points that are being used.

29:23 Speaker 2: And now we're going to switch gears a little bit and focus specifically on the sat PM two point five data set produced at Washington University in Saint Louis by the Atmospheric Compositional Analysis Group or AGAC. The sat PM two point five data product is the culmination of fifteen to twenty years of research using aerosol optical depth to estimate near surface PM two point five concentrations. The goal of this long running project is to provide a consistent global, long term PM two point five data set, and its current iteration is observationally construc using both satellite AOD and ground based observations within the hybrid geophysical framework that we've been discussing is monthly with regular updates provided annually and at a fairly high resolution on a zero point zero one degree grid, which is about one kilometer.

30:16 Speaker 2: A key feature of this data set from its inception is that it is publicly available currently with its own dedicated website at SATPM dot org. On the right you can see its basic development structure, again following the hybrid geophysical framework, where AOD are converted to geophysical PM twointy five using a simulated representation of ATA, and these values are refined using a hybrid approach. At present, we do maintain a geographically weighted regression type algorithm such as we've been discussing, but primarily we encourage users to make use of a more current algorithm that makes use of a more modern machine learning method convolutional neural networks.

30:59 Speaker 2: I'm going to take a few minutes and address some of the unique challenges in setting up a full fledged hybrid geophysical PM two point five algorithm and how we've addressed these within the sat PM data set. Geophysical PM twoint five all starts with AOD, So the first questioning phase is which AOD retrieval is best suited for PM two point five estimation. The answer, of course can varya both space and time depending on the specific conditions. So to give you a better idea, of how AOD retrievals can vary.

31:29 Speaker 2: I've plotted here four separate AOD retrievals, all monthly means of available data for April twenty fourteen, taken from the same satellite platform. Two instruments represented MODUS and MISER, with the dark Target, Deep blue and MAIC retrievals being applied to the Modus instrument and the MISER instrument having its own specific retrieval algorithm. What will strike you is how in some ways these retrievals are also similar, but in other ways they look quite different. In mind, these are all good, high quality retrievals.

32:02 Speaker 2: Some differences are by design. Dark Target, for example, was never intended to retrieve AOD over deserts, and so data over such regions is simply missing from this retrieval. Deep Blues development initially focused on this very type of surface and well captures the features we might expect over such regions. MEEK specializes in complex terrain, using multiple overpasses to build up a representation of surface reflectants, something which MISER can do in a single overpass, but with far less spatial coverage each time, which results in almost stuttered look for MISER.

32:36 Speaker 2: Despite its high quality of retrieval owing to its reduced sampling of the monthly means shown in all cases, you can see data missing from the far North, where snow cover inhibits the use of any of these methods. So what can we do? We have multiple satellite instruments with multiple AOI retrievals, all which contain useful information, but not necessarily equally under all conditions. In our case, we turned to airnet. Aaronnet was briefly introduced last week, and for our purposes, all we really need to know is that aaronet is a ground based network of sun photometers that provides accurate long term measurements of AOD at different locations around the world.

33:20 Speaker 2: This allows the evaluation of satellite based AOD, but only at locations that have an AARNet site. So the question becomes, how can we extend this site specific information globally to understand the quality of a satellite retrieval anywhere in the world. What we've done is group AARNet locations together and perform separate month specific comparisons for each satellite instrument and or retrieval. These comparisons are grouped by land type DDI or normalized difference, vegetation index, and weighted by distance.

33:54 Speaker 2: We combine the findings of these comparisons based on local conditions around the world to produce a contents consistent definition of uncertainty for each AOD data source. You can see an example of this on the right, where I've shown the normalized root means square difference for Dark Target. Based on this approach. For those of you familiar with this retrieval, it shows a number of features we might expect, such as higher uncertainties over the Western US, where the surface tends to be brighter, presenting a challenge to some of dark Target's inherent assumptions, compared to the Eastern US, where richly vegetated surfaces align very well with those same assumptions.

34:32 Speaker 2: Once we've completed these comparisons for different AOD data sources, we need a way to bring this information together so that we locally rely on each data set relative to its accuracy at any given location. To do this, we employ the relationship shown in the lower right to represent each retrieval's overall uncertainty compared to the total uncertainty of all the other AOD data sources combined. Here are location's final AOD is determined as the weighted average of all available AOD data sources, with each one's weight based on a combination of its local normalized routing square difference.

35:09 Speaker 2: Bias is represented by the slope of the line of best fit and its availability. This equation can be evaluated anywhere in the world, which gives us the overall weighting factors for each retrieval shown here, which provide a measure of the relative local amount of uncertainty for each AOD retrieval. What's encouraging about these results is that these spatial features make intuitive sense based on an understanding of each AOD data source. You can see, for example, dark target contributing heavily over more vegetative regions, deep blue over many of the so called brighter surfaces, and may contributing most over some of the more complicated conditions.

35:49 Speaker 2: We've also brought in CTM simulated AOD as an additional AOD data source, which contributes predominantly over those regions where satellite based retrievals cannot find function, such as snow covered regions in the far North. Here, simulated AOD uncertainty follows a similar concept of comparisons against airnet, but rather it uses categories based on speciation and elevation, which are more globally connected to CTM uncertainty than land surface types in NDBI, and bringing this all together we can produce an overall combined AOD such as shown here, which takes advantage of each data set where it is most accurate.

36:28 Speaker 2: An additional challenge to bringing together multiple AOD data sets is that often there are different spatial scales or time periods. Here I have listed the satellites of retrievals as well as the model we currently use in our sat PM data set. You'll note that some of the earlier retrievals available from seaweeds using deep Blue are available from nineteen ninety eight to twenty ten at a resolution of fourteen kilometers, whereas the most recent editions include a veer's instrument starting in twenty eighteen at resolutions ranging from one to six kilometers.

36:59 Speaker 2: Gs. Chemical transfer model, for its part, is run at a resolution about fifty kilometers throughout this whole time period. One option to combine all these sources would be to simply interplate each source down to the finest one klometer resolution and combined, but this runs the risk of washing out fine scale features when relatively coarse AOD sources are weighted against truly fine ones. To address this possibility, we combine these data sets in a tiered fashion, first averaging all data sets onto the coursest fifty klounder grid.

37:35 Speaker 2: We then apply the ten kilometer relative variation in AOD within that fifty klomter grid according to the satellite based sources. And lastly, we apply the one clometer relative variation within the ten clumdra grid according to MAIC algorithm, which is the only satellite retrieval providing information at such a fine spatial scale. Finally, we linearly interprelate the simulated AOD to PM twenty five relationship down to the one cloonder grid which assumes smooth variation and ATA, and apply it to the one cloonder AOD.

38:13 Speaker 2: As I've stressed throughout this presentation, the geophysical method relies heavily on the inclusion of a chemical transport model. The Atmospheric Compositional Analysis Group is fortunate to be a proud part of the GSM community. GSM is an advanced chemical transport model with a specific mission to advance the understanding of human and natural influences on the environment through a comprehensive state of the science, readily accessible Global Model of Atmospheric Composition. It is continually developed and maintained by an international host of scientists, many of whom are shown in this photograph taken two years ago during the biannual GSM meeting, and from the map below, which shows locations of various GSM contributors for our particular purposes, representing the aerosol driven geophysical relationship between AOD and m PO point five.

39:01 Speaker 2: It is particularly relevant that GSM is open source and community driven. This allows us to include developments specific to our needs, such as recent developments and mineral dust emissions, or particular diagnostics that are well suited to ADA itself. GSM also boasts a detailed aerosol oxiden model representing the inorganic sulfate nitrate ammonium system. Carbonaceous aerosols and natural sources such as mineral dust and sea salts. Emissions are up to date and include meteorologically driven sources such as mineral dust and biogenic VOCs, as well as high resolution anthroogenic inventories such as SAIDs.

39:35 Speaker 2: Lastly, biomass spurning is informed using satellite observations using inventories such as GFAs and g FED. The whole system is driven using assimilated meteorology with variable resolution, but in our case we typically run at about fifty kilometers hybrid PM two point five, as you're aware, depends not only on satellite retrievals and CTM simulations, but requires high quality ground based observations. At AGAC, we actually maintain a database covering approximately fifteen thousand monitor locations around the world.

40:10 Speaker 2: The coverage of this database continues to grow, as you can see from the coverage map shown here, with different colors representing ground based monitors that were brought in during successive SAPPM releases. This database predominantly relies on government based networks and is limited to high quality monitors as opposed to low cost sensors which would complicate the usage in its context. Most of the data sources are publicly available, but there are exceptions which are obtained through direct contact.

40:38 Speaker 2: I've listed the various countries and sources of the monitors we use. This includes data from open AQ, which is an online nonprofit specifically devoted to the aggregation and harmonization of open access air quality data, but additionally, as I mentioned, direct access to many government environmental agencies. Akin to the challenge we face combining multiple AOD sources over a long time period, ground based observations are not temporarily complete. That is, there a very few ground based monitors that have been in continuous operation at the same location for the past twenty five years.

41:13 Speaker 2: As a result, changes to the distribution of monitors over time has the potential to weight the adjustments inferred from these monitors towards regions with the longest histories of monitors or worse introduced temporal changes in these adjustments that are based on changes to the distribution of available monitors over time rather than a meaningful representation of how the impact of bias itself has changed. To help limit this potential impact, we produce a temporarily complete PM two point five record at all ground based monitors prior to incorporating them into the hybrid framework.

41:47 Speaker 2: To complete this task, missing ground based values are inferred using relationships developed during overlapping periods with other available data sources. These other sources include not only geophysical and simulated PIN twoin five, but also other ground based observations from nearby or related stations. Multiple techniques are used to fill in this missing data. All are compared using a temporal holdback, and then combined based on the relative uncertainties. You can see the overall agreement between the inferred and observed P and twoint five and the lower scatterplots shown here, with a left hand panel showing the agreement when other ground based monitors are included within the inference calculations, compared to the right hand panel, which showed the agreement when based strictly on geophysical and simulated P and Twoiny five.

42:35 Speaker 2: A couple sample time series are shown above, with the red line showing the directly observed concentrations, the blue line showing the inferred values that are benefited from other local monitors, and the cyan color showing inferred values where no other local monitors were available. This representation of missing observation obviously adds some of its own certainties, but is overall very much a net benefit for during long term consistency and applying the insight gain from more recent heavily monitored time periods to earlier years.

43:09 Speaker 2: For the actual hybrid calculations, our most recent algorithms make use of a state of the science convolutional neural network or CNN. These types of machine learning algorithms are specifically designed to identify and learn from images, and in this way are highly effective for spatial tasks such as the relationship of geophysical PM twointy five with surrounding predictor variables. We train models separately for each month, which allows us to capture the temporal variability that we would expect relationships between the predictors and the geophysical bias.

43:40 Speaker 2: We additionally maintain an earlier GWR based algorithm, but this is a little bit more of a niche market. It's quite useful for a few particular applications, but in general we recommend the CNN based version. Six. Predictor variables are in line with the GWR discussion we had previously, including satellite based products, CTM simulations and related factors, emissions, meteorology, monitor location details, and geological type of information such as land type and elevation.

44:13 Speaker 2: We spend a lot of time developing cross validation strategies to help us understand the impacts of and ensure an effective model away from ground based monitor locations. These monitors provide an invaluable source of information and constraint, but they also run the risk of fooling us into misrepresenting or misunderstanding the qualities of our data set, especially as we get farther away from these training locations. On the left, here i've show on the effect on our squared of changes to buffer radius during our buffered cross validation analysis.

44:47 Speaker 2: In effect, this can be interpreted is how the agreement of hybrid PM two point five changes with distance from the nearest ground based monitor. For a series of differency and N models that were created during the development of our version six SAT PMD data set, the agreement of a purely geophysical PM twoenty five estimate shown in red, is independent of monitor distance, as shown the flat straight line. Interestingly, all the hybrid models we developed performed similarly at the monitor locations themselves if a buffer were included.

45:20 Speaker 2: However, models that excluded information from a CTM that did not include a physically and chemically meaningful process based understanding the atmosphere fell below the quality of a purely geophysical approach within about fifty kilometers. More advanced models that did include predictors based on CTM output fared better, but again within about one hundred two one hundred and fifty klometers underperformed the results provided by the geophysical PM twoenty five. Our CNN model has been optimized in a way that recognizes the relative strength of the purely geophysical estimates with increasing distance to the training sites.

45:59 Speaker 2: This allows improved agreement compared to the geophysical lessons alone, even at great distance from modern locations.

46:08 Speaker 2: Another way to look at this result is through what's called a shaply additive explanation or SHAP analysis. I'm not going to go through the details of this approach here, but only to say that a SHAP analysis is a method to rank the relative importance of individual predictor variables within a machine learning algorithm. I've shown such a result on the right, with the variables again ranked by their impact on the results of our hybrid CNN. As you can see, the process based predictors and the geophysical PM two pint five itself rank as the most influential predictors, again highlighting the limitations that would be present in a more purely statistical approach that didn't build upon a physically sound framework.

46:49 Speaker 2: And lastly, the results here these figures are briefly cycling through the global monthly hybrid geophysical PM two point five data set that were how to produce and please to make publicly available. The evaluations you see are against the highest possible standards, being both temporarily and spatially buffered. The sat PM two point five data set is particularly well suited for users that want to include the impact of features at a high space resolution and want to ensure consistency both over a long time period but also a distance from ground based constraints.

47:25 Speaker 2: So, in summary, AGAX sat PM two point five offers a global PM two point five data set that uses the geophysical hybrid methodology with multiple satellite based AOD products, an advanced chemical transfer model, and thousands of ground based observations, all within the state of the science convolutional neural network. We focus on consistency and quality, and we're proud to make this whole data set freely and publicly available. To the last section of my part of this talk, I'd like to give a brief walkthrough of the SATPM website and how to access the SATPIM data sets.

48:00 Speaker 2: For that, I'm going to switch over to a demo of the website itself. All right, So the easiest way to access the SATPM two point five data set is via its dedicated website, which can be found at SATPM dot org. The main homepage shown here gives a brief description of the data set with links to many of the associated information sources we've already covered today. One element I particularly like to draw your attention to is this form at the bottom of the page, which allows you to sign up for our mailing list.

48:34 Speaker 2: We primarily use this list to let our users know when an update has been released. Given the active nature of this data set, this happens at least annually is we extend sat PM two point five's time series forward in time, which we also coincide with expanded and updated ground based monitors and any other algorithm developments we've been working on. We really encourage our users to sign up so that they can stay informed about these updates and releases.

49:02 Speaker 2: The menus are fairly self explanatory, with access to our most recent data sets via the data access tab. Here, you'll find our latest global CNN based product V six GLO three, as well as the corresponding GWR based version V five GL six, and while not our focused today, there's also a North American regional CNN based product V six and AO one, which offers a compositional data set over North America following a similar methodology. Selecting any of these such as V six YLO three will then direct you to the product specific page.

49:37 Speaker 2: Here you'll find more detailed information related to this specific version, with a particular focus on any updates compared to the previous version. You also find reference to each data sets associated publication, which will provide you a detailed understanding of the data sets development. A little further down, you'll find details about the format and usage of the data itself, along with with access to the data set. For ease of access, we currently post this data set at both zero point zero one and zero point one degree resolution or about one in ten kilometer, and available from two separate repositories.

50:13 Speaker 2: This courser zero point one degree resolution is much easier for file size to work with and especially for the global files, but the zero point zero one degree files are therefo any projects that require it. For access, the first access point is via box folders using the links provided. To avoid downloading the whole data set, we've parsed the data by regions, say for South America, AF for Africa, GL for Global and A for North America, AS for Asia and EU for Europe. Within these you'll find subjectories for either monthly and or annual mean values, and then the net CDF formatted files themselves.

50:52 Speaker 3: Access is also.

50:53 Speaker 2: Available via the AWS Registry of Open Data via a dedicated S three bucket. Browsing the data via this repository is similar to the box folders, with subdirectories for region and timescale. Data can also be downloaded via the AWS command line interface, with specific instructions given here. Lastly, we recognize that often the questions being asked of our data set don't require access to the full high resolution data set. In fact, it can even be cumbersome to do so. To simplify such applications, we provide a select number of process data sets via the links here at the bottom of each version's page.

51:36 Speaker 2: These links will direct you to process national and regional summaries in a simple text based CSV format. These files can be downloaded and viewed within any number of programs, such as Excel, which I'm bringing onto the screen here. They contain national or subnational population and geographically weighted mean p and two point five concentrations by year covering the entire sat PM time series, as well as a breakdown of the percent of the population above certain thresholds. So if for example, you're interested in how exposures have changed over China for the past fifteen years, you could do a very quick searchdown to where China is being stored here the Chinese data that is, of course, select the last fifteen years or so, as well as the corresponding p twenty five concentrations, and end up by just simply plying that with an insert scatter plot you can see how concentrations have changed or China or any country you're interested in for the past little while.

52:37 Speaker 2: Of course, the exact structure of doing this would depend on your exact software of choice, but it's meant to be a very simple access point for answering these very questions quite quickly. Well, not our main focus, we do offer few tools and examples of accessing and reformatting the data, mainly provided by some of our users, which can help get you started on more advanced applications. The archive in Publications tab provide access to historic versions and to our group's publications related to sat PM two point five, and finally, the support team tab gives you a point of contact if you have any questions need to reach out, as well as specific contact for the members of a group responsible for supporting and overseeing this product.

53:31 Speaker 2: Lastly, if this SATPM product is of interest to you, I'll put in one final plug for joining our mailing list. We don't use extensively and so you won't be fluttered with emails, but it really does allow us to most effectively inform our users when new versions are released. And with that, I'll turn over to Pawan and Junhayung who will be discussing a direct machine learning approach for surface PM point five estimation.

54:15 Speaker 3: Thanks Eren for covering that. Now I'm going to talk about some of the direct machine learning approach to get the PM two point five using satellite data and model outputs.

54:28 Speaker 3: So before I start, I like to introduce three main categories of approaches used to estimate surface PM two point five from satellite data and how direct machine learning fit into that landscape. The first two approach are geophysical and hybrid approach, which we have covered in previous section by Eron. The third is direct mL approach. In this there is no city in the loop. Instead, machine learning models learn the mapping directly from the inputs such as satellite davids and reanalysis variable straight to surface PM two point five.

55:06 Speaker 3: Everything is learned end to end from the data. There are many examples in the community, and we'll explore one of them in here that is marred to CNN.

55:21 Speaker 2: Now.

55:21 Speaker 3: This slide shows how a machine learning based PM point four five product can support applications across three time horizon, the past, the present, and the future. On the left is the historical case answering question like what was PM two point five in twenty ten? Here we use complete retrospective archives, long term satellite data, reanalysis, and ground monitors. These products are ideal for exposure studies, trend analysis, and building decade long records. In the middle one is near real time what is PM to pint five right now?

56:04 Speaker 3: This relies streaming satellite data such as geostationary AEROSO, optical depth and short term metrilogical forecast. These rapid updates supports operational monitoring, wildfire response, dust storms, and public health alerts. And on the right is forecasting what will be PM to point five tomorrow. These models rely only on forecast metrilogy since feature satellite observations are ut available, and emission forecast forecasts can help with planning public alerts and proactive exposer mitigation strategies.

56:47 Speaker 3: Together, these three time horizons shows how machine learning can help us understand past air quality, monitor current conditions, and anticipate future. The typical workflow for machine learning based PM two point five estimation. We start with satellite aerosoloptical depth and reanalysis AEROSO product as the primary input. Then we bring in axillary variables such as humidity, boundary air, high temperature, windland cover, mostly from re analysis or forecasting models like AR five and MIREMERATU.

57:33 Speaker 3: These fields helps the model understand meteorological context. Finally, we use ground based PM to point five observation from networks like air now and open Aqui platform as target variables. All of these inputs feeds into machine learning models such as random forest ex you boost deep learning to produce surface PM to pint five estimates. Here I walked through the full pipeline we used to estimate surface PM to point five from satellite observation. We begin by acquiring data satellite aud and ground based PM to point five measurement.

58:17 Speaker 3: Next, we co located these data sets in time and space to ensure they represent the same atmospheric conditions. Then we build statistical or machine learning models that learn the relationship between aerosoloptical depth and surface PM too point five. Once train, the model estimate PM too point five across the satellite grade, providing continuous special coverage. As an optional final step, we convert those PM to point five values into air quality index categories to support public health communication.

58:56 Speaker 3: This is a reference like showing some example of machine learning use cases. Overall, this progression shows how machine learning is increasingly integrated into both observation driven and model driven PM to point five system improving accuracies across time scales. Here I'm highlighting the key strength and limitation of machine learning based PM to point five estimates. Machine learning approaches are powerful because they can capture nonlinear relationship between aerostoptical depth and PM to point five, incorporate diverse predictors or inputs, and perform well even when emission inventories are spares.

59:48 Speaker 3: They also reduce some systematic biases found in physics based models and allow fast inference one strand. However, there performance depends heavily on the density and quality of ground monitors. They may struggle with extreme events if those conditions are underrepresented in training. Database Generalization across region and time period is sometimes difficult in machine learning approaches. Okay, now, let's move on to the examples of machine learning. Deride PM to point five data sets called bias corrected MARA two PM two point five at our ly scale produced as a NASA's hay Cost project.

1:00:39 Speaker 3: Before we get to bias corrected MARAP to PM two point five, take a look at comparison of several major global PM too point five product. These data sets offer in different temporal coverage special resolution in the method used to generate PM two point five. For example, deepcamp and HAP combine the satellite ground and deep learning approach to provide high resolution hourly or daily estimates. Berkeley Earth relies primarily on the ground reasurement with interpolation, while the Washington University data sets which we just learned about, integrate satellite model and ground input at coursal temporal scale.

1:01:38 Speaker 3: Then, finally, MARA to CNN hay Cast PM two point five product, we will learn more on it incoming slide. The slide provides some more specific details on the MARA to CNN PM two point five data sets, although it says only available until twenty twenty four, but periodically continue to produce the data and currently as we will see, available until twenty twenty five. Temporal resolution is one hour with special resolution is about half a degree, which is original special resolution of MARA to data product.

1:02:20 Speaker 3: Each daily file contained global data in net city of format and the recently publication provides more details on the methodology and validation efforts. Each PM two point five estimation over each Mara to grid comes with a qaflag. This qaflag provides a simple grid level measure of confidence in the estimated PM two point five value. It determines by factors such as proximity to the surface, monitor line covered, type and latitude. Area with more monitoring or a stable surface condition generally receives a higher scores.

1:03:07 Speaker 3: The Quality Assurance flag range from one to four, where one represent low quality and four represent highest. For most quantitative analysis, we recommend using data with a flag value of three or four. The map on the right shows how this quality level vary across the globe. As you will notice that the flags are higher where the ground monitored density is high. Now we will get into some details on how these PM to point five products are derived. Marito provides PM to point five component at surface such as does sulfate, carbon, etc. And this equation is often used to combine these components to calculate total surface PM two point five.

1:04:12 Speaker 3: You'll notice that it is missing nitrate, which can be significant contributed in certain regions. When we will validate these across the globe, we found large biases and comparison with individual station, although it shows good consistency over larger areas. Here is the map showing its global performance against ground monitors in year twenty twenty. What we see is that bias pattern varies significantly by region and bi aerosol types. In places like California, Eastern China, and part of Middle East, marit tend to overestimate PM too point five.

1:05:08 Speaker 3: In contrast, regions such as the indogntetic plane and Southern and South America shows under estimation. These regions regionally varying biases highlight y machine learning correction is so important for producing PM too point five astimates.

1:05:35 Speaker 3: In this slide, we summarize the key mara to input be used to model PM too point five by capturing the major process that derives aerosol formation, transport and removal. On the aerosol side, we include black carbon, organic carbon, dust, sulfate, ESOTO, surface mass concentration as well as total extinction at five hundred fifteen nanometer that is our alsoloptical app These variables help represent both primary and second reparticle sources. On the meteorological site, we use surface pressure, humidity, temperature at multiple levels and ten meter wind component to capture boundary dynamics and atmospheric mixing.

1:06:30 Speaker 3: All inputs are taken at their native merit to spatial resolution and at hourly timing scale, ensuring consistent alignment across the data set. Here we describe how we use a reference grade PM two point five measurement collected from open eq platform as ground truth for training and validating our model. We begin by collecting measurement from more than five thousand monitoring station worldwide and collocating each site with the nearest Marato grid cell. We then temporarily match the ely PM to point failue with the crossmending Marato inputs.

1:07:19 Speaker 3: After collocation, we apply a strict quality control to remove outliers and invalid records, resulting in more than thirty million varied hourly observations. For model development, we used year twenty eighteen and nineteen data and for training, and for reserved the data from twenty twenty as an independent validation period to assess the performance. Here is a full workflow used to produce the bias corrected PM to point five data sets. As we described earlier, we start with especially and temporarily collocating marat meteorological and aerosol input with open EQ surface observation.

1:08:12 Speaker 3: Then we apply five different aerosol aware data processing strategies, each capturing PM two point five behavior from a slightly different perspective. The final product is a single bias corrected PM to point five estimate for every single grid cell every hour, with improved accuracies and regional consistency. Here is some more details on five day different strategies. Each strategy train its own independent convolution neural network, but they differ in how aerosol information is organized. For example, M one is our baseline model using the full data set with no studyfication.

1:09:01 Speaker 3: M two classified data by dominating aerosol species h each other, while M three assigned dominating species species by station for the data period. M four assign each mirror to grid cell in dominating species across the three years data sets, and M find find that by merging sulfate and cars aerosol as a single dominating species. Finally, the ensemble model, which is M six, combine output from all five approaches to produce a single more robust PM to pint five estimates. Here we show how the ensemble model M six improve PM to pint five estimates compared to the RAWMERA to data products.

1:09:59 Speaker 3: On on the left you see the training data which come which has data from year twenty twenty eight to twenty twenty nine. The red line shows MARA II and the blue line shows the M six output, and on the right is the validation period which is an INDEPENDENTATA set are coming from year twenty twenty two. So we perform this evaluation by binning observed PM two pointween two contiles and comparing model values across full concentration range. The blue line represent ensemble which closely track the oneine one line in both the training years and the independent twenty twenty validation journalists within the plus minus two sigma certainty weight.

1:10:51 Speaker 3: In contrast, the red line shows that the raw ERA to systematically overestimate low PM two point five values and understimate high concentration. These concentrations depend bias are clearly visible, and the results demonstrate how the ensemble approach using CNN substantially reviews these biases.

1:11:21 Speaker 3: Here we compare regional PM two point five biases in raw MARA to product with CNN corrected data. So as we have seen earlier. In the top panel, MARA to exhibit large regionally consistent error over estimation by more than twenty microgram per cubic meter in places like California, China, Medalist and understand entition by more than fifteen microgramperm meter over endopendetic plane in parts of South America. The bottom panel shows how CNN greatly reduce these biases, producing more localized and better calibrated pattern across diverse aerosol regimes.

1:12:11 Speaker 3: CNN correct PM two point five also has some limitation and it over predicts in eastern Southeast Asia at high concentrations.

1:12:24 Speaker 3: Here is another performance metric showing the performance of CNN versus RA MARA two PM two point five. The bar shows the medium index of agreement, which measures how well the estimations capture both the magnitude and pattern of observed pm too point five closer to one is better. Across every region, CNN performs sustentionally better. The improvement are especially strong in South Asia and Southeast Asia, where agreement nearly doubles compared to MARA two. Europe and the United States also shows clear gains.

1:13:04 Speaker 3: The key TAKEABA is that CNN more accurately represents both special and temporal variability a PM to point five, providing a more reliable global data sets than the raw MARA to PM two point five. So, in summary, Mara tow CNN hey cast PM too point five data sets provides long term global hourly data improve PM TOO point five estimation over Mara I bias corrected using ensembles, CNN approach and quality AWAID data sets. Now we'll look how to access these data sets. The slide kind of summarize the different ways in which a user can access the PM to point five data sets.

1:13:58 Speaker 3: The data ending page is the best starting point, offering an overview of product along with key links. The read we file provide detailed documentation on file structure and variable conditions. Users who prefer direct download can use the online archive or Earth Data search to retrieve files for their specific time period. For cloud based workflow. The data set is also available on AWS. Open Deep is another option which enables remote access through tools like Python, matlib or R and Giovanni for an easy browser based interface for subsetting, averaging and visualizing the data sets.

1:14:50 Speaker 3: Here is just screenshot of data lending page at NASA disk which hosts the pm to my five data sets from this project. Next we'll walk through a quick demo of how to access these data in NASA Giovanni tool. So now we are going to look at an exercise using NASA's Giovanni tool to access the merit to CNN PM to PINT five data sets and see a couple of example how we can access and map and do the time stage type of analysis online. So in order to search Giovanni page, I would just go to Google and search Giovanni.

1:15:44 Speaker 3: NASA Giovanni will be much better and you will see the first link which is NASA Giovanni. I'll just click on that. When you first time click on that link, you will notice it will ask for a username and passwords. So if you do not have Earth data log in user name and password, you might want to register and create. It's a free If you already have it, you should be able to log in. My computer already had so it already logged in. And if you do not log into the Earth Data log in, then you will have limited capabilities from the Giovanni.

1:16:24 Speaker 3: You wan't to be able to do everything. So this is homepage of Giovanni. It's a tool which actually allows visualization and analysis of many many NASA's data sets. As you can see on the left side is the list of all the different platforms, variable disciplines the data sets are available. We if you actually want to explore, you can click each of these taps like measurements and you will see a lot of different types of measurements which are listed here. There's like five hundred or more. You can also select by platform or instruments such as like Modi, is Mara or whatever you want.

1:17:14 Speaker 3: You can also subsidt based on the resolution and other things. And then if you go on the top that's where the analysis tools are available. So if you go to the select part, it will give you the list of different type of analysis you can do. You can do maps, comparison, time series, histograms, and so on and so forth. And this is the selection of date starting an end date. And then finally you have a geographical region selection and the geographical reasons selection. There are various ways we can explore that.

1:17:53 Speaker 3: First, so since we are going to look the MARA to c in NPM two point five, instead of going through this whole list, I will just try to search here, I'll just say CNN and hopefully that should pop up. So here is as soon as I do the CNN and do the search. The parameter which I'm looking for is called bias corrected surface total PM too point five, mass concentration all quality level. And then you needs and source and resolutions. And as I mentioned earlier, the data is actually now available until end of last year twenty twenty five, so we process the data frequently.

1:18:33 Speaker 3: So first thing we are going to look we selected the data. Now, first thing we are going to do a time averaged map. Okay, so I'm going to just select a map type and then I will select a date in twenty twenty five last year. And for some reason, I'm going to pick August, let's say August eight, and that is my starting and then same date, I'm going to pick August eighty five, so I'm just for quick analysis. I'm just picking one day, so it has since this is hourly data, so the total data will be twenty four timestamp.

1:19:23 Speaker 3: And then I will select the geographical area. Now, in the geographical area, I can actually select a box rectangular box around any geographic region, or I can actually select specific countries like or many other US states and many other things. So for simplicity, we're going to just select a simple box over continental northern America. I'm going to select something which covers both Canada and US to show a specific case here. So and you can if you're walking through with me, you can select any region of the world you like to select.

1:20:05 Speaker 3: Once you do that, so now we have selected the type of analysis we want to do, which is time abridge map. We have selected the dat range, we have selected the region of interest, and we have selected the variable which we want to plot. Once you've done all of that, you just click on that bottom right corner plot data. Once you do that, the system is going to go and get the data for that particular day, all twenty four time of stems and then start making plot. So here is the plot. It's very quick.

1:20:47 Speaker 3: You have a color scale here on the right, and then you have a map. Now what you see here these high values in Canada is actually there huge fire during this period. So that is why I selected that specific month and day to demonstrate the capture of smoke loom within these data sets. So we can make this plot look a little bit better. So to do that, I will click on the option and under the options again there is option I will change the color scales to a little bit more, to seventy five, and then this palette I can actually select another palette.

1:21:39 Speaker 3: I'm going to select something which is a little bit more. I'm going to select this orange from yellow to oranges. And then also I'm going to apply some smoothing so that the map looks a little bit better, and then the projection is fine, and just say replot. Once I do, you can see that the color scale has changed the values. Since I've only selected the maxim value up to fifty of the high values are masked. So let's go back and actually revise that and maybe go to hundreds and

1:22:27 Speaker 3: do the replotting again. And you can use the lab ramathic scale also if you like to see the plots and difference. So now we can see really nice smoke prooms from these fires showing high concentration of PM two point five here. So this is very simple to use tool. You can do this for any parts of the world. Now let's try another. We will use the same day and we will just do a quick time series over this region and see how the PM two point five is evolving throughout the day. So to do that, I'll just click again bottom right corner which says back to the data selection.

1:23:13 Speaker 3: So once I do that, it goes back to my previous selection. Now I can change the plot type. I'm going to do time series area averaged, okay, and I will keep the day same because I want to see the hourly evaluation of the PM two point five. But I will just focus on this small area where we saw very high concentration of PM too point five. Just the small area because we want to see the time step evaluation. Right, So again I changed the plot type to time series date. I kept the same from zero to twenty three hours and the area I have changed.

1:23:56 Speaker 3: Again, I'll say plot the data and since it is averaging very small area, so it should be very fast if you're doing this over longer a period of time. Many many years or larger area, then the tool can take much longer time. So now you can see a dinal variability on August eight, twenty twenty five, and all the times are in UTC, so make sure that you remember that you can convert that into local time if you want. But you can see the concentrations are very high at the start of the day and they started going down actually, so again I would suggest you to actually play with this tool.

1:24:51 Speaker 3: It's really nice. I can show you some other example which I have run earlier. This is another day where we have done the times. There is from July first to August July thirty first, and this is a different part of the US actually, but it does show some increasing trend and again this increase is actually resulting from the transport of the smoke in that part of the region. This is another one from August. You can see from earlier August there were a lot more PM to point five concentration and August and this again influenced by the those wildfires which we have seen.

1:25:32 Speaker 3: So it's a really nice and easy tool. I would strongly recommend you play with different types of analysis you have here. You can do the histogram how the PM too point five actually looked on that particular area. So it will basically take all the data in that box which we have selected for that particular day and do a quick histogram to understand what kind of values we see. So you can see very often the values are very low most of the time, but it does shows longer tail with values ranging all the two hundred hundred twenty which we saw earlier in the map.

1:26:21 Speaker 3: Also, so you can actually do a lot of good things. You can also download this as a PNG or net city of files, and then there are other options. So I hope this tool is useful and next week CALL will actually go through this data sets a little bit more and provide you more ways in which you can actually analyze this data along with the ground data and the satellite based PM to point five station. Thank you back to the call.

1:26:58 Speaker 1: So thank you very much doctor Goop and doctor von Donklar for those overviews of the data products and their associated methodologies. To summarize what we discussed in this part, the two data sets we discussed are the Washington University in Saint Louis SAT PM two point five data set and the NASA bias corrected Merit two PM two point five data set. For the SATPM data set, the methodology it employs is a hybrid approach where a geophysical estimate of PM two point five from model simulations and aerosolo optical depth is corrected for biases using ground based measurements of PM two point five on a global scale.

1:27:36 Speaker 1: The key strengths of this data set are its fine spatial resolution. Gridded products are available at point zero one degree roughly one kilometer spatial resolution, as well as having a long record going back to nineteen ninety eight and continuing till twenty twenty four. The data set has consistent performance even far from the location of ground based monitors, as assessed through robust spatial cross validation. A possible limitation to consider when using this data set are the monthly temporal resolution of the products, so this would be unable to resolve variability at daily or hourly scales.

1:28:15 Speaker 1: The data are free and openly accessible via the SATPM website. For the bias corrected Merit II data set, the methodology used by that data set is an ensemble approach where a series of convolutional neural network models are developed and calibrated for different dominant aerosol conditions, and then a final ensemble model is used to select the appropriate model depending on the dominant conditions for any location. Key strengths of this data set are its high temporal resolution. It's available at hourly temporal resolution.

1:28:51 Speaker 1: It also has a rather long data record from twenty to twenty twenty four, and this data set represents PM two point five spatial and ten p poral variability much better compared to the Merit II Global reanalysis data set on which it was based. A possible limitation to consider when using this data set is its relatively coarse spatial resolution. It's available on the original MERRITI grid, which is a point five zero point six two five degree or roughly fifty kilometer spatial resolution grid. These data are also free and openly accessible via NASA Earth Data and for example, you can use tools like Giovanni to do online analysis with this data product.

1:29:33 Speaker 1: In Part three, we'll be looking at a case study where we will download and compare the sat PM the bias correct Merit two data sets to ground based PM two point five measurements for specific region and time period of interest. This will demonstrate both how to access these data sets. And how to evaluate them in a real world setting for practical applications. To be sure that you're ready for Part three, we're asking you to prepare it advance three different things. First, ensure that you have access to Google Collab.

1:30:08 Speaker 1: This is accessible to anyone with a Google account such as a Gmail address if you have that. If not, those are free to sign up for, and accessing a Gmail address or having a Google account will enable you to access Google Collab and use it during the training. Second, you'll need a account with NASA Earth Data. Again, this is a free account. You can sign up for one if you don't already have one, and for the training, you should have your username and password ready to input into the code we're using to allow you to access the NASA data.

1:30:44 Speaker 1: And finally, we ask you to have an open aq account to access an open aq API key, which will also be using to download data from that global data aggregation service that we discussed in part one. If you don't have an account, again, that's free to sign up for, so you can go to the open Aq website and register for an account and they will provide you with an API key as a reminder, the homework for this training will be issued after the last part of the training, which will be Part three next week, and in order to receive a certificate for a completion of the training, you should attend all three parts of the training through the webinars and also complete the homework assignment by the deadline, which will be two weeks after the homework is issued.

1:31:30 Speaker 1: This is the contact information for myself and the trainers who contribute to this part of the training, as well as for the RSET program in general. And here are the references included in the material. So thank you all very much for your attention, and we'll move on to the question and answer portion of the training. All right, So as we transition to the Q and A, I just want to remind everyone we'll try to answer as many questions as possible in the rest of this session. We'll be recording all our answers in a document, including the questions we don't get to in the live session, and we'll be posting that document to the to the training website as soon as we've gone through and completed answering all the questions.

1:32:26 Speaker 1: So let me just check. Okay, here we go. So the first question is related given the non unique relationship between satellite observed AOD and PMT one five, Can conditional density estimation models such as mixed density networks represent which of the uncertainty threshod exceed the probability early warning and so forth? And what validation framework is needed to demonstrate spatial transferability to monitor sparse regions. We had a couple of questions similar to this, so I'll just answer sort of generally here.

1:33:05 Speaker 3: Uh.

1:33:05 Speaker 1: You know, the techniques we've talked about in this training are are two of the techniques, but there are you know, other techniques such as the one you mentioned, can also be applicable, and they may have their own strengths and weaknesses which would have to be determined for a given application. But in general, the validation framework needed to demonstrate the transferability of any methodology, including the methodologies we talked about today. Were the methodologies which which Aaron von Donkel are covered in his portion.

1:33:36 Speaker 1: Uh. Those include buffer leave, cluster out cross validation to assess spatial generalizability and temporal hold back to general to assess the temporal uh generalizability or transferability. So I'll just use again, I'll just use that as a general opportunity general opportunity to answer questions about UH validation, and similarly for a question too. This question is specifically about how models trained in a country can be transferred to Korea, but I'll use this as an opportunity to ask kind of the general question of generalizability and transferability of models, including the models presented here.

1:34:18 Speaker 1: So in this case, maybe i'll I'll ask Aaron and then maybe pol On to comment on this on the more general question of transferability. So Aaron, let's start with you.

1:34:32 Speaker 2: Fuir, So, I mean, in general I would not advise using a statistically trained model that was trying to want region to go to another. It really depends on the structure. Using One of the strengths you know when I discussed the geophysical method was it is really designed to try to be as globally obflitable as possible. But really, in general, you don't want to train a model one addict another, meaning in either event. Really, as you know, Carl, answer the first question, I mean, good robust validation of any model that you want to use is really helpful.

1:35:03 Speaker 2: In such cases. You want to make sure that if you're trying to extend the model beyond where it's been trained you have some sort of independent data sets to try to validate that.

1:35:13 Speaker 1: Okay, thank you, Paul. On anything to add to that or.

1:35:20 Speaker 3: No, I think that is true. Only thing I would add is it really depends. There are many things which you can consider. One is what kind of PM two pint five concentrations and aerosols types are in different regions. So let's say you trend a model over US and you want to apply over Europe. In some cases it will work. But then, as Aaron says, if you have if you do not have any ground measurement to validate, then it is going to be somebody's guests.

1:35:52 Speaker 1: Right.

1:35:53 Speaker 3: We don't know what is the quality of that. So depending on specific situation type of ASO concentration range, meteorological condition, geographical terrain, and other kind of look constraint, the model may or may not work. So it's it's a really case by case determination whether it should, whether it will work or not.

1:36:20 Speaker 1: Okay, thank you. So moving on to question three, was the minimum exploitable spatial and temporal resolution for estimating PM two point five at the scale of a Sephlian city like Waga, dougu or Kaya using mayak or Veer's products, So the Mayak, the Modus Mayak, and the Veers MAAC products specifically are available at a one kilometer spatial resolution, so that would be the maximum possible, the best possible spatial resolution, and these would be these are both polar orbiting satellites, so those would be available at a one day temporal resolution.

1:36:54 Speaker 1: So if you restricted yourself to just those sources of information, that would be the the best possible spatial and temporal resolution you can get. As discussed in the training, there are ways to bring in other types of information to get for the biace correcting Mara two product, for example, to get the higher tempore resolution. So moving on to question four, I think we basically answered this. This is kind of a similar to question one, So we'll just to sort of reiterate yes, it may be applicable and robust validation strategy would be needed to assess that.

1:37:33 Speaker 1: Question five. Are the AODPM two point five conversion coefficients already calibrated for West Africa or the the hell? Or is it necessary to estimate them locally? Yeah, I'll just kind of quickly We talked about that a little bit before as well, in the sort of general question about generalizability. So, yes, these data sets, in the data sets we discussed today include localized calibrations that would make these locally relevant. But if you want to create your own data set you would have to perform those local calibrations yourself.

1:38:08 Speaker 1: Anything else to add on that from Aaron or on?

1:38:14 Speaker 2: No, not, not really, I mean that's yeah. I mean one of the reasons we precee these, that's just to avoid having to your own calibrations. But there are cases string that aren't covered by what we do. In those cases, yes, some local calibration can be can be helpful.

1:38:28 Speaker 1: Well I don't have anything, okay, okay, thank you. So question six, how can surface PM two point five be distinguished from elevated smoke dust or transported aerosol layers when they produce similar AOD values? This is a good question. We alluded to this a little bit in Part one, and then in Part two you kind of saw the specific strategies that the two data sets we talked about use. So, in particular, the SATPM product uses the geos Kim simulation to simulate the vertical profiles of aerosols, and MERIT two uses the go Kart simulation.

1:39:03 Speaker 1: The Aerosol Simulation module, and there are also observational based methods to look at, especially the vertical distribution of aerosol from either ground or space based light ar observations which can provide the vertical distribution of atmospheric particles. Question seven, how is the correlation between AOD and maybe PM two point five at shorter temporal scales such as days or weeks. I'd say there's no sort of general answer to this question. It can depend on a lot of factors. For example, you know, averaging over longer periods of time could average out some noise in the signal, But then you're also looking at changing meteorological conditions, and as we discussed in part one, those meteorological conditions such as humidity of mounted boundary layer height can have a big impact on the AOD to PM two point five relationships.

1:39:58 Speaker 1: So looking at a longer period of time or a shorter period of time may may not help. Yeah, I'll maybe leave it there. Question eight, how can I obtain MYCAOD data to estimate PM two point five? We put some links there for you to access those data. They're available through NASA's Earth Data site, and once we post this document, to the website, you'll be able to access those links directly, but in general, if you go to the NASA Earth Data website and search for the data product, you should be able to find information on how to download it.

1:40:35 Speaker 1: To the method question nine, to the methodologies in estimating pm slash aerosol apply to all other air pollutants, So that's a good question. While this specific methodology wouldn't directly apply, there are certainly ways to modify the methodology or use similar methodologies for other pollutants. Aaron, maybe just briefly comment on your group's work there, sure so.

1:41:01 Speaker 2: I mean, as Carl mentioned, a lot of the concepts become the same, a lot of the details become slightly different. So N two is when our group has worked with a little bit. You know, the idea is very similar. You have a satellite retrieval of total column N two, you can relate that to near service N two using a chemical transfer model, and you can use a statistical framework to further refine those estimates. So by broad strokes, it can be very similar. But a lot of the details are often quite different and the unique challenges that come with that, so there can be a lot of work to just to copy that those concepts over to a different species, but the concepts to broadly hold.

1:41:41 Speaker 1: Okay, thank you. Question ten. Can MLR model for PM two point five bias correction be applied in regions with limited ground based monitoring stations like North Africa? How do you handle missing data? Again? I think, Aaron, if you want to comment on this, but we've sort of touched on this already with some of the previous questions about generalizability. But if you have anything else to add or Pollen, if you have anything else to add.

1:42:10 Speaker 2: Not dramatically beyond what was covered in the presentation. I think this question might have come in a bit earlier on in the presentation, So hopefully we've covered some of this during our talk. But I mean probably a lot of the methods that we're covering here were designed specifically to be more globally applicable trying to handle missing data, fill in missing data, relate those relationships or feel those relationships over broader regions as well, So they're inherently trying to account for a lot of those challenges.

1:42:37 Speaker 2: But again, each region is slightly unique.

1:42:43 Speaker 1: Okay, thank you. Question eleven, which is recommended for bright sparsely vegetated surfaces. I interpreted this as a question about the AOD products, and in general, we tend to think the deep blue AOD algorithm is better for those types of surfaces. And I believe, Aaron, you did cover that in slides twenty eight through thirty four the presentation, where there's some examples of comparing the performance of the different algorithms in different regions. I believe this question did come in, you know, before you got to those slides, So hopefully by referring to those slides, this person can get an answer to their question question twelve.

1:43:23 Speaker 1: Okay, this is pretty specific. When evaluating interurban environmental exposure inequality using the version six Global product TATPM product in dense megacities, we notice the gridded product heavily flattened spatial gradients, i e. Very low spatial coefficient of variation across districts compared to localized ground monitors that show very large microclimatic and seasonal peaks. How do you recommend researchers to account for or bias correct this localized spatial smoothing when conducting fine scale subdistrict exposure and Genie coefficient analyses.

1:44:01 Speaker 1: So, Aaron, I think that's a question for you.

1:44:04 Speaker 2: Sure. So, I mean there can be a number of factors affecting the scatter that you're seeing between the local monitors. I think Carl showed a slide last week that that really highlighted the amount of variability within monitors within a given city, how much monitors can vary even within a one kilometer distance. So it's worth braining mind that the estimates we're providing our average over one kilometer. Beyond that, of course, it is challenging to capture these fine scale features. One of the things within the soldering my presentation is a lot of different resolutions of data sets to go into it.

1:44:36 Speaker 3: And so.

1:44:38 Speaker 2: As much as these statistcal models are attempting to account for this fine scale of variability, there are a lot of resolutions going on, and so we tend to think of it as the one klumeter product may not fully capture those one those subparometer features, even one klmeter features fully, and so we typically recommend averaging over slightly larger areas. Dramatically, you know, up to ten colmers you can see some pretty dramatic improvement, although in my experience a lot of health studies, people tend to move around, and so I tend to think this is not necessarily for most applications a severe limitation, but that may depend on the particular study you have in mind.

1:45:19 Speaker 1: Okay, thank you. Question thirteen. When we build local machine learning models or downscale or validate satellite estimates with limited ground data, our leave one station out spatial cross validation yields a near zero spatial R squared even if temporal validation is high. What are your recommended best practices for validating spatial models and establishing local confidence in satellite derive PM two point five when ground stations are so severely limited. So I would actually say that you're probably doing pretty much the best you can.

1:45:56 Speaker 1: So the leave one station out is probably a good cross validation method for you, especially considering a condition of of sparse ground monitoring stations. If you had more of a dense network, then probably a buffered leave one station out or a buffered clustered leave a cluster out basically method would be probably more recommended. But in the case of not so dense ground measurements that there's probably not much statistical difference between these different cross validation techniques so yeah, I would say, there's there's no really no simple answer here.

1:46:31 Speaker 1: You're I think the believe one station out strategy is probably a good strategy for your case. And the fact that may just be that the methodology you're looking at is not able to give you a good spatial generalization. There may just be too much variability, or you're you're you're missing an information on which would explain that variability. And then my my other recommendation here would be, in such situations where ground monitoring is very sparse, it may actually be better to use a pre existing product like the saven PM product or the bias corrected Barriti product that we talked about today, and use your limited ground based monitoring information purely for evaluation of the quality of that existing product.

1:47:20 Speaker 1: Oh and sorry, I see polland stands up.

1:47:22 Speaker 3: Go ahead, Well, no, go ahead, finish your question. Answer. I wanted to just make an announcement.

1:47:27 Speaker 1: Okay, yeah, so yeah, I think that's basically what I want to say. So I would say the most efficient use of such sparse data is it would really be instead of trying to develop your own model, use that data to evaluate the models that are already out there and see which one is the most appropriate for your setting. Okay, go ahead, Polan, Yeah.

1:47:47 Speaker 3: I just want to, I think, put a node out there. I know several of you are trying to access the merit to CNN using geo Wanni and you're not able to find the data. And I was not aware, but I just found out that Giovanni team is moving that data sets to the cloud. So for time being, they have actually taken it off, so it will take about ten days for them to move it. So after July twenty fourth, the data should be available back in the Giovanna So we apologize for this inconvenience. I think we didn't plant ahead in time, but hopefully the data sets will be available soon back in the system and you will be accessed.

1:48:41 Speaker 1: Okay, yeah, so thank you for that announcement, Pollen. We'll maybe see I don't think that should affect your the homework that we're assigning for this. We're not using Giovanni specifically for the homework assignment, but maybe we'll just double check that that won't affect people's ability to complete the homework assignment. But yeah, thank you for that announcement. All right, So question fourteen, what proportions of nitrate dominated urb in PM two point five are formed from local emissions generated and surrounding regionals or transported across national boundaries, and how do these change under different meteorological and pollution conditions.

1:49:21 Speaker 1: So that's a very complicated question. You may be able to use something like a chemical transport model to get an answer to that question. It's really a little bit beyond the scope of today's training, so we won't go into more detail than that, but feel free to maybe contact us or first explore that training that we put a link to, and if that doesn't answer your question, you might need to contact us for further discussions. Question fifteen, how well does the vertical scaling of GEOSCM capture highly localized boundary layer dynamics and low altitude micro level emission sources in extremely dense megacities And are their future efforts to integrate local high resolution meteorology to improve this?

1:50:02 Speaker 1: So, Aaron, I think you spoke to this a little bit in your previous answers, but if you want to add anything onto this, go ahead, sure.

1:50:09 Speaker 2: I mean, so it's a good question, and I mean I like the idea of trying to integrate higher resolution meteorology to improve local conditions. There has been some work done on like ship emissions representing subgrid variation stuff. I'm not aware of any of that specifically looks at megacities for now. Really, what we're relying on is more of the statistical models to capture those any shortcomings within just came to capture those really high resolution features that maybe may come about because of those features such as megacities.

1:50:43 Speaker 2: It's it's pretty good, we think, but it's certainly an actual physically based model would be even better if we could to pull it off.

1:50:53 Speaker 1: Okay, thank you. Question sixteen. For developing countries which cannot afford ground measurements and they have a little to no ground stations and there's limited technical capacity for advanced machine learning, what would you recommend as the ideal data and method to analyze air pollution there? So, I think that's, you know, partial part of the motivation for this training is the fact that both the SATPM and the bias corrected MERRIT two data sets are global, provide global coverage, and are publicly freely available everywhere in the world.

1:51:25 Speaker 1: The reason, but you know, the reasoning behind that is exactly to provide this kind of information to those who have more limited resources. So using these data sets that are already available to you, and wherever possible, comparing them to whatever local information you do have to establish their data quality would be kind of my recommendation for the best place to start your analysis. Okay, So for question seventeen, given random cafolds, spatially buffered and temporal holdout cross byalidation, which of these gives the most reliable estimate of error when the model is applied to unmonitored locations in future data?

1:52:06 Speaker 1: So I think the recommendation of best practice here is to use both a spatially buffered cross validation to assess the spatial generalizability, that is, the applicability to unmonitored locations, and a temporal holdout strategy to assess the temporal generalizability, so the applicability to future data. Question eighteen, will the retrievals be biased over airnet locations? I'm not sure if this refers to the aerosoloptical depth retrievals or the PM two point five data sets, but I think in either case there really shouldn't be any kind of bias specific to the airnet stations.

1:52:47 Speaker 1: In either of these methodologies. Question nineteen for deep investigations about the problems and possible solutions for Shalian countries and the Sahelian Alliance countries, what kind of partnership is possible for NASA R SET if the University of Thomas Sunkar University. So two points here. First, our set is a training program. We don't really do our own research and investigations. That's sort of something that's done separately by others at our set. But with that being said, you know, the people who participate in this training, myself and doctor von Donkler and doctor Gupta are involved in research around the world.

1:53:29 Speaker 1: So if you have specific questions or you know specific research topics which you think you want to collaborate on, you can reach out to us directly about that. Question twenty. Since process based constraints are interpolated smoothly from the course fifty kilometer model, what limitations or what are the limitations of using the downscaled one kilometer product for subgrid, micro environment or local buffer analyses in a highly complex urban area is Again I guess this is a question for Aaron and anything to add, you know, beyond what you've already said basically.

1:54:09 Speaker 2: No, I mean, it's a great question, as I mentioned before, I mean, actually one thing, I'm really glad to hear that you're thinking this way, whoever asked us questions, is a great way to think about the limitations of the model, and it really highlights why we talk about the value you know, as much as possible average to a larger area. So as much as we do have stysical models and things trying to capture that fine scale resolution down to one kilometer, it's quite reasonable to assume that those really really fine scale features will not be fully represented and they'll higher uncertainties.

1:54:37 Speaker 2: So it's a really good question we are relying on the statistical models to capture that really fine scale and how things will vary over that really really fine scale at least, how that relationship will over fine scale.

1:54:52 Speaker 1: Okay, thank you. Question twenty one, Again, this was a pretty detailed specific question and a bit beyond the scope of this training. I think this is sort of a whole research topic in and of itself, so we can't really provide a simple answer to it, unfortunately. So question twenty two, to what extent does visitor traffic inside and closed museum environments contribute to indoor PM two point five. Again, indoor PM concentrations is not really the be on the scope of this training, and I don't think it's really in any of our expertise unfortunately, so we can't really comment on that in a very informed way.

1:55:29 Speaker 1: So let's go to question twenty three next, scroll down a little bit. There we go, due to the lack of ground based measurements in Africa, would you expect there to be significant uncertainties in the derived PM two point five concentrations there from these data sets? Would you expect similar performance or performance to be similar, better or worse than merit to over Africa? So I think, yeah, maybe to Aaron and then to poll on about regionally specific differences in uncertainties and the products.

1:56:06 Speaker 2: Sure, so yeah, sorry, As the question you know quite rightly implies Africa is a certainly going to be a more uncertain region. There are some really there's some challenging conditions for models to capture both from major dust sources, major byomass bringing sources, and both these methods and painals weak because I think we are trying our best over those regions to speak to some of the strengths of the set PM data set. We really have that observational constraint from satellite, very just hardwired in what we're doing.

1:56:35 Speaker 2: The methods are designed in our by design using sparse ground based measurements as much as we have available to try and calibrate that better. It's definitely a region of more uncertainties, but I would still argue that what we're producing is probably one of the better estimates that is available out there.

1:56:57 Speaker 1: Okay, I just want to.

1:57:01 Speaker 3: Add some the Africa, as Aaron mentioned, is always very interesting reason and there's a lake up ground measurement. We all know about it, right, so it's very very hard to actually assess what kind of uncertaintives we might have. But I just want to let people know that there have been a lot of efforts going on in Africa in deploying low cost census although we know they are not best to use for this kind of validation, but they do provide useful information, and there are several groups in Africa and from other parts of the world who are actually trying and deploying and collecting the data.

1:57:42 Speaker 3: So we are hoping to actually collaborate with these groups to validate our products in future and provide more accurate assessment as we move forward in processing more data.

1:57:58 Speaker 1: Okay, thank you. So question twenty four was a question about actually like future funding opportunities, which is again not really something we're qualified to speak to, but I encourage you to take a look at the nspire's website, which is a place where all NASA funding opportunities are posted for you to look at. So question twenty five, in my own PM two point five machine learning framework SHAP analysis ranked my separate satellite AOD retrieval as the weakest predictor compared to continuous meteorological and spatial features.

1:58:30 Speaker 1: Have you observed similar trends where the model downweights physical AOD in favor of net data proxies. So I'll just speak maybe to my own experience, I would say that this is certainly possible in specific conditions that especially where PM two point five might be highly meteorology dependent. There's also the possibility that there are multiple layers of aerosols in the atmosphere and so the total AOD is really not reflective of the surface concentration very well. And also especially if you smoke, if you focus your analysis on a very small region where the meteorology is similar, and the PM two point five is also similar.

1:59:11 Speaker 1: Statistical methods might just be able to pick out on similar patterns and meteorology and AOD and made the connection between them much more easily than if you tried the same methodology on a larger region or even on a global scale, where the relationship between AOD and PM two point five is is much less consistent, and therefore other information sources like AOD would have a bigger role to play. Paul On or Aaron if you want to add on to that.

1:59:41 Speaker 2: No, no, I think you covered it. Sorry, go ahead, paul On.

1:59:44 Speaker 3: No, No, I was just saying, good thing.

1:59:46 Speaker 2: Well, you covered it very nicely, Carl.

1:59:48 Speaker 1: Okay, thanks, Okay. Question twenty six, how do we ensure the data is used accurately? Okay, this is the pretty broad question, but at a simple level, I'd say, you know, the this course, for one, is trying to convey to you basically what the limitations and the capabilities are of the data. So keeping those in mind as you use the data will help you to use it accurately. And as I think we've said several times, wherever possible, performing your own local assessments with locally collected data is really the best way to establish the quality of these data products for your specific region of interest.

2:00:30 Speaker 1: So anything to add on to that, I'll ho or eron.

2:00:36 Speaker 3: I can add one thing about specifically related to the statistical modeling or machine learning method. One thing which is very very critical whenever we do the statistical or machine learning method is the data distribution. So if you have several input parameter and you're trying to map with the output PM two point five, do check how the data has distributed. Because machine learning algorithm and statistical methods often try to learn what is around the mean and the extreme values become actually has less impact on the relationship which are which is formed, and often you will notice that these methods either underestimate the very high values because there are very few data points in your training and overestimate the low values because that's again very small values in your OWLD data set.

2:01:35 Speaker 3: So the data distribution, checking and adjusting your validation data versus the training data can be an important aspect of preparing your data before you start modeling. Whether you use a multiple linear regression or linear regression or machine learning method, it applies to all of those.

2:01:57 Speaker 1: Yeah, think that's a yeah, that's a really good point of the extreme values, especially if you have an a special interest in the extreme values as opposed to the mean, then that's that's an important caveat to keep in mind. Okay, yeah, let's maybe move on to the next question. Let's scroll down a little bit to question twenty seven. Will the mL based estimate capture events such as dust storms and smoke from fires? So, powin, do you want to speak to this a little bit?

2:02:37 Speaker 3: Sorry I was muted. I think it will capture the mara to as written here in the response, does assimilate satellite based aerosol optical depth and the fire emissions. And if that event is captured by the satellite is specifically from the modies, then the merit should have it captured and the machine learning based bias corrections should actually capture it. In some cases it might if the machine learning training didn't have dust or a smoke type of cases, then it might miss. So there are certain cases where smoke is very thick or dust plume is very thick, and the satellite sometimes get confused whether it is dust or cloud and mask it as a cloud and it does not get propagated into the assimilation stream and that can actually be not seen in the MARI too.

2:03:43 Speaker 3: So I would say it veries case by case, but in general it should be able to capture qualitatively. Quantitatively things can be very different, and depending on which event and how far we are looking from the location of the source, the quantitative numbers can be different. Qualitative lids should be captured.

2:04:06 Speaker 1: Okay, thank you. Question twenty eight, what are the main challenges in making PM two point five products like the Questionington University SAT PM two point five or the Meritus and then hay Cass product from retrospective processing to near real time to delivery. So this is a question about like the data latency and how maybe how that might be reduced in the future. So I guess yeah, Aaron, Aaron, go ahead, sure.

2:04:34 Speaker 2: So, as I wrote, I mean, that's a great question. We'd love to be able to get data sets, data sets out even faster. It's unforced, its often not practical. There is you know, as I've written here, as you gather from the presentation, is an awful lot of different data sets that come together to make these final PM twoint five data sets possible. From biomass burning emission inventories, which seems to be brought you brought into the kepital trust models models needs to be run. One very practical example of a delay that we face is to get the highest quality ground based monitors.

2:05:06 Speaker 2: Often governmental agencies will perform their own internal checks for usually about six months. So really for next year's release, you know which we're playing later this year, we're just starting to work with the ground based monitors now, which are the highest reference grade model that as well as our own internal validations take time. It's really multifaceted why it takes us so long, but I really feel that the time gives us an opportunity to reach the highest quality data product that we can.

2:05:38 Speaker 1: Okay, thank you, I on it.

2:05:41 Speaker 3: I think it applies to the matra TO or any other satellite product in the same ways because they all depends on some ancile data sets. Specifically in case of matter TO, it assimilates to so many data sets, so to get all those data prepared that for assimilation some time, and MERI to is re analysis is always I think several weeks behind because it's a re analysis. The idea is to retrospective analysis. So I think that is why we want to produce the best availabled data sets, and there are some real time and forecasting more data products which have a different way of processing.

2:06:27 Speaker 1: Okay, thank you, so question twenty nine this refers to a specific slide where we're talking about the index of agreement and asking about some of the other metrics. So some of the other slides did have different metrics, especially bias. And then also the paper describing the data set itself, which we'll link here obviously has a much more detailed description of the performance. Okay. Question thirty is this at PM data set available for North Africa? Yes, it is, and was the spatial and temporal resolution it's point zero one degree spatial end monthly ten or a resolution.

2:07:06 Speaker 1: We covered that in the presentations. Question thirty one, okay. Another regional question. In Indonesia, the days with the worst air pollution, for example hayes from fires are often the same days covered by clouds, so these days get removed from the data the AOD data. Specifically, how do we fix PM two point five estimates so they don't end up too low because of this? Aaron, maybe I think you answered this question here, so.

2:07:34 Speaker 2: Sure, I mean, I'll speak to how we address that within the SATPM data set, and maybe Poland has more to add after within sat PM. This is we account for these sampling bias in a few different ways. First off, the ground based monitors themselves we okay you said. First off, the sat AOD we directed to a monthly mean based upon sampling adjustment factors provided by the CTM, so again using that chepical transt model to infer how the days we observe relate to the days we haven't observed, and then providing a more representative overall monthly mean.

2:08:09 Speaker 2: Subsequent to that, the ground based monitors themselves will sample irrespective of that cloud cover. So when we apply those machine learning models or GBR models as a subsequent calibration to the geophysical estimates, that inherently also corrects for any sampling biases that may be from limitations to the satellite retrievals themselves. So there's a few ways in which this is hopefully addressed. None of them are obviously perfect, but there should not be inherently a sampling bias due to cloud covered days in our product.

2:08:44 Speaker 3: Okay, yeah, for Merit too, I think as it is a model produced data, so the data should be available even if there is our cloud. Now, in terms of the bias, I think the bias can arise is because if the satellite a woods are missing, then the model is relying completely on its own capability. I don't think we do any specific correction to actually account for that sampling bias there, but we haven't seen any clear evident in all validation studies that it should it should have a biases.

2:09:28 Speaker 1: Okay, thank you, So question two question thirty two, rather as a follow on, I think asking about which of these satellites would work best for Indonesia where there are cloud covered So just in terms of the this is maybe a little bit more general, but a geostationary satellite like Kimawari offering more frequent observations would give you more opportunities for cloud free observations. However, you know, any of these instruments would be I think similarly affected by the presence of clouds. And I also just want to note we've gone a little bit over our allotted time.

2:10:09 Speaker 1: If it's all right with with Aaron and Pallen, we might keep going for maybe ten more minutes and answer some more questions and then everything anything. We don't get to live we can answer in this document offline. Yeah, so let's keep going for a little bit longer. Question thirty four, you mentioned that multiliner regression is used to predict you a physical bias in PM two point five. Could you specify which independent variables or predictors are typically used in the set pm MLR model. So, I think to save time, we'll just sort of refer you to the publication where these are listed, and then in general they fall into the categories of physical and process based parameters, emissions, meteorology, and site location.

2:10:59 Speaker 1: Believe Aaron covered this in the slides as well. So question thirty five, within the workflow of the six ensemble predictive model developed for the Maritu sine n Heyksta, is that what is the scientific rationale for training five distinct convolitional neural network strategies rather than relying on a single baseline strategy basically the M one model. So, poland if you want to, yeah, speak to that freequently.

2:11:28 Speaker 3: Sure. Sure. So. I think, as we have discussed and seen in many cases, that the relationship between input parameter and the output target variable PM too point five is not linear and it varies as a function of aerosols type, meteorology, range of Pm two point five and many other parameters. And our goal was to what best we can do right, So in order to actually get the best of the data sets, we divided the data into different nominating aerosol component time so that we can get a better sense of how the input to output mapping can be done with more reliability.

2:12:20 Speaker 3: And that is why we actually try to play with different schemes and then we came up with these five different model which provide us an optimized solution. It may not be the best, but it provided an optimal solution which worked globally. And then the ensemble kind of bring them together in a more unified consistent data sets across the globe. So to use the strength of narrow too and their aerosol component, use those different approach to get an optimized global Team nine five data sient.

2:13:04 Speaker 1: All right, thank you. Question thirty six Quick clarification Slide sixty three notes that the mL generalization across region and time isn't guaranteed well. Slide seventy eight cosmeritive product robust onscing regions in years, given that it's trained on twenty eighteen to twenty nineteen dow date on twenty twenty. Could you help me understand what a bust onceing readings and years is based on Powen if you want to, Yeah.

2:13:31 Speaker 3: So, I think we continue to evaluate this data, although in those years which we have mentioned in the those are based on the published work which we did, but we continue to evaluate and I think we haven't seen any degression in the model performance over even beyond twenty twenty one and twenty twenty four, which we have done recently, which is not part of this paper. So I think what we can say is robust over time now over the region. The problem is the same as we have been discussing in earlier question is that if they're not ground stations available over certain regions, then it's very very hard for us to actually even predict what kind of uncertainties or error might be there.

2:14:23 Speaker 3: And that is why we try to provide a quality assurance flat with the data sets which basically rely on the ground density of the ground measurements and few other parameters which we are important in obour training algorithm, and that gives you an idea about whether a product is reliable in certain revision or not. So I think the quality assurance is one way to kind of see whether the product is robost or not.

2:14:58 Speaker 1: Okay, thank you. Question thirty seven, how do do you phittic physical models handle sudden extreme local emissions like crop burning or wildfires, especially when satellite AAD observations are missing due to cloud cover, Aaron, if you want to add anything on that.

2:15:16 Speaker 2: Sure mean this probably ties a little bit back into the earlier question why it takes so long to put these estimates together each time. So within the geophyscal model, it's obviously driven by a chemical transport model, and so in our case, and we use daily biomass burning emissions, and so those would capture within those daily inventories. They specific events like a sudden residual burning or wildfires, And those emission inventories themselves aren't necessarily dependent on the same limiting, don't have the same dependence sort of limitations, don't have the same limitations as the AOD retrieval.

2:15:52 Speaker 2: So a number of those daily biomass burning inventories can work as fine even when there is some cloud cover. Now that's not to say they're perfect, of course, I know crop resume burning has received a little bit more attention in a few papers I've read recently, thing that can be a higher uncertainty. But for the geophysical lestments, they're trying to within those daily inventories. And one of the reasons we of course enhance the geophysical lestments using the hybrid approach is to try to account for any misunderstandings or underrepresentations that occur within those inventories.

2:16:24 Speaker 2: So that's kind of how we we account for it.

2:16:30 Speaker 1: Okay, thank you. And question thirty eight another question about Indonesia due to the persistent cloud cover, which for the machine learning methods, which meteorological variables or predictors are the most crucial to include so the model doesn't become biased in a human tropical tropical region that experiences extreme humidity fluctuations during hotspot seasons. I guess Fallen, if you want to offer suggestions there or this might just be something that would need to be studied, but go ahead, go ahead, Yeah, let me.

2:17:06 Speaker 3: Just read the tropical system cloud co throat the US most moder Yeah. This this is a real problem, I guess. And it is not only over Indonesia.

2:17:23 Speaker 1: Uh.

2:17:24 Speaker 3: This is if you look across the ITCZ, which is has a consistent clouder cover even over South America or Africa. Part of that, and satellite the current passive satellite observation of aerosols will always have that problem because if there are cloud, the signal is so dominated by the cloud we cannot really get anything any information about aerosols. So I don't think there is any observational solution there. We will have to rely on the model for that, and more we can improve on the model, I think the better answer we can get.

2:18:06 Speaker 3: The other option is rely on ground measurement and active satellite based observation, which are a few of them right now in orbit and the few morees coming in the future which should be able to provide some more details or aerosol information in presence of cloud, and once they get assimilated into models, we will have a better idea what is happening in those reasons.

2:18:34 Speaker 1: Okay, thank you, And then I also saw another question just came in which was also referring to Indonesia. I think you have probably answered answered this in your response as well. About the general challenges so well, I think we'll, you know, we'll read through these and make sure that we've captured all of them. But I think for the sake of time, we're gonna I think call it call an end to the Q and A here. So thank you very much doctor Owen Dunkelar and doctor Kupta for contributing your time and your expertise to this training.

2:19:12 Speaker 1: And thank you very much to our attendees for your interest and all these very interesting and insightful questions about how to use these data sets appropriately. As a reminder, the next and final part of the training will be next week, and to prepare yourself with that training, you should test out that you're able to access Google Collab the sorry the have an Earth Data login password for accessing the NASA Earth data and also have an account with open Aq to access the open aq data set via their API, and there'll be some links to those.

2:19:56 Speaker 1: The slides will have links to those when they're posts the training website. So yeah, again, thank you everyone, and we will see you next week.