Hands-On Analysis of AERONET Data

0:00 Speaker 1: Browser and get ready to walk along with me okay great so before we start on the web demos let's find the aeronet site so it's easy to remember the aeronet website it's aeronet.gssc.nasa.gov but if you cannot the best way to do is just search Aeronet NASA in the Google search engine or any other search engine you like, and hit, and the first one will come up as Aerosol Robotic Network homepage. And if I click on that, you'll reach to this page. Now, let me make this little bit bigger in terms of font so that everybody is able to visualize, see that more clearly.

0:52 Speaker 1: So when you go to this website, this is our home page. On the top, we have different components of Eronet, which we talked about in part one. This gives you a little bit introduction about Eronet. Again, we talked about this all in part one and describe what different component does. This is the map showing all the stations. And if you come on the bottom here, We update this particular section with the news and announcement. So, for example, NASA, their only data, which is simply a data worldview, which I talked earlier, is a news item here.

1:35 Speaker 1: We have some specific about maintenance issues for people who are using data on a regular basis. We have a big meeting coming up. in September. So that announcement is here. We have a newsletter, so that announcement. So anything, sometimes we also put out some quick analysis on interesting aerosol events which are happening. So this is from the last year when Canadian wildfire actually happened and a lot of the smoke was transporting here in the East Coast in the US. We put together how Aeronet is actually measuring those smoke aerosols in the atmosphere and how it is impacting their quality in the Eastern Coast.

2:16 Speaker 1: So there's a lot which we put here time to time. So anybody who's really looking for this Aeronet data, watch out. There's a space for new and updated information. One more thing I want to point out, we have a list of So on homepage, you can see this yellow highlighted part. So if you are interested in receiving regular updates about Eronet and any specific things through Eronet, you can click on that link and join Eronet listserv and you will get regular updates from Eronet team. On the left, you will see different parts, right?

3:00 Speaker 1: So you have networks, campaign, collaborators, data, logistics, all different things are there listed here, okay? So I will go over some of these more detail, but let me go through the homepage first. This section is where the Eronet data section so anybody who's interested in accessing data or displaying data this is the section to watch the panel vertical panel on the left side where you have visualization tool we have the data display download in different ways and these are divided into different types of data sets which we talked in part one.

3:46 Speaker 1: So these are direct sun measurement aerosol optical depth. These are inversion data product like single scattering albedo and other things. We have some other things also like solar flux. These are the ocean color data, which we talked about, Aeronet OC. These are the nighttime data using the lunar. And then we have some other cloud mode, which we have not talked in more great detail, but these are some experimental data sets. Okay, so before we move on, let me show you one of the things which often people use, specifically people who have used a lot of datasets.

4:24 Speaker 1: So what is called is site list, okay? So if you click on the text format, then you will get a list like this, and this list combines all the sites Eronet ever has created. So it has site name, the first one, and then the longitude, and then latitude, and then the elevation in meters. So this site is generated, I think, every day, and it is get updated. So there are, if you look counts, I think there are 1,600, some odd number of sites which ever created. So this can be a good source for people who want to analyze global data sets.

5:04 Speaker 1: The site list is also available in Google Earth format. So if you want to download that and display in Google Earth, you will see all those dots all over the world. And if you click on the all list, you can get that the same list in Google Earth format or text format by year. So you can generate this list by any year. So let's click on 2024. and then you will get a list like this. This list is very similar to what we saw earlier, but in addition to those station information, it also has information in terms of data availability.

5:46 Speaker 1: So the way data availability is displayed we have done in one way. So for each month, we have set up the flag one or zero. So if there is a data available for that particular month, it is saying one. If there is no data available, then it says zero. So there is the flags for each month is available and this again can be very useful for people who wants to analyze and understand how much data is available from particular site on particular year. Again, there are so many lists here people can browse through.

6:23 Speaker 1: Okay, so let's go back to the homepage after browsing those lists. Now, first thing I want to go over is called what we call site information. So again, on the left side panel, if you click on the site information page, you will reach to this page. Now, you will see a notice here that if you want to stabilize a new Aeronet site, if anyone is interested in getting a new Aeronet site near them, they can contact us and we can see if Aeronet... Just remember, we cannot fulfill every single request. There are so many factors which determines where we will put the Aeronet station.

7:10 Speaker 1: But this is a way to start the process, whether it is possible or not. Now we have a similar map as we saw on homepage. This map is very interactive. You can zoom in, zoom out, any parts of the world. There are four buttons on the top. By default, all the stations are displayed, but you can also display active stations. So remember in part one, we talked about that although there are 17, 1800 stations displayed,

7:45 Speaker 1: deployed during entire time window of Eronet, but currently about 600 stations are active. So out of all, if I just click active, then these are the sites which are active in last one year. The way we define the active sites are sites which have produced any data in last 365 days. That's the site we call it active. Then there are inactive sites. which have not produced any data in last one year. And then there are some sites which are limited. So these limited sites are created sometime to test the instrument or to make measurement for just for a few days.

8:28 Speaker 1: Sometimes they can be very specific to field campaign or something, but they are very, very rare. Limited sites, I think we define them If the data is available for less than 10 days, then we don't call them permanent or campaign site. It comes into the limited site page. So the most important one is the active site page. Now, on the bottom of this map, you will see a list of all the sites, okay? And as you start zooming into the map, you will notice that that station list also start reducing. It only shows the list which is displaying on the map.

9:09 Speaker 1: So I'm going to go to the NASA Goddard area. That's where we are located. That's from where we are doing this webinar series. So I'm going to, I have already clicked all the active sites. So this is the area where we are located. And if I go down to the map, then you will see there are only nine stations which are displaying in the map which we have. And this particular site is the one where we are located. It's called GSFC. There are two sites actually on the same location. One is called GSFC, which is the oldest one.

9:49 Speaker 1: And more recently, we started a GSFC polar, which is a polar instrument on the Goddard site. So if I click on the GSFC you get to this page. Let me zoom out a little bit so that we can see what is the online page. You have site name, you have coordinates, and the elevation, same information as we saw earlier in the file list. Here is the picture. Typically, we have two, three pictures. So this is our aeronet. Since it is the calibration center, we have mounted many, many instruments here for calibration purpose, which gets swapped out on a daily basis.

10:36 Speaker 1: There is some site description. There are some information about if you want to contact the PII site manager, there are some information here. So for each site, we have this information. So let's go back to the map. Now, as exercise, I want you to take two minutes of your time and zoom into the area where you are physically located and find a site which is nearest to your location and note down the name in a piece of paper or on a notepad on your computer. And that is the site we will use throughout the rest of the presentation to actually evaluate and download the data.

11:22 Speaker 1: So I'm giving two minutes time for everyone to look, browse through this map, find the site which is closest to you, and note down the name.

11:33 Speaker 2: Thank you.

13:36 Speaker 1: Okay, great. So I hope everyone with internet connection, you have ability and you have explored the site information page and find the nearest Eronet site which is close to you. Okay, now let's move on and just remember that site name and we'll use it later on. Now, let me also go through some of the other pages on this website. We have a link which says campaigns. So again, I talked about this in part one. Aeronet supports several NASA and non-NASA field campaigns specifically designed to make aerosol measurements, sometimes for air quality applications, sometimes to do the climate applications, sometimes to understand more atmospheric behavior of atmospheric aerosols.

14:34 Speaker 1: in chemistry and energy budget. So these measurements, Aeronet has been taking to support this field campaign since the beginning of the program, as you can see, in 1993. And since then, it has participated in many, many field campaigns. Starting 2011, we started deploying this high-density network called Dragon. The most recent one we deployed, Aeronet, in the Asia AQ so I'll just give you a flavor of how this campaign information looks on our web page so once you click on this campaign page you will get to that You'll have some background information about the campaign itself.

15:20 Speaker 1: And then we will display and provide the near real-time data to actually not only people who are doing the campaign on the ground, but to the community as well. And this data helps them to actually do flight planning. If they're flying aircrafts, they can actually change their flight flight plans based on where aerosol events are happening based on their own data, which becomes almost in real time available. So again, during HIAQ, we deployed in South Korea, Philippines, Taiwan, and Thailand. These are the locations specifically actually done just for the field campaign.

16:06 Speaker 1: There are additional sites which you don't see here, which are permanent sites in that particular regions of the world. So this is just example here from the same page. You can download the data also. You can also get the list of location list here. There's a map where the new sites are displayed in blue color. The permanent sites are displayed in green color, very similar to what we saw earlier. And if I just click on the campaign, then you will just see the blue color sites which are specifically falls in the And then there are some more details, contact information.

16:47 Speaker 1: So very important, if you are interested in field campaign datasets, this is the Aeronet contribution to those campaigns. Okay, now let me go to the operation. Operation is a specific page may not be so much useful for the data users. But this is more for site managers and principal investigator of each site. People who actually manage the instrument on remote locations at universities or institutions or remote individual houses, this is for them. And what this does is it provides a lot of information about the health of the instrument.

17:34 Speaker 1: So if I click on, for example, instrument status, let me click, these are the different Aeronet networks, partner networks, as remember we talked about that Aeronet is a networks of networks, so these are the different networks. So if I just click on the Aeronet, then you will see all the instruments which are actually deployed and you'll see many, many stations, many, many with the GSFC because these are all deployed for calibration purposes. So you will see a lot of specific details about those instrument.

18:07 Speaker 1: Again, not so much useful for the data users, but for more for site managers and PI. Okay, now one of the most important thing is about the system description. We talked about very briefly in part one, what kind of measurements sun photometer or what kind of instrument aeronet use, here you will get a lot more details about that. So you can click on the measurement system and you will learn about all that same instrument, what each number in that name means, different wavelength combinations, all the specific things including the communication.

18:49 Speaker 1: In the calibration section, you will read about entire calibration process, which I have briefly gone through in part one. But if you are interested in more details, you will get a lot of details here. There is a operation page. There is a data transfer page. You get a lot of details, specifics. If you're transferring data to GSM or internet or satellite, We have a data processing process set up, how the data are processed. We get the data, and then it gets to the different. And then finally, the data distribution, which I will go over a little bit more.

19:34 Speaker 1: Okay, so that is what system description takes you. Now, let me take you to one more important topic, which is logistics. Again, this is not so much for data users, but this is for people who manage those site but it has a lot of important information so we have all the manuals of all the different models of some photometer which we ever used we have several videos to troubleshoot the instrument, to install the instrument. So these are very instructional videos. We have some tools, how to actually do some data transfer.

20:17 Speaker 1: We do a lot of shipping, as I mentioned earlier in the part one. The system has to come every 12 to 18 months to NASA Goddard or any other calibration center. So there's a lot of shipping happens back and forth from the site to one of the calibration centers. So there's a lot of information here. Again, not so much useful for the data users, but for the site managers and PI. Okay, one more thing, publication. Since the beginning of program, there have been a lot of publications research article which are published by AeroNet team and AeroNet partners from all around the world and we have tried to archive them here.

21:02 Speaker 1: Most of them you can directly get to the link. Some of them are very old one and you can actually find through the search. We also have links to the quarterly newsletters. We also have several reports on Aeronet, different data sets, some details about a specific version, specific products. There's a lot of reading material available for people who wants to use the Aeronet data. Okay, so this is main parts I have gone through. There are some other things which you can go over in your own. They're very...

21:45 Speaker 1: clearly written. Now, I also want to show quickly about ocean color, which we talked about, Aeronet OC in part one. This is a specific page on that. They have their own news, although it's a component of Aeronet, but it operates only over ocean, their sites, their data downloads, all the details are there about Aeronet OC. Similarly, we have for maritime aerosols, which I discussed earlier using handheld microtops. This is a old paper, but it described the main network very clearly. We provide the datasets here.

22:27 Speaker 1: Again, all the datasets are produced here and they are named based on the ship cruise and time window when it happens and you can download this data actually. So you will see a long list of shift curves from where we collect the data and the data are reported here. Okay, so I think that kind of completes the first part. Now I want to get into the data part. So, aeronet data access, right? So let's start with Ansel optical depth because that is the most common parameters used So the first one is called data display.

23:12 Speaker 1: I will click on the data display. And it will take me to this page. So on the data display now again this map is there. You can actually filter this map in many different ways. How many years of data are available? What level of data you want? You can also pick individual years, you can pick month, day, you can pick active and inactive sites, all kind of things you can do here. But I'm going to just look the GSFC site and then what you can do is when you're walking through with me, you can select the site which you identify earlier as the nearest site to your location.

23:55 Speaker 1: So instead of GSFC, you can pick that site. So I'll just, so I picked the GSFC site. click on that the map shows the location of the site and then when I click on the GSFC it will take me to the data display page for the GSFC site so once you click on the GSFC site you will come here the page look like this okay now let me walk you through very quickly so Again, this is the site information page link, which we saw earlier. Here are some details about the people who manage the site. Here are some statistics on the data availability.

24:41 Speaker 1: So from this particular site, we have 25 years of data record. It started in 1992, 93, 92, 93, and currently data is. So the total number of days are these, almost 30 years of the data are available from this particular site. data arranged in different years. You can select any years. These are the parameters which you want to display. You can select. I have selected the aerosol optical depth first. And these are the different levels. Again, level one is the raw data without cloud clearing. Level 1.5 are automatically cloud clear data.

25:21 Speaker 1: And level two are quality assured and calibration applied. So the For on-quantity purposes, we recommend that users use Level 2 data. You can display daily averages or all points. So let me go to Level 1.5 data because that is available in real time. Okay, Level 2 data takes one to two years. And this is 2024, August.

25:57 Speaker 1: and today is August 2nd. So this is showing the August 1 and August 2 data and then this is just for the August 2nd. These are the time in UTC. So you can see these different lines shows aerosol optical depth corresponding to different wavelength which are represented here. AOD at around 500 nanometer is 0.344, which is actually pretty high for this location. So there is a wildfire going on in the West Coast, which may have some influence transported, but also during summertime, sometimes AOD can be high because of the hygroscopic effect on aerosols.

26:47 Speaker 1: On the bottom, you have several link you can download the data directly from here or there are other ways to download which i will show you so this is what the first visualization looks like now let me go to the download tool next the interface is very similar okay you can select any site so since i am going directly to gsfc on the search box i'll click put GSFC, click on the GSFC, and then we'll take it. Once I click on the station, it will take me to the download page, and you can see various options here, okay?

27:30 Speaker 1: You can select the date range. So I'm going to just select 2024 for a sample download from January to December 2024. And then there are different types of measurement which you can download, which we talked earlier, and you can get. Now, what you can do here is you can click on this data description page and you will get description of all the things which are written there, how the daily averages are calculated, how the monthly data are calculated, what it means for different parameters, all the raw data and inversion.

28:07 Speaker 1: So very specific and detailed information are actually provided on this page. The unit page gives you the units of all the parameters which we provide, so you have specific units for each of those parameters which we provide. So just for sake of demo, I'm going to choose level 1.5, aerosol, optical depth, precipitation, water, and then angstrom parameter. and then I will choose all points, means all the measurements which are taken, and then the daily average is just the daily average. So to keep things faster, I will choose daily average level 1.5, January 1st to December 31st, 2024, and click on the data download.

28:59 Speaker 1: Once I click on the data download, you will get to this page, which is basically Eronet data policy. So since Eronet is a collaboration among many countries, many individuals, we try to make sure that people who manage the individual sites get credit when their data is used for publication for a such purpose. And that's what exactly this data policy page shows, how to refer the data before you use the data in research and publishing. Do you want to give Sankar data the people who are managing and collecting the data from that particular station?

29:45 Speaker 1: Believe me, it takes a lot of efforts to maintain these instruments. They often get problems and the site managers have to continuously watch for them and make sure the data continuity remains. So that is the whole purpose of this data policy page. You click accept and then you immediately it downloads the file. The files are downloaded in this form. This is the initial date and this is the ending date of the data and the site name and level 1.5. So these are downloaded as ASCII files. I can open this using Excel or text.

30:28 Speaker 1: I will just open in the text here just to show you how it looks, but you can also, these are basically CSV files, so you can easily open them in an Excel spreadsheet. This is the header of the file. You will see all the details about the datasets, different columns, so you will notice there are many, many different columns presented here. But this provides you everything you need to use the data for various scientific research or application purposes.

31:01 Speaker 1: So it's very simple format, very easy to use data, and very easy to download the data. Okay, now let's say you want to download all the global data for the entire timestamp, everything together, and we have this as well. This is called download all the data, all the sites, all timestamps. And they are provided in two different formats. One is star.gz and then just the zip format. So depending on your requirement, you can download any of these. They are again separated by all points data, daily average or monthly average.

31:41 Speaker 1: And again, these are different parameters. These are SGA, aerosol optical data, whatever you need. And again, user's guides are also given here. So this is another way to download. It's a huge file. When you click on the download, you will get several gigabytes of files. So it's a little bit bigger file, but you get everything into single file. Climatological tables. Now this is another piece of information which we provide. Again, very similar interface as you see earlier. What it does is it provides climatological values for each site.

32:20 Speaker 1: Again, this is the data policy. I'll just say accept, and then you get to these tables. So what these tables provides you are the climatological mean for each month and year. So this is entire time window for each month, how the aerosol optical depth is, and this is yearly mean 0.18, and then this is one standard deviation. This is Engstrom exponent. This is strength division in angstrom exponent. This is precipitable water vapor. This is strength division in water vapor. These are the number of days which are used to average, get this, and these are the number of months for which annual data are averaged.

33:09 Speaker 1: So this is for the entire time window, but you can also get into each year. This is 1993. And remember, whenever we don't have the data, we don't provide it, we don't try to fill it. So if any of the month is missing, yearly means are not calculated. User can calculate their only annual mean, but we don't provide it here. And you can download the data using these files which we have here, overall download, combined download. So this climatology is very, very useful. They are calculated with a lot of precaution, quality control and people who are interested in trend analysis, they can actually get these quantities easily.

33:54 Speaker 1: Okay, so now the next thing I want to do is click on the web services. So on the web service, we have data access to API. Again, different types of the data. We are going to look into the AOD and SDA. These are different parameters and their explanations in the API, okay? And in the bottom, these are the different examples how the APIs are called. So let's say I want to get level 1.5 data from all sites available from the AOD, right? Or I want to get level 1 data for one of the sites. So let's click on this particular link.

34:43 Speaker 1: This is a web link which you can actually define. So instead of what I will do is to get the data for the guarded site, I will change in the web link

35:00 Speaker 1: site equal to card site and sub card site. My site name is JSX. And you can change to the site name which you have noted down earlier. So once I change the site name, I rerun. and that's what now the site is GSFC and this is for 2000 data. Now, instead of 2000, I'm going to get the data for 2024. So this is 2024. And then, now this AOD, 10

35:42 Speaker 1: is level one, so I'm going to change this to AOD 15 equal to one. It means I'm getting the data for level 1.5. So this is level 1.5 data, all the points which were given for that particular group. This is GSFC, this is the day, month, and this is, since we are doing average equal to 20, it means it is going to give me daily average. If I do average equal to 10, I think this will give me all points. So this gives me all points. Now, this API is very useful to people who wants to access data in more automated ways.

36:29 Speaker 1: We will show you some example how we use this API in Python code when we get to that part. Okay, so I hope you are able to follow along and get up to this point. Let me go back to this website again. So this was data display and data download part. Now, you will notice that although I have only gone through aerosol optical depth, but these similar capabilities are built for each type of data sets, which you can explore in your own time. Now, in the next few minutes, I will quickly walk you through two more tools which we have.

37:13 Speaker 1: One is called Map Explorer. This is a, I'm going to reduce, this is a quickly visualize the data, and it can visualize either near real-time data or daily main data. So select the mode, it's near real-time, and then if I select the daily, then this day-night flag should go away and you will see, let me select from yesterday.

37:44 Speaker 1: So you can actually map the aerosol optical depth over the entire globe. And let's, since we know already, there have been a lot of fires in the US currently going on, so I will just go back a few days, and let's look July 26th, for example. So this is July 6th, 2024. You can see some of the stations, like we're zooming this area. They are really high aerosol optical depth in this particular part of the world. The gray area stations shows inactive sites on that particular day, and I can turn them on and off if I want.

38:35 Speaker 1: If I don't want, they can be turned off. Now, if you mouse over any of these sites, you will see specific details about the site. It can take you to the data page also from here. But if you click on them, you will get a time series of the data from last 30 days. So you can see how on this particular site, aerosol optical depth was so low earlier in the July. AOD values were less than 0.1, very clean condition. As soon as FIRE started, AUD started jumping up, up, and then it reached to all the 1.24.

39:14 Speaker 1: And these are daily mean values. If you look hourly values, they can be much more higher. So this is a nice tool to visualize the data. If I do on the near real time basis, then it basically takes the current time and display the data which are available in last one or two hour time window. And you will also notice that this also displays a day night separator. So the area which are kind of hidden, gray colored, they are nighttime and this is the daytime. ArrowNet is an optical measurement. Most of the sun data are during daytime.

39:56 Speaker 1: There are some night data which we don't display here, but we do have nighttime data as well. So again, very nice tool if you want to visualize data on the map. Okay, now let me go to the Synergy tool now. So we can click, go to the Data Visualization, click on the Synergy tool. Here, I won't spend a lot of time But again, you can do similar things. You can choose the date, time, site information, and then you can select different things from here, what you want to visualize. Some of them may not be readily available, and some of them may be available.

40:39 Speaker 1: So AOD, inversion products, right? So you can display, select again what parameter you want to display. You can choose water vapor or angstrom parameter. So you can actually decide which parameter you want to display. Here are the size distribution, reflect index, or other parameter which you want to display. So for this particular date, probably we don't have the data available, so let me switch to level 1.5. and then here also level 1.5. And let's pick another date. Yeah, now this is for January 14th, so you can see some of the parameters.

41:35 Speaker 1: So this is refractive index, this is size distribution. Yeah, so if you click on that, you'll get a size distribution. Basically, it's showing that there are two modes in January. One is dominating in the smaller size particles and one in the coarser size particles. And depending on time of the year, the size mode can change. So let's look at the size mode just now, what is happening during the fires when we have some impact of fires, right? So size distribution, now you can see a little bit different now.

42:13 Speaker 1: dominating mode is the fine mode, smaller size particles, whereas the coarse mode is not that much of domination. So this is also a good tool to actually look several data sets together. Okay, great. So that was ArrowNet website tools and data download. Now I'm going to pass along to my colleague, Peter. He is also part of Aeronet team. And he has developed some of the Python tools to visualize, map, do time series, and do intercomparison with the satellite data. So he will show you how to access these codes and how to run them using Google Colab.

43:01 Speaker 1: So for this specifically, I'm hoping everybody has Google account. So please sign into your Google account if you have not. because we are going to use Google account to do that. So Peter, take over.

43:18 Speaker 2: Thank you, Pawan. Hi, everyone. My name is Peter Gregorov, and I'm AirNet's new scientific programmer. I developed the following Google Colab codes in the infancy of my career, and we think those codes are very important because it gives people outside of our network the capability of interacting with AirNet data and producing some cool plots. So there are five codes that we're going to go over. The first two read and map ARINET's aerosol data onto a global map. One of them produces a series of static plots that can be saved onto your machine, while the other ones produce more interactive visuals with some widgets.

43:55 Speaker 2: Another script accomplishes a similar task except it reads ARINET inversion data. And then another script reads aerosol data once again, but it instead produces a variety of time series plots. And the final script collocates aeronet aerosol data with the VIIRS satellite data and then does a comparison. So first code we're going to open up is the read and map aeronet data, or AOD. And just note that these codes are on the AireNet GitHub and they're open. And the link to this repository is I think on slide 17.

44:43 Speaker 2: Okay, so now that we have this code open. So the purpose of this code is to read and map AireNet data directly from our web API. The data includes our AOD measurements taken with different wavelengths, as well as Angstrom exponent parameters.

45:05 Speaker 2: Different parameters can be specified by the user, such as the geographical box, the coordinate box of latitude, longitude, the type of average, whether it's daily or hourly, the timeframe, channel, et cetera. And the output of this code is those geographical maps with the circle markers and color bar, which represent the average daily or hourly values of the products. The main use of this tool is to track the movement and density of aerosols over time. And this tool was first deployed in June of 2023 and mapping the aerosols produced by the Canadian wildfires.

45:43 Speaker 2: And this time, we focus on the timeframe from July 25th to August 1st, where we would track some of the wildfires that are affecting the west coast of the U.S. and the spreading throughout So this module makes use of certain libraries. So the first thing that we will have to do once we run the script is we'll have to mount our Google, sorry, we have to mount our Google Drive onto this Colab notebook. Otherwise, if we do not do that, we're not gonna be able to save our produced blocks. So just have to click continue, make sure you have a Google account as Pawan had previously mentioned.

46:28 Speaker 2: And these are some of the libraries that we make use of. The NumPy library is for array manipulation. Pandas is for data cleaning and data processing. Some of our map features, such as coastlines, state boundaries, et cetera, are done with Car2P. And in addition to that, some of the projections that we use, such as the orthographic or the plate carry projection, those are imported from the Car2P library as well. Beautiful Soup, kind of an odd name, but that one is the most popular web scraping library from Python where it would access a website, send the request using the request module, and then it would read that data, the text file on the website as a text file.

47:20 Speaker 2: And then GeoPandas is similar to Pandas except it's for geographical data. So now we've gone over some of the relevant libraries and we've mounted our Google Drive. The next part will be the input parameters. So how the Web API works is we have some parameters and those parameters, such as the date, the level data, the features, the coordinates, they're used to construct the URL hyperlink. And depending on those parameters, it'll produce a different hyperlink, sorry, a different URL that contains that relevant data.

48:05 Speaker 2: So our initial and final dates are in four-digit year, two-digit month, and two-digit date format. Our level, as Pavan mentioned, Eridan has level 1.0 data, level 1.5, which is cloud screen and quality assured, then level 2.0 data is, in addition to, Quality Assert and Cloud Screen, it's also calibration verified. The average type, so we have 3,000 averages here. Either we can produce daily plots, which is the average for the day. We can produce by hourly bin, or the time average is the total average for the entire selected data frame or the time frame, which will produce only one plot.

48:51 Speaker 2: These two variables, vizmin and vizmax, they are the color bar bounds. So we've defined the AOD value greater than 1.0 is going to show as magenta on the color map. Anything less than zero would show up as gray or unavailable. And anything in between will follow this color pattern where for low AODs will be dark green. For high AODs, it will be dark red. and then anything in between.

49:25 Speaker 2: So after defining this color bar, some of our final parameters for data aggregation will be selecting our feature and our channels. So if you would like to select AOD, or an angstrom exponent, we either do one or two. Otherwise,

49:46 Speaker 2: after that is done, we select our specific channel. So we have dozens of different wavelengths, but the most popular AOD that almost always has data is the 500 nanometer.

50:01 Speaker 2: And then here's the bounding box. So they have to be in a decimal format. because the web API does only take a decimal format when constructing the links. So we can run that. And since everyone can download a copy of this code, they're free to do so, and everyone can change their parameters as they wish. This next cell, what it does is it validates some of the data. and it prepares those inputs for constructing the link. So first, this block here takes the initial and final dates and extracts their year, month, and day components.

50:52 Speaker 2: Here in this block, we take the user input of the level data and we convert it to an integer number. So if it's level 1.0, it'll become 10. If it's level 1.5, it'll become 15. The purpose of that is the Web API reads it in such a way. And then the same thing, some data verification for VizMin and VizMax, make sure none of them are negative, make sure they're not flipped. And then this is the last verification check, which prevents users from selecting the current year and level 2.0 data, because as Pavon earlier mentioned, it takes a couple of months, maybe up to a year for level 2.0 data to show up.

51:35 Speaker 2: And after that is done, we construct the URL as it's seen here from all of the input parameters. And then we pass this URL onto this request module, which sends the HTTP request using the web and uses the beautiful suit package to convert that to a text file that is saved. So that is, after that is being saved, the file would show up onto your Google Drive.

52:13 Speaker 2: So once that is done, we read that text file and we assign it to a Pandas data frame. So data frame is kind of like, it resembles an Excel file. It has column and then this is where the, some of the data manipulation begins because our date and time formats for error in the data, they're a little different. than the conventional ones. So this block of code basically converts them to Python's datetime format, which follows a four-digit year dash two-digit month dash two-digit day. For error net data, it's using two-digit day colon two-digit month colon four-digit year.

52:55 Speaker 2: And then depending on the average type that we specify, we group them based on the error net site and date. If it's daily, if it's hourly, it's site, date, and hour. and if it's for total average, we just group everything by the site. And the numeric columns argument pretty much makes sure that string columns do not get aggregated, otherwise they will cause an error, because we cannot average non-numeric columns.

53:27 Speaker 2: And there's a check here that if, for example, if

53:32 Speaker 2: our data frame is does not exist or if there is no data, it will print the message that there's no data to parse and then it will prompt the user to retry with different parameters. So now we have some data. So the next, after we have our data processed, it's time to select the data that we want to work with. So this cell, this particular cell does exactly that. It extracts all the AOD columns and the, sorry, it reads all the AOD columns or the action exponent columns then it matches the proper columns based on the user input.

54:08 Speaker 2: So depending on if your feature choice is AOD or if your feature choice is Angstrom or what AOD value you use, it parses through the file and it matches your input with the proper file. And at the end, it isolates the data frame to just include the side latitude, side longitude, and the AOD column or the actual exponent column.

54:37 Speaker 2: After that is done, we produce a, first thing is very important to drop all values that are undefined and to reset the index. So resetting the index does, remember we group by error in a site and date for daily averages. To do further manipulation, we have to reset the index so we can include the site and date back into our data set. So now we have data set that is ready for plotting. So here we make use of all of those plotting packages such as car2p and matplotlib. Then we produce those plots for each day.

55:23 Speaker 2: So now the code should generate a plot for the first day, which is July 25th, and then it will keep loading until it goes to the last date, which is August 2nd. So here we have for July 25th, you can see some high AOD valleys in the Midwest. And here's some forest fires on the western side of the US around Idaho and Montana. And some more for that day. And here's on the 28th, they're spreading to the North Great Plains. Came for the 29th. and 30th and here's where they're kind of like subdued and they've already pushed, they've moved a little eastward.

56:10 Speaker 2: You can see some higher AOD values around the Great Lakes region. And now here, the AOD values are pretty much stable. So that was for the plate carry projection and a similar thing can be done except in the orthographic projection. which some people may prefer because it gives you the, it's like, you know, looking at a globe instead of just a map, right? So the computational expense is a little greater for this type of a projection. So it will take a little longer for the PLOS to produce, but should still not take that biggest time.

56:59 Speaker 2: So this is, I'll have to zoom out a little bit so everyone can see. So we have.

57:06 Speaker 3: For the 27th, 28, 29, 30th, 31st.

57:21 Speaker 2: And after this code finishes running, everyone has the opportunity to decide if they want to save all of these files All these files that have been produced here, we would like to download as a zip file or not. So I'll say yes. So they will compress them. Could have chosen a more creative name, but feel free to modify the code and put a different name for the output.

57:58 Speaker 2: And I'll just put in my downloads folder. And it's going to be there. Let's see. Okay. So, that is the end of the read and map Airnet code. So, now, we will go back and launch a different one. So, let's try the more interactive code for reading and mapping Airnet data. So the only difference between this interactive code and the previous code is that this one makes use of widgets from Python's ipywidgets library, as well as matplotlib's animation package. So it's the same procedure as always. We define the same input parameters.

58:53 Speaker 2: We produce the link the exact same way. We process the data in the exact same way. and as well as the angstrom exponent AOD selections. The only difference is that in terms of the plotting. At the end, we get this map, which it's one plot, but it works like an image carousel where you select the slider. Let's put the 30th. There's going to be a few second delay depending on how big the data is, but that will go directly to the specified date. And similarly, this is done for the orthographic projection.

59:42 Speaker 2: And again, if you define hourly average, then actually will produce two sliders, one for date and one for hour. But for demonstration purposes, we're only gonna stick to date. So our plots have been produced and we can select, let's say 28 and here's the plot. And the second piece of interactive widget that I decided to use here is an animation. Similar to a GIF, except there's some control buttons such as pause, play. We can either run it to go through once, or we can just, we can loop it. And we can also adjust the speed of, you know, how fast the images switch.

1:00:54 Speaker 2: Or you could also go in reverse. I think reflect was, Yes, reflect is where you would go in reverse.

1:01:06 Speaker 1: Yeah.

1:01:09 Speaker 2: And then same thing for the orthographic projection. And there is no, I did not put in the script to save these. I think this interactive code is better for just the purpose of displaying and viewing the data, which is why I did not put anything to save it. But yeah, this will work the same way. Okay, so moving on to one of the more recent developments. Sorry, the more recent developments. It is the Descript, which reads and maps inversion products. So this is no longer AOD, this plots inversion products like single scattering, of albedo or

1:02:07 Speaker 2: like a single scattering albedo or AOD coincident inputs or extinction and absorptions, et cetera. And we can select our inversion type, whether it's alma contra or hybrid. And we can specify again our wavelength of interest. And similarly, like before, there's some data validation and constriction of the, of the data year, month, or sorry, construction of the converting the timestamp to individual year, month, and day, constructing the hyperlink, and then reading it through beautiful soup. And so here I combined all the data processing and plotting into one big cell.

1:02:50 Speaker 2: So I should start plotting some of them.

1:02:56 Speaker 2: So this particular, this script, just because I combined everything in one cell, it would take a few minutes to run, maybe between three and four, but at the end it would produce everything for you. So we have for each hourly bin, I guess this is why it took a little long this time because instead of daily averages, I did hourly averages for each day. So there it is, you can see some, So this plots the single scattering albedo at 440 nanometers each hour. And I think it also does it for the orthographic projection eventually.

1:03:42 Speaker 2: But you can just take one plot and you can see in detail. So yeah, this script runs Same, very similar fashion, the way that the original Read & Map, Aeronet, Aerosol data works, except there's some more data validation and I've combined all the cells together. So the next script we're going to run will be for the read and map Airenet time series.

1:04:30 Speaker 2: So we can just close out of that. We can go back to our Google CoLab. Okay, so now that everybody has that open, so the purpose of this script is to read Airenet data, again, from the awareness API. and plot time series graph to visualize changes in AOD or action exponent parameters, except this time it does not do it for the whole geographical domain, it does it for the specific error net site. Again, we can specify site name, average type, data level, wavelength, timeframe, and it produces three plots.

1:05:19 Speaker 2: It will produce a standard time series graph, It'll produce a standard time series graph. It will also produce some calendar plots, a tile plot, and at the end it will produce an annual variability plot with some standard deviations superimposed on the graph as a shaded region, right?

1:05:56 Speaker 2: So we pretty much use almost the same packages. We have Cal, except, yeah, CalPlot is another package that we have to install. That one's different from the previous one. All the other ones are pre-installed already. We have Math for some more complex equations. We have some other Matplotlib components like mdates, which converts our date time to numeric.

1:06:33 Speaker 2: And then, again, we'll have some hard drive onto this Colab notebook. Just click continue. So, obviously, the only difference with the input parameters is that we also have to specify site name and actually the web, an ARNS web API, you can put the site as a parameter and it will generate a link with data just for that specific site. So in this case, we choose NASA Goddard, just GSSC, and we're doing it for a period of 10 years. So starting, yeah. So averages we can either choose daily or monthly.

1:07:24 Speaker 2: So daily averages is preferable here because you're also going to get the calendar plots and the tile plots. And unfortunately you cannot produce a calendar plot with monthly averages because the purpose of a calendar plot is to show the value for each day. So now we read our URL, same data validation as before. same data processing,

1:07:56 Speaker 2: and same feature selection. And at the end, for our specific site, notice that the error in the site name is no longer there because now it's redundant because we're doing that for each specific site. So you have the date, day of the year, and the AOD measurement. Day of the year is important to have here because date is, it's an object, it's not an integer value, so it's gonna be difficult to manipulate whenever we're doing counters. The first plot is a time series graph. Depending on the average type, it will produce either monthly average or daily average.

1:08:38 Speaker 2: Little clutter here because, again, it's daily average, so it's valid for each date, and this is over the course of 10 years, so, but we can spot some seasonality here with some of the peaks and obviously here we know this is during the period where there was a lot of wildfires in Northeast Canada. So that was roughly, yeah, it looks like that's around summer of 2023.

1:09:04 Speaker 3: And this is where.

1:09:07 Speaker 2: We produce our tile map. So tile map is, let me zoom out a little bit. It resembles a tile. Basically, the months are on the y-axis, the x-axis, sorry, and the years are on the y-axis. You can actually specify a monthly average, except for the month of January, instead of having 30 values, it will have one value. So it's just gonna look like a less cluttered tile plot. You can spot specific days where the AOD might be higher. And this calendar plot is for, again, for daily averages only. It will not, reproduce if average type is not one, or it's not daily.

1:09:53 Speaker 2: So we wait for it to produce.

1:10:04 Speaker 2: And it looks like it is done. So I cannot zoom out far enough to see the whole thing because it's a very large data frame, but here it is, the side GSFC. And not only does it show the month and the year, it also shows the day of the week. Here that gives us a more clear view. We can spot particular days where the AOD will be really high, and then once, and again, we can see for the month of June, the wildfire season, we can see some of those particularly high AOD days. And finally is the annual variability plot.

1:10:45 Speaker 2: which again can only be shown for daily averages. So it aggregates the data, it shows for a particular year what the average AOD was at that site, and it creates a linear regression line along with a standard deviation. And you can see the long-term trend of AODs, and you can see that for the Goddard site, the average has been pretty consistent. But if you generate some plots for AOD, Other places where recently the governments have implemented some measures to reduce the level of pollution, you can see that the trends have gone down over the years.

1:11:30 Speaker 2: Once again, this last part will prompt the user if they want to save those plots to a zip folder. In this case, I would not want to save it. And so this is it for the time series script. So, now we can transition to our final script, which is the one that co-locates error net data with VIIRS data. Wait a little bit for everyone. Open this one up. Okay, so, This code, I would say, is probably the most involved in terms of what the user has to do, okay? So first, we obviously want to install some of the libraries here.

1:12:28 Speaker 2: And while that runs, I can explain what this code does. So after mounting the Google Drive, click Continue, Continue.

1:12:41 Speaker 4: Yeah.

1:12:43 Speaker 2: This code includes, so yeah, it collocates ARINET data with VIIRS data, includes some functions for spatial averaging and statistical comparison. It retrieves and processes of VIIRS aerosol data from the NASA's LADS query, data query. And it also retrieves and processes ARINET's aerosol data in a similar manner using the web scraping capabilities from the web API. The catch is VIIRS uses the 550 nanometer channel, which ARA does not have. So for proper comparison, we do some statistical interpolation using the 440, 500, and 675 channels to infer what the AOD measurement would be at the 550 nanometer channel.

1:13:36 Speaker 2: And I think we use a cubic spline interpolation. Yep, it's a cubic spline. Correct. And after the collocation is done, we produce two outputs. One is an Excel spreadsheet with the collocated data. And the final one is a visual,

1:13:57 Speaker 2: a collocation plot. So we use a 5.5 pixel grid to get those, to obtain those spatial averages. And this is the statistical function which where we compare the actual and then the predicted values. And we obtain some values, some metrics such as the bias, root mean squared error and the percent error. So the setting up pass in directory, this script, what this script does is, So in our case, we made it a little easier. We have put our query file, our last query file into our GitHub. So now we can just read the file directly from the GitHub.

1:14:58 Speaker 2: And what that file contains is a URL, a link to all the NetCDF files in our query. So in this case, we chose July 15th through July 31st, and we focused on the region of South and Southeast Asia. So that's all the AOD 550 nanometer measurements. It then constructs the file name. It sets up some of the working directories. And this URL is the one where those measurements are obtained.

1:15:35 Speaker 1: From.

1:15:36 Speaker 2: So running this code is more involved as it requires ERP data credentials and the ability to use the NASA LADS archive to query a website, the user will need to generate a token, which serves as authentication to download and access that data. That token is unique for each user and it expires every 60 days. And that token is actually specified right here after the bearer attribute. So once logged in in Earth data, the token can be generated by clicking the generate token menu which Pavan had went over and we inserted in line six here.

1:16:23 Speaker 2: Sorry, not line six, this is cell six. So this cell, we download and process this VIIRS data. So the data set is being read. We calculate the moving averages. we obtain the latitude and longitude, we format the date and time, and at the end, we concat all the data sets together because, again, this produces one data set per hour or per, I think it's per hour, correct? So depending on the length of the Daylight, they could be up maybe anywhere between 8 and 14 files per day. And each of these net CDF files are saved in your local drive, my drive, and we can see RSET workshop, and we can see the ARDT query.

1:17:33 Speaker 2: We have all those BIERS files indeed saved for that timeframe. So now we're done with downloading and processing BIERS data. Now the next step would be to download and process error net data in a very similar fashion, right? So we can choose minimax AOD. So we only chose standard AOD between zero and one, not too high AOD. We specify our coordinates in a similar way. Now make sure that the boundaries, the coordinates that you use in your lab data resemble the coordinates that you're using when querying the error net data.

1:18:11 Speaker 2: because otherwise, at the end, the code will crash because there will not be any data to collocate. So now, after processing the error in the data, this is where the collocation happens. The collocation is, we use the cubic spline to, where we interpolate the 550 nanometer data. value based on the three channels, the 440, 500, 675. This section of the code will take the longest time because it uses some not complex, but computationally consuming arithmetics. And after this code is done, it saves the collocated data as a CSV file, which is also going to be found in your working directory.

1:19:07 Speaker 2: And that can be downloaded from your Google Drive. And if you'll notice, you don't have to specify date here, because why you don't specify dates for Airnet? Because it takes the dates that is produced, the dates from the VIIRS data set, it assigns them to a list, it's called date time, and then that list is then being fed in a loop, right? The loop is called a date time. For each index of the date time list is used to access the URL and then download that AirNet data. So in this case, we're not producing one URL from AirNet.

1:20:01 Speaker 2: We're using it multiple times, as many as their unique date times in the VIIRS data set. So after the Excel file is produced, there's going to be the final section of this code, which is plotting. So to plot it, we isolate the necessary columns and we filter the values based on AOD.

1:20:41 Speaker 2: It produced the following visual.

1:20:42 Speaker 3: Let me zoom out.

1:20:46 Speaker 2: So we have the measured VIIRS AOD at 550 and the interpolated ARINET AOD at 550. So the solid line represents the line of best fit. And the dashed line represents what theoretically the line should be if there's perfect collocation between ARINET and VIIRS data. So our R is 0.851. So R squared will be roughly around 0.65 or 0.7, which means that the collocation is fairly reliable and deterministic. The samples are 93, which means that there's 93 sites that this collocation has been done for. So yeah.

1:21:45 Speaker 2: More details on how to run the script, like how to set up your account, how to query LADS, how to authorize the LADS applications on your account, and how to generate a token can be found on the VEERS AirNet comparison readme file, which is once again on the AirNet GitHub.

1:22:10 Speaker 2: So thank you everybody for watching. running these scripts with me. I hope you enjoy them and I hope they are not too complicated. And now back over to Bhavan for the closing remarks. Thank you very much.

1:22:28 Speaker 1: Okay, great. Thank you, Peter, very much for excellent overview of the Python codes. Again, if anybody has trouble in running up those codes, please contact Peter. All the links, his contact and Details are there on the GitHub and on those individual codes. And we will be happy to help if anybody has struggles in accessing or running those codes. So, quickly, I want to summarize all the things which we have covered today. Again, Aeronet is a network of ground-based sun photometer systems. It makes aerosol measurements from around 600 active stations from all around the world.

1:23:14 Speaker 1: It's not surface measurement, but it's the total column measurements of black mass spheres. The data has been around for the last 30 years, extensively used in air quality, air pollution, climate change research, to validate the satellite, to validate the model outputs. In this particular part two, we have actually gone through different parts of our own web sites, which include interactive site maps, data visualization in many different ways, and then most importantly, data download, which is free for everyone to download and access in simple-to-use ASCII format.

1:23:54 Speaker 1: We have Bave API also for super data users, people who want to automate and display that data into their own system. APIs does that very cleverly and easily for everyone. And then Peter finally covered some of the Python tools which are available through the Aeronet GitHub page to download and analyze the data along with the satellite data in more quantitative analysis. And then finally, again, I think as I mentioned in the part one, Aeronet is a collaborative effort international community actually contribute to all the Aeronet data which are we distributing.

1:24:42 Speaker 1: So I really acknowledge every single person directly or indirectly involved in Aeronet, either site managers, PIs from all over the world, and the various network teams from partner networks, including NASA Goddard team, who actually works very hard to make this data available for the community. So, yeah, I think that's all from me. I thank you, RSAID team, for organizing this. I want to hand over back to Carl to continue from here. All right.

1:25:21 Speaker 4: Thank you very much, Pawan, and thank you, Peter, for that overview of how to access Aeronet data. Looking forward to the next part of this training, we'll be learning about Pandora, which is another ground-based passive remote sensor. which actually provides complementary information to the aerosol network, the aeronet network rather, on trace gases like ozone and nitrogen dioxide in the atmosphere. Just as a reminder, the homework for this will be posted at the end of our last session, which will be August 22nd, and you'll have about two weeks to complete that homework assignment before the deadline.

1:25:58 Speaker 4: Here's our contact information if you want to contact us. And again, there's an Aeronet mailing list you can register for if you're interested in using Aeronet data. And here's a summary of some of the links that we showed you to the different resources. So thank you very much for your attention today, and we'll now move into the question and answer session. And because we're... over our scheduled time, we're going to try to go through some of the high priority questions that are really focused on the tools and the data sets we talked about in the training.

1:26:36 Speaker 4: So I think we're going to start off a little bit lower. Again, we'll cover these and these answers will be posted to the website after the training, but because of time, we'll just focus on a few questions. So first, we'll start with question six. which is in the Aeronet downloading tool, what is the meaning of the different levels? And what are the implications of using the different AOD levels? This was, I think, yeah, go ahead.

1:27:07 Speaker 1: Yeah, thanks, Carl. So I think we discussed this in part one, but just I think it's nice to refresh that. So we have three levels of data, level one, level 1.5, and level two. Level 2 are the highest quality of the data which are done after the post-field calibration, and we recommend that for using for scientific research analysis. Level 1 are basically raw data without any cloud clearing, so we don't recommend those data unless you're looking for some other piece of information. Level 1.5 data are very similar to level 2 data in many ways, except it does not have the post-field calibration.

1:27:50 Speaker 1: So for many applications, people who need the data in real time, they can also use level 1.5. But for quantitative scientific research analysis, we always recommend using level 2 data.

1:28:07 Speaker 4: Okay, great. Thank you. I think our next question that we've prioritized was question 15 here. So on the tool and specific.