In this example, we are using the sgclear platform to handle the reflectance spectra of small areas (3-10km in diameter) on the lunar surface. There are about 400 spectra acquired by the 2.2m telescope on Mauna Kea, Hawaii in the late 1970s and 1980s. Each spectrum has 120 spectral channels from 0.62 to 2.6 micrometers, measured using the McCord two-component circular-variable-filter (CVF) near-infrared photometer.

Prepare the experiment datasets

1. Download the data: the spectral reflectance measurements in this project are all open for public download mkls0001.zip, unzip the .zip file.
2. go to the experiment folder,such as ~/lunarspec/ and create soft link to the data folder
• cd ~/lunarspec
• ln -s /data/lunarspec/mkls_0001/data DATA
You should sees folders like h8xxxx, h9xxxx in DATA. Each of the folder is an indepedent measurement.

About the experiment script

There are seven measurement experiments in this project, done over more than 10 years, stored in subfolder, as h8xxxx, h9xxxx, haxxxx, hbxxxx, hcxxxx, hdxxxx, hexxxx in DATA. In each experiment, there tens of measurements, stored as DATA/h8xxxx/h80409.tab and together with the description file DATA/h8xxxx/h80409.lbl.

In order to make the experiment tidy, we define a function named loadspec in lunarspec.py to load and merge all the spectra in a measurement experiment. This function is called in the main script SConstruct.

lunarspec.py

1. the function is defined as general python functions, with only one parameter name as the experiment folder, such as name=h8xxxx

from experiment import *

def loadspec(name):

2. in the function, we loop all .tab files in the specific experiment, and load then into sigclear dataset

	lst1=Glob('DATA/%s/*.tab'%name)
	lst2 =[]
	for tab in lst1:
		it1 = str(tab)
		it=it1[12:18]
		group=int(it[1], base=16)-int("a", base=16)+10
		Process(it, it,
			'''
			sed '1iwavelength,reflectance,uncertainty'
			| sgloadcsv 
			| sgsort key=group,wavelength
			| sgfieldmath group=%d id:i=%s 
			'''%(group, it[2:]),
			sprefix='DATA/%s/'%name, ssuffix=".tab")
		lst2.append(it)

3. we combine all the loaded files into one file

	Process('raw-'+name, lst2, 
		'sgcat ${SOURCES[1:]} | sgsort key=id,wavelength')

4. There are lots of invalid measurements in each datasets. We show an simple example to add an data editing by selecting measured reflectance value larger than 0.5, but smaller than 2.0 as follows

	Process('filter-'+name, 'raw-'+name,
		'''
		sgwindow reflectance.min=0.5 reflectance.max=2
		''')

SConstruct

1. With the defined loadspec, we loop and load all the measurement experiments, in the main experiment script as easy as follows

from lunarspec import *
lst1=Glob('DATA/h*')

for it1 in lst1:
	it2 = str(it1)
	it=it2[5:]
	loadspec(it)

2. we QC all the raw measurements in one experiments

Figure('./raw-h8xxxx.png',
	'''
	sgfieldout fields=wavelength,reflectance,uncertainty
	| sgsort key=group,id
	| sgplotps
	''')

the raw data is as follows



3. we can either QC the edited reflectance as

   Figure('./filter-h8xxxx.png', 
   	'''
   	sgfieldout fields=wavelength,reflectance
   	| sgsort key=group,id
   	| sggraphps 
   	  x=wavelength y=reflectance reflectance.color=id
   	''')
   

   The graph of the filtered reflectance spectra is as

   

More about this experiment

• As there are about 400 measurements (.tab files) in this project, to accelerate the experiment, we can run in parallel simply as
• scons -j 8
These 400 .tab files will loaded into sigclear datasets with 8 in parallel (if there are at least 8 cores in the computing node).
• Please be free to change the Figure parameters or add more processes in the SConstruct script. As you rerun the script, the loading processes for the 400 .tab files should never be executed. The benefit is powered by sigclear-experiment.
• Please be free to add more your customized experiment processes in lunarspec.py. Moreover, you can build a package for lunarspec.py, distribute it to your colleagues and install it in other computing nodes.

References:

1. Learn more about the NASA project of lunar reflectance spectra measurements