Electromagnetic Transfer Functions File Conversion Utilities (EMTF FCU), (C) 2007-2018 Anna Kelbert, is a set of Fortran 90 routines for electromagnetic transfer function file conversion and rotation.
EMTF FCU is written in Fortran 90 and may be compiled with any command-line or visual modern Fortran compiler. However, this EMTF FCU compilation guide assumes a command-line environment. This is straightforward under Linux or on a Mac: just open any shell terminal. On Windows, command-line environment is most easily achieved by installing the Cygwin package from https://www.cygwin.com. Windows 10 and later versions have an inbuilt Linux shell option: https://docs.microsoft.com/en-us/windows/wsl/install-win10
-
Install gfortran or similar Fortran 95 compiler.
-
Download the newest version of FoX library, https://github.com/andreww/fox and unpack it to a directory (call it
$FOX_LIB). -
Compile FoX package
cd $FOX_LIB
./configure
makeNB: if you move FoX in the file system, reconfigure
-
Go back to the EMTF FCU directory. Change the
$FOX_LIBvariable in the Makefile to your FoX path. Change$BIN_DIRvariable in the Makefile to an existing directory where you wish your executables to reside. -
Optional: to create XML files, set
homedirvariable in the header ofglobal.f90to the directory whereDATATYPESandCOPYRIGHTfolder reside (usually, same as code).If you intend to only run the codes from the file directory Where source codes reside,
homediris not needed.
NB: if you set homedir to a directory name that is longer than
80 chars, can encounter compile problems with fortran. In that
case, add the -ffree-line-length-none option to the compile flags.
Other compilers such as Intel should not encounter this problem.
- Compile EMTF FCU by typing
make.
The following standalone programs are available:
edi2xml
z2xml
xml2z
xml2edi
z2edi
Additionally, these standalone codes may be used for quick coordinate rotation:
xml2xml
edi2edi
z2z
Note that, other than for the XML files, there is no guarantee that the metadata will be preserved in the same exact format upon a coordinate rotation, or that no information will be lost.
For the above conversion codes, Perl script counterparts are available to perform batch processing.
edi2xml & z2xml:
depend on the FoX library and require additional metadata files for full functionality (see below)
xml2z & xml2edi:
depend on the FoX library but do not require any additional files; have limited functionality in the sense that they will only work correctly with those XML files that contain the right data types and statistical estimates for this type of storage
z2edi:
standalone code that requires no additional inputs
All conversion codes have an additional capability to rotate the transfer functions to an arbitrary coordinate system. This capability ought to be executed with great care for EDI files, which are inherently limited in that they do not store full covariance matrices. Rotation, therefore, invalidates the error bars in the files. While we try to do something reasonable in this case, rotation without a covariance matrix is mathematically invalid.
This comment only applies if you would like to use the software to create XML files for archiving or self-describing data storage, using z2xml or edi2xml.
In this case, you need to create an XML configuration file called config.xml and place it in the same directory as your input data. Example file pasted below.
This is where any user-defined information about the experiment is stored. You could also use the configuration file to specify one or all XML lists with additional metadata: Sites.xml, Runs.xml, Channels.xml. The lists, if specified, should contain experiment metadata information about runs and the sites, as described in module read_lists.f90. Examples below.
NB: For the XML lists, the attributes are optional; they are automatically inserted for easier reading with Matlab. All elements are also optional.
The program looks for the configuration file and the optional XML lists in the same directory where the input file is located. If the lists are not found, the programs run without the additional information.
NB: If you have the lists, use them by first running z2xml, then xml2edi. Note that the program z2edi does not require FoX library to be installed, and it does not use the additional XML information.
Use Case A: Convert EDI to XML, while rotating to geographic coordinates.
-
First, create a config.xml file and place it in the same directory as the original EDI file.
-
Make sure that the code is able to locate f90/DATATYPES and f90/COPYRIGHT. This is done by either copying them to your local directory, or by setting the "homedir" variable in global.f90 to your f90 folder.
-
Run
./edi2xml filename.edi filename.xml [verbose|silent] 0.0
Setting the third input variable to "silent" will suppress most output.
The forth input variable is any number (for an orthogonal coordinate system), or "sitelayout" to revert back to the original site orientations.
Rotation will be lossless is the original EDI file is an EDI SPECTRA. Otherwise, rotation will still be attempted but the output error bars may not be physically interpreted, since the original EDI does not have sufficient information for an unambiguous coordinate rotation.
Use Case B: Rotate EMTF XML to original site orientation.
Since the complete original site layout is now stored in the XML channels details, rotations of XML files are fully reversible.
./xml2xml geographic.xml original.xml [verbose|silent] sitelayout
Use Case C: Convert XML to Z-file without additional rotation.
./xml2z filename.xml filename.zrr
This code as previous released publicly and can be found here:
https://www.usgs.gov/software/electromagnetic-transfer-function-file-conversion-utilities-emtf-fcu
Bug fixes:
This release includes two major bug fixes.
- z2xml & z2edi.
In devising the XML schema, we have attempted to maintain compatibility with the historical Society of Exploration Geophysicists (SEG) Data Interchange Standard 1987, also known as EDI, to the extent possible. This includes maintaining consistency in the element names (e.g., Z, T, etc) and the statistical estimates, where relevant. However, the definition of variances in the EDI files, as we surmised after reading the manual extra carefully, deviates from the definition that we've been using for computation of variances from the covariance matrices in Z-files, by a factor of 2. Specifically, variances in EDI files are given for a complex number, not for the real or imaginary part, and are therefore twice those that we computed from Z-files. This inconsistency resulted in different definitions of Z.VAR and T.VAR between the data originally converted from Z-files, and data that came from the historic EDIs.
This update eliminates this inconsistency. From now on, all XML files include the variance
as defined by the historic EDI format, i.e., the variance of the complex transfer function
component. To compute the standard error of a real or imaginary part, e.g., for plotting,
one would divide the variance by 2, then take square root (i.e.
- edi2xml & xml2edi.
A second major bug fix involves correctly interpreting the rotations in EDI files. The old database had a warning to the effect that EDI files were not rotated. In the previous release of this database, we were trying to be careful about the error bars which can be corrupted if an EDI file is rotated in the absence of the full error covariance matrices. Unfortunately, this resulted in some occasions of file archiving where the data were not rotated to geographic, while the rotation information (ZROT) was not preserved in the archive. This is now corrected, and all data can now be rotated to geographic with a caveat that the error bars are not to be trusted on these occasions.
This fix also involves the rotation of tipper data in EDI files. Specifically, in the case of an EDI file that only has ZROT information (i.e., impedance rotation) but no indication as to how the tipper are rotated (e.g., no TROT), the code defaulted to assuming that the tippers are oriented according to the channels orientations. However, it is much more typical for people who omit TROT information to mean that the tippers are oriented the same way as the impedance. So we have now changed the code's behavior to default to ZROT information for the tippers, if TROT or any other indication of tipper rotations is missing.
Conceptual and schema changes:
We have updated the XML schema to include unambiguous rotation information. TF orientation is now stored in tag. Original site layout is always preserved in the block.
Code improvements:
The updated EMTF FCU v4.0 code includes numerous improvements to correctly parse and rotate many variants of EDI files. Among the major improvements are the ability to correctly read EDI SPECTRA files and compute the full error covariances from such files. This allows for arbitrary rotations of the transfer functions without loss of information, even after conversion to the XML.
We have also now implemented frequency-by-frequency rotations that allow conversion to XML of data that went through a principal axis rotation algorithm. This can now be undone for EDI files that include variable ZROT values, even though the resultant error estimates may suffer.
Finally, we have implemented a general rotation algorithm that allows for rotation from an arbitrary (not necessarily orthogonal) set of channel orientations, to an arbitrary orthogonal coordinate system, or back. All of these improvements have allowed us to archive a large number of historic MT TFs that were previously unaccessible for archiving.
Enjoy!
Anna Kelbert, USGS
Release Date: 11 Nov 2007 Last Update: 18 Sep 2018
Example config.xml as of 18 Sep 2018:
<Configuration>
<!-- set this to zero or omit if the time series are not archived at IRIS DMC -->
<TimeSeriesArchived>0</TimeSeriesArchived>
<Network>EM</Network>
<!-- optionally, specify an external URL for the data or metadata -->
<ExternalUrl></ExternalUrl>
<ExternalUrlInfo></ExternalUrlInfo>
<!-- Project / Survey combination used to allocate survey DOI; Project should contain no spaces -->
<Project>EMSOC</Project>
<Survey>Sant Andreas Fault</Survey>
<!-- Project.SiteID.YearCollected used to create a unique data ID in SPUD -->
<YearCollected>2005-2006</YearCollected>
<Country>USA</Country>
<!-- required, comma-separated list of data types. See list of possible types -->
<Tags>
impedance,tipper
</Tags>
<!-- data citation -->
<Citation>
<Title>Magnetotelluric transfer functions in the San Andreas Fault zone, CA, USA</Title>
<Authors>Becken, M., Ritter, O., Park, S. K., Bedrosian, P. A., Weckmann, U. and Weber, M.</Authors>
<Year>1999-2010</Year>
<SurveyDOI>doi:10.17611/DP/EMTF/EMSOC/SAF</SurveyDOI>
</Citation>
<THANKS>THANKS.txt</THANKS>
<PAPERS>PAPERS.txt</PAPERS>
<!-- optional additional copyright information -->
<README>readme.txt</README>
<!-- supported options: Unrestricted Release / Academic Use Only / Conditions Apply -->
<ReleaseStatus>Paper Citation Required</ReleaseStatus>
<!-- allows to supplement missing provenance and processing information from files -->
<AcquiredBy>Stephen K. Park</AcquiredBy>
<Creator>
<Name>Stephen K. Park</Name>
<Email>magneto@ucrmt.ucr.edu</Email>
<Org>University of California Riverside</Org>
<OrgUrl>http://www.ucr.edu/</OrgUrl>
</Creator>
<Submitter>
<Name>Lana Erofeeva</Name>
<Email>serofeev@coas.oregonstate.edu</Email>
<Org>Oregon State University</Org>
<OrgUrl>http://oregonstate.edu</OrgUrl>
</Submitter>
<ProcessedBy>Stephen K. Park</ProcessedBy>
<ProcessingSoftware>
<Name>Jimmy Larsen's robust processing code (using negative conjugate of output)</Name>
<LastMod>01/21/06</LastMod>
<Author></Author>
</ProcessingSoftware>
<SignConvention>exp(+ i\omega t)</SignConvention>
<!-- tells the conversion codes how to parse the EDI and whether to write the INFO block to XML -->
<ComputeSiteCoords>0</ComputeSiteCoords>
<DateFormat>MM/DD/YY</DateFormat>
<DummyDataValue>.10000000E+33</DummyDataValue>
<ChannelsOnTwoLines>1</ChannelsOnTwoLines>
<UseImpedanceRotationForAll>1</UseImpedanceRotationForAll>
<ParseEDIInfo>0</ParseEDIInfo>
<WriteEDIInfo>0</WriteEDIInfo>
<IgnoreSiteNameInFile>1</IgnoreSiteNameInFile>
<DefaultSiteName>Parkfield, CA, USA</DefaultSiteName>
<DefaultStartTime>2006-06-01T00:00:00</DefaultStartTime>
<DefaultEndTime>2006-06-01T00:00:00</DefaultEndTime>
<DefaultDataQuality>5</DefaultDataQuality>
<DataQualityComment>Default data quality assigned to the survey at archiving</DataQualityComment>
<!-- this option is for those who only wish to upload the metadata -->
<MetadataOnly>0</MetadataOnly>
<!-- if these extensions are present, the corresponding files are also submitted to SPUD, and displayed -->
<Image>png</Image>
<Original>edi</Original>
<!-- optional attachment - full name required -->
<Attachment></Attachment>
<AttachmentInfo></AttachmentInfo>
</Configuration>Example site list element as of 9 Sep 2013:
<Site idx="1" type="struct" size="1 1">
<ID idx="1" type="char" size="1 5">CAM01</ID>
<Description idx="1" type="char" size="1 18">Earl Lake, CA, USA</Description>
<Location idx="1" type="struct" size="1 1">
<Latitude idx="1" type="double" size="1 1">41.87624</Latitude>
<Longitude idx="1" type="double" size="1 1">-124.19000791667</Longitude>
<Elevation idx="1" type="double" size="1 1">0.975</Elevation>
</Location>
<Declination idx="1" type="double" size="1 1">16.6</Declination>
<TimePeriod idx="1" type="struct" size="1 1">
<Start idx="1" type="char" size="1 19">2007-08-31T21:51:38</Start>
<End idx="1" type="char" size="1 19">2007-09-20T18:42:08</End>
</TimePeriod>
<QualityRating idx="1" type="double" size="1 1">5</QualityRating>
<GoodFromPeriod idx="1" type="double" size="1 1">10</GoodFromPeriod>
<GoodToPeriod idx="1" type="double" size="1 1">20000</GoodToPeriod>
<QualityComments idx="1" type="char" size="1 42">great TF from 10 to 10000 secs (or longer)</QualityComments>
<BestTF idx="1" type="char" size="1 15">CAM01bc_K1x.zrr</BestTF>
<RunList idx="1" type="char" size="1 20">CAM01a CAM01b CAM01c</RunList>
<NoGPS idx="1" type="logical" size="1 1">0</NoGPS>
<Comments idx="1" type="char" size="1 65">Data Problem Report available at http://www.iris.edu/data/dpr.htm</Comments>
</Site>Example run list element as of 9 Sep 2013:
<Run idx="1" type="struct" size="1 1">
<ID idx="1" type="char" size="1 6">CAM01a</ID>
<siteID idx="1" type="char" size="1 5">CAM01</siteID>
<Manufacturer idx="1" type="char" size="1 11">Barry Narod</Manufacturer>
<Instrument idx="1" type="char" size="1 12">NIMS 2612-04</Instrument>
<Location idx="1" type="struct" size="1 1">
<Latitude idx="1" type="double" size="1 1">41.87636</Latitude>
<Longitude idx="1" type="double" size="1 1">-124.19003166667</Longitude>
<Elevation idx="1" type="double" size="1 1">2.8</Elevation>
</Location>
<Declination idx="1" type="double" size="1 1">16.6</Declination>
<TimePeriod idx="1" type="struct" size="1 1">
<Start idx="1" type="char" size="1 19">2007-08-31T21:51:38</Start>
<End idx="1" type="char" size="1 19">2007-08-31T22:52:15</End>
</TimePeriod>
<Ex_wire_length idx="1" type="double" size="1 1">100</Ex_wire_length>
<Ey_wire_length idx="1" type="double" size="1 1">100</Ey_wire_length>
<SamplingInterval idx="1" type="double" size="1 1">1</SamplingInterval>
<ClockOffset idx="1" type="double" size="1 1">0</ClockOffset>
<NewEpoch idx="1" type="double" size="1 1">1</NewEpoch>
<Count idx="1" type="double" size="1 1">1</Count>
<NChannels idx="1" type="double" size="1 1">5</NChannels>
<SiteInstalledBy idx="1" type="char" size="0 0"/>
<FieldComments idx="1" type="char" size="0 0"/>
<MetaDataCheckedBy idx="1" type="char" size="0 0"/>
<Comments idx="1" type="char" size="0 0"/>
<Errors idx="1" type="char" size="0 0"/>
</Run>Example channel list element as of 9 Sep 2013:
<Channel idx="1" type="struct" size="1 1">
<ID idx="1" type="char" size="1 3">LFN</ID>
<Name idx="1" type="char" size="1 2">Hx</Name>
<siteID idx="1" type="char" size="1 5">CAM01</siteID>
<runID idx="1" type="char" size="1 6">CAM01a</runID>
<Instrument idx="1" type="char" size="1 12">Magnetometer</Instrument>
<InstrumentType idx="1" type="char" size="1 8">fluxgate</InstrumentType>
<Manufacturer idx="1" type="char" size="1 11">Barry Narod</Manufacturer>
<InstrumentName idx="1" type="char" size="1 4">NIMS</InstrumentName>
<InstrumentID idx="1" type="char" size="1 7">2612-04</InstrumentID>
<InstrumentConfig idx="1" type="char" size="0 0"/>
<TimePeriod idx="1" type="struct" size="1 1">
<Start idx="1" type="char" size="1 19">2007-08-31T21:51:38</Start>
<End idx="1" type="char" size="1 19">2007-08-31T22:52:15</End>
</TimePeriod>
<SamplingRate idx="1" type="struct" size="1 1">
<Value idx="1" type="double" size="1 1">1</Value>
<Units idx="1" type="char" size="1 2">Hz</Units>
</SamplingRate>
<DipoleLength idx="1" type="double" size="1 1">0</DipoleLength>
<Orientation idx="1" type="double" size="1 1">16.6</Orientation>
<LowPassCutoff idx="1" type="double" size="1 1">0.5</LowPassCutoff>
<HighPassCutoff idx="1" type="double" size="1 1">0</HighPassCutoff>
<TimeOffset idx="1" type="double" size="1 1">-0.192</TimeOffset>
<Correction idx="1" type="struct" size="1 1">
<Value idx="1" type="double" size="1 1">0</Value>
<Type idx="1" type="char" size="1 10">base shift</Type>
</Correction>
<Conversion idx="1" type="double" size="1 1">0.01</Conversion>
<Comments idx="1" type="char" size="1 36">x 0.01 to get nT, Hx base shift 0 nT</Comments>
</Channel>
```xml
-------------------------------------------------------------------
FoX usage example:
``` Fortran
doc => parseFile("input.xml")
nlist => getElementsByTagname(doc, "interestingElement")
do i = 0, getLength(nlist)-1
np => item(nlist, i)
call extractDataAttribute(np, "att", value)
if (value==somethingInteresting) then
call extractDataContent(np, data)
endif
enddo
call destroy(doc)