Skip to content

Commit 1348eb7

Browse files
authored
Merge branch 'master' into cmr-10213-new-jetty
2 parents 1b7f866 + 327c850 commit 1348eb7

28 files changed

Lines changed: 4539 additions & 789 deletions

File tree

common-app-lib/src/cmr/common_app/config.clj

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
(defconfig collection-umm-version
99
"Defines the latest collection umm version accepted by ingest - it's the latest official version.
1010
This environment variable needs to be manually set when newer UMM version becomes official"
11-
{:default "1.18.3"})
11+
{:default "1.18.4"})
1212

1313
(defconfig launchpad-token-enforced
1414
"Flag for whether or not launchpad token is enforced."

es-spatial-plugin/README.md

Lines changed: 167 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,172 @@ To package the spatial script for use in elastic:
2727

2828
This will create a zip in target/ ready for installation in elasticsearch.
2929

30+
31+
Elastic Search 8.x - tested on 8.0.0, 8.5.0, 8.10.4, 8.15.5 - These instructions were generated May 2025.
32+
1) compile CMR
33+
34+
2) After CMR has been compiled change the following:
35+
es-spatial-plugin/project.clj:
36+
change the provided profile:
37+
set the elastic version to "8.15.5"
38+
in the exclusions section for both cmr-common-lib and cmr-spatial-lib comment out the following:
39+
;[com.fasterxml.jackson.core/jackson-core]
40+
;[com.fasterxml.jackson.dataformat/jackson-dataformat-cbor]
41+
;[com.fasterxml.jackson.dataformat/jackson-dataformat-smile]
42+
43+
es-spatial-plugin/resources/plugin/plugin-descriptor.properties so that elasticsearch.version=8.15.5
44+
45+
To get the spatial plug in to work on Elastic Search 8.x I had to only use the cmr-es-spatial-plugin-0.1.0-SNAPSHOT.jar
46+
47+
then create the jar file
48+
49+
lein with-profile es-plugin,provided jar
50+
51+
52+
3) Create a directory to work in.
53+
54+
4) create a zip directory in your working directory. I called mine zip8.15 and copied in
55+
es-spatial-plugin/target/cmr-es-spatial-plugin-0.1.0-SNAPSHOT.jar
56+
es-spatial-plugin/resources/plugin/plugin-descriptor.properties
57+
58+
In the directory above the zip file (the working directory) copy
59+
es-spatial-plugin/resources/plugin/plugin-security.policy
60+
61+
The cmr-es-spatial-plugin-0.1.0-SNAPSHOT.jar contains the compiled .class bytecode of the plugin classes.
62+
63+
5) in the es-spatial-plugin project run
64+
lein deps :tree
65+
This command gives a heirarchical listing of dependency libraries. Most of these libraries we do not
66+
need because they are either for testing or for parts of cmr-common, that the spatial library does not use.
67+
68+
Then I decided on a set of jar files that I knew I needed and went into my personal maven repository and copied the
69+
libraries into my zip8.15 directory. My maven repository is located in my home directory under a directory called .m2
70+
71+
The set of jar files that I ended up with are:
72+
clojure-utils-0.6.1.jar core.matrix-0.54.0.jar jackson-dataformat-cbor-2.13.3.jar
73+
math.combinatorics-0.1.4.jar randomz-0.3.0.jar assertions-0.2.0.jar
74+
cmr-common-lib-0.1.1-SNAPSHOT.jar edn-java-0.4.4.jar jackson-dataformat-smile-2.13.3.jar
75+
mathz-0.3.0.jar spec.alpha-0.3.218.jar cheshire-5.12.0.jar
76+
cmr-es-spatial-plugin-0.1.0-SNAPSHOT.jar jafama-2.3.1.jar plugin-descriptor.properties
77+
vectorz-0.44.0.jar clojure-1.11.2.jar cmr-spatial-lib-0.1.0-SNAPSHOT.jar
78+
jackson-core-2.13.3.jar joda-time-2.8.1.jar primitive-math-0.1.4.jar
79+
vectorz-clj-0.28.0.jar
80+
81+
The way that I figured out the above jars is the following:
82+
83+
Bring up Elastic Search 8.15.8:
84+
6) start up docker - I use docker desktop
85+
7) Use the directory that was created in step 3. You will run elastic search from here.
86+
8) Create a file in this directory called docker-compose.yml. Put in the following into the file and save it:
87+
Note: it is best not to use a gui to create the file - upon another developer testing these instructions extra spaces
88+
were inserted and we had to fix the copy and paste errors.
89+
90+
```
91+
services:
92+
elasticsearch:
93+
image: elasticsearch:8.15.5
94+
ports:
95+
- 9210:9200
96+
- 9300:9300
97+
environment:
98+
- discovery.type=single-node
99+
- xpack.security.enabled=false
100+
- ES_JAVA_OPTS=-Djava.security.policy=/usr/share/elasticsearch/config/plugin-security.policy
101+
volumes:
102+
- ./esdata:/usr/share/elasticsearch/data
103+
networks:
104+
elasticsearch-network:
105+
aliases:
106+
- elasticsearch
107+
# kibana:
108+
# container_name: cmr-kibana
109+
# depends_on:
110+
# - elasticsearch
111+
# image: docker.elastic.co/kibana/kibana:8.17.5
112+
# ports:
113+
# - 5601:5601
114+
# networks:
115+
# - elasticsearch-network
116+
117+
networks:
118+
elasticsearch-network: {}
119+
```
120+
121+
9) Note that you don't need the kibana section - as it just creates more junk in the logs to have to look at, so it is commented out.
122+
I included here so that others may see how to bring up both if they need it sometime in the future.
123+
124+
10) Open up another termial window to the same directory. In this new termainal and in the same directory where docker-compose.yml exists run the command:
125+
docker compose up
126+
127+
This will load up the elastic search 8.15.5 image from the web and store it in your docker image. Then will will start up elastic search 8.15.5.
128+
It will also create a new directory called esdata in your directory. This is where elastic search will store its data so that the data is retained
129+
after elastic search is stopped and restarted. This window will also print the elastic search logs.
130+
131+
11) Once it has started, go back INTO your zip directory (mine is called zip8.15) and zip up the files using the following command:
132+
zip -r cmr-es-spatial-plugin-8.15.5.zip \
133+
clojure-1.11.2.jar \
134+
spec.alpha-0.3.218.jar \
135+
cmr-common-lib-0.1.1-SNAPSHOT.jar \
136+
cmr-spatial-lib-0.1.0-SNAPSHOT.jar \
137+
jackson-dataformat-cbor-2.13.3.jar \
138+
jackson-core-2.13.3.jar \
139+
jackson-dataformat-smile-2.13.3.jar \
140+
joda-time-2.8.1.jar \
141+
primitive-math-0.1.4.jar \
142+
jafama-2.3.1.jar \
143+
core.matrix-0.54.0.jar \
144+
vectorz-clj-0.28.0.jar \
145+
clojure-utils-0.6.1.jar \
146+
vectorz-0.44.0.jar \
147+
mathz-0.3.0.jar \
148+
randomz-0.3.0.jar \
149+
edn-java-0.4.4.jar \
150+
assertions-0.2.0.jar \
151+
math.combinatorics-0.1.4.jar \
152+
cheshire-5.12.0.jar \
153+
cmr-es-spatial-plugin-0.1.0-SNAPSHOT.jar \
154+
plugin-descriptor.properties
155+
156+
12) copy cmr-es-spatial-plugin-8.15.5.zip to your working directory
157+
158+
13) run the following to get the plugin into the docker container - my docker container name is es-spatial-plugin-elasticsearch-1.
159+
if yours is different then you will need to modify the command to use your container name. You will get an error when trying to copy if
160+
your container name is wrong. You can find your container name in docker desktop.
161+
162+
docker cp cmr-es-spatial-plugin-8.15.5.zip es-spatial-plugin-elasticsearch-1:/tmp/
163+
164+
14) copy the plugin-security.policy that is in your working directory from step 4 to the config directory in the elastic container:
165+
docker cp plugin-security.policy es-spatial-plugin-elasticsearch-1:/usr/share/elasticsearch/config/plugin-security.policy
166+
167+
15) run the following command to run a shell that executes inside the docker container
168+
docker exec -it es-spatial-plugin-elasticsearch-1 /bin/bash
169+
170+
16) run the following command to install the spatial plugin
171+
elasticsearch-plugin install file:///tmp/cmr-es-spatial-plugin-8.15.5.zip
172+
173+
The plugin should install
174+
175+
17) restart Elastic search: in the docker desktop click on the vertical ... on the right side of the elastic search container and select restart
176+
177+
18) Bring up your external CMR database
178+
179+
19) bring up CMR
180+
since I am using an external database and elastic, after the first time I ran the CMR, I needed to comment out (bootstrap/bootstrap started-system)
181+
from access-control-app/src/cmr/access_control/system.clj because the ACLs already exist. The CMR won't come up because access control fails because the ACLs
182+
already exist.
183+
184+
20) ingest a collection record that contains a spatial extent - mine used a bounding box.
185+
186+
21) search for that record using a spatial extent search ( a spatial extent search will invoke the spatial plugin.) You should get your search
187+
response back without crashing elastic search.
188+
189+
During the process of figuring out the libraries, elastic search would crash. In the elastic search logs, either from the container or the terminal
190+
where I ran docker compose up, look for Cause and also ClassNotFoundException. Then I figured out which library contained that class. Then I copied
191+
that jar file into my zip directory, zipped up my file. Then I had to delete my elastic search container, delete the esdata directory, bring down CMR.
192+
Then I had to follow the above instructions to bring everything back up and try again. I kept up this process, until I got search results back without
193+
crashing elastic search.
194+
195+
30196
## License
31197

32-
Copyright © 2021 NASA
198+
Copyright © 2021-2025 NASA

schemas/resources/schemas/collection-draft/v1.0.0/metadata.json

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -264,8 +264,8 @@
264264
"DOI" : "10.1234/DOIID",
265265
"Authority" : "https://doi.org/"
266266
},
267-
"ShortName" : "Mapping Short Name 1.18.3",
268-
"EntryTitle" : "Mapping Example for UMM-C 1.18.3",
267+
"ShortName" : "Mapping Short Name 1.18.4",
268+
"EntryTitle" : "Mapping Example for UMM-C 1.18.4",
269269
"PublicationReferences" : [ {
270270
"OnlineResource" : {
271271
"Linkage" : "http://www.yahoo.com",
@@ -814,9 +814,9 @@
814814
"LongName" : "MUSES"
815815
} ],
816816
"MetadataSpecification" : {
817-
"URL" : "https://cdn.earthdata.nasa.gov/umm/collection/v1.18.3",
817+
"URL" : "https://cdn.earthdata.nasa.gov/umm/collection/v1.18.4",
818818
"Name" : "UMM-C",
819-
"Version" : "1.18.3"
819+
"Version" : "1.18.4"
820820
},
821821
"ArchiveAndDistributionInformation" : {
822822
"FileArchiveInformation" : [ {

system-validation-test/Gemfile

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,10 @@
33
source 'https://rubygems.org'
44

55
group :test do
6-
gem 'cucumber'
6+
gem 'cucumber', '~> 9.2.1'
77
gem 'httparty'
88
gem 'jsonpath'
99
gem 'nokogiri', "= 1.18.3"
1010
gem 'rspec'
11+
gem 'ffi', '~> 1.16.3'
1112
end

system-validation-test/Gemfile.lock

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
GEM
22
remote: https://rubygems.org/
33
specs:
4-
bigdecimal (3.1.8)
5-
bigdecimal (3.1.8-java)
6-
builder (3.2.4)
4+
bigdecimal (3.1.9)
5+
bigdecimal (3.1.9-java)
6+
builder (3.3.0)
77
csv (3.3.0)
8-
cucumber (9.2.0)
8+
cucumber (9.2.1)
99
builder (~> 3.2)
1010
cucumber-ci-environment (> 9, < 11)
1111
cucumber-core (> 13, < 14)
@@ -18,18 +18,18 @@ GEM
1818
multi_test (~> 1.1)
1919
sys-uname (~> 1.2)
2020
cucumber-ci-environment (10.0.1)
21-
cucumber-core (13.0.2)
21+
cucumber-core (13.0.3)
2222
cucumber-gherkin (>= 27, < 28)
2323
cucumber-messages (>= 20, < 23)
2424
cucumber-tag-expressions (> 5, < 7)
2525
cucumber-cucumber-expressions (17.1.0)
2626
bigdecimal
2727
cucumber-gherkin (27.0.0)
2828
cucumber-messages (>= 19.1.4, < 23)
29-
cucumber-html-formatter (21.3.1)
30-
cucumber-messages (> 19, < 25)
29+
cucumber-html-formatter (21.9.0)
30+
cucumber-messages (> 19, < 28)
3131
cucumber-messages (22.0.0)
32-
cucumber-tag-expressions (6.1.0)
32+
cucumber-tag-expressions (6.1.2)
3333
diff-lcs (1.5.1)
3434
ffi (1.16.3)
3535
ffi (1.16.3-java)
@@ -66,18 +66,20 @@ GEM
6666
diff-lcs (>= 1.2.0, < 2.0)
6767
rspec-support (~> 3.13.0)
6868
rspec-support (3.13.1)
69-
sys-uname (1.2.3)
69+
sys-uname (1.3.1)
7070
ffi (~> 1.1)
7171

7272
PLATFORMS
7373
arm64-darwin-21
7474
arm64-darwin-23
7575
universal-java-17
7676
x86_64-darwin-22
77+
x86_64-darwin-23
7778
x86_64-linux
7879

7980
DEPENDENCIES
80-
cucumber
81+
cucumber (~> 9.2.1)
82+
ffi (~> 1.16.3)
8183
httparty
8284
jsonpath
8385
nokogiri (= 1.18.3)

umm-lib/resources/schema/dif10/UmmCommon_1.2.xsd

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -585,6 +585,7 @@ technical contact
585585
<xs:enumeration value="Other"/>
586586
<xs:enumeration value="IsPreviousVersionOf"/>
587587
<xs:enumeration value="IsNewVersionOf"/>
588+
<xs:enumeration value="IsDescribedBy"/>
588589
</xs:restriction>
589590
</xs:simpleType>
590591
<xs:simpleType name="AssociatedDOITypeOtherEnum">

umm-lib/resources/schema/echo10/MetadataCommon.xsd

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -819,6 +819,7 @@
819819
<xs:enumeration value="Other"/>
820820
<xs:enumeration value="IsPreviousVersionOf"/>
821821
<xs:enumeration value="IsNewVersionOf"/>
822+
<xs:enumeration value="IsDescribedBy"/>
822823
</xs:restriction>
823824
</xs:simpleType>
824825
</xs:element>

umm-lib/resources/schema/iso_mends/resources/Codelist/eosCodelists.xml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -245,6 +245,11 @@
245245
<gml:description>part of a set of larger works that is identified by a digital object identifier.</gml:description>
246246
<gml:identifier codeSpace="gov.nasa.esdis.umm">DOI_Previous_Version</gml:identifier>
247247
</gmx:CodeDefinition>
248+
</gmx:codeEntry> <gmx:codeEntry>
249+
<gmx:CodeDefinition gml:id="EOS_AssociationTypeCode_isDescribedBy">
250+
<gml:description>This collection is described by the associated DOI.</gml:description>
251+
<gml:identifier codeSpace="gov.nasa.esdis.umm">Is_Described_By</gml:identifier>
252+
</gmx:CodeDefinition>
248253
</gmx:codeEntry>
249254
</gmx:CodeListDictionary>
250255
</gmx:codelistItem>

umm-lib/resources/schema/iso_smap/resources/Codelist/eosCodelists.xml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -246,6 +246,12 @@
246246
<gml:identifier codeSpace="gov.nasa.esdis.umm">DOI_Previous_Version</gml:identifier>
247247
</gmx:CodeDefinition>
248248
</gmx:codeEntry>
249+
<gmx:codeEntry>
250+
<gmx:CodeDefinition gml:id="EOS_AssociationTypeCode_isDescribedBy">
251+
<gml:description>The collection is described by the associated DOI.</gml:description>
252+
<gml:identifier codeSpace="gov.nasa.esdis.umm">Is_Described_By</gml:identifier>
253+
</gmx:CodeDefinition>
254+
</gmx:codeEntry>
249255
</gmx:CodeListDictionary>
250256
</gmx:codelistItem>
251257
</gmx:CT_CodelistCatalogue>

0 commit comments

Comments
 (0)