Errors when connecting to our ElasticSearch 5.6.16 deployment

[reefdog]

We've successfully setup and run the local Dockerized query parser 🎉. However, when we configure it to point to our new ElasticSearch 5.6.16 deployment and run a search, we receive this error:

08-Aug-2019 19:22:52.878 SEVERE [http-nio-5000-exec-2] com.sun.jersey.spi.container.ContainerResponse.mapMappableContainerException The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
    org.codehaus.jettison.json.JSONException: JSONObject["raw"] not found.
        at org.codehaus.jettison.json.JSONObject.get(JSONObject.java:360)
        at org.codehaus.jettison.json.JSONObject.getJSONObject(JSONObject.java:454)
        at com.cloud.uspto.service.UsptoService.getResults(UsptoService.java:147)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
...

@shikham01 has been made aware, has been provided access to our ElasticSearch deployment, and is currently debugging.

[shikham01]

I can see following missing in v2

{
  "meta": {
    "raw": {
      "UploadDate": "2016-01-27T17:01:47.000Z",
      "PushDate": "Mon May  5 18:53:46 PDT 1997"
    }
  }
}

[reefdog]

@shikham01 Thanks for looking into that!

I'm not sure where this is supposed to be returned. Is that data that's supposed to be returned by the search endpoint? Does that mean we need to modify one of the templates or index definitions? Basically I'm not sure how to resolve the issue given the above.

[shikham01]

do we have the pipeline(fscrawler_pipeline) applied to this instance. we may want to delete all the records, then apply the pipeline and then index few docs

[reefdog]

Yes, if you go to the API Console in the cloud.elastic.co dashboard and perform a GET to _ingest/pipeline/fscrawler_pipeline, you'll see the output below. Do we need to change something?

{
  "fscrawler_pipeline": {
    "description": "Converts fscrawler crawled docs into uspto searchable docs",
    "processors": [
      {
        "script": {
          "lang": "painless",
          "source": "if (params.extension1.contains(ctx.file.extension)) {ctx.custom_meta_data.customFileType = params.custom1} else if (params.extension2.contains(ctx.file.extension)) {ctx.custom_meta_data.customFileType = params.custom2} else if (params.extension3.contains(ctx.file.extension)) {ctx.custom_meta_data.customFileType = params.custom3}else if (params.extension4.contains(ctx.file.extension)) {ctx.custom_meta_data.customFileType = params.custom4}else if (params.extension5.contains(ctx.file.extension)) {ctx.custom_meta_data.customFileType = params.custom5} else if (params.extension6.contains(ctx.file.extension)) {ctx.custom_meta_data.customFileType = params.custom6}else {ctx.custom_meta_data.customFileType =params.custom7}",
          "params": {
            "custom4": "SpreadSheets",
            "custom5": "Word & Text Docs",
            "custom6": "Videos",
            "custom7": "Others",
            "custom1": "Web Pages",
            "custom2": "PDFs",
            "custom3": "Presentations",
            "extension6": [
              "3gp",
              "3g2",
              "aiff",
              "asf",
              "au",
              "avi",
              "flac",
              "flv",
              "m4a",
              "m4p",
              "mid",
              "midi",
              "rmi",
              "mov",
              "mp2",
              "mp3",
              "mp4",
              "mpg4",
              "m4v",
              "mp4v",
              "divx",
              "mpg",
              "m2v",
              "ogg",
              "rm",
              "ram",
              "swf",
              "wav",
              "wma",
              "wmv",
              "wm",
              "wmz"
            ],
            "extension5": [
              "docx",
              "dotx",
              "docm",
              "dotm",
              "txt",
              "ini",
              "props",
              "properties",
              "text",
              "c",
              "c++",
              "conf",
              "cpp",
              "css",
              "cxx",
              "h",
              "hh",
              "js",
              "log",
              "lst",
              "pas",
              "pl",
              "s",
              "sh",
              "tsv",
              "uri",
              "uris",
              "wmls",
              "dtd",
              "java",
              "jsf",
              "jav",
              "nt",
              "pv",
              "latex",
              "tex",
              "faq",
              "csv",
              "asc",
              "ans",
              "me"
            ],
            "extension4": [
              "xls",
              "xlt",
              "xlsx",
              "xltx",
              "xlsm",
              "xltm",
              "xlsb",
              "xlam"
            ],
            "extension3": [
              "ppt",
              "pot",
              "pps",
              "pptx",
              "potx",
              "pptm",
              "potm",
              "ppsx",
              "ppam",
              "ppsm"
            ],
            "extension2": [
              "pdf",
              "PDF"
            ],
            "extension1": [
              "html",
              "htm"
            ]
          }
        }
      },
      {
        "date": {
          "timezone": "UTC",
          "field": "meta.raw.UploadDate",
          "ignore_failure": true,
          "target_field": "meta.raw.UploadDate",
          "formats": [
            "EEE MMM d HH:mm:ss zzz yyyy"
          ]
        }
      },
      {
        "script": {
          "lang": "painless",
          "source": "if (ctx.custom_meta_data.customFileType==params.custom1) {ctx.custom_meta_data.publishDate=ctx.meta.raw.PushDate} else {ctx.custom_meta_data.publishDate=ctx.meta.date}",
          "params": {
            "custom1": "Web Pages",
            "custom2": "PDFs"
          }
        }
      },
      {
        "lowercase": {
          "field": "_index"
        }
      },
      {
        "script": {
          "lang": "painless",
          "source": "if (ctx._index==params.usptoreindex) {ctx._index='reidx-'+ctx.custom_meta_data.companyid} else {ctx._index='idx-'+ctx.custom_meta_data.companyid}"
        }
      },
      {
        "date": {
          "timezone": "UTC",
          "field": "custom_meta_data.publishDate",
          "ignore_failure": true,
          "target_field": "custom_meta_data.publicationDate",
          "formats": [
            "yyyy-MM-dd'T'HH:mm:ss",
            "EEE MMM  d HH:mm:ss zzz yyyy"
          ]
        }
      },
      {
        "remove": {
          "field": [
            "path",
            "file.filename",
            "file.content_type",
            "meta.format",
            "meta.raw.date",
            "meta.raw.modified",
            "meta.raw.created",
            "meta.raw.Creation-Date",
            "meta.raw.resourceName",
            "meta.raw.Author",
            "meta.raw.producer",
            "meta.raw.description",
            "meta.raw.Last-Modified",
            "meta.raw.title",
            "meta.raw.Last-Save-Date",
            "meta.raw.Content-Type",
            "meta.raw.creator",
            "meta.raw.access_permission:fill_in_form",
            "meta.author",
            "meta.created",
            "meta.description",
            "meta.raw.pdf:PDFVersion",
            "meta.raw.pdf:docinfo:title",
            "meta.raw.access_permission:can_print_degraded",
            "meta.raw.subject",
            "meta.raw.pdf:docinfo:custom:copyrightYear",
            "meta.raw.dc:format",
            "meta.raw.pdf:encrypted",
            "meta.raw.cp:subject",
            "meta.raw.pdf:docinfo:subject",
            "meta.raw.pdf:docinfo:creator",
            "meta.raw.meta:author",
            "meta.raw.meta:creation-date",
            "meta.raw.access_permission:extract_for_accessibility",
            "meta.raw.pdf:docinfo:custom:UploadDate",
            "meta.raw.pdf:docinfo:custom:title",
            "meta.raw.pdf:docinfo:producer",
            "meta.raw.dc:description",
            "meta.raw.pdf:docinfo:created",
            "meta.raw.access_permission:can_modify",
            "meta.raw.pdf:docinfo:custom:copyright",
            "meta.raw.access_permission:can_print",
            "meta.raw.access_permission:extract_content",
            "meta.raw.xmpTPg:NPages",
            "meta.raw.access_permission:assemble_document",
            "meta.raw.meta:save-date",
            "meta.raw.pdf:docinfo:modified",
            "meta.raw.dcterms:modified",
            "meta.raw.dcterms:created",
            "meta.raw.dc:creator",
            "meta.raw.access_permission:modify_annotations",
            "meta.raw.dc:title",
            "meta.raw.X-Parsed-By"
          ],
          "on_failure": [
            {
              "remove": {
                "field": []
              }
            }
          ]
        }
      },
      {
        "remove": {
          "field": [
            "meta.raw.X-Parsed-By"
          ],
          "on_failure": [
            {
              "remove": {
                "field": []
              }
            }
          ]
        }
      },
      {
        "remove": {
          "field": [
            "meta.raw.dc:title"
          ],
          "on_failure": [
            {
              "remove": {
                "field": []
              }
            }
          ]
        }
      },
      {
        "remove": {
          "field": [
            "meta.raw.Content-Encoding"
          ],
          "on_failure": [
            {
              "remove": {
                "field": []
              }
            }
          ]
        }
      },
      {
        "remove": {
          "field": [
            "meta.raw.resourceName"
          ],
          "on_failure": [
            {
              "remove": {
                "field": []
              }
            }
          ]
        }
      },
      {
        "remove": {
          "field": [
            "meta.raw.Content-Type"
          ],
          "on_failure": [
            {
              "remove": {
                "field": []
              }
            }
          ]
        }
      },
      {
        "remove": {
          "field": [
            "meta.raw.title"
          ],
          "on_failure": [
            {
              "remove": {
                "field": []
              }
            }
          ]
        }
      }
    ]
  }
}

[shikham01]

When we apllied this pipeline. Just want to confirm that we created fresh index after applying this pipeline and deleted all the old records.

[reefdog]

This was a brand new deployment, so there was no existing records. After applying this and the other configs, we uploaded a few documents (which we can successfully search for using simple queries on the Lambda query parser).

Have you been able to log in to the cloud.elastic.co dashboard and verify configuration through the API Console?

[shikham01]

can we comment out following lines (238-250) in usptoservice.java and then deploy the code once

if(esSingleHitSourceMeta.has(SearchConstants.raw)){
				JSONObject esSingleHitSourceMetaRaw = esSingleHitSourceMeta.getJSONObject(SearchConstants.raw);
				
				if(esSingleHitSourceMetaRaw.has(SearchConstants.uploadDateValue)){
					usptoResponse.append(SearchConstants.uploadDate,
						esSingleHitSourceMetaRaw.get(SearchConstants.uploadDateValue));
				}
				// Check whether Copyright Year Exists
				if (esSingleHitSourceMetaRaw.has(SearchConstants.copyrightYear)) {
					usptoResponse.append(SearchConstants.copyright,
							esSingleHitSourceMetaRaw.get(SearchConstants.copyrightYear));
				}
			}

[shikham01]

Just want to make sure its an issue with date fields

[reefdog]

@shikham01: You should be able to configure your local query-parser-docker to interact with our deployment endpoint and debug it directly that way, right? (That’s how we discovered the issues in the first place; configuring our own query-parser-docker and hitting the new endpoint.) You shouldn’t actually need to deploy the query-parser anywhere in order to iterate tests on this. Let me know if there’s some miscommunication, or if I do need to do something else on our end to help you debug!