Failing to scan repository with FossID: Cannot deserialize value of type `java.util.LinkedHashMap<java.lang.String,java.lang.String>`

[realbigflo]

When trying to create a scan with FossID (workbench version: 2023.3.0) ORT scan step fails.
I tried different internal repositories all failing with the same pattern. Is there anything I can provide as debug more than the traceback? Is there any option to increase the debug output even further to get the details of the API calls that are made towards FossID (assuming, that there is a mismatch between what ORT gets and what is expected)?

16:31:44.135 [main] INFO  org.ossreviewtoolkit.plugins.scanners.fossid.FossId - Default branch is 'develop'.
16:31:44.135 [main] WARN  org.ossreviewtoolkit.plugins.scanners.fossid.FossId - No project revision has been given.
16:31:44.136 [main] INFO  org.ossreviewtoolkit.plugins.scanners.fossid.FossIdUrlProvider - URL mapping applied to <https git url>: Mapped to <ssh git url>.
16:31:44.137 [main] WARN  java.util.List - No recent scan found for project revision null. Falling back to default branch scans.
16:31:44.137 [main] WARN  java.util.List - No recent default branch scan found. Falling back to old behavior.
16:31:44.137 [main] INFO  org.ossreviewtoolkit.plugins.scanners.fossid.FossId - Found 0 scans.
16:31:44.137 [main] INFO  org.ossreviewtoolkit.plugins.scanners.fossid.FossId - No scan found for <ssh git url> and revision <revision>. Creating origin scan...
16:31:44.137 [main] INFO  org.ossreviewtoolkit.plugins.scanners.fossid.FossIdNamingProvider - Parameterizing the name with pattern '$Var1_$Var3'.
16:31:44.138 [main] INFO  org.ossreviewtoolkit.plugins.scanners.fossid.FossId - Creating scan 'jenkins_library_test'...
Exception in thread "main" com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.util.LinkedHashMap<java.lang.String,java.lang.String>` from Array value (token `JsonToken.START_ARRAY`)
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 49] (through reference chain: org.ossreviewtoolkit.clients.fossid.EntityResponseBody["data"])
        at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59)
        at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1767)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1541)
        at com.fasterxml.jackson.databind.deser.std.StdDeserializer._deserializeFromArray(StdDeserializer.java:222)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:457)
        at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:545)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:570)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:440)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1493)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:348)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:185)
        at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
        at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2125)
        at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1501)
        at retrofit2.converter.jackson.JacksonResponseBodyConverter.convert(JacksonResponseBodyConverter.java:33)
        at retrofit2.converter.jackson.JacksonResponseBodyConverter.convert(JacksonResponseBodyConverter.java:23)
        at retrofit2.OkHttpCall.parseResponse(OkHttpCall.java:246)
        at retrofit2.OkHttpCall$1.onResponse(OkHttpCall.java:156)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

[sschuberth]

@nnobelis could you please look into this?

[sschuberth]

@alexandruz any insights what might have changed in FOSSID'S data model?

[nnobelis]

@realbigflo Unfortunately, we cannot log the requests to FossID as they contain credentials.

We are also running this version and we don't get the the error.
Looking at your stack trace it seems FossID returns now for service.createScan an array and not a map anymore, but ORT should be able to handle this case.

If you want, reach me internally and we can look at this problem together.

[realbigflo]

While debugging with @nnobelis we figured out, the problem is caused by an access problem of Fossid for the specific git repository. I thought initially, that the ssh key was added correctly to the bitbucket instance but unfortunately this wasn't the case.

This is the request that should be send to the FossID instance:

curl --header "Content-Type: application/json" --request POST --data "{\"action\":\"create\",\"group\":\"scans\",\"data\":{\"username\":\"username\",\"key\":\"mykey\",\"project_code\":\"test_project\",\"scan_code\":\"test_ssh_cred_fail\",\"scan_name\":\"test_ssh_cred_fail\",\"git_repo_url\":\"ssh git repo url\",\"git_branch\":\"develop\"}}" <fossid REST endpoint>

{"operation":"scans_create",
 "status":"0",
 "data":[{"code": "RequestData.Base.issue_with_executing_command",
              "message":"Field git_repo_url: there was an issue executing command: timeout 200 git ls-remote 'ssh git repo' 2>&1. Exit status: 128. Output: Repository not found The requested repository does not exist, or you do not have permission to access it. fatal: Could not read from remote repository.  Please make sure you have the correct access rights and the repository exists.",
              "message_parameters":{"fieldname":"git_repo_url",
                                                    "cmd":"timeout 200 git ls-remote 'ssh git repo' 2>&1",
                                                    "exitStatus":128,
                                                    "out":"Repository not found The requested repository does not exist, or you do not have permission to access it. fatal: Could not read from remote repository.  Please make sure you have the correct access rights and the repository exists."}
              }],
"error":"RequestData.Base.issues_while_parsing_request",
"message":"These issues were found while parsing the request:",
"message_parameters":[]
}

After correcting the authentication issue, the result of the request looks like:

{"operation":"scans_create",
 "status":"1",
 "data":{"scan_id":23732},
 "message":"Scan test_ssh_cred_fail has been created."
  }

The data field in the failure case is an array with a single element (map) while in the good case, it is only a map.

[nnobelis]

Thanks a lot @realbigflo : Yeah, it seems they changed the syntax of the error response, with even more polymorphism :(
I will have a look.

[nnobelis]

@sschuberth Could you maybe assign the issue to me ?

[nnobelis]

@sschuberth I started to work on it here: #8541

Unfortunately, the exception is still present: the polymorphic deserializer currently allows to process multiple responses for the same function: T, [T], { "xxx", T }.

Unfortunately, in this case, the responses are either data: { "scan_id": N } or something totally different e.g.

"data":[{"code": "RequestData.Base.issue_with_executing_command",
              "message":"Field git_repo_url: there was an issue executing command: timeout 200 git ls-remote 'ssh git repo' 2>&1. Exit status: 128. Output: Repository not found The requested repository does not exist, or you do not have permission to access it. fatal: Could not read from remote repository.  Please make sure you have the correct access rights and the repository exists.",
              "message_parameters":{"fieldname":"git_repo_url",
                                                    "cmd":"timeout 200 git ls-remote 'ssh git repo' 2>&1",
                                                    "exitStatus":128,
                                                    "out":"Repository not found The requested repository does not exist, or you do not have permission to access it. fatal: Could not read from remote repository.  Please make sure you have the correct access rights and the repository exists."}
              }],

I still need to work this out.

[nnobelis]

As I don't see how to fix this without an ugly hack, I asked our OPS to create a ticket at FossID support.

[sschuberth]

As I don't see how to fix this without an ugly hack, I asked our OPS to create a ticket at FossID support.

Any update here @nnobelis?

[nnobelis]

Unfortunately no progress at all. I guess we'll have to wait for the promised OpenApi-conform new API.

In the meantime, I will ask our ops regarding the support ticket.