Skip to content

Out-of-bound access in SetConsensus in the case of degenerate symbols. #341

@althonos

Description

@althonos

The p7_hmm_SetConsensus can be used to set the consensus for a P7_HMM object, either by reading from the HMM matrix directly, or from a sequence given as argument.

That function executes the following loop:

  for (k = 1; k <= hmm->M; k++) 
    {
      x = (sq ?  sq->dsq[k] : esl_vec_FArgMax(hmm->mat[k], hmm->abc->K));
      hmm->consensus[k] = ((hmm->mat[k][x] >= mthresh) ? toupper(hmm->abc->sym[x]) : tolower(hmm->abc->sym[x]));
    }

In there, x is a symbol from the HMM alphabet, and is used to index hmm->mat[k], meaning it is expected to be in the 0..K range. This is the case if x is obtained from the esl_vec_FArgMax call; however, if x is read directly from the sequence, it may be in the 0..Kp range due to degenerate symbols and/or special letters. In that case, the indexing of hmm->max[k] can result in out-of-bound access (and valgrind reports so). This may cause inconsistent lower/upper-case consensus to be created.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions