Skip to content

Debug related issues - CheckMemory should consider margin when checking memory guard #495

Description

@msg7086

When _DEBUG is enabled on Windows,

int VideoFrame::CheckMemory() const {
#ifdef _DEBUG
  if (vfb->data && vfb->device->device_type == DEV_TYPE_CPU) {
    // check buffer overrun
    int *pInt = (int *)(vfb->data + vfb->data_size);
    if (pInt[0] != 0xDEADBEEF ||
      pInt[1] != 0xDEADBEEF ||
      pInt[2] != 0xDEADBEEF ||
      pInt[3] != 0xDEADBEEF)
    {
      return 1;
    }
    return 0;
  }
#endif
  return -1;
}

vfb->data + vfb->data_size didn't consider the margin passed into VFBStorage, so it almost always return 1 and fail the memory checks.


When _DEBUG is enabled on Linux,

static void DebugOut(char* s)
{
#ifdef AVS_POSIX
  LogMsg(LOGLEVEL_DEBUG, s);
#else
  _RPT0(0, s);
#endif
}

LogMsg call lost ScriptEnvironment thus it can't be compiled.
Once LogMsg is replaced with fputs to console, compilation passes, and I'm able to see the same memory checks fail.

After adding margin to VideoFrameBuffer and changing some code, memory checks now all pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions