Skip to content

Outputs.share() under-allocates the data wire whenever check_type() promotes the dtype #484

Description

@prawnsgupta

Outputs.share() decides how much shared memory the data wire needs before it decides what dtype it is going to store there, and the two do not agree.

What happens

check_type() deliberately widens the dtype — int32/float32 are promoted to 64-bit, and any Unicode/bytes dtype is promoted to at least 64 characters:

https://github.com/JdeRobot/VisualCircuit/blob/master/backend/staticfiles/synthesis/lib/outputs.py#L21-L36

But the wire is allocated from the original, un-promoted nbytes:

data_size = data.nbytes if data.nbytes > 256 else 256
data_wire = self._create_wire(self.outputs[name]["wire"], data_size)
...
self.outputs[name]["data"] = create_ndbuffer(shape, type[0], data_wire.buf)   # promoted dtype

So whenever promotion actually changes the itemsize and the promoted size exceeds both data.nbytes and the 256-byte floor, the view is larger than the buffer and numpy refuses it.

Reproduction

o = Outputs({"Out": {"wire": "demo_b", "lock": multiprocessing.Lock()}})
o.share("Out", ["cat", "dog", "bird"])

I'm on Windows, where #483 fires first, so to isolate this bug I patched only the shape/dim dtype line out and ran a few payloads through the shipped code:

payload                dtype  -> promoted  allocated  needed   result
list of strings        <U4    -> <U64          256      768    TypeError: buffer is too small for requested array
100 float32 samples    float32 -> <f8          400      800    TypeError: buffer is too small for requested array
100 int32 samples      int32  -> <i8           400      800    TypeError: buffer is too small for requested array
100 float64 samples    float64 -> <f8          800      800    OK

Only the case where no promotion happens survives. On Linux the shipped file reaches this point directly, so the failure is the same there.

The practical effect is that outputs.share() works for float64/int64 payloads and fails for anything else once it crosses 256 bytes — e.g. a block emitting a list of class labels, or one sharing a float32 array straight out of a model.

Related, same block of code: scalar payloads

A scalar goes down a different path but breaks on the same line group. np.array("hello").shape is (), so shape.nbytes is 0 and the shape wire is created with size 0:

  File "lib/outputs.py", line 64, in share
    shape_wire = self._create_wire(wire_name + "_shape", shape.nbytes)
  File "lib/outputs.py", line 17, in _create_wire
    shm = shared_memory.SharedMemory(name=name, create=True, size=size)
ValueError: 'size' must be a positive number different from zero

outputs.share("Out", 5) and outputs.share("Out", "hello") both hit this. Mentioning it here because it lives in the same four lines and would naturally be fixed in the same patch — happy to split it out if you'd rather track it separately.

Suggested fix

Compute the size from the promoted dtype rather than the input dtype, roughly:

final_type = self.check_type(data.dtype.str)
needed = np.dtype(final_type).itemsize * max(data.size, 1)
data_size = max(needed, 256)

Happy to open a PR with this plus a small test that covers the string / float32 / int32 cases, if you'd like me to take it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions