Somewhat evil concurrency question about running parallel tasks on a computer cluster

[fechan]

Context

I'm coming up with ways for SIGILS to reduce the number of getItemDetail() and getItemLimit() calls needed when piping items from collections of slots from different inventory peripherals, which I am currently inefficiently doing by running these as parallel tasks on every slot in the collection in batches of 128 (or some other number smaller than the concurrent event limit).

As I was thinking about this, I had a somewhat evil thought:

The question

What would happen if I set up a cluster of CC computers hooked up to a network, then when a master computer needed to get the item details of a huge number of inventory slots, asked the computer cluster to retrieve these item details, to bypass the concurrent event limit for each individual computer?

Obviously this is inadvisable, but I'm curious whether the execution of this would somehow fail, or what the performance impact on the Minecraft server would be.

Also, I'm curious how the getItemDetail() and other inventory interaction methods work under the hood. Why does it take a full tick to fulfill?

[Wojbie]

I think your solution would work, tho i question need to query getItemDetails() this many times?

To decrease amount of details calls you have to make I would suggest creating cache of ItemDetails data, that used item name + nbthash of item to store Details of unique item into file (if details other then count change, nbthash would change too, so it would be different item to such cache). That way, if your system interacted with item it already seen before, it can use item name + nbthash you get from single list() call to retrieve item details from your locally stored cache, without needing to wait for CC to get this data from world on next world tick. It would mean instead of needing to getItemDetails for every item in inventory, you already know details of all items you seen before, and only need to get details of ones you didn't. Considering this data could/should be stored into file, future executions of program would not need to getItemDetails for that item again.

As for why getItemDetails() takes a tick to work, while my explanation might not be most technical here, i understand its because CC itself runs outside of MC world, in separate thread, so it only can do operations that interact/change things in world on next world tick, and after that operation is preformed it can return result if it worked.

[fatboychummy]

For caching, here's a basic implementation:

local cache = {}

-- Returns the unique name for the cached item.
local function cache_name(item)
  return item.name .. (item.nbthash and ":" .. item.nbthash or "")
end

-- Cache detailed info about a single item.
local function cache_item(inv, slot, item)
  local name = cache_name(item)
  if not cache[name] then
    local detailed_item = inv.getItemDetail(slot)
    if detailed_item.name ~= item.name or detailed_item.nbthash ~= item.nbthash then return end -- prevent collecting info about wrong item if the slot has changed.

    cache[name] = detailed_item
  end
end

-- Cache an inventory, storing detailed info about each item
local function cache_inv(inv)
  local list = inv.list()
  for slot, item in pairs(list) do
    cache_item(inv, slot, item)
  end
end

-- cache all connected inventories
local function cache_invs()
  local invs = table.pack(peripheral.find("inventory"))
  
  for _, inv in ipairs(invs) do
    cache_inv(inv)
  end
end

Note that this isn't parallelized. Parallelizing will make it much faster too. The first cache iteration will usually take the longest, but after that it will be much quicker, as only items that are new to the system need to be cached.

Then, when you need detailed info about an item...

local function get_details(item)
  return cache[cache_name(item)]
end

setmetatable(cache, {
  __index = function(self, idx)
    if type(idx) == "table" then
      return get_details(idx)
    end
    
    return rawget(self, idx) or
  end
})

Then, can simply do cache[item] to get the detailed info about that item, if it exists. If not, call cache_item and move on.

[fechan]

I didn't realize that itemId+nbthash was enough to uniquely identify the item, but in hindsight it makes sense. With this in mind, I've introduced a much more aggressive caching scheme to my program, leading to a substantial speedup.

In any case, I do find the idea of spinning up a whole cluster of ComputerCraft computers for parallel computation to be kind of funny, though perhaps impractical!