Are token features and node/type identifiers added or concatenated?

[hmfu]

Thanks for providing the great implementation codes!

After looking into the codes, I have a quick question about the formation of input features to the TokenGT model. Specifically, if I understood the paper correctly, the node features, token identifiers, and token type identifiers are concatenated (C + 2 * d_p + d_e dimensions according to Section 2 - Main Transformer in the paper), while in the code here, they seem to be added together rather than concatenated. Am I misunderstanding the paper or the codes? Or are these two approaches actually equivalent or achieving similar performances?

Thank you for any help on this!

[ryoji-kubo]

Hi, we were also wondering about the same issue and would appreciate it if the authors could provide insight into how the proof will still hold if addition is used instead of concatenation. Thanks!

[xribene]

I'm also interested in any insights. Thanks!

[Jostarndt]

I am also interested in any further information. Thanks! :)

[jw9730]

Hi all, the two can be understood as different ways of writing (expressing) the same thing.

Let x and y be column vectors, and A and B be matrices with same numbers of rows. Then:

Concatenation: z = [A, B][x; y]
Addition: z = Ax + By

One thing to keep in mind is to add x and y after linear projections by A and B.

[michailmelonas]

@jw9730 I see the projection is not being applied to the type identifiers as it is specified in the paper. Is this because applying such a projection directly to the output of the embedding would be obsolete?