-
Notifications
You must be signed in to change notification settings - Fork 66
Open
Labels
documentationImprovements or additions to documentationImprovements or additions to documentation
Description
Currently this snippet:
from nnsight import LanguageModel
model = LanguageModel("gpt2")
toks = model.tokenizer("Hello, world!", return_tensors="pt")
input_ids = toks["input_ids"]
attention_mask = toks["attention_mask"]
with model.trace(
input_ids=input_ids, attention_mask=attention_mask
):
x = model.transformer.h[0].outputFails with this very uninformative message:
---------------------------------------------------------------------------
NNsightException Traceback (most recent call last)
Cell In[6], line 5
2 from nnsight import LanguageModel
4 model = LanguageModel("gpt2")
----> 5 with model.trace(
6 input_ids=input_ids, attention_mask=attention_mask
7 ):
8 x = model.transformer.h[0].output
File ~/projects/diffing-toolkit/.venv/lib/python3.12/site-packages/nnsight/intervention/tracing/base.py:665, in Tracer.__exit__(self, exc_type, exc_val, exc_tb)
662 if self.asynchronous:
663 return self.backend(self)
--> 665 self.backend(self)
667 return True
File ~/projects/diffing-toolkit/.venv/lib/python3.12/site-packages/nnsight/intervention/backends/execution.py:24, in ExecutionBackend.__call__(self, tracer)
21 return tracer.execute(fn)
22 except Exception as e:
---> 24 raise wrap_exception(e, tracer.info) from None
25 finally:
26 Globals.exit()
NNsightException:
Traceback (most recent call last):
File "<ipython-input-6-77795bbc4ead>", line 8, in <module>
x = model.transformer.h[0].output
ValueError: Cannot return output of Envoy that is not interleaving nor has a fake output set.I guess this is a LanguageModel specific issue and not a general NNsight issue, but this can be confusing for people used to the hf interfact where this works:
from transformers import AutoModelForCausalLM
hf_model = AutoModelForCausalLM.from_pretrained("gpt2")
hf_model(input_ids=input_ids, attention_mask=attention_mask)if there are some limitations that do not allow to support this synthax a nicer error message would be nice
Metadata
Metadata
Assignees
Labels
documentationImprovements or additions to documentationImprovements or additions to documentation