NoneType has no len() error.

[joshuaLW]

Describe the bug

I'm getting a cryptic error message after removing a module.: TypeError: object of type 'NoneType' has no len()

Steps To Reproduce

dbt clean
dbt run
(I'm sure the problem is something to do with my project.yml but i'm not sure what exactly is throwing the error.... will report back when i can.)

Expected behavior

An error message that will help me fix my configs/project.yml

Screenshots and log output

2021-01-15 11:19:42.774812 (MainThread): Traceback (most recent call last):
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/main.py", line 124, in main
results, succeeded = handle_and_check(args)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/main.py", line 202, in handle_and_check
task, res = run_from_args(parsed)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/main.py", line 255, in run_from_args
results = task.run()
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/task/runnable.py", line 399, in run
self._runtime_initialize()
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/task/runnable.py", line 118, in _runtime_initialize
super()._runtime_initialize()
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/task/runnable.py", line 78, in _runtime_initialize
self.load_manifest()
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/task/runnable.py", line 65, in load_manifest
self.manifest = get_full_manifest(self.config)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/perf_utils.py", line 31, in get_full_manifest
adapter.connections.set_query_header,
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/manifest.py", line 678, in load_manifest
return ManifestLoader.load_all(config, macro_manifest, macro_hook)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/manifest.py", line 348, in load_all
loader.load(macro_manifest=macro_manifest)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/manifest.py", line 204, in load
self.parse_project(project, macro_manifest, old_results)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/manifest.py", line 177, in parse_project
self.parse_with_cache(path, parser, old_results)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/manifest.py", line 133, in parse_with_cache
parser.parse_file(block)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/base.py", line 429, in parse_file
self.parse_node(file_block)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/base.py", line 402, in parse_node
self.render_update(node, config)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/base.py", line 377, in render_update
self.render_with_context(node, config)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/parser/base.py", line 289, in render_with_context
parsed_node.raw_sql, context, parsed_node, capture_macros=True
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/clients/jinja.py", line 577, in get_rendered
return render_template(template, ctx, node)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/dbt/clients/jinja.py", line 528, in render_template
return template.render(ctx)
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/Users/JoshuaLayton-Wood/venvs/dbt/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "", line 111, in top-level template code
TypeError: object of type 'NoneType' has no len()

System information

Which database are you using dbt with?

• [X ] bigquery

The output of dbt --version:

0.18.0

The operating system you're using:
MAC OS
The output of python --version:
3.7.3

Additional context

I've recently removed the snowplow module and copied some of the models into my own models folder. reversing these steps gets rid of the problem.

[jtcohen6]

Hey @joshuaLW, this is indeed a cryptic error. The question of removing the Snowplow package seems like a good place to start debugging.

• Do you have anything else in packages.yml?
• Do you still have references to the snowplow package in dbt_project.yml?

[joshuaLW]

@jtcohen6 three things... I never actually checked that the snowplow modules were removed from packages.yml when i ran "dbt clean" but now having reversed these changes with "dbt deps" I've only this in that file:
packages:

• package: fishtown-analytics/snowplow
  version: 0.11.0

I did comment out all variables and models referencing the snowplow package in the project.yml

I didn't remove all dependencies upon snowplow from my models but i expected to get an error of the kind model x refers to model y that does not exist.

I'm going to go at it that way now actually: i'm going to remove references to snowplow models in my models first and then try removing snowplow module again.

[jtcohen6]

I did comment out all variables and models referencing the snowplow package in the project.yml

If I had to pick one of the three as the likeliest cause, I'd say it's this one. Try removing snowplow from packages.yml, run dbt clean, leave the vars in place, and then see what error you get back. (I hope it will be of the missing-ref kind.)

[joshuaLW]

@jtcohen6 good shout. I've just tried emptying the packages.yml then running dbt clean and i'm getting the errors i expected eg. "model x depends on y" so i'll keep going with cleaning up my models till all dependancies are removed... and hopefully it was just that... :) FWIW I thought dbt clean would have cleaned my packages.yml too... but computers can't read my mind can they?

[joshuaLW]

@jtcohen6 So the plot thickens: the error returns even with packages.yml empty. It seems to be connected to one of the models I pinched from the package.
so to recap:

1. i installed snowplow with dbt deps
2. copied three models from the module into my models
3. cleared packages.yml of its content
4. ran dbt clean
5. dbt docs generate (any other cmd get's the same result) and i get the "object of type 'NoneType' has no len()"

[joshuaLW]

@jtcohen6 yup i found the line that was causing the problem. It was a reference to a variable that was nested in the package i'd removed.

[jtcohen6]

There it is! Well found.

I think it would be an improvement to the snowplow package if its models handled better the case when var('snowplow:pass_through_columns') is undefined in the project (None), rather than merely empty ([]).

Perhaps by rewriting that line to be:

{% if var('snowplow_pass_through_columns', []) | length > 0 %}

That way, if you do want to copy-paste the model code into your own project, and (sensibly) remove the snowplow-specific vars from your project config, you won't be stymied by a cryptic error.

I'm going to transfer this issue to the snowplow package repository, since that's where the change should take place.