A bug with CrawlSpider with multiple rules

[Red-Pheonix]

When there is a crawlspider with multiple rules, if one of them has no methods attached as a callback, then it gets "TypeError: join() argument must be str, bytes, or os.PathLike object, not 'NoneType'". It tries to make a directory based on the name, but fails to do so. Adding a check prevents this issue.

I found another issue as well. Although the fix works(as in the spider doesn't crash), the generated tests don't work. It tries to call the non existent method and gets an error TypeError: NoneType object is not callable

[danilomr12]

Found the same here. Just to add a more detailed explanation.
Given a CrawlSpider like this:

class MyCrawlSpiderX(CrawlSpider):

    name = "mycrawlspiderx"
    allowed_domains = ['testdomain.com']
    start_urls = ['https://www.testdomain.com/products/category-x']

    rules = (
        Rule(LinkExtractor(allow=[r'(.+)[-]?test-product-page/(.+)/(\d+)$']), 'parse_item'),
        Rule(LinkExtractor(allow=[r'/(.+)-products/category-x\?(from|adState|pageSize)=(\d+|.*)&(from|adState|pageSize)=(\d+|.*)$']), ),
        Rule(DaftPaginationExtractor(allow=[r'/(.+)-products-for-sale/country\?(from|adState|pageSize)=(\d+|.*)&(from|adState|pageSize)=(\d+|.*)$']),)
    )
   def parse_item(self, response):

The two rules without callback function parse_item will break the cassette recorder when trying to get the test_dir.
_get_test_dir function appends a blank callback_name to components here:

scrapy-autounit/scrapy_autounit/recorder.py

Line 65 in 0858bee

components.append(callback_name)

And later in the loop it tries to join a blank comp item from components, throwing the error

test_dir = os.path.join(test_dir, comp) if test_dir else comp

Basically, AutoUnit is trying to store/cache the responses from URLs that should be put away.