Any way to speed up read_gtf()?

[gernophil]

Hey, thanks for this awesome package. It's really, really helpful. I just realized that it takes quite some time to read a GTF using read_gtf(). For the Ensembl GTF for the GRCh38 it takes around 35 seconds. Is there a way to speed this up? The gencodegenes.Gencode() function for example takes around 5 seconds. Of course, the created object is less complex, but that's still quite a difference. Even with full=False, it's not much faster and I need the column gene_name. Is there any way to optimize this?

[endrebak]

Thanks for the request.

What is gencodegenes.Gencode? Does it read any of the attribute (i.e. ;-delimited) columns?

[gernophil]

You can find gencodegenes here. Not sure, which it all reads, but it reads the HGNC gene symbol, which is in the ;-delimited column, isn't it?

[endrebak]

Nice. It is written in C++ so it might be less flexible than pyranges. GTF is not well-defined so some hacks are needed to parse every file found in the wild. That makes our code slower.

I haven't tried the library though, it might handle everything flawlessly.

[endrebak]

The best way to speed it up is to store the dataframe as a tsv and read it as a tsv the next time.

[gernophil]

That's actually a nice idea. That would at least speed it up, after the first run. Btw, I get this warning with read_gtf():

FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning.
  return {k: v for k, v in df.groupby(grpby_key)}

Not sure, if this is of impact or can just be ignored.

[endrebak]

We should fix it eventually.