JBrowse Update #141

hexylena · 2015-05-13T22:29:51Z

New Features

Preview:

Questions:

is it OK to run bgzip/tabix on vcf data, or is there another way I should be accessing that? (Yes)
should I expose colouring options? Isn't there a new UI element for colours? (Not yet)
should I re-add wig support? I removed it in lieu of bigWig only (No)
should I support pre-set min/max values for BigWig data (normally it auto-scales)? (Yes)

bgruening · 2015-05-14T12:08:08Z

Questions:

is it OK to run bgzip/tabix on vcf data, or is there another way I should be accessing that?

Galaxy has an integrated converter for it: https://github.com/galaxyproject/galaxy/blob/dev/config/datatypes_conf.xml.sample#L204 I also hope to make use of this for visualizations. Not sure about this though.

should I expose colouring options? Isn't there a new UI element for colours?

Yes, @guerler is working on this. Imho it is not yet in dev.

should I re-add wig support? I removed it in lieu of bigWig only

I would got with bigWig support only. Galaxy has converters if someone really cares.

should I support pre-set min/max values for BigWig data (normally it auto-scales)?

I guess this is a important options, but than again it depends on your use cases.

bgruening · 2015-05-14T12:08:39Z

tools/jbrowse/jbrowse.py

+    if len(json_data.keys()) == 0:
+        return
+
+    print json_data


debug output?

yep, will remove before it's out of WIP

hexylena · 2015-05-14T13:58:49Z

@bgruening

Galaxy has an integrated converter for it

okay, then we just need to make sure the indices are attached like for bam files. Added to TODO

I would got with bigWig support only. Galaxy has converters if someone really cares.

Great, sounds good. I want it to be easy, and as long as Galaxy will list wig files that can be auto-converted for use...

should I support pre-set min/max values for BigWig data (normally it auto-scales)?

I guess this is a important options, but than again it depends on your use cases.

Just want a general utility JBrowse tool which exposes a lot of the tougher to deal with JBrowse configuration in a user-friendly way. Trying to find a happy place where it isn't an overwhelming amount of configuration. I'll probably add in the min/max then.... I really want people to use this and tell me what needs to be fixed for their datasets...

hexylena · 2015-05-20T23:56:33Z

BlastXML to gapped GFF3 will be part of this PR soon.

@peterjc you have any thoughts/comments/opinions on the following graphics?

peterjc · 2015-05-21T10:01:00Z

Pretty graphics :)

See also peterjc/galaxy_blast#61 plus there have been some recent discussion about returning BLAST XML etc from the Tool Shed to the Galaxy core. If that happens then BLAST XML to gapped GFF3 could also go in the core...

hexylena · 2015-05-21T14:10:06Z

Thanks @peterjc, thought I'd ping you since you're the blast person...

Definitely, I'd love to see all the blast stuff in core, I'll be happy to PR that PR with a datatype converter.

hexylena · 2015-05-21T18:11:34Z

Now with colouring for e-value.
~~Getting pretty close to a full replacement for, with more features than, trackster ...~~

hexylena · 2015-05-22T21:51:30Z

I don't expect anyone to review this before next week, but if anyone feels like it, this is out of WIP and ready to be merged. :)

(Additionally I recognise that pet-projects have a different relationship with the IUC repo than normal packaging efforts.)

bgruening · 2015-05-23T15:44:53Z

tools/jbrowse/jbrowse.py

+        gff3_unrebased.close()
+
+        gff3_rebased = tempfile.NamedTemporaryFile(delete=False)
+        cmd = ['python', os.path.join(INSTALLED_TO, 'gff3_rebase.py')]


Both files are located in the same directory, why not importing the main function and using it without tempfile and subprocess?

the thinking with tempfile was that it will be better behaved than reading/storing the entire data structure in memory. However, the parser isn't written in a very "streaming" fashion right now, so huge amounts of data will still be stored in memory.

You're absolutely right though, I'll switch over to importing :)

Do you want to switch over?

Shoot, hadn't tested. Will test now.

What happened to this one? I'm fine with it being called as subprocess, just wanted to be sure.

Let's leave it as is for this round, since I know it is already working, and the testing for jbrowse isn't easy yet.

I'll be PRing another jbrowse update soon to do the following:

switch over to importing (maybe?)

add test cases

remove the sample_data folder from use in the package

add more configurability for track plotting

figure out whatever bug makes viewing bam/bigwig data nearly impossible inside of galaxy. outside of galaxy everything works fine.

bgruening · 2015-05-23T16:03:37Z

@erasche;

you need to add BCBio & biopython, samtools (aka tabix, bgzip), perl? as dependency
I would put blastxml_to_gapped_gff3 into a separate repository

See my other small comments inline.

hexylena · 2015-05-24T15:35:37Z

@bgruening

bcbio/biopython should already be there.
samtools and a bunch of perl libs will have to be added, thanks.
I would like to separate out blastxml_to_gapped_gff3 and gff3_rebase since those are useful standalone tools, but I didn't know where to put them. xml2gff3 makes sense as a datatype converter if blast datatypes move into galaxy, but it has user-editable options and I don't remember datatype conversion supporting that. gff3_rebase will be useful for our annotation pipelines, but the IUC repo is mostly tool suites rather than individual tools. Should I open a PR with those in separate folders?
moved back to WIP until I can fix your suggestions.

bgruening · 2015-05-24T17:34:34Z

@erasche I don't see the IUC repo only for tool suites. Feel free to add those converters as standalone tools.

hexylena · 2015-05-24T17:36:02Z

@bgruening okay! Will do!

hexylena · 2015-05-25T17:02:09Z

@bgruening do you have an examples of having a tool repo be a dependency? e.g.

<?xml version="1.0"?>
<tool_dependency>
  <package name="jbrowse" version="1.11.6">
    <repository name="package_jbrowse_1_11_6" owner="iuc"/>
  </package>
  <tool name="blastxml_to_gff3" version="1.0">
    <repository name="blastxml_to_gff3" owner="iuc"/>
  </tool>

? How does one write the XML for that, I'm not seeing any examples.

Or do I need to:

publish the python code somewhere
make that into a package_
separate out the XML files in to tools?

That seems awfully complex.

Or should I just copy+paste the scripts into this directory and keep them in a separate repo as well?

bgruening · 2015-06-10T19:24:38Z

tools/blastxml_to_gapped_gff3/blastxml_to_gapped_gff3.py

+                # may be longer than the parent feature, so we use the supplied
+                # subject/hit length to calculate the real ending of the target
+                # protein.
+                #print hsp.align_length, hit.length


yep, that was from testing/debugging the math. Removed.

bgruening · 2015-06-10T19:58:16Z

@erasche please also see my old comments.
What about symlinking your files as much as you can to avoid duplication?
Overall, impressive work!

hexylena · 2015-06-10T20:06:32Z

@bgruening thanks for the amazing feedback, I really appreciate it. I'll get fixes implemented shortly.

hexylena · 2015-06-11T00:11:29Z

@bgruening okay, should be closer to ready. Used symlinks everywhere possible.

bgruening · 2015-06-11T05:28:48Z

@erasche you can also symlink on git level to avoid code duplication. Thanks for the fixes.

hexylena · 2015-06-11T06:40:22Z

Okay, travis is failing now due to the bundle. I'll have to correct that first thing tomorrow along with git symlinks.

hexylena · 2015-06-11T15:49:51Z

@bgruening should be good to go, if travis tests pass.

JBrowse Update

hexylena self-assigned this May 13, 2015

bgruening reviewed May 14, 2015
View reviewed changes

hexylena added the enhancement label May 22, 2015

hexylena changed the title ~~[WIP] JBrowse Update~~ JBrowse Update May 22, 2015

hexylena removed their assignment May 22, 2015

hexylena added the reviewer needed label May 22, 2015

bgruening reviewed May 23, 2015
View reviewed changes

hexylena changed the title ~~JBrowse Update~~ [WIP] JBrowse Update May 24, 2015

Eric Rasche added 3 commits May 25, 2015 12:09

Make parameters more generic

9674e5a

reorganise code layout, support bigwig/bam

890cdbb

Add test data

d46fddb

bgruening reviewed Jun 10, 2015
View reviewed changes

Eric Rasche added 8 commits June 10, 2015 18:50

Use symlinks instead of copying datasets

6b6838b

More concise min/max routine

5dbbb62

Remove duplicate packages

84bf22d

Move to download_store

bedaeb4

Fix english

ace42ff

Slightly less error prone code

f5a7b25

Better output name

ebe33fa

Whitespace fix

edc6e40

nsoranzo removed the reviewer needed label Jun 11, 2015

nsoranzo assigned bgruening Jun 11, 2015

Eric Rasche added 2 commits June 11, 2015 10:48

Corret XML error

0587868

Replace scripts with symlinks

26e6b5a

bgruening added a commit that referenced this pull request Jun 11, 2015

Merge pull request #141 from galaxyproject/jbrowse_02

fd63087

JBrowse Update

bgruening merged commit fd63087 into master Jun 11, 2015

bgruening deleted the jbrowse_02 branch June 11, 2015 17:55

hexylena mentioned this pull request Apr 16, 2017

More useful output name than "default" hexylena/argparse2tool#23

Open

JBrowse Update #141

JBrowse Update #141

Uh oh!

Conversation

hexylena commented May 13, 2015

New Features

Preview:

Questions:

Uh oh!

bgruening commented May 14, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hexylena commented May 14, 2015

Uh oh!

hexylena commented May 20, 2015

Uh oh!

peterjc commented May 21, 2015

Uh oh!

hexylena commented May 21, 2015

Uh oh!

hexylena commented May 21, 2015

Uh oh!

hexylena commented May 22, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bgruening commented May 23, 2015

Uh oh!

hexylena commented May 24, 2015

Uh oh!

bgruening commented May 24, 2015

Uh oh!

hexylena commented May 24, 2015

Uh oh!

hexylena commented May 25, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bgruening commented Jun 10, 2015

Uh oh!

hexylena commented Jun 10, 2015

Uh oh!

hexylena commented Jun 11, 2015

Uh oh!

bgruening commented Jun 11, 2015

Uh oh!

hexylena commented Jun 11, 2015

Uh oh!

hexylena commented Jun 11, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants