<div dir="ltr">Hi Helen<div><br></div><div>FYI, if I m not wrong at this time of the day, VEP run (--tab output) stops<span style="background-color:rgb(255,255,255)"> </span>(w/o any msg or warning file created) as soon as a line with REF==ALT is reached, so it does not seem to need to 'accumulate' a number of these 'non-variant' entries </div><div><div><br></div><div>on the other hand, as you say, when using vcf as output with the --allow_non_variant flag, entries as e.g.</div><div><br></div><div>12 111352091 16500 C C . PASS<br></div><div><br></div><div>are included in the output w/o any annotation, </div><div><br></div><div>thanks!</div><div>and have a nice weekedn</div><div>d</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">El vie., 7 jun. 2019 a las 18:25, Helen Schuilenburg (<<a href="mailto:helens@ebi.ac.uk">helens@ebi.ac.uk</a>>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi David</p>
<p>Thanks for the information.</p>
<p>The VEP should skip non-variant lines of input by default and not
stop.</p>
<p>We will look at updating the stats to report the lines skiped.</p>
<p>When using VCF format as input and output, by default VEP will
skip non-variant lines of input. Please could you try running
your sample variants with vcf output (--vcf) and
--allow_non_variant.</p>
<p><a class="gmail-m_-8637669723232668235moz-txt-link-freetext" href="https://www.ensembl.org/info/docs/tools/vep/script/vep_options.html#filt" target="_blank">https://www.ensembl.org/info/docs/tools/vep/script/vep_options.html#filt</a><br>
</p>
<p>With the text output, it should also skip the non-variant lines.
VEP could be stopping on your input file, if it has skipped a
number of non-variant lines. We will look into this</p>
<p>Regards<br>
Helen <br>
</p>
<p><br>
</p>
<div class="gmail-m_-8637669723232668235moz-cite-prefix">On 07/06/2019 13:02, David Tamborero
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Hi Helen, wow, thanks for the quick reply, you re
awesome guys !
<div><br>
</div>
<div>there was no warnings file created, that s why I was
assuming that the VEP run was ok and it just stopped at some
point 'naturally'.<br>
</div>
<div><br>
</div>
<div>But after your reply, I took the lines 16500-17000 (to
catch the supposed to be wrong one) and inputted to VEP; </div>
<div><br>
</div>
<div>-> In the ouptut tab file, there are 465 variants</div>
<div><br>
</div>
<div>-- > the stats file show that 465 (not 500) variants
are read; and 464 processed; and 0 filtered out. </div>
<div><br>
</div>
<div><font size="1">Lines of input read 465<br>
Variants processed 464<br>
Variants filtered out 0</font><br>
</div>
</div>
<div><br>
</div>
<div>-> the ids of the variants that are in the input but not
ias you pointed outn the output (note that the id is the
variant number), are from 16965 to 17000. <span style="background-color:rgb(255,242,204)">This would be
compatible with VEP finding a line that does not like and
stop, </span><span style="background-color:rgb(255,255,255)">as you pointed out</span><br>
</div>
<div><br>
</div>
<div>-> the first 'not mapped' variant is ( GRCh37) </div>
<div>2 167099158 16965 A A </div>
<div>which o course it makes no sense to consider it as a
variant, but I do not know i<span style="background-color:rgb(255,242,204)">f it s such a
weird thing to make everything stop</span> (indeed in the
web VEP this is not mapped neither)<br>
</div>
<div><br>
</div>
<div>--> And<span style="background-color:rgb(255,242,204)"> there
is no warning file<b> </b></span><br>
</div>
<div><span style="background-color:rgb(255,255,255)"><br>
</span></div>
<div><span style="background-color:rgb(255,255,255)">--> I
removed this 'conflictive' entry and VEP consequently run
the 499 variants</span></div>
<div><span style="background-color:rgb(255,255,255)"><br>
</span></div>
<div><span style="background-color:rgb(255,255,255)">So I think
this is a bug, since --as far as i know-- the normal
behaviour when VEP encounters a line that does not like is
to ignore it and generate the corresponding warning file,
right ?</span></div>
<div><span style="background-color:rgb(255,242,204)"><b><br>
</b></span></div>
<div><br>
</div>
<div><span style="background-color:rgb(255,242,204)"><b><br>
</b></span></div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">El vie., 7 jun. 2019 a las
12:39, Helen Schuilenburg (<<a href="mailto:helens@ebi.ac.uk" target="_blank">helens@ebi.ac.uk</a>>)
escribió:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi David</p>
<p>VEP can run with 100K variants. The VEP publication
(McLaren W et al. 2016, doi:10.1186/s13059-016-0974-4)
gives examples of<br>
number of variants and timings e.g for 4,474,140
variants.<br>
<br>
Were any warnings reported in your warnings_file (
--warning_file
/home/david/Desktop/tmp/tmp/vep_warnings.txt)?</p>
<p>From the stats file, there may be a problem with your
input file causing to to stop processing at Line 16856
of your input file.<br>
<br>
Regards<br>
Helen<br>
</p>
<div class="gmail-m_-8637669723232668235gmail-m_2995700720703143231moz-cite-prefix">On
07/06/2019 10:17, David Tamborero wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi all,
<div><br>
</div>
<div>I m working with VEP (command line) v95 to
analyse (large) vcf files and generate VEP -tab
files</div>
<div><br>
</div>
<div>I ve realized that the output does not contain
all the entries that I m feeding to the tool. I ve
been doing several tests for vcf files with 1K, 5K,
10K, 20K, 50K, 100K entries and it looks to me that
VEP reads 'a maximum' of 16K8 variants (see VEP
stats output below)</div>
<div><br>
</div>
<div>I was not aware that it existed such a limit, nor
I am capable of finding any flag regarding this
issue. I feel that I m missing a big point, but I do
not know which one --sorry in advance if so.</div>
<div><br>
</div>
<div>command:</div>
<div>./vep -i blablabla.vcf -o blbabla.txt --tab
--warning_file blabla.txt --format vcf --cache
--offline --force_overwrite --hgvs --symbol
--stats_file blablabla.txt --stats_text </div>
<div><br>
</div>
<div>version:</div>
<div>release 95<br>
sub 4f834538054c1aee24098c72f31f92d4c5aa303b<br>
</div>
<div><br>
</div>
<div>and the stats file content for the run with a vcf
file with 100K entries :</div>
<div> [<font size="1">VEP run statistics]<br>
VEP version (API) 95 (95)<br>
Annotation sources Cache:
/home/david/.vep/homo_sapiens/95_GRCh37<br>
Species homo_sapiens<br>
Command line options --cache --dir
/home/david/.vep --force_overwrite --format vcf
--hgvs --input_file
/home/david/Desktop/tmp/tmp/hq.vcf --offline
--output_file /home/david/Desktop/tmp/tmp/vep.txt
--stats_file
/home/david/Desktop/tmp/tmp/stats_file.txt
--stats_text --symbol --tab --warning_file
/home/david/Desktop/tmp/tmp/vep_warnings.txt<br>
Start time 2019-06-07 10:50:24<br>
End time 2019-06-07 10:59:46<br>
Run time 562 seconds<br>
Input file /home/david/Desktop/tmp/tmp/hq.vcf<br>
Output file /home/david/Desktop/tmp/tmp/vep.txt<br>
<br>
[General statistics]<br>
<span style="background-color:rgb(255,217,102)">Lines
of input read 16856</span><br>
Variants processed 16855<br>
Variants filtered out 0<br>
Novel / existing variants -<br>
Overlapped genes 2809<br>
Overlapped transcripts 15672<br>
Overlapped regulatory features -</font></div>
</div>
<br>
<fieldset class="gmail-m_-8637669723232668235gmail-m_2995700720703143231mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-8637669723232668235gmail-m_2995700720703143231moz-quote-pre">_______________________________________________
Dev mailing list <a class="gmail-m_-8637669723232668235gmail-m_2995700720703143231moz-txt-link-abbreviated" href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a>
Posting guidelines and subscribe/unsubscribe info: <a class="gmail-m_-8637669723232668235gmail-m_2995700720703143231moz-txt-link-freetext" href="https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org" target="_blank">https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org</a>
Ensembl Blog: <a class="gmail-m_-8637669723232668235gmail-m_2995700720703143231moz-txt-link-freetext" href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a>
</pre>
</blockquote>
</div>
_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org" rel="noreferrer" target="_blank">https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" rel="noreferrer" target="_blank">http://www.ensembl.info/</a>-<br>
</blockquote>
</div>
</div>
<br>
<fieldset class="gmail-m_-8637669723232668235mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-8637669723232668235moz-quote-pre">_______________________________________________
Dev mailing list <a class="gmail-m_-8637669723232668235moz-txt-link-abbreviated" href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a>
Posting guidelines and subscribe/unsubscribe info: <a class="gmail-m_-8637669723232668235moz-txt-link-freetext" href="https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org" target="_blank">https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org</a>
Ensembl Blog: <a class="gmail-m_-8637669723232668235moz-txt-link-freetext" href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a>
</pre>
</blockquote>
<pre class="gmail-m_-8637669723232668235moz-signature" cols="72"></pre>
</div>
_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org" rel="noreferrer" target="_blank">https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" rel="noreferrer" target="_blank">http://www.ensembl.info/</a><br>
</blockquote></div>