<div dir="ltr">Hi Duarte,<div style><br>Do you have some input that causes this error?</div><div style><br></div><div style>Thanks</div><div style><br></div><div style>Will</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On 14 May 2013 12:57, Duarte Molha <span dir="ltr"><<a href="mailto:duartemolha@gmail.com" target="_blank">duartemolha@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Another bug using the updated version... now using the --check_alleles and --check_existing options the script dies at line 4759<div><br></div><div><div>Use of uninitialized value in string ne at /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 4759.</div>
</div><div><br></div><div><br></div><div>Best regards</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Duarte</div></font></span></div><div class="gmail_extra"><div class="im"><br clear="all"><div><font style color="#999999">=========================<br>
Duarte Miguel Paulo Molha <br></font><div><font style color="#999999"> <a href="http://about.me/duarte" target="_blank">http://about.me/duarte</a> <br>=========================</font></div>
</div>
<br><br></div><div><div class="h5"><div class="gmail_quote">On Tue, May 14, 2013 at 11:22 AM, Will McLaren <span dir="ltr"><<a href="mailto:wm2@ebi.ac.uk" target="_blank">wm2@ebi.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Thanks - try <a href="http://cvs.sanger.ac.uk/cgi-bin/viewvc.cgi/ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils/VEP.pm?revision=1.93&root=ensembl" target="_blank">http://cvs.sanger.ac.uk/cgi-bin/viewvc.cgi/ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils/VEP.pm?revision=1.93&root=ensembl</a><span><font color="#888888"><div>
<br></div><div>Will</div></font></span></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On 14 May 2013 11:09, Duarte Molha <span dir="ltr"><<a href="mailto:duartemolha@gmail.com" target="_blank">duartemolha@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">It seems the problems are still there :<div><br></div><div>Here is my output:</div><div><br></div><div>
<div>
perl <a href="http://variant_effect_predictor.pl" target="_blank">variant_effect_predictor.pl</a> --config vep_human.ini -i INPUT.vcf --fork 16</div>
<div><br></div><div>2013-05-14 10:33:38 - Read configuration from vep_human.ini</div><div>#----------------------------------#</div><div># ENSEMBL VARIANT EFFECT PREDICTOR #</div><div>#----------------------------------#</div>
<div><br></div><div>version 71</div><div><br></div><div>By Will McLaren (<a href="mailto:wm2@ebi.ac.uk" target="_blank">wm2@ebi.ac.uk</a>)</div><div><br></div><div>Configuration options:</div><div><br></div><div>###</div>
<div>allow_non_variant 1</div>
<div>buffer_size 500000</div><div>cache 1</div><div>canonical 1</div><div>ccds 1</div><div>check_alleles 1</div><div>check_existing 1</div><div>config vep_human.ini</div>
<div>core_type core</div><div>custom /ReferenceData/vep_additional_annotations/Somatic_variation_phenotypes.bed.gz,Somatic,bed,exact </div><div> /ReferenceData/vep_additional_annotations/dbsnp135_ensembl_variation_phenotype.bed.gz,dbsnp135,bed,exact</div>
<div>dir /ReferenceData/vep_cache/</div><div>domains 1</div><div>force_overwrite 1</div><div>fork 16</div><div>gmaf 1</div><div>hgnc 1</div>
<div>host <a href="http://ensembldb.ensembl.org" target="_blank">ensembldb.ensembl.org</a></div><div>individual all</div><div>input_file INPUT.vcf</div><div>numbers 1</div>
<div>plugin Blosum62 Condel,/ReferenceData/vep_cache/Plugins/config/Condel/config,b Carol</div>
<div>polyphen b</div><div>port 5306</div><div>protein 1</div><div>regulatory 1</div><div>sift b</div><div>species homo_sapiens</div><div>stats HASH(0x35a8000)</div>
<div>terms SO</div><div>toplevel_dir /ReferenceData/vep_cache/</div><div>verbose 1</div><div>xref_refseq 1</div><div><br></div><div>--------------------</div><div><br></div><div>
Will only load v71 databases</div><div>Species 'homo_sapiens' loaded from database 'homo_sapiens_core_71_37'</div><div>Species 'homo_sapiens' loaded from database 'homo_sapiens_cdna_71_37'</div>
<div>Species 'homo_sapiens' loaded from database 'homo_sapiens_vega_71_37'</div><div>Species 'homo_sapiens' loaded from database 'homo_sapiens_otherfeatures_71_37'</div><div>Species 'homo_sapiens' loaded from database 'homo_sapiens_rnaseq_71_37'</div>
<div>homo_sapiens_variation_71_37 loaded</div><div>homo_sapiens_funcgen_71_37 loaded</div><div>Bio::EnsEMBL::Compara::DBSQL::DBAdaptor not found so the following compara databases will be ignored: ensembl_compara_71</div>
<div>ensembl_ancestral_71 loaded</div><div>ensembl_ontology_71 loaded</div><div>2013-05-14 10:33:39 - Connected to core version 71 database and variation version 71 database</div><div>2013-05-14 10:33:39 - Read existing cache info</div>
<div>2013-05-14 10:33:39 - Loaded plugin: Blosum62</div><div>2013-05-14 10:33:39 - Loaded plugin: Condel</div><div>2013-05-14 10:33:39 - Loaded plugin: Carol</div><div>2013-05-14 10:33:40 - Starting...</div><div>2013-05-14 10:33:40 - Detected format of input file as vcf</div>
<div>2013-05-14 10:33:46 - Read 195789 variants into buffer</div><div>2013-05-14 10:33:46 - Skipping 67552 non-variant loci</div><div>2013-05-14 10:33:46 - Reading transcript data from cache and/or database</div><div>[======================================================================================================] [ 100% ]</div>
<div>2013-05-14 10:40:19 - Retrieved 189344 transcripts (0 mem, 202901 cached, 0 DB, 13557 duplicates)</div><div>2013-05-14 10:40:19 - Reading regulatory data from cache and/or database</div><div>[======================================================================================================] [ 100% ]</div>
<div>2013-05-14 10:50:09 - Retrieved 872092 regulatory features (0 mem, 872351 cached, 0 DB, 259 duplicates)</div><div>2013-05-14 10:50:12 - Calculating consequences</div><div>Use of uninitialized value $_ in pattern match (m//) at /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1022.</div>
<div>Use of uninitialized value $_ in pattern match (m//) at /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1030.</div><div>Use of uninitialized value $_ in pattern match (m//) at /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1037.</div>
<div>Use of uninitialized value $_ in pattern match (m//) at /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1089.</div><div>Use of uninitialized value $_ in concatenation (.) or string at /NGS_Test/duarte/vep_71/Bio/EnsEMBL/Variation/Utils/VEP.pm line 1097.</div>
<div><br></div><div>ERROR: Forked process failed</div><div><br></div><div><br></div><div><br></div></div></div><div class="gmail_extra"><div><br clear="all"><div><font color="#999999">=========================<br>
Duarte Miguel Paulo Molha <br></font><div><font color="#999999"> <a href="http://about.me/duarte" target="_blank">http://about.me/duarte</a> <br>=========================</font></div>
</div>
<br><br></div><div><div><div class="gmail_quote">On Tue, May 14, 2013 at 10:42 AM, Duarte Molha <span dir="ltr"><<a href="mailto:duartemolha@gmail.com" target="_blank">duartemolha@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Thanks<div><br></div><div>Running a annotation using 16 forks... lets see how it handles :)</div><div>I'll report back any issues.</div><div><br></div><div>Thanks for the update</div><span><font color="#888888">
<div><br></div><div>Duarte</div><div><br></div></font></span></div><div class="gmail_extra"><div><br clear="all"><div><font color="#999999">=========================<br> Duarte Miguel Paulo Molha <br>
</font><div><font color="#999999"> <a href="http://about.me/duarte" target="_blank">http://about.me/duarte</a> <br>=========================</font></div></div>
<br><br></div><div><div><div class="gmail_quote">On Tue, May 14, 2013 at 10:16 AM, Will McLaren <span dir="ltr"><<a href="mailto:wm2@ebi.ac.uk" target="_blank">wm2@ebi.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Stuart, Guillermo, Duarte,<div><br></div><div>I'm currently working on some code as I stated above to improve stability and performance under forking.</div><div><br></div><div>I've committed some code to the HEAD of our CVS tree which should help the problems you are encountering. You'd all be welcome to test this out, with the obvious proviso that this is development code and may contain bugs!</div>
<div><br></div><div>To use this, you should download the copy of VEP.pm from:</div><div><br></div><div><a href="http://cvs.sanger.ac.uk/cgi-bin/viewvc.cgi/ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils/VEP.pm?revision=1.92&root=ensembl" target="_blank">http://cvs.sanger.ac.uk/cgi-bin/viewvc.cgi/ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils/VEP.pm?revision=1.92&root=ensembl</a><br>
</div><div><br></div><div>and replace the VEP.pm under ensembl-variation/modules/Bio/EnsEMBL/Variation/Utils (or just Bio/EnsEMBL/Variation/Utils if you use INSTALL.pl) with this one.</div><div><br></div>
<div>This code will appear in production in the next proper release of Ensembl.</div><div><br></div><div>Regards</div><span><font color="#888888"><div><br></div><div>Will</div></font></span></div><div>
<div><div class="gmail_extra"><br><br><div class="gmail_quote">
On 14 May 2013 09:55, Stuart Meacham <span dir="ltr"><<a href="mailto:sm766@cam.ac.uk" target="_blank">sm766@cam.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hi,<br>
<br>
I certainly don't want to hijack this thread but it seemed daft to
start another. I am also getting forking errors. I don't use any
custom plugins and am using a validated VCF as input (with about
600,000 variants). Trying to fork more than 4 threads is unstable
even on my machine which has 64 cores and half a TB of RAM.<br>
<br>
I haven't found anything reproducible, however if I do I'll report
back to the list.<br>
<br>
Thanks<span><font color="#888888"><br>
<br>
Stuart</font></span><div><div><br>
<br>
On 14/05/2013 09:42, Will McLaren wrote:<br>
</div></div></div><div><div>
<blockquote type="cite">
<div dir="ltr">Hello,
<div><br>
</div>
<div>Your aa_grantham_distance plugin is somewhat inefficient -
it retrieves the peptide alleles from the HGVS annotation,
which itself requires some database fetching and processing to
produce. This is why it is slow.</div>
<div><br>
</div>
<div>You can get the peptides from the transcript
variation object:</div>
<div><br>
</div>
<div>my @peps = split "/",
$tva->transcript_variation->pep_allele_string();</div>
<div>
<br>
</div>
<div>This will give you single-letter AA codes, but you
could either modify your hash or use BioPerl to convert:</div>
<div><br>
</div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px">$seqobj
= Bio::PrimarySeq->new ( -seq => $single_letter_aa); </span><br style="font-family:Arial,Helvetica,sans-serif;font-size:13px">
<span style="font-family:Arial,Helvetica,sans-serif;font-size:13px">$three_letter_aa
= Bio::SeqUtils->seq3($seqobj); </span><br>
</div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px">You
should also declare your distances hash in the new() sub and
store it on $self; this will also marginally speed up your
plugin.</span></div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px">Regarding
the forking issues, we are working on improving stability
under forking.</span></div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px">Thanks
for your patience</span></div>
<div>
<span style="font-family:Arial,Helvetica,sans-serif;font-size:13px"><br>
</span></div>
<div><span style="font-family:Arial,Helvetica,sans-serif;font-size:13px">Will</span></div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
On 14 May 2013 07:37, Guillermo Marco Puche <span dir="ltr"><<a href="mailto:guillermo.marco@sistemasgenomicos.com" target="_blank">guillermo.marco@sistemasgenomicos.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000066">
<div>Hello,<br>
<br>
I'm not really sure which one of those plugins is
causing the fork error. I cannot recreate it now running
each one of them separately.<br>
<br>
Here are both:<br>
<br>
<a href="https://github.com/guillermomarco/vep_plugins_71" target="_blank">https://github.com/guillermomarco/vep_plugins_71</a><br>
<br>
They also slow the calculating consequences process a
lot. <a href="http://aa_grantham_distance.pm" target="_blank">aa_grantham_distance.pm</a>
is just a hardcoded plugin from one of the biologists in
my work. It was just a pure copy paste and adaptation to
make it work as a VEP plugin. Maybe the problem is in
the matrix definition every time the sub routine is
called. I'm not running out of memory nor CPU. I'm
currently using it with 2 threads and buffersize of 500
for a 5000 variant vcf file.<br>
<br>
I'm my honest opinion, I think one (or even both) of
those plugins are slowing so much the calculating
process that sometimes the fork just dies. Like when you
have a timeout during to heavy network traffic. So when
you use them together with lot of other plugins like
Condel, Consequence, etc.. they may be causing the
process to handle and die.<br>
<br>
Best regards,<br>
Guillermo.
<div>
<div><br>
<br>
On 05/13/2013 03:55 PM, Duarte Molha wrote:<br>
</div>
</div>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr">I also get this error... it is so
prevalent and so difficult to pinpoint what is
causing it that I have given up on forking my
annotation process.
<div><br>
</div>
<div>I do think it is related to the number of
forks. It seems to crash less often if you use a
low number of forks... anything above 5
will undoubtedly crash the script at least in my
experience.</div>
<div><br>
</div>
<div>Cheers</div>
<div><br>
Duarte</div>
</div>
<div class="gmail_extra"><br clear="all">
<div><font color="#999999">=========================<br>
Duarte Miguel Paulo Molha <br>
</font>
<div><font color="#999999"> <a href="http://about.me/duarte" target="_blank">http://about.me/duarte</a>
<br>
=========================</font></div>
</div>
<br>
<br>
<div class="gmail_quote">On Mon, May 13, 2013 at
2:50 PM, Will McLaren <span dir="ltr"><<a href="mailto:wm2@ebi.ac.uk" target="_blank">wm2@ebi.ac.uk</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Guillermo,
<div><br>
</div>
<div>Test each plugin individually until you
find the one that causes the error. It is
highly unlikely that a particular
combination of plugins is causing the
crash.</div>
<div><br>
</div>
<div>Check that there are no "print" (to
STDOUT or STDERR) statements in your
plugin - forking assumes that code remains
silent otherwise it will throw errors like
this.</div>
<div><br>
</div>
<div> Also, check what, if anything, is
cached between runs of your plugin. If you
are caching things (for example to avoid
re-querying a database), you may need to
write storable hooks to ensure the data is
getting cached between forks - see <a href="https://github.com/ensembl-variation/VEP_plugins/blob/master/ProteinSeqs.pm" target="_blank">https://github.com/ensembl-variation/VEP_plugins/blob/master/ProteinSeqs.pm</a>
for an example.</div>
<div><br>
</div>
<div>If you still have no luck, send me the
code and an input file that recreates the
problem.</div>
<div><br>
</div>
<div>Regards</div>
<div><br>
</div>
<div>Will</div>
</div>
<div class="gmail_extra"> <br>
<br>
<div class="gmail_quote">
<div>
<div>On 13 May 2013 13:18, Guillermo
Marco Puche <span dir="ltr"><<a href="mailto:guillermo.marco@sistemasgenomicos.com" target="_blank">guillermo.marco@sistemasgenomicos.com</a>></span>
wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>
<div bgcolor="#FFFFFF" text="#000066"> Hello,<br>
<br>
I've started to recently having
problems with VEP script while
using different plugins (most of
them own plugins).<br>
<br>
<pre>2013-05-13 13:59:44 - Connected to core version 71 database and variation version 71 database
2013-05-13 13:59:44 - Loaded plugin: vcf_input
2013-05-13 13:59:44 - Loaded plugin: biobase
2013-05-13 13:59:44 - Loaded plugin: aa_grantham_distance
2013-05-13 13:59:44 - Loaded plugin: flanking_sequence
2013-05-13 13:59:44 - Loaded plugin: Condel
2013-05-13 13:59:44 - Output fields redefined (37 defined)
2013-05-13 13:59:44 - Starting...
2013-05-13 13:59:45 - Read 3888 variants into buffer
2013-05-13 13:59:54 - Reading transcript data from cache and/or database
[===============================================] [ 100% ]
2013-05-13 14:02:38 - Retrieved 6463 transcripts (0 mem, 0 cached, 13743 DB, 7280 duplicates)
2013-05-13 14:02:38 - Calculating consequences
[===================================> ] [ 78% ]
ERROR: Forked process failed
</pre>
I'm not getting any other error
message. So I cannot debug
properly. I thought my plugins
were OK but it's seems they don't.
I think the problem occurs when I
use "aa_grantham_distance plugin"
together with "flanking_sequence".
I've no idea what could be causing
this.<br>
<br>
I'm running VEP on verbose mode
but I can't get any usefull
information. How could I debug
that?<br>
<br>
Best regards,<br>
Guillermo.<br>
<br>
</div>
<br>
</div>
</div>
_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and
subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br>
<br>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br>
<br></blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br>
<br></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br>
<br></blockquote></div><br></div>