<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have a library of about 93 million variants that I want to run VEP on. I have downloaded VEP version 84 onto a local linux machine running CentOS Linux release 7.1.1503. The machine has 125G and 40 cores @ 2.3GHz. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My variant library is stored in an impala table in hdfs. To generate my input into VEP, I run a simple impala query as follows:<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">impala -B -o ~/vep/knownvariants.tsv --output_delimiter="\\t" -q "select contig, pos, pos+length, concat(ref,'\/',alt), '+' from ingest.central_variant_store where variant_type != 'COMPLEX'"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal">This writes the variants to a tab delimited file in ensembl format:<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">10 57676885 57676886 C/G +<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">7 34456697 34456698 A/G +<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">8 62679252 62679253 G/A +<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">7 9184853 9184854 C/A +<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">3 29205854 29205855 C/T +<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">10 42815272 42815273 C/T +<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">8 117963405 117963406 C/T +<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">12 53054550 53054551 C/T +<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">6 105515195 105515196 T/C +<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">20 665650 665651 G/C +<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The entire library is about 93 million. I filtered out the complex variants (i.e. CNVs) as my experience showed that these just threw warnings in VEP and remained unannotated. My idea was to break this output into smaller chunks for processing,
ideally around 25M variants each. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I then feed each chunk into VEP using the following arguments:<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">perlbrew switch 5.16.3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">perlbrew exec perl ./variant_effect_predictor.pl --cache --offline --everything --json --buffer_size 25000000 --force_overwrite --verbose -i knownvariants.tsv -o knownvariants.json 2>&1 | tee knownvariants.log<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal">VEP will crunch on this for a while. It will usually clear the following steps:<o:p></o:p></p>
<p class="MsoNormal">-load variants into buffer<o:p></o:p></p>
<p class="MsoNormal">-check for existing variations<o:p></o:p></p>
<p class="MsoNormal">-read transcript data from cache<o:p></o:p></p>
<p class="MsoNormal">-read regulatory data from cache<o:p></o:p></p>
<p class="MsoNormal">-begin analyzing chromosomes<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Somewhere in the last step, VEP will crash with the following error message:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Command [perl ./variant_effect_predictor.pl --cache --offline --everything --json --buffer_size 25000000 --force_overwrite --verbose -i knownvariants.tsv -o knownvariants.json] terminated with exit
code 0 ($? = 9) under the following perl environment:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Command terminated with non-zero status.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Current perl:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> Name: perl-5.16.3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> Path: /opt/perl5/perls/perl-5.16.3/bin/perl<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> Config: -de -Dprefix=/opt/perl5/perls/perl-5.16.3 -Aeval:scriptdir=/opt/perl5/perls/perl-5.16.3/bin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> Compiled at: Apr 15 2016 06:06:22<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">perlbrew:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> version: 0.75<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> ENV:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> PERLBREW_ROOT: /opt/perl5<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> PERLBREW_HOME: <o:p>
</o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> PERLBREW_PATH: /opt/perl5/bin:/opt/perl5/perls/perl-5.16.3/bin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> PERLBREW_MANPATH: /opt/perl5/perls/perl-5.16.3/man<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This exit code is not very specific so it is hard to know what is going on. Is it running out of memory? I lean away from that because it seems that I have seen specific error messages related to memory usage when that happens. I have tried
inputting smaller numbers of variants such as 12.5M, 10M, 5M, 2.5M, 1M, 100K, and 10K. So far, I can generally execute just fine on up to 2.5M variants. Anything bigger and I get this same error message. I also got this once on a 2.5M run, only this time the
program crashed after analysis of all the chromosomes. It was actively writing to the json output when it died.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This leads me to ask if there is a known upper limit to how many variants one can practically push through at a time? Or perhaps a timeout limit? Processing 93M in 2.5M chunks is a bit tedious. Any thoughts on how to improve or optimize
this would be appreciated. I have attached the log files from several runs for reference.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Sean Taylor<o:p></o:p></p>
</div>
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information protected by law. Any unauthorized review, use, disclosure or distribution is prohibited.
If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
</body>
</html>