<div> </div>
<div>Hi Guillermo,</div>
<div> </div>
<div>The plugins are not using threads but they will use forks when the -fork option is specified, as mentioned in the documentation you quoted previously.</div>
<div> </div>
<div>Different batches of variants will have different data and therefore take different lengths of time to analyse. When a batch of variants is submitted for analysis it is divided in equal numbers between the required number of forks which will take different lengths of time to complete before the next batch is submitted for analysis.It is more likely one fork will run longer than the others than they all complete together.</div>
<div> </div>
<div>Best wishes,</div>
<div> </div>
<div>Sarah<br></div>
<div> </div>
<div class="gmail_quote">On Thu, Jun 6, 2013 at 1:24 PM, Guillermo Marco Puche <span dir="ltr"><<a href="mailto:guillermo.marco@sistemasgenomicos.com" target="_blank">guillermo.marco@sistemasgenomicos.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div text="#000066" bgcolor="#FFFFFF">
<div>Hello,<br><br>Are plugins using multiple threads feature? If not how can i force plugin to benefit from multiple threads when specified?<br><br>I've noticed launching vep with 8 CPUs at start of calculating consequences I can see the 8 perl processes running cpu at 100%. Then after 1hour or so just 1 thread is active with CPU at 100% the rest are still alive but not using cpu. Then after some time number of variants specified in buffer size is written into output file then then all the threads resume to use CPU.<br>
<br>So for VEP with 8 threads it would be something like:<br>
<ul>
<li>Retrieve info from database</li>
<li>Calculate consequences with all threads at max CPU usage</li>
<li>Calculate consequences with 1 thread at max CPU usage</li>
<li>Write output</li></ul>
<p>Repeat.<br></p>
<p>I would like to know if the time where only 1 CPU is being is due to the fact that plugins are maybe not using multiple threads option, or if it's because VEP script is doing any other thing that I don't know.<br>
</p>
<p>Thank you.<br></p>
<p>Best regards,<br>Guillermo.<br></p>
<div>
<div>On 06/05/2013 12:41 PM, Guillermo Marco Puche wrote:<br></div></div></div>
<blockquote type="cite">
<div>
<div>Hello dear developers,<br><br>I think I've a stable configuration that fits my needs for huge number of variants.<br><br>My last VCF input has 400.000 variants.<br><br>Currently on my cluster, with one compute node (x8 cpu and 32GB RAM) using 8 threads and a buffer size of 15000 variants the time for VEP with all the plugins and options I need it takes VEP 2h 56min to calculate 15000 variants.<br>
<br>I'm using a local ensembl71 database replica + cache for homo_sapiens. So the time to load vars into memory is very small.<br><br>The 99% of time it takes the VEP script it's obviously from "Calculating consequences".<br>
<br>I've also noticed that VEP with 8 threads consumes the 100% of my 8 CPUs with 8 threads, it's really great. But RAM load being used is very low 8GB.<br><br>So I've a few questions. <br><br>
<ul>
<li>Has someone achieved to parallelize VEP process with MPI or OpenMPI? It would be awesome being able to select for example 16 threads and being able to distribute 8 and 8 threads between two different machines (compute nodes).<br>
<br></li>
<li>In order to optimize self coded plugins, I've been reading into this from VEP ensembl website: <i>"VEP users writing plugins should be aware that while the VEP code attempts to preserve the state of any plugin-specific cached data between separate forks, there may be situations where data is lost. If you find this is the case, you should disable forking in the new() method of your plugin by deleting the "fork" key from the $config hash."<br>
<br></i>I had no problems with my plugins after fixing them (thanks to the great support of developers on this list). But I feel they're slowing VEP I'm sure they can be optimized. I really would like a direction, guide or some tips that I could use to optimize my code. <br>
<br></li>
<li>I hope a new way to share plugins between VEP users is available soon, so we can help, give tips between all devs to improve the code, speed, results etc..</li></ul>
<p><br>Thank you !<br></p>
<p>Best regards,<br>Guillermo.<br></p><br><br><br>
<fieldset></fieldset> <br></div></div><pre>_______________________________________________
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a>
</pre></blockquote></div><br>_______________________________________________<br>Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br><br></blockquote></div><br>