<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">We extracted all variants, uniqued them, ran vep, then tabix index the vcf files.<div>These are your cache. You then can annotate your VCFs from this cache using a script to lookup variants in your VEP vcfs.</div><div><br></div><div><br></div><div><br></div><div><br><div><div>On 6 Mar 2015, at 07:25, Jan Vogel <<a href="mailto:jan.vogel@gmail.com">jan.vogel@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div><br></div>Hi Will, <div><br></div><div>thank you for the explanation on the —write_cache - as the option does not work with the —fork together, it’s use is pretty limited to only small regions. </div><div><br></div><div>Concerning the idea of a pre-VEP stage: I like your idea to work with vcf-annotate; a nice solution which will scale pretty well as don’t have a DB bottleneck.</div><div><br></div><div>To summarize </div><div><br></div><div><ul class="MailOutline"><li>run VEP on a few samples</li><li>collect VEP output files and write (unique) file for vcf-annotate</li><li>a script to combine vcf-annotae for known variants and VEP for missing variants ( output has the same format produced by VEP )</li></ul><div><br></div></div><div>If anyone has done this before and has some code on github, I’d love to take a peek. </div><div><br></div><div>Cheers, </div><div> </div><div> Jan </div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br><div><div>On Mar 4, 2015, at 1:25 AM, Will McLaren <<a href="mailto:wm2@ebi.ac.uk">wm2@ebi.ac.uk</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div dir="ltr">Hi Jan,<div><br></div><div class="gmail_extra"><div class="gmail_quote">On 4 March 2015 at 03:40, Jan Vogel<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:jan.vogel@gmail.com" target="_blank">jan.vogel@gmail.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex; position: static; z-index: auto;"><div style="word-wrap: break-word;"><div><br></div><div><br></div><div>Hello Will, </div><div><br></div><div>I’m annotating some large scale data and I was thinking to create my own cache with the —write_cache option - my idea is that everytime I annotate a VCF, I add the variations, which have not been cached preciously, to the cache.<b><span class="Apple-converted-space"> </span>Is this what the write_cache option is intend to do ?</b> </div><div><br></div></div></blockquote><div><br></div><div>No, this is not how the cache works. The cache only contains data read from the Ensembl DBs. The --write_cache option is an alternative to e.g. "--build all" that creates the cache bit by bit according to your input:</div><div><br></div><div>1) Read variants</div><div><br></div><div>2) Get overlapped genomic region</div><div><br></div><div>3) If region overlapped found in cache, read data (transcripts, regulatory features, known variant locations) from cache</div><div><br></div><div>4) If region not found in cache, read data from database then write data to cache (then next time same region found use 3)</div><div><br></div><div>I don't expect many people to use this flag and in fact I'm considering removing it as it does seem to cause confusion amongst users. My initial idea was that it would be of use to people whose input always only spans a particular genomic region or regions (for example those doing targetted sequencing).</div><div><br></div><div>I think now it is just easier for everyone to download the whole genome cache file (or generate it using --build all if you have custom data in your Ensembl DBs) and not have to worry about any of the above.</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;"><div></div><div>I ran into a bit of trouble when using it, as </div><div> - I had the same CODE ref exception from Storable.pm ( fixed it with your —no_adaptor_cache option) - might be a good idea to add this to <a href="http://uswest.ensembl.org/info/docs/tools/vep/script/vep_example.html" target="_blank">http://uswest.ensembl.org/info/docs/tools/vep/script/vep_example.html</a> </div></div></blockquote><div><br></div><div>As I said, this is fixed in newer versions, to which the documentation refers.</div><div> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;"><div><br></div><div>I also run into trouble when forking the script - it seems to me that there is a race condition, and that the forking processes are modifying the same cache files - so I end up with corrupted cache, and error message like: </div><div><br></div><div>gzip: /.vep/homo_sapiens/77_GRCh38/1/2000001-3000000_var.gz: unexpected end of file</div><div>gzip: /.vep/homo_sapiens/77_GRCh38/1/3000001-4000000_var.gz: unexpected end of file</div><div>gzip: /.vep/homo_sapiens/77_GRCh38/1/11000001-12000000_var.gz: unexpected end of file</div><div><br></div><div>Have you seen this before ? </div></div></blockquote><div><br></div><div>I'm not surprised this happens, there is no file-level locking so using fork and write_cache would indeed give you this issue. I'll add them to the list of incompatible option pairs.</div><div> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;"><div><br></div><div>Also, I can’t get VEP to work with two different cache files - my ideal setup would be </div><div><br></div><div>a)<span class="Apple-converted-space"> </span><b>a system-wide cache</b><span class="Apple-converted-space"> </span>with pre-computed cache data from EnsEMBL </div><div>b)<b><span class="Apple-converted-space"> </span>a “by-user” cache</b><span class="Apple-converted-space"> </span>- once a user computed a single variation which is not in the system-wide cache, it would be great to add it to the user-cache - so it does not get re-computed. </div></div></blockquote><div><br></div><div>I've looked into this before but there are a number of issues with the by-user cache:</div><div><br></div><div>1) the cache would need to contain all possible annotations for each variant, or be able to be updated if user requests e.g. HGVS when they didn't the first time it was cached</div><div><br></div><div>2) the cache would become out of date if the user updated VEP (transcripts etc may change in a newer version of Ensembl)</div><div><br></div><div>3) the cache would need to operate fast enough so that the performance benefit would outweigh just re-computing every time</div><div> </div><div>Other VEP users have created pipelines using VCFs or a noSQL DB as the VEP results cache, with a pre-VEP stage that looks up results from those resources then only runs VEP on the novel variants. A super simple way would be to use VCF output from the VEP, then use vcf-annotate to copy results to any subsequent input VCFs.</div><div><br></div><div>HTH</div><div><br></div><div>Will</div><div><br></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;"><div><br></div><div>Ideally, it would also be<span class="Apple-converted-space"> </span><b>possible to merge both caches</b><span class="Apple-converted-space"> </span>( user + system-wide) - so other users can benefit from pre-calculated variations. </div><div><br></div><div><br></div><div>I’m in a multi-user environment, that’s why I am hesitant to have all users write to the same system-wide cache. </div><div><br></div><div>Do such options currently exist and did I just not find them ? Or am I running VEP the wrong way ? I was hoping that the<span class="Apple-converted-space"> </span><font face="Menlo">—dir</font><span class="Apple-converted-space"> </span>and<span class="Apple-converted-space"> </span><font face="Menlo">—dir_cache</font><span class="Apple-converted-space"> </span>options can be used this way ... </div><div><br></div><div><div>Here’s my command line : </div><div><br></div><div><font face="Menlo">perl ensembl-tools-release-78/scripts/variant_effect_predictor/<a href="http://variant_effect_predictor.pl/" target="_blank">variant_effect_predictor.pl</a> </font></div><div><font face="Menlo">—write_cache </font></div><div><font face="Menlo"> —verbose</font></div><div><font face="Menlo"> —cache </font></div><div><font face="Menlo"> <span class="Apple-converted-space"> </span>—force_overwrite</font></div><div><font face="Menlo"> -i test.vcf -o test.out </font></div><div><font face="Menlo"> <span class="Apple-converted-space"> </span>--dir_cache /gne/research/workspace/vogelj4/variant_effect_predictor/jensenmann/igis_cache/new_cache --cache_version 77 </font></div><div><font face="Menlo"> <span class="Apple-converted-space"> </span>--species homo_sapiens </font></div><div><font face="Menlo"> --db_version 77</font></div><div><font face="Menlo"> --dir /gne/research/workspace/vogelj4/variant_effect_predictor/jensenmann/igis_cache/e77.1/VEP/ </font></div><div><font face="Menlo"> <span class="Apple-converted-space"> </span>--fork 12</font></div></div><div><br></div><div><br></div><div>Thanks for this great tool ! </div><span class="HOEnZb"><font color="#888888"><div><br></div><div> Jan </div><div><br></div><div><br></div><div><br></div><div><br></div><div><font face="Menlo"><br></font></div><div><br></div><div><br></div></font></span></div><br>_______________________________________________<br>Dev mailing list <span class="Apple-converted-space"> </span><a href="mailto:Dev@ensembl.org">Dev@ensembl.org</a><br>Posting guidelines and subscribe/unsubscribe info:<span class="Apple-converted-space"> </span><a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a><br>Ensembl Blog:<span class="Apple-converted-space"> </span><a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br><br></blockquote></div><br></div></div>_______________________________________________<br>Dev mailing list <a href="mailto:Dev@ensembl.org">Dev@ensembl.org</a><br>Posting guidelines and subscribe/unsubscribe info:<span class="Apple-converted-space"> </span><a href="http://lists.ensembl.org/mailman/listinfo/dev">http://lists.ensembl.org/mailman/listinfo/dev</a><br>Ensembl Blog:<span class="Apple-converted-space"> </span><a href="http://www.ensembl.info/">http://www.ensembl.info/</a></div></blockquote></div><br></div></div>_______________________________________________<br>Dev mailing list <a href="mailto:Dev@ensembl.org">Dev@ensembl.org</a><br>Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev">http://lists.ensembl.org/mailman/listinfo/dev</a><br>Ensembl Blog: <a href="http://www.ensembl.info/">http://www.ensembl.info/</a><br></blockquote></div><br></div></body></html>