<div dir="ltr">Hi Jan,<div><br></div><div class="gmail_extra"><div class="gmail_quote">On 4 March 2015 at 03:40, Jan Vogel <span dir="ltr"><<a href="mailto:jan.vogel@gmail.com" target="_blank">jan.vogel@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div><br></div><div>Hello Will, </div><div><br></div><div>I’m annotating some large scale data and I was thinking to create my own cache with the —write_cache option - my idea is that everytime I annotate a VCF, I add the variations, which have not been cached preciously, to the cache.<b> Is this what the write_cache option is intend to do ?</b> </div><div><br></div></div></blockquote><div><br></div><div>No, this is not how the cache works. The cache only contains data read from the Ensembl DBs. The --write_cache option is an alternative to e.g. "--build all" that creates the cache bit by bit according to your input:</div><div><br></div><div>1) Read variants</div><div><br></div><div>2) Get overlapped genomic region</div><div><br></div><div>3) If region overlapped found in cache, read data (transcripts, regulatory features, known variant locations) from cache</div><div><br></div><div>4) If region not found in cache, read data from database then write data to cache (then next time same region found use 3)</div><div><br></div><div>I don't expect many people to use this flag and in fact I'm considering removing it as it does seem to cause confusion amongst users. My initial idea was that it would be of use to people whose input always only spans a particular genomic region or regions (for example those doing targetted sequencing).</div><div><br></div><div>I think now it is just easier for everyone to download the whole genome cache file (or generate it using --build all if you have custom data in your Ensembl DBs) and not have to worry about any of the above.</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div></div><div>I ran into a bit of trouble when using it, as </div><div> - I had the same CODE ref exception from Storable.pm ( fixed it with your —no_adaptor_cache option) - might be a good idea to add this to <a href="http://uswest.ensembl.org/info/docs/tools/vep/script/vep_example.html" target="_blank">http://uswest.ensembl.org/info/docs/tools/vep/script/vep_example.html</a> </div></div></blockquote><div><br></div><div>As I said, this is fixed in newer versions, to which the documentation refers.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>I also run into trouble when forking the script - it seems to me that there is a race condition, and that the forking processes are modifying the same cache files - so I end up with corrupted cache, and error message like: </div><div><br></div><div>gzip: /.vep/homo_sapiens/77_GRCh38/1/2000001-3000000_var.gz: unexpected end of file</div><div>gzip: /.vep/homo_sapiens/77_GRCh38/1/3000001-4000000_var.gz: unexpected end of file</div><div>gzip: /.vep/homo_sapiens/77_GRCh38/1/11000001-12000000_var.gz: unexpected end of file</div><div><br></div><div>Have you seen this before ? </div></div></blockquote><div><br></div><div>I'm not surprised this happens, there is no file-level locking so using fork and write_cache would indeed give you this issue. I'll add them to the list of incompatible option pairs.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>Also, I can’t get VEP to work with two different cache files - my ideal setup would be </div><div><br></div><div>a) <b>a system-wide cache</b> with pre-computed cache data from EnsEMBL </div><div>b)<b> a “by-user” cache</b> - once a user computed a single variation which is not in the system-wide cache, it would be great to add it to the user-cache - so it does not get re-computed. </div></div></blockquote><div><br></div><div>I've looked into this before but there are a number of issues with the by-user cache:</div><div><br></div><div>1) the cache would need to contain all possible annotations for each variant, or be able to be updated if user requests e.g. HGVS when they didn't the first time it was cached</div><div><br></div><div>2) the cache would become out of date if the user updated VEP (transcripts etc may change in a newer version of Ensembl)</div><div><br></div><div>3) the cache would need to operate fast enough so that the performance benefit would outweigh just re-computing every time</div><div> </div><div>Other VEP users have created pipelines using VCFs or a noSQL DB as the VEP results cache, with a pre-VEP stage that looks up results from those resources then only runs VEP on the novel variants. A super simple way would be to use VCF output from the VEP, then use vcf-annotate to copy results to any subsequent input VCFs.</div><div><br></div><div>HTH</div><div><br></div><div>Will</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>Ideally, it would also be <b>possible to merge both caches</b> ( user + system-wide) - so other users can benefit from pre-calculated variations. </div><div><br></div><div><br></div><div>I’m in a multi-user environment, that’s why I am hesitant to have all users write to the same system-wide cache. </div><div><br></div><div>Do such options currently exist and did I just not find them ? Or am I running VEP the wrong way ? I was hoping that the <font face="Menlo">—dir</font> and <font face="Menlo">—dir_cache</font> options can be used this way ... </div><div><br></div><div><div>Here’s my command line : </div><div><br></div><div><font face="Menlo">perl ensembl-tools-release-78/scripts/variant_effect_predictor/<a href="http://variant_effect_predictor.pl" target="_blank">variant_effect_predictor.pl</a> </font></div><div><font face="Menlo">—write_cache </font></div><div><font face="Menlo"> —verbose</font></div><div><font face="Menlo"> —cache </font></div><div><font face="Menlo"> —force_overwrite</font></div><div><font face="Menlo"> -i test.vcf -o test.out </font></div><div><font face="Menlo"> --dir_cache /gne/research/workspace/vogelj4/variant_effect_predictor/jensenmann/igis_cache/new_cache --cache_version 77 </font></div><div><font face="Menlo"> --species homo_sapiens </font></div><div><font face="Menlo"> --db_version 77</font></div><div><font face="Menlo"> --dir /gne/research/workspace/vogelj4/variant_effect_predictor/jensenmann/igis_cache/e77.1/VEP/ </font></div><div><font face="Menlo"> --fork 12</font></div></div><div><br></div><div><br></div><div>Thanks for this great tool ! </div><span class="HOEnZb"><font color="#888888"><div><br></div><div> Jan </div><div><br></div><div><br></div><div><br></div><div><br></div><div><font face="Menlo"><br></font></div><div><br></div><div><br></div></font></span></div><br>_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br>
<br></blockquote></div><br></div></div>