<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div>Hello Johanne,<br class=""><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">
</div>
<div class="">The only exception I get now from time to time, is:</div>
<div class="">DBD::mysql::st execute failed: Lost connection to MySQL server during query at /Users/Johanne/src/ensembl/modules//Bio/EnsEMBL/DBSQL/BaseAdaptor.pm line 482. </div>
<div class=""><br class="">
</div>
<div class="">Is it caused by something concerning my local MySQL installation or Internet connection?</div></div></div></blockquote><div><br class=""></div><div>This is probably related to a long run time of the script. I will look into this.</div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">As our thread contained two conversations, I wonder if you got the same results on the LD variant expansion script as I did with the two reference genomes?</div></div></div></blockquote><div><br class=""></div>I still need to run the comparison. But a difference of 13 variants in the results doesn’t seem to be a problem to me. You are using an updated assembly with hg38 which has consequences on the set of variants you are looking at. The blog post (<a href="http://genomeref.blogspot.co.uk/2013/12/announcing-grch38.html" class="">http://genomeref.blogspot.co.uk/2013/12/announcing-grch38.html</a>) explains this in more detail in the section 'General assembly updates'. </div><div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">
</div>
<div class="">And does your database contain all information from dbSNP + 1000G? I have set the script to use both database and VCF data ($variation_adaptor->db->use_vcf(1);), but did not find info on where exactly the data in the database comes from.</div></div></div></blockquote><div><br class=""></div>We store all 1000G variants in our database. The variants are imported from dbSNP. The genotypes for 1000G variants are stored in VCF files. To use the genotypes from VCF files you need to set use_vcf to 1.<br class=""><div><br class=""></div><div><br class=""></div><div>Best,</div><div>Anja</div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">
</div>
<div class="">Best,</div>
<div class="">Johanne</div>
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">23. mai 2016 kl. 16.45 skrev Will McLaren <<a href="mailto:wm2@ebi.ac.uk" class="">wm2@ebi.ac.uk</a>>:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Hello,
<div class=""><br class="">
</div>
<div class="">The files there are symbolic links to another directory - it's possible your FTP client is not following these links.</div>
<div class=""><br class="">
</div>
<div class="">The "real" files are here:</div>
<div class=""><br class="">
</div>
<div class=""><a href="ftp://ftp.ensembl.org/pub/release-82/variation/vcf/homo_sapiens/1000GENOMES-phase_3-genotypes/" class="">ftp://ftp.ensembl.org/pub/release-82/variation/vcf/homo_sapiens/1000GENOMES-phase_3-genotypes/</a><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Try downloading from that path instead.</div>
<div class=""><br class="">
</div>
<div class="">Regards</div>
<div class=""><br class="">
</div>
<div class="">Will</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On 23 May 2016 at 15:31, Johanne Håøy Horn <span dir="ltr" class="">
<<a href="mailto:johannhh@ifi.uio.no" target="_blank" class="">johannhh@ifi.uio.no</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Hi,
<div class=""><br class="">
</div>
<div class="">I got the same error message with the full path.
<div class="">
<div class=""><br class="">
</div>
<div class="">I think the problem is with the hg38 .gz and .gz.tbi files. I went back to the URL you gave me: <a href="ftp://ftp.ensembl.org/pub/variation_genotype/homo_sapiens/" target="_blank" class="">ftp://ftp.ensembl.org/pub/variation_genotype/homo_sapiens/</a></div>
<div class="">But when I click any of the files, I get the error message "The operation can’t be completed because the original item for <file name> can’t be found». This is not a problem for the hg19 ftp connection, whose files I can open just fine.</div>
<div class="">I have restarted my computer (Max OS X 10.11.5) and remounted the connection several times. I log on using guest.</div>
<div class=""><br class="">
</div>
<div class="">So the files I downloaded were not existing after all, and the error message were correct.</div>
<div class=""><br class="">
</div>
<div class="">Do you have any suggestions as to how I can mount it correctly?</div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class="">Johanne</div>
<div class="">
<div class="h5">
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">23. mai 2016 kl. 15.31 skrev Will McLaren <<a href="mailto:wm2@ebi.ac.uk" target="_blank" class="">wm2@ebi.ac.uk</a>>:</div>
<br class="">
<div class="">
<div dir="ltr" class="">I think possibly Perl doesn't like using "~" to represent your home directory - try replacing it with the full path, or possibly $ENV{HOME}
<div class=""><br class="">
</div>
<div class="">Will</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On 23 May 2016 at 13:59, Johanne Håøy Horn <span dir="ltr" class="">
<<a href="mailto:johannhh@ifi.uio.no" target="_blank" class="">johannhh@ifi.uio.no</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Thank you for your wonderful support!
<div class=""><br class="">
</div>
<div class="">I tried now with the following JSON struct:
<div class=""><span class="">
<div class="">{</div>
<div class=""> "id": "1000genomes_phase3",</div>
<div class=""> "species": "homo_sapiens",</div>
<div class=""> "assembly": "GRCh38",</div>
<div class=""> "type": "local",</div>
<div class=""> "strict_name_match": 1,</div>
</span>
<div class=""> "filename_template": "~/src/ensembl-vcf/ALL.chr###CHR###.phase3_shapeit2_mvncall_integrated_v3plus_nounphased.rsID.genotypes.GRCh38_dbSNP.vcf.gz",</div>
<span class="">
<div class=""> "chromosomes": [</div>
<div class=""> "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14",</div>
<div class=""> "15", "16", "17", "18", "19", "20", "21", "22", "X", "Y"</div>
<div class=""> ],</div>
<div class=""> "sample_prefix": "1000GENOMES:phase_3:"</div>
<div class=""> },</div>
</span></div>
<div class=""><br class="">
</div>
<div class="">I got this error message:</div>
<div class="">MSG: ERROR: VCF file ~/src/ensembl-vcf/ALL.chr1.phase3_shapeit2_mvncall_integrated_v3plus_nounphased.rsID.genotypes.GRCh38_dbSNP.vcf.gz not found</div>
<div class=""><br class="">
</div>
<div class="">Should the tbi files be where I call the script? Or is it something else I am doing wrong?</div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class="">Johanne</div>
<div class="">
<div class="">
<div class=""><br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">23. mai 2016 kl. 14.04 skrev Will McLaren <<a href="mailto:wm2@ebi.ac.uk" target="_blank" class="">wm2@ebi.ac.uk</a>>:</div>
<br class="">
<div class="">
<div dir="ltr" class="">Hi Johanne,
<div class=""><br class="">
</div>
<div class="">You need the filename part of the template too, so:</div>
<div class=""><br class="">
</div>
<div class=""><span style="font-size:12.8px" class=""> "filename_template": "~/src/ensembl-vcf/ALL.chr###CHR###.phase3_shapeit2_mvncall_integrated_v3plus_nounphased.rsID.genotypes.GRCh38_dbSNP.vcf.gz",</span></div>
<div class=""><span style="font-size:12.8px" class=""><br class="">
</span></div>
<div class=""><span style="font-size:12.8px" class="">Regards</span></div>
<div class=""><span style="font-size:12.8px" class=""><br class="">
</span></div>
<div class=""><span style="font-size:12.8px" class="">Will</span></div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On 23 May 2016 at 12:57, Johanne Håøy Horn <span dir="ltr" class="">
<<a href="mailto:johannhh@ifi.uio.no" target="_blank" class="">johannhh@ifi.uio.no</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Hello again!
<div class=""><br class="">
</div>
<div class="">I tried set the following in the JSON file:</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""> {</div>
<div class=""> "id": "1000genomes_phase3",</div>
<div class=""> "species": "homo_sapiens",</div>
<div class=""> "assembly": "GRCh38",</div>
<div class=""> "type": "local",</div>
<div class=""> "strict_name_match": 1,</div>
<div class=""> "filename_template": "~/src/ensembl-vcf/",</div>
<div class=""> "chromosomes": [</div>
<div class=""> "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14",</div>
<div class=""> "15", "16", "17", "18", "19", "20", "21", "22", "X", "Y"</div>
<div class=""> ],</div>
<div class=""> "sample_prefix": "1000GENOMES:phase_3:"</div>
<div class=""> },</div>
<div class=""><br class="">
</div>
<div class="">But I get this error message:</div>
<div class="">MSG: ERROR: VCF file ~/src/ensembl-vcf/ not found</div>
<div class=""><br class="">
</div>
<div class="">I downloaded all the hg38 files you linked to in the folder ~/src/ensembl-vcf/. When you say that I need to change filename_template to the path where the files were downloaded, is it the full path of all the 48 files rather than the path to
the folder they are in?</div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class="">Johanne</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class=""><span class="">
<div class="">23. mai 2016 kl. 11.57 skrev Will McLaren <<a href="mailto:wm2@ebi.ac.uk" target="_blank" class="">wm2@ebi.ac.uk</a>>:</div>
<br class="">
</span>
<div class="">
<div class="">
<div class="">
<div dir="ltr" class="">Hi Johanne,
<div class=""><br class="">
</div>
<div class="">It looks like the API is intermittently losing connection to the remote VCF files hosted on our FTP site.</div>
<div class=""><br class="">
</div>
<div class="">You can bypass this connection by downloading the files to your local machine:</div>
<div class=""><br class="">
</div>
<div class="">GRCh38: <a href="ftp://ftp.ensembl.org/pub/variation_genotype/homo_sapiens/" target="_blank" class="">
ftp://ftp.ensembl.org/pub/variation_genotype/homo_sapiens/</a><br class="">
</div>
<div class="">GRCh37: <a href="ftp://ftp.ensembl.org/pub/grch37/release-82/variation/vcf/homo_sapiens/1000GENOMES-phase_3-genotypes/" target="_blank" class="">
ftp://ftp.ensembl.org/pub/grch37/release-82/variation/vcf/homo_sapiens/1000GENOMES-phase_3-genotypes/</a><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">You will then need to edit [module_path]/ensembl-variation/modules/Bio/EnsEMBL/Variation/DBSQL/vcf_config.json, changing the "filename_template" entry to the path where you downloaded the files, and "type" from "remote" to "local".</div>
<div class=""><br class="">
</div>
<div class="">Regarding the warning message, this should not affect your analyses in any way, but I have put in a fix on release/84 of ensembl-variation to suppress it.</div>
<div class=""><br class="">
</div>
<div class="">Regards</div>
<div class=""><br class="">
</div>
<div class="">Will McLaren</div>
<div class="">Ensembl Variation</div>
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On 21 May 2016 at 12:57, Johanne Håøy Horn <span dir="ltr" class="">
<<a href="mailto:johannhh@ifi.uio.no" target="_blank" class="">johannhh@ifi.uio.no</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word" class="">Dear ensembl dev team,
<div class=""><br class="">
</div>
<div class="">I have been using your variation API for some time now, and get a range of errors from time to time, without knowing exactly why. It is not because of the scripts, I think, as the same script producing the error can work just fine if I run it
again.</div>
<div class=""><br class="">
</div>
<div class="">The different error messages are:</div>
<div class="">
<div class=""><font face="Courier New" class="">/Parser/BaseVCF4.pm line 891, <IN> line 5.</font></div>
<div class=""><font face="Courier New" class="">Use of uninitialized value in list assignment at /Users/Johanne/src/ensembl-io/modules/Bio/EnsEMBL/IO/Parser/BaseVCF4.pm line 891, <IN> line 5.</font></div>
<div class=""><span style="white-space:pre-wrap" class=""><font face="Courier New" class=""></font></span></div>
<div class=""><font face="Courier New" class="">connect: Operation timed out</font></div>
<div class=""><font face="Courier New" class="">[kftp_connect_file] 350 Restarting at 654385206. Send STORE or RETRIEVE to initiate transfer</font></div>
<div class=""><font face="Courier New" class=""> </font></div>
<div class=""><font face="Courier New" class="">[kftp_connect_file] 227 Entering Passive Mode (193,62,203,85,220,250).</font></div>
<div class=""><font face="Courier New" class="">Tabix::tabix_query: t is not of type tabix_tPtr at /Users/Johanne/src/ensembl-io/modules/Bio/EnsEMBL/IO/TabixParser.pm line 70.</font></div>
<div class=""><font face="Courier New" class=""> </font></div>
<div class=""><font face="Courier New" class="">[kftp_connect_file] 227 Entering Passive Mode (193,62,203,85,157,134).</font></div>
<div class=""><font face="Courier New" class="">[main] fail to open the data file.</font></div>
<div class=""><font face="Courier New" class="">Can't use an undefined value as an ARRAY reference at /Users/Johanne/src/ensembl-io/modules/Bio/EnsEMBL/IO/Parser/BaseVCF4.pm line 730.</font></div>
</div>
<div class=""><br class="">
</div>
<div class="">Usually just one of these error occur at a time. I suspect it might have something to do with the connection between my computer and the ensembl database, as the first error at least always show up in repeats when I lose my Internet connection.
However, are all of them caused by Internet trouble? I have checked that the MySQL instance is up and running, and can visit web pages through a browser when some of the errors occur. Could it be something on the server/database side?</div>
<div class=""><br class="">
</div>
<div class="">Also, if I use the GRCh37 database:</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""><font face="Courier New" class="">$registry->load_registry_from_db(</font></div>
<div class=""><font face="Courier New" class=""> -host => '<a href="http://ensembldb.ensembl.org/" target="_blank" class="">ensembldb.ensembl.org</a>',</font></div>
<div class=""><font face="Courier New" class=""> -user => 'anonymous',</font></div>
<div class=""><font face="Courier New" class=""> -port => 3337,</font></div>
<div class=""><font face="Courier New" class="">);</font></div>
</div>
<div class=""><font face="Courier New" class=""><br class="">
</font></div>
<div class="">I get this warning/printout:</div>
<div class=""><font face="Courier New" class="">Use of uninitialized value $nums{"."} in numeric comparison (<=>) at /Users/Johanne/src/ensembl-variation/modules/Bio/EnsEMBL/Variation/VCFCollection.pm line 770, <IN> line 6.</font></div>
<div class=""><font face="Courier New" class=""><br class="">
</font></div>
<div class="">I use version 84 of the Ensembl API, OS X 10.11.5, and the script I use when all of these errors occur, is attached. Note that the attached script by default uses hg38, but will produce the last printout mentioned when switching to hg37.</div>
<div class=""><br class="">
</div>
<div class="">And something different I have been wondering about:</div>
<div class="">The VCF files that are downloaded locally (ALL.chr1.phase3_shapeit2_mvncall_integrated_v3plus_nounphased.rsID.genotypes.GRCh38_dbSNP.vcf.gz.tbi, for instance) - should they be deleted and re-downloaded from time to time to get the latest 1000G
data? And where exactly are the VCFs downloaded from? Is it dbSNP, as indicated in the file name?</div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class="">Johanne Håøy Horn</div>
<div class=""></div>
</div>
<br class="">
_______________________________________________<br class="">
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank" class="">Dev@ensembl.org</a><br class="">
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" rel="noreferrer" target="_blank" class="">
http://lists.ensembl.org/mailman/listinfo/dev</a><br class="">
Ensembl Blog: <a href="http://www.ensembl.info/" rel="noreferrer" target="_blank" class="">
http://www.ensembl.info/</a><br class="">
<br class="">
</blockquote>
</div>
<br class="">
</div>
_______________________________________________<br class="">
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank" class="">Dev@ensembl.org</a><br class="">
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank" class="">
http://lists.ensembl.org/mailman/listinfo/dev</a><br class="">
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank" class="">http://www.ensembl.info/</a><br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
<br class="">
_______________________________________________<br class="">
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank" class="">Dev@ensembl.org</a><br class="">
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" rel="noreferrer" target="_blank" class="">
http://lists.ensembl.org/mailman/listinfo/dev</a><br class="">
Ensembl Blog: <a href="http://www.ensembl.info/" rel="noreferrer" target="_blank" class="">
http://www.ensembl.info/</a><br class="">
<br class="">
</blockquote>
</div>
<br class="">
</div>
_______________________________________________<br class="">
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank" class="">Dev@ensembl.org</a><br class="">
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank" class="">
http://lists.ensembl.org/mailman/listinfo/dev</a><br class="">
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank" class="">http://www.ensembl.info/</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</div>
<br class="">
_______________________________________________<br class="">
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank" class="">Dev@ensembl.org</a><br class="">
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" rel="noreferrer" target="_blank" class="">
http://lists.ensembl.org/mailman/listinfo/dev</a><br class="">
Ensembl Blog: <a href="http://www.ensembl.info/" rel="noreferrer" target="_blank" class="">
http://www.ensembl.info/</a><br class="">
<br class="">
</blockquote>
</div>
<br class="">
</div>
_______________________________________________<br class="">
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank" class="">Dev@ensembl.org</a><br class="">
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank" class="">
http://lists.ensembl.org/mailman/listinfo/dev</a><br class="">
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank" class="">http://www.ensembl.info/</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</div>
</div>
<br class="">
_______________________________________________<br class="">
Dev mailing list <a href="mailto:Dev@ensembl.org" class="">Dev@ensembl.org</a><br class="">
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" rel="noreferrer" target="_blank" class="">
http://lists.ensembl.org/mailman/listinfo/dev</a><br class="">
Ensembl Blog: <a href="http://www.ensembl.info/" rel="noreferrer" target="_blank" class="">
http://www.ensembl.info/</a><br class="">
<br class="">
</blockquote>
</div>
<br class="">
</div>
_______________________________________________<br class="">
Dev mailing list <a href="mailto:Dev@ensembl.org" class="">Dev@ensembl.org</a><br class="">
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" class="">
http://lists.ensembl.org/mailman/listinfo/dev</a><br class="">
Ensembl Blog: <a href="http://www.ensembl.info/" class="">http://www.ensembl.info/</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">Dev mailing list <a href="mailto:Dev@ensembl.org" class="">Dev@ensembl.org</a><br class="">Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" class="">http://lists.ensembl.org/mailman/listinfo/dev</a><br class="">Ensembl Blog: <a href="http://www.ensembl.info/" class="">http://www.ensembl.info/</a><br class=""></div></blockquote></div><br class=""></body></html>