<div dir="ltr">Hello,<div><br></div><div>The files there are symbolic links to another directory - it's possible your FTP client is not following these links.</div><div><br></div><div>The "real" files are here:</div><div><br></div><div><a href="ftp://ftp.ensembl.org/pub/release-82/variation/vcf/homo_sapiens/1000GENOMES-phase_3-genotypes/">ftp://ftp.ensembl.org/pub/release-82/variation/vcf/homo_sapiens/1000GENOMES-phase_3-genotypes/</a><br></div><div><br></div><div>Try downloading from that path instead.</div><div><br></div><div>Regards</div><div><br></div><div>Will</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 23 May 2016 at 15:31, Johanne Håøy Horn <span dir="ltr"><<a href="mailto:johannhh@ifi.uio.no" target="_blank">johannhh@ifi.uio.no</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
Hi,
<div><br>
</div>
<div>I got the same error message with the full path.
<div>
<div><br>
</div>
<div>I think the problem is with the hg38 .gz and .gz.tbi files. I went back to the URL you gave me: <a href="ftp://ftp.ensembl.org/pub/variation_genotype/homo_sapiens/" target="_blank">ftp://ftp.ensembl.org/pub/variation_genotype/homo_sapiens/</a></div>
<div>But when I click any of the files, I get the error message "The operation can’t be completed because the original item for <file name> can’t be found». This is not a problem for the hg19 ftp connection, whose files I can open just fine.</div>
<div>I have restarted my computer (Max OS X 10.11.5) and remounted the connection several times. I log on using guest.</div>
<div><br>
</div>
<div>So the files I downloaded were not existing after all, and the error message were correct.</div>
<div><br>
</div>
<div>Do you have any suggestions as to how I can mount it correctly?</div>
<div><br>
</div>
<div>Best,</div>
<div>Johanne</div><div><div class="h5">
<div><br>
<div>
<blockquote type="cite">
<div>23. mai 2016 kl. 15.31 skrev Will McLaren <<a href="mailto:wm2@ebi.ac.uk" target="_blank">wm2@ebi.ac.uk</a>>:</div>
<br>
<div>
<div dir="ltr">I think possibly Perl doesn't like using "~" to represent your home directory - try replacing it with the full path, or possibly $ENV{HOME}
<div><br>
</div>
<div>Will</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 23 May 2016 at 13:59, Johanne Håøy Horn <span dir="ltr">
<<a href="mailto:johannhh@ifi.uio.no" target="_blank">johannhh@ifi.uio.no</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Thank you for your wonderful support!
<div><br>
</div>
<div>I tried now with the following JSON struct:
<div><span>
<div>{</div>
<div> "id": "1000genomes_phase3",</div>
<div> "species": "homo_sapiens",</div>
<div> "assembly": "GRCh38",</div>
<div> "type": "local",</div>
<div> "strict_name_match": 1,</div>
</span>
<div> "filename_template": "~/src/ensembl-vcf/ALL.chr###CHR###.phase3_shapeit2_mvncall_integrated_v3plus_nounphased.rsID.genotypes.GRCh38_dbSNP.vcf.gz",</div>
<span>
<div> "chromosomes": [</div>
<div> "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14",</div>
<div> "15", "16", "17", "18", "19", "20", "21", "22", "X", "Y"</div>
<div> ],</div>
<div> "sample_prefix": "1000GENOMES:phase_3:"</div>
<div> },</div>
</span></div>
<div><br>
</div>
<div>I got this error message:</div>
<div>MSG: ERROR: VCF file ~/src/ensembl-vcf/ALL.chr1.phase3_shapeit2_mvncall_integrated_v3plus_nounphased.rsID.genotypes.GRCh38_dbSNP.vcf.gz not found</div>
<div><br>
</div>
<div>Should the tbi files be where I call the script? Or is it something else I am doing wrong?</div>
<div><br>
</div>
<div>Best,</div>
<div>Johanne</div>
<div>
<div>
<div><br>
<div>
<blockquote type="cite">
<div>23. mai 2016 kl. 14.04 skrev Will McLaren <<a href="mailto:wm2@ebi.ac.uk" target="_blank">wm2@ebi.ac.uk</a>>:</div>
<br>
<div>
<div dir="ltr">Hi Johanne,
<div><br>
</div>
<div>You need the filename part of the template too, so:</div>
<div><br>
</div>
<div><span style="font-size:12.8px"> "filename_template": "~/src/ensembl-vcf/ALL.chr###CHR###.phase3_shapeit2_mvncall_integrated_v3plus_nounphased.rsID.genotypes.GRCh38_dbSNP.vcf.gz",</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">Regards</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">Will</span></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 23 May 2016 at 12:57, Johanne Håøy Horn <span dir="ltr">
<<a href="mailto:johannhh@ifi.uio.no" target="_blank">johannhh@ifi.uio.no</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Hello again!
<div><br>
</div>
<div>I tried set the following in the JSON file:</div>
<div><br>
</div>
<div>
<div> {</div>
<div> "id": "1000genomes_phase3",</div>
<div> "species": "homo_sapiens",</div>
<div> "assembly": "GRCh38",</div>
<div> "type": "local",</div>
<div> "strict_name_match": 1,</div>
<div> "filename_template": "~/src/ensembl-vcf/",</div>
<div> "chromosomes": [</div>
<div> "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14",</div>
<div> "15", "16", "17", "18", "19", "20", "21", "22", "X", "Y"</div>
<div> ],</div>
<div> "sample_prefix": "1000GENOMES:phase_3:"</div>
<div> },</div>
<div><br>
</div>
<div>But I get this error message:</div>
<div>MSG: ERROR: VCF file ~/src/ensembl-vcf/ not found</div>
<div><br>
</div>
<div>I downloaded all the hg38 files you linked to in the folder ~/src/ensembl-vcf/. When you say that I need to change filename_template to the path where the files were downloaded, is it the full path of all the 48 files rather than the path to
the folder they are in?</div>
<div><br>
</div>
<div>Best,</div>
<div>Johanne</div>
<div><br>
</div>
<div>
<blockquote type="cite"><span>
<div>23. mai 2016 kl. 11.57 skrev Will McLaren <<a href="mailto:wm2@ebi.ac.uk" target="_blank">wm2@ebi.ac.uk</a>>:</div>
<br>
</span>
<div>
<div>
<div>
<div dir="ltr">Hi Johanne,
<div><br>
</div>
<div>It looks like the API is intermittently losing connection to the remote VCF files hosted on our FTP site.</div>
<div><br>
</div>
<div>You can bypass this connection by downloading the files to your local machine:</div>
<div><br>
</div>
<div>GRCh38: <a href="ftp://ftp.ensembl.org/pub/variation_genotype/homo_sapiens/" target="_blank">
ftp://ftp.ensembl.org/pub/variation_genotype/homo_sapiens/</a><br>
</div>
<div>GRCh37: <a href="ftp://ftp.ensembl.org/pub/grch37/release-82/variation/vcf/homo_sapiens/1000GENOMES-phase_3-genotypes/" target="_blank">
ftp://ftp.ensembl.org/pub/grch37/release-82/variation/vcf/homo_sapiens/1000GENOMES-phase_3-genotypes/</a><br>
</div>
<div><br>
</div>
<div>You will then need to edit [module_path]/ensembl-variation/modules/Bio/EnsEMBL/Variation/DBSQL/vcf_config.json, changing the "filename_template" entry to the path where you downloaded the files, and "type" from "remote" to "local".</div>
<div><br>
</div>
<div>Regarding the warning message, this should not affect your analyses in any way, but I have put in a fix on release/84 of ensembl-variation to suppress it.</div>
<div><br>
</div>
<div>Regards</div>
<div><br>
</div>
<div>Will McLaren</div>
<div>Ensembl Variation</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 21 May 2016 at 12:57, Johanne Håøy Horn <span dir="ltr">
<<a href="mailto:johannhh@ifi.uio.no" target="_blank">johannhh@ifi.uio.no</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Dear ensembl dev team,
<div><br>
</div>
<div>I have been using your variation API for some time now, and get a range of errors from time to time, without knowing exactly why. It is not because of the scripts, I think, as the same script producing the error can work just fine if I run it
again.</div>
<div><br>
</div>
<div>The different error messages are:</div>
<div>
<div><font face="Courier New">/Parser/BaseVCF4.pm line 891, <IN> line 5.</font></div>
<div><font face="Courier New">Use of uninitialized value in list assignment at /Users/Johanne/src/ensembl-io/modules/Bio/EnsEMBL/IO/Parser/BaseVCF4.pm line 891, <IN> line 5.</font></div>
<div><span style="white-space:pre-wrap"><font face="Courier New"></font></span></div>
<div><font face="Courier New">connect: Operation timed out</font></div>
<div><font face="Courier New">[kftp_connect_file] 350 Restarting at 654385206. Send STORE or RETRIEVE to initiate transfer</font></div>
<div><font face="Courier New"> </font></div>
<div><font face="Courier New">[kftp_connect_file] 227 Entering Passive Mode (193,62,203,85,220,250).</font></div>
<div><font face="Courier New">Tabix::tabix_query: t is not of type tabix_tPtr at /Users/Johanne/src/ensembl-io/modules/Bio/EnsEMBL/IO/TabixParser.pm line 70.</font></div>
<div><font face="Courier New"> </font></div>
<div><font face="Courier New">[kftp_connect_file] 227 Entering Passive Mode (193,62,203,85,157,134).</font></div>
<div><font face="Courier New">[main] fail to open the data file.</font></div>
<div><font face="Courier New">Can't use an undefined value as an ARRAY reference at /Users/Johanne/src/ensembl-io/modules/Bio/EnsEMBL/IO/Parser/BaseVCF4.pm line 730.</font></div>
</div>
<div><br>
</div>
<div>Usually just one of these error occur at a time. I suspect it might have something to do with the connection between my computer and the ensembl database, as the first error at least always show up in repeats when I lose my Internet connection.
However, are all of them caused by Internet trouble? I have checked that the MySQL instance is up and running, and can visit web pages through a browser when some of the errors occur. Could it be something on the server/database side?</div>
<div><br>
</div>
<div>Also, if I use the GRCh37 database:</div>
<div><br>
</div>
<div>
<div><font face="Courier New">$registry->load_registry_from_db(</font></div>
<div><font face="Courier New"> -host => '<a href="http://ensembldb.ensembl.org/" target="_blank">ensembldb.ensembl.org</a>',</font></div>
<div><font face="Courier New"> -user => 'anonymous',</font></div>
<div><font face="Courier New"> -port => 3337,</font></div>
<div><font face="Courier New">);</font></div>
</div>
<div><font face="Courier New"><br>
</font></div>
<div>I get this warning/printout:</div>
<div><font face="Courier New">Use of uninitialized value $nums{"."} in numeric comparison (<=>) at /Users/Johanne/src/ensembl-variation/modules/Bio/EnsEMBL/Variation/VCFCollection.pm line 770, <IN> line 6.</font></div>
<div><font face="Courier New"><br>
</font></div>
<div>I use version 84 of the Ensembl API, OS X 10.11.5, and the script I use when all of these errors occur, is attached. Note that the attached script by default uses hg38, but will produce the last printout mentioned when switching to hg37.</div>
<div><br>
</div>
<div>And something different I have been wondering about:</div>
<div>The VCF files that are downloaded locally (ALL.chr1.phase3_shapeit2_mvncall_integrated_v3plus_nounphased.rsID.genotypes.GRCh38_dbSNP.vcf.gz.tbi, for instance) - should they be deleted and re-downloaded from time to time to get the latest 1000G
data? And where exactly are the VCFs downloaded from? Is it dbSNP, as indicated in the file name?</div>
<div><br>
</div>
<div>Best,</div>
<div>Johanne Håøy Horn</div>
<div></div>
</div>
<br>
_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" rel="noreferrer" target="_blank">
http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" rel="noreferrer" target="_blank">
http://www.ensembl.info/</a><br>
<br>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">
http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" rel="noreferrer" target="_blank">
http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" rel="noreferrer" target="_blank">
http://www.ensembl.info/</a><br>
<br>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">
http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" rel="noreferrer" target="_blank">
http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" rel="noreferrer" target="_blank">
http://www.ensembl.info/</a><br>
<br>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank">
http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank">http://www.ensembl.info/</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div></div></div>
</div>
</div>
<br>_______________________________________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" rel="noreferrer" target="_blank">http://lists.ensembl.org/mailman/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" rel="noreferrer" target="_blank">http://www.ensembl.info/</a><br>
<br></blockquote></div><br></div>