<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Kurt,<div class=""><br class=""></div><div class="">No problem- I’m glad to see that you have been able to resolve the issue downloading files from the FTP site from your end. Please do get back in touch if you continue to experience any problems.<br class=""><div class=""><br class=""></div><div class="">Best wishes</div><div class=""><br class=""></div><div class="">Ben<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 31 Jan 2019, at 20:00, Kurt Wheeler <<a href="mailto:kurt.wheeler91@gmail.com" class="">kurt.wheeler91@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Sorry!!!<div class=""><br class=""></div><div class="">So I didn't realize that when you use the `--spider` flag, wget will return a status code of 8 for some reason. This means that my script was backwards and I was seeing success ~66% of the time. However, I think the failures may have been related to a networking configuration we had set up. We corrected this and now it seems like we can reliable pull this file.</div><div class=""><br class=""></div><div class="">Sorry again for any confusion or distraction this caused. If it ends up being a problem again with the network configuration changes we made, I'll follow back up.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class=""><br class=""></div><div class="">- Kurt</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 31, 2019 at 2:24 PM Kurt Wheeler <<a href="mailto:kurt.wheeler91@gmail.com" class="">kurt.wheeler91@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class="">Hello again,<br class=""><div class=""><br class=""></div><div class="">I wrote a small script to make sure that I'm not just getting unlucky with when I try to hit it. This is the script I used:<br class=""><br class=""><div class=""><br class=""></div><div class="">#!/bin/bash</div><div class=""><br class=""></div><div class="">while true; do</div><div class=""> if $(wget --spider --timeout=30 --tries=1 "<a href="ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz" target="_blank" class="">ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz</a>"); then</div><div class=""> status="up"</div><div class=""> else</div><div class=""> status="down"</div><div class=""> fi</div><div class=""> echo "$(date), $status" >> ensembl_statuses.csv</div><div class=""> sleep 300</div><div class="">done</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">And here is the output for almost 3 hours:<br class=""><br class=""><div class="">Thu Jan 31 10:26:05 EST 2019, up</div><div class="">Thu Jan 31 10:47:00 EST 2019, up</div><div class="">Thu Jan 31 10:54:26 EST 2019, down</div><div class="">Thu Jan 31 10:59:28 EST 2019, down</div><div class="">Thu Jan 31 11:04:29 EST 2019, down</div><div class="">Thu Jan 31 11:22:58 EST 2019, down</div><div class="">Thu Jan 31 11:25:03 EST 2019, down</div><div class="">Thu Jan 31 11:30:05 EST 2019, down</div><div class="">Thu Jan 31 11:35:06 EST 2019, down</div><div class="">Thu Jan 31 11:40:37 EST 2019, down</div><div class="">Thu Jan 31 11:46:08 EST 2019, up</div><div class="">Thu Jan 31 11:51:10 EST 2019, down</div><div class="">Thu Jan 31 11:56:41 EST 2019, down</div><div class="">Thu Jan 31 12:02:12 EST 2019, down</div><div class="">Thu Jan 31 12:07:43 EST 2019, down</div><div class="">Thu Jan 31 12:13:14 EST 2019, down</div><div class="">Thu Jan 31 12:18:45 EST 2019, down</div><div class="">Thu Jan 31 12:24:16 EST 2019, up</div><div class="">Thu Jan 31 12:29:48 EST 2019, up</div><div class="">Thu Jan 31 12:35:19 EST 2019, down</div><div class="">Thu Jan 31 12:40:49 EST 2019, down</div><div class="">Thu Jan 31 12:46:20 EST 2019, down</div><div class="">Thu Jan 31 12:51:52 EST 2019, down</div><div class="">Thu Jan 31 12:57:23 EST 2019, up</div><div class="">Thu Jan 31 13:02:54 EST 2019, up</div><div class="">Thu Jan 31 13:08:25 EST 2019, down</div><div class="">Thu Jan 31 13:13:56 EST 2019, up</div><div class="">Thu Jan 31 13:19:27 EST 2019, down</div><div class="">Thu Jan 31 13:24:29 EST 2019, down</div><div class="">Thu Jan 31 13:29:30 EST 2019, down</div><div class="">Thu Jan 31 13:35:01 EST 2019, down</div><div class="">Thu Jan 31 13:40:32 EST 2019, up</div><div class="">Thu Jan 31 13:46:03 EST 2019, down</div><div class="">Thu Jan 31 13:51:34 EST 2019, down</div><div class="">Thu Jan 31 13:57:05 EST 2019, down</div><div class="">Thu Jan 31 14:02:36 EST 2019, down</div><div class="">Thu Jan 31 14:08:07 EST 2019, up</div><div class="">Thu Jan 31 14:13:09 EST 2019, down</div><div class="">Thu Jan 31 14:18:40 EST 2019, down</div><br class=""><br class="">As you can see, only about 34% of the requests I've made to it have been successful.<br class=""><br class="">Hope this helps,</div><div class=""><br class=""></div><div class="">- Kurt</div></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 31, 2019 at 10:21 AM Kurt Wheeler <<a href="mailto:kurt.wheeler91@gmail.com" target="_blank" class="">kurt.wheeler91@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class="">Hi Ben,<div class=""><br class=""></div><div class="">It came back this morning:</div><div class=""><br class=""></div><div class=""><div class="">2019-01-31 15:08:44,914 local [volume: 0] data_refinery_workers.downloaders.transcriptome_index DEBUG [downloader_job: 1]: Downloading file from <a href="ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz" target="_blank" class="">ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz</a> to /home/user/data_store/Caenorhabditis_elegans/Caenorhabditis_elegans.fa.gz.</div><div class="">2019-01-31 15:10:55,982 local [volume: 0] data_refinery_workers.downloaders.transcriptome_index ERROR [downloader_job: 1]: Exception caught while downloading file from: <a href="ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz" target="_blank" class="">ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz</a></div><div class="">Traceback (most recent call last):</div><div class=""> File "/usr/lib/python3.5/urllib/request.py", line 1478, in ftp_open</div><div class=""> fp, retrlen = fw.retrfile(file, type)</div><div class=""> File "/usr/lib/python3.5/urllib/request.py", line 2347, in retrfile</div><div class=""> conn, retrlen = self.ftp.ntransfercmd(cmd)</div><div class=""> File "/usr/lib/python3.5/ftplib.py", line 360, in ntransfercmd</div><div class=""> source_address=self.source_address)</div><div class=""> File "/usr/lib/python3.5/socket.py", line 711, in create_connection</div><div class=""> raise err</div><div class=""> File "/usr/lib/python3.5/socket.py", line 702, in create_connection</div><div class=""> sock.connect(sa)</div><div class="">TimeoutError: [Errno 110] Connection timed out</div><div class=""><br class=""></div><div class="">During handling of the above exception, another exception occurred:</div><div class=""><br class=""></div><div class="">Traceback (most recent call last):</div><div class=""> File "/home/user/data_refinery_workers/downloaders/transcriptome_index.py", line 42, in _download_file</div><div class=""> with closing(urllib.request.urlopen(download_url)) as request:</div><div class=""> File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen</div><div class=""> return opener.open(url, data, timeout)</div><div class=""> File "/usr/lib/python3.5/urllib/request.py", line 466, in open</div><div class=""> response = self._open(req, data)</div><div class=""> File "/usr/lib/python3.5/urllib/request.py", line 484, in _open</div><div class=""> '_open', req)</div><div class=""> File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain</div><div class=""> result = func(*args)</div><div class=""> File "/usr/lib/python3.5/urllib/request.py", line 1489, in ftp_open</div><div class=""> raise exc.with_traceback(sys.exc_info()[2])</div><div class=""> File "/usr/lib/python3.5/urllib/request.py", line 1478, in ftp_open</div><div class=""> fp, retrlen = fw.retrfile(file, type)</div><div class=""> File "/usr/lib/python3.5/urllib/request.py", line 2347, in retrfile</div><div class=""> conn, retrlen = self.ftp.ntransfercmd(cmd)</div><div class=""> File "/usr/lib/python3.5/ftplib.py", line 360, in ntransfercmd</div><div class=""> source_address=self.source_address)</div><div class=""> File "/usr/lib/python3.5/socket.py", line 711, in create_connection</div><div class=""> raise err</div><div class=""> File "/usr/lib/python3.5/socket.py", line 702, in create_connection</div><div class=""> sock.connect(sa)</div><div class="">urllib.error.URLError: <urlopen error ftp error: TimeoutError(110, 'Connection timed out')></div><div class="">2019-01-31 15:10:56,025 local [volume: 0] data_refinery_workers.downloaders.utils INFO [downloader_job: 1] [failure_reason: Exception caught while downloading file from: <a href="ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz" target="_blank" class="">ftp://ftp.ensembl.org/pub/release-95/fasta/caenorhabditis_elegans/dna/Caenorhabditis_elegans.WBcel235.dna.toplevel.fa.gz</a>] [downloader_task: TRANSCRIPTOME_INDEX]: Downloader job failed!<br class=""><br class=""><br class=""></div></div><div class="">It's somewhat unfortunate that it was working when you tried it. We've been seeing it work occasionally, but recently we have been finding it be down more than it has been up. Do the servers have any monitoring set up?</div><div class=""><br class=""></div><div class="">Thanks,</div><div class=""><br class=""></div><div class="">- Kurt</div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 31, 2019 at 6:20 AM Ben Moore <<a href="mailto:bmoore@ebi.ac.uk" target="_blank" class="">bmoore@ebi.ac.uk</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Hi Kurt,<div class=""><br class=""></div><div class="">I’ve tried to replicate the error, but have been able to download the <i class="">C. elegans</i> GTF file from the FTP server. It is possible that you have encountered intermittent errors, but we believe that there are no issues currently affecting the FTP server. Please try again, and get back in touch if you continue to experience the same issues.</div><div class=""><br class=""></div><div class="">We apologise for any inconvenience you’ve experienced.</div><div class=""><div class=""><br class=""></div><div class="">Best wishes</div><div class=""><br class=""></div><div class="">Ben<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 30 Jan 2019, at 19:06, Kurt Wheeler <<a href="mailto:kurt.wheeler91@gmail.com" target="_blank" class="">kurt.wheeler91@gmail.com</a>> wrote:</div><br class="gmail-m_6495845816236427037gmail-m_-8020691285054179569gmail-m_-7937476832506945184gmail-m_-1498981642396525237Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><div dir="ltr" class="">Hello,<div class=""><br class=""></div><div class="">My application has been having trouble downloading files from the Ensembl FTP server. One specific file I have had trouble downloading is:</div><div class=""><br class=""></div><div class=""> <a href="ftp://ftp.ensembl.org/pub/release-95/gtf/caenorhabditis_elegans/Caenorhabditis_elegans.WBcel235.95.gtf.gz" target="_blank" class="">ftp://ftp.ensembl.org/pub/release-95/gtf/caenorhabditis_elegans/Caenorhabditis_elegans.WBcel235.95.gtf.gz</a></div><div class=""><br class=""></div><div class="">However, I also am usually unable to even view a directory such as:</div><div class=""><br class=""></div><div class=""> <a href="ftp://ftp.ensembl.org/pub/release-95/gtf/" target="_blank" class="">ftp://ftp.ensembl.org/pub/release-95/gtf/</a></div><div class=""><br class=""></div><div class="">This makes me think the FTP server has not been running for large amounts of time. While I was writing this email it seems to have come back up, but a colleague of mine and I have observed it coming up for small periods of time over the last week or so while remaining generally unreachable.</div><div class=""><br class=""></div><div class="">Is anyone able to look into this?</div><div class=""><br class=""></div><div class="">Thanks,</div><div class=""><br class=""></div><div class="">- Kurt</div></div></div></div>
_______________________________________________<br class="">Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank" class="">Dev@ensembl.org</a><br class="">Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" target="_blank" class="">http://lists.ensembl.org/mailman/listinfo/dev</a><br class="">Ensembl Blog: <a href="http://www.ensembl.info/" target="_blank" class="">http://www.ensembl.info/</a><br class=""></div></blockquote></div><br class=""><div class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><div class="">Ben Moore</div><div class="">Ensembl Outreach Officer</div><div class=""><br class=""></div><div class="">European Bioinformatics Institute (EMBL-EBI)</div><div class="">European Molecular Biology Laboratory</div><div class="">Wellcome Trust Genome Campus</div><div class="">Hinxton</div><div class="">Cambridge</div><div class="">CB10 1SD</div><div class="">UK</div><div class=""><br class=""></div><div class=""><a href="mailto:bmoore@ebi.ac.uk" target="_blank" class="">bmoore@ebi.ac.uk</a></div><div class="">+44 (0)1223 494265</div></div></div>
</div>
<br class=""></div></div></div>_______________________________________________<br class="">
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank" class="">Dev@ensembl.org</a><br class="">
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" rel="noreferrer" target="_blank" class="">http://lists.ensembl.org/mailman/listinfo/dev</a><br class="">
Ensembl Blog: <a href="http://www.ensembl.info/" rel="noreferrer" target="_blank" class="">http://www.ensembl.info/</a><br class="">
</blockquote></div>
</blockquote></div>
</blockquote></div>
_______________________________________________<br class="">Dev mailing list <a href="mailto:Dev@ensembl.org" class="">Dev@ensembl.org</a><br class="">Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" class="">http://lists.ensembl.org/mailman/listinfo/dev</a><br class="">Ensembl Blog: <a href="http://www.ensembl.info/" class="">http://www.ensembl.info/</a><br class=""></div></blockquote></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Ben Moore</div><div class="">Ensembl Outreach Officer</div><div class=""><br class=""></div><div class="">European Bioinformatics Institute (EMBL-EBI)</div><div class="">European Molecular Biology Laboratory</div><div class="">Wellcome Trust Genome Campus</div><div class="">Hinxton</div><div class="">Cambridge</div><div class="">CB10 1SD</div><div class="">UK</div><div class=""><br class=""></div><div class=""><a href="mailto:bmoore@ebi.ac.uk" class="">bmoore@ebi.ac.uk</a></div><div class="">+44 (0)1223 494265</div></div></div>
</div>
<br class=""></div></div></body></html>