<div dir="ltr">Thanks for the answer.<div><br></div><div>Indeed, I had some memory issues. But after memory is increased, I still get an error:</div><div><br></div><div>Loading the file</div><div><a href="ftp://ftp.ncbi.nlm.nih.gov/genomes/H_sapiens/GFF/ref_GRCh38.p12_top_level.gff3.gz" target="_blank" style="color:rgb(17,85,204);font-size:12.8px;background-color:rgb(255,255,255)">ftp://ftp.ncbi.nlm.nih.gov/<wbr>genomes/H_sapiens/GFF/ref_<wbr>GRCh38.p12_top_level.gff3.gz</a><br></div><div><br></div><div><div style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">I use the following perl script to do that:</div><div style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">ensembl-pipeline/scripts/<wbr>refseq_import/<a href="http://parse_ncbi_gff3.pl/" target="_blank" style="color:rgb(17,85,204)">parse_ncbi_gff3.<wbr>pl</a></div><br></div><div><br></div><div>Error Message:</div><div>
<span></span>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">No translation for transcript NR_038231.1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">No translation for transcript XR_002956503.1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">No translation for transcript NR_135168.1</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Can't call method "add_Transcript" on unblessed reference at /home/ensembl/release-92/ensembl-pipeline/scripts/refseq_import/<a href="http://parse_ncbi_gff3.pl">parse_ncbi_gff3.pl</a> line 965, <__ANONIO__> line 3695780.</span></p>
<br></div><div>The error message points to the last line of the GFF file which is ### (the end of the file). The error itself occures at the following code, while adding the transcript. $genes($parent_id) seems to have a diiferent type. It is always Bio::Ensembl::Gene expect in case of the error. Then it is HASH. Any ideas?</div><div><br></div><div>
<span></span>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(51,187,200);background-color:rgb(255,255,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">if</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> (</span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">exists</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> </span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">$genes{$parent_id}</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">) {</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(51,187,200);background-color:rgb(255,255,255)"><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">my</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> </span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">$stabid</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> = </span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">$transcript->stable_id</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">();</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(195,55,32);background-color:rgb(255,255,255)"><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">say</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">(</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">"Adding transcript "</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> . </span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)">$stabid</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> . </span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">" to gene "</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> . </span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)">$parent_id</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">) </span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">if</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> (</span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)">$verbose</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">);</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(51,187,200);background-color:rgb(255,255,255)"><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">$genes{$parent_id}-></span><span class="gmail-s5" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,166,179);background-color:rgb(230,230,0)">add_Trans</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">cript</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">(</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">$transcript</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">);</span></p>
<p class="gmail-p3" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>} </span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">else</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"> {</span></p>
<p class="gmail-p4" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(82,48,225);background-color:rgb(255,255,255)"><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"># this should never happen (edit: so why isn't it thrown?)</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(195,55,32);background-color:rgb(255,255,255)"><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">say</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">(</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">"Parent Gene not found for transcript: "</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> . </span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)">$transcripts{$k}{</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">stable_id</span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)">}</span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">);</span></p>
<p class="gmail-p3" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">next</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"> TRANSCRIPT;</span></p>
<p class="gmail-p3" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span></span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)">$tcnt</span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">--;</span></p>
<p class="gmail-p3" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:19px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(255,255,255)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>}</span></p>
<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 4, 2018 at 3:23 PM, Thomas Danhorn <span dir="ltr"><<a href="mailto:danhornt@njhealth.org" target="_blank">danhornt@njhealth.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I doubt this is a problem with the script per se, but rather with the configuration of the machine it runs on (and therefore more a job for your sysadmin than for the Ensembl team).<br>
<br>
I am just speculating, but here is my best guess:<br>
<br>
- The script is growing a hash whenever it finds a slice (see the code).<br>
- With that amount of memory required to hold the data grows as well.<br>
- If the import is sorted, it appears that the issue happens toward the end (chromosome X), i.e. when most of the data has already been read and stored in memory.<br>
- It is likely that your machine has a memory limit in place, which prevents processes to use more RAM than physically available. (One could use more, but that would necessitate swapping to disk, which ini turn essentially freezes the machine until done, which might be long time, therefore it is typically preferable on shared machines to just kill the process.)<br>
- The reason why your other organisms worked is most likely that they had less data to import (the human genome is large and well studied) and stayed under the memory limit.<br>
- A solution to this to find a machine/node with more memory and a higher limit.<br>
<br>
Hope this helps,<br>
<br>
Thomas<div><div class="h5"><br>
<br>
<br>
On Wed, 4 Jul 2018, Herzig, David wrote:<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Hi Ensembl Dev Team<br>
<br>
I have set up a mysql db containing homo_sapiens_core_92_38.<br>
<br>
After that I tried to import the NCBI file:<br>
<br>
<a href="ftp://ftp.ncbi.nlm.nih.gov/genomes/H_sapiens/GFF/ref_GRCh38.p12_top_level.gff3.gz" rel="noreferrer" target="_blank">ftp://ftp.ncbi.nlm.nih.gov/gen<wbr>omes/H_sapiens/GFF/ref_GRCh38.<wbr>p12_top_level.gff3.gz</a><br>
<br>
I use the following perl script to do that:<br>
<br>
ensembl-pipeline/scripts/refse<wbr>q_import/<a href="http://parse_ncbi_gff3.pl" rel="noreferrer" target="_blank">parse_ncbi_gff3.pl</a><br>
<br>
The process will be killed during the loading:<br>
<br>
Last output lines:<br>
***<br>
Slice NT_187386.1 found (scaffold:GRCh38:KI270731.1:1:<wbr>150754:1)<br>
Slice NT_187388.1 found (scaffold:GRCh38:KI270733.1:1:<wbr>179772:1)<br>
Slice NT_187389.1 found (scaffold:GRCh38:KI270734.1:1:<wbr>165050:1)<br>
Slice NC_000023.11 found (chromosome:GRCh38:X:1:1560408<wbr>95:1)<br>
Killed<br>
***<br>
<br>
Any ideas from your side?<br>
<br>
I did the same for other species and this works all fine.<br>
<br>
regards,<br>
David<br>
<br>
-- <br>
David Herzig<br>
Senior Scientist<br>
Roche Pharma Research and Early Development<br>
Roche Innovation Center Basel<br>
<br>
F. Hoffmann-La Roche Ltd<br>
Grenzacherstrasse 124<br>
4070 Basel<br>
Switzerland<br>
Phone +41 61 687 31 70<br>
<br></div></div>
Learn more about pRED Informatics at <a href="http://go.roche.com/*pREDi*" rel="noreferrer" target="_blank">http://go.roche.com/*pREDi*</a><br>
<<a href="http://go.roche.com/pREDi" rel="noreferrer" target="_blank">http://go.roche.com/pREDi</a>><br>
<br>
</blockquote>
<br>
NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.<br>
______________________________<wbr>_________________<br>
Dev mailing list <a href="mailto:Dev@ensembl.org" target="_blank">Dev@ensembl.org</a><br>
Posting guidelines and subscribe/unsubscribe info: <a href="http://lists.ensembl.org/mailman/listinfo/dev" rel="noreferrer" target="_blank">http://lists.ensembl.org/mailm<wbr>an/listinfo/dev</a><br>
Ensembl Blog: <a href="http://www.ensembl.info/" rel="noreferrer" target="_blank">http://www.ensembl.info/</a><br>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><font face="tahoma, sans-serif">David Herzig</font></div><div dir="ltr"><font face="tahoma, sans-serif">Senior Scientist<br>Roche Pharma Research and Early Development<br>Roche Innovation Center Basel<br><br>F. Hoffmann-La Roche Ltd<br>Grenzacherstrasse 124<br>4070 Basel<br>Switzerland<br>Phone +41 61 687 31 70</font><p style="font-family:arial;font-size:small"><span lang="DE" style="font-family:Imago"><span style="font-family:arial,sans-serif;color:black;font-size:10pt"><font face="tahoma, sans-serif">Learn more about pRED Informatics at </font></span><a href="http://go.roche.com/pREDi" style="color:rgb(17,85,204);font-family:arial,sans-serif;font-size:13.3333339691162px" target="_blank"><span style="font-size:12.8000001907349px">http://go.roche.com/</span><b style="font-size:12.8000001907349px">pREDi</b></a><br></span></p></div></div></div></div></div></div></div></div></div>
</div>