<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Assier,<div class=""><br class=""></div><div class="">I’d like to get the the list of variants with data issues. However, I would be even more interested in the ones that fail because of hitting a memory limit if that is possible.</div><div class=""><br class=""></div><div class="">Thank you,</div><div class="">Anja</div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 21 Aug 2019, at 16:16, Asier Gonzalez <<a href="mailto:gonzaleza@ebi.ac.uk" class="">gonzaleza@ebi.ac.uk</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Hi Anja,</p><p class="">I understand that there may be some data issues, which is why I
capture 400 errors and flag them with a message in the output. I
will ammend the code to retry when 400 errors occur as per your
suggestion.</p><p class="">Just to finish, do you want me to send you a list of the variants
that seem to have data issues?</p><p class="">Thank you,<br class="">
Asier<br class="">
</p>
<div class="moz-cite-prefix">On 21/08/2019 16:08, Anja Thormann
wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:0FC6F8DC-7C3A-46E5-93CE-35955F8D6977@ebi.ac.uk" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
Hi Asier,
<div class=""><br class="">
</div>
<div class="">
<div class="">this is helpful. Apart from the memory errors you
are actually seeing genuine problems with the input data. For
the example variant you have sent we only have missing
information and VEP cannot calculate any consequences with the
incomplete data. In our production pipeline we are more
permissive and try to calculate consequences even with
incomplete data. As a side note for our next release
rs1057518506 has all required alleles present and VEP will be
able to calculate consequences for it.</div>
<div class=""><br class="">
</div>
<div class="">To conclude, looping a few times over variants
with 400 errors seems to be the way to go for now. We will
look into catching memory limit errors better and return a
more appropriate error code. But keep in mind that there are
variants for which VEP cannot caclulate consequences for
example because of missing or wrong allele or location
information.</div>
<div class=""><br class="">
</div>
<div class="">Best,</div>
<div class="">Anja</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On 21 Aug 2019, at 15:16, Asier Gonzalez
<<a href="mailto:gonzaleza@ebi.ac.uk" class="" moz-do-not-send="true">gonzaleza@ebi.ac.uk</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Hi Anja,</p><p class="">This is the part of the code that manages
the queries:
<a class="moz-txt-link-freetext" href="https://github.com/opentargets/snp_to_gene/blob/master/snp_assignment.pl#L153-L179" moz-do-not-send="true">https://github.com/opentargets/snp_to_gene/blob/master/snp_assignment.pl#L153-L179</a>.
If you look further down you will see that it cannot
be a 429 code as they are handled explicitly. I
could add repeats in case of 400 codes if you think
that it is the best solution. I have asked about the
memory allocation error because that is the one that
gets resolved if I try again but there are around 50
variants that always give a 400 code even though
they exist in Ensembl. One example is rs1057518506
which is a <a moz-do-not-send="true" href="https://www.ncbi.nlm.nih.gov/snp/rs1057518506" class="">deletion according to dbSNP</a>, is
described as a <a moz-do-not-send="true" href="https://www.ensembl.org/Homo_sapiens/Variation/Explore?db=core;r=17:7220304-7221313;v=rs1057518506;vdb=variation;vf=371667648" class="">frameshift variant in Ensembl but it
lacks alleles</a> and the <a moz-do-not-send="true" href="https://rest.ensembl.org/vep/human/id/rs1057518506?content-type=application/json" class="">API returns a 400 code saying that the
length of the reference allele is 0</a>. I am
happy to share these cases with you if you are
interested in them.<br class="">
</p><p class="">I understand that using the POST endpoints
would be a better solution, at least because we
could use a single query to retrieve data about
multiple variants thus reducing the burden on the
API. However,I am afraid that this is a piece of
code that we run twice every two months and it is
not a priority for us to refactor it unless you have
a good reason for me to convince my managers.</p><p class="">Thank you,<br class="">
Asier<br class="">
</p>
<div class="moz-cite-prefix">On 21/08/2019 14:54, Anja
Thormann wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:C2007FF7-7E3A-4349-8E3A-99A50B8C0E08@ebi.ac.uk" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
Hi Asier,
<div class=""><br class="">
</div>
<div class="">I would like to take a look at your
script please. I recommend for the first part that
you use our VEP POST endpoints for <a href="https://rest.ensembl.org/documentation/info/vep_region_post" class="" moz-do-not-send="true">region</a> and <a href="https://rest.ensembl.org/documentation/info/vep_id_post" class="" moz-do-not-send="true">id</a>. At this point I
recommend that you rerun your requests a few times
on a 400 error and if the requests keep failing
contact us with details (variant id or region) of
your failed requests. Can you rule out that the
error message has a 429 code due to too many
requests?</div>
<div class=""><br class="">
</div>
<div class="">Thank you,</div>
<div class="">Anja<br class="">
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 21 Aug 2019, at 14:28,
Asier Gonzalez <<a href="mailto:gonzaleza@ebi.ac.uk" class="" moz-do-not-send="true">gonzaleza@ebi.ac.uk</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Hello Anja,<br class="">
<br class="">
Thank you for your email. This is an
script that calls both the id
(/vep/human/id/) and region
(/vep/human/region/) endpoints depending
on whether the variant is defined by a
rsID or a genomic coordinates. It calls
the API thousands of times using GET
requests, once per variant. I don't see
any other settings but I can point you
to the few code lines that control it on
Github if you want to have a look
yourself.<br class="">
<br class="">
Please let me know if I can help. I just
need to know whether this is an expected
behaviour as our script retries calling
the API if the response is a 5XX error
but it passes if it's a 400 as it should
have been caused by the query and
retrying should not make any difference.<br class="">
<br class="">
Best wishes,<br class="">
Asier<br class="">
<br class="">
On 21/08/2019 13:39, Anja Thormann
wrote:<br class="">
<blockquote type="cite" class="">Dear
Asier,<br class="">
<br class="">
thank you for your feedback. Could you
please let me know which VEP endpoint
and settings you use? If you are using
a POST endpoint, how many variant ids
or regions are you sending? I assume
that the problem happens at a point
during the VEP calculation where it is
difficult to differentiate the cause
of the problem. But we will
investigate further and hopefully be
able to provide a more accurate error
report.<br class="">
<br class="">
Best wishes,<br class="">
Anja<br class="">
<br class="">
<blockquote type="cite" class="">On 21
Aug 2019, at 12:47, Asier Gonzalez
<<a href="mailto:gonzaleza@ebi.ac.uk" class="" moz-do-not-send="true">gonzaleza@ebi.ac.uk</a>>
wrote:<br class="">
<br class="">
Hi,<br class="">
<br class="">
I have an script that calls the VEP
API for a few thousand variants and
does some processing. The script
captures HTTP errors and messages
and I have found a few cases where
the 400 error is accompanied by a
"ERROR: Cannot allocate memory"
message. If I query the API again
with the ids that produced that
error I get a results so I
understand that this is a temporary
issue. I could handle the 400 errors
further to control these cases but I
wonder if it is an expected case or
if it an issue as it sounds as this
should be an 5XX error instead of a
400.<br class="">
<br class="">
Kind regards,<br class="">
Asier<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
Dev mailing list <a href="mailto:Dev@ensembl.org" class="" moz-do-not-send="true">Dev@ensembl.org</a><br class="">
Posting guidelines and
subscribe/unsubscribe info: <a href="https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org" class="" moz-do-not-send="true">https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org</a><br class="">
Ensembl Blog: <a href="http://www.ensembl.info/" class="" moz-do-not-send="true">http://www.ensembl.info/</a><br class="">
</blockquote>
<br class="">
_______________________________________________<br class="">
Dev mailing list <a href="mailto:Dev@ensembl.org" class="" moz-do-not-send="true">Dev@ensembl.org</a><br class="">
Posting guidelines and
subscribe/unsubscribe info: <a href="https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org" class="" moz-do-not-send="true">https://lists.ensembl.org/mailman/listinfo/dev_ensembl.org</a><br class="">
Ensembl Blog: <a href="http://www.ensembl.info/" class="" moz-do-not-send="true">http://www.ensembl.info/</a><br class="">
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
</div></blockquote></div><br class=""></div></body></html>