[OSM-dev] [OSM-talk] problem inserting nodes through the OSM API

[Shaun McDonald]

Moving to the dev mailing list.

What is the request that you are sending to the API?

Are you using the test API at http://api06.dev.openstreetmap.org/ rather than production for the testing of your app?

Shaun

Hi, I am currently testing a new POI editor I'm developing.
ActiveRecord::StatementInvalid: PGError: ERROR: insert or update on table "current_node_tags" violates foreign key constraint "current_node_tags_id_fkey"
DETAIL: Key (id)=(717587768) is not present in table "current_nodes".
: INSERT INTO "current_node_tags" ("k", "v", "id") VALUES(E'name', E'Caseratto', 717587768) RETURNING "id"
Does anyone know if it is a bug, or if there is any issue with adding POI through the API? If it is a bug, is there any workaround?
Thanks in advance,
Ignacio.
_______________________________________________
talk mailing list
talk at openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100502/cdbec5a7/attachment.html>

[IgnacioZ]

Hi I'm using real data for "testing" and I'm uploading the data to
production. Then I check it with Potlach and it's working fine for most
nodes, but sometimes I get the errors I mentioned (during the upload) and
then the nodes are not in the database.

These are three examples of XML that had the error:

<osm><node changeset="4588121" lat="-31.427919" lon="-64.187429"><tag
k="name" v="Caseratto"/><tag k="amenity" v="bar"/></node></osm>

<osm><node changeset="4588121" lat="-31.428072" lon="-64.187143"><tag
k="name" v="Palestra"/><tag k="type" v="climbing"/></node></osm>

<osm><node changeset="4588121" lat="-31.428261" lon="-64.186582"><tag
k="name" v="Beto's"/><tag k="amenity" v="fast_food"/></node></osm>

If you check out that changeset you will find that there are other nodes
that were uploaded without problems.


Thanks,
Ignacio.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100502/fa4cbc4e/attachment.html>

[IgnacioZ]

Hi I'm a bit lost here. For me this looks like a bug: it's trying to insert
a tuple that represents the tag of a node before inserting the node (at
least that looks like).

I would like to know if there is a workaround, or where should I report the
bug?

If you need some help fixing it, let me know. Who maintains the
code/scripts?

Regards,
Ignacio.

Here I copy the error again:

ActiveRecord::StatementInvalid: PGError: ERROR: insert or update on table
"current_node_tags" violates foreign key constraint
"current_node_tags_id_fkey"
DETAIL: Key (id)=(717587768) is not present in table "current_nodes".
: INSERT INTO "current_node_tags" ("k", "v", "id") VALUES(E'name',
E'Caseratto', 717587768) RETURNING "id"

Post by IgnacioZ
Hi I'm using real data for "testing" and I'm uploading the data to
production. Then I check it with Potlach and it's working fine for most
nodes, but sometimes I get the errors I mentioned (during the upload) and
then the nodes are not in the database.
<osm><node changeset="4588121" lat="-31.427919" lon="-64.187429"><tag
k="name" v="Caseratto"/><tag k="amenity" v="bar"/></node></osm>
<osm><node changeset="4588121" lat="-31.428072" lon="-64.187143"><tag
k="name" v="Palestra"/><tag k="type" v="climbing"/></node></osm>
<osm><node changeset="4588121" lat="-31.428261" lon="-64.186582"><tag
k="name" v="Beto's"/><tag k="amenity" v="fast_food"/></node></osm>
If you check out that changeset you will find that there are other nodes
that were uploaded without problems.
Thanks,
Ignacio.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100503/12c0d186/attachment.html>

[Peter Körner]

Post by IgnacioZ
I would like to know if there is a workaround, or where should I report
the bug?

You can report it at http://trac.openstreetmap.org/ but you still should
get an answer from an api maintainer here :)

Peter

Post by IgnacioZ
ActiveRecord::StatementInvalid: PGError: ERROR: insert or update on
table "current_node_tags" violates foreign key constraint
"current_node_tags_id_fkey"
DETAIL: Key (id)=(717587768) is not present in table "current_nodes".
: INSERT INTO "current_node_tags" ("k", "v", "id") VALUES(E'name',
E'Caseratto', 717587768) RETURNING "id"
On Sun, May 2, 2010 at 4:53 PM, IgnacioZ <zignacio at gmail.com
Hi I'm using real data for "testing" and I'm uploading the data to
production. Then I check it with Potlach and it's working fine for
most nodes, but sometimes I get the errors I mentioned (during the
upload) and then the nodes are not in the database.
<osm><node changeset="4588121" lat="-31.427919"
lon="-64.187429"><tag k="name" v="Caseratto"/><tag k="amenity"
v="bar"/></node></osm>
<osm><node changeset="4588121" lat="-31.428072"
lon="-64.187143"><tag k="name" v="Palestra"/><tag k="type"
v="climbing"/></node></osm>
<osm><node changeset="4588121" lat="-31.428261"
lon="-64.186582"><tag k="name" v="Beto's"/><tag k="amenity"
v="fast_food"/></node></osm>
If you check out that changeset you will find that there are other
nodes that were uploaded without problems.
Thanks,
Ignacio.
_______________________________________________
dev mailing list
dev at openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

[Tom Hughes]

Post by Peter Körner

Post by IgnacioZ
I would like to know if there is a workaround, or where should I report
the bug?

You can report it at http://trac.openstreetmap.org/ but you still should
get an answer from an api maintainer here :)

I don't see why he should get a response here (or indeed anywhere) as
that suggests that we have paid api maintainers whose job it would be to
respond rather than volunteers who help out as and when they wish to.

The correct place to report bugs is trac, and getting an FK constraint
violation is pretty much automatically a bug, although probably only one
that is being triggered by a malformed request of some sort.

IgnacioZ - please file a bug in trac, but when you do please indicate
what URL you are calling and what HTTP method you are using, as well as
what XML you are sending.

Tom

[IgnacioZ]

Hello, if you can point me to the place where the code is, I could help fix
it.

There's no one maintaining that code?

Regards,
Ignacio.

Post by IgnacioZ
I would like to know if there is a workaround, or where should I report

Post by IgnacioZ
the bug?

You can report it at http://trac.openstreetmap.org/ but you still should
get an answer from an api maintainer here :)

I don't see why he should get a response here (or indeed anywhere) as that
suggests that we have paid api maintainers whose job it would be to respond
rather than volunteers who help out as and when they wish to.
The correct place to report bugs is trac, and getting an FK constraint
violation is pretty much automatically a bug, although probably only one
that is being triggered by a malformed request of some sort.
IgnacioZ - please file a bug in trac, but when you do please indicate what
URL you are calling and what HTTP method you are using, as well as what XML
you are sending.
Tom
--
Tom Hughes (tom at compton.nu)
http://compton.nu/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100504/b77cc830/attachment.html>

[Tom Hughes]

Post by IgnacioZ
Hello, if you can point me to the place where the code is, I could help
fix it.

How do you plan to do that before you know what the bug is?

Post by IgnacioZ
There's no one maintaining that code?

I didn't say that. What I said was that there is nobody obligated to fix
any particular bugs in any particular timescale.

That said, if there is a serious bug (which I doubt, as many people are
editing every day without any problem) then obviously it will probably
be fixed sooner rather than later.

My suspicion is that you have made a mistake in your request and that
the only bug is that we're not catching that at the application level
and hence the database is detecting the problem.

If you do what I asked and file a ticket in trac with the information I
requested, or even provide that information here, then I might actually
be able to tell you what the problem is.

Tom

Post by IgnacioZ
Regards,
Ignacio.
On Tue, May 4, 2010 at 4:46 AM, Tom Hughes <tom at compton.nu
I would like to know if there is a workaround, or where
should I report
the bug?
You can report it at http://trac.openstreetmap.org/ but you still should
get an answer from an api maintainer here :)
I don't see why he should get a response here (or indeed anywhere)
as that suggests that we have paid api maintainers whose job it
would be to respond rather than volunteers who help out as and when
they wish to.
The correct place to report bugs is trac, and getting an FK
constraint violation is pretty much automatically a bug, although
probably only one that is being triggered by a malformed request of
some sort.
IgnacioZ - please file a bug in trac, but when you do please
indicate what URL you are calling and what HTTP method you are
using, as well as what XML you are sending.
Tom
--
Tom Hughes (tom at compton.nu <mailto:tom at compton.nu>)
http://compton.nu/

[IgnacioZ]

Post by Tom Hughes
How do you plan to do that before you know what the bug is?

By looking at the code...

Post by Tom Hughes
If you do what I asked and file a ticket in trac with the information I

requested, or even provide that information here, then I might actually be
able to tell you what the problem is.

I will provide it here:

URL:
http://api.openstreetmap.org//api/0.6/node/create

HTTP Method: POST

XML:
<osm><node changeset="4588121" lat="-31.428261" lon="-64.186582"><tag
k="name" v="Beto's"/><tag k="amenity" v="fast_food"/></node></osm>

If you take a look at that changeset you will see that there are other
elements that where added correctly.


Ignacio.

Post by Tom Hughes
Hello, if you can point me to the place where the code is, I could help

Post by IgnacioZ
fix it.

How do you plan to do that before you know what the bug is?
There's no one maintaining that code?
I didn't say that. What I said was that there is nobody obligated to fix
any particular bugs in any particular timescale.
That said, if there is a serious bug (which I doubt, as many people are
editing every day without any problem) then obviously it will probably be
fixed sooner rather than later.
My suspicion is that you have made a mistake in your request and that the
only bug is that we're not catching that at the application level and hence
the database is detecting the problem.
If you do what I asked and file a ticket in trac with the information I
requested, or even provide that information here, then I might actually be
able to tell you what the problem is.
Tom
Regards,

Post by IgnacioZ
Ignacio.
On Tue, May 4, 2010 at 4:46 AM, Tom Hughes <tom at compton.nu
I would like to know if there is a workaround, or where
should I report
the bug?
You can report it at http://trac.openstreetmap.org/ but you still should
get an answer from an api maintainer here :)
I don't see why he should get a response here (or indeed anywhere)
as that suggests that we have paid api maintainers whose job it
would be to respond rather than volunteers who help out as and when
they wish to.
The correct place to report bugs is trac, and getting an FK
constraint violation is pretty much automatically a bug, although
probably only one that is being triggered by a malformed request of
some sort.
IgnacioZ - please file a bug in trac, but when you do please
indicate what URL you are calling and what HTTP method you are
using, as well as what XML you are sending.
Tom
--
Tom Hughes (tom at compton.nu <mailto:tom at compton.nu>)
http://compton.nu/

--
Tom Hughes (tom at compton.nu)
http://compton.nu/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100504/8407a7be/attachment.html>

[Tom Hughes]

Post by IgnacioZ

Post by Tom Hughes
How do you plan to do that before you know what the bug is?

By looking at the code...

Good luck with that...

Post by IgnacioZ

Post by Tom Hughes
If you do what I asked and file a ticket in trac with the information

I requested, or even provide that information here, then I might
actually be able to tell you what the problem is.
http://api.openstreetmap.org//api/0.6/node/create
HTTP Method: POST
<osm><node changeset="4588121" lat="-31.428261" lon="-64.186582"><tag
k="name" v="Beto's"/><tag k="amenity" v="fast_food"/></node></osm>
If you take a look at that changeset you will see that there are other
elements that where added correctly.

Am I right in suspecting that you were uploading multiple objects in
parallel by sending different requests on different connections?

Tom

[IgnacioZ]

They were done in a for loop and before I start the next request, I receive
the answer, though by looking at my code it could happen that two requests
are done in parallel, after receiving the Id of the previous one, I continue
and the previous connection may not have been finalized (they are different
instances).

Do you think that could be the reason ?

Post by Tom Hughes

Post by IgnacioZ

Post by Tom Hughes
How do you plan to do that before you know what the bug is?

By looking at the code...

Good luck with that...

Post by IgnacioZ
If you do what I asked and file a ticket in trac with the information
I requested, or even provide that information here, then I might
actually be able to tell you what the problem is.
http://api.openstreetmap.org//api/0.6/node/create
HTTP Method: POST
<osm><node changeset="4588121" lat="-31.428261" lon="-64.186582"><tag
k="name" v="Beto's"/><tag k="amenity" v="fast_food"/></node></osm>
If you take a look at that changeset you will see that there are other
elements that where added correctly.

Am I right in suspecting that you were uploading multiple objects in
parallel by sending different requests on different connections?
Tom
--
Tom Hughes (tom at compton.nu)
http://compton.nu/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100504/d78cbac6/attachment.html>

[Jan Sandbrink]

As far as I know the API (and I only know it from wiki-doc) it could
only be a problem if you are trying to upload a way, whose nodes are not
uploaded at that moment. So this should not happen in parallel.

Post by IgnacioZ
They were done in a for loop and before I start the next request, I
receive the answer, though by looking at my code it could happen that
two requests are done in parallel, after receiving the Id of the
previous one, I continue and the previous connection may not have been
finalized (they are different instances).
Do you think that could be the reason ?
On Tue, May 4, 2010 at 8:29 AM, Tom Hughes <tom at compton.nu

Post by Tom Hughes
How do you plan to do that before you know what the bug is?

By looking at the code...
Good luck with that...

Post by Tom Hughes
If you do what I asked and file a ticket in trac with the

information
I requested, or even provide that information here, then I might
actually be able to tell you what the problem is.
http://api.openstreetmap.org//api/0.6/node/create
HTTP Method: POST
<osm><node changeset="4588121" lat="-31.428261"
lon="-64.186582"><tag
k="name" v="Beto's"/><tag k="amenity" v="fast_food"/></node></osm>
If you take a look at that changeset you will see that there are other
elements that where added correctly.
Am I right in suspecting that you were uploading multiple objects
in parallel by sending different requests on different connections?
Tom
--
Tom Hughes (tom at compton.nu <mailto:tom at compton.nu>)
http://compton.nu/
_______________________________________________
dev mailing list
dev at openstreetmap.org
http://lists.openstreetmap.org/listinfo/dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100504/047a6297/attachment.html>

[Tom Hughes]

Post by IgnacioZ
They were done in a for loop and before I start the next request, I
receive the answer, though by looking at my code it could happen that
two requests are done in parallel, after receiving the Id of the
previous one, I continue and the previous connection may not have been
finalized (they are different instances).
Do you think that could be the reason ?

Once you get the ID back everything should be complete on the server so
if you were waiting that long there shouldn't be a problem. That isn't
what I'm seeing in the server logs though.

I have the request to add 717587768 starting:

[2010-05-02 19:02:06.985698 #9411] Processing NodeController#create (for
...) [PUT]

then the request add 717587767 starting:

[2010-05-02 19:02:06.998618 #9413] Processing NodeController#create (for
...) [PUT]

then they both deadlocked (probably against each other):

[2010-05-02 19:02:08.136362 #9413] Deadlock detected on retry 1,
restarting transaction
[2010-05-02 19:02:08.282322 #9411] Deadlock detected on retry 1,
restarting transaction

after which they both report FK violations.

Now there is presumably a bug in the way we restart the deadlocked
transaction that leads to the FK violations but I suspect it was the
parallel upload that led to the deadlock and that it explains why other
people don't see the same problems.

Tom

[IgnacioZ]

Ok thank you very much.

I will add some more POI today changing my code to wait for the new request
to be created when the previous one finished (which is not always exactly
the same as to when I get the response).

If you need some help with the code let me know. What language do you work
with ?

Regards,
Ignacio.

Post by IgnacioZ
They were done in a for loop and before I start the next request, I

Post by IgnacioZ
receive the answer, though by looking at my code it could happen that
two requests are done in parallel, after receiving the Id of the
previous one, I continue and the previous connection may not have been
finalized (they are different instances).
Do you think that could be the reason ?

Once you get the ID back everything should be complete on the server so if
you were waiting that long there shouldn't be a problem. That isn't what I'm
seeing in the server logs though.
[2010-05-02 19:02:06.985698 #9411] Processing NodeController#create (for
...) [PUT]
[2010-05-02 19:02:06.998618 #9413] Processing NodeController#create (for
...) [PUT]
[2010-05-02 19:02:08.136362 #9413] Deadlock detected on retry 1, restarting
transaction
[2010-05-02 19:02:08.282322 #9411] Deadlock detected on retry 1, restarting
transaction
after which they both report FK violations.
Now there is presumably a bug in the way we restart the deadlocked
transaction that leads to the FK violations but I suspect it was the
parallel upload that led to the deadlock and that it explains why other
people don't see the same problems.
Tom
--
Tom Hughes (tom at compton.nu)
http://compton.nu/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100504/b60bb342/attachment.html>

[Peter Körner]

Post by IgnacioZ
I will add some more POI today changing my code to wait for the new
request to be created when the previous one finished (which is not
always exactly the same as to when I get the response).

Can't this be done using the diff-upload? I could imagine that this is
less error prone in bulk uploads and maybe as well more efficient.

Peter

[IgnacioZ]

Yes I believe it can. I will try that. Thanks!

Ignacio.

Post by IgnacioZ
I will add some more POI today changing my code to wait for the new

request to be created when the previous one finished (which is not always
exactly the same as to when I get the response).

Can't this be done using the diff-upload? I could imagine that this is less
error prone in bulk uploads and maybe as well more efficient.
Peter

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstreetmap.org/pipermail/dev/attachments/20100504/a5e6b636/attachment.html>

[Andy Allan]

Post by IgnacioZ
Ok thank you very much.
I will add some more POI today changing my code to wait for the new request
to be created when the previous one finished (which is not always exactly
the same as to when I get the response).

If you are testing your application please do so against the following API:

http://api06.dev.openstreetmap.org/api

and not the main live api.

Cheers,
Andy

[Jan Sandbrink]

Perhaps one should add a hint to
http://wiki.openstreetmap.org/wiki/API_v0.6 that parallel uploads _may_
cause problems.
I could add this to the wiki if you want, but I would only do so if you
think this is useful.

Post by Tom Hughes
but I suspect it was the
parallel upload that led to the deadlock and that it explains why other
people don't see the same problems.

[Tom Hughes]

Post by Jan Sandbrink
Perhaps one should add a hint to
http://wiki.openstreetmap.org/wiki/API_v0.6 that parallel uploads _may_
cause problems.

Well they're not supposed to cause problems... As I said, that is most
likely a bug of some sort that we are currently looking into.

Tom