[QFJ-119] No JDBC transactions around storing messages and updating
sequence number. Created: 28/Dec/06 Updated: 20/Dec/11 Resolved: 09/Jan/07
Status:
Project:
Component/s:
Affects
Version/s:
Fix Version/s:
Closed
QuickFIX/J
Engine
None
Type:
Reporter:
Resolution:
Labels:
Other
Paul Haefele
Fixed
None
None
Priority:
Assignee:
Votes:
Default
Steve Bate
0
Description
I am curious as to the fact that their is no enclosing transaction around the storing of the
messages to the database and the updating of the sequence numbers that happens in
Session.sendRaw(). In here, QuickFixJ (if configured for JDBC) updates the database after it
has send out the message using:
state.set(msgSeqNum, messageString);
state.incrNextSenderMsgSeqNum();
Both of these update the database, however they don't do it in a single transaction. Doesn't this
mean the DB would be in an inconsistent state if the server where to go down between the
above two statements.
Indeed I was hoping to be able to hook into this transaction myself to update my own tables so
as to ensure that my OMS is always inline with what QuickFix believes it has sent.
I am a relative newbie to QuickFixJ, so please let me know if I am missing the boat entirely.
Many thanks
Paul
Comments
Comment by Paul Haefele [ 28/Dec/06 ]
I am curious as to the fact that their is no enclosing transaction around the storing of the
messages to the database and the updating of the sequence numbers that happens in
Session.sendRaw(). In here, QuickFixJ (if configured for JDBC) updates the database after it
has send out the message using:
state.set(msgSeqNum, messageString);
state.incrNextSenderMsgSeqNum();
Both of these update the database, however they don't do it in a single transaction. Doesn't this
mean the DB would be in an inconsistent state if the server where to go down between the
above two statements.
Indeed I was hoping to be able to hook into this transaction myself to update my own tables so
as to ensure that my OMS is always inline with what QuickFix believes it has sent.
I am a relative newbie to QuickFixJ, so please let me know if I am missing the boat entirely.
Many thanks
Paul
Comment by Paul Haefele [ 28/Dec/06 ]
Sorry for the duplicate message above - meant to send an additional comment (below) but ended
up re-pasting original...
Looking further at Session.sendRaw() I see that it is doing two things:
a) sending message on network (Session.send() and then
b) logging this message to the database (state.set())
However I'm wondering about the failure semantics here of what would happen if our FixServer
went down between (a) and (b). This could mean that the receiving Fix engine could get a
message that we don't know about when restart. Wouldn't this lead us to sending a different
message with the an already sent MessageID on a restart. I would have expected that (a) and (b)
would have been reversed.
Many thanks,
Paul
Comment by Steve Bate [ 09/Jan/07 ]
Hi Paul,
Quickfix and Quickfix/J have no transactional guarantees. This would be an interesting
extension. Remember, that most of the messages stores and logs are not themselves
transactional. However, the interfaces could be extended to support the potential for this
behavior even if it's a no-op for most implementations.
If the rare event (hopefully!) that a server crash happens between the two calls you mention, it
may require some manual intervention to patch up sequence numbers or resend messages. I
think if the calls are reversed it possibly lead to a dropped message, depending on exactly where
the crash occurs. However, I'll look into it further.
Comment by Paul Haefele [ 10/Jan/07 ]
Hi Steve,
Thanks for this reply. In our use of quickfixj, we want to expose the underlying transaction that
jdbc will be using. This is so our own DB update can be transactionally guaranteed to occur
when quickfix has transactionally committed (i.e. written to its internal store.) I've looked at
Session, SessionState and MessageStore (and its implementers) and think I can expose any
underlying transaction (assuming there is one) to the Application in a clean and minimal
manner.
Indeed I've already implemented this, but want to review (and test) it further. I'm happy to email
you the code, and if you feel it will be useful and is implemented correctly, to have this
committed to the quickfixj code base.
In terms of the ordering, my sense is that mina is not a guaranteed delivery mechanism, and that
FIX itself guarantees this via its message sequence numbers. As such a dropped message is no
big issue. If we write to the store and then crash (without sending the message), when we come
back up and send another message, the counterparty will simply ask for the missed message
which we've stored.
In the case where we send the message first (and let us say it arrives) and then crash before
saving to our own store, the next message we send would have the same sequence number, but
indeed be a different message. We would not know what message was send prior to the crash.
Thus updating our own internal store before sending any message is I believe the preferable way
to go.
In any event, thanks for the reply, and most impressed with the way QuickFixJ works - great
stuff. I've been working with a commercial OMS that exposes Fix messages, and the quickfixJ
implementation is far more consistent and clean.
Thanks again,
Paul
Comment by shaun barker [ 20/Dec/11 ]
Hi
I realise that this is closed, but we recently came upon the same problem, came to the same
conclusions and also have the code available. If you want I am happy to ping up the changes? let
me know if so. Otherwise - thanks for the great product!
Generated at Tue Feb 09 19:03:49 CET 2016 using JIRA 7.0.10#70120sha1:37e3d7a6fc4d580639533e7f7c232c925e554a6a.