FIX Primer

FIX Primer (Optional)

Stop: If you are familiar with the FIX protocol you can most likely skip this section. The OneChronos FIX Spec contains everything that you need to integrate OneChronos. If you are looking for documentation on FIX itself, please proceed.

The FIX protocol is a transport agnostic text based protocol widely used by financial institutions to send orders, receive execution reports, and disseminate market data. There are many versions of FIX, and individual institutions usually customize a specific version of FIX for their use case. OneChronos FIX is based on the OneChronos FIX Spec.

FIX is a session based, asynchronous, guaranteed message delivery protocol. Sessions are bidirectional streams of messages between two parties that may arrive out-of-order, but will always be processed by the receiving application in-order. A single FIX session can span multiple physical connections; sessions are independent of network connects and disconnects. Message sequence numbers are used to ensure orderly message processing and recovery of lost messages.

FIX messages are broadly divided into two types: administrative, which control the FIX session itself, and business, which exchange information on the application level. The messages themselves are key/value pairs, delimited by the SOH character (ASCII #001, hex 0x01). Given that SOH is an unprintable character, we’ll use ”|” in its place when displaying messages. All FIX messages start with the Standard Message Header and end with the Standard Message Trailer. The set of key/value pairs between the header and the trailer (referred to as the Body) are message type dependent. The type of the message is always given by a tag in the header: MsgType (tag #35).

Message Format

Every FIX message is a stream of <tag>=<value> fields with a field delimiter (ASCII SOH) used between fields in the stream. Depending on the message type, some fields are required, others are conditionally required, and some are optional. All tags must have a value specified. Optional fields without values should not be included in the FIX message - the message will be rejected as malformed if they are. The general format of a FIX message is the standard header followed by the message body fields and terminated with the standard trailer.

Except where noted, fields within a message can be defined in any sequence. The relative position of a field within a message is inconsequential (although by convention they are displayed in ascending tag sorted order).

With the exception of data type fields discussed in data types FIX uses ASCII 128 encoding for all message content (tags, values, and delimiters).

Field Delimiter

All fields (including the last field in a message) are terminated by the ASCII SOH character (#001, hex 0x01). Given that this is an unprintable character, our documentation uses ”|” in place of SOH, but SOH should always be used in actual FIX messages.

Data Types

The following data type descriptions were taken from the Financial Information Exchange Protocol (FIX) Version 4.2 with Errata 20010501. Data types are mapped to ASCII strings as follows:

int: Sequence of digits without commas or decimals and optional sign character (ASCII characters ”-” and “0” - “9” ). The sign character utilizes one byte (i.e. positive int is “99999” while negative int is “-99999”). Note that int values may contain leading zeros (e.g. “00023”= “23”). Examples: 723 in field 21 would be mapped int as |21=723|. -723 in field 12 would be mapped int as |12=-723|
float: Sequence of digits with optional decimal point and sign character (ASCII characters ”-”, “0” - “9” and ”.”); the absence of the decimal point within the string will be interpreted as the float representation of an integer value. All float fields must accommodate up to fifteen significant digits. The number of decimal places used should be a factor of business/market needs and mutual agreement between counterparties. Note that float values may contain leading zeros (e.g. “00023.23” = “23.23”) and may contain or omit trailing zeros after the decimal point (e.g. “23.0” = “23.0000” = “23”).
qty: float field (see definition of “float” above) capable of storing either a whole number (no decimal places) of “shares” or a decimal value containing decimal places for non-share quantity asset classes.
price: float field (see definition of “float” above) representing a price. Note the number of decimal places may vary.
char: Single character value, can include any alphanumeric character or punctuation except the delimiter. All char fields are case sensitive (i.e. m ≠ M).
boolean: a char field (see definition of “char” above) containing one of two values:
- ‘Y’ = True/Yes
- ‘N’ = False/No
string: Alpha-numeric free format strings, can include any character or punctuation except the delimiter. All char fields are case sensitive (i.e. morstatt ≠̸ Morstatt).
multi_value_string: String field (see definition of “String” above) containing one or more space delimited values.
exchange: String field (see definition of “String” above) representing a market or exchange.
utctimestamp: Time/date combination represented in UTC (Universal Time Coordinated, also known as “GMT”) in either YYYYMMDD-HH:MM:SS (whole seconds) or YYYYMMDD-HH:MM:SS.sss (milliseconds) format, colons, dash, and period required. Valid values:
- YYYY = 0000-9999, MM = 01-12, DD = 01-31, HH = 00-23, MM = 00-59, SS = 00-5960 (60 only if UTC leap second*) (without milliseconds).
- YYYY = 0000-9999, MM = 01-12, DD = 01-31, HH = 00-23, MM = 00-59, SS = 00-5960 (60 only if UTC leap second*), sss=000-999 (indicating milliseconds).

* See the International Earth Rotation Service (IERS) guide to leap seconds for additional details on leap second handling.

Checksum Calculation

FIX uses a weak checksum to protect against obvious forms message corruption. The CheckSum (tag #10) is the last field of any FIX message. It is computed via the following procedure:

Initialize an unsigned 32 bit integer (henceforth referred to as checksumTotal) to zero.
Iterate over every character in the FIX message up to, but not including, the checksum field, and add its integer value to checksumTotal.
Take the value of checksumTotal mod 256 and format it as a zero padded, three digit string.

In pseudocode:

char* GenerateCheckSum(char* buf, long bufLen)
{
    static char tmpBuf[4];
    long idx;
    unsigned int cks;

    for (idx = 0L, cks = 0; idx < bufLen; cks += (unsigned int)buf[idx++]) {
    }

    sprintf(tmpBuf, "%03d", (unsigned int)(cks % 256));
    return tmpBuf;
}

Message Delivery

FIX guarantees full, ordered, once-and-only-once processing (of application messages) between parties regardless of the network transport (e.g. TCP/IP, UDP) used. Formally, FIX provides bidirectional at-least-once message delivery semantics, and once-and-only-once processing semantics. A system of inbound and outbound sequence numbers tracked by both sides of the session dialogue makes this possible. Henceforth we will refer to the two parties in a FIX dialogue as P1 and P2. Note that P1’s outbound sequence is P2’s inbound sequence, and P1’s inbound sequence is P2’s outbound sequence. By tracking sequences and providing facilities for P1 to request that P2 resends messages (and vice versa) the protocol allows for asynchronous but orderly communication. Note that FIX uses an optimistic delivery model. Message receipt is never acknowledged on the protocol layer but message processing is (e.g. NewOrderSingle is acknowledged with an ExecutionReport indicating the order’s status).

Sequence Numbers

Every FIX message contains a sequence number - MsgSeqNum (tag #34) - in its header. Parties to a FIX session use these sequence numbers to ensure that messages arriving from their counterparty are processed in the correct order, and that no messages were lost.

At the beginning of a FIX session both parties initialize their “inbound” and “outbound” sequence numbers to one. When a message is sent, they increment their outbound counter. When a message is received, they check that the message sequence number is equal to their inbound sequence number. If it is, they increment their inbound sequence number and process the message. If it is not, the process described in ordered message processing is used to recover the dropped messages. Note that in both cases the inbound/outbound sequence numbers were read and the appropriate action performed before incrementing.

Example:

Start of session: P1 sets her inbound and outbound sequence number to one, as does P2
P1 sends P2 a Logon message with sequence number one. After sending, she increments her sequence number to two
P2 receives the Logon message from P1, noting that one is the expected sequence number at the start of the session. He sends a Logon message in acknowledgement. The sequence number on his response is one (he hasn’t sent anything yet). After sending, he increments his outbound sequence number to two
P1 receives the message, noting that one was the expected sequence number for P2’s response
Both P1 and P2 now have their inbound and outbound sequence numbers set to two. Going forward P1 and P2 will not always increment their sequence numbers in lock step. Some messages never need a reply (market data) and others may need several replies (multiple execution reports when an order is filled). Having P1 and P2 maintain independent inbound/outbound sequence numbers ensures that this is never a problem

Ordered Message Processing

FIX supports two options for processing out-of-order/dropped messages. In both cases, a ResendRequest as outlined in the section on recovery is used to initiate retransmission. Of the two options below, option one is preferable, but both are valid:

When a message arrives with a sequence number higher than the expected inbound sequence number…

Request the missing message(s) explicitly by sequence number while placing new arrivals into to a priority queue sorted by message sequence number. When the missing message is received as a retransmission, process it, process the queue, and resume processing of the inbound message stream.
- Example: the receiver misses message five and receives messages six through ten. The receiver requests that message five be replayed and adds messages six through ten to a sorted list for processing after message five is received.
Ignore all new messages after the gap and request replay of messages between the inbound sequence number and the largest message sequence number seen (or preferably set EndSeqNo=0 in the ResendRequest to indicate that all messages from BeginSeqNo on should be replayed). If new messages arrive while the replayed message(s) are in-flight, repeat the process iteratively.
- Example: the receiver processes messages one through four, misses message five, and receives messages six through ten. It requests replay for messages five through ten by sending a ResendRequest with BeginSeqNo=5 and EndSeqNo=10 or ideally EndSeqNo=0.

Recovery

As noted in the section on ordered message processing, counterparties can request that missed messages are replayed via a ResendRequest message. Any message that is sent in response to a replay request must be sent with the PossDupFlag (tag #43) set to Y. The only fields that can change in a retransmission are: CheckSum (tag #10), OrigSendingTime (tag #122), SendingTime (tag #52), BodyLength (tag #9) and PossDupFlag (tag #43).

A counterparty may elect to send a SequenceReset message with the GapFillFlag (tag #123) in response to ResendRequest in lieu of replaying the original message. This is useful in situations where the original message is no longer relevant, e.g. an outdated order, or when the message requested is an administrative message (which typically aren’t replayed).

When consecutive administrative messages are present it is recommended that only one SequenceReset be sent in their place. This is accomplished by setting the MsgSeqNum of the SequenceReset to the next expected outbound sequence number and the NewSeqNo (tag #36) to the sequence number of the highest administrative message in the group, plus one. Example: there are five consecutive administrative messages being resent. The first message has a sequence number of ten, the last a sequence number of fourteen. A SequenceReset is sent in their place, with MsgSeqNum=10 and NewSeqNo=15.

Logon Messages

Under the standard ordered message processing and recovery paradigm, out-of-order messages are never processed on the application layer. An exception is made for Logon messages. The recipient of a Logon should always process it immediately, even if their sequence number is too high. After sending a Logon confirmation back, a ResendRequest is sent if a message gap was detected in the Logon sequence number.

Duplicates

Messages may be duplicated in the course of typical ordered message processing and recovery procedures. Any message that may be a duplicate (transmitted in response to a ResendRequest) must be sent with PossDupFlag=Y. If PossDupFlag=Y is set and the message sequence number is less than the receiver’s inbound sequence number, the message is dropped. Doing so ensures once-and-only-once processing semantics for the application layer.

If at any point a message is seen with an incoming sequence number less than expected and the PossDupFlag flag is not set to Y, a serious error has occurred. The only caveat to this rule which involves multiple overlapping ResendRequests, is covered in the Sequence Resets section. The recipient of a such a message should terminate the FIX session immediately via a Logout message. Further intervention may be required to ensure that no orders are outstanding and that all filled orders are accounted for.

Retransmissions

The FIX protocol specifies a method for application level retransmission when no gap was detected but there is concern that a message was not handled by the counterparty (e.g. an order was sent but no execution reports received). This is usually accomplished by setting PossResend (tag #97) to Y. OneChronos does not support this aspect of FIX. Resending messages in this fashion is unnecessary, potentially dangerous, and impossible to implement performantly.

Sequence Resets

A counterparty’s incoming sequence number (the sender’s outgoing sequence number) can be reset during message recovery via the SequenceReset message. There are two modes of doing so:

A SequenceReset message with GapFillFlag (tag #123) set to Y (henceforth referred to SeqReset-GapFill)
A SequenceReset message with GapFillFlag (tag #123) set to N (henceforth referred to SeqReset-Reset)

SequenceReset messages can also be sent to a counterparty, unsolicited, when standard ordered message handling procedures need to be bypassed (this is almost never necessary or advisable). Sending a SeqReset-Reset tells the counterparty to ignore the MsgSeqNum field and reset its expected incoming sequence number explicitly.

In all cases NewSeqNo (tag #36) should be set to the sequence number of the message being skipped, plus one. If multiple messages are being skipped, it should be set to the highest sequence number out of all messages being skipped, plus one.

SeqReset-GapFill messages must conform to the standard sequencing rules (i.e. their sequence numbers should match the counterpary’s expected inbound sequence number). Both SeqReset-GapFill and SeqReset-Reset requests should only ever attempt to increase the counterparty’s incoming sequence number. The note in the section on duplicates holds - any attempt to decrease a counterparty’s sequence should (and will, for OneChronos) result in immediate termination of the session.

It is possible to have multiple ResendRequests in-flight simultaneously. When this happens a SeqReset-GapFill sent in response may be an implicit duplicate, even if the PossDupFlag is not set. This condition is easily detected by checking if the MsgSeqNum for a SeqReset-GapFill is less than the expected incoming sequence number. Such messages should be silently dropped, even though PossDupFlag is not set. As an example (adapted from the FIX 4.2 guide):

P1 sends a ResendRequests for sequence numbers five through ten followed immediately by a second request for five through eleven.
Messages eight, ten, and eleven are application messages. Messages five through seven, and nine, are administrative messages that should be skipped.
P2 replies to the first ResendRequests with (SeqReset-GapFill, MsgSeqNum=5, NewSeqNo=8), (Message eight, MsgSeqNum=8, PossDupFlag=Y), (SeqReset-GapFill, MsgSeqNum=9, NewSeqNo=10), (Message ten, MsgSeqNum=10, PossDupFlag=Y).
P2 replies to the second ResendRequests with (SeqReset-GapFill, MsgSeqNum=5, NewSeqNo=8), (Message eight, MsgSeqNum=8, PossDupFlag=Y), (SeqReset-GapFill, MsgSeqNum=9, NewSeqNo=10), (Message ten, MsgSeqNum=10, PossDupFlag=Y), (Message eleven, MsgSeqNum=11, PossDupFlag=Y).
P1 receives P2’s first series of replies and processes them, after which P2’s expected incoming sequence number is eleven.
P1 receives P2’s second series of replies. The first of which, SeqReset-GapFill, MsgSeqNum=5, NewSeqNo=8, attempts to decrease P2’s incoming sequence number. However P2 can determine that this is an implicit duplicate given that MsgSeqNum=5 is less than the expected sequence, so it ignores it. Message eight, MsgSeqNum=8, PossDupFlag=Y is ignored as a duplicate because PossDupFlag=Y and its sequence number is less than eleven. SeqReset-GapFill, MsgSeqNum=9, NewSeqNo=10 is ignored for the same reason that the first SeqReset-GapFill was ignored, and message ten is ignored for the same reason that message eight was. Message eleven is processed because MsgSeqNum=11 is equal to the expected incoming sequence number.

Logon Procedures

FIX sessions start with a Logon message and end with a Logout. Given the complexities of sequence number handling several scenarios must be accounted for. We will refer the party initiating the logon as Initiator and the party receiving it as Acceptor.

Logging in and Resetting Sequence Numbers

A session Initiator can reset sequence numbers while logging in by including the field ResetSeqNum (tag #141)=Y. In such cases, Acceptor will reset their expected incoming sequence number to one before processing the Logon message, per the message delivery rules. Messages from any previous session will be dropped. Note: this is only advisable when establishing a new FIX session after the daily sequence number reset period.

The following logic applies when Acceptor receives a Logon message with ResetSeqNum (tag #141)=Y:

Logon message received by Acceptor is valid and authentication succeeds?
- Yes
  - Acceptor responds with Logon ResetSeqNum (tag #141)=Y. Initiator should set their expected incoming sequence number to one before processing the response from Acceptor. A session is now established.
- No, the message was valid but authentication failed
  - Acceptor responds with Logout with the Text (tag #58) field set to a description of the reason for authentication failure.
- No, the message was valid but the session was in use
  - Acceptor will not respond, given that doing so might corrupt sequence numbers for the active session.
- No, the message was well formed but invalid
  - Acceptor will reply with a Reject message with SeqNum=1. An explanation as to why the message was invalid will be given in the Text (tag #58) field. When applicable SessionRejectReason (tag #373) and RefTagID (tag #371) will be populated.
- No, the message was malformed
  - Acceptor will silently drop any packets containing malformed data.

Logging in Without Resetting Sequence Numbers

The following logic is used to negotiate a FIX session when Acceptor receives a Logon message with ResetSeqNum (tag #141)=N or unset:

Logon message received by Acceptor is valid, authentication succeeds, and SeqNum is greater than or equal to the incoming sequence number that Acceptor expects?
- Yes
  - Proceed to step 2.
- No, the message was valid but authentication failed
  - Acceptor responds with Logout with the Text (tag #58) field set to a description of the reason for authentication failure.
- No, the message was valid but the session was in use
  - Acceptor will not respond, given that doing so might corrupt sequence numbers for the active session.
- No, the message was well formed but invalid
  - Acceptor will reply with a Reject message with SeqNum=1. An explanation as to why the message was invalid will be given in the Text (tag #58) field. When applicable SessionRejectReason (tag #373) and RefTagID (tag #371) will be populated.
- No, the sequence number was less than expected
  - Per the note in the section on duplicate messages, Acceptor will not acknowledge the message as doing so might cause further problems.
- No, the message was malformed
  - Acceptor will silently drop any packets containing malformed data.
Are sequence numbers in sync for both the Initiator and Acceptor?
- Yes
  - Proceed to step 3.
- No, the Logon message received by Acceptor from Initiator has a sequence number that is higher than expected
  - Acceptor sends a ResendRequest for the missing messages. Initiator may wish to respond with SeqReset-GapFill if the original messages are no longer relevant. Repeat step 2.
- No, the Logon message received by Initiator from Acceptor has a sequence number that is higher than expected
  - The Initiator sends Acceptor a ResendRequest for the missing sequence range per the standard message recovery protocol. Repeat step 2.
- No, the Logon message received by Initiator from Acceptor has a sequence number that is lower than expected
  - Per the note on duplicate messages, an serious error has occurred. Initiator should not attempt to establish another session.
Has one second elapsed since a Logon message or a ResendRequest was sent or received by either party?
- Yes
  - Acceptor sends a TestRequest indicating that a FIX session was established. Initiator is free to send business type messages to Acceptor, and vice versa.
- No
  - Repeat step 3.

Logout

Once a FIX session is established, either party to the session can terminate it by sending a Logout message. Initiator should wait for Acceptor to respond with a reciprocal Logout message before closing the session. Heartbeat timeout rules apply if Acceptor does not acknowledge the Logout message in a timely fashion. The logout workflow is as follows:

Initiator sends Acceptor a Logout message.
Is the sequence number equal to Acceptors’s expected incoming sequence number?
- Yes
  - Acceptor responds with a Logout message
- No
  - Acceptor sends Initiator a ResendRequest
  - Acceptor sends a Logout messages after receiving all missing messages. Initiator can now consider the session ended.

Heartbeats

FIX uses a bidirectional “heartbeat” to ensure that a session is logically alive, regardless of network state. When Initiator sends a Logon message, it must populate the HeartBtInt (tag #108) field with a value stipulating the maximum length of time (in seconds) that should elapse between messages. For example, if P1 sets an interval of thirty seconds (expressed as an integer number of seconds) in her Logon message, she expects to see a message from P2 at least every thirty seconds. If a message is not sent as part of normal session activity, P2 must send a Heartbeat message instead. When P2 replies to P1’s Logon message he stipulates a heartbeat interval as well.

When either party to a FIX session notes that the heartbeat interval plus one second has elapsed without a message being sent they are responsible for sending a TestRequest message. If no message is received within another heartbeat interval, the session is considered lost. A Logout message should be sent (with the expectation that the counterparty might not receive it), after which logon procedures can be reinitiated.