Handle when Exchange Server performs MarketClosed on an order

[dfelton]

At times when gemini performs maintenance on their servers they will close all open limit orders an a trading pair. When this happens, the application does not recognize that the order is not live anymore and will not re-place the original buy / sell order.

Example order status after this event has occurred:

Order Id           	71173822663
Id                 	71173822663
Symbol             	batusd
Exchange           	gemini
Avg Execution Price	0.00
Side               	sell
Type               	exchange limit
Timestamp          	1637985141
Timestampms        	1637985141681
Is Live            	false
Is Cancelled       	true
Is Hidden          	false
Was Forced         	false
Executed Amount    	0
Client Order Id    	repeater_42249_sell_1637985141.1653
Reason             	MarketClosed
Options            	
	0: maker-or-cancel
Price              	2.24251
Original Amount    	1.1725
Remaining Amount   	1.1725

[dfelton]

Note: the below verbiage is written under the context that we are only trading on "USD" quoted pairs. However, problem is quote currency agnostic.

After today's 9 hour service outage Gemini encountered which resulted in the cancellation of all limit orders on the exchange it brought to light the importance of handling this properly for partially filled orders. Up until now I have been resolving this scenario by performing the following manual steps:

• ensure the TradeRepeater suite of classes are not running (touch var/emergency_shutdown if necessary to halt trading)
• if the local database record is in a BUY_PLACED status, reset the record by manipulating it in the following way:
  • SET status = "BUY_READY", buy_order_id = null, buy_client_order_id = NULL, meta = null
• if the local database record is in a SELL_PLACED status, reset the record by manipulating it in the following way:
  • SET status = "BUY_FILLED", sell_order_id = null, sell_client_order_id = NULL, meta = '{"buy_price":"1234.56"}'
    • (regarding the meta column, the sell_price would be unset, and the buy_price value of the original buy for the record left alone)
• After performing these steps on all necessary records, start the application back up and allow the bot to replace all impacted limit orders.

This worked well enough for the time being. It was already known that there was the possibility that some orders had been partially filled. For these situations, the following was accepted:

• Partially filled buy orders would result in what is to be considered "dust" accumulations of purchased coin
  • The assumption was made that for re-placed buy orders, there would always be enough extra USD on hand to replace the entirety of the original order. This may come from recent profits sitting in the usd bucket, else the reserve funds that are used for the temporary hold of transaction fees beyond the current maker API fees that get returned to available balance upon order fill (no automated tracking here, manual execution of the bin/gemini repeater:account command would yield the necessary information to determine if any manual re-allocation of bucket funds would be necessary afterwards).
• Partially filled sell orders would result in what is to be considered "dust" accumulations of acquired USD
  • The assumption was made that for re-placed sell orders, there would always be enough available coin from the profit bucket of that coin to cover re-placing the entirety of the original order.
• Accumulated "dust" would not be tracked by the application (although still show up in amount and amount available records from the exchange)

What had not been considered was the following:

• When resetting limit buy orders which have been partially filled, it is certainly possible to put the bot into a situation where it attempts to place more buy orders than it has USD funds available to work with. This hold certainly true when the exchange performs a MarketClosed event on the exchange's entire set of order books across all pairs.
  • this is especially true when operating on a slim margin of extra funds.
• When resetting limit sell orders which have been partially filled, the profit bucket for that pair becomes inaccurate in that it will state there are more coins available in the bucket than there truly are. This holds true on the USD bucket as well for buy orders, however this bucket is frequently adjusted anyway.
• In the event that there is a trading pair where none of the coin is ever kept with each sale (thus, never growing a profit bucket of), resetting of all the SELL_PLACED limit orders will result in a situation where the bot runs into insufficient funds unless the user has manually purchased enough of that coin in the past and is currently hodling.

In today's MarketClosed of all Limit Orders on the exchange, the bot put itself into a situation where it was $40 away from running out of USD funds for more buy orders. Furthermore, because all shibusd records had been configured to never hodl any of the profits of coin's sales into profit buckets, the last record it attempted to re-place a sell order for, encountered an insufficient funds error (Attempted to place a sell order of 100,000 SHIB, however only 65983.79777000 was available. Indicating that roughly 34,000 SHIB had already been sold in prior partially filled sell orders).