We will investigate the effectiveness of Multi-Agent Reinforcement Learning(MARL) for obtaining optimal liquidation strategies in Financial Markets. Liquidation is a sequential decision-making problem where an agent strategises to sell a large number of assets in a given time period taking into account a financial market environment. Since state changes in such an environment are not subject to one but multiple such agents, we seek to frame this as a Multi-agent Reinforcement Learning Problem.

Introduction

We will investigate the effectiveness of Multi-Agent Reinforcement Learning(MARL) for obtaining optimal liquidation strategies in Financial Markets. Liquidation is a sequential decision-making problem where an agent strategises to sell a large number of assets in a given time period taking into account a financial market environment. Since state changes in such an environment are not subject to one but multiple such agents, we seek to frame this as a Multi-agent Reinforcement Learning Problem.

Contribution

As the first step, we will reproduce the result from the paper ‘Multi-agent Deep Reinforcement Learning for Liquidation Strategy Analysis’[1]. It uses Deep Deterministic Policy Gradients, an actor-critic method to generate optimal liquidation strategies.

Subsequently, we hope to make the some of the following modifications:

Change the underlying market model

  • Almgren-Chriss Market Impact Model[2] is the standard framework for modelling financial markets. This paper also uses the same.
  • We plan to incorporate adaptive trading[3] which improves upon the Almgren-Chriss model.

Change the liquidation problem

  • The paper assumes that multiple agents only sell assets in a given time period. We aim to experiment with both buying and selling so as to make a more realistic market environment.
  • We also hope to experiment with optimistic bull, pessimistic bear[4], and anomaly events[5] if time permits. These would also contribute towards making the environment closer to real-life.

Modify the state space

  • The state space currently includes collected rewards, trades left and assets left. We hypothesize that adding some signals that represent the current market situation might lead to better liquidation strategies.

Modify the reward function

  • Currently the reward function includes risk aversion, implementation shortfall, and trading trajectory. We aim to include Sharpe ratio, Profit rate, Sortino Ratio, Return, and Profit & Loss and compare their effectiveness.

Modify the learning algorithm

  • We hope to implement Advanced Actor Critic(A2C) which includes value-based agents in addition to policy-based agents.
  • We also seek to experiment with Proximal Policy Optimization(PPO) for this multi-agent setting.

We understand that it might not be feasible to implement all these modifications. We will prioritise the changes that we feel are most impactful.

Potential Difficulties

We predict that the following difficulties could arise when carrying out the project:

  • Implementing adaptive trading will probably require changes in the learning algorithm itself. We can address this by making the trade rate a predicted variable of the model.
  • Permitting both buying and selling of assets will require some modifications to the Algren-Chriss model. Carefully analysing the impact factors can provide insight on how to do this.
  • A2C and PPO have their own problems which will need addressing. For example, PPO is an on-policy method and thus could suffer from the problem of being stuck in a local minima. One way to address this is by adding entropy coefficients to environment.

Conclusion

We expect to deliver a learning agent trained using MARL, which is more robust and effective in real-world trading scenarios. We also seek to understand multi-agent’s cooperate-compete behaviour to open up new directions for future research.

Team Information

Team 29

  • Shivam Patel (805626050)
  • Abirami Anbumani (005526158)

Reference

[1] Bao, W. and Liu, X.Y., 2019. Multi-agent deep reinforcement learning for liquidation strategy analysis. arXiv preprint arXiv:1906.11046.

[2] Almgren, Robert and Chriss, Neil A., Optimal Liquidation (November 24, 1997). Available at SSRN: https://ssrn.com/abstract=53501 or http://dx.doi.org/10.2139/ssrn.53501

[3] Lorenz, Julian M. 2008. Optimal trading algorithms - Portfolio transactions, multiperiod portfolio selection, and competitive online search. Available at https://doi.org/10.3929/ethz-a-005687539

[4] Li, X., Li, Y., Zhan, Y., and Liu, X.-Y. Optimistic bull or pessimistic bear: adaptive deep reinforcement learning for stock portfolio allocation. In ICML Workshop on Applications and Infrastructure for Multi-Agent Learning, 2019b.

[5] Li, X., Li, Y., , Liu, X.-Y., and Wang, C. Risk management via anomaly circumvent: Mnemonic deep learning for midterm stock prediction. In KDD Workshop on Anomaly Detection in Finance, 2019a.