Hostname: page-component-7f64f4797f-rcs6z Total loading time: 0 Render date: 2025-11-04T04:58:26.237Z Has data issue: false hasContentIssue false
  • English
  • Français

Robust flow control and optimal sensor placement using deep reinforcement learning

Published online by Cambridge University Press:  26 February 2021

Romain Paris Open the ORCID record for Romain Paris [Opens in a new window] ,
Samir Beneddine Open the ORCID record for Samir Beneddine [Opens in a new window]  and
Julien Dandois Open the ORCID record for Julien Dandois [Opens in a new window]
Romain Paris*
Affiliation:
DAAA, ONERA, Université Paris Saclay, 92190 Meudon, France
Samir Beneddine
Affiliation:
DAAA, ONERA, Université Paris Saclay, 92190 Meudon, France
Julien Dandois
Affiliation:
DAAA, ONERA, Université Paris Saclay, 92190 Meudon, France
*
†Email address for correspondence: romain.paris@onera.fr
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper focuses on finding a closed-loop strategy to reduce the drag of a cylinder in laminar flow conditions. Deep reinforcement learning algorithms have been implemented to discover efficient control schemes, using two synthetic jets located on the cylinder's poles as actuators and pressure sensors in the wake of the cylinder as feedback observation. The present work focuses on the efficiency and robustness of the identified control strategy and introduces a novel algorithm (S-PPO-CMA) to optimise the sensor layout. An energy-efficient control strategy reducing drag by $18.4\,\%$ at a Reynolds number of $120$ is obtained. This control policy is shown to be robust both to a Reynolds-number variation in the range $[100;216]$ and to measurement noise, for signal-to-noise ratios as low as $0.2$ with negligible impact on performance. Along with a systematic study on sensor number and location, the proposed sparsity-seeking algorithm has achieved a successful optimisation to a reduced five-sensor layout while keeping state-of-the-art performance. These results further highlight the interesting possibilities of reinforcement learning for active flow control and pave the way to efficient, robust and practical implementations of these control techniques in experimental or industrial systems.

JFM classification

Flow Control: Drag reduction Flow Control: Instability control Vortex Flows: Vortex shedding

Information

Type
JFM Papers
Information
Journal of Fluid Mechanics , Volume 913 , 25 April 2021 , A25
Check for updates
Copyright
© The Author(s), 2021. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

REFERENCES

Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G. & Isard, M. 2016 Tensorflow: a system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265–283.Google Scholar
Arakeri, J.H. & Shukla, R.K. 2013 A unified view of energetic efficiency in active drag reduction, thrust generation and self-propulsion through a loss coefficient with some applications. J. Fluids Struct. 41, 22–32.CrossRefGoogle Scholar
Atam, E., Mathelin, L. & Cordier, L. 2016 Identification-based closed-loop control strategies for a cylinder wake flow. IEEE Trans. Control Syst. Technol. 25 (4), 1488–1495.CrossRefGoogle Scholar
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B. & Mordatch, I. 2019 Emergent tool use from multi-agent autocurricula. arXiv:1909.07528.Google Scholar
Barkley, D. 2006 Linear analysis of the cylinder wake mean flow. Europhys. Lett. 75 (5), 750.CrossRefGoogle Scholar
Beintema, G., Corbetta, A., Biferale, L. & Toschi, F. 2020 Controlling Rayleigh–Bénard convection via reinforcement learning. J. Turbul. 21 (9–10), 585–605.Google Scholar
Belus, V., Rabault, J., Viquerat, J., Che, Z., Hachem, E. & Reglade, U. 2019 Exploiting locality and translational invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling liquid film. AIP Adv. 9 (12), 125014.CrossRefGoogle Scholar
Beneddine, S. 2017 Characterization of unsteady flow behavior by linear stability analysis. PhD thesis, Université Paris-Saclay.Google Scholar
Benoit, C., Péron, S. & Landier, S. 2015 Cassiopee: a CFD pre-and post-processing tool. Aerosp. Sci. Technol. 45, 272–283.CrossRefGoogle Scholar
Bergmann, M. & Cordier, L. 2008 Optimal control of the cylinder wake in the laminar regime by trust-region methods and POD reduced-order models. J. Comput. Phys. 227 (16), 7813–7840.CrossRefGoogle Scholar
Bergmann, M., Cordier, L. & Brancher, J.-P. 2005 Control of the cylinder wake in the laminar regime by trust-region methods and POD reduced order models. In Proceedings of the 44th IEEE Conference on Decision and Control, pp. 524–529.Google Scholar
Bergmann, M., Cordier, L. & Brancher, J.-P. 2006 On the generation of a reverse Von Kármán street for the controlled cylinder wake in the laminar regime. Phys. Fluids 18 (2), 028101.CrossRefGoogle Scholar
Braza, M., Chassaing, P.H.H.M. & Minh, H.H. 1986 Numerical study and physical analysis of the pressure and velocity fields in the near wake of a circular cylinder. J. Fluid Mech. 165, 79–130.CrossRefGoogle Scholar
Bright, I., Lin, G. & Kutz, J.N. 2013 Compressive sensing based machine learning strategy for characterizing the flow around a cylinder with limited pressure measurements. Phys. Fluids 25 (12), 127102.CrossRefGoogle Scholar
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J. & Zaremba, W. 2016 OpenAI gym. arXiv:1606.01540.Google Scholar
Brunton, S.L. & Noack, B.R. 2015 Closed-loop turbulence control: progress and challenges. Appl. Mech. Rev. 67 (5), 050801.CrossRefGoogle Scholar
Brunton, S.L., Noack, B.R. & Koumoutsakos, P. 2020 Machine learning for fluid mechanics. Annu. Rev. Fluid Mech. 52, 477–508.CrossRefGoogle Scholar
Bucci, M.A., Semeraro, O., Allauzen, A., Wisniewski, G., Cordier, L. & Mathelin, L. 2019 Control of chaotic systems by deep reinforcement learning. Proc. R. Soc. A 475, 20190351.Google Scholar
Chen, Z. & Aubry, N. 2005 Active control of cylinder wake. Commun. Nonlinear Sci. Numer. Simul. 10 (2), 205–216.CrossRefGoogle Scholar
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H. & Bengio, Y. 2014 Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078.CrossRefGoogle Scholar
Cohen, K., Siegel, S. & McLaughlin, T. 2006 A heuristic approach to effective sensor placement for modeling of a cylinder wake. Comput. Fluids 35 (1), 103–120.CrossRefGoogle Scholar
Cohen, K., Siegel, S., Seidel, J., Aradag, S. & McLaughlin, T. 2012 Nonlinear estimation of transient flow field low dimensional states using artificial neural nets. Expert Syst. Appl. 39 (1), 1264–1272.CrossRefGoogle Scholar
Colabrese, S., Gustavsson, K., Celani, A. & Biferale, L. 2017 Flow navigation by smart microswimmers via reinforcement learning. Phys. Rev. Lett. 118 (15), 158004.CrossRefGoogle ScholarPubMed
Curtiss, C.F. & Hirschfelder, J.O. 1952 Integration of stiff equations. Proc. Natl Acad. Sci. USA 38 (3), 235.CrossRefGoogle ScholarPubMed
Dandois, J., Garnier, E. & Pamart, P.-Y. 2013 NARX modelling of unsteady separation control. Exp. Fluids 54 (2), 1445.CrossRefGoogle Scholar
Dandois, J., Mary, I. & Brion, V. 2018 Large-eddy simulation of laminar transonic buffet. J. Fluid Mech. 850, 156–178.CrossRefGoogle Scholar
DeVries, L. & Paley, D.A. 2013 Observability-based optimization for flow sensing and control of an underwater vehicle in a uniform flowfield. In 2013 American Control Conference, pp. 1386–1391.Google Scholar
Edwards, J.R. & Liou, M.-S. 1998 Low-diffusion flux-splitting methods for flows at all speeds. AIAA J. 36 (9), 1610–1617.CrossRefGoogle Scholar
Foures, D.P.G., Dovetta, N., Sipp, D. & Schmid, P.J. 2014 A data-assimilation method for Reynolds-averaged Navier–Stokes-driven mean flow reconstruction. J. Fluid Mech. 759, 404–431.CrossRefGoogle Scholar
Fujisawa, N., Kawaji, Y. & Ikemoto, K. 2001 Feedback control of vortex shedding from a circular cylinder by rotational oscillations. J. Fluids Struct. 15 (1), 23–37.CrossRefGoogle Scholar
Gerhard, J., Pastoor, M., King, R., Noack, B., Dillmann, A., Morzynski, M. & Tadmor, G. 2003 Model-based control of vortex shedding using low-dimensional Galerkin models. In 33rd AIAA Fluid Dynamics Conference and Exhibit, p. 4262.Google Scholar
Gutmark, E.J. & Grinstein, F.F. 1999 Flow control with noncircular jets. Annu. Rev. Fluid Mech. 31 (1), 239–272.CrossRefGoogle Scholar
Hämäläinen, P., Babadi, A., Ma, X. & Lehtinen, J. 2020 PPO-CMA: proximal policy optimization with covariance matrix adaptation. In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1–6. IEEE.CrossRefGoogle Scholar
Hansen, N. 2016 The CMA evolution strategy: a tutorial. arXiv:1604.00772.Google Scholar
Hansen, N., Müller, S.D. & Koumoutsakos, P. 2003 Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Comput. 11 (1), 1–18.CrossRefGoogle Scholar
He, J.-W., Glowinski, R., Metcalfe, R., Nordlander, A. & Periaux, J. 2000 Active control and drag optimization for flow past a circular cylinder: I. Oscillatory cylinder rotation. J. Comput. Phys. 163 (1), 83–117.CrossRefGoogle Scholar
He, K., Zhang, X., Ren, S. & Sun, J. 2016 Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778.Google Scholar
Henderson, R.D. 1997 Nonlinear dynamics and pattern formation in turbulent wake transition. J. Fluid Mech. 352, 65–112.CrossRefGoogle Scholar
Huh, M., Agrawal, P. & Efros, A.A. 2016 What makes imagenet good for transfer learning? arXiv:1608.08614.Google Scholar
Jin, B., Illingworth, S.J. & Sandberg, R.D. 2019 Feedback control of vortex shedding using a resolvent-based modelling approach. arXiv:1909.04865.CrossRefGoogle Scholar
Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Campbell, R.H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P. & Levine, S. 2019 Model-based reinforcement learning for Atari. arXiv:1903.00374.Google Scholar
Kim, K., Kerr, M., Beskok, A. & Jayasuriya, S. 2006 Frequency-domain based feedback control of flow separation using synthetic jets. In 2006 American Control Conference, p. 6. IEEE.Google Scholar
Kingma, D.P. & Ba, J. 2014 Adam: a method for stochastic optimization. arXiv:1412.6980.Google Scholar
Leclerc, E., Sagaut, P. & Mohammadi, B. 2006 On the use of incomplete sensitivities for feedback control of laminar vortex shedding. Comput. Fluids 35 (10), 1432–1443.CrossRefGoogle Scholar
Leclercq, C., Demourant, F., Poussot-Vassal, C. & Sipp, D. 2019 Linear iterative method for closed-loop control of quasiperiodic flows. J. Fluid Mech. 868, 26–65.CrossRefGoogle Scholar
Louizos, C., Welling, M. & Kingma, D.P. 2017 Learning sparse neural networks through $l_0$ regularization. arXiv:1712.01312.Google Scholar
Manohar, K., Kutz, J.N. & Brunton, S.L. 2018 Optimal sensor and actuator placement using balanced model reduction. arXiv:1812.01574.Google Scholar
Marquet, O., Sipp, D. & Jacquin, L. 2008 Sensitivity analysis and passive control of cylinder flow. J. Fluid Mech. 615, 221–252.CrossRefGoogle Scholar
Min, C. & Choi, H. 1999 Suboptimal feedback control of vortex shedding at low Reynolds numbers. J. Fluid Mech. 401, 123–156.CrossRefGoogle Scholar
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K. & Ostrovski, G. 2015 Human-level control through deep reinforcement learning. Nature 518 (7540), 529.CrossRefGoogle ScholarPubMed
Mons, V., Chassaing, J.-C., Gomez, T. & Sagaut, P. 2016 Reconstruction of unsteady viscous flows using data assimilation schemes. J. Comput. Phys. 316, 255–280.CrossRefGoogle Scholar
Mons, V., Chassaing, J.-C. & Sagaut, P. 2017 Optimal sensor placement for variational data assimilation of unsteady flows past a rotationally oscillating cylinder. J. Fluid Mech. 823, 230–277.CrossRefGoogle Scholar
Muddada, S. & Patnaik, B.S.V. 2010 An active flow control strategy for the suppression of vortex structures behind a circular cylinder. Eur. J. Mech. B/Fluids 29 (2), 93–104.CrossRefGoogle Scholar
Nair, A.G., Taira, K., Brunton, B.W. & Brunton, S.L. 2020 Phase-based control of periodic fluid flows. arXiv:2004.10561.Google Scholar
Nishioka, M. & Sato, H. 1978 Mechanism of determination of the shedding frequency of vortices behind a cylinder at low Reynolds numbers. J. Fluid Mech. 89 (1), 49–60.CrossRefGoogle Scholar
Nørgård, P.M., Ravn, O., Poulsen, N.K. & Hansen, L.K. 2000 Neural Networks for Modelling and Control of Dynamic Systems: A Practitioner's Handbook. Springer.CrossRefGoogle Scholar
Oehler, S.F. & Illingworth, S.J. 2018 Sensor and actuator placement trade-offs for a linear model of spatially developing flows. J. Fluid Mech. 854, 34–55.CrossRefGoogle Scholar
Protas, B. & Styczek, A. 2002 Optimal rotary control of the cylinder wake in the laminar regime. Phys. Fluids 14 (7), 2073–2087.CrossRefGoogle Scholar
Protas, B. & Wesfreid, J.E. 2002 Drag force in the open-loop control of the cylinder wake in the laminar regime. Phys. Fluids 14 (2), 810–826.CrossRefGoogle Scholar
Rabault, J., Kuchta, M., Jensen, A., Réglade, U. & Cerardi, N. 2019 Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. J. Fluid Mech. 865, 281–302.CrossRefGoogle Scholar
Rabault, J. & Kuhnle, A. 2019 Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach. Phys. Fluids 31 (9), 094105.CrossRefGoogle Scholar
Rabault, J., Ren, F., Zhang, W., Tang, H. & Xu, H. 2020 Deep reinforcement learning in fluid mechanics: a promising method for both active flow control and shape optimization. J. Hydrodyn. 32, 234–246.CrossRefGoogle Scholar
Rashidi, S., Hayatdavoodi, M. & Esfahani, J.A. 2016 Vortex shedding suppression and wake control: a review. Ocean Engng 126, 57–80.CrossRefGoogle Scholar
Ren, F., Rabault, J. & Tang, H. 2020 Applying deep reinforcement learning to active flow control in turbulent conditions. arXiv:2006.10683.Google Scholar
Schulman, J., Levine, S., Abbeel, P., Jordan, M. & Moritz, P. 2015 a Trust region policy optimization. In International Conference on Machine Learning, pp. 1889–1897. PMLR.Google Scholar
Schulman, J., Moritz, P., Levine, S., Jordan, M. & Abbeel, P. 2015 b High-dimensional continuous control using generalized advantage estimation. arXiv:1506.02438.Google Scholar
Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. 2017 Proximal policy optimization algorithms. arXiv:1707.06347.Google Scholar
Seidel, J., Siegel, S., Fagley, C., Cohen, K. & McLaughlin, T. 2009 Feedback control of a circular cylinder wake. Proc. Inst. Mech. Engs G 223 (4), 379–392.CrossRefGoogle Scholar
Selby, G.V., Lin, J.C. & Howard, F.G. 1992 Control of low-speed turbulent separated flow using jet vortex generators. Exp. Fluids 12 (6), 394–400.CrossRefGoogle Scholar
Shimomura, S., Sekimoto, S., Oyama, A., Fujii, K. & Nishida, H. 2020 Closed-loop flow separation control using the deep Q network over airfoil. AIAA J. 58 (10), 4260–4270.CrossRefGoogle Scholar
Siegel, S., Cohen, K. & McLaughlin, T. 2003 Feedback control of a circular cylinder wake in experiment and simulation. In 33rd AIAA Fluid Dynamics Conference and Exhibit, p. 3569.Google Scholar
Singh, A.K. & Hahn, J. 2005 Determining optimal sensor locations for state and parameter estimation for stable nonlinear systems. Ind. Engng Chem. Res. 44 (15), 5645–5659.CrossRefGoogle Scholar
Singha, S. & Sinhamahapatra, K.P. 2011 Control of vortex shedding from a circular cylinder using imposed transverse magnetic field. Intl J. Numer. Meth. Heat & Fluid Flow 21 (1), 32–45.CrossRefGoogle Scholar
Sipp, D. 2012 Open-loop control of cavity oscillations with harmonic forcings. J. Fluid Mech. 708, 439–468.CrossRefGoogle Scholar
Sipp, D., Marquet, O., Meliga, P. & Barbagallo, A. 2010 Dynamics and control of global instabilities in open-flows: a linearized approach. Appl. Mech. Rev. 63 (3).CrossRefGoogle Scholar
Sipp, D. & Schmid, P.J. 2016 Linear closed-loop control of fluid instabilities and noise-induced perturbations: a review of approaches and tools. Appl. Mech. Rev. 68 (2).CrossRefGoogle Scholar
Sohankar, A., Khodadadi, M. & Rangraz, E. 2015 Control of fluid flow and heat transfer around a square cylinder by uniform suction and blowing at low Reynolds numbers. Comput. Fluids 109, 155–167.CrossRefGoogle Scholar
Sutskever, I., Vinyals, O. & Le, Q.V. 2014 Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112.Google Scholar
Tang, H., Rabault, J., Kuhnle, A., Wang, Y. & Wang, T. 2020 Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning. Phys. Fluids 32 (5), 053605.CrossRefGoogle Scholar
Verma, S., Papadimitriou, C., Lüthen, N., Arampatzis, G. & Koumoutsakos, P. 2020 Optimal sensor placement for artificial swimmers. J. Fluid Mech. 884.CrossRefGoogle Scholar
Williams, R.J. 1992 Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8 (3–4), 229–256.CrossRefGoogle Scholar
Williamson, C.H.K. 1996 Vortex dynamics in the cylinder wake. Annu. Rev. Fluid Mech. 28 (1), 477–539.CrossRefGoogle Scholar
Zielinska, B.J.A., Goujon-Durand, S., Dusek, J. & Wesfreid, J.E. 1997 Strongly nonlinear effect in unstable wakes. Phys. Rev. Lett. 79 (20), 3893.CrossRefGoogle Scholar