Department of Computing

City School of Informatics People Eduardo Alonso

I AM RECRUITING PhD STUDENTS

Requirements: Good programming skills (preferably but not limited to Java/C++; MATLAB) and expertise in two of the following areas: neuroscience (learning and behaviour), machine learning, control and optimisation, dynamic systems. Applicants would also need to have a strong mathematical background.

• Simulators of Psychological Models of Conditioning: It is universally accepted that conditioning is at the basis of most learning phenomena. It is thus paramount that we develop accurate simulators of models of conditioning. In collaboration with Esther Mondragon at the Centre for Computational and Animal Learning Research and Alberto Fernandez-Gil (Universidad Rey Juan Carlos), I am developing simulators of classical conditioning models. We have just released version 3.1 of our Rescorla and Wagner Simulator (CAL-RWSim, a Java simulator of Rescorla and Wagner, 1972), and plan to extend it with other models in the near future. Details on the simulator have been published in Computer Methods and Programs in Biomedicine.
  
  With Jonathan Gray we have developed a simulator of Temporal Difference, CAL-TDSim, and are simulating results obtained by Charlotte Bonardi (University of Nottimgham) and Domhnall Jennings (University of Newcastle), on the role of time variance and uncertainty in overshadowing.
  
  
• Reinforcement Learning and Adaptive Dynamic Programming: Michael Fairbank's PhD work extends Temporal Difference methods to continuous state spaces and deterministic environments: He has invented a new algorithm, Value Gradient Learning-VGL(lambda), that learns gradients of values rather than values and proved that it converges to local optimality under certain conditions when lambda=1; he has also proved that VGL(1) is equivalent to Policy Gradient Learning and to Backpropagation through time (BPTT), and shown some interesting experimental results (published as three papers in the Proceedings of the IEEE IJCNN 2012). We would like to complement this line of research and investigate new Adaptive Dynamic Programming methods that make use of model functions --in line with Pontryagin's maximum principle . The long-term aim is to develop a set of Reinforcement Learning methods that apply to both traditional stochastic model-free problems as well as to model-based deterministic ones. In this, with are collaborating with Danil Prokhorov, from Toyota Research Institute, Michigan.
  
  We are also working with Shuhui Li at The University of Alabama, and Don Wunsch at the Missouri University of Science and Technology, in applying VGL to real-life control problems, to power systems in particular.
  
  
• Computational Models of Learning, Behaviour and Decision-Making (Neuroeconomics): As a member of the Society for Computational Modeling of Associative Learning (SOCMAL) I am interested in developing computational models of learning, behaviour and decision-making. In particular, I am interested in exploring variational principles in learning and behaviour. I am investigating classical conditioning and instrumental conditioning using calculus of variations and optimal control methods. Such methods have been proved useful in expressing extremal principles that reflect constitutive and conservation laws as well as underlying symmetries in Nature.
  
  I am co-editing along with Nestor Schmajuk (Duke Institute for Brain Sciences) a Special Issue on Computational Models of Classical Conditioning for the journal Learning & Behavior. In addition, during Autumn 2012 I will be visiting the Gatsby Computational Neuroscience Unit at University College London and the Centre for Philosophy of Natural and Social Science at the London School of Economics to pursue joint projects on neuroeconomics.
  
  
  NEWS
  
  
  • Michael Fairbank and Eduardo Alonso, Efficient Calculation of the Gauss-Newton Approximation of the Hessian Matrix in Neural Networks, Neural Computation, March 2012, Vol. 24, No. 3: 607-610.

  • Eduardo Alonso, Esther Mondragon and Alberto Fernandez, A Java simulator of Rescorla and Wagner's prediction error model and configural cue extensions, to appear in Computer Methods and Programs in Biomedicine.

  • Michael Fairbank, Eduardo Alonso and Danil Prokhorov, Simple and Fast Calculation of the Second Order Gradients for Globalized Dual Heuristic Dynamic Programming in Neural Networks, to appear in IEEE Transactions on Neural Networks and Learning Systems.

  • Eduardo Alonso and Nestor Schmajuk, Computational Models of Classical Conditioning, to appear in Learning & Behavior.

  • Michael Fairbank and Eduardo Alonso, Value-Gradient Learning, in the Proceedings of the IEEE International Joint Conference on Neural Networks (IEEE IJCNN 2012), pp. 3062-3069. Brisbane, Australia, 10-15 June, 2012. Nominated Best Paper Award.

  • Michael Fairbank and Eduardo Alonso, A Comparison of Learning Speed and Ability to Cope Without Exploration between DHP and TD(0), in the Proceedings of the IEEE International Joint Conference on Neural Networks (IEEE IJCNN 2012), pp. 1478-1485. Brisbane, Australia, 10-15 June, 2012. Nominated Best Paper Award.

  • Michael Fairbank and Eduardo Alonso, The Divergence of Reinforcement Learning Algorithms with Value-Iteration and Function Approximation, in the Proceedings of the IEEE International Joint Conference on Neural Networks (IEEE IJCNN 2012), pp. 3070-3077. Brisbane, Australia, 10-15 June, 2012.

  • Shuhui Li, Michael Fairbank, Donald Wunsch and Eduardo Alonso, Vector Control of a Grid-Connected Rectifier/Inverter Using an Artificial Neural Network, in the Proceedings of the IEEE International Joint Conference on Neural Networks (IEEE IJCNN 2012), pp. 1783-1789. Brisbane, Australia, 10-15 June, 2012.

  • Accepted chapter in Frank Lewis and Derong Liu (Eds.), Handbook of Learning and Approximate Dynamic Programming, Volume 2, Wiley-IEEE Press: Approximating Optimal Control with Value Gradient Learning,with Michael Fairbank and Danil Prokhorov.

  • I am contributing to The Cambridge Handbook of Artificial Intelligence, Cambridge University Press, Keith Frankish and William Ramsey (Eds.) with a chapter on Actions and Agents.

  • Eduardo Alonso, Esther Mondragon and Niclas Kjall-Ohlsson, Internally Driven Q-learning: Convergence and Generalization Results, in Joaquim Filipe and Ana Fred (Eds.), Proceedings of the The Fourth International Conference on Agents and Artificial Intelligence (ICAART-2012), Vol. 1, 491-494, SciTe Press, Vilamoura, Portugal, 6-8 February 2012.

  • Accepted paper for The 38th ABAI (Association for Behavior Analysis International) Annual Convention, on variational principles of classical and operant conditioning. Seattle, WA, May 2012.
    
    
    
  • Co-ordinating with Prof. Nicos Karcanias the new City Complexity Science Group.
  • Co-ordinating with Mark Broom the Mathematical and Computational Behaviour and Evolution Research Group (MCBE).
  • Awarded a Royal Society Research Grant, The British History of Artificial Intelligence --as we speak I am writing the history of The Society for the Study of Artificial Intelligence and the Simulation of Behaviour (AISB), of which I served as vice-chair between 2003 and 2006 and as co-editor of the AISB Journal.
  • Appointed Member of the Engineering and Physical Sciences Research Council (EPSRC) Peer Review College for 2009-2012.
  • Acting as reviewer for Artificial Intelligence.
  • I have edited with Esther Mondragon the book Computational Neuroscience for Advancing Artificial Intelligence: Models, Methods and Applications, IGI Glogal, 2011.