Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

A virtual rodent predicts the structure of neural activity across behaviours

Abstract

Animals have exquisite control of their bodies, allowing them to perform a diverse range of behaviours. How such control is implemented by the brain, however, remains unclear. Advancing our understanding requires models that can relate principles of control to the structure of neural activity in behaving animals. Here, to facilitate this, we built a ‘virtual rodent’, in which an artificial neural network actuates a biomechanically realistic model of the rat1 in a physics simulator2. We used deep reinforcement learning3,4,5 to train the virtual agent to imitate the behaviour of freely moving rats, thus allowing us to compare neural activity recorded in real rats to the network activity of a virtual rodent mimicking their behaviour. We found that neural activity in the sensorimotor striatum and motor cortex was better predicted by the virtual rodent’s network activity than by any features of the real rat’s movements, consistent with both regions implementing inverse dynamics6. Furthermore, the network’s latent variability predicted the structure of neural variability across behaviours and afforded robustness in a way consistent with the minimal intervention principle of optimal feedback control7. These results demonstrate how physical simulation of biomechanically realistic virtual animals can help interpret the structure of neural activity across behaviour and relate it to theoretical principles of motor control.

This is a preview of subscription content, access via your institution

Access options

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Comparing biological and artificial control across the behavioural repertoire with MIMIC.
Fig. 2: Training artificial agents to imitate rat behaviour with MIMIC.
Fig. 3: Neural activity in DLS and MC is best predicted by an inverse dynamics model.
Fig. 4: The representational structure of neural populations in DLS and MC across behaviours resembles that of an inverse model.
Fig. 5: Stochastic controllers regulate motor variability as a function of behaviour by changing latent variability.

Similar content being viewed by others

Data availability

The data generated from real animals are publicly available on Harvard Dataverse, https://doi.org/10.7910/DVN/FB0MZT. To help us understand use, provide support, fulfil custom requests and encourage collaboration, we ask that users contact us when considering using this dataset. Because of their size, the data generated in simulation will be made available on reasonable request.

Code availability

Code for all analyses will be made available from the corresponding authors on reasonable request. Repositories for skeletal registration (STAC), behavioural classification (motion-mapper) and inverse dynamic model inference are available at https://github.com/diegoaldarondo/virtual_rodent.

References

  1. Merel, J. et al. Deep neuroethology of a virtual rodent. In Proc. 8th International Conference on Learning Representations 11686–11705 (ICLR, 2020).

  2. Todorov, E., Erez, T. & Tassa, Y. MuJoCo: a physics engine for model-based control. In Proc. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems 5026–5033 (IEEE, 2012).

  3. Hasenclever, L., Pardo, F., Hadsell, R., Heess, N. & Merel, J. CoMic: complementary task learning & mimicry for reusable skills. In Proc. 37th International Conference on Machine Learning (eds Daumé, H. & Singh, A.) 4105–4115 (PMLR, 2020).

  4. Merel, J. et al. Neural probabilistic motor primitives for humanoid control. In Proc. 7th International Conference on Learning Representations (ICLR, 2019).

  5. Peng, X. B., Abbeel, P., Levine, S. & van de Panne, M. DeepMimic: example-guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph. 37, 1–14 (2018).

    CAS  Google Scholar 

  6. Jordan, M. I. in Handbook of Perception and Action, Vol. 2 (ed. Heuer, H.) Ch. 2 (Academic Press, 1996).

  7. Todorov, E. & Jordan, M. I. Optimal feedback control as a theory of motor coordination. Nat. Neurosci. 5, 1226–1235 (2002).

    CAS  PubMed  Google Scholar 

  8. Todorov, E. Direct cortical control of muscle activation in voluntary arm movements: a model. Nat. Neurosci. 3, 391–398 (2000).

    CAS  PubMed  Google Scholar 

  9. Lillicrap, T. P. & Scott, S. H. Preference distributions of primary motor cortex neurons reflect control solutions optimized for limb biomechanics. Neuron 77, 168–179 (2013).

    CAS  PubMed  Google Scholar 

  10. Ijspeert, A. J., Crespi, A., Ryczko, D. & Cabelguen, J.-M. From swimming to walking with a salamander robot driven by a spinal cord model. Science 315, 1416–1420 (2007).

    ADS  CAS  PubMed  Google Scholar 

  11. Kalidindi, H. T. et al. Rotational dynamics in motor cortex are consistent with a feedback controller. eLife 10, e67256 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  12. Georgopoulos, A. P., Kalaska, J. F., Caminiti, R. & Massey, J. T. On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex. J. Neurosci. 2, 1527–1537 (1982).

    CAS  PubMed  PubMed Central  Google Scholar 

  13. Evarts, E. V. Relation of pyramidal tract activity to force exerted during voluntary movement. J. Neurophysiol. 31, 14–27 (1968).

    CAS  PubMed  Google Scholar 

  14. Ashe, J. Force and the motor cortex. Behav. Brain Res. 87, 255–269 (1997).

    CAS  PubMed  Google Scholar 

  15. Kalaska, J. F. From intention to action: motor cortex and the control of reaching movements. Adv. Exp. Med. Biol. 629, 139–178 (2009).

    PubMed  Google Scholar 

  16. Churchland, M. M. & Shenoy, K. V. Temporal complexity and heterogeneity of single-neuron activity in premotor and motor cortex. J. Neurophysiol. 97, 4235–4257 (2007).

    PubMed  Google Scholar 

  17. Yamins, D. L. K. et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc. Natl Acad. Sci. USA 111, 8619–8624 (2014).

    ADS  CAS  PubMed  PubMed Central  Google Scholar 

  18. Kar, K., Kubilius, J., Schmidt, K., Issa, E. B. & DiCarlo, J. J. Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nat. Neurosci. 22, 974–983 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  19. Khaligh-Razavi, S.-M. & Kriegeskorte, N. Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comput. Biol. 10, e1003915 (2014).

    PubMed  PubMed Central  Google Scholar 

  20. Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V. & McDermott, J. H. A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron 98, 630–644 (2018).

    CAS  PubMed  Google Scholar 

  21. Wang, P. Y., Sun, Y., Axel, R., Abbott, L. F. & Yang, G. R. Evolving the olfactory system with machine learning. Neuron 109, 3879–3892 (2021).

    CAS  PubMed  Google Scholar 

  22. Singh, S. H., van Breugel, F., Rao, R. P. N. & Brunton, B. W. Emergent behaviour and neural dynamics in artificial agents tracking odour plumes. Nat. Mach. Intell. 5, 58–70 (2023).

    PubMed  PubMed Central  Google Scholar 

  23. Haesemeyer, M., Schier, A. F. & Engert, F. Convergent temperature representations in artificial and biological neural networks. Neuron 103, 1123–1134.e6 (2019).

    CAS  PubMed  PubMed Central  Google Scholar 

  24. Mante, V., Sussillo, D., Shenoy, K. V. & Newsome, W. T. Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature 503, 78–84 (2013).

    ADS  CAS  PubMed  PubMed Central  Google Scholar 

  25. Higgins, I. et al. Unsupervised deep learning identifies semantic disentanglement in single inferotemporal face patch neurons. Nat. Commun. 12, 6456 (2021).

    ADS  CAS  PubMed  PubMed Central  Google Scholar 

  26. Banino, A. et al. Vector-based navigation using grid-like representations in artificial agents. Nature 557, 429–433 (2018).

    ADS  CAS  PubMed  Google Scholar 

  27. Cueva, C. J. & Wei, X.-X. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. In Proc. 6th International Conference on Learning Representations (ICLR, 2018).

  28. Grillner, S. et al. in Progress in Brain Research, Vol. 165 (eds Cisek, P. et al.) 221��234 (Elsevier, 2007).

  29. Knüsel, J., Crespi, A., Cabelguen, J.-M., Ijspeert, A. J. & Ryczko, D. Reproducing five motor behaviors in a salamander robot with virtual muscles and a distributed CPG controller regulated by drive signals and proprioceptive feedback. Front. Neurorobot. 14, 604426 (2020).

    PubMed  PubMed Central  Google Scholar 

  30. Michaels, J. A., Schaffelhofer, S., Agudelo-Toro, A. & Scherberger, H. A goal-driven modular neural network predicts parietofrontal neural dynamics during grasping. Proc. Natl Acad. Sci. USA 117, 32124–32135 (2020).

    ADS  CAS  PubMed  PubMed Central  Google Scholar 

  31. Sussillo, D., Churchland, M. M., Kaufman, M. T. & Shenoy, K. V. A neural network that finds a naturalistic solution for the production of muscle activity. Nat. Neurosci. 18, 1025–1033 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  32. Chiel, H. J. & Beer, R. D. The brain has a body: adaptive behavior emerges from interactions of nervous system, body and environment. Trends Neurosci. 20, 553–557 (1997).

    CAS  PubMed  Google Scholar 

  33. Scott, S. H. & Loeb, G. E. The computation of position sense from spindles in mono- and multiarticular muscles. J. Neurosci. 14, 7529–7540 (1994).

    CAS  PubMed  PubMed Central  Google Scholar 

  34. Latash, M. L., Scholz, J. P. & Schöner, G. Motor control strategies revealed in the structure of motor variability. Exerc. Sport Sci. Rev. 30, 26–31 (2002).

    PubMed  Google Scholar 

  35. Dunn, T. W. et al. Geometric deep learning enables 3D kinematic profiling across species and environments. Nat. Methods 18, 564–573 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  36. Mimica, B., Dunn, B. A., Tombaz, T., Bojja, V. P. T. N. C. S. & Whitlock, J. R. Efficient cortical coding of 3D posture in freely behaving rats. Science 362, 584–589 (2018).

    ADS  CAS  PubMed  Google Scholar 

  37. Markowitz, J. E. et al. The striatum organizes 3D behavior via moment-to-moment action selection. Cell 174, 44–58.e17 (2018).

    CAS  PubMed  PubMed Central  Google Scholar 

  38. Klaus, A. et al. The spatiotemporal organization of the striatum encodes action space. Neuron 95, 1171–1180.e7 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  39. Mimica, B. et al. Behavioral decomposition reveals rich encoding structure employed across neocortex in rats. Nat. Commun. 14, 3947 (2023).

    ADS  CAS  PubMed  PubMed Central  Google Scholar 

  40. Marshall, J. D. et al. Continuous whole-body 3D kinematic recordings across the rodent behavioral repertoire. Neuron 109, 420–437.e8 (2021).

    CAS  PubMed  Google Scholar 

  41. Berman, G. J., Choi, D. M., Bialek, W. & Shaevitz, J. W. Mapping the stereotyped behaviour of freely moving fruit flies. J. R. Soc. Interface 11, 20140672 (2014).

    PubMed  PubMed Central  Google Scholar 

  42. Klibaite, U. et al. Deep phenotyping reveals movement phenotypes in mouse neurodevelopmental models. Mol. Autism 13, 12 (2022).

    PubMed  PubMed Central  Google Scholar 

  43. Pereira, T. D. et al. Fast animal pose estimation using deep neural networks. Nat. Methods 16, 117–125 (2018).

    PubMed  PubMed Central  Google Scholar 

  44. Wu, T., Tassa, Y., Kumar, V., Movellan, J. & Todorov, E. STAC: simultaneous tracking and calibration. In Proc. 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids) 469–476 (IEEE, 2013).

  45. Peng, X. B., Ma, Z., Abbeel, P., Levine, S. & Kanazawa, A. AMP: adversarial motion priors for stylized physics-based character control. ACM Trans. Graph. 40, 1–20 (2021).

  46. Fussell, L., Bergamin, K. & Holden, D. SuperTrack: motion tracking for physically simulated characters using supervised learning. ACM Trans. Graph. 40, 1–13 (2021).

    Google Scholar 

  47. Dhawale, A. K., Wolff, S. B. E., Ko, R. & Ölveczky, B. P. The basal ganglia control the detailed kinematics of learned motor skills. Nat. Neurosci. 24, 1256–1269 (2021).

    CAS  PubMed  PubMed Central  Google Scholar 

  48. Kriegeskorte, N., Mur, M. & Bandettini, P. Representational similarity analysis - connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008).

    PubMed  PubMed Central  Google Scholar 

  49. Jordan, M. I. & Rumelhart, D. E. Internal world models and supervised learning. In Proc. 8th International Workshop on Machine Learning (eds Birnbaum, L. A. & Collins, G. C.) 70–74 (Morgan Kaufmann, 1991).

  50. Nagabandi, A., Kahn, G., Fearing, R. S. & Levine, S. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. Preprint at https://arxiv.org/abs/1708.02596 (2017).

  51. Valero-Cuevas, F. J., Venkadesan, M. & Todorov, E. Structured variability of muscle activations supports the minimal intervention principle of motor control. J. Neurophysiol. 102, 59–68 (2009).

    PubMed  PubMed Central  Google Scholar 

  52. Diedrichsen, J., Shadmehr, R. & Ivry, R. B. The coordination of movement: optimal feedback control and beyond. Trends Cogn. Sci. 14, 31–39 (2010).

    PubMed  Google Scholar 

  53. Flash, T. & Hogan, N. The coordination of arm movements: an experimentally confirmed mathematical model. J. Neurosci. 5, 1688–1703 (1985).

    CAS  PubMed  PubMed Central  Google Scholar 

  54. Harris, C. M. & Wolpert, D. M. Signal-dependent noise determines motor planning. Nature 394, 780–784 (1998).

    ADS  CAS  PubMed  Google Scholar 

  55. Wolpert, D. M. Probabilistic models in human sensorimotor control. Hum. Mov. Sci. 26, 511–524 (2007).

    PubMed  PubMed Central  Google Scholar 

  56. Lai, L. & Gershman, S. J. in Psychology of Learning and Motivation, Vol. 74 (ed. Federmeier, K. D.) Ch. 5 (Academic Press, 2021).

  57. Ramalingasetty, S. T. et al. A whole-body musculoskeletal model of the mouse IEEE Access 9, 163861–163881 (2021).

    PubMed  PubMed Central  Google Scholar 

  58. Golub, M., Chase, S. & Yu, B. Learning an internal dynamics model from control demonstration. In Proc. 30th International Conference on Machine Learning (eds Dasgupta, S. & McAllester, D.) 606–614 (PMLR, 2013).

  59. Shidara, M., Kawano, K., Gomi, H. & Kawato, M. Inverse-dynamics model eye movement control by Purkinje cells in the cerebellum. Nature 365, 50–52 (1993).

    ADS  CAS  PubMed  Google Scholar 

  60. Kawai, R. et al. Motor cortex is required for learning but not for executing a motor skill. Neuron 86, 800–812 (2015).

    CAS  PubMed  PubMed Central  Google Scholar 

  61. Faisal, A. A., Selen, L. P. J. & Wolpert, D. M. Noise in the nervous system. Nat. Rev. Neurosci. 9, 292–303 (2008).

    CAS  PubMed  PubMed Central  Google Scholar 

  62. Dhawale, A. K. et al. Automated long-term recording and analysis of neural activity in behaving animals. eLife 6, e27702 (2017).

    PubMed  PubMed Central  Google Scholar 

  63. Chung, J. E. et al. A fully automated approach to spike sorting. Neuron 95, 1381–1394.e6 (2017).

    CAS  PubMed  PubMed Central  Google Scholar 

  64. Merel, J. et al. Hierarchical visuomotor control of humanoids. In Proc. 7th International Conference on Learning Representations (ICLR, 2019).

  65. Chentanez, N., Müller, M., Macklin, M., Makoviychuk, V. & Jeschke, S. Physics-based motion capture imitation with deep reinforcement learning. In Proc. 11th Annual International Conference on Motion, Interaction, and Games 1–10 (ACM, 2018).

  66. Abdolmaleki, A. et al. A distributional view on multi-objective policy optimization. In Proc. 37th International Conference on Machine Learning (eds Daumé, H. & Singh, A.) 11–22 (PMLR, 2020).

  67. Francis Song, H. et al. V-MPO: on-policy maximum a posteriori policy optimization for discrete and continuous control. In Proc. 8th International Conference on Learning Representations (2020).

  68. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In Proc. 3rd International Conference on Learning Representations (eds Bengio, Y. & LeCun, Y.) (2015).

  69. Maas, A. L., Hannun, A. Y. & Ng, A. Y. Rectifier nonlinearities improve neural network acoustic models. In Proc. 30th International Conference on Machine Learning (ICML) (2013).

  70. Seabold, S. & Perktold, J. Statsmodels: econometric and statistical modeling with Python. In Proc. 9th Python in Science Conference (eds van der Walt, S. & Millman, J.) 92–96 (SciPy, 2010); https://doi.org/10.25080/majora-92bf1922-011.

  71. Diedrichsen, J. et al. Comparing representational geometries using whitened unbiased-distance-matrix similarity. Preprint at https://arxiv.org/abs/2007.02789 (2020).

  72. Schütt, H. H., Kipnis, A. D., Diedrichsen, J. & Kriegeskorte, N. Statistical inference on representational geometries. Preprint at https://arxiv.org/abs/2112.09200 (2021).

  73. Nili, H. et al. A toolbox for representational similarity analysis. PLoS Comput. Biol. 10, e1003553 (2014).

    PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank M. Shad and the team at Harvard Research Computing for their technical support. We are grateful to S. Wolff, K. Hardcastle and J. Casas for their support with experimental procedures. We would also like to thank S. Escola for feedback on our manuscript. This work was supported by a National Institutes of Health D-SPAN Award (1F99NS125834-01A1) to D.A. and National Institutes of Health grants (nos. R01NS099323, R01GM136972) to B.P.Ö. The illustration of the rat in Fig. 1a was hand drawn by D.A. from a model licensed from Biosphera3D.

Author information

Authors and Affiliations

Authors

Contributions

D.A., J.M., J.D.M., G.W., M.B. and B.P.Ö. conceived the project idea. D.A., U.K. and A.G. carried out the experiments. D.A. and U.K. processed the data. J.M. and L.H. trained the inverse dynamics models. D.A., J.M., J.D.M. and Y.T. contributed to the biomechanical model. D.A. analysed the data. D.A., J.M., J.D.M., L.H., M.B. and B.P.Ö. contributed to the interpretation of the results. D.A., J.M., J.D.M. and B.P.Ö. wrote the manuscript with input from all authors.

Corresponding authors

Correspondence to Diego Aldarondo or Bence P. Ölveczky.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature thanks Jonathan Whitlock and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended Data Fig. 1 Recording neural activity in freely behaving rats.

A) Schematic of custom 128-channel tetrode drive. B, Tetrodes record electrical events of several putative neurons from the DLS or MC. Shown are recordings from a tetrode in DLS. C) Individual putative cells are extracted based on their unique spike waveforms using a custom spike-sorting software, FAST. D) Tetrodes allows for the recording of hundreds of putative single units simultaneously. E-F) Representative examples of Nissl-stained brain slices from animals with electrophysiological implants in DLS and MC. Red ellipses indicate the lesions remaining from the tetrode implants. G, Dorsal view denoting the position of implants for DLS and MC. The position of the implant with the dashed circle could not be verified with histology as the recording headstage was dislodged prior to electric lesion. The position was instead estimated using scarring at the cortical surface and the recorded depth of implantation. The other implants were verified with electric lesions or scarring from the implant tip. H) Coronal plane indicating the location of implants in the DLS across 3 animals. I) Coronal plane indicating the location of implants in MC across 3 animals.

Extended Data Fig. 2 High fidelity 3D pose estimation and skeletal registration.

A) In DANNCE, a 3D U-Net processes multi-view images to estimate the positions of 23 3D keypoints across the rat’s body. B, DANNCE keypoint estimates show high concordance with manual annotations, deviating from manual labels to a similar degree as repeated manual annotations of the same testing frames. C) Visualization of median DANNCE keypoint discrepancy relative to manual annotation. Grey circles indicate the bounds of the sphere with radius equal to the median keypoint discrepancy for each keypoint. D) Schematic depicting the relevant variables in STAC. STAC operates by jointly optimizing a set of offsets relating the skeletal model to different keypoints and the pose of the model in each frame. E) STAC registration is highly accurate across body parts and F) across different behaviours. For all boxplots in this figure, coloured lines indicate the median, boxes indicate the interquartile range, and whiskers indicate the 10th and 90th percentiles.

Extended Data Fig. 3 Comparing imitation performance for held-out data across different classes of control networks.

A) The proportion of episodes exceeding a given duration for the four classes of controllers. Results for each class are averaged across models with all KL regularization coefficients for that class. B, Violin plots showing the distribution of rewards by each model class on the held-out testing set. Models with LSTM decoders outperform other classes. C) Average reward as a function of the center of mass speed for each class of controller. LSTM models outperform other model classes across all speeds, but especially at slow speeds. D) Box plots denoting the distribution of rewards for each model class as a function of behavior category. LSTM models outperform other classes across all behavior, but especially those with slow center of mass speed. White lines indicate the median, box limits indicate the interquartile range, box whiskers indicate the 10th and 90th percentiles. E) The proportion of episodes exceeding a given duration for models with LSTM decoders across all KL regularization coefficients. Models with higher KL regularization are generally less robust than those with lower KL regularization, consistent with an increase in latent noise. F) Violin plots denoting the distribution of rewards on held-out natural behavior for each model as a function of KL regularization. Increasing the KL regularization coefficient marginally decreases the reward distribution of the models. White lines indicate the median. G, We trained five models with different reference window lengths using an LSTM decoder with a KL regularization of 1e-4. Violin plots denote the distribution of rewards on held-out natural behavior for each model. Models with reference windows of length 5 or shorter exhibit comparable performance, while a reference window of 10 exhibits poorer performance. Grey lines indicate the quartiles. H) The proportion of episodes exceeding a given duration. Models with longer reference window length are generally more robust than those with shorter reference window lengths, with the most robust model being that with a reference window length of 5. Shaded regions indicate the standard error of the mean over sessions. I) The distribution of joint angles during imitation closely match those of STAC-registered skeletal models during imitation. Data is from a model with an LSTM decoder and a KL regularization of 1e-4. Box centers indicate the median, box limits indicate the interquartile range, box whiskers indicate the maximum or minimum values up to 1.5 times the interquartile range from the box limits.

Extended Data Fig. 4 Neurons in the DLS and MC encode posture across many body parts to a degree consistent with previous reports during unrestrained behavior.

A, C) Proportion of neurons in DLS and MC best predicted by each feature class. B, D) Violin plots showing the distribution of cross-validated log-likelihood ratios (CV-LLR) of GLMs trained to predict spike counts using different feature classes. E, F) Box plots showing the distribution of deviance-ratio pseudo r-squared values of GLMs trained to predict spike counts using different feature classes. White lines indicate the median, boxes indicate the interquartile range, and whiskers indicate the 10th and 90th percentiles. G, H) Empirical cumulative distribution functions denoting the proportion of neurons in DLS and MC with peak GLM predictivity below a given pseudo r-squared value. The distributions resemble previous reports in rats during spontaneous behavior42.

Extended Data Fig. 5 Encoding properties are similar across striatal cell types.

A-C) Proportion of neurons in DLS and MC best predicted by each feature class for each cell type. D-F) Box plots showing the distribution of cross-validated log-likelihood ratios relative to a mean firing rate model for GLMs trained to predict spike counts using different feature classes. White lines indicate the median, boxes indicate the interquartile range, and whiskers indicate the 10th and 90th percentiles. G-H) Comparison of the best computational feature derived from the network and representational feature GLM CV-LLRs for each neuron. GLMs based on the inverse dynamics models (computational features) outperform those based on representational features for the majority of classified neurons for all cell types (p < .001, permutation test).

Extended Data Fig. 6 Neurons in the DLS and MC encode future movement during natural behavior.

We trained GLMs to predict neural activity from measurable features of movement and from features of the ANN controllers while introducing time lags ranging from -1000 ms to 300 ms between neural activity and the features. A) Histograms depicting the distribution of time lags for maximally predictive GLMs when using joint angle predictors. Time lags less than zero correspond to neurons whose future movements better predict neural activity (premotor), while time lags greater than zero correspond to neurons whose past movements best predict neural activity (postmotor). B) CVLLR relative to models trained with a time lag of 0 ms averaged across neurons. Shaded regions indicate the standard error of the mean. The peak average CVLLR occurs at -200 ms for all cell types. C, D) Same as A-B, except using features from the inverse dynamics model (LSTM hidden layer 1) as GLM predictors for a model with an LSTM decoder and a KL regularization of 1e-4. Peak predictivity occurs closer to a time lag of zero, consistent with the network’s representation of desired future state and inverse dynamics. E, F) Same as A-B for neurons in MC. G, H) Same as C-D for neurons in MC.

Extended Data Fig. 7 Comparing imitation performance and neural predictivity of models trained to control bodies of different masses.

A) We trained five models with an LSTM decoder and a KL regularization of 1e-4 to control bodies of different masses. Violin plots denote the distribution of rewards on held-out natural behavior for each model. Several models controlling bodies with masses other than the standard mass exhibited reduced performance. White lines indicate medians. B) The proportion of episodes exceeding a given duration. Shaded regions indicate S.E.M across individuals. C-D) Box plots depicting the distribution of cross-validated log-likelihood ratios across neurons of GLMs trained to predict neural activity from network features. The CVLLR for each neuron is expressed relative to the likelihood of a GLM trained to predict neural activity using network features from the standard mass model. Values greater than zero imply a model more predictive of neural activity than those derived from the standard mass model, and vice versa. White lines indicate the median, box limits indicate the quartiles, whiskers indicate the 10th and 90th percentiles. Stars indicate that a greater proportion of neurons are better predicted by GLMs trained using features from the standard mass model than from the alternative mass model (Bonferroni corrected, α = .05, permutation test). E-F) Average WUC similarity between RDMs derived from network layers and neural activity in DLS or MC. Error bars indicate S.E.M across individuals. Arrows indicate significantly different similarity distributions across animals (Benjamini-Hochberg corrected, false discovery rate α = .05, one-sided t-test).

Extended Data Fig. 8 Comparing imitation performance and neural predictivity of models trained to control bodies of the same total mass with different head masses.

A) We trained five models with an LSTM decoder and a KL regularization of 1e-4 to control bodies of the same total mass with different relative masses between the head and the rest of the body. Violin plots denote the distribution of rewards on held-out natural behavior for each model. Several models controlling bodies with masses other than the standard mass exhibited reduced performance. White lines indicate medians. B) The proportion of episodes exceeding a given duration. Shaded regions indicate S.E.M across individuals. C-D) Box plots depicting the distribution of cross-validated log-likelihood ratios across neurons of GLMs trained to predict neural activity from network features. The CVLLR for each neuron is expressed relative to the likelihood of a GLM trained to predict neural activity using network features from the standard mass model. Values greater than zero imply a model more predictive of neural activity than those derived from the standard mass model, and vice versa. White lines indicate the median, box limits indicate the quartiles, whiskers indicate the 10th and 90th percentiles. Stars indicate that a greater proportion of neurons are better predicted by GLMs trained using features from the standard mass model than from the alternative mass model (Bonferroni corrected, α = .05, permutation test). E-F) Average WUC similarity between RDMs derived from network layers and neural activity in DLS or MC. Error bars indicate S.E.M across individuals. Arrows indicate significantly different similarity distributions across animals (Benjamini-Hochberg corrected, false discovery rate α = .05, one-sided t-test).

Extended Data Fig. 9 The representational structures of DLS and MC resemble an inverse model more than alternative control models.

A) To compare the representational structure of neural activity in DLS and MC across different candidate computational models we used B) rollouts from an inverse model to collect state-action pairs to train C) forward and sequential models with supervised learning. D-F) Across-subject representational similarity between control models and neural activity. The latent representation of an inverse model more closely resembles the structure of neural activity in DLS and MC than the latent representation of forward or sequential models. G-I) The latent variability of an inverse model better predicts the structure of neural variability than representational models. Error bars indicate S.E.M. Icicles and dew drops indicate significant differences from the noise ceiling and zero (Bonferroni corrected, α = .05, one-sided t-test). Gray bars indicate the estimated noise ceiling of the true model. Arrows indicate significant differences between features (Benjamini-Hochberg corrected, false discovery rate α = .05, one-sided t-test). Points indicate individual animals.

Extended Data Fig. 10 Inverse dynamics models predict putative single-unit neural activity better than alternative control models and feedback.

A-B) Box plots showing the distribution of cross-validated log-likelihood ratios (CV-LLR) relative to mean firing-rate models of GLMs trained to predict spike counts using different feature classes. White lines indicate the median, boxes indicate the interquartile range, and whiskers indicate the 10th and 90th percentiles.

Supplementary information

Supplementary Information

This file contains Supplementary Discussion and Tables 1–3.

Reporting Summary

Supplementary Video 1

Overview of the MIMIC pipeline. The MIMIC pipeline consists of multicamera video acquisition.

Supplementary Video 2

Accurate 3D pose estimation with DANNCE. We used DANNCE to estimate the 3D pose of freely moving rats from multicamera recordings. This video depicts the DANNCE keypoint estimates overlain atop the original video recordings from all six cameras. Keypoint estimates are accurate across a wide range of behaviours.

Supplementary Video 3

Accurate skeletal registration with STAC. We used a custom implementation of STAC.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aldarondo, D., Merel, J., Marshall, J.D. et al. A virtual rodent predicts the structure of neural activity across behaviours. Nature (2024). https://doi.org/10.1038/s41586-024-07633-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1038/s41586-024-07633-4

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing