Alberto Lumbreras

State-space models are a low-complexity alternative to transformers for encoding long sequences and capturing long-term dependencies. We propose LOCOST: an encoder-decoder architecture based on state-space models for conditional text generation with long context inputs. With a computational complexity of 𝒪(L log L), this architecture can handle significantly longer sequences than state-of-the-art models that are based on sparse attention patterns. We evaluate our model on a series of long… Show more

State-space models are a low-complexity alternative to transformers for encoding long sequences and capturing long-term dependencies. We propose LOCOST: an encoder-decoder architecture based on state-space models for conditional text generation with long context inputs. With a computational complexity of 𝒪(L log L), this architecture can handle significantly longer sequences than state-of-the-art models that are based on sparse attention patterns. We evaluate our model on a series of long document abstractive summarization tasks. The model reaches a performance level that is 93-96% comparable to the top-performing sparse transformers of the same size while saving up to 50% memory during training and up to 87% during inference. Additionally, LOCOST effectively handles input texts exceeding 600K tokens at inference time, setting new state-of-the-art results on full-book summarization and opening new perspectives for long input processing. Show less

Data-to-text (D2T) and text-to-data (T2D) are dual tasks that convert structured data, such as graphs or tables into fluent text, and vice versa. These tasks are usually handled separately and use corpora extracted from a single source. Current systems leverage pre-trained language models fine-tuned on D2T or T2D tasks. This approach has two main limitations: first, a separate system has to be tuned for each task and source; second, learning is limited by the scarcity of available corpora. This… Show more

Data-to-text (D2T) and text-to-data (T2D) are dual tasks that convert structured data, such as graphs or tables into fluent text, and vice versa. These tasks are usually handled separately and use corpora extracted from a single source. Current systems leverage pre-trained language models fine-tuned on D2T or T2D tasks. This approach has two main limitations: first, a separate system has to be tuned for each task and source; second, learning is limited by the scarcity of available corpora. This paper considers a more general scenario where data are available from multiple heterogeneous sources. Each source, with its specific data format and semantic domain, provides a non-parallel corpus of text and structured data. We introduce a variational autoencoder model with disentangled style and content variables that allows us to represent the diversity that stems from multiple sources of text and data. Our model is designed to handle the tasks of D2T and T2D jointly. We evaluate our model on several datasets, and show that by learning from multiple sources, our model closes the performance gap with its supervised single-source counterpart and outperforms it in some cases. Show less

Binary data matrices can represent many types of data such as social networks, votes, or gene expression. In some cases, the analysis of binary matrices can be tackled with nonnegative matrix factorization (NMF), where the observed data matrix is approximated by the product of two smaller nonnegative matrices. In this context, probabilistic NMF assumes a generative model where the data is usually Bernoulli-distributed. Often, a link function is used to map the factorization to the [0, 1] range,… Show more

Binary data matrices can represent many types of data such as social networks, votes, or gene expression. In some cases, the analysis of binary matrices can be tackled with nonnegative matrix factorization (NMF), where the observed data matrix is approximated by the product of two smaller nonnegative matrices. In this context, probabilistic NMF assumes a generative model where the data is usually Bernoulli-distributed. Often, a link function is used to map the factorization to the [0, 1] range, ensuring a valid Bernoulli mean parameter. However, link functions have the potential disadvantage to lead to uninterpretable models. Mean-parameterized NMF, on the contrary, overcomes this problem. We propose a unified framework for Bayesian mean-parameterized nonnegative binary matrix factorization models (NBMF). We analyze three models which correspond to three possible constraints that respect the mean-parameterization without the need for link functions. Furthermore, we derive a novel collapsed Gibbs sampler and a collapsed variational algorithm to infer the posterior distribution of the factors. Next, we extend the proposed models to a nonparametric setting where the number of used latent dimensions is automatically driven by the observed data. We analyze the performance of our NBMF methods in multiple datasets for different tasks such as dictionary learning and prediction of missing data. Experiments show that our methods provide similar or superior results than the state of the art, while automatically detecting the number of relevant components. Show less

We present novel understandings of the Gamma-Poisson (GaP) model, a probabilistic matrix factorization model for count data. We show that GaP can be rewritten free of the score/activation matrix. This gives us new insights about the estimation of the topic/dictionary matrix by maximum marginal likelihood estimation. In particular, this explains the robustness of this estimator to over-specified values of the factorization rank and in particular its ability to automatically prune spurious… Show more

We present novel understandings of the Gamma-Poisson (GaP) model, a probabilistic matrix factorization model for count data. We show that GaP can be rewritten free of the score/activation matrix. This gives us new insights about the estimation of the topic/dictionary matrix by maximum marginal likelihood estimation. In particular, this explains the robustness of this estimator to over-specified values of the factorization rank and in particular its ability to automatically prune spurious dictionary columns, as empirically observed in previous work. The marginalization of the activation matrix leads in turn to a new Monte-Carlo Expectation-Maximization algorithm with favorable properties. Show less

Some structural characteristics of online discussions have been successfully modeled in the recent years. When parameters of these models are properly estimated, the models are able to generate synthetic discussions that are structurally similar to the real discussions. A common aspect of these models is that they consider that all users behave according to the same model. In this paper, we combine a growth model with an Expectation–Maximization algorithm that finds different parameters for… Show more

Some structural characteristics of online discussions have been successfully modeled in the recent years. When parameters of these models are properly estimated, the models are able to generate synthetic discussions that are structurally similar to the real discussions. A common aspect of these models is that they consider that all users behave according to the same model. In this paper, we combine a growth model with an Expectation–Maximization algorithm that finds different parameters for different latent groups of users. We use this method to find the different roles that coexist in the community. Moreover, we analyze whether we can predict users behaviors based on their roles. Indeed, we show that predictions are improved for some of the roles when compared with a simple growth model. Show less

We present a dual-view mixture model to cluster users based on their features and
latent behavioral functions. Every component of the mixture model represents a probability
density over a feature view for observed user attributes and a behavior view for latent behavioral
functions that are indirectly observed through user actions or behaviors. Our task is to infer the
groups of users as well as their latent behavioral functions. We also propose a non-parametric
version based on a… Show more

We present a dual-view mixture model to cluster users based on their features and
latent behavioral functions. Every component of the mixture model represents a probability
density over a feature view for observed user attributes and a behavior view for latent behavioral
functions that are indirectly observed through user actions or behaviors. Our task is to infer the
groups of users as well as their latent behavioral functions. We also propose a non-parametric
version based on a Dirichlet Process to automatically infer the number of clusters. We test the
properties and performance of the model on a synthetic dataset that represents the participation
of users in the threads of an online forum. Experiments show that dual-view models outperform
single-view ones when one of the views lacks information. Show less

Analyser les rôles dans les communautés virtuelles nous permet de mieux comprendre, voire de prédire, le comportement individuel des internautes. Bien que de nombreuses approches aient été proposées, on constate un manque de généralisation des méthodes existantes et des résultats obtenus. Dans ce papier, nous passons en revue quelques théories développées à propos des rôles sociaux et nous cherchons une définition compatible à une automatisation par les machines de la détection des rôles joués… Show more

Analyser les rôles dans les communautés virtuelles nous permet de mieux comprendre, voire de prédire, le comportement individuel des internautes. Bien que de nombreuses approches aient été proposées, on constate un manque de généralisation des méthodes existantes et des résultats obtenus. Dans ce papier, nous passons en revue quelques théories développées à propos des rôles sociaux et nous cherchons une définition compatible à une automatisation par les machines de la détection des rôles joués par les individus dans des fils de discussions sur internet. Nous analysons ensuite le site Web IMDb afin d’illustrer notre discours. Show less

El estudio recogido en este documento corresponde a una experiencia de investigación colectiva del grupo @datAnalysis15M sobre el movimiento #15M, también denominado movimiento de l@s indignad@s o #spanishrevolution. El estudio, apoyado en la experiencia interna del movimiento y en literatura previa para establecer el marco teórico, se ha realizado a través de distintos métodos experimentales de análisis de datos, desde el análisis de redes sociales y de sistemas complejos al procesamiento de… Show more

El estudio recogido en este documento corresponde a una experiencia de investigación colectiva del grupo @datAnalysis15M sobre el movimiento #15M, también denominado movimiento de l@s indignad@s o #spanishrevolution. El estudio, apoyado en la experiencia interna del movimiento y en literatura previa para establecer el marco teórico, se ha realizado a través de distintos métodos experimentales de análisis de datos, desde el análisis de redes sociales y de sistemas complejos al procesamiento de lenguaje y el análisis de emociones. La investigación atiende a los factores de antecedentes, gestación y desencadenantes, al mismo tiempo que muestra la explosión y desarrollo del #15M. También se explica la importancia de la relación entre la multiplicación de las prácticas tecnopolíticas a través de redes sociales humanas y digitales y las prácticas resultantes de organización, acción y comunicación colectiva. En el estudio se consideran la toma del espacio urbano y la explosión emocional de indignación y empoderamiento que, junto a estas prácticas, dan lugar a un sistema-red autónomo y autoorganizado, en otras palabras, una multitud conectada inteligente. Show less

Recommender systems have been strongly researched within the last decade. With the arising and popularization of digital social networks a new field has been opened for social recommendations. Considering the network topology, users interactions, or estimating trust between users are some of the new strategies that recommender systems can take into account in order to adapt their techniques to these new scenarios. We introduce MarkovTrust, a way to infer trust from Twitter interactions and to… Show more

Recommender systems have been strongly researched within the last decade. With the arising and popularization of digital social networks a new field has been opened for social recommendations. Considering the network topology, users interactions, or estimating trust between users are some of the new strategies that recommender systems can take into account in order to adapt their techniques to these new scenarios. We introduce MarkovTrust, a way to infer trust from Twitter interactions and to compute trust between distant users. MarkovTrust is based on Markov chains, which makes it simple to be implemented and computationally efficient. We study the properties of this trust metric and study its application in a recommender system of tweets. Show less