Multimodal Feature Learning for Understanding Consumer Produced Multimedia Data

ICSI is working with LLNL on ongoing work on feature extraction and analytic techniques that map raw data from multiple input modalities (e.g., video, images, text) into a joint semantic space. This requires cutting edge research in multiple modalities, as well as in the mathematical methods to learn the semantic mappings. ICSI's role is developing computational algorithms, systems, and methods to handle content composed of multiple types of data (such as news videos and social network posts) and performing research on the scientific problems arising from the complementary nature of different data sources, each of which captures only partial information. The development of these technologies will ultimately help analysts find patterns of significance in large unlabeled multimodal datasets.

Funding provided by LLNL