Deep Learning · Multimodal Fusion

Multimodal Sentiment Classification

One model, three senses. Text, image, and audio are encoded by RoBERTa, ViT, and wav2vec2, fused into a single representation, and classified into negative, neutral, or positive sentiment — trained on the MSCTD dialogue dataset.

View source on GitHub Try the playground

Architecture

Late fusion of frozen-vocabulary transformer encoders with a compact MLP head.

Text

RoBERTa-large

Mean-pooled sentence embedding, 1024-d

Vision

DINOv2-large

Scene embedding from the frame, 1024-d

Audio

wav2vec2-base

Mean-pooled waveform features, optional

→

Fusion

Concat → MLP

Negative · Neutral · Positive

Playground

Runs against your own inference server — nothing is uploaded anywhere else. Start it locally, then analyze anything.

Text — optional when an image or audio is provided

This is the best day of my whole life! The package arrived broken and nobody answers my emails. The meeting was moved to Thursday afternoon.

Optional modalities

Image — upload or use your camera

Audio — upload or record your voice

Inference server

Couldn't reach your server. Start it on your machine, then try again:

git clone https://github.com/Nikelroid/multimodal-sentiment-classification
cd multimodal-sentiment-classification
pip install -r requirements.txt
uvicorn app.main:app --port 8000

In this mode the playground talks only to your endpoint — inputs never leave your machine.