DeepSaber is a deep-learning system that automatically generates beatmaps — the note choreography synced to a song — for the VR rhythm game Beat Saber. The whole project rests on one insight: a beatmap is a sentence. Each move is a word, so the obvious tool is NLP, not hand-tuned heuristics.
Beatmaps as sentences
I encoded every beat element as <hand><y><x><rotation> and concatenated them into “actions” — a discrete vocabulary of about 2,450 unique moves. Those actions were then embedded exactly like words, with Word2Vec and FastText via Gensim. On the audio side I fed in MFCC audio features (13 coefficients, 25ms/10ms), with a small negative time-shift so notes land pre-emptively — you swing before the beat arrives.
I compared six models: a stacked-LSTM baseline, a reproduction of DDC (Dance Dance Convolution), a custom model (per-stream Conv1D preprocessing, stacked LSTMs, RectifiedAdam/Lookahead, label smoothing, and a cosine-similarity loss on the embeddings), and several Keras-Tuner-searched variants — including a multi-LSTM with randomized cross-layer connections.
How it was judged, and how it did
Because actions live in an embedding space, raw accuracy is a poor measure. So evaluation also used embedding-based metrics — cosine distance and MAE between predicted and true action vectors — alongside perplexity and top-k accuracy. The multi-LSTM reached perplexity ~14.87 and top-5 accuracy ~0.69, against the DDC baseline’s ~16.0 perplexity. Generated maps were compared against Beat Sage and OxAI’s “DeepSaberv2”, on a custom dataset of ~700 maps (~38h of audio) across Normal/Hard/Expert difficulties.
Code is on GitHub, and the full write-up is in the thesis PDF. Co-authored with Jan Pokorny.