Audio-Animated 3D Piano: On-Device AI Pitch Detection & MIDI Sync

Effortlessly turn piano audio into realistic 3D animation. Beautiful hand-crafted piano model integrated with on-device AI pitch detection & MIDI import. Fully plug&play. Includes Timeline support.

Technical Features & Specs:

• AI Pitch Detection model uses Unity Sentis/Inference Engine for cross-platform local execution.
• Piano key animation requires Unity Audio Source for audio playback
• Audio Formats: formats supported by Unity Audio Clips.
• Included samples: 24bit 48KHz Stereo FLAC live grand piano recording.
• Supoprted OS and platforms:
• Windows, MacOS, iOS, Web.
• Android: Not tested.

Limitations:

• AI Pitch-detection is optimized specifically for smooth visual playback of solo piano audio. It may cause minor missed or additional key presses.
• Audio-to-Music-Note and MIDI conversion scripts run in Editor mode only (generated animation events are stored for runtime playback).
• Streaming audio pitch detection is not supported.
• Live real-time MIDI input stream is not supported.
• MIDI importer ingests all tracks/channels; multi-track MIDI files should be split prior to import for targeted track channel animation.
• On Web platform, Audio Source Pitch setting other than 1 is not supported.

Third-Party Components:

• DryWetMIDI: licensed under MIT (Commercial use allowed).
• Basic-Pitch ONNX Model: licensed under Apache 2.0 (Commercial use allowed).
• Quintessential Font, licensed under SIL Open Font License (Commercial use allowed).

Designed for game developers, interactive storytellers, 3D and music artists, this complete package (3D model, Editor tools, Sentis AI integration) provides everything you need to bring realistic, frame-accurate piano performances to your Unity project.

Key Features

Detailed 3D Parlor Grand Piano Model:

• Advanced Rendering: Built for Real-time and Baked Global Illumination with texture & normal maps, and custom lightmap UVs.
• Fully Rigged & Posable: Dimension-accurate model featuring an openable/poseable lid, lid flap, music desk stand, and fallboard. Customizable fallboard text/logo.
• Pre-wired Piano Keys: Individual piano keys pre-configured for instant, note-driven animations out of the box.

Automatic Music Note Converter (Editor Tools)

• AI Audio-to-Note Converter: Uses a fast, local neural network for offline pitch detection. No cloud APIs or subscription fees required.
• MIDI-to-Note File Importer: Import standard MIDI files for 100% precise key animations.
• Inspector Preview & Scrubbing: Scrub and preview note data directly inside the Unity Inspector without entering Play mode.
• Note: AI is specifically optimized for responsive animation timing, and not intended for precise music transcription.

Real-Time Programmatic Controller:

• Frame-Accurate Sync: Automatically synchronizes AudioSource playback with note data to deliver frame-accurate key animations in real time.
• Developer-Friendly API: Full runtime C# control via standard Unity AudioSource workflows.

No-Code Configuration & Control

• All-in-One Prefab: Pre-configured with the piano model, programmatic animation controller, and audio source out of the box.
• Inspector Customization: Easily configure piano pose, audio clips, and music note data directly in the Inspector.
• Cinematic Timeline Support: Includes a custom, fully featured Piano Timeline Track for no-code effortless cinematic sequencing.

Ready-to-Use Demos

• Timeline Demo: Showcases no-code cinematic setup.
• C# Jukebox Demo: Interactive player featuring song selection, playlists, seeking, and real-time speed/pitch controls.
• Audio Samples: Features high-quality, live acoustic grand piano recordings for instant out-of-the-box testing.

100% Handcrafted: Models, artwork, and audio created entirely by human artists.

Questions, comments, and feature requests are welcome!

support@aidevpartner.com