Transcription and Diarization - Mattias Kastensson

📸 Application Screenshots

See Audio Scribe AI in action with these screenshots from the actual application interface:

Main Interface & Recording Controls

The main application interface featuring browser recording controls, drag-and-drop file upload, and real-time processing feedback.

Transcription Results & Speaker Diarization

The results interface showing transcribed text with speaker identification, editable speaker names, and the interactive karaoke-style playback feature.

🎯 Key Interface Features Shown

🎙️ Recording Controls

One-click browser recording with real-time duration display and audio format information.

📁 Drag & Drop Upload

Intuitive upload area supporting multiple audio formats, plus video files that are auto-converted to audio.

📊 Processing Visualization

Real-time chunk processing progress with visual indicators and current text preview.

👥 Speaker Diarization

Color-coded speaker segments with editable names and precise timestamps.

🎵 Karaoke Player

Interactive playback with synchronized text highlighting and audio controls.

📤 Export Options

Download results as structured JSON, plain text, or a formatted Word document — with support for custom templates.

Project Overview

Audio Scribe AI is a comprehensive web application that combines the power of KB-whisper (a Swedish-optimized variant of OpenAI Whisper) with Pyannote.audio for advanced speaker diarization. The application runs entirely locally, ensuring privacy and offline processing capabilities. It features a modern, responsive interface with real-time processing feedback and advanced visualization tools.

4 AI Backends

10 Max Speakers

14 Audio & Video Formats

100% Local & Private

Key Features

🎙️ Browser Recording: Record audio directly in your browser with real-time duration display
📁 File Upload: Support for multiple audio formats (MP3, WAV, M4A, FLAC, OGG, WebM) with drag-and-drop
🎬 Video Support: Drop in a video file (MP4, AVI, MKV, MOV, WMV, FLV, MPEG, 3GP) and it's automatically converted to audio before transcription
🤖 AI Transcription: Powered by KB-whisper (Swedish-optimized), OpenAI Whisper, local Whisper models, or a vLLM server
🏠 Local Processing: Complete privacy with offline processing capabilities
⚡ vLLM Support: High-performance transcription via a vLLM server with PagedAttention, OpenAI-compatible API, concurrent request handling, and automatic chunking for large files
🌐 Distributed Processing: Optional standalone Whisper and Pyannote servers so heavy models can run on a separate GPU machine
👥 Speaker Diarization: Automatic speaker identification using Pyannote.audio
✏️ Editable Speaker Names: Rename speakers from "Speaker 1, 2, 3..." to actual names
⏱️ Timestamped Results: Precise start/end times for each speaker segment
🎵 Karaoke Player: Interactive playback with synchronized text highlighting
📤 Export Options: Download results as JSON, plain text, or a formatted Word document (.docx)
📝 Custom Word Templates: Bring your own .docx template to control the exported document's formatting
🎨 Modern UI: Clean, responsive interface with dark theme and visual feedback
⚡ GPU Acceleration: Automatic CUDA support for faster processing
📊 Processing Visualization: Real-time chunk processing progress with visual indicators

Technical Implementation

Audio Scribe AI is built with a modern FastAPI backend and vanilla JavaScript frontend. The system uses a unified service architecture that supports both local and cloud-based AI models, with automatic fallback to mock services for development.

FastAPI Backend Architecture

The backend uses a modular service-oriented architecture with separate services for audio processing, transcription, and diarization:

# backend/app.py - Main FastAPI application
from fastapi import FastAPI, File, UploadFile, HTTPException
from services.unified_whisper_service import UnifiedWhisperService
from services.pyannote_service import PyannoteService
from services.audio_service import AudioService

app = FastAPI(title="Audio Scribe AI", version="1.0.0")

# Initialize services with automatic fallback
whisper_service = UnifiedWhisperService()
pyannote_service = PyannoteService()
audio_service = AudioService()

@app.post("/api/transcribe/{file_id}")
async def transcribe_audio(file_id: str):
    """Transcribe and diarize audio file with real-time progress"""
    # Implementation handles chunked processing with progress updates
      

Unified Whisper Service

The unified service supports both KB-whisper (Swedish-optimized) and standard OpenAI Whisper models with automatic model switching and local processing capabilities:

# services/unified_whisper_service.py
class UnifiedWhisperService:
    def __init__(self):
        self.current_service = None
        self.local_service = None
        self.openai_service = None

    async def transcribe_with_progress(self, audio_path: str,
                                     progress_callback=None):
        """Transcribe with chunked processing and progress updates"""
        chunks = self.split_audio_into_chunks(audio_path)
        results = []

        for i, chunk in enumerate(chunks):
            result = await self.transcribe_chunk(chunk)
            results.append(result)

            if progress_callback:
                await progress_callback({
                    "chunk": i + 1,
                    "total_chunks": len(chunks),
                    "current_text": result.get("text", ""),
                    "progress": (i + 1) / len(chunks) * 100
                })

        return self.combine_chunk_results(results)
      

Pyannote Speaker Diarization

Advanced speaker diarization using Pyannote.audio with configurable speaker limits and automatic speaker identification:

# services/pyannote_service.py
from pyannote.audio import Pipeline
import torch

class PyannoteService:
    def __init__(self):
        self.pipeline = Pipeline.from_pretrained(
            "pyannote/speaker-diarization-3.1",
            use_auth_token=os.getenv("HUGGINGFACE_TOKEN")
        )

    def diarize_audio(self, audio_path: str,
                     min_speakers: int = 1,
                     max_speakers: int = 10):
        """Perform speaker diarization with configurable parameters"""
        diarization = self.pipeline(
            audio_path,
            min_speakers=min_speakers,
            max_speakers=max_speakers
        )

        # Convert to structured format
        speakers = []
        for turn, _, speaker in diarization.itertracks(yield_label=True):
            speakers.append({
                "start": float(turn.start),
                "end": float(turn.end),
                "speaker": speaker,
                "duration": float(turn.end - turn.start)
            })

        return speakers
      

Real-time Processing with Server-Sent Events

The application streams live progress updates using Server-Sent Events (SSE), so the browser gets per-chunk status without polling:

// frontend/static/js/app.js - Real-time progress handling
class AudioProcessor {
    async processAudio(fileId) {
        // Open a Server-Sent Events stream for progress updates
        const eventSource = new EventSource(`/api/transcribe-stream/${fileId}`);

        eventSource.onmessage = (event) => {
            const progress = JSON.parse(event.data);
            this.updateProgressVisualization(progress);

            if (progress.status === 'transcription_complete') {
                eventSource.close();
            }
        };

        eventSource.onerror = () => {
            // Falls back to simulated progress if streaming fails
            eventSource.close();
        };
    }

    updateProgressVisualization(progress) {
        // Update chunk visualization and progress bars
        const progressFill = document.getElementById('overallProgressFill');
        progressFill.style.width = `${progress.progress}%`;
        this.highlightCurrentChunk(progress.chunk);
    }
}
      

# backend/app.py - Server-Sent Events streaming endpoint
@app.get("/api/transcribe-stream/{file_id}")
async def transcribe_audio_stream(file_id: str):
    """Transcribe and diarize audio file with SSE progress updates"""

    async def generate_progress():
        yield f"data: {json.dumps({'status': 'starting'})}\n\n"

        async for progress_data in whisper_service.transcribe_with_progress(wav_path):
            yield f"data: {json.dumps(progress_data)}\n\n"

    return StreamingResponse(generate_progress(), media_type="text/event-stream")
      

vLLM High-Performance Inference

For production workloads, transcription can be offloaded to a vLLM server exposing an OpenAI-compatible API. vLLM takes priority over local and OpenAI Whisper when enabled, and large files are automatically split into 30-second chunks before being sent:

# Enable vLLM (takes priority over local and OpenAI Whisper)
export WHISPER_USE_VLLM=true
export VLLM_BASE_URL="http://localhost:8000/v1"
export VLLM_API_KEY="token-abc123"
export VLLM_MODEL_NAME="KBLab/kb-whisper-large"
export VLLM_MAX_AUDIO_FILESIZE_MB="25"   # larger files are chunked automatically

# Install and start the vLLM server
pip install vllm[audio]
vllm serve KBLab/kb-whisper-large --api-key token-abc123
      

Technology Stack

Python 3.8+ FastAPI PyTorch KB-whisper OpenAI Whisper vLLM Pyannote.audio Transformers python-docx NumPy FFmpeg Uvicorn JavaScript ES6+ WebRTC Server-Sent Events CUDA (optional)

API Endpoints

The application provides a comprehensive REST API for all transcription and diarization operations:

# Main Application
GET  /                              # Main application interface
GET  /static/*                      # Static files (CSS, JS, assets)

# Audio Processing
POST /api/upload                    # Upload audio file
POST /api/save-recording            # Save browser recording
POST /api/transcribe/{file_id}      # Transcribe and diarize audio
GET  /api/transcribe-stream/{file_id}   # SSE stream with live per-chunk progress
GET  /api/health                    # Health check endpoint
GET  /api/settings                  # Current service/model configuration

# Whisper Service Management
GET  /api/whisper/status            # Get detailed Whisper service status
POST /api/whisper/switch-to-local   # Switch to local Whisper service
POST /api/whisper/switch-to-openai  # Switch to OpenAI Whisper service
POST /api/whisper/download-model    # Download a local Whisper model

# Export
POST /api/export/word               # Export results to a Word document (.docx)
POST /api/export/txt                # Export results to a plain text file
GET  /api/export/templates          # List available custom Word templates
      

API Usage Examples

Here are practical examples of how to interact with the API:

# Upload an audio file
curl -X POST "http://localhost:8000/api/upload" \
     -H "Content-Type: multipart/form-data" \
     -F "file=@audio.mp3"

# Response:
{
  "success": true,
  "file_id": "uuid-string",
  "filename": "audio.mp3",
  "size": 1048576,
  "message": "File uploaded successfully"
}

# Start transcription and diarization
curl -X POST "http://localhost:8000/api/transcribe/uuid-string"
      

Example Output

Here's an example of the comprehensive JSON output from the transcription and diarization process:

{
  "success": true,
  "transcription": "Välkommen till dagens möte. Vi ska idag diskutera projektets framsteg. Tack. Jag har uppdaterat tidslinjen och vi ligger i fas med planen. Utmärkt. Kan du gå igenom de tekniska utmaningarna vi stött på?",
  "segments": [
    {
      "start": 0.5,
      "end": 5.2,
      "speaker": "SPEAKER_00",
      "text": "Välkommen till dagens möte. Vi ska idag diskutera projektets framsteg."
    },
    {
      "start": 6.0,
      "end": 10.8,
      "speaker": "SPEAKER_01",
      "text": "Tack. Jag har uppdaterat tidslinjen och vi ligger i fas med planen."
    },
    {
      "start": 11.5,
      "end": 16.3,
      "speaker": "SPEAKER_00",
      "text": "Utmärkt. Kan du gå igenom de tekniska utmaningarna vi stött på?"
    }
  ],
  "speakers": ["SPEAKER_00", "SPEAKER_01"],
  "duration": 16.3,
  "processing_time": 8.7,
  "model_info": {
    "whisper_model": "kb-whisper-large",
    "diarization_model": "pyannote/speaker-diarization-3.1"
  }
}
      

Installation & Setup

Get Audio Scribe AI running on your local machine in just a few steps:

Prerequisites

Python 3.8 or higher
CUDA-capable GPU (optional, but recommended for faster processing)
FFmpeg installed on your system
4GB+ RAM (8GB+ recommended for large models)

Quick Start

# 1. Clone the repository
git clone https://github.com/your-repo/kb-whisper-pyannote-transcription-diarization
cd kb-whisper-pyannote-transcription-diarization

# 2. Install Python dependencies
pip install -r requirements.txt

# 3. Install FFmpeg (Windows)
# Download from https://ffmpeg.org/download.html and add to PATH

# 4. Set up environment (optional for some Pyannote models)
cp .env-example .env
# Edit .env and add your HUGGINGFACE_TOKEN if needed

# 5. Start the application
python run.py

# 6. Open your browser
# Navigate to http://localhost:8000
      

Configuration Options

# Environment variables for customization
export WHISPER_MODEL="base"              # tiny, base, small, medium, large
export WHISPER_LANGUAGE="auto"           # auto, sv, en, etc.
export WHISPER_USE_LOCAL=true            # Use local models for privacy
export PYANNOTE_MODEL="pyannote/speaker-diarization-3.1"
export MIN_SPEAKERS="1"
export MAX_SPEAKERS="10"
export MAX_FILE_SIZE="104857600"         # 100MB limit

# vLLM server (takes priority over local and OpenAI Whisper)
export WHISPER_USE_VLLM=true
export VLLM_BASE_URL="http://localhost:8000/v1"
export VLLM_API_KEY="token-abc123"
export VLLM_MODEL_NAME="KBLab/kb-whisper-large"
export VLLM_MAX_AUDIO_FILESIZE_MB="25"
      

Applications

Audio Scribe AI has numerous real-world applications across various industries:

📝 Meeting Minutes: Automatic generation with speaker attribution and timestamps
🎙️ Interview Transcription: Perfect for journalists, researchers, and content creators
⚖️ Legal Documentation: Court proceedings and legal consultations with speaker identification
🏥 Medical Records: Doctor-patient consultations with privacy-focused local processing
📺 Media Indexing: Content search and accessibility for video/audio libraries
♿ Accessibility Services: Real-time transcription for hearing-impaired individuals
📞 Call Analysis: Customer service quality assurance and training
🎓 Educational Content: Lecture transcription and student accessibility

Project Structure

The application follows a clean, modular architecture that separates concerns and enables easy maintenance and extension:

kb-whisper-pyannote-transcription-diarization/
├── backend/
│   ├── app.py                          # FastAPI main application
│   ├── services/
│   │   ├── audio_service.py            # Audio & video processing utilities
│   │   ├── whisper_service.py          # OpenAI Whisper integration
│   │   ├── local_whisper_service.py    # Local Whisper models
│   │   ├── vllm_whisper_service.py     # vLLM server integration
│   │   ├── remote_whisper_service.py   # Standalone remote Whisper server client
│   │   ├── unified_whisper_service.py  # Unified service interface
│   │   ├── export_service.py           # Word (.docx) / text export
│   │   ├── pyannote_service.py         # Pyannote diarization
│   │   ├── pyannote_service_simple.py  # Simplified diarization
│   │   ├── simple_diarization.py       # Basic speaker separation
│   │   └── mock_services.py            # Development/testing mocks
│   ├── templates/                      # Custom Word export templates
│   └── utils/
│       └── config.py                   # Configuration management
├── frontend/
│   ├── index.html                      # Main application interface
│   └── static/
│       ├── css/style.css               # Application styling
│       └── js/
│           ├── app.js                  # Core application logic
│           ├── karaoke-player.js       # Interactive playback
│           └── recorder.js             # Audio recording
├── screenshots/                        # Application screenshots
├── whisper_server.py                   # Standalone remote Whisper REST service
├── pyannote_server.py                  # Standalone remote Pyannote REST service
├── setup_word_export.py                # Word export setup script
├── requirements.txt                    # Python dependencies
├── run.py                             # Application entry point
├── setup.py                           # Package configuration
├── LOCAL_WHISPER_SETUP.md             # Local setup guide
├── WORD_EXPORT_GUIDE.md               # Word export template guide
└── README.md                          # Project documentation
      

Performance & Optimization

Audio Scribe AI is optimized for both speed and accuracy:

🚀 GPU Acceleration: Automatic CUDA detection and utilization
⚡ Chunked Processing: Large files processed in manageable segments
🔄 Parallel Processing: Concurrent transcription and diarization
💾 Memory Management: Efficient handling of large audio files
📊 Model Selection: Choose optimal model size for your hardware
🎯 Format Optimization: Automatic audio format conversion

Future Enhancements

Planned improvements and features in development:

🎭 Emotion Analysis: Sentiment and emotion detection in speech
📝 Smart Summarization: AI-powered content summarization
🌍 Multi-language Support: Enhanced support for mixed-language content
🎯 Custom Vocabularies: Domain-specific terminology recognition
🔊 Noise Reduction: Advanced audio preprocessing
📱 Mobile App: Native iOS and Android applications

Audio Scribe AI - KB-whisper + Pyannote

📸 Application Screenshots

Main Interface & Recording Controls

Transcription Results & Speaker Diarization

🎯 Key Interface Features Shown

🎙️ Recording Controls

📁 Drag & Drop Upload

📊 Processing Visualization

👥 Speaker Diarization

🎵 Karaoke Player

📤 Export Options

Project Overview

Key Features

Technical Implementation

FastAPI Backend Architecture

Unified Whisper Service

Pyannote Speaker Diarization

Real-time Processing with Server-Sent Events

vLLM High-Performance Inference

Technology Stack

API Endpoints

API Usage Examples

Example Output

Installation & Setup

Prerequisites

Quick Start

Configuration Options

Applications

Project Structure

Performance & Optimization

Future Enhancements