Speech Releases

v3

3.0 - TBD

BREAKINGFeature

Concurrent playback — clips overlap instead of interrupting each other. IAudioPlayer could only ever hold one native player, so every PlayAsync began by tearing down whatever was already playing: background music plus a sound effect plus a voice line was impossible, and the only cancellation you had was “stop the one thing”. New IAudioPlayer.StartAsync(stream | source, ct) returns an IAudioPlayback handle (Id, Source, IsPlaying, Completion, StopAsync(), IAsyncDisposable) as soon as the clip is playing, so a clip is a thing you hold rather than a call you wait on. PlayAsync is now StartAsync plus a wait on that handle’s Completion — it waits for its clip and leaves the rest alone — and a CancellationToken passed to either stops that clip only, completing it normally rather than throwing OperationCanceledException. IAudioPlayer.Active lists what is playing right now, oldest first; IAudioPlayer.StopAsync() still stops everything. Breaking: PlayAsync no longer stops what is already playing — call await player.StopAsync() first for the old exclusive behavior — and the platform players implement StartAsync instead of PlayAsync (PlayAsync is now a default interface member, so it is only reachable through IAudioPlayer, not through a concrete AppleAudioPlayer/AndroidAudioPlayer/… reference). Every backend mixes natively — one AVAudioPlayer, MediaPlayer, <audio> element or PCM stream per clip — so this is real simultaneous audio, not a queue. On Apple the session category is still applied per clip and still ORs in MixWithOthers over whatever options are set, so a concurrent DuckOthers (e.g. Shiny.Music’s Duck()) is preserved rather than clobbered. Cloud TTS got more precise, not less: ITextToSpeechService.StopAsync() and IsSpeaking now track only the utterance it started, so stopping speech no longer silences the app’s other audio. Metering stays one stream per player — while clips overlap, AudioLevelChanged reports the loudest of them, which is what a VU meter should show. Also fixed on the way through: in the browser, PlayAsync never completed at a clip’s natural end (the ended event was not wired up) and would hang until something else stopped it. See Concurrent playback

Feature

Emotion & tone, portable across providers — new SpeechTone on TextToSpeechOptions (Emotion, Intensity, free-text Instructions) expresses expressive delivery once and lets each provider project it onto whatever it actually supports. Expressive TTS arrived with four incompatible APIs — ElevenLabs reads bracketed audio tags out of the text ([excited]), Typecast takes an emotion_preset + emotion_intensity field, Azure takes SSML mstts:express-as style/styledegree, OpenAI takes a free-text instructions string, and no on-device engine takes anything — so the cheap move (pass tags through, strip where unsupported) would have made expressiveness ElevenLabs-only and silently degraded it to nothing everywhere else. The 12-value SpeechEmotion is deliberately small and provider-neutral; mapping down is lossy on purpose (Typecast has seven presets total, so Excited and Friendly both land on happy, while Fearful and Sarcastic have no analogue and fall back to TypecastConfig.Emotion), and Instructions is the escape hatch for direction that doesn’t fit. Bracketed annotations in the text are normalized rather than passed through, because only eleven_v3 performs them — every other engine, including older ElevenLabs models, reads them aloud as words. The new SpeechAnnotationHandling defaults to Auto: promote the first emotion-bearing tag to a SpeechTone, then keep tags for providers that understand them and strip them for everyone else, so SpeakAsync("[excited] We shipped it. [laughs] Everything is live.") performs on eleven_v3 and says the clean sentence with the closest available emotion everywhere else. [laughs] and other performance beats have no portable equivalent and are dropped rather than promoted; an explicit Tone always beats a tag found in the text. Preserve passes text through untouched (for prose that legitimately contains brackets — tags are matched by shape, up to three alphabetic words, so [1] and [a, b] are already left alone) and Strip removes them unconditionally. This is the path for LLM-authored speech: prompt the model to write tags and whatever provider is registered does the right thing instead of the tags leaking into the audio. Capabilities are derived, not hardcoded — ITextToSpeechProvider.ToneCapabilities (a default interface member returning None, so existing custom providers keep compiling and get stripping for free) is computed by the ElevenLabs provider from TextToSpeechModel, so switching models automatically switches between performing tags and stripping them. Also new: the OpenAI provider now sets instructions at all (it previously ignored the field — needs gpt-4o-mini-tts), the Azure provider emits the mstts namespace and wraps in express-as only when a style maps, and the Typecast provider now sends PresetPrompt (which carries emotion_type) instead of the legacy Prompt type and honours a per-call tone over the configured default. SpeechAnnotations is public for direct use: Strip, Extract, ToEmotion, ToAnnotation, Resolve. See Emotion & Tone

Feature

Real-time microphone effects — new AudioEffectChain (in Shiny.Audio) applies DSP to captured audio: PitchShiftEffect, EchoEffect, ReverbEffect, ChorusEffect, DistortionEffect, RingModEffect, BiquadFilterEffect, NoiseGateEffect and GainEffect, plus AudioEffectPresets.Create(...) for ready-made combinations (Robot, Chipmunk, DeepVoice, Cathedral, Telephone, Megaphone, Ensemble). Pass a chain via the new StartCaptureAsync(AudioCaptureOptions, ct) overload. This is pure managed code over the 16 kHz mono PCM every backend already produces, so it behaves identically on every platform — which is the point: no platform offers pitch shift on capture at all, Windows and iOS offer echo/reverb, and Android and Linux offer neither, so a native-first implementation would have been four divergent feature sets. Effects are objects you own, driven live — chain.Add(...) returns the instance, and assigning a parameter or toggling Enabled applies on the next audio buffer with no restart and nothing to “apply”. Three levels of on/off (chain.Enabled master, effect.Enabled, and per-effect Mix), and composition can change mid-stream because Add/Remove publish a new array atomically while the audio thread snapshots it once per buffer. Live control is click-free by construction: parameters ramp over ~15 ms instead of stepping, bypass crossfades over ~12 ms instead of hard-switching, and an effect that has fully faded out is reset so switching it back on can’t replay a stale reverb tail. All parameters are float (a double write can tear on 32-bit ARM) and every setter clamps, so a slider can bind straight to one. PitchShiftEffect is the only one with meaningful latency (~50 ms, the crossfade window) and holds duration exactly while changing pitch; everything else is latency-free and cheap enough for a Raspberry Pi. Effects run before metering, so InputLevelChanged reflects what you will actually hear. Do not put effects on audio feeding speech recognition or wake-word detection — they destroy accuracy. IAudioMonitor (the live mic-to-output PA) does not run the chain

Feature

Recording to WAV — new IAudioRecorder (registered by AddAudioServices(), also on the IAudio facade as Recorder) records the microphone to a 16 kHz mono PCM16 WAV file and owns the capture session and drain loop, so there is no PCM stream to pump by hand. AudioRecordingOptions takes a Path (null gives a timestamped file under LocalApplicationData/shiny.audio/recordings), an AudioEffectChain, AudioProcessingOptions, an optional MaxDuration, and a Mode: Wet (the processed take, default), Dry (the untouched mic), or Both — which writes two files, with the clean one at AudioRecording.DryPath. Recording always captures dry from the source and applies the chain in the recorder’s own loop, which is what makes Both possible without splitting the capture stream (PipeStream is single-reader, so there was nothing to tee). StopAsync() returns null when nothing was captured and leaves no unplayable header-only file behind, so null-check it. Because a dry take is kept, AudioEffectProcessor.ProcessFile(input, output, chain) re-renders it with different settings later instead of asking anyone to perform it again. Output is WAV/PCM16 only — AAC/MP3 would need a different native encoder per platform and Linux has none. It is platform-agnostic, built on whatever IAudioSource is registered, so it works everywhere capture does including Linux via AddLinuxAudio()

Feature

WavWriter / WavReader are now public (Shiny.Audio). The writer streams — it emits the header with placeholder sizes and patches them on close — so a long recording never has to sit in memory, and WavWriter.CreateFile(pcm) still builds a whole file at once for short clips. The reader walks the RIFF chunk list rather than assuming a canonical 44-byte header, so files carrying LIST/fact chunks parse correctly. This replaces two private copies of the same header-building code that previously lived in the ElevenLabs Scribe provider and the MAUI sample

Enhancement

IAudioSource.StartCaptureAsync gains an AudioCaptureOptions overload carrying both Processing and Effects; the existing StartCaptureAsync(AudioProcessingOptions?, ct) signature is retained as a default interface method and keeps working unchanged. Internally, the level metering, throttling and pipe-write logic that was duplicated across all five platform backends is now a single shared capture sink

BREAKINGFeature

Structured AI turns — typed questions instead of guessing at the wording. A voice conversation has to know whether the AI is waiting on the user: if it is, the listener stays open and they answer without repeating the wake word. IChatClient gives you nothing to decide that with — ChatFinishReason is only Stop/Length/ToolCalls/ContentFilter, and Microsoft.Extensions.AI’s InputRequestContent hierarchy covers tool approvals, not conversational questions — so the service previously inspected the reply for a trailing ?. It now asks the model for a structured turn and reads the signal off the shape: AiTurn(string Reply, AiQuestion[]? Questions), where each AiQuestion(Id, Text, AiChoice[]? Choices, bool AllowMultiple) may carry a fixed set of AiChoice(Id, Label) answers. Reply is what the user sees and hears; Questions drives the interface — the model still phrases the question naturally in Reply, nothing speaks the structured text aloud. New on IAiConversationService: PendingQuestions (the live queue — each turn replaces it, since the model is the source of truth for what it still needs and re-asks whatever is outstanding), FollowUpTimeout (default 20s, null waits forever) which clears the queue and returns to the wake word when nobody answers — an unbounded open mic means the next unrelated thing said in the room becomes the answer — and StructuredOutputMode to override the provider. AiResponse gains Text (the parsed reply), Turn, and Questions; ExpectsResponse is now typed off Turn.Questions. Answers go back as plain text — the model already has the question and its choices in context, so nothing is fuzzy-matched locally. When the reply is spoken aloud (Acknowledgement above AudioBlip) the model is told to ask at most one question per turn; in text mode it may ask several, since three questions in one breath renders fine as chips and is unusable as audio. Provider support is declared, not assumed: IChatClientProvider.StructuredOutputMode (a default interface member) picks JsonSchema (native schema-constrained, the default), Json (JSON format + shape in the prompt), Prompt (prompt only) or None — both GitHub Copilot providers default to Json because the proxy passes json_schema support through per-model. Nothing hard-fails: parsing tolerates markdown fences and surrounding prose, a rejected request is retried unstructured, and any failure leaves Turn null with AiResponse.Text as the raw reply and the old trailing-? heuristic as the fallback — so null-check Turn and treat PendingQuestions as possibly empty. Opt out entirely with AiStructuredOutputMode.None. Breaking: IMessageStore.Store takes a new string? assistantMessage parameter and implementations must persist that, not response.Text — the latter is now the raw JSON envelope and would push JSON into chat history and back into later prompts through the chat lookup tool

Feature

Multiple-choice answers in AiChatView — when a turn carries AiChoice options they render as tappable chips under the AI bubble, and tapping one sends its Label as the user’s answer. Questions with AllowMultiple toggle their chips and commit with a send chip, so several picks go out as one answer; a bubble’s chips disable once answered. New properties ShowChoiceButtons (default true) and ChoiceSendText (default Send). The built-in AiChoiceTemplateSelector is only installed when you have set neither MessageTemplate nor MessageTemplateSelector — with your own template it stays out of the way, and you can render the choices yourself via AiChoiceTemplateSelector.ReadQuestions(chatMessage) off the message metadata (AiTurnSerializer.SerializeQuestions/DeserializeQuestions round-trip the payload). AiChatView also now renders and speaks AiResponse.Text rather than Response.Text, so the raw JSON envelope never reaches a bubble

Feature

Drop-in MAUI chat UI — new Shiny.AiConversation.Maui package ships AiChatView, a complete chat screen for IAiConversationService. It derives from the Shiny.Maui.Controls ChatView, so every style, template and behavior property of the base control still applies (MyBubbleColor, OtherTextColor, ChatBackgroundColor, BubbleCornerRadius, MessageTemplateSelector, InputActions, …), but there is nothing to bind: the control resolves the conversation service from the app’s service provider and builds its own session. Typed messages go to TalkTo; replies (AiResponded) render as AI bubbles; utterances heard by speech-to-text (SpeechOccurred) render as user bubbles, so a wake-word conversation happening elsewhere in the app still fills in the transcript; the typing indicator follows AiState with a heartbeat so long turns keep the bubble alive; and TalkTo failures plus ErrorOccurred render as AI bubbles carrying Identifier = "error" for template targeting. History comes from the registered IMessageStore — the newest page is backfilled on load and scrolling to the top pages further back through GetChatHistory; with no message store registered the chat simply starts empty and stays live-only rather than throwing. New properties: BotName, BotAvatar, BotBubbleColor, UserName, UserAvatar, UserBubbleColor, LoadHistory, GreetingMessage (shown when there is no history), ShowTokenUsage (per-message token footer), ShowMicrophoneAction + MicrophoneActionText (push-to-talk input-bar action over ListenAndTalk, invoke again to cancel), AiService (to pass the service explicitly) and a Refresh() method for reloading after ClearChatHistory. XAML namespace is http://shiny.net/maui/aiconversation. Prefer a hand-rolled layout? opts.AddChatSessionProvider() registers the same bridge as an IChatSessionProvider for a plain ChatView

FeatureLinux

On-device, offline speech-to-text on Linux — new Shiny.Speech.Linux.Whisper package runs OpenAI’s Whisper locally through whisper.cpp (via Whisper.net): no cloud account, no API key, no network at runtime, no per-minute billing. This closes the one real gap left on Linux — it is the only platform in the library with no OS speech engine to wrap, so recognition there previously meant shipping audio to somebody else’s server. Register with AddLinuxWhisperSpeechToText(GgmlType.BaseEn, QuantizationType.Q5_1) (or a full WhisperConfig) after AddLinuxAudio(), which supplies the IAudioSource it captures from; registration is a no-op off Linux, so it is safe in shared startup code. It registers an ordinary ISpeechToTextService, so ListenUntilSilence(), StatementAfterKeyword(), WaitListenForKeywords() and the KeywordHeard event all work unchanged. Speech-to-text only — Whisper is a recognition model, there is no Whisper TTS; pair it with a cloud TTS provider for a full voice loop. No partial results: Whisper is a batch model with a 30-second window, not a streaming recognizer, so the provider runs client-side voice activity detection over the PCM mic stream and turns each speech→silence segment into one inference pass and one IsFinal = true result — the same shape as the ElevenLabs Scribe provider — tunable via SilenceRmsThreshold / MinUtteranceDurationMs / MaxUtteranceDurationMs. The ggml model downloads from Hugging Face on first use and caches in ~/.local/share/shiny.speech/whisper; call PrepareAsync() at startup so the first utterance doesn’t pay the download plus multi-second model load, or set AutoDownloadModel = false to require a pre-provisioned file. Whisper’s hallucination on silence is handled by default: FilterNonSpeechAnnotations strips [BLANK_AUDIO]/(wind blowing)/♪ and discards results that filter down to nothing, and CarryContextBetweenUtterances is off so one bad transcription can’t poison everything after it. Also configurable: ModelType/Quantization, Threads, Language (overridden per session by SpeechRecognitionOptions.Culture), Translate, InitialPrompt for domain vocabulary, and UseGpu. Model sizing — Tiny/Base are the realistic choices on a Raspberry Pi 4/5 (roughly realtime at Base, slower above Small), which suits push-to-talk and wake-word-then-command far better than continuous dictation; Small+ is comfortable on x64. The *En variants are meaningfully more accurate than the multilingual model of the same size for English-only apps. Supported on all mainstream Linux, not just Pi — linux-x64, linux-arm64 and linux-arm natives ship; requires libstdc++6 and glibc 2.31+ (Debian 11+ / Ubuntu 20.04+ / Pi OS Bullseye+), and on x86/x64 a CPU with AVX, AVX2, FMA and F16C (older CPUs add a Whisper.net.Runtime.NoAvx reference; ARM has no such requirement). Publish with an explicit RID (-r linux-arm64) to avoid copying every platform’s native binaries, and add Whisper.net.Runtime.Cuda/.Vulkan with UseGpu = true for GPU acceleration

FeatureLinux

Linux support — new Shiny.Audio.Linux package implements all four audio services (IAudioSource, IAudioPlayer, IAudioDevices, IAudioMonitor) over PulseAudio/PipeWire, falling back to ALSA where no sound server is running (headless servers, minimal containers, Raspberry Pi). Register with AddLinuxAudio() — a no-op off Linux, so it is safe in shared startup code, but it must be called before AddAudioServices() / AddSpeechServices() / AddCloudSpeechToText, which use TryAdd and register nothing on Linux. This unblocks cloud speech-to-text and text-to-speech on Linux: the Azure / OpenAI / ElevenLabs / Typecast / Microsoft.Extensions.AI providers are pure HTTP over the PCM stream and needed only an IAudioSource to exist. There is still no native Linux STT/TTS — Linux has no OS speech engine to wrap — so AddSpeechToText()/AddTextToSpeech() remain no-ops there; pair AddLinuxAudio() with a cloud provider. Notes: playback decodes MP3 and WAV in managed code (NLayer) since Linux has no system decoder, which is why this ships as a separate package — and which is also why playback VU metering works on Linux where it doesn’t on Windows/Browser; no resampler ships in the library, as streams open at the exact format wanted and PulseAudio (server-side) or ALSA (plug plugin) converts; Volume is settable on PulseAudio/PipeWire but IsVolumeControlSupported is false on the ALSA fallback; echo cancellation is honoured by selecting PulseAudio’s module-echo-cancel virtual source when that module is loaded; ShowOutputPicker() is a no-op and IAudioDevices.Changed fires on PulseAudio/PipeWire only. Runtime dependency is the distro’s libpulse-simple.so.0 or libasound.so.2 — check IsLinuxAudioAvailable at startup to fail fast

FixiOS

Microphone capture no longer forces voice processing on. AppleAudioSource set the session to VoiceChat mode with AllowBluetooth/AllowBluetoothA2DP on every capture, regardless of AudioProcessingOptions — so “capture raw input” wasn’t raw: VoiceChat engages Apple’s AEC/NS/AGC chain, and a paired headset moved the mic onto 8 kHz HFP. That is invisible for transcription but destroys anything that measures the signal itself; a speaker-recognition stack measured two recordings of the same person as unrelated voices. Capture now uses Measurement mode (minimum system input processing) unless effects are actually requested, matching the rule IAudioMonitor already followed. Requesting AudioProcessingOptions.VoiceChat/EchoCancellation behaves exactly as before

Feature

AudioProcessingOptions.Analysis — the options preset for audio that feeds a model rather than a listener (speaker recognition/verification, wake words, anything producing an embedding): no effects, and no Bluetooth route. The new AllowBluetooth property (default true, iOS/Mac Catalyst) drops the Bluetooth options from the capture session category, because an HFP mic caps capture at 8 kHz narrowband and quietly makes recorded bandwidth depend on what happens to be paired. Android’s raw mic source and Windows capture never route to a Bluetooth mic implicitly, so it is a no-op there. Record enrollment and matching through the same options — the embedding encodes the channel

Feature

Wired / Bluetooth route classification — new AudioDeviceExtensions (in Shiny.Audio) adds IsWired(), IsBluetooth(), IsBuiltIn(), IsHeadphones() and HasMicrophone() on both AudioDevice and AudioDeviceType, so “are wired headphones plugged in?” is one call instead of a hand-written match over enum members. Wired routes were already normalized on both platforms — WiredHeadphones (output only) and WiredHeadset (output plus mic), from AVAudioSession.PortHeadphones/PortHeadsetMic on iOS and WiredHeadphones/WiredHeadset on Android. IsWired() deliberately includes Usb, because on handsets with no 3.5mm jack the wired option is USB-C and neither platform surfaces those as a Wired* type (Android reports UsbHeadset/UsbDevice, iOS reports PortUsbAudio) — the trade-off is that a USB audio interface or DAC also matches, so use IsHeadphones() when you specifically mean something worn on the head. HasMicrophone() excludes Usb for the inverse reason: neither platform reveals mic presence from the output device alone — check GetInputs() for a matching Usb entry. Bluetooth cannot be narrowed to earbuds-vs-speaker on any platform, so IsHeadphones() counts all Bluetooth as private

FixAndroid

IAudioDevices.CurrentOutput returned null for a wired headset with a microphone, and for any USB-C headset. The current route was derived from the deprecated AudioManager.BluetoothA2dpOn/WiredHeadsetOn flags, and WiredHeadsetOn is true for headsets and headphones while only ever reporting WiredHeadphones — so a mic-equipped headset (mapped as WiredHeadset) matched no enumerated device, and USB routes were never considered at all. This hit most current handsets, where USB-C is the only wired option. The active output is now resolved by ranking the real AudioManager.GetDevices list by routing priority (Bluetooth → wired → USB → car/HDMI → built-in), and IsCurrent matches on device id rather than type, so two routes of the same type no longer both report as current

Feature

Microphone VU metering — the level signal now covers the input side, matching what playback already had. New IAudioSource.InputLevelChanged emits a normalized 0.0–1.0 mic level while capturing on every platform (Apple taps the capture node, Android meters the AudioRecord read loop, Windows meters each AudioGraph quantum, the browser meters the worklet’s PCM), and new ISpeechToTextService.InputLevelChanged + IsInputAnalysisSupported expose it while listening. Cloud STT (Azure / OpenAI / ElevenLabs / Microsoft.Extensions.AI / custom) forwards the capture source’s level, so a “listening” meter works everywhere; Apple native recognition meters the recognizer’s own input tap and Android forwards SpeechRecognizer.OnRmsChanged; Windows and Browser native recognizers own the mic and report IsInputAnalysisSupported = false. Capture-side events are throttled to ~20/sec and peak-held in between so a bound bar stays smooth

Feature

AudioLevel is now public (Shiny.Audio) — FromRms, FromPcm16, and FromSamples expose the dBFS mapping (-50 dB noise floor) that every meter in the library runs through, so PCM you consume yourself meters on the same scale as the events

Feature

Volume control on IAudioPlayer — new Volume (device media volume, 0.0–1.0), a VolumeChanged event, and an IsVolumeControlSupported guard. Reading works on every platform; setting is platform-limited. Android reads/writes the system STREAM_MUSIC level (AudioManager) and observes changes via a settings ContentObserver. Windows reads/writes the default render endpoint’s master volume through WASAPI IAudioEndpointVolume, with an IAudioEndpointVolumeCallback for changes. macOS reads/writes the default output device’s virtual main volume via the CoreAudio HAL (with a property listener) — settable when the current device supports it. iOS / Mac Catalyst read AVAudioSession.OutputVolume and observe it via KVO, but the setter throws NotSupportedException (Apple exposes no supported API to set the system volume — use the hardware buttons or an MPVolumeView). Browser maps Volume to the app’s own HTMLAudioElement volume (the OS volume is sandboxed), persisted across plays. VolumeChanged fires for hardware buttons, the OS volume UI, or a successful set; marshal it to the UI thread

Feature

Live microphone monitor — new IAudioMonitor (in Shiny.Audio) routes the mic straight to the current output in near-real-time (a PA / “talk over a Bluetooth speaker” scenario). Start/Stop, adjustable Gain, an InputLevelChanged VU signal, per-session AudioMonitorOptions (voice processing + preferred input/output device), and SetInputDevice/SetOutputDevice. iOS/Mac Catalyst route input → main mixer → output through AVAudioEngine and route to a Bluetooth A2DP speaker (phone mic + BT output) — the session uses no DefaultToSpeaker and Default mode so output follows a Bluetooth/wired route, auto-preferring an external output and rebuilding the engine on route changes. Android bridges AudioRecord → AudioTrack. The audio session is snapshotted and restored on Stop. Trade-off (iOS): enabling AudioProcessingOptions (echo cancellation) engages the voice-processing unit which forces Bluetooth onto the low-quality HFP profile, so a Bluetooth speaker (A2DP) drops back to the phone — leave processing off to reach a BT speaker. AirPlay (HomePod/Apple TV) is not supported for a live mic, since iOS only permits AirPlay for playback, not while recording

Feature

Audio device enumeration & selection — new IAudioDevices (in Shiny.Audio) lists input/output routes (GetInputs/GetOutputs), reports the active CurrentInput/CurrentOutput, and raises Changed when routes come and go. Each AudioDevice has a normalized Type (BuiltInMic, BluetoothA2dp, WiredHeadphones, BuiltInSpeaker, …). Reads AVAudioSession route/AvailableInputs + RouteChangeNotification on Apple and AudioManager.GetDevices + AudioDeviceCallback on Android. Selection is applied via IAudioMonitor.SetInputDevice/SetOutputDevice: Android enumerates and selects both input and output fully; iOS can select the input but treats output as observe-only (no app-level output enumeration/selection — AirPlay/Bluetooth output is owned by the system route picker). Use CurrentInput/CurrentOutput as a display property everywhere

FixiOS

AppleAudioPlayer no longer forces DefaultToSpeaker, which pinned playback to the built-in speaker. Playback now follows the current output route — headphones / Bluetooth / an AirPlay device — so a recorded clip can play out a Bluetooth speaker

Feature

New IAudio facade — one injectable that exposes Player / Source / Monitor / Devices, so the whole audio surface is discoverable from a single dependency. The focused interfaces remain independently injectable; lifetimes are preserved (IAudioSource stays a fresh transient per access). Registered by AddAudioServices(), which now also wires AddAudioMonitor() and AddAudioDevices()

FixiOS

Recorded/played-back audio was quiet after a capture session on iOS — AppleAudioSource left the shared AVAudioSession in the record-oriented PlayAndRecord + VoiceChat profile (attenuated, earpiece-routed), which subsequent IAudioPlayer playback inherited. Capture now snapshots the session category/options/mode on start and restores them on stop, so later playback returns to full-volume Playback routing

BREAKINGEnhancement

Native speech & audio now build on Shiny.Core instead of hand-rolled platform plumbing. Android runtime permission requests (RECORD_AUDIO) and current-activity tracking are delegated to Shiny.Core’s AndroidPlatform, replacing the library’s internal ActivityProvider + PermissionRequestFragment. Apps must now reference Shiny.Hosting.Maui and call .UseShiny() on the MauiAppBuilder so AndroidPlatform is registered and receives permission callbacks — without it, RequestAccess() throws TimeoutException on Android. AccessState now comes from Shiny.Core and lives in the Shiny namespace (parent of Shiny.Audio/Shiny.Speech, so most code resolves it with no change). The duplicated Android permission-check code that previously lived in both AndroidAudioSource and the Android SpeechToTextImpl is gone

BREAKINGEnhancement

The browser JS interop module now ships inside the Shiny.Audio package as a static web asset at _content/Shiny.Audio/shiny-audio.js (renamed from shiny-speech.js), loaded on demand via JSHost.ImportAsync. Blazor WebAssembly apps no longer need to copy the file into wwwroot or add a <script src="shiny-speech.js"> tag — just reference the NuGet package. Delete any existing wwwroot/shiny-speech.js left over from a previous version. Shiny.Audio is now built with the Razor SDK to produce the static web asset

FixBrowser

Browser raw audio capture callbacks (BrowserAudioSource.OnAudioData / OnCaptureError) were dispatched to the wrong assembly after audio was extracted into Shiny.Audio, breaking IAudioSource capture in the browser. The interop module now resolves exports from the correct Shiny.Audio and Shiny.Speech assemblies

Feature

Microphone voice processing — new AudioProcessingOptions (in Shiny.Audio) requests platform echo cancellation, noise suppression, and automatic gain control on a capture session. Pass it to IAudioSource.StartCaptureAsync(processing, ct) or set SpeechRecognitionOptions.AudioProcessing (honored by the cloud STT providers). Echo cancellation subtracts the device’s own speaker/TTS output from the mic so it isn’t re-captured during barge-in. Maps to the Voice-Processing I/O unit on Apple (bundled AEC+NS+AGC), AcousticEchoCanceler/NoiseSuppressor/AutomaticGainControl (+ VoiceCommunication source) on Android, the Communications capture category on Windows, and getUserMedia constraints (WebRTC AEC3) in the browser. Effects are best-effort/device-dependent; native on-device recognizers manage their own mic and are unaffected

Feature

Cloud provider credentials can now be changed at runtime. The provider config objects (AzureSpeechConfig, ElevenLabsConfig, OpenAiSpeechConfig, TypecastConfig) are mutable singletons — set a new ApiKey/SubscriptionKey (or region/model/voice) on the instance you registered (or resolve it from DI) and the provider uses it on its next call, with no re-registration. Configuration APIs (AddAzureSpeech, AddElevenLabsSpeech, AddOpenAiSpeech, AddTypecastSpeech) are unchanged. Client-caching providers rebuild their SDK/HTTP client on key change via the new RefreshableClient<T> helper in Shiny.Speech.Cloud

Feature

New Shiny.Speech.Typecast provider — cloud text-to-speech via the official typecast-csharp SDK. Register with AddTypecastSpeech(apiKey) (or a TypecastConfig for model / default voice / language / emotion / audio format). TTS-only; pair with Azure/ElevenLabs/OpenAI or native STT for recognition

BREAKINGEnhancement

Audio capture and playback extracted into a new standalone Shiny.Audio package. IAudioSource, IAudioPlayer, and PipeStream moved from the Shiny.Speech namespace to the new Shiny.Audio namespace — add using Shiny.Audio; where you consume them. Shiny.Speech references Shiny.Audio automatically, so AddSpeechServices() still registers everything; no package reference changes are needed for existing speech apps

Feature

Shiny.Audio is usable on its own for recording/playback without the speech stack — register via the new AddAudioServices() (or AddAudioSource() / AddAudioPlayer()) extension methods

Feature

IAudioPlayer.PlayAsync(string source) — play audio from a remote http/https URL or a local file path. You pass a plain URL/path and each platform resolves the source natively (no platform-specific file URI required); remote sources stream progressively on Android, Windows, and Browser, and are buffered on Apple

Chore

All Shiny.Speech / Shiny.Audio / Shiny.AiConversation packages now share a single version defined by the repo-root version.json

FixAndroid

Speech recognition reported itself unsupported on every modern device. SpeechRecognizer.isRecognitionAvailable() resolves the recognition service through PackageManager.queryIntentServices, which API 30+ filters by package visibility — so without a <queries> declaration the lookup came back empty, ISpeechToTextService.IsSupported was false, and RequestAccess() returned AccessState.NotSupported before the microphone permission was ever requested. That reads like “speech recognition is broken”, and every consumer had to rediscover the manifest declaration to fix it. Shiny.Speech now ships its own Android library manifest carrying the declaration, merged into your app automatically — remove it from your own manifest if you added it. Permissions are deliberately still yours to declare: RECORD_AUDIO is a dangerous permission and would otherwise surface in the store listing of an app that only uses text-to-speech.

Fix

A continuous session no longer dies silently on one transient error. Both native recognizers are single-utterance underneath and the service re-arms them to keep the mic open — but a hard error (the mic taken by another capture, a busy recognizer, a dropped round trip) skipped that re-arm. The session stopped recognizing while IsListening stayed true, so nothing downstream could tell it was dead: one blip early on and the rest of the session detected nothing. Recoverable errors are now still surfaced through Error, then re-armed behind the new SpeechRetryPolicy backoff — 250ms doubling to a 4 second ceiling, reset by the next result. After 5 consecutive failures, or immediately on an error retrying cannot fix (missing permission, unsupported language), the session stops itself rather than looping.

FixAndroid

SpeechRecognitionOptions.PreferOnDevice was silently ignored. It worked on Apple and did nothing on Android, which quietly put long or offline sessions on the network path. Android now uses the on-device recognizer when the device has one installed (API 31+) and otherwise falls back to the system recognizer with EXTRA_PREFER_OFFLINE set — best-effort either way, never a failure.

EnhancementiOS

SpeechRecognitionOptions.AudioProcessing now applies to native Apple recognition, which owns its own AVAudioEngine: it selects the session category options, VoiceChat vs Measurement mode, and the voice-processing I/O unit, exactly as AppleAudioSource already did. Previously the option existed but only the cloud providers read it. Leaving it null keeps the behavior Apple recognition has always had (the full VoiceChat chain), so nothing changes unless you ask for it — pass AudioProcessingOptions.Analysis for an unaltered signal off a narrowband Bluetooth route. It remains unavailable on Android, where the platform SpeechRecognizer runs out-of-process and opens the mic itself; setting it there now logs a warning instead of being dropped in silence.

FixAndroid

Stop() cleared its Handler field while a teardown callback was still queued on it, so a restart racing with Stop() could start a recognizer that was about to be destroyed.

2.1 - May 21, 2026

Feature

ISpeechToTextProvider.Error event — providers can surface non-fatal errors (e.g. transient network failures between chunked requests in continuous mode) without aborting the RecognizeAsync enumerator. CloudSpeechToText subscribes and forwards to the service-level ISpeechToTextService.Error event automatically. Azure / OpenAI / ElevenLabs providers updated to use it instead of throwing out of the enumerator

BREAKINGChore

ISpeechToTextProvider now requires implementers to expose an event EventHandler<SpeechRecognitionError>? Error; — existing custom providers must add the event declaration (it can be unraised for one-shot providers)

Feature

ElevenLabs Scribe speech-to-text provider — AddElevenLabsSpeech() now registers both STT and TTS, or use the new AddElevenLabsSpeechToText() helper. Buffers captured PCM, wraps in a WAV container, and posts a single request to /v1/speech-to-text; yields one final SpeechRecognitionResult per session

Feature

ElevenLabsConfig.SpeechToTextModel property (default scribe_v1) — configurable Scribe model id

BREAKINGChore

Renamed ElevenLabsConfig.ModelId → ElevenLabsConfig.TextToSpeechModel to disambiguate from the new SpeechToTextModel property

Fix

KeywordHeard no longer re-fires for the same final transcription within a 3-second window — eliminates duplicate keyword events caused by trailing-audio carry-over between recognition tasks (iOS SFSpeechRecognizer re-arm, Android SpeechRecognizer restart). Applied uniformly to Apple, Android, Browser, Windows, and CloudSpeechToText

Feature

ITextToSpeechService.AudioLevelChanged event and IsPlayerAnalysisSupported flag — normalized 0.0–1.0 RMS level for driving VU-meter UI during speech playback

Feature

IAudioPlayer.AudioLevelChanged event and IsPlayerAnalysisSupported flag — same RMS signal raised during generic audio playback (e.g. cloud TTS audio streams)

EnhancementiOS

Apple native TTS now routes AVSpeechSynthesizer through AVAudioEngine + AVAudioPlayerNode with a player-node tap, enabling AudioLevelChanged for built-in iOS / macOS / Mac Catalyst voices. Engine is created lazily on first speak and kept warm across utterances

EnhancementAndroid

Android native TTS taps UtteranceProgressListener.OnAudioAvailable to compute RMS from PCM bytes without rerouting playback

EnhancementAndroid

AndroidAudioPlayer attaches Android.Media.Audiofx.Visualizer to the MediaPlayer audio session for cloud TTS / generic playback metering (no RECORD_AUDIO permission needed for per-session capture; MODIFY_AUDIO_SETTINGS recommended)

EnhancementiOS

AppleAudioPlayer enables AVAudioPlayer.MeteringEnabled and polls AveragePower for VU metering during cloud / generic audio playback

Chore

CloudTextToSpeech forwards AudioLevelChanged and IsPlayerAnalysisSupported from the underlying IAudioPlayer — Azure / OpenAI / ElevenLabs / custom providers get VU metering for free

EnhancementiOS

CarPlay compatible — iOS audio session uses PlayAndRecord with AllowBluetooth, AllowBluetoothA2dp, and DefaultToSpeaker so audio automatically routes through the car’s microphone and speakers when CarPlay is active

2.0 - May 13, 2026

BREAKINGChore

ISpeechToTextService redesigned from IAsyncEnumerable-based to event-based Start/Stop model — ContinuousRecognize() and ListenUntilSilence() removed from the interface

Feature

Start(SpeechRecognitionOptions?) / Stop() methods — long-lived listening sessions with explicit lifecycle control; Start() throws if already listening, Stop() is a safe no-op

Feature

ResultReceived event — fires for every recognition result (partial and final) with full SpeechRecognitionResult including Text, IsFinal, and Confidence; multiple subscribers supported

Feature

KeywordHeard event — fires when a keyword from SpeechRecognitionOptions.Keywords is detected in a final result using case-insensitive whole-word matching

Feature

Error event — fires on recognition errors with SpeechRecognitionError containing Message and optional Exception

Feature

SpeechRecognitionError record — new type for error reporting via the Error event

Feature

SpeechRecognitionOptions.Keywords property (string[]?) — built-in keyword detection at the platform level; keywords are matched with compiled regex on final results

BREAKINGChore

ListenWithWakeWord() extension method removed — replaced by StatementAfterKeyword()

BREAKINGChore

ListenForKeyword() extension method removed — replaced by WaitListenForKeywords() and ListenForKeywords()

Feature

ListenUntilSilence() extension method — starts listening, waits for first final result, then stops; replaces the former interface method

Feature

StatementAfterKeyword(string[]) extension method — waits for a keyword to be heard, then returns the next final statement (replaces ListenWithWakeWord)

Feature

WaitListenForKeywords(string[], TimeSpan?) extension method — returns the first keyword heard with optional timeout

Feature

ListenForKeywords(string[]) extension method — yields keywords continuously as IAsyncEnumerable<string>

Enhancement

All extension methods handle Start/Stop/event wiring automatically — no manual lifecycle management needed for simple scenarios

Enhancement

Multiple classes can now subscribe to speech recognition events simultaneously — eliminates the single-consumer limitation of IAsyncEnumerable

Enhancement

Cloud provider (CloudSpeechToText) adapted to consume ISpeechToTextProvider.RecognizeAsync() internally on a background task and raise events — ISpeechToTextProvider interface unchanged

v1

1.2.1 - May 11, 2026

Feature

OpenAI cloud provider — AddOpenAiSpeech() registers OpenAI STT (Whisper / GPT-4o Transcribe) and TTS (GPT-4o Mini TTS) with configurable model and voice selection

Feature

OpenAI TTS supports 10 built-in voices: alloy, ash, ballad, coral, echo, fable, onyx, nova, sage, shimmer

Feature

OpenAI STT/TTS follows the same cloud provider pattern as Azure and ElevenLabs — platform IAudioSource and IAudioPlayer handle audio I/O

Feature

Microsoft.Extensions.AI adapter — AddShinySpeechClients() exposes any registered cloud provider as ISpeechToTextClient and ITextToSpeechClient from Microsoft.Extensions.AI

Feature

AddShinySpeechToTextClient() / AddShinyTextToSpeechClient() for registering M.E.AI adapters individually

Feature

M.E.AI streaming support — GetStreamingTextAsync() emits SessionOpen, TextUpdating, TextUpdated, SessionClose update kinds mapped from SpeechRecognitionResult.IsFinal

Feature

M.E.AI TTS streaming — GetStreamingAudioAsync() emits SessionOpen, AudioUpdated, SessionClose update kinds with audio as DataContent

Feature

M.E.AI options mapping — SpeechToTextOptions.SpeechLanguage → CultureInfo, TextToSpeechOptions voice/speed/pitch/volume mapped to Shiny equivalents

1.2 - May 6, 2026

FeatureWASM

Browser IAudioSource implementation — raw PCM microphone capture via the Web Audio API (getUserMedia + ScriptProcessorNode), downsampled to 16kHz 16-bit mono, matching the output format of Android, iOS, and Windows

EnhancementWASM

Cloud STT providers (Azure, custom) now work in the browser — IAudioSource provides the raw audio stream that CloudSpeechToText requires

1.1.2 - May 6, 2026

Enhancement

Cloud provider extensions (AddAzureSpeech, AddElevenLabsTextToSpeech, AddCloudSpeechToText, AddCloudTextToSpeech) now automatically register IAudioSource and IAudioPlayer — manual AddAudioSource() / AddAudioPlayer() calls are no longer required

1.1.1 - May 5, 2026

Feature

ISpeechToTextService.IsListening property — indicates whether speech recognition is currently active, analogous to ITextToSpeechService.IsSpeaking

1.1 - May 4, 2026

Feature

ListenWithWakeWord() extension method — “Hey Siri” style wake word activation that continuously listens for a wake phrase, then captures the spoken command after it until silence

Feature

ListenForKeyword() extension method — listens continuously until one of the specified keywords is detected (case-insensitive, whole-word matching), returns the matched keyword

Feature

Wake word supports pause-then-speak — if the user says the wake phrase and pauses before speaking, the method waits for the next utterance as the command

Feature

Both methods are extension methods on ISpeechToTextService composing over ContinuousRecognize — no platform-specific code changes required

Feature

Sample apps updated with Wake Word and Keyword listening modes (MAUI + Blazor)

1.0 - May 2, 2026

Feature

ISpeechToTextService interface — platform-native speech recognition with permission management, continuous streaming, and listen-until-silence modes

Feature

ITextToSpeechService interface — platform-native text-to-speech with voice selection, speech rate, pitch, and volume control

Feature

IAudioSource interface — raw PCM audio capture from the device microphone (16kHz, 16-bit, mono)

Feature

IAudioPlayer interface — MP3 audio stream playback with play/stop control

Feature

SpeechRecognitionOptions — configurable culture, silence timeout, and on-device preference for STT

Feature

TextToSpeechOptions — configurable culture, voice, speech rate, pitch, and volume for TTS

Feature

ContinuousRecognize() — streaming recognition results via IAsyncEnumerable<SpeechRecognitionResult> with partial and final results

Feature

ListenUntilSilence() — simple dictation mode that returns the final transcription after silence is detected

Feature

GetVoicesAsync() — enumerate available TTS voices with optional culture filtering

Feature

AddSpeechServices() — single extension method to register all core services (STT, TTS, AudioSource, AudioPlayer)

FeatureAndroid

Android STT implementation using SpeechRecognizer with streaming partial results

FeatureAndroid

Android TTS implementation using Android.Speech.Tts.TextToSpeech

FeatureAndroid

Android audio capture via AudioRecord with 16kHz PCM streaming

FeatureAndroid

Android audio playback via MediaPlayer

FeatureiOS

iOS STT implementation using SFSpeechRecognizer with SFSpeechAudioBufferRecognitionRequest

FeatureiOS

iOS TTS implementation using AVSpeechSynthesizer

FeatureiOS

iOS audio capture via AVAudioEngine with PCM tap

FeatureiOS

iOS audio playback via AVAudioPlayer

Feature

Cloud provider abstraction — ISpeechToTextProvider and ITextToSpeechProvider interfaces for pluggable cloud backends

Feature

CloudSpeechToText and CloudTextToSpeech — bridge classes that combine platform audio with cloud provider APIs

Feature

AddCloudSpeechToText<T>() and AddCloudTextToSpeech<T>() — generic DI registration for custom cloud providers

Feature

Azure AI Speech provider — AddAzureSpeech() registers Azure STT and/or TTS with subscription key and region

Feature

Azure TTS with SSML prosody control — speech rate, pitch, and volume mapped to SSML elements

Feature

ElevenLabs TTS provider — AddElevenLabsTextToSpeech() registers ElevenLabs cloud TTS with configurable voice and model

Feature

PipeStream utility — thread-safe producer-consumer stream using System.IO.Pipelines for bridging audio capture with cloud providers

FeatureWASM

Browser/WebAssembly support — STT and TTS via Web Speech API, auto-detected at runtime via OperatingSystem.IsBrowser()

FeatureWASM

Browser STT implementation using SpeechRecognition API with streaming partial and final results

FeatureWASM

Browser TTS implementation using SpeechSynthesis API with voice selection, rate, pitch, and volume control

FeatureWASM

Browser audio playback via HTML5 Audio element with base64 data URL conversion

Feature

Blazor WebAssembly sample app demonstrating STT, TTS, and voice listing