Repository navigation
feat(lyrics): AI singable translations #351
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
bchhngsaygez
wants to merge
11
commits into
tryliner:main
from
bchhngsaygez:feat/ai-lyrics-and-canvas-fx
Closed
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
ca4fa7a
feat(lyrics): add AI singable translation and dynamic canvas syllable…
bchhngsaygez acefddf
fix(lyrics): address code review feedback on AI translation and canva…
bchhngsaygez c45f634
fix(lyrics): resolve CodeRabbit review findings on AI translation and…
bchhngsaygez 984c151
fix(lyrics): address CodeRabbit follow-up review findings on line mat…
bchhngsaygez 68a79f5
test(ai-translate): add line-matching regression tests and verify tar…
bchhngsaygez 2eda898
fix(canvas): eliminate syllable scale tearing and enable dynamic lumi…
bchhngsaygez 36f428a
feat(canvas): restore unified syllable bounce lift and pop for all to…
bchhngsaygez 9983948
feat(lyrics): remove canvas bounce and enhance AI settings with provi…
bchhngsaygez 8d556f0
feat(ai-translate): add Claude and OpenRouter providers, fix provider…
bchhngsaygez 0fdbe24
feat(ai-translate): add Ollama provider for balanced 4x2 grid and fix…
bchhngsaygez ac01e20
fix(ai-translate): persist translation cache across restarts and prev…
bchhngsaygez File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| import { describe, it, expect } from "vitest"; | ||
| import { | ||
| SONG_TRANSLATION_SYSTEM_PROMPT, | ||
| SONG_SINGABLE_TRANSLATION_SYSTEM_PROMPT, | ||
| SONG_ROMANIZATION_SYSTEM_PROMPT, | ||
| } from "./services/aiTranslateService"; | ||
|
|
||
| describe("AI Lyrics Translation Prompts", () => { | ||
| it("generates correct system prompt enforcing line-by-line parity", () => { | ||
| const prompt = SONG_TRANSLATION_SYSTEM_PROMPT("English", "Vietnamese"); | ||
| expect(prompt).toContain("Maintain line-by-line parity"); | ||
| expect(prompt).toContain("poetic song translator"); | ||
| expect(prompt).toContain("Vietnamese"); | ||
| }); | ||
|
|
||
| it("generates a romanization prompt that forbids translation", () => { | ||
| expect(SONG_ROMANIZATION_SYSTEM_PROMPT).toContain("Maintain line-by-line parity"); | ||
| expect(SONG_ROMANIZATION_SYSTEM_PROMPT).toContain("Hepburn Romaji"); | ||
| expect(SONG_ROMANIZATION_SYSTEM_PROMPT).toContain("Do NOT translate"); | ||
| }); | ||
|
|
||
| it("generates musical singable prompt with syllable and rhyme rules", () => { | ||
| const singablePrompt = SONG_SINGABLE_TRANSLATION_SYSTEM_PROMPT( | ||
| "Japanese", | ||
| "English", | ||
| ); | ||
| expect(singablePrompt).toContain("singable"); | ||
| expect(singablePrompt).toContain("SYLLABLE COUNT & CADENCE"); | ||
| expect(singablePrompt).toContain("RHYME & TONAL MELODY"); | ||
|
|
||
| const prompt = SONG_TRANSLATION_SYSTEM_PROMPT("Japanese", "English", true); | ||
| expect(prompt).toContain("SYLLABLE COUNT & CADENCE"); | ||
| }); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| export * from "./types"; | ||
| export * from "./prompts"; | ||
| export * from "./services/aiTranslateService"; | ||
| export * from "./store/aiTranslateStore"; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| export const SONG_SINGABLE_TRANSLATION_SYSTEM_PROMPT = ( | ||
| sourceLang = "auto", | ||
| targetLang = "English", | ||
| ) => `You are a master musical lyricist and poetic song adapter specializing in writing singable ${targetLang} song lyrics. | ||
|
|
||
| Your task is NOT a dry or literal translation. You must adapt and recreate the lyrics into ${targetLang} so that an artist can SING IT DIRECTLY to the original vocal melody, rhythm, beat, and BPM. | ||
|
|
||
| CRITICAL MUSICAL & POETIC RULES: | ||
| 1. STRICT LINE-BY-LINE PARITY: If the input has N lines, return EXACTLY N lines in the same order. | ||
| 2. SYLLABLE COUNT & CADENCE: Match the syllable count and musical cadence of each original line closely so that every translated word fits naturally on the musical notes without squeezing or stretching. | ||
| 3. RHYME & TONAL MELODY: | ||
| - Weave harmonious internal and end rhymes and balance melodic rhythm typical of professional vocal songwriting. | ||
| - For tonal languages (such as Vietnamese or Chinese), accents and tones MUST align naturally with the pitch contour of the song melody, avoiding awkward or comical mispronunciations when sung. | ||
| 4. EMOTIONAL ESSENCE & POETIC BEAUTY: Preserve the emotional heart, storyline, atmosphere, and metaphors of the song. Express the meaning with rich, artistic, and evocative ${targetLang} imagery. | ||
| 5. VOCALIZATIONS: Keep non-lexical musical interjections and hooks (e.g., 'Oh', 'Yeah', 'La la la', 'Na na na') unchanged. | ||
| 6. OUTPUT FORMAT: Output ONLY the singable lyrics lines. Never output line numbers, romanizations, commentary, or markdown formatting.`; | ||
|
|
||
| export const SONG_TRANSLATION_SYSTEM_PROMPT = ( | ||
| sourceLang = "auto", | ||
| targetLang = "English", | ||
| singable = false, | ||
| ) => { | ||
| if (singable) { | ||
| return SONG_SINGABLE_TRANSLATION_SYSTEM_PROMPT(sourceLang, targetLang); | ||
| } | ||
| return `You are an elite, poetic song translator specializing in musical context and emotional nuance. | ||
| Your task is to translate the following song lyrics from ${sourceLang} into ${targetLang}. | ||
| STRICT RULES: | ||
| 1. Maintain line-by-line parity. If the input has N lines, return EXACTLY N lines in the same order. | ||
| 2. Preserve emotional resonance, metaphors, and rhythm. Avoid literal, robotic word-by-word translation. | ||
| 3. Natural phrasing: The translated line should sound melodic and natural when sung or read aloud. | ||
| 4. Do NOT output line numbers, markdown formatting, prefixes, or commentary. Output ONLY the translated lines. | ||
| 5. If a line is purely an interjection or musical vocalization (e.g., 'Oh', 'Yeah', 'La la la'), keep it as is.`; | ||
| }; | ||
|
|
||
| export const SONG_ROMANIZATION_SYSTEM_PROMPT = `You are an expert lyrics romanizer and transliteration specialist. | ||
| Your task is to transliterate song lyrics into standard Latin alphabet (pronunciation): | ||
| - For Japanese lyrics, provide standard Hepburn Romaji with spaces between words (e.g. "zankoku na tenshi no te-ze"). | ||
| - For Korean lyrics, provide Revised Romanization. | ||
| - For Chinese lyrics, provide Pinyin. | ||
| - If a line is already in English or Latin alphabet, leave it exactly as is. | ||
| STRICT RULES: | ||
| 1. Maintain line-by-line parity. If the input has N lines, return EXACTLY N lines in the same order. | ||
| 2. Do NOT translate the meaning. Output ONLY the pronunciation/romanization in Latin alphabet. | ||
| 3. Do NOT output line numbers, markdown formatting, commentary, or quotes. Output ONLY the romanized lines.`; |
142 changes: 142 additions & 0 deletions
142
src/features/ai-translate/services/aiTranslateService.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| import { describe, it, expect, vi, beforeEach } from "vitest"; | ||
| import { | ||
| translateSongLyrics, | ||
| romanizeSongLyrics, | ||
| } from "./aiTranslateService"; | ||
|
|
||
| function mockFetchJson(data: unknown): typeof fetch { | ||
| return vi.fn<typeof fetch>(async () => | ||
| new Response(JSON.stringify(data), { status: 200 }), | ||
| ); | ||
| } | ||
|
|
||
| function mockGoogleTranslate( | ||
| dictionary: Record<string, string>, | ||
| onQuery?: (q: string, tl: string | null) => void, | ||
| ): typeof fetch { | ||
| return vi.fn<typeof fetch>(async (input: RequestInfo | URL) => { | ||
| const url = new URL(String(input)); | ||
| const q = url.searchParams.get("q") ?? ""; | ||
| const tl = url.searchParams.get("tl"); | ||
| onQuery?.(q, tl); | ||
| const translated = dictionary[q] ?? q; | ||
| return new Response(JSON.stringify([[[translated, q]]]), { status: 200 }); | ||
| }); | ||
| } | ||
|
|
||
| describe("AI Lyrics Translation Service", () => { | ||
| beforeEach(() => { | ||
| vi.restoreAllMocks(); | ||
| }); | ||
|
|
||
| it("uses free translator for google_translate provider", async () => { | ||
| const targetLangs: (string | null)[] = []; | ||
| const mockFetch = mockGoogleTranslate( | ||
| { | ||
| "Hello World": "Xin chào thế giới", | ||
| Peace: "Hòa bình", | ||
| }, | ||
| (_q, tl) => targetLangs.push(tl), | ||
| ); | ||
| global.fetch = mockFetch; | ||
|
|
||
| const result = await translateSongLyrics(["Hello World", "", "Peace"], { | ||
| provider: "google_translate", | ||
| targetLang: "vi", | ||
| }); | ||
|
|
||
| expect(result).toEqual(["Xin chào thế giới", "", "Hòa bình"]); | ||
| expect(mockFetch).toHaveBeenCalledTimes(2); | ||
| expect(targetLangs.length).toBe(2); | ||
| expect(targetLangs.every((tl) => tl === "vi")).toBe(true); | ||
| }); | ||
|
|
||
| it("throws when API key is missing for api key required providers", async () => { | ||
| for (const provider of ["gemini", "claude", "openai", "deepseek", "openrouter"] as const) { | ||
| const name = | ||
| provider === "gemini" | ||
| ? "Gemini" | ||
| : provider === "claude" | ||
| ? "Claude" | ||
| : provider === "openai" | ||
| ? "OpenAI" | ||
| : provider === "deepseek" | ||
| ? "DeepSeek" | ||
| : "OpenRouter"; | ||
| await expect( | ||
| translateSongLyrics(["Hello"], { provider, targetLang: "vi" }), | ||
| ).rejects.toThrow(`API key is required for ${name}`); | ||
| await expect( | ||
| romanizeSongLyrics(["Hello"], { provider, targetLang: "vi" }), | ||
| ).rejects.toThrow(`API key is required for ${name}`); | ||
| } | ||
| }); | ||
|
|
||
| it("keeps subtitle indices aligned when an LLM drops blank lines", async () => { | ||
| global.fetch = mockFetchJson({ | ||
| candidates: [{ content: { parts: [{ text: "Dòng một\nDòng ba" }] } }], | ||
| }); | ||
|
|
||
| const result = await translateSongLyrics( | ||
| ["Line one", "", "Line three"], | ||
| { provider: "gemini", apiKey: "test-key", targetLang: "vi" }, | ||
| ); | ||
|
|
||
| expect(result).toEqual(["Dòng một", "", "Dòng ba"]); | ||
| }); | ||
|
|
||
| it("romanizes Japanese lyrics and skips lines that are already Latin", async () => { | ||
| global.fetch = vi.fn<typeof fetch>(async (input: RequestInfo | URL) => { | ||
| const q = decodeURIComponent( | ||
| new URL(String(input)).searchParams.get("q") ?? "", | ||
| ); | ||
| const result = q === "残酷な天使のテーゼ" ? "Zankoku na tenshi no tēze" : q; | ||
| return new Response(JSON.stringify([[[null, null, null, result]]]), { status: 200 }); | ||
| }); | ||
|
|
||
| const result = await romanizeSongLyrics( | ||
| ["残酷な天使のテーゼ", "Hello world"], | ||
| { provider: "google_translate", targetLang: "vi" }, | ||
| ); | ||
|
|
||
| expect(result).toEqual(["Zankoku na tenshi no tēze", ""]); | ||
| }); | ||
|
|
||
| it("translates via OpenAI compatible provider with custom baseUrl and model", async () => { | ||
| let capturedUrl = ""; | ||
| interface ChatPayload { | ||
| model?: string; | ||
| messages?: Array<{ content?: string }>; | ||
| } | ||
| let capturedBody: ChatPayload = {}; | ||
| let capturedHeaders: HeadersInit | undefined = undefined; | ||
|
|
||
| global.fetch = vi.fn<typeof fetch>(async (input: RequestInfo | URL, init?: RequestInit) => { | ||
| capturedUrl = String(input); | ||
| capturedHeaders = init?.headers; | ||
| capturedBody = JSON.parse(String(init?.body ?? "{}")) as ChatPayload; | ||
| return new Response(JSON.stringify({ | ||
| choices: [ | ||
| { message: { content: "Xin chào bầu trời xanh\nNắng chiếu lấp lánh quanh" } }, | ||
| ], | ||
| }), { status: 200 }); | ||
| }); | ||
|
|
||
| const result = await translateSongLyrics(["Blue sky", "Bright sun"], { | ||
| provider: "openai_compatible", | ||
| apiKey: "custom-token-123", | ||
| baseUrl: "https://api.groq.com/openai/v1", | ||
| model: "llama-3.3-70b-versatile", | ||
| targetLang: "vi", | ||
| singableMode: true, | ||
| }); | ||
|
|
||
| expect(capturedUrl).toBe("https://api.groq.com/openai/v1/chat/completions"); | ||
| const headers = new Headers(capturedHeaders); | ||
| expect(headers.get("Authorization")).toBe("Bearer custom-token-123"); | ||
| expect(capturedBody.model).toBe("llama-3.3-70b-versatile"); | ||
| const content = capturedBody.messages?.[0]?.content ?? ""; | ||
| expect(content).toContain("SYLLABLE COUNT"); | ||
| expect(result).toEqual(["Xin chào bầu trời xanh", "Nắng chiếu lấp lánh quanh"]); | ||
| }); | ||
| }); |
115 changes: 115 additions & 0 deletions
115
src/features/ai-translate/services/aiTranslateService.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| import type { TranslationConfig } from "../types"; | ||
| import { dropUnchangedRomanization } from "./lineMatching"; | ||
| import { | ||
| translateWithGemini, | ||
| romanizeWithGemini, | ||
| } from "./geminiProvider"; | ||
| import { | ||
| translateWithClaude, | ||
| romanizeWithClaude, | ||
| } from "./claudeProvider"; | ||
| import { | ||
| translateWithOpenAI, | ||
| romanizeWithOpenAI, | ||
| translateWithDeepSeek, | ||
| romanizeWithDeepSeek, | ||
| translateWithOpenRouter, | ||
| romanizeWithOpenRouter, | ||
| translateWithOpenAiCompatible, | ||
| romanizeWithOpenAiCompatible, | ||
| } from "./openAiProvider"; | ||
| import { | ||
| translateWithOllama, | ||
| romanizeWithOllama, | ||
| } from "./ollamaProvider"; | ||
| import { | ||
| translateWithGoogleFree, | ||
| romanizeWithGoogleFree, | ||
| } from "./googleFreeProvider"; | ||
|
|
||
| export * from "../prompts"; | ||
|
|
||
| export async function translateSongLyrics( | ||
| lines: string[], | ||
| config: TranslationConfig, | ||
| ): Promise<string[]> { | ||
| if (lines.length === 0) return []; | ||
|
|
||
| const { provider, apiKey, targetLang, model, baseUrl, singableMode } = config; | ||
|
|
||
| if (provider === "gemini") { | ||
| if (!apiKey?.trim()) throw new Error("API key is required for Gemini"); | ||
| return translateWithGemini(lines, apiKey, targetLang, model, singableMode); | ||
| } | ||
| if (provider === "claude") { | ||
| if (!apiKey?.trim()) throw new Error("API key is required for Claude"); | ||
| return translateWithClaude(lines, apiKey, targetLang, model, singableMode); | ||
| } | ||
| if (provider === "openai") { | ||
| if (!apiKey?.trim()) throw new Error("API key is required for OpenAI"); | ||
| return translateWithOpenAI(lines, apiKey, targetLang, model, singableMode); | ||
| } | ||
| if (provider === "deepseek") { | ||
| if (!apiKey?.trim()) throw new Error("API key is required for DeepSeek"); | ||
| return translateWithDeepSeek(lines, apiKey, targetLang, model, singableMode); | ||
| } | ||
| if (provider === "openrouter") { | ||
| if (!apiKey?.trim()) throw new Error("API key is required for OpenRouter"); | ||
| return translateWithOpenRouter(lines, apiKey, targetLang, model, singableMode); | ||
| } | ||
| if (provider === "ollama") { | ||
| return translateWithOllama(lines, apiKey, targetLang, baseUrl, model, singableMode); | ||
| } | ||
| if (provider === "openai_compatible") { | ||
| return translateWithOpenAiCompatible( | ||
| lines, | ||
| apiKey || "", | ||
| targetLang, | ||
| baseUrl, | ||
| model, | ||
| singableMode, | ||
| ); | ||
| } | ||
|
|
||
| return translateWithGoogleFree(lines, targetLang); | ||
| } | ||
|
|
||
| export async function romanizeSongLyrics( | ||
| lines: string[], | ||
| config: TranslationConfig, | ||
| ): Promise<string[]> { | ||
| if (lines.length === 0) return []; | ||
|
|
||
| const { provider, apiKey, model, baseUrl } = config; | ||
|
|
||
| let result: string[]; | ||
| if (provider === "gemini") { | ||
| if (!apiKey?.trim()) throw new Error("API key is required for Gemini"); | ||
| result = await romanizeWithGemini(lines, apiKey, model); | ||
| } else if (provider === "claude") { | ||
| if (!apiKey?.trim()) throw new Error("API key is required for Claude"); | ||
| result = await romanizeWithClaude(lines, apiKey, model); | ||
| } else if (provider === "openai") { | ||
| if (!apiKey?.trim()) throw new Error("API key is required for OpenAI"); | ||
| result = await romanizeWithOpenAI(lines, apiKey, model); | ||
| } else if (provider === "deepseek") { | ||
| if (!apiKey?.trim()) throw new Error("API key is required for DeepSeek"); | ||
| result = await romanizeWithDeepSeek(lines, apiKey, model); | ||
| } else if (provider === "openrouter") { | ||
| if (!apiKey?.trim()) throw new Error("API key is required for OpenRouter"); | ||
| result = await romanizeWithOpenRouter(lines, apiKey, model); | ||
| } else if (provider === "ollama") { | ||
| result = await romanizeWithOllama(lines, apiKey, baseUrl, model); | ||
| } else if (provider === "openai_compatible") { | ||
| result = await romanizeWithOpenAiCompatible( | ||
| lines, | ||
| apiKey || "", | ||
| baseUrl, | ||
| model, | ||
| ); | ||
| } else { | ||
| result = await romanizeWithGoogleFree(lines); | ||
| } | ||
|
|
||
| return dropUnchangedRomanization(result, lines); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| import { describe, it, expect, vi, beforeEach } from "vitest"; | ||
| import { translateWithClaude, romanizeWithClaude } from "./claudeProvider"; | ||
|
|
||
| describe("claudeProvider", () => { | ||
| beforeEach(() => { | ||
| vi.restoreAllMocks(); | ||
| }); | ||
|
|
||
| it("calls Claude API with proper headers and payload", async () => { | ||
| let capturedUrl = ""; | ||
| let capturedHeaders: HeadersInit | undefined; | ||
| interface ClaudePayload { | ||
| model?: string; | ||
| messages?: Array<{ content?: string }>; | ||
| } | ||
| let capturedBody: ClaudePayload = {}; | ||
|
|
||
| global.fetch = vi.fn<typeof fetch>(async (input, init) => { | ||
| capturedUrl = String(input); | ||
| capturedHeaders = init?.headers; | ||
| capturedBody = JSON.parse(String(init?.body ?? "{}")) as ClaudePayload; | ||
| return new Response( | ||
| JSON.stringify({ | ||
| content: [{ type: "text", text: "Xin chào\nTạm biệt" }], | ||
| }), | ||
| { status: 200 }, | ||
|
bchhngsaygez marked this conversation as resolved.
Resolved
|
||
| ); | ||
| }); | ||
|
|
||
| const res = await translateWithClaude(["Hello", "Goodbye"], "sk-ant-test", "vi"); | ||
| expect(capturedUrl).toBe("https://api.anthropic.com/v1/messages"); | ||
| const headers = new Headers(capturedHeaders); | ||
| expect(headers.get("x-api-key")).toBe("sk-ant-test"); | ||
| expect(headers.get("anthropic-version")).toBe("2023-06-01"); | ||
| expect(capturedBody.model).toBe("claude-3-5-haiku-20241022"); | ||
| expect(res).toEqual(["Xin chào", "Tạm biệt"]); | ||
| }); | ||
|
|
||
| it("handles romanization with Claude", async () => { | ||
| global.fetch = vi.fn<typeof fetch>(async () => | ||
| new Response( | ||
| JSON.stringify({ | ||
| content: [{ type: "text", text: "Konnichiwa" }], | ||
| }), | ||
| { status: 200 }, | ||
| ), | ||
| ); | ||
|
|
||
| const res = await romanizeWithClaude(["こんにちは"], "sk-ant-test"); | ||
| expect(res).toEqual(["Konnichiwa"]); | ||
| }); | ||
| }); | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.