TypeScript types and mock SDK for building external apps that integrate with Choreo Display.
# Configure npm for @choreoai scope (one-time setup)
echo "@choreoai:registry=https://npm.pkg.github.com" >> .npmrc
# Authenticate with GitHub (use your GitHub credentials)
npm login --registry=https://npm.pkg.github.com
# Install the package
npm install @choreoai/display-sdk-types
In production, include the Choreo Display SDK script in your HTML:
<script src="https://display2.choreo.ai/sdk/choreo-display-sdk.js"></script>
This exposes window.ChoreoSDK when your app runs inside Choreo Display.
For the development environment, load the matching development host's script:
<script src="https://canvas-dev.choreo.ai/sdk/choreo-display-sdk.js"></script>
The generic asset API is included in SDK types version 0.4.0. Before production
rollout, install that exact version or the dev tag from GitHub Packages and test
against the development host. A package's dev tag does not update existing apps
or the production browser SDK. Older apps can continue using their existing SDK
and video APIs.
Create a sdk.ts file that switches between mock (development) and real SDK (production):
// src/sdk.ts
import type { IChoreoSDK } from "@choreoai/display-sdk-types";
import { createMockSDK } from "@choreoai/display-sdk-types/mock";
// Use mock in development, real SDK in production
export const sdk: IChoreoSDK = import.meta.env.DEV
? createMockSDK({
assets: [
{ id: "poster", name: "Poster", mimeType: "image/png", url: "/poster.png" },
{ id: "v1", name: "Sample Video", mimeType: "video/mp4", url: "/sample.mp4" },
],
debug: true,
})
: (window as unknown as { ChoreoSDK: IChoreoSDK }).ChoreoSDK;
import { sdk } from "./sdk";
// Wait for SDK to be ready
await sdk.ready();
// Get configured assets, including videos
const assets = await sdk.getAssets();
const poster = assets.find(asset => asset.id === "poster");
if (poster) document.querySelector<HTMLImageElement>("#poster")!.src = poster.url;
// Existing video APIs remain available and return only videos
const videos = await sdk.getVideos();
const activeVideoUrl = videos[0]?.url;
// Start recording
await sdk.startRecording({ durationMs: 10000 });
// Listen for completion
sdk.once("recordingComplete", ({ recordingId }) => {
console.log("Recording complete:", recordingId);
});
// Exit with result
await sdk.exitApp({ result: "success" });
// Send logs to parent Sentry
sdk.log?.info("Experience completed", { phase: 4, energy: "pink" });
sdk.log?.error("Camera failed", { error: "NotAllowedError" });
// Release SDK-managed blob URLs when you no longer need them
if (activeVideoUrl) {
sdk.releaseUrl?.(activeVideoUrl);
}
sdk.destroy?.();
In the Gallery Editor, open the External App node and add files under Assets. Any file selectable from Choreo Drive can be included: images, audio, videos, PDFs, models, JSON, text, fonts, or other binary files. Publish the gallery to make these files available to the app. Use the configured asset ID to retrieve a file; filenames and display names are not identifiers.
| Method | Result | Behavior |
|---|---|---|
getConfig() |
SDKConfig |
assets lists configured metadata without transferring file contents. |
getAssets() |
Promise<AssetInfo[]> |
Returns every configured asset in editor order with a usable URL; cached files are resolved eagerly. |
getAssetUrl(id) |
Promise<string> |
Resolves one configured asset and reuses its URL for the current iframe session. |
AssetInfo contains id, name, url, and mimeType, with optional
fileName, size (bytes), and metadata. SDKConfig.assets contains the same
metadata without url. Unknown file types use application/octet-stream.
An unknown or unconfigured ID rejects with an error. An app can access only the
assets configured on its own node.
await sdk.ready();
// Resolve just the files you need, without loading the whole library.
const imageUrl = await sdk.getAssetUrl("poster");
document.querySelector<HTMLImageElement>("#poster")!.src = imageUrl;
const audioUrl = await sdk.getAssetUrl("soundtrack");
document.querySelector<HTMLAudioElement>("#audio")!.src = audioUrl;
const pdfUrl = await sdk.getAssetUrl("brochure");
document.querySelector<HTMLIFrameElement>("#pdf")!.src = pdfUrl;
const configUrl = await sdk.getAssetUrl("settings");
const settings = await fetch(configUrl).then(response => response.json());
console.log(settings);
sdk.releaseUrl?.(configUrl);
const binaryUrl = await sdk.getAssetUrl("model-data");
const bytes = await fetch(binaryUrl).then(response => response.arrayBuffer());
console.log(bytes.byteLength);
sdk.releaseUrl?.(binaryUrl);
The SDK returns file contents unchanged. Your app chooses how to parse or render them. Files that reference other files, such as a glTF model with separate textures, require those dependencies to be configured and resolved by the app; the SDK does not rewrite embedded relative URLs.
The display precaches configured assets as part of gallery preparation. When a file is cached, its bytes are transferred to the app and the SDK creates a Blob URL in the iframe's own origin. When a file is not cached, the API returns its published network URL. Fetching or displaying that fallback requires a network connection and, where applicable, the server's CORS permission.
Offline retrieval requires a completed preload. getAssets() eagerly resolves
all cached files, so use getConfig().assets and getAssetUrl(id) when a large
library only needs a few files. URLs are session resources, not durable links;
save asset IDs instead of Blob URLs.
getVideos(), listVideos(), getVideoUrl(), and playVideo() keep their
existing return shapes and expose only videos. Legacy galleries using
appVideos continue to work and are also visible through the new asset API on
an updated player. Gallery publishing retains a video compatibility projection
for older players and SDK scripts.
Both the player and the SDK script must support asset access. Updating only the TypeScript package does not update the installed player or its script. Check both sides before offering generic asset functionality:
const config = await sdk.getConfig();
const supportsAssets = typeof sdk.getAssets === "function"
&& typeof sdk.getAssetUrl === "function"
&& Array.isArray(config.assets);
if (supportsAssets) {
const assets = await sdk.getAssets();
console.log(assets);
} else {
// Older deployments retain the original video functionality.
const videos = await sdk.getVideos();
console.log(videos);
}
An older player rejects unsupported generic methods; the SDK does not silently replace a mixed asset library with videos.
Methods like getAssets(), getAssetUrl(), getVideos(), getVideoUrl(),
playVideo(), and getRecording()
can return SDK-managed blob URLs. These URLs remain valid for the current iframe
session until one of the following happens:
sdk.releaseUrl(url)sdk.destroy()The SDK does not revoke an earlier URL just because you fetched the same asset again later in the same session.
Asset and video methods share the same URL for a video. Release it only after
every element or other consumer using that URL is finished. Network URLs are
not revoked by releaseUrl().
Call sdk.releaseUrl(url) when your app has replaced a previously returned URL
in a <video>/<audio>/<img> element or dropped it from component state.
let currentVideoUrl: string | null = null;
async function playVideo(videoId: string) {
const { url } = await sdk.playVideo(videoId);
if (currentVideoUrl && currentVideoUrl !== url) {
sdk.releaseUrl?.(currentVideoUrl);
}
currentVideoUrl = url;
videoElement.src = url;
}
Call sdk.destroy() during app teardown or page unload. This revokes every
tracked SDK blob URL and removes SDK-owned event listeners.
window.addEventListener("beforeunload", () => {
sdk.destroy?.();
});
| Option | Type | Default | Description |
|---|---|---|---|
assets |
AssetInfo[] |
derived from videos | Complete mock asset library; when supplied, takes precedence over videos, including an empty array. |
videos |
VideoInfo[] |
2 samples | Mock video library |
latencyMs |
number |
0 |
Simulate network delay |
recordingDurationMs |
number |
5000 |
Recording duration |
failOnUpload |
boolean |
false |
Simulate upload failure |
failOnRecording |
boolean |
false |
Simulate recording failure |
debug |
boolean |
false |
Log SDK calls |
Mixed asset library:
const sdk = createMockSDK({
assets: [
{ id: "poster", name: "Poster", mimeType: "image/png", url: "/poster.png" },
{ id: "settings", name: "Settings", mimeType: "application/json", url: "/settings.json" },
{ id: "intro", name: "Introduction", mimeType: "video/mp4", url: "/intro.mp4" },
],
});
const assets = await sdk.getAssets(); // All three files
const videos = await sdk.getVideos(); // Introduction only
Custom video library:
const sdk = createMockSDK({
videos: [
{ id: "hip-hop", name: "Hip Hop Routine", url: "/videos/hiphop.mp4" },
{ id: "ballet", name: "Ballet Solo", url: "/videos/ballet.mp4" },
{
id: "jazz",
name: "Jazz Combo",
url: "/videos/jazz.mp4",
metadata: { difficulty: "advanced" },
},
],
});
Simulate network latency:
const sdk = createMockSDK({
latencyMs: 500, // 500ms delay on all SDK calls
debug: true, // Log calls to console
});
Test error handling:
// Test recording failures
const sdk = createMockSDK({
failOnRecording: true,
});
sdk.on("recordingError", ({ error }) => {
console.error("Recording failed:", error);
});
// Test upload failures
const sdk = createMockSDK({
failOnUpload: true,
});
sdk.on("uploadError", ({ taskId, error }) => {
console.error("Upload failed:", taskId, error);
});
Unit testing:
import { createMockSDK } from "@choreoai/display-sdk-types/mock";
import { describe, it, expect, vi } from "vitest";
describe("Recording Flow", () => {
it("completes recording and emits event", async () => {
const sdk = createMockSDK({ recordingDurationMs: 100 });
const onComplete = vi.fn();
sdk.on("recordingComplete", onComplete);
await sdk.startRecording();
await new Promise((r) => setTimeout(r, 150));
expect(onComplete).toHaveBeenCalledWith(
expect.objectContaining({ recordingId: expect.any(String) }),
);
});
});
External apps can route logs to the parent app's Sentry and console using the SDK log methods. Logs are automatically tagged with source: 'external-app', the app URL, and the slide name.
// Convenience methods (recommended)
sdk.log?.debug("Frame processed", { fps: 30 }); // Console only
sdk.log?.info("Face detected", { phase: 3 }); // Console + Sentry Logs
sdk.log?.warn("Model load slow", { duration: 5000 }); // Console + Sentry Logs
sdk.log?.error("Camera failed", { error: "denied" }); // Console + Sentry Logs + Sentry issue
// Direct call (equivalent)
sdk.log?.("info", "Face detected", { phase: 3 });
The log method silently fails if the parent doesn't support it (backward compatible with older display versions). Logging is fire-and-forget — there's no need to await the result.
The SDK provides parent-hosted face detection powered by face-api.js. Models are loaded once in the parent's web worker and persist across iframe reloads — no model re-loading overhead and no tensor memory leaks.
Face detection must be enabled in the gallery editor for each external-app slide that needs it. The gallery config specifies which models to pre-load.
Use detectFace for person identification (Phase 2-3 in interactive experiences):
// Draw camera frame to an offscreen canvas
const canvas = document.createElement("canvas");
const ctx = canvas.getContext("2d");
ctx.drawImage(cameraVideo, 0, 0, 640, 480);
// Create ImageBitmap (transferred to parent, zero-copy)
const bitmap = await createImageBitmap(canvas);
// Run detection via parent worker
const { descriptor, box, error } = await sdk.detectFace(bitmap, {
detector: "ssdMobilenetv1", // default
minConfidence: 0.5,
});
if (descriptor) {
// Rehydrate the 128-D face descriptor
const vec = new Float32Array(descriptor);
// Use for person matching
const distance = euclideanDistance(vec, lockedDescriptor);
}
if (box) {
// Use bounding box for face tracking / zoom
console.log("Face at:", box.x, box.y, box.width, box.height);
}
Use detectFacePresence for "is someone there?" checks (Phase 1):
const bitmap = await createImageBitmap(canvas);
const { detected, box } = await sdk.detectFacePresence(bitmap, {
detector: "tinyFaceDetector", // default, faster
inputSize: 416, // larger = better at distance
scoreThreshold: 0.2, // lower = more sensitive
});
if (detected) {
enterPhase2();
}
| Option | Type | Default | Description |
|---|---|---|---|
detector |
'tinyFaceDetector' | 'ssdMobilenetv1' |
varies | Detector model |
inputSize |
number | 320 | TinyFaceDetector resolution (128-608) |
scoreThreshold |
number | 0.5 | TinyFaceDetector confidence threshold |
minConfidence |
number | 0.5 | SSD MobileNet confidence threshold |
Both methods return an error field instead of throwing:
| Error | Meaning |
|---|---|
SERVICE_UNAVAILABLE |
Face detection not enabled for this slide, or worker failed |
UNSUPPORTED_MODEL |
Requested model not loaded in gallery config |
TIMEOUT |
Detection took >2 seconds |
NOT_READY |
Worker still initializing |
INVALID_INPUT |
ImageBitmap was closed or corrupted |
const result = await sdk.detectFace(bitmap);
if (result.error) {
console.warn("Detection failed:", result.error);
}
In iframe mode, the SDK returns camera constraints rather than a MediaStream
(streams cannot be transferred across iframe boundaries via postMessage).
Your app should call getUserMedia with the provided constraints:
const result = await sdk.getCameraStream({ width: 1280, height: 720 });
if (result instanceof MediaStream) {
// Mock/standalone mode - stream provided directly
videoElement.srcObject = result;
} else if (result.constraints) {
// Iframe mode - get stream yourself with provided constraints
const stream = await navigator.mediaDevices.getUserMedia(result.constraints);
videoElement.srcObject = stream;
// Store reference for cleanup
window._activeStream = stream;
}
When releasing the camera, stop all tracks on the local stream:
// Release local stream if you acquired it
if (window._activeStream) {
window._activeStream.getTracks().forEach((track) => track.stop());
window._activeStream = null;
}
// Also notify the SDK
await sdk.releaseCamera();
For recording, the SDK manages the stream internally on the parent side. Just call the recording methods:
// Start recording (10 seconds)
await sdk.startRecording({ durationMs: 10000 });
// Listen for completion
sdk.on("recordingComplete", async ({ recordingId, duration, size }) => {
console.log("Recording saved:", recordingId);
// Play back the recording
const { url } = await sdk.getRecording(recordingId);
if (!url) return;
const previousUrl = videoElement.dataset.recordingUrl;
if (previousUrl && previousUrl !== url) {
sdk.releaseUrl?.(previousUrl);
}
videoElement.src = url;
videoElement.dataset.recordingUrl = url;
// Or queue for upload (see Upload section below)
await sdk.queueUpload({ recordingId });
});
When a camera is mounted sideways (common with Logitech cameras in kiosks), the display startup menu allows selecting a rotation angle (0°, 90°, 180°, 270°). Logitech cameras default to 90°. The SDK exposes this angle so your app can apply a CSS rotation to its <video> element.
The cameraAngle is available from both getConfig() and getCameraStream():
// Via getConfig()
const config = await sdk.getConfig();
console.log(config.cameraAngle); // 0, 90, 180, or 270
// Via getCameraStream()
const result = await sdk.getCameraStream({ width: 1280, height: 720 });
if (!(result instanceof MediaStream)) {
console.log(result.cameraAngle); // 0, 90, 180, or 270
}
Apply a CSS transform on the <video> element to match the physical camera orientation.
Important: When rotating 90° or 270°, a plain rotate() will shrink the video to fit inside its original bounding box. You must also apply scale() to compensate for the dimension swap.
const result = await sdk.getCameraStream({ width: 1280, height: 720 });
if (!(result instanceof MediaStream)) {
const stream = await navigator.mediaDevices.getUserMedia(result.constraints);
videoElement.srcObject = stream;
const angle = result.cameraAngle || 0;
const isSwapped = angle === 90 || angle === 270;
videoElement.addEventListener("loadedmetadata", () => {
if (isSwapped) {
// Video dimensions are swapped after rotation — scale to fill container
const container = videoElement.parentElement!;
const scale = Math.min(
container.clientWidth / videoElement.videoHeight,
container.clientHeight / videoElement.videoWidth,
);
videoElement.style.transform = `rotate(${angle}deg) scale(${scale})`;
videoElement.style.transformOrigin = "center center";
} else if (angle !== 0) {
videoElement.style.transform = `rotate(${angle}deg)`;
}
});
}
Container CSS — the parent element should clip the overflow:
.camera-container {
position: relative;
width: 100%;
height: 100%;
overflow: hidden;
}
.camera-container video {
width: 100%;
height: 100%;
object-fit: cover;
}
import type { IChoreoSDK } from "@choreoai/display-sdk-types";
const sdk: IChoreoSDK = (window as any).ChoreoSDK;
await sdk.ready();
const video = document.querySelector("video")!;
const container = video.parentElement!;
// Get camera with rotation info
const result = await sdk.getCameraStream({ width: 1280, height: 720 });
if (result instanceof MediaStream) {
video.srcObject = result;
} else {
const stream = await navigator.mediaDevices.getUserMedia(result.constraints);
video.srcObject = stream;
const angle = result.cameraAngle || 0;
const isSwapped = angle === 90 || angle === 270;
video.addEventListener("loadedmetadata", () => {
if (isSwapped) {
const scale = Math.min(
container.clientWidth / video.videoHeight,
container.clientHeight / video.videoWidth,
);
video.style.transform = `rotate(${angle}deg) scale(${scale})`;
video.style.transformOrigin = "center center";
} else if (angle !== 0) {
video.style.transform = `rotate(${angle}deg)`;
}
});
}
When the camera is physically rotated (e.g. Logitech cameras mounted sideways), recordings need rotation handling. The SDK supports two approaches — choose based on your needs.
Record via sdk.startRecording() and attach the cameraAngle as upload metadata. The video file is stored unrotated; the consumer applies rotation at playback. No re-encoding, no quality loss.
const config = await sdk.getConfig();
const cameraAngle = config.cameraAngle; // 0, 90, 180, or 270
// Record using SDK (parent-side, unrotated)
await sdk.startRecording({ durationMs: 10000 });
sdk.once("recordingComplete", async ({ recordingId }) => {
const { taskId } = await sdk.queueUpload({
recordingId,
metadata: {
cameraAngle: String(cameraAngle),
capturedAt: new Date().toISOString(),
},
});
});
When to use: You control the playback viewer and can apply CSS rotation there, or the backend handles rotation in post-processing.
Record rotated video locally via a canvas and upload the blob directly. The uploaded file is correctly oriented everywhere — no consumer-side rotation needed.
const result = await sdk.getCameraStream({ width: 1280, height: 720 });
const stream = await navigator.mediaDevices.getUserMedia(result.constraints);
const angle = result.cameraAngle || 0;
// Set up canvas with rotated dimensions
const video = document.createElement("video");
video.srcObject = stream;
await video.play();
const isSwapped = angle === 90 || angle === 270;
const canvas = document.createElement("canvas");
canvas.width = isSwapped ? video.videoHeight : video.videoWidth;
canvas.height = isSwapped ? video.videoWidth : video.videoHeight;
const ctx = canvas.getContext("2d")!;
const radians = (angle * Math.PI) / 180;
// Draw rotated frames
function drawFrame() {
ctx.save();
ctx.translate(canvas.width / 2, canvas.height / 2);
ctx.rotate(radians);
ctx.drawImage(video, -video.videoWidth / 2, -video.videoHeight / 2);
ctx.restore();
requestAnimationFrame(drawFrame);
}
drawFrame();
// Record from canvas
const canvasStream = canvas.captureStream(30);
const recorder = new MediaRecorder(canvasStream, { mimeType: "video/webm" });
const chunks: Blob[] = [];
recorder.ondataavailable = (e) => {
if (e.data.size > 0) chunks.push(e.data);
};
recorder.start();
setTimeout(() => recorder.stop(), 10000); // 10 seconds
recorder.onstop = async () => {
const rotatedBlob = new Blob(chunks, { type: "video/webm" });
// Upload the pre-rotated blob via SDK
const { taskId } = await sdk.queueUpload({
blob: rotatedBlob,
metadata: {
cameraAngle: String(angle),
rotated: "true",
},
});
};
When to use: The video must be correctly oriented when downloaded or viewed outside your app, or you want to apply mirror/filters during recording.
| Metadata-Only | Canvas Blob Upload | |
|---|---|---|
| Complexity | Minimal — 3 lines of code | More involved — canvas + MediaRecorder setup |
| Video quality | Original quality preserved | Re-encoded through canvas |
| Upload latency | Immediate | Immediate (recorded already rotated) |
| Consumer requirement | Must read metadata and rotate | None — video is correctly oriented |
| Best for | Internal pipelines, controlled viewers | Public sharing, downloads, third-party viewers |
The SDK supports background uploads that survive tab close, powered by a service worker and the Background Sync API. Uploads are queued and processed reliably in the background.
When the gallery has recording upload enabled, the simplest upload requires no configuration:
sdk.once("recordingComplete", async ({ recordingId }) => {
const { taskId } = await sdk.queueUpload({ recordingId });
console.log("Upload queued:", taskId);
});
You can attach key-value pairs to uploads. These are stored as x-amz-meta-* headers on the S3 object and persisted alongside the file. This works with any S3-compatible storage provider including AWS S3, Cloudflare R2, and Choreo Drive.
sdk.once("recordingComplete", async ({ recordingId }) => {
const { taskId } = await sdk.queueUpload({
recordingId,
metadata: {
sessionId: "sess_abc123",
participantName: "Jane Doe",
danceName: "waltz",
capturedAt: new Date().toISOString(),
},
});
});
Metadata constraints:
sessionId becomes x-amz-meta-sessionid)const { taskId } = await sdk.queueUpload({ recordingId });
// Poll for completion
const interval = setInterval(async () => {
const status = await sdk.getUploadStatus(taskId);
if (status.status === "completed") {
clearInterval(interval);
console.log("Uploaded to:", status.retrievalUrl);
} else if (status.status === "failed") {
clearInterval(interval);
console.error("Upload failed:", status.error);
}
}, 2000);
import type { IChoreoSDK } from "@choreoai/display-sdk-types";
const sdk: IChoreoSDK = (window as any).ChoreoSDK;
await sdk.ready();
// 1. Start recording
await sdk.startRecording({ durationMs: 5000 });
// 2. Wait for completion, then upload with metadata
sdk.once("recordingComplete", async ({ recordingId, duration, size }) => {
console.log(`Recorded ${duration}ms, ${size} bytes`);
// 3. Queue upload with metadata
const { taskId } = await sdk.queueUpload({
recordingId,
metadata: {
sessionId: "my-session",
danceName: "tango",
},
});
// 4. Wait for upload to complete
sdk.once("uploadQueued", ({ taskId: uploadedTaskId }) => {
console.log("Upload started:", uploadedTaskId);
});
});
When the gallery has recordingUpload.enabled = true, recordings are auto-uploaded on completion without metadata. To attach metadata, use explicit queueUpload() instead. Note that if auto-upload is enabled, the recording may be uploaded twice (once automatically, once via your explicit call).
Once uploaded, metadata can be retrieved using any S3-compatible client via the HeadObject operation.
aws s3api head-object \
--bucket my-bucket \
--key recordings/rec-123.webm
# Response includes:
# "Metadata": {
# "sessionid": "sess_abc123",
# "participantname": "Jane Doe",
# "dancename": "waltz"
# }
R2 is fully S3-compatible. Use the AWS CLI with R2's endpoint:
aws s3api head-object \
--endpoint-url https://<account-id>.r2.cloudflarestorage.com \
--bucket my-bucket \
--key recordings/rec-123.webm
Or via the Cloudflare dashboard under R2 → Object Details → Custom Metadata.
Choreo Drive exposes an S3-compatible API that supports custom metadata on all object operations.
# Using curl
curl -I https://<bucket>.storage.choreo.ai/recordings/rec-123.webm
# Response headers:
# x-amz-meta-sessionid: sess_abc123
# x-amz-meta-dancename: waltz
# Using AWS CLI with Choreo Drive endpoint
aws s3api head-object \
--endpoint-url https://storage.choreo.ai \
--bucket my-bucket \
--key recordings/rec-123.webm
import { S3Client, HeadObjectCommand } from "@aws-sdk/client-s3";
const client = new S3Client({
region: "us-east-1",
endpoint: "https://storage.choreo.ai", // or R2/S3 endpoint
credentials: {
accessKeyId: "...",
secretAccessKey: "...",
},
});
const response = await client.send(
new HeadObjectCommand({
Bucket: "my-bucket",
Key: "recordings/rec-123.webm",
}),
);
console.log(response.Metadata);
// { sessionid: 'sess_abc123', dancename: 'waltz' }
Proprietary - Choreo, LLC