Image generation

title: ‘Image Generation’ description: “Blend Animus’ automatic media workflows with direct API control for character renders, photo edits, and more.” icon: ‘image’

Image generation now ships with a unified set of REST endpoints (/generation/*) alongside the automatic workflows inside the Animus SDK. You can let the SDK detect when an assistant promises an image, or call the media API directly when you need precise control.

Looking for the HTTP endpoints? Jump to the REST Image Generation guide.

How It Works

Two-Phase Process

Image generation operates in two phases:

Detection Phase: The API analyzes the conversation to determine if the assistant has agreed to send an image
Generation Phase: If an image is requested, the SDK automatically generates it using the provided prompt

import { AnimusClient } from 'animus-client';

const client = new AnimusClient({
  tokenProviderUrl: 'https://your-backend.com/api/get-animus-token',
  chat: {
    model: 'vivian-llama3.1-70b-1.0-fp8',
    systemMessage: 'You are a helpful assistant.',
    check_image_generation: true // Enable automatic image generation
  }
});

// When the AI agrees to send an image, it's automatically generated
client.chat.send("Can you show me a beautiful sunset over the ocean?");

// The SDK handles everything automatically:
// 1. AI responds with agreement and creates image prompt
// 2. Image is generated using that prompt
// 3. Image is added to chat history
// 4. Events are emitted for UI updates

Basic Usage

Enable Image Generation

Enable automatic image generation in your client configuration:

const client = new AnimusClient({
  tokenProviderUrl: 'https://your-backend.com/api/get-animus-token',
  chat: {
    model: 'vivian-llama3.1-70b-1.0-fp8',
    systemMessage: 'You are a creative assistant who can generate images.',
    check_image_generation: true // Enable automatic detection and generation
  }
});

Important: Automatic image generation and adding images to message history only happens when using the chat.send() method. If you’re using chat.completions(), you’ll need to manually call client.generateImage() or use your own custom/3rd party image generation endpoint.

Listen for Image Generation Events

Set up event listeners to handle the image generation process:

// Image generation lifecycle events
client.on('imageGenerationStart', (data) => {
  console.log('Starting image generation:', data.prompt);
  showImageGenerationIndicator();
});

client.on('imageGenerationComplete', (data) => {
  console.log('Image generated:', data.imageUrl);
  displayGeneratedImage(data.imageUrl);
  hideImageGenerationIndicator();
});

client.on('imageGenerationError', (data) => {
  console.error('Image generation failed:', data.error);
  showImageGenerationError(data.error);
  hideImageGenerationIndicator();
});

// Send a message that might trigger image generation
client.chat.send("Can you show a painting you've created that you're the most proud of?");

Direct Image Generation

Manual Image Generation

You can also generate images directly without going through chat:

try {
  const imageUrl = await client.generateImage("A serene mountain landscape at sunset with a lake reflection");
  console.log('Generated image URL:', imageUrl);
  
  // The image is automatically added to chat history
  displayImage(imageUrl);
} catch (error) {
  console.error('Image generation failed:', error);
}

Custom Image Prompts

Generate images with detailed prompts:

const detailedPrompt = `
  A photorealistic image of a cozy coffee shop interior during golden hour.
  Warm lighting streaming through large windows, wooden furniture,
  plants on shelves, steam rising from coffee cups, soft shadows,
  inviting atmosphere, high quality, detailed textures.
`;

const imageUrl = await client.generateImage(detailedPrompt);

Integration with Other Features

Image Generation with AutoTurn

Image generation works seamlessly with conversational turns:

const client = new AnimusClient({
  tokenProviderUrl: 'https://your-backend.com/api/get-animus-token',
  chat: {
    model: 'vivian-llama3.1-70b-1.0-fp8',
    systemMessage: 'You are a creative assistant.',
    autoTurn: true,                    // Enable conversational turns
    check_image_generation: true       // Enable image generation
  }
});

// Listen for both auto-turn and image generation events
client.on('messageComplete', (data) => {
  console.log(`${data.messageType} message completed: ${data.content}`);
  
  // Check if all messages in the sequence are complete
  if (data.totalMessages) {
    console.log('All messages completed - image generation may start now');
  }
});

client.on('imageGenerationStart', (data) => {
  console.log('Image generation started after conversation turns');
});

// This might trigger both auto-turn responses AND image generation
client.chat.send("I love Van Gogh's paintings! Can you paint something in his style for me?");

Image Generation Coordination

Image generation intelligently coordinates with other features:

After AutoTurn: Images are generated only after all conversation turns are complete
Follow-up Protection: Follow-up requests are blocked immediately after image generation to prevent loops
Event Sequencing: Events are emitted in the correct order for proper UI updates

Advanced Image Generation

Custom Image Generation Settings

While the SDK handles image generation automatically, you can customize the process:

// The SDK uses these default settings for image generation
// (These are handled internally, but shown for reference)
const imageGenerationSettings = {
  model: 'flux-schnell',           // Image generation model
  width: 1024,                     // Image width
  height: 1024,                    // Image height
  guidance_scale: 7.5,             // How closely to follow the prompt
  num_inference_steps: 20,         // Quality vs speed tradeoff
  seed: null                       // Random seed (null for random)
};

Handling Image Generation Responses

When image generation is enabled, responses may include image prompts. However, automatic generation behavior differs between methods:

// Using completions() - requires manual image generation
const response = await client.chat.completions({
  messages: [
    { role: 'user', content: 'You have any more photos from your vacation in Spain?' }
  ],
  check_image_generation: true
});

const message = response.choices[0].message;

if (message.image_prompt) {
  console.log('AI created image prompt:', message.image_prompt);
  console.log('AI response:', message.content);
  
  // With completions(), you must manually generate the image
  const imageUrl = await client.generateImage(message.image_prompt);
  
  // You also need to manually add it to history if desired
  // (This is handled automatically with chat.send())
}

Key Difference: When using chat.completions(), the SDK will detect image prompts but will not automatically generate images or add them to chat history. You must handle this manually. Only chat.send() provides fully automatic image generation and history management.

Image History Management

Image Format in Chat History

Generated images are automatically added to chat history in a structured format:

// Images appear in chat history like this:
const historyMessage = {
  role: 'assistant',
  content: '<image description="A serene mountain landscape at sunset with a lake reflection" />',
  timestamp: '2024-01-15T10:30:00Z'
};

// Get chat history including images
const history = client.chat.getChatHistory();
history.forEach(message => {
  if (message.content.includes('<image')) {
    console.log('Found image in history:', message.content);
  }
});

Managing Image Context

Generated images provide context back to the AI:

// After generating an image, the AI knows what was shown
client.chat.send("Can you show me something peaceful?");
// Image is generated and added to history

// The AI now has context about the image
client.chat.send("Can you make the colors more vibrant?");
// AI knows what image you're referring to

Error Handling

Comprehensive Error Handling

Implement robust error handling for image generation:

client.on('imageGenerationError', (data) => {
  console.error('Image generation failed:', data.error);
  
  // Handle different types of errors
  if (data.error.includes('content policy')) {
    showContentPolicyError();
  } else if (data.error.includes('rate limit')) {
    showRateLimitError();
  } else if (data.error.includes('timeout')) {
    showTimeoutError();
  } else {
    showGenericImageError(data.error);
  }
});

// Fallback for direct image generation
try {
  const imageUrl = await client.generateImage(prompt);
} catch (error) {
  if (error.message.includes('Invalid prompt')) {
    console.error('Prompt was rejected:', error.message);
  } else if (error.message.includes('Service unavailable')) {
    console.error('Image service is down:', error.message);
  } else {
    console.error('Unexpected error:', error.message);
  }
}

Graceful Degradation

Handle image generation failures gracefully:

class ImageGenerationHandler {
  private retryCount = 0;
  private maxRetries = 3;

  async handleImageGeneration(prompt: string) {
    try {
      const imageUrl = await client.generateImage(prompt);
      this.retryCount = 0; // Reset on success
      return imageUrl;
    } catch (error) {
      console.error(`Image generation attempt ${this.retryCount + 1} failed:`, error);
      
      if (this.retryCount < this.maxRetries) {
        this.retryCount++;
        console.log(`Retrying image generation (${this.retryCount}/${this.maxRetries})...`);
        
        // Wait before retry
        await new Promise(resolve => setTimeout(resolve, 2000 * this.retryCount));
        return this.handleImageGeneration(prompt);
      } else {
        this.retryCount = 0;
        throw new Error('Image generation failed after maximum retries');
      }
    }
  }
}

UI Implementation

Complete Image Generation UI

Here’s a complete example of handling image generation in your UI:

class ImageGenerationUI {
  private imageContainer: HTMLElement;
  private loadingIndicator: HTMLElement;
  private errorContainer: HTMLElement;

  constructor() {
    this.setupEventListeners();
  }

  setupEventListeners() {
    client.on('imageGenerationStart', (data) => {
      this.showImageGenerationStart(data.prompt);
    });

    client.on('imageGenerationComplete', (data) => {
      this.showGeneratedImage(data.imageUrl);
    });

    client.on('imageGenerationError', (data) => {
      this.showImageGenerationError(data.error);
    });
  }

  showImageGenerationStart(prompt: string) {
    // Show loading state
    this.loadingIndicator.style.display = 'block';
    this.loadingIndicator.innerHTML = `
      <div class="image-loading">
        <div class="spinner"></div>
        <p>Generating image: "${prompt.substring(0, 50)}..."</p>
      </div>
    `;
    
    // Hide any previous errors
    this.errorContainer.style.display = 'none';
  }

  showGeneratedImage(imageUrl: string) {
    // Hide loading indicator
    this.loadingIndicator.style.display = 'none';
    
    // Create image element
    const imageElement = document.createElement('img');
    imageElement.src = imageUrl;
    imageElement.alt = 'Generated image';
    imageElement.className = 'generated-image';
    
    // Add loading and error handlers
    imageElement.onload = () => {
      imageElement.classList.add('loaded');
    };
    
    imageElement.onerror = () => {
      this.showImageLoadError();
    };
    
    // Add to container
    this.imageContainer.appendChild(imageElement);
    
    // Scroll to show the image
    imageElement.scrollIntoView({ behavior: 'smooth' });
  }

  showImageGenerationError(error: string) {
    // Hide loading indicator
    this.loadingIndicator.style.display = 'none';
    
    // Show error message
    this.errorContainer.style.display = 'block';
    this.errorContainer.innerHTML = `
      <div class="image-error">
        <p>Failed to generate image: ${error}</p>
        <button onclick="this.retryImageGeneration()">Try Again</button>
      </div>
    `;
  }

  showImageLoadError() {
    this.errorContainer.style.display = 'block';
    this.errorContainer.innerHTML = `
      <div class="image-error">
        <p>Failed to load generated image</p>
      </div>
    `;
  }

  retryImageGeneration() {
    // Implement retry logic
    console.log('Retrying image generation...');
  }
}

// Initialize the UI handler
const imageUI = new ImageGenerationUI();

CSS for Image Generation UI

.image-loading {
  display: flex;
  flex-direction: column;
  align-items: center;
  padding: 20px;
  background: #f5f5f5;
  border-radius: 8px;
  margin: 10px 0;
}

.spinner {
  width: 40px;
  height: 40px;
  border: 4px solid #e0e0e0;
  border-top: 4px solid #007bff;
  border-radius: 50%;
  animation: spin 1s linear infinite;
  margin-bottom: 10px;
}

@keyframes spin {
  0% { transform: rotate(0deg); }
  100% { transform: rotate(360deg); }
}

.generated-image {
  max-width: 100%;
  height: auto;
  border-radius: 8px;
  box-shadow: 0 4px 8px rgba(0,0,0,0.1);
  opacity: 0;
  transition: opacity 0.3s ease;
  margin: 10px 0;
}

.generated-image.loaded {
  opacity: 1;
}

.image-error {
  background: #fee;
  border: 1px solid #fcc;
  border-radius: 8px;
  padding: 15px;
  margin: 10px 0;
  color: #c33;
}

.image-error button {
  background: #007bff;
  color: white;
  border: none;
  padding: 8px 16px;
  border-radius: 4px;
  cursor: pointer;
  margin-top: 10px;
}

Best Practices

Prompt Engineering for Images

// Good: Specific, detailed prompts
const goodPrompts = [
  "A photorealistic portrait of a golden retriever sitting in a sunny meadow, soft natural lighting, shallow depth of field",
  "Minimalist line art illustration of a coffee cup with steam, black ink on white background, simple and clean",
  "Digital art of a cyberpunk cityscape at night, neon lights reflecting on wet streets, futuristic architecture"
];

// Avoid: Vague or problematic prompts
const avoidPrompts = [
  "Something cool",           // Too vague
  "A picture",               // No specific content
  "Make it look good"        // No clear direction
];

Performance Optimization

// Cache generated images to avoid regeneration
class ImageCache {
  private cache = new Map<string, string>();

  async getOrGenerateImage(prompt: string): Promise<string> {
    // Create a hash of the prompt for caching
    const promptHash = await this.hashPrompt(prompt);
    
    if (this.cache.has(promptHash)) {
      console.log('Using cached image for prompt');
      return this.cache.get(promptHash)!;
    }

    const imageUrl = await client.generateImage(prompt);
    this.cache.set(promptHash, imageUrl);
    return imageUrl;
  }

  private async hashPrompt(prompt: string): Promise<string> {
    const encoder = new TextEncoder();
    const data = encoder.encode(prompt);
    const hashBuffer = await crypto.subtle.digest('SHA-256', data);
    const hashArray = Array.from(new Uint8Array(hashBuffer));
    return hashArray.map(b => b.toString(16).padStart(2, '0')).join('');
  }
}

Content Guidelines

// Implement content filtering for image prompts
function validateImagePrompt(prompt: string): boolean {
  const prohibitedTerms = [
    'violence', 'gore', 'explicit', 'nsfw',
    // Add your content policy terms
  ];

  const lowerPrompt = prompt.toLowerCase();
  return !prohibitedTerms.some(term => lowerPrompt.includes(term));
}

// Use before generating images
if (validateImagePrompt(userPrompt)) {
  await client.generateImage(userPrompt);
} else {
  console.warn('Prompt violates content policy');
}

Next Steps

Event System

Master the comprehensive event system for image generation

Auto-Turn Conversations

Combine image generation with natural conversation flow

Tool Calling

Create tools that can generate images programmatically

Media & Vision

Analyze generated images with vision capabilities

Get Started

SDK Features

Advanced SDK

REST API Integration

Models

title: ‘Image Generation’ description: “Blend Animus’ automatic media workflows with direct API control for character renders, photo edits, and more.” icon: ‘image’

How It Works

Two-Phase Process

Basic Usage

Enable Image Generation

Listen for Image Generation Events

Direct Image Generation

Manual Image Generation

Custom Image Prompts

Integration with Other Features

Image Generation with AutoTurn

Image Generation Coordination

Advanced Image Generation

Custom Image Generation Settings

Handling Image Generation Responses

Image History Management

Image Format in Chat History

Managing Image Context

Error Handling

Comprehensive Error Handling

Graceful Degradation

UI Implementation

Complete Image Generation UI

CSS for Image Generation UI

Best Practices

Prompt Engineering for Images

Performance Optimization

Content Guidelines

Next Steps

Event System

Auto-Turn Conversations

Tool Calling

Media & Vision

​title: ‘Image Generation’ description: “Blend Animus’ automatic media workflows with direct API control for character renders, photo edits, and more.” icon: ‘image’

​How It Works

​Two-Phase Process

​Basic Usage

​Enable Image Generation

​Listen for Image Generation Events

​Direct Image Generation

​Manual Image Generation

​Custom Image Prompts

​Integration with Other Features

​Image Generation with AutoTurn

​Image Generation Coordination

​Advanced Image Generation

​Custom Image Generation Settings

​Handling Image Generation Responses

​Image History Management

​Image Format in Chat History

​Managing Image Context

​Error Handling

​Comprehensive Error Handling

​Graceful Degradation

​UI Implementation

​Complete Image Generation UI

​CSS for Image Generation UI

​Best Practices

​Prompt Engineering for Images

​Performance Optimization

​Content Guidelines

​Next Steps

Event System

Auto-Turn Conversations

Tool Calling

Media & Vision

title: ‘Image Generation’ description: “Blend Animus’ automatic media workflows with direct API control for character renders, photo edits, and more.” icon: ‘image’

How It Works

Two-Phase Process

Basic Usage

Enable Image Generation

Listen for Image Generation Events

Direct Image Generation

Manual Image Generation

Custom Image Prompts

Integration with Other Features

Image Generation with AutoTurn

Image Generation Coordination

Advanced Image Generation

Custom Image Generation Settings

Handling Image Generation Responses

Image History Management

Image Format in Chat History

Managing Image Context

Error Handling

Comprehensive Error Handling

Graceful Degradation

UI Implementation

Complete Image Generation UI

CSS for Image Generation UI

Best Practices

Prompt Engineering for Images

Performance Optimization

Content Guidelines

Next Steps