marketing-pipeline-share-ai-content-automation

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Marketing Pipeline Share - AI Content Automation

营销流水线分享 - AI内容自动化

Skill by ara.so — Marketing Skills collection.

This project is an end-to-end AI-powered content automation system that handles research (web scraping), content generation (Claude/OpenAI), and video rendering (Remotion). It automates content creation from keyword input to publishable video output.

由 ara.so 提供的技能 — 营销技能合集。

本项目是一个端到端的AI驱动内容自动化系统，可处理调研（网页抓取）、内容生成（Claude/OpenAI）和视频渲染（Remotion）。它能实现从关键词输入到可发布视频输出的全流程内容创作自动化。

What It Does

功能介绍

Auto-Research: Scrapes TechCrunch, a16z, Twitter/X, LinkedIn for recent news (24h window)
AI Content Generation: Creates diverse formats (toplist, POV, case study, how-to) using Claude 3 or OpenAI
Multi-language Support: Generates content in both English and Vietnamese
Video Rendering: Automatically creates infographics and short-form videos using Remotion
Multi-platform Export: Optimizes videos for Reels, TikTok, Shorts

自动调研：抓取TechCrunch、a16z、Twitter/X、LinkedIn上的近期新闻（24小时窗口）
AI内容生成：使用Claude 3或OpenAI创建多种格式内容（排行榜、观点文、案例研究、教程）
多语言支持：生成英文和越南语内容
视频渲染：使用Remotion自动创建信息图和短视频
多平台导出：针对Reels、TikTok、Shorts优化视频

Installation

安装

bash

undefined

bash

undefined

Clone the repository

git clone https://github.com/pennydinh/marketing-pineline-share.git cd marketing-pineline-share

Install dependencies

npm install

or

yarn install

or

pnpm install

undefined

pnpm install

undefined

Environment Configuration

环境配置

Create a

.env.local

file in the root directory:

env

undefined

在根目录创建

.env.local

文件：

env

undefined

AI Provider Keys

ANTHROPIC_API_KEY=your_claude_api_key OPENAI_API_KEY=your_openai_api_key

Scraping/Research APIs (RapidAPI)

RAPIDAPI_KEY=your_rapidapi_key

Optional: Custom API endpoints

NEXT_PUBLIC_API_URL=http://localhost:3000

Remotion Configuration

REMOTION_AWS_ACCESS_KEY_ID=your_aws_key REMOTION_AWS_SECRET_ACCESS_KEY=your_aws_secret

undefined

REMOTION_AWS_ACCESS_KEY_ID=your_aws_key REMOTION_AWS_SECRET_ACCESS_KEY=your_aws_secret

undefined

Project Structure

项目结构

marketing-pineline-share/
├── src/
│   ├── app/              # Next.js app directory
│   ├── components/       # React components
│   ├── lib/
│   │   ├── ai/          # AI provider integrations
│   │   ├── scraper/     # Web scraping modules
│   │   └── video/       # Remotion video rendering
│   ├── types/           # TypeScript definitions
│   └── utils/           # Utility functions
├── remotion/            # Remotion video templates
└── public/              # Static assets

marketing-pineline-share/
├── src/
│   ├── app/              # Next.js应用目录
│   ├── components/       # React组件
│   ├── lib/
│   │   ├── ai/          # AI提供商集成模块
│   │   ├── scraper/     # 网页抓取模块
│   │   └── video/       # Remotion视频渲染模块
│   ├── types/           # TypeScript类型定义
│   └── utils/           # 工具函数
├── remotion/            # Remotion视频模板
└── public/              # 静态资源

Core Usage Patterns

核心使用模式

1. Content Research & Scraping

1. 内容调研与抓取

typescript

import { scrapeNews } from '@/lib/scraper/news-scraper';
import { analyzeTrends } from '@/lib/scraper/trend-analyzer';

// Scrape recent news from multiple sources
async function researchTopic(keyword: string) {
  const sources = ['techcrunch', 'a16z', 'twitter', 'linkedin'];
  
  const newsData = await scrapeNews({
    keyword,
    sources,
    timeframe: '24h',
    limit: 50
  });

  // Analyze and extract insights
  const insights = await analyzeTrends(newsData);
  
  return {
    articles: newsData,
    insights,
    keywords: insights.topKeywords,
    stats: insights.statistics
  };
}

typescript

import { scrapeNews } from '@/lib/scraper/news-scraper';
import { analyzeTrends } from '@/lib/scraper/trend-analyzer';

// 从多个来源抓取近期新闻
async function researchTopic(keyword: string) {
  const sources = ['techcrunch', 'a16z', 'twitter', 'linkedin'];
  
  const newsData = await scrapeNews({
    keyword,
    sources,
    timeframe: '24h',
    limit: 50
  });

  // 分析并提取洞察
  const insights = await analyzeTrends(newsData);
  
  return {
    articles: newsData,
    insights,
    keywords: insights.topKeywords,
    stats: insights.statistics
  };
}

2. AI Content Generation with Claude

2. 基于Claude的AI内容生成

typescript

import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});

async function generateContent(
  topic: string,
  format: 'toplist' | 'pov' | 'case-study' | 'how-to',
  language: 'en' | 'vi',
  researchData: any
) {
  const systemPrompt = `You are an expert content creator. 
  Create a ${format} article about ${topic} in ${language}.
  Use the following research data: ${JSON.stringify(researchData)}`;

  const message = await anthropic.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 4096,
    temperature: 0.7,
    system: systemPrompt,
    messages: [
      {
        role: 'user',
        content: `Create an engaging ${format} article about ${topic}. 
        Include data-backed insights and recent trends.`
      }
    ]
  });

  return message.content[0].text;
}

typescript

import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});

async function generateContent(
  topic: string,
  format: 'toplist' | 'pov' | 'case-study' | 'how-to',
  language: 'en' | 'vi',
  researchData: any
) {
  const systemPrompt = `You are an expert content creator. 
  Create a ${format} article about ${topic} in ${language}.
  Use the following research data: ${JSON.stringify(researchData)}`;

  const message = await anthropic.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 4096,
    temperature: 0.7,
    system: systemPrompt,
    messages: [
      {
        role: 'user',
        content: `Create an engaging ${format} article about ${topic}. 
        Include data-backed insights and recent trends.`
      }
    ]
  });

  return message.content[0].text;
}

3. Alternative: OpenAI Integration

3. 替代方案：OpenAI集成

typescript

import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

async function generateWithOpenAI(
  topic: string,
  context: string
) {
  const completion = await openai.chat.completions.create({
    model: 'gpt-4-turbo-preview',
    messages: [
      {
        role: 'system',
        content: 'You are a marketing content expert.'
      },
      {
        role: 'user',
        content: `Create content about ${topic}. Context: ${context}`
      }
    ],
    temperature: 0.8,
    max_tokens: 3000
  });

  return completion.choices[0].message.content;
}

typescript

import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

async function generateWithOpenAI(
  topic: string,
  context: string
) {
  const completion = await openai.chat.completions.create({
    model: 'gpt-4-turbo-preview',
    messages: [
      {
        role: 'system',
        content: 'You are a marketing content expert.'
      },
      {
        role: 'user',
        content: `Create content about ${topic}. Context: ${context}`
      }
    ],
    temperature: 0.8,
    max_tokens: 3000
  });

  return completion.choices[0].message.content;
}

4. Video Generation with Remotion

4. 基于Remotion的视频生成

typescript

import { bundle } from '@remotion/bundler';
import { renderMedia, selectComposition } from '@remotion/renderer';
import { webpackOverride } from './remotion/webpack-override';

async function renderContentVideo(
  content: string,
  title: string,
  format: 'reels' | 'tiktok' | 'shorts'
) {
  // Video dimensions based on format
  const dimensions = {
    reels: { width: 1080, height: 1920 },
    tiktok: { width: 1080, height: 1920 },
    shorts: { width: 1080, height: 1920 }
  };

  const bundleLocation = await bundle({
    entryPoint: './remotion/index.ts',
    webpackOverride,
  });

  const composition = await selectComposition({
    serveUrl: bundleLocation,
    id: 'ContentVideo',
    inputProps: {
      title,
      content,
      ...dimensions[format]
    },
  });

  await renderMedia({
    composition,
    serveUrl: bundleLocation,
    codec: 'h264',
    outputLocation: `out/${title}-${format}.mp4`,
    inputProps: {
      title,
      content,
    },
  });

  return `out/${title}-${format}.mp4`;
}

typescript

import { bundle } from '@remotion/bundler';
import { renderMedia, selectComposition } from '@remotion/renderer';
import { webpackOverride } from './remotion/webpack-override';

async function renderContentVideo(
  content: string,
  title: string,
  format: 'reels' | 'tiktok' | 'shorts'
) {
  // 根据格式设置视频尺寸
  const dimensions = {
    reels: { width: 1080, height: 1920 },
    tiktok: { width: 1080, height: 1920 },
    shorts: { width: 1080, height: 1920 }
  };

  const bundleLocation = await bundle({
    entryPoint: './remotion/index.ts',
    webpackOverride,
  });

  const composition = await selectComposition({
    serveUrl: bundleLocation,
    id: 'ContentVideo',
    inputProps: {
      title,
      content,
      ...dimensions[format]
    },
  });

  await renderMedia({
    composition,
    serveUrl: bundleLocation,
    codec: 'h264',
    outputLocation: `out/${title}-${format}.mp4`,
    inputProps: {
      title,
      content,
    },
  });

  return `out/${title}-${format}.mp4`;
}

5. Complete Pipeline Example

5. 完整流水线示例

typescript

import { scrapeNews } from '@/lib/scraper/news-scraper';
import { generateContent } from '@/lib/ai/claude-generator';
import { renderContentVideo } from '@/lib/video/remotion-renderer';

async function runContentPipeline(
  keyword: string,
  config: {
    format: 'toplist' | 'pov' | 'case-study' | 'how-to';
    languages: ('en' | 'vi')[];
    videoFormats: ('reels' | 'tiktok' | 'shorts')[];
  }
) {
  // Step 1: Research
  console.log('🔍 Researching topic...');
  const research = await scrapeNews({
    keyword,
    sources: ['techcrunch', 'a16z'],
    timeframe: '24h'
  });

  // Step 2: Generate content for each language
  const contents = await Promise.all(
    config.languages.map(async (lang) => {
      console.log(`✍️ Generating ${lang} content...`);
      const content = await generateContent(
        keyword,
        config.format,
        lang,
        research
      );
      return { lang, content };
    })
  );

  // Step 3: Render videos
  const videos = await Promise.all(
    contents.flatMap(({ lang, content }) =>
      config.videoFormats.map(async (format) => {
        console.log(`🎬 Rendering ${format} video (${lang})...`);
        const videoPath = await renderContentVideo(
          content,
          `${keyword}-${lang}`,
          format
        );
        return { lang, format, videoPath };
      })
    )
  );

  return {
    research,
    contents,
    videos
  };
}

// Usage
const result = await runContentPipeline('AI Marketing Trends', {
  format: 'toplist',
  languages: ['en', 'vi'],
  videoFormats: ['reels', 'tiktok']
});

typescript

import { scrapeNews } from '@/lib/scraper/news-scraper';
import { generateContent } from '@/lib/ai/claude-generator';
import { renderContentVideo } from '@/lib/video/remotion-renderer';

async function runContentPipeline(
  keyword: string,
  config: {
    format: 'toplist' | 'pov' | 'case-study' | 'how-to';
    languages: ('en' | 'vi')[];
    videoFormats: ('reels' | 'tiktok' | 'shorts')[];
  }
) {
  // 步骤1：调研
  console.log('🔍 正在调研主题...');
  const research = await scrapeNews({
    keyword,
    sources: ['techcrunch', 'a16z'],
    timeframe: '24h'
  });

  // 步骤2：为每种语言生成内容
  const contents = await Promise.all(
    config.languages.map(async (lang) => {
      console.log(`✍️ 正在生成${lang}语言内容...`);
      const content = await generateContent(
        keyword,
        config.format,
        lang,
        research
      );
      return { lang, content };
    })
  );

  // 步骤3：渲染视频
  const videos = await Promise.all(
    contents.flatMap(({ lang, content }) =>
      config.videoFormats.map(async (format) => {
        console.log(`🎬 正在渲染${format}格式视频（${lang}语言）...`);
        const videoPath = await renderContentVideo(
          content,
          `${keyword}-${lang}`,
          format
        );
        return { lang, format, videoPath };
      })
    )
  );

  return {
    research,
    contents,
    videos
  };
}

// 使用示例
const result = await runContentPipeline('AI Marketing Trends', {
  format: 'toplist',
  languages: ['en', 'vi'],
  videoFormats: ['reels', 'tiktok']
});

API Routes (Next.js)

API路由（Next.js）

Research Endpoint

调研接口

typescript

// app/api/research/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { scrapeNews } from '@/lib/scraper/news-scraper';

export async function POST(request: NextRequest) {
  const { keyword, sources, timeframe } = await request.json();

  try {
    const data = await scrapeNews({ keyword, sources, timeframe });
    return NextResponse.json({ success: true, data });
  } catch (error) {
    return NextResponse.json(
      { success: false, error: error.message },
      { status: 500 }
    );
  }
}

typescript

// app/api/research/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { scrapeNews } from '@/lib/scraper/news-scraper';

export async function POST(request: NextRequest) {
  const { keyword, sources, timeframe } = await request.json();

  try {
    const data = await scrapeNews({ keyword, sources, timeframe });
    return NextResponse.json({ success: true, data });
  } catch (error) {
    return NextResponse.json(
      { success: false, error: error.message },
      { status: 500 }
    );
  }
}

Content Generation Endpoint

内容生成接口

typescript

// app/api/generate/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { generateContent } from '@/lib/ai/claude-generator';

export async function POST(request: NextRequest) {
  const { topic, format, language, research } = await request.json();

  try {
    const content = await generateContent(topic, format, language, research);
    return NextResponse.json({ success: true, content });
  } catch (error) {
    return NextResponse.json(
      { success: false, error: error.message },
      { status: 500 }
    );
  }
}

typescript

// app/api/generate/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { generateContent } from '@/lib/ai/claude-generator';

export async function POST(request: NextRequest) {
  const { topic, format, language, research } = await request.json();

  try {
    const content = await generateContent(topic, format, language, research);
    return NextResponse.json({ success: true, content });
  } catch (error) {
    return NextResponse.json(
      { success: false, error: error.message },
      { status: 500 }
    );
  }
}

Development Commands

开发命令

bash

undefined

bash

undefined

Start development server

启动开发服务器

npm run dev

Build for production

构建生产版本

npm run build

Start production server

启动生产服务器

npm start

Render Remotion video (development)

渲染Remotion视频（开发模式）

npm run remotion:dev

Render Remotion video (production)

渲染Remotion视频（生产模式）

npm run remotion:render -- --props='{"title":"My Video"}'

Type checking

类型检查

npm run type-check

Linting

代码检查

npm run lint

undefined

npm run lint

undefined

Remotion Video Templates

Remotion视频模板

Create a custom video template:

typescript

// remotion/compositions/ContentVideo.tsx
import { AbsoluteFill, useCurrentFrame, useVideoConfig } from 'remotion';

export const ContentVideo: React.FC<{
  title: string;
  content: string;
}> = ({ title, content }) => {
  const frame = useCurrentFrame();
  const { fps, width, height } = useVideoConfig();

  const opacity = Math.min(1, frame / 30);

  return (
    <AbsoluteFill
      style={{
        backgroundColor: '#000',
        justifyContent: 'center',
        alignItems: 'center',
      }}
    >
      <div style={{ opacity }}>
        <h1 style={{ color: 'white', fontSize: 80 }}>{title}</h1>
        <p style={{ color: 'white', fontSize: 40, maxWidth: width * 0.8 }}>
          {content}
        </p>
      </div>
    </AbsoluteFill>
  );
};

typescript

// remotion/index.ts
import { registerRoot } from 'remotion';
import { ContentVideo } from './compositions/ContentVideo';

registerRoot(() => {
  return (
    <>
      <Composition
        id="ContentVideo"
        component={ContentVideo}
        durationInFrames={300}
        fps={30}
        width={1080}
        height={1920}
        defaultProps={{
          title: 'Default Title',
          content: 'Default content',
        }}
      />
    </>
  );
});

创建自定义视频模板：

typescript

// remotion/compositions/ContentVideo.tsx
import { AbsoluteFill, useCurrentFrame, useVideoConfig } from 'remotion';

export const ContentVideo: React.FC<{
  title: string;
  content: string;
}> = ({ title, content }) => {
  const frame = useCurrentFrame();
  const { fps, width, height } = useVideoConfig();

  const opacity = Math.min(1, frame / 30);

  return (
    <AbsoluteFill
      style={{
        backgroundColor: '#000',
        justifyContent: 'center',
        alignItems: 'center',
      }}
    >
      <div style={{ opacity }}>
        <h1 style={{ color: 'white', fontSize: 80 }}>{title}</h1>
        <p style={{ color: 'white', fontSize: 40, maxWidth: width * 0.8 }}>
          {content}
        </p>
      </div>
    </AbsoluteFill>
  );
};

注册合成组件：

typescript

// remotion/index.ts
import { registerRoot } from 'remotion';
import { ContentVideo } from './compositions/ContentVideo';

registerRoot(() => {
  return (
    <>
      <Composition
        id="ContentVideo"
        component={ContentVideo}
        durationInFrames={300}
        fps={30}
        width={1080}
        height={1920}
        defaultProps={{
          title: 'Default Title',
          content: 'Default content',
        }}
      />
    </>
  );
});

Common Patterns

通用模式

Rate Limiting & Error Handling

速率限制与错误处理

typescript

import { RateLimiter } from '@/lib/utils/rate-limiter';

const limiter = new RateLimiter({
  maxRequests: 50,
  perMinutes: 1
});

async function safeAPICall<T>(
  fn: () => Promise<T>,
  retries = 3
): Promise<T> {
  await limiter.waitForSlot();

  for (let i = 0; i < retries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (i === retries - 1) throw error;
      await new Promise(resolve => setTimeout(resolve, 1000 * (i + 1)));
    }
  }
  
  throw new Error('Max retries exceeded');
}

typescript

import { RateLimiter } from '@/lib/utils/rate-limiter';

const limiter = new RateLimiter({
  maxRequests: 50,
  perMinutes: 1
});

async function safeAPICall<T>(
  fn: () => Promise<T>,
  retries = 3
): Promise<T> {
  await limiter.waitForSlot();

  for (let i = 0; i < retries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (i === retries - 1) throw error;
      await new Promise(resolve => setTimeout(resolve, 1000 * (i + 1)));
    }
  }
  
  throw new Error('Max retries exceeded');
}

Content Quality Validation

内容质量验证

typescript

interface ContentQuality {
  wordCount: number;
  readabilityScore: number;
  hasData: boolean;
  hasHeadings: boolean;
}

function validateContent(content: string): ContentQuality {
  const wordCount = content.split(/\s+/).length;
  const hasData = /\d+%|\$\d+|statistics|data shows/i.test(content);
  const hasHeadings = /#+ /m.test(content);

  return {
    wordCount,
    readabilityScore: calculateReadability(content),
    hasData,
    hasHeadings,
  };
}

typescript

interface ContentQuality {
  wordCount: number;
  readabilityScore: number;
  hasData: boolean;
  hasHeadings: boolean;
}

function validateContent(content: string): ContentQuality {
  const wordCount = content.split(/\s+/).length;
  const hasData = /\d+%|\$\d+|statistics|data shows/i.test(content);
  const hasHeadings = /#+ /m.test(content);

  return {
    wordCount,
    readabilityScore: calculateReadability(content),
    hasData,
    hasHeadings,
  };
}

Troubleshooting

故障排除

API Rate Limits

API速率限制

If you encounter rate limit errors:

typescript

// Implement exponential backoff
const delay = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));

async function retryWithBackoff<T>(
  fn: () => Promise<T>,
  maxRetries = 5
): Promise<T> {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (error.status === 429 && i < maxRetries - 1) {
        const waitTime = Math.pow(2, i) * 1000;
        console.log(`Rate limited. Waiting ${waitTime}ms...`);
        await delay(waitTime);
      } else {
        throw error;
      }
    }
  }
  throw new Error('Max retries exceeded');
}

如果遇到速率限制错误：

typescript

// 实现指数退避
const delay = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));

async function retryWithBackoff<T>(
  fn: () => Promise<T>,
  maxRetries = 5
): Promise<T> {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (error.status === 429 && i < maxRetries - 1) {
        const waitTime = Math.pow(2, i) * 1000;
        console.log(`速率受限，等待${waitTime}ms...`);
        await delay(waitTime);
      } else {
        throw error;
      }
    }
  }
  throw new Error('Max retries exceeded');
}

Scraping Failures

抓取失败

Check for blocked requests or invalid selectors:

typescript

// Add user agent and retry logic
import axios from 'axios';

const scrapeWithHeaders = async (url: string) => {
  return axios.get(url, {
    headers: {
      'User-Agent': 'Mozilla/5.0 (compatible; ContentBot/1.0)',
      'Accept': 'text/html,application/xhtml+xml',
    },
    timeout: 10000,
  });
};

检查请求是否被拦截或选择器是否无效：

typescript

// 添加用户代理和重试逻辑
import axios from 'axios';

const scrapeWithHeaders = async (url: string) => {
  return axios.get(url, {
    headers: {
      'User-Agent': 'Mozilla/5.0 (compatible; ContentBot/1.0)',
      'Accept': 'text/html,application/xhtml+xml',
    },
    timeout: 10000,
  });
};

Video Rendering Memory Issues

视频渲染内存问题

For large video renders:

typescript

// Reduce quality or split into chunks
await renderMedia({
  composition,
  serveUrl: bundleLocation,
  codec: 'h264',
  quality: 80, // Reduce from default 100
  concurrency: 1, // Limit concurrent frames
  outputLocation: outputPath,
});

针对大型视频渲染：

typescript

// 降低质量或拆分片段
await renderMedia({
  composition,
  serveUrl: bundleLocation,
  codec: 'h264',
  quality: 80, // 从默认100降低
  concurrency: 1, // 限制并发帧数
  outputLocation: outputPath,
});

TypeScript Configuration

TypeScript配置

Ensure proper types for AI responses:

typescript

// types/content.ts
export interface GeneratedContent {
  title: string;
  body: string;
  metadata: {
    format: string;
    language: string;
    wordCount: number;
  };
  sources: string[];
}

// Use with AI generation
const content = await generateContent(...) as GeneratedContent;

确保AI响应有正确的类型：

typescript

// types/content.ts
export interface GeneratedContent {
  title: string;
  body: string;
  metadata: {
    format: string;
    language: string;
    wordCount: number;
  };
  sources: string[];
}

// 在AI生成中使用
const content = await generateContent(...) as GeneratedContent;