MediaCrawler
An open-source Python crawler for seven major Chinese social platforms — Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Tieba and Zhihu — driven by a real Playwright browser so no signature reverse-engineering is needed.
Try it now
GPT-4oWhat is MediaCrawler?
MediaCrawler (by NanmiCoder, ~60k GitHub stars) is a widely used self-media data-collection framework for the Chinese internet. It covers seven platforms — Xiaohongshu (RED), Douyin, Kuaishou, Bilibili, Weibo, Baidu Tieba and Zhihu — and can gather posts, videos, notes, comments (including second-level replies), creator-homepage content and search results. Instead of manually reverse-engineering each platform's encrypted signatures, it uses Playwright to drive a real browser, keeps the logged-in context (QR-code or cookie login, cached for reuse) and obtains encryption parameters by evaluating JavaScript, which greatly lowers the barrier. Across platforms it supports keyword search, crawl-by-post-ID, creator-homepage crawl, login-state caching, an IP proxy pool and comment word-cloud generation, with results saved to CSV/JSON or MySQL/PostgreSQL/SQLite. Setup uses Python 3.11 (uv recommended), Node.js 16+ for some platforms and Playwright browsers. The project is open for learning and research on public data only; commercial use requires authorization, and users must respect each platform's terms, data-protection law and account-safety rules.
Pros
- One unified codebase across seven major Chinese platforms.
- Playwright approach avoids hard signature reverse work.
- Rich modes: search, ID, creator and nested comments.
- Flexible storage from files to relational databases.
- Large, active community and extensive tutorials.
Cons
- Requires a Python 3.11 + Node.js + Playwright environment.
- You must log in, and aggressive crawling risks account restrictions.
- Public data and platform terms must be respected; legal risk sits with the user.
- Open for research; commercial use needs separate authorization.
Detailed Ratings
Capabilities
Compare with Alternatives
| Feature | MediaCrawler | Semantic Kernel | Warp | Vercel AI SDK |
|---|---|---|---|---|
| Rating | 4.4 | 4.8 | 4.8 | 4.8 |
| Pricing | Free | Free | Free | Free |
| Starting From | Free | Free | Free | Free |
| API | — | — | — | — |
| Free Tier | — | — | — | — |
| Mobile App | — | — | — | — |
Who is it for
Competitive monitoring
Track competitors' notes, videos and engagement across platforms.
Sentiment & PR
Collect public comments and build word clouds for opinion analysis.
Content planning
Mine trending topics and high-performing posts for editorial ideas.
Academic research
Gather public social datasets for communication or market studies.
Engineering learning
Study browser automation and how anti-bot signatures work.