Social Picker API

Extract media from various posting platforms like Twitter, Reddit, Pixiv, Youtube, Tiktok, Osnova and many others. This project written in Typescript for Node.js and it works as REST service for other services, e.g. Anime Ultra Bot or Social Picker Vue.

Usage

Start a server, make HTTP-request to it with a link to the post and receive that post's content back.

Configs

There are some configuration files:

• service.json – service port, base URL of external service for viewing some and proxy for some platforms – see Platform specific/Proxies
• tokens.json – tokens for platforms and other filepaths. See Platform specific and types with comments
• pm2.production.json – config for pm2 daemon
• Other configuration files for particular platforms – see Platform specific

Development config files can be created and placed along production ones (e.g. tokens.dev.json or service.dev.json). You can also install all npm modules (including dev) one with npm install. npm run dev will run service in dev-environment, npm run lint will use eslint.

How to run

1. Install necessary dependencies – npm i --omit=dev
2. Install necessary binaries needed for some platforms
3. Compile Typescript – npm run compile
4. Start server – npm run production

After launching you can access Picker with fetching it like curl http://localhost:8080/?url=__LINK_TO_ANY_POST__ (change 8080 to real port you specified in service.json).

Open access

If you're planning opening your Picker instance to the world, I'd suggest creating your own middleware service for handling user access, downloading combined videos, etc. (something like that). After creating such service set CUSTOM_IMG_VIEWER_SERVICE at service.json in this app pointing to that service (you can delete proxy search param from queries – see Platform specific/Proxies for more).

List of supported platforms

• Twitter – images, videos, GIFs and direct media from *.twimg.com. More details
• Pixiv – images, single direct media from *.pximg.net and Ugoira-GIFs. More details
• Reddit – images, videos, GIFs and galleries. More details
• Youtube – video with response in default type containing all streams (via yt-dlp)
• Tiktok – Any video in multiple formats. More details
• Instagram – images, videos, galleries and Reels. More details
• Osnova – images, videos, GIFs and galleries. More details
• Coub – looped videos with linear audio. More details
• Joyreactor – images and GIFs from multiple subdomains and direct links to media files. More details
• Tumblr – images and galleries. More details
• Kemono – images and galleries. More details
• Danbooru – images (unstable)
• Gelbooru – images (unstable)
• Konachan – images (unstable)
• Yandere – images (unstable)
• Eshuushuu – images (unstable)
• Sankaku – images (unstable)
• Zerochan – images (unstable)
• AnimePictures – images (unstable)

Platform specific

Binaries

Some platform modules utilizes binary executable, here is the list:

• Social Picker Twitter Scrapper – read more about Twitter specific
• ffmpeg – for media stream merging and convering. Follow instructions on the official site. After installing make sure you added its location to $PATH
• yt-dlp – for picking some video-platforms. Download built release for your platform, make it executable and ensure its location is in the $PATH

Proxies

Some platforms have very harsh limitation of allowed IP addresses/AS. Therefore service.json contains two fields for setting up host and port of SOCKS5 proxy to some remote server (PROXY_HOSTNAME and PROXY_PORT); you can set them to null if accessing such platforms from your IP address is not being limited. For the same reason base URL of external service hash template for proxy flag in its search query (see Open access).

Twitter

Supports images, videos, GIFs (temporary unstable) and direct media from *.twimg.com.

This project uses Social Picker Twitter Scrapper – wrapper for this Go lib – to fetch tweets without API. Thus if you want to get Tweets, before running this app you should consider reading that project README and placing its compiled/created executable and config-files into these folders:

1. Move executable file Social-Picker-Twitter-Scrapper into bin folder
2. Move generated cookies.json into config, choose a explicit name for it, e.g. twitter-scrapper-cookies.json
   • For convenience you can move credentials.json next to cookies.json (this app and executable binary use only cookies file).
3. Check and edit tokens.json – give a look to properties TWITTER_SCAPPER.binary_file_path and TWITTER_SCAPPER.cookies_file_path: paths are relative to project's root, default ones are in accordance to this example.

Pixiv

Supports images, single direct media from *.pximg.net and Ugoira-GIFs.

Uses external service for end-user viewing of high-res images due to Referer Header issues. Uses ugoira-builder for creating mp4 video from Ugoira ZIP with PNGs (via ffmpeg).

Reddit

Supports images, videos, GIFs and galleries. Because Reddit's videos are divided from audios, this app utilizes video-audio-merge for merging separated streams via ffmpeg.

Requires setup of the site's cookies in tokens.json

Tiktok

Any video in multiple formats. Does not support image galleries.

Uses yt-dlp for extracting metadata and then video-codec-convert to convert original highest quality HEVC video without watermark to H264 file (via ffmpeg) for compatibility. Still prioritizes HEVC one.

Instagram

Supports images, videos, galleries and Reels.

To pick default regular post (square photos, videos and galleries) set property INSTAGRAM_COOKIE_ONE_LINE_FOR_POSTS in tokens.json. Place there contents of cookies from your web-browser's DevTools (Press F12 —> Console tab —> type document.cookie —> press Enter).

Question: Are those cookies enough?
Answer: Works well for me! If they're not sufficient anymore, copy from DevTools tabs Network or Application

To pick Reels:

1. Install this browser extension – it's open source and it supports Chrome and Firefox
2. Go to instagram.com
3. Copy cookies (click Copy button in extension popup)
4. Create file config/instagram.cookies.txt and paste there contents from clipboard (it starts with "# Netscape HTTP Cookie File")
5. Check and edit tokens.json – give a look to property INSTAGRAM_COOKIE_FILE_LOCATION_FOR_REELS: set path relative to project's root, default one is in accordance to this example.

Osnova

Supports images, videos, GIFs and galleries. Also extracts Twitter and Instagram blocks/links from within and handles them with parsers above.

Coub

Creates looped videos with linear audio, limits itself only with audio length and filesize if it ever exceed a reasonable size (20 MB). Uses video-audio-merge for merging separated streams (via ffmpeg).

Joyreactor

Supports images, videos, GIFs and galleries, requires setup of the site's cookies in tokens.json.

Tumblr

Supports images and galleries, set OAuth keys and stuff in tokens.json.

Kemono

Supports images and galleries, requires setup of the site's cookies in tokens.json.

Read more

• Social Picker Twitter Scrapper – wrapper for twitter-scraper lib in Go
• Telegram bot based on this service
• Frontend client built for this service
• ffmpeg docs
• yt-dlp docs
• Tumblr API
• Osnova API
• What is Pixiv's Ugoira

BSL-1.0 License