public-roadmap icon indicating copy to clipboard operation
public-roadmap copied to clipboard
verified publisher icon serpapi

[New Bing Search API] Bing Image search

Open andypple83 opened this issue 3 years ago • 6 comments

Let's support Bing image search. We can search by keyword:

Screen Shot 2022-10-15 at 14 18 53

Or image file/url:

Screen Shot 2022-10-15 at 14 21 17

andypple83 avatar Oct 15 '22 07:10 andypple83

Adding old Canny.io link for reference: https://forum.serpapi.com/feature-requests/p/add-bing-image-search-api

hartator avatar Nov 03 '22 21:11 hartator

Reverse image search results are obtained through the /images/api/custom/knowledge API endpoint.

Related content tab

Bash script

curl 'https://www.bing.com/images/api/custom/knowledge?q=&rshighlight=true&textDecorations=true&internalFeatures=share&FORM=SBIHMP&skey=aeCUosWoYCPQ7_1wAr_HO_O746BHv_fxH8UAUBIvp1k&safeSearch=Moderate&mkt=en-ww&setLang=en-us&IG=0EE09CCA53BE4ACAA95BD9B092C45CF3&IID=idpins&SFX=1' \
  -s \
  -H 'authority: www.bing.com' \
  -H 'accept: */*' \
  -H 'accept-language: en-US,en;q=0.8' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundaryybS6WM5BOLApSZSk' \
  -H 'cookie: SUID=M; MUID=1FF2510B90A86CE407E0438891A86D52; MUIDB=1FF2510B90A86CE407E0438891A86D52; _EDGE_S=F=1&SID=3701616DA2BD6F32240D73EEA3BD6E8E; _EDGE_V=1; SRCHD=AF=SBIHMP; SRCHUID=V=2&GUID=963F1DC6DA524E3C8E2B419534E8DD4F&dmnchg=1; SRCHUSR=DOB=20221223; _SS=SID=3701616DA2BD6F32240D73EEA3BD6E8E; MMCASM=ID=FF993A37A0AC4FA4AD7DAD03BC6E8D52; SRCHHPGUSR=SRCHLANG=en&BRW=HTP&BRH=M&CW=976&CH=730&SCW=976&SCH=730&DPR=1.3&UTC=120&DM=0' \
  -H 'origin: https://www.bing.com' \
  -H 'pragma: no-cache' \
  -H 'referer: https://www.bing.com/images/search?view=detailV2&insightstoken=bcid_RLKVsIV2BwkFXg*ccid_spWwhXYH&form=SBIHMP&iss=SBIUPLOADGET&sbisrc=ImgPicker&idpbck=1&sbifsz=927+x+524+%c2%b7+25.15+kB+%c2%b7+png&sbifnm=serpapi-serpbear.png&thw=927&thh=524&ptime=223&dlen=34344&expw=798&exph=451&selectedindex=0&id=-1051855017&ccid=spWwhXYH&vt=2&sim=11' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sec-gpc: 1' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36' \
  --data-raw $'------WebKitFormBoundaryybS6WM5BOLApSZSk\r\nContent-Disposition: form-data; name="knowledgeRequest"\r\n\r\n{"imageInfo":{"imageInsightsToken":"bcid_RLKVsIV2BwkFXg*ccid_spWwhXYH","source":"Gallery"},"knowledgeRequest":{"invokedSkills":["ImageById","BestRepresentativeQuery","Offline","ObjectDetection","OCR","EntityLinkingFace","EntityLinkingDog","EntityLinkingAnimal","EntityLinkingPlant","EntityLinkingLandmark","EntityLinkingFood","EntityLinkingBook","SimilarImages","RelatedSearches","PagesIncluding","TextAds","ProductAds","SponsoredAds","Annotation","Recipes","Travel"],"invokedSkillsRequestData":{"adsRequest":{"textRequest":{"mainlineAdsMaxCount":2}}},"index":1}}\r\n------WebKitFormBoundaryybS6WM5BOLApSZSk--\r\n' \
  --compressed | jq -cr '.tags[] | select(.displayName == "") | .actions[] | select(.actionType == "VisualSearch") | .data.value[] | { name, cDNContentUrl }'

Not all HTTP headers are necessary, for sure.

Output

{"name":"What is Lead Management | Lead Management Process | Freshsales","cDNContentUrl":"https://th.bing.com/th/id/R.b9f567b5396fa177a4f4dc31be4ee7df?rik=QfNkfgDMOLdGug&pid=ImgRaw&r=0"}
{"name":"Sandglaz","cDNContentUrl":"https://th.bing.com/th/id/R.0b9cd275326b080d90611c41eed0ba53?rik=ZNQETsfqL%2bnJ%2bA&pid=ImgRaw&r=0"}
{"name":"How to Use Live Chat Auto Started Chats | onWebChat","cDNContentUrl":"https://th.bing.com/th/id/R.9c0f994e00e6a557d2bff1766f395960?rik=tYN%2f%2fgBqV8F1NA&pid=ImgRaw&r=0"}

# Omitted ...

Screenshot

image

MIT-licensed open-source parser

https://sourcegraph.com/github.com/d4n3436/Fergun@88a5b915f7b09c61a52655d605a77f9139088841/-/blob/src/Apis/Bing/BingVisualSearch.cs?L77-89

Text tab

Response schema on Azure documentation: https://github.com/MicrosoftDocs/azure-docs/blob/65798f88a769256202438ed9f956d5ecd48c918a/articles/cognitive-services/bing-visual-search/concepts/sending-queries.md#text-recognition

Bash script

curl 'https://www.bing.com/images/api/custom/knowledge?q=&rshighlight=true&textDecorations=true&internalFeatures=share&FORM=SBIHMP&skey=aeCUosWoYCPQ7_1wAr_HO_O746BHv_fxH8UAUBIvp1k&safeSearch=Moderate&mkt=en-ww&setLang=en-us&IG=0EE09CCA53BE4ACAA95BD9B092C45CF3&IID=idpins&SFX=1' \
  -s \
  -H 'authority: www.bing.com' \
  -H 'accept: */*' \
  -H 'accept-language: en-US,en;q=0.8' \
  -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundaryybS6WM5BOLApSZSk' \
  -H 'cookie: SUID=M; MUID=1FF2510B90A86CE407E0438891A86D52; MUIDB=1FF2510B90A86CE407E0438891A86D52; _EDGE_S=F=1&SID=3701616DA2BD6F32240D73EEA3BD6E8E; _EDGE_V=1; SRCHD=AF=SBIHMP; SRCHUID=V=2&GUID=963F1DC6DA524E3C8E2B419534E8DD4F&dmnchg=1; SRCHUSR=DOB=20221223; _SS=SID=3701616DA2BD6F32240D73EEA3BD6E8E; MMCASM=ID=FF993A37A0AC4FA4AD7DAD03BC6E8D52; SRCHHPGUSR=SRCHLANG=en&BRW=HTP&BRH=M&CW=976&CH=730&SCW=976&SCH=730&DPR=1.3&UTC=120&DM=0' \
  -H 'origin: https://www.bing.com' \
  -H 'pragma: no-cache' \
  -H 'referer: https://www.bing.com/images/search?view=detailV2&insightstoken=bcid_RLKVsIV2BwkFXg*ccid_spWwhXYH&form=SBIHMP&iss=SBIUPLOADGET&sbisrc=ImgPicker&idpbck=1&sbifsz=927+x+524+%c2%b7+25.15+kB+%c2%b7+png&sbifnm=serpapi-serpbear.png&thw=927&thh=524&ptime=223&dlen=34344&expw=798&exph=451&selectedindex=0&id=-1051855017&ccid=spWwhXYH&vt=2&sim=11' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sec-gpc: 1' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36' \
  --data-raw $'------WebKitFormBoundaryybS6WM5BOLApSZSk\r\nContent-Disposition: form-data; name="knowledgeRequest"\r\n\r\n{"imageInfo":{"imageInsightsToken":"bcid_RLKVsIV2BwkFXg*ccid_spWwhXYH","source":"Gallery"},"knowledgeRequest":{"invokedSkills":["ImageById","BestRepresentativeQuery","Offline","ObjectDetection","OCR","EntityLinkingFace","EntityLinkingDog","EntityLinkingAnimal","EntityLinkingPlant","EntityLinkingLandmark","EntityLinkingFood","EntityLinkingBook","SimilarImages","RelatedSearches","PagesIncluding","TextAds","ProductAds","SponsoredAds","Annotation","Recipes","Travel"],"invokedSkillsRequestData":{"adsRequest":{"textRequest":{"mainlineAdsMaxCount":2}}},"index":1}}\r\n------WebKitFormBoundaryybS6WM5BOLApSZSk--\r\n' \
  --compressed | jq -cr '.tags[] | select(.displayName == "##TextRecognition") | .actions[] | select(.actionType == "TextRecognition") | .data.regions[].lines[] | { text }'

Output

{"text":"README. md"}
{"text":"SerpApi"}
{"text":"SerpBear"}
{"text":"SerpBea"}
{"text":"et notice Position Tracking App. It"}
{"text":"positions in Google and get notified of their positions."}
{"text":"Documentation"}

Screenshot

image

ilyazub avatar Dec 23 '22 14:12 ilyazub

Thanks @ilyazub a lot. I looked into this API too, I can not find the JSON containing the Related content - Images.

andypple83 avatar Dec 23 '22 14:12 andypple83

@andypple83 Glad it helps!

I can not find the JSON containing the Related content - Images.

Here's my process:

  1. Ctrl+F in the Network tab of browser dev tools.

image

  1. Go to the Preview tab of the JSON response
  2. Expand JS object recursively (my Brave Browser doesn't search in collapsed JSON :confused:)

image

  1. Ctrl+F the target string

image

  1. Copy property path

image

  1. Navigate up and down in JS object (with arrow keys) to learn its structure and create an adapter.
  2. Copy as cURL and transform response with jq to check my assumption.

Tbh, Ctrl+Shift+F in the browser dev tools no longer searches across all responses.

image

I've initially proxied the browser network connections via mitmproxy because. Then filtered response bodies with ~bs "TEXT_FROM_THE_HTML_ELEMENT_I_"LOOKING_FOR".

Start mitmproxy with view filter

$ mitmproxy --view-filter '~bs "Freshsales"'

Start chromium-based browser with the target URL and the following flags and parameters

  • Proxy requests via mitmproxy: --proxy-server='http://127.0.0.1:8080'.
  • Use incognito mode (1) with a temporary user profile (2) ignoring insecure connections (3) and certificate errors (4): --temp-profile -incognito --user-data-dir="mktemp -d" --no-first-run --ignore-certificate-errors --allow-insecure-localhost. (I ignore certificate errors in a temporary browser profile to not install mitmproxy's certificates system-wide.)
$ brave-browser 'https://www.bing.com/images/search?view=detailV2&insightstoken=bcid_RLKVsIV2BwkFXg*ccid_spWwhXYH&form=SBIHMP&iss=SBIUPLOADGET&sbisrc=ImgPicker&idpbck=1&sbifsz=927+x+524+%c2%b7+25.15+kB+%c2%b7+png&sbifnm=serpapi-serpbear.png&thw=927&thh=524&ptime=223&dlen=34344&expw=798&exph=451&selectedindex=0&id=-1051855017&ccid=spWwhXYH&vt=2&sim=11' --proxy-server='http://127.0.0.1:8080'  --temp-profile -incognito --user-data-dir="`mktemp -d`" --no-first-run --ignore-certificate-errors --allow-insecure-localhost

image

mitmproxy will display the matched requests

image

PS. It's fun to ask ChatGPT about developer tools.

image

ilyazub avatar Dec 24 '22 06:12 ilyazub

A high-volume customer requested this feature.

Intercom

hilmanski avatar Mar 31 '24 23:03 hilmanski

Another high-volume customer requested this:

Intercom

martin-serpapi avatar Apr 30 '24 13:04 martin-serpapi

A user requested this feature (search via image URL).

Intercom

hilmanski avatar Jul 11 '24 04:07 hilmanski