爬虫实战项目分享

# winrm的使用

## 自定义端口
1. 删除现有的WinRM监听器
首先，需要删除现有的WinRM监听器。打开命令提示符（以管理员身份），运行以下命令：
```bash
 winrm delete winrm/config/listener?Address=*+Transport=HTTP
 winrm delete winrm/config/listener?Address=*+Transport=HTTPS
```
1. 创建新的监听器并指定端口
接下来，创建一个新的监听器并指定新的端口（例如8050）。运行以下命令：
```bash
 winrm create winrm/config/listener?Address=*+Transport=HTTP @{Port="8050"}
 winrm create winrm/config/listener?Address=*+Transport=HTTPS @{Port="8051"}
```



1. 验证监听器配置
```bash
winrm e winrm/config/listener
```
1. 确保开启服务
```bash
winrm quickconfig 
```
确保WinRM服务已正确配置以允许使用提供的凭据进行认证。运行以下命令以启用基本认证和允许未加密通信：
```bash
winrm set winrm/config/service/auth @{Basic="true"}
winrm set winrm/config/service @{AllowUnencrypted="true"}
```

## python调用

```python
import winrm

session = winrm.Session('http://ip:8050/wsman', auth=('user', 'pwd'))
result = session.run_cmd('ipconfig')
print(result.std_out.decode())

```

winrm的使用

# Xhopee协议之ds cookie算法
Xhopee的协议里有很多值得逆向的内容，这一篇文章我们主要讲解cookie中的ds的生成
![image.png](https://www.xsblog.site/static/img/c0b7a247acfdd4a82e288b45a1d489a5.image.webp)
我们可以看到，这是一个32位的字符串，首先想到会不会是一个md5算法呢？我们可以全局搜索md5的特征值



> A = 0x67452301  // 1732584193
> 
> B = 0xefcdab89  // -271733879
> 
> C = 0x98badcfe  // -1732584194
> 
> D = 0x10325476  // 271733878

经过一些断点分析和搜搜，可以最终定位到下面这段函数，这里就是魔改后的md5算法的位置
![image.png](https://www.xsblog.site/static/img/1128140478b37ef3ce2e42c6f613c791.image.webp)
下面我们把这个代码修改为nodeJS可以运行的模式
```javascript
function long_token_sign(token) {
    function n(e) {
        for (var t = "", n = 7; 0 <= n; n--)
            t += "0123456789abcdef".charAt(e >> 4 * n & 15);
        return t
    }

    function r(e, t) {
        var n = (65535 & e) + (65535 & t);
        return (e >> 16) + (t >> 16) + (n >> 16) << 16 | 65535 & n
    }

    function o(e, t) {
        return e << t | e >>> 32 - t
    }

    var t = {}
    t.default = function (e, t) {
        for (var a, i, s, c = function (e) {
            for (var t = 1 + (e.length + 8 >> 6), n = new Array(16 * t), r = 0; r < 16 * t; r++)
                n[r] = 0;
            for (r = 0; r < e.length; r++)
                n[r >> 2] |= e.charCodeAt(r) << 24 - r % 4 * 8;
            return n[r >> 2] |= 128 << 24 - r % 4 * 8,
                n[16 * t - 1] = 8 * e.length,
                n
        }(e), l = new Array(80), u = 1732584193, d = -271733879, f = -1732584194, p = 271733878, h = -1009589776, v = 0; v < c.length; v += 16) {
            for (var _ = u, m = d, g = f, b = p, y = h, x = 0; x < 80; x++) {
                l[x] = x < 16 ? c[v + x] : o(l[x - 3] ^ l[x - 8] ^ l[x - 14] ^ l[x - 16], 1);
                var E = r(r(o(u, 5), (E = d,
                    i = f,
                    s = p,
                    (a = x) < 20 ? E & i | ~E & s : !(a < 40) && a < 60 ? E & i | E & s | i & s : E ^ i ^ s)), r(r(h, l[x]), (a = x) < 20 ? 1518500249 : a < 40 ? 1859775393 : a < 60 ? -1894007588 : -899497514));
                h = p,
                    p = f,
                (t || (f = o(d, 30)) && t) && t || (d = u),
                    u = E
            }
            (u = r(u, _)) && t || (d = r(d, m)),
                f = r(f, g),
            (t || (p = r(p, b)) && t) && t || (h = r(h, y))
        }
        return n(u) + n(d) + n(f) + n(p) + n(h)
    }
    return t.default(token)
}
```

![image.png](https://www.xsblog.site/static/img/32779e83e7566da4f5cf61837965b1a4.image.webp)
可以看到ds的生成逻辑就是由shopee_webUnique_ccd拼接一个uuid计算而来，下面是node运行结果

![image.png](https://www.xsblog.site/static/img/2ffcedac03a17302d709508128a76c83.image.webp)
可以看到是正确运行的结果
# 后话
这个参数生成依赖的这两个值一个是shopee返回的设备短签名，这个值在之前的文章中有提到，第二个uuid也不是随机的uuid，经过逆向分析后，我们可以确定这个值是这个文件的之前的代码中生成的__WIR_SZ_UNIQ_DC放置在localStorage中

![image.png](https://www.xsblog.site/static/img/9711064790076391764c7dc24f8baa68.image.webp)

xhopee之cookie中的ds生成协议

## 引言
本文主要探讨python环境下的文件上传请求，包含各种情况以及最终的完美方案

在接到图搜需求的时候，首先抓包看一下图搜的请求是什么样的，看到curl是这个样子的：

```bash
curl -X POST '****' -H 'User-Agent: ****' -H 'content-type: multipart/form-data; boundary=BlFoMb-m.gnAh.sIVeQqAVu3Z.1abMrT2QIu8kmAqv-9ttGN5dLrZGkw5eTp3x5YjyGUrp' -H '****: ****' -H 'x-csrftoken: RRGBA6ten9P0gfSoQJsJBAB1chLbYcbe' -H 'x-search-entrance: HOME' -H 'x-api-source: rn' -H '****: ****' -H '****: ****' -H '****: ****' -H 'x-search-image-source: gallery_panel' -H '****: ****' -H 'referer: ****' -H 'accept-language: ko-KR,ko,en-US,en,en-GB' -H 'Cookie: ****' -F 'md5=imagesearch_ce651b0886b615bcd0cad0c0cd7754c6' -F 'language=1' -F 'shop_id=undefined' -F 'item_id=undefined' -F 'result_type=0' -F 'offset=0' -F 'limit=20' -F 'athenaCameraParams={}' -F 'file=@"/var/mobile/Containers/Data/Application/4AB0A55A-10A4-4DC0-A188-85042D232D28/Library/Application Support/tmp/91ffcd2d-5fe8-4ea4-b5aa-cb9b628cb86d";filename="91ffcd2d-5fe8-4ea4-b5aa-cb9b628cb86d"
```
我们可以看到这个请求和平时我们经常遇到的发送json荷载的post请求不同，***multipart/form-data***，并且请求体是以-F结尾，当时看到这个请求的时候其实还是有点蒙圈的，然后我去看了一下请求的原始数据，看到下面的内容：

![image.png](https://www.xsblog.site/static/img/c5153d7ad3c3bcc11a89681bdc0c7c38.image.webp)
经过查阅一些资料知道这是一个表单请求，web也有类似的请求，是用的form表单发起的post请求，在浏览器上的请求体长这个样子：

![image.png](https://www.xsblog.site/static/img/35f0f42e4f6f1c67a82967ff79d599e6.image.webp)
这个时候我们问题分析已经结束了，下面就要考虑如何发送这个请求了。
## 浏览器
类似的请求在浏览器上发送是很简单的，我们只需创建一个表单对象然后发送这个表单对象即可，请求头的content-type浏览器会自己帮我们加上
```javascript
const form = new FormData();
form.append('images', '');
form.append('thumbnail_size', '120');

fetch('****', {
  method: 'POST',
  body: form
});
```
这样我们就可以发送请求了，但是这个是面向自己开发服务的情况，那么我们如果要做爬虫开发，就需要用脚本开发，这个时候就需要考虑用请求库。
## requests
python的requests作为我们最常用的请求库，自然是支持这样的请求的。
```python
import requests
files = {
    'images': ('图片.png', open('图片.png', 'rb'), 'image/png'),
    'thumbnail_size': (None, '120'),
}
headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36'
}
response = requests.post('****', headers=headers, files=files)
```
同时也不需要指定请求的content-type，因为requests会自动帮我们生成请求头的content-type

一般的情况我们分析到这里就可以愉快的发送请求了，但是经过测试发现，请求什么参数都是准确的，但是过不了服务器的风控，这里就是requests的一些局限性，他不能过爬虫的tls检测。这个时候我们就需要找一个支持tls和文件上传的请求库。
## go
go语言是天生容易修改tls的信息的，作者在这里放几个有名的go第三方请求库
[https://req.cool/zh/docs/tutorial/set-body/](https://req.cool/zh/docs/tutorial/set-body/)

[https://github.com/bogdanfinn/tls-client](https://github.com/bogdanfinn/tls-client)

[https://github.com/wangluozhe/requests](https://github.com/wangluozhe/requests)

由于作者的go语言水平一般，这里就不赘述go的请求怎么写了，有兴趣的同学可以自行尝试，本文主要讨论如何用python实现，接下来我们继续探索python的写法。

## requests-go
在写go语言的时候，突然发现了这样一个python第三方库

[https://github.com/wangluozhe/requests-go](https://github.com/wangluozhe/requests-go)

这个第三方库是开源作者基于go版requests以及python版的requests写的一个python工具，可以十分方便的转化tls信息，同时看源码的时候发现作者也保留了files类型的请求，我们这个时候很愉快的开始写脚本
```python
import requests_go as requests
from requests_go import tls_config

url = "****"
tc = {
    # 这个可以见github怎么获取
}

files = [
    ('language', (None, '1')),
    ('result_type', (None, '0')),
    ('offset', (None, '0')),
    ('limit', (None, '40')),
    ('athenaCameraParams', (None, 'undefined')),
]


headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36'
}
tls_conf = tls_config.to_tls_config(tc)

response = requests.post(url, files=files, headers=headers, tls_config=tls_conf, proxies={
    "http": "http://localhost:17890",
    "https": "http://localhost:17890",
})

print(response.json())
```
经过测试发现，requesst-go可以模拟表单请求，但是表单里面只支持发送字符串，不能发送图片的信息，如果添加图片的话会出现异常：
```python
import requests_go as requests
from requests_go import tls_config

url = "****"
tc = {
    # 这个可以见github怎么获取
}

files = [
    ('file', ('图片.png', open('图片.png', 'rb'), 'image/png')),
]


headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36'
}
tls_conf = tls_config.to_tls_config(tc)

response = requests.post(url, files=files, headers=headers, tls_config=tls_conf, proxies={
    "http": "http://localhost:17890",
    "https": "http://localhost:17890",
})

print(response.json())
```

![image.png](https://www.xsblog.site/static/img/55a41f9f3097896d0c570a2ba118a4c9.image.webp)

这里应该是请求库对body进行了编码，但是图片的字节无法被编码，不知道是不是我的写法有问题还是开源作者的一个小bug。
## tls_client
这个时候我又去寻找了另外一个第三方库，这个请求库在python的热度也是很高的，群友在github上找到一个写法可以进行文件上传：
```python
import tls_client, random, string
from io import BytesIO
from os.path import basename

__all__ = ['File', 'FileUploader']

def randSeq(length: int, chars: str, encode: str = 'ascii'):
    return bytes().join(random.choice(chars).encode(encode) for _ in range(length))

class File:
    def __init__(self, file: str|BytesIO, *, content_type: str = None, name: str = None):
        self.name = name
        self.content_type = content_type
        if type(file) == str:
            self.file = open(file, 'rb')
            if name == None or len(name) == 0:
                self.name = basename(self.file.name)
        elif isinstance(file, BytesIO):
            self.file = file
            self.name = name
        else:
            raise TypeError("'file' object must be string or BytesIO")
    def extract(self):
        return self.name, self.file, self.content_type

class FileUploader:
    def __init__(self, sess: tls_client.Session):
        self.new_session(sess)
    def new_session(self, sess: tls_client.Session):
        self.sess = sess
        self.reset()
    def reset(self):
        self.__new_boundary()
        self.body = b''
    def __new_boundary(self):
        self.boundary = b"----WebKitFormBoundary" + randSeq(16, string.ascii_lowercase + string.ascii_uppercase + string.digits)
    def __generate_file_header(self, name: str, filename: str = None, content_type: str = None):
        header = b'Content-Disposition: form-data; name="' + name.encode() + b'"; '
        if filename != None:
            header += b'filename="' + filename.encode() + b'"; '
        if content_type != None:
            header += b'\r\nContent-Type: ' + content_type.encode()
        return header
    def addFile(self, name: str, file: File):
        filename, fp, mime = file.extract()

        self.body = b'--' + self.boundary + b'\r\n'
        self.body += self.__generate_file_header(name, filename, mime)
        self.body += b'\r\n\r\n'
        self.body += fp.read()
        self.body += b'\r\n--' + self.boundary + b'--\r\n'
    def upload(self, url: str, *, files = None, data = None, json = None, **kwargs):
        headers = kwargs.pop('headers', dict())
        headers['Content-Type'] = 'multipart/form-data; boundary=' + self.boundary.decode()
        return self.sess.post(url, data=self.body, headers=headers, **kwargs)
```
用法是：
```python
session = tls_client.Session(client_identifier="chrome_120", random_tls_extension_order=True)

url = "****"
headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36'
}
form = FileUploader(session)
img = File('图片.png', content_type="image/png")
form.addFile("images", img)
resp = form.upload(url=url, headers=headers, proxy="http://127.0.0.1:17890")
print(resp.text)
```

这样确实是可以发送表单格式的请求，但是缺点是只能添加一个文件，没发添加多个文件。（也可能是我自己不会写）

## 最终尝试
经过上面几种尝试方法，似乎文件上传的方法都有缺陷，难道python真的不能做到完美的文件上传请求并且模拟tls指纹吗（不进行大量二次开发的前提，当然有实力的同学可以重写一个requests做到完美tls）

这个时候一个第三方库引起了我的注意：
[https://github.com/requests/toolbelt](https://github.com/requests/toolbelt)

这个库可以帮助我们编写表单信息以及content-type

```python
from requests_toolbelt.multipart.encoder import MultipartEncoder
import hashlib
import tls_client

session = tls_client.Session(client_identifier="chrome_120", random_tls_extension_order=True)

encoder = MultipartEncoder(
    fields=[
        ('language', (None, '1')),
        ('file', ('图片.png', open('图片.png', 'rb'), 'image/png')),
    ]
)
headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36'
    'content-type': encoder.content_type,
}
response = session.post(
    "****", headers=headers,
    data=encoder.to_string(), proxy="http://127.0.0.1:17890"
)
print(response.json())

```
我们只要这样就可以做到文件上传以及模拟tls指纹了
## 结语

我们使用MultipartEncoder用requests-go发送请求同样也会出现前文的异常，所以目前还是用tls_client发起请求比较好。

这次也学到了很多新的知识，自己也摸索了几天，尝试了各种方法，最后在群友的分享整合出了这个方案，所以也写这一篇博客分享。

附上成功请求数据的图片哈哈哈

![image.png](https://www.xsblog.site/static/img/58e9c9640f7f599bf865e75779718099.image.webp)

从图搜谈关于python请求与multipart/form-data的一些心得

[原始git链接](https://github.com/xishandong/crawlProject/blob/main/%E8%BF%9B%E9%98%B6%E7%AF%87/%E5%9F%BA%E7%A1%80%E7%BB%BC%E5%90%88/Instagram/downloader.py)

## INS爬虫
最近有很多朋友发邮件和私信询问ins爬虫的问题，我重新去看了一下，然后把新版更新了一下。

还有由于tls指纹，我讲请求的第三方库调整为tls_client模拟浏览器调用。

出现问题的主要原因是请求头的参数提取有问题，新版的请求头已经不是原来的那个样子了，这样会导致请求的返回是这样的：
```json
{'message': 'useragent mismatch', 'status': 'fail'}
```



我们修改请求头后就可以得到准确的返回结果
```json
{
    "biography": "",
    "username": "renebaebae",
    "fbid": "17841419632210822",
    "full_name": "IRENE",
    "id": "19446582407",
    "followed_by": 11689096,
    "follow": 4,
    "avatar": "https://scontent-nrt1-1.cdninstagram.com/v/t51.2885-19/69067759_957976681207922_446652332218777600_n.jpg?stp=dst-jpg_s320x320&_nc_ht=scontent-nrt1-1.cdninstagram.com&_nc_cat=1&_nc_ohc=lqyt1uLD7bkQ7kNvgElH_Vc&_nc_gid=da9e9a13bbbb40afb7ca800ef16e8ded&edm=AOQ1c0wBAAAA&ccb=7-5&oh=00_AYDS_6vHqjNGgGPP1uGP2VCpuzkcf-UR4zaatJ_IPdTFJg&oe=66F7168F&_nc_sid=8b3546",
    "noteCount": 165,
    "is_private": false,
    "is_verified": true
}
```
### cookie的样例
还有部分朋友不知道cookie的格式，这里我也把cookie的格式贴下：

```json
{
    "wd": "",
    "dpr": "",
    "mid": "",
    "datr": "",
    "ig_did": "",
    "ig_nrcb": "",
    "ps_l": "1",
    "ps_n": "1",
    "csrftoken": "",
    "ds_user_id": "",
    "sessionid": "",
    "shbid": "",
    "shbts": "",
    "rur": ""
    }
```
大概需要这些内容，简单的方法是把页面的cookie字符串贴下，然后找一个在线转化的转为python的字典即可。
### 代理
还有一部分朋友询问代理如何添加，这里我也将代理抽离到全局进行添加了
```json
proxy = {
    "http": "http://127.0.0.1:17890",
    "https": "http://127.0.0.1:17890",
}
```
按照python的格式修改即可
### 原始代码

```python
import time

from tls_client import Session

# which need login cookie
# user info, not necessarily
# post, need
# comment, need

# cookie example
# {
#     "wd": "",
#     "dpr": "",
#     "mid": "",
#     "datr": "",
#     "ig_did": "",
#     "ig_nrcb": "",
#     "ps_l": "1",
#     "ps_n": "1",
#     "csrftoken": "",
#     "ds_user_id": "",
#     "sessionid": "",
#     "shbid": "",
#     "shbts": "",
#     "rur": ""
#     }
cookie = {
    # your cookie
}

PARAMS = r'("app_id":\s*"[^"]+")|("claim":\s*"[^"]+")|("csrf_token":\s*"[^"]+")|(["LSD",[],{"token":\s*"[^"]+")'

URLS = [
    'https://i.instagram.com/',
    'https://i.instagram.com/api/v1/users/web_profile_info/',
    'https://i.instagram.com/api/v1/feed/user',
    'https://i.instagram.com/api/v1/media/'
]
# proxy, selectable
proxy = {
    "http": "http://127.0.0.1:17890",
    "https": "http://127.0.0.1:17890",
}


class Ins:
    def __init__(self, cookies: dict):
        self.cookies = cookies
        self.session = Session(client_identifier="chrome_104", random_tls_extension_order=True)
        self.session.proxies.update(proxy)
        self.headers = {
            'sec-fetch-mode': 'cors',
            'referer': 'https://www.instagram.com/',
            'x-ig-app-id': '936619743392459',
            'sec-fetch-site': 'same-site',
            'accept-language': 'zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7',
            'x-asbd-id': '198387',
            'accept': '*/*',
            'sec-ch-ua': 'Chromium";v="104", " Not A;Brand";v="99", "Google Chrome";v="104"',
            'sec-ch-ua-mobile': '?0',
            'x-ig-www-claim': 'hmac.AR11qy__GsvLpiS4wKBygLGdxs2DxJB1esTkBw7b2QFaHH2d',
            'authority': 'i.instagram.com',
            'sec-ch-ua-platform': 'Windows"',
            'x-instagram-ajax': '1006400593',
            'sec-fetch-dest': 'empty',
            'user-agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36'
        }

    def ajax_request(self, url: str, /, params=None):
        """
        do requests, the engine of class
        :param url: api url
        :param params: api params
        :return: json object
        """
        for _ in range(5):
            try:
                resp = self.session.get(url, headers=self.headers, params=params, cookies=self.cookies)
                return resp.json()
            except:
                time.sleep(15)
        else:
            return None

    def get_userInfo(self, userName: str):
        """
        get user info by username
        :param userName: name of user
        :return: dict of user info
        """
        params = {
            'username': userName,
        }
        resp = self.ajax_request(URLS[1], params=params)
        if resp:
            try:
                # to avoid exception? Internet went wrong may return wrong information
                data = resp['data']['user']
            except KeyError:
                raise 'Could not get user information...'
            return {
                'biography': data.get('biography'),
                'username': data.get('username'),
                'fbid': data.get('fbid'),
                'full_name': data.get('full_name'),
                'id': data.get('id'),
                'followed_by': data.get('edge_followed_by', {}).get('count'),
                'follow': data.get('edge_follow', {}).get('count'),
                'avatar': data.get('profile_pic_url_hd'),
                'noteCount': data.get('edge_owner_to_timeline_media', {}).get('count'),
                'is_private': data.get('is_private'),
                'is_verified': data.get('is_verified')
            } if data else 'unknown User'

    def get_userPosts(self, userName: str):
        """
        get all posts from the username
        :param userName:  name
        :return: generator
        """
        continuations = [{
            'count': '12',
        }]
        temp = userName + '/username/'
        while continuations:
            continuation = continuations.pop()
            # url will change when second request and later
            url = URLS[2] + f'/{temp}'
            resp = self.ajax_request(url, params=continuation)
            # no such user
            if not resp.get('user'):
                yield 'checking cookie or unknown/private User: {}'.format(userName)
            else:
                _items = resp.get('items')
                # simulate the mousedown
                if resp.get('more_available'):
                    continuations.append({'count': '12', 'max_id': resp.get('next_max_id')})
                    user = resp.get('user')
                    temp = user.get('pk_id') if user.get('pk_id') else user.get('pk')
                yield from self.extract_post(_items)

    def get_comments(self, id):
        """
        get comments by given post id
        :param id:
        :return: generator of comments
        """
        continuations = [{
            'can_support_threading': 'true',
            'permalink_enabled': 'false',
        }]
        # base url
        url = URLS[3] + f'{id}/comments/'
        while continuations:
            continuation = continuations.pop()
            resp = self.ajax_request(url, params=continuation)
            if resp.get('next_min_id'):
                continuations.append({
                    'can_support_threading': 'true',
                    'min_id': resp.get('next_min_id')
                })
            comments = resp.get('comments')
            if comments:
                for comment in comments:
                    yield {
                        'id': comment.get('pk'),
                        'user_name': comment.get('user', {}).get('username'),
                        'user_fullname': comment.get('user', {}).get('full_name'),
                        'text': comment.get('text'),
                        'created_at': comment.get('created_at'),
                        'comment_like_count': comment.get('comment_like_count'),
                        'reply_count': comment.get('child_comment_count')
                    }
                    if comment.get('child_comment_count') > 0:
                        yield from self.get_child_comment(id, comment.get('pk'))
            else:
                yield 'no comments or losing login cookies'

    def get_child_comment(self, main_id, id):
        """
        get child of the comment by comment_id, only used in function get_comments().
        :param main_id: post id
        :param id: comment_id
        :return: to comments generator
        """
        url = f'https://www.instagram.com/api/v1/media/{main_id}/comments/{id}/child_comments/'
        continuations = [{'max_id': ''}]
        while continuations:
            continuation = continuations.pop()
            resp = self.ajax_request(url, params=continuation)
            cursor = resp.get('next_max_child_cursor')
            if cursor:
                continuations.append({'max_id': cursor})
            comments = resp.get('child_comments')
            if comments:
                for comment in comments:
                    yield {
                        'id': comment.get('pk'),
                        'user_name': comment.get('user', {}).get('username'),
                        'user_fullname': comment.get('user', {}).get('full_name'),
                        'text': comment.get('text'),
                        'created_at': comment.get('created_at'),
                        'comment_like_count': comment.get('comment_like_count'),
                    }

    @staticmethod
    def extract_post(posts):
        """
        to extract a post from a list of posts
        :param posts: original instagram posts
        :return: dict of posts
        """
        for post in posts:
            caption = post.get('caption')
            item = {
                'code': post.get('code'),
                'id': post.get('pk'),
                'pk_id': post.get('id'),
                'comment_count': post.get('comment_count'),
                'like_count': post.get('like_count'),
                'text': caption.get('text') if caption else None,
                'created_at': caption.get('created_at') if caption else post.get('taken_at'),
            }
            # other type can be added by yourself
            types = post.get('media_type')
            item.update({
                'photo': [_.get('image_versions2', {}).get('candidates', [{}])[0].get('url') for _ in
                          post.get('carousel_media')]
            }) if types == 8 else None
            item.update({
                'video': post.get('video_versions', [{}])[0].get('url')
            }) if types == 2 else None
            item.update({
                'photo': post.get('image_versions2', {}).get('candidates', [{}])[0].get('url')
            }) if types == 1 else None
            yield item


if __name__ == '__main__':
    INS = Ins(cookie)
    # items = INS.get_userPosts('renebaebae')
    items = INS.get_comments('3092771276598639274')
    for item in items:
        print(item)
        break
    item = INS.get_userInfo('renebaebae')
    print(item)


```

### 是否需要添加cookie

| 类型 | 是否需要cookie |
| --- | --- |
| user_info | ❌ |
| post | ✅ |
|comment | ✅ |

爬虫项目实战

Star History

winrm的使用

自定义端口

Xhopee协议之ds cookie算法

引言

INS爬虫