河小马
在本网页爬取教程中，我们将了解如何爬取RightMove – 英国流行的房地产列表网页。
我们将爬取在英国出售或出租的房源的房地产信息。为此，我们将使用 Python 和一些社区库。
我们还将了解如何使用 RightMove 的搜索来查找列表，我们还将使用该搜索来跟踪新列表，以便在房地产市场中获得优势。
为什么要爬取 RightMove.co.uk？
RightMove.com 是英国颇受欢迎的新兴房地产网站。它包含有关房地产销售的大型数据集，并包含房地产价格、挂牌地点和销售日期、卖家联系方式和一般房产信息等字段。
这对于市场分析、房地产行业研究以及竞争对手的总体概述来说都是有价值的信息。通过网络爬取 RightMove，我们可以轻松访问主要的公共房地产数据集。
可用的 Rightmove 数据字段
我们可以爬取 Rightmove 的几个流行的房地产数据字段和目标：
- 价钱
- 地点详情
- 开放日活动
- 详细照片
- 房地产经纪人信息
- 房产特点和细节
请参阅我们将要爬取的内容的演示数据集：
爬取结果预览
{
"id": "122875838",
"available": true,
"archived": false,
"phone": "01822 667990",
"bedrooms": 5,
"bathrooms": 4,
"type": "BUY",
"property_type": "Detached",
"tags": [],
"description": "NOW WITH 7 ACRES AND OUTBUILDINGS. A simply incredible, converted Victorian church, neither Listed nor consecrated, finished to a truly exceptional standard, coupled with a separate log cabin for holiday letting and approximately 7 acres of land. Church Conversion of Immense Stature, Outstanding, High-specification Finish, Gated Drive and Four Garages, Separate Holiday Letting Log Cabin, Pasture Field and Outbuildings, Peaceful and Picturesque Setting, 7.69 Acres In All, No Chain, Freehold, Council Tax Band G. EPC Band: D<br /><br /><b>Situation</b> - This incredible home is located outside of the quiet and unspoilt village of Latchley, which falls within the Tamar Valley Area of Outstanding Natural Beauty (AONB) and a UNESCO World Heritage Site, occupying a commanding position above the river Tamar and enjoying some superb, panoramic views of the valley below. Only two miles away, the self-contained village of Gunnislake has a Post Office, two general stores catering for day-to-day needs, a Co-op fuel station and mini-supermarket, health centre, public houses and a railway station offering a regular service into Plymouth city (a 25-minute journey). Tavistock, 6.5 miles to the east, is a hugely popular market town in West Devon, rich in history and tradition dating to the 10th century. Today, the town offers a superb range of shopping, recreational and educational facilities, including the sought-after private and independent school, Mount Kelly.<br /><br /><b>Description</b> - A true one-of-a-kind, this remarkable home was originally built as the parish church for nearby Latchley, commissioned by the diocese of Truro and designed by the Victorian architect, J P St Aubyn, in the Gothic revival style. Converted into a residence some 100 years later (having never been consecrated), the property has more recently undergone a programme of refurbishment, and now represents one of the finest and most interesting homes in the area. Combined with a turn-key holiday let and 7 acres of land overlooking the breath-taking valley view, this unique residence would make a wonderful home for the aspiring smallholder, a perfect equestrian opportunity or an idyllic family home, and a perfect home-and-income opportunity for those seeking a desirable lifestyle move. The property is offered with no onward chain.<br /><br /><b>Thimble Hall & Thimble Lodge</b> - This impressive statement home has magnificent accommodation arranged over three floors, totalling 2,853 sq.ft. The building has retained many original features, including several substantial arched timber doors, extensive exposed structural timbers, the church bell and pulley, and a circular turret and stone spiral staircase within. Contemporary finishing touches includes a handmade English oak kitchen with Italian marble work surfaces, numerous bespoke Gothic-arched double-glazed windows, bespoke American oak joinery, floor-level lighting and cast-iron radiators. On the ground floor is the original church vestibule leading into a hallway, two double bedrooms and a beautifully finished shower room. Double doors lead into the striking, vaulted living room with a separate study to one side. On the lower ground floor is the hugely impressive kitchen and dining area, complete with breakfast bar, Everhot range cooker, integral dishwasher, wine cooler and walk-in larder. On the first floor are the beautiful master suite, including two walk-in wardrobes and a sublime en-suite shower room, plus the final bedroom and separate bathroom. From the master bedroom, a direct view of the river Tamar can be enjoyed across the vaulted sitting room.<br /><br />In the grounds is a charming, solid oak log cabin, comprising a bedroom/reception, kitchenette, WC and shower area, which sits within its own large garden featuring a timber pagoda, enjoying an uninterrupted view of the wooded Tamar valley. It has its own entrance and parking, and lies directly adjacent to the main dwelling. Our client generates an income through holiday letting of the cabin, which is extremely popular. Details of turnover etc. can be shared with interested parties following a viewing.<br /><br /><b>Gardens & Land</b> - Both Thimble Hall and Thimble Lodge sit in their own large separate gardens with incredible views over the Tamar Valley. The property is complemented by approximately 7 acres of land, providing an excellent opportunity to create a smallholding and live the self-sufficient lifestyle, an equestrian facility in the quiet countryside or a re-wilded natural haven for wildlife. In the southern corner of the field are two small fenced paddocks served by a field shelter. Adjacent to the paddocks are a stone-built tractor shed and storage container, both with power and water connected. There is also a raised decking area with a firepit, the perfect discreet area for barbeques or evening drinks in the warmer months.<br /><br /><b>Agents Note</b> - 1. Some of the furniture within the house may also be available by separate negotiation.<br />2. Services are mains water & electricity, biomass boiler providing central heating and hot water, 2x septic tanks. Please note that we have neither inspected nor tested these services. Council Tax Band G.<br /><br />",
"title": "5 bedroom detached house for sale in Latchley, Tamar Valley, PL18",
"subtitle": "5 bedroom detached house",
"price": "\u00a31,275,000",
"price_sqft": "\u00a3446.90 per sq. ft.",
"address": {
"displayAddress": "Latchley, Tamar Valley",
"countryCode": "GB",
"deliveryPointId": 90875566,
"ukCountry": "England",
"outcode": "PL18",
"incode": "9AX"
},
"latitude": 50.536003,
"longitude": -4.243573,
"features": [
"Unique Lifestyle Opportunity",
"Church Conversion of Immense Stature",
"Outstanding, High-specification Finish",
"Gated Drive and Four Garages",
"Separate Holiday Letting Log Cabin",
"Pasture Field and Outbuildings",
"Equestrian Potential - 7.69 Acres In All",
"Peaceful and Picturesque Setting",
"Freehold (No Chain)",
"Council Tax Band G"
],
"history": {
"listingUpdateReason": "Reduced on 17/01/2023"
},
"photos": [
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_46_0000.jpeg",
"caption": "Elevated Aspect"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_44_0000.jpeg",
"caption": "Rear Elevated Showing Log Cabin"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_50_0000.jpeg",
"caption": "Fenced Paddocks and Field Shelter"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_02_0000.jpeg",
"caption": "Sitting Room"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_03_0000.jpeg",
"caption": "Sitting Room"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_04_0000.jpeg",
"caption": "Kitchen/Dining"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_05_0000.jpeg",
"caption": "Bespoke Kitchen"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_06_0000.jpeg",
"caption": "Kitchen"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_07_0000.jpeg",
"caption": "Stone Spiral Staircase"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_08_0000.jpeg",
"caption": "Master Bedroom"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_09_0000.jpeg",
"caption": "Master Shower Room"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_10_0000.jpeg",
"caption": "View from Master Bedroom"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_11_0000.jpeg",
"caption": "Bedroom 2"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_12_0000.jpeg",
"caption": "Bedroom 3"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_13_0000.jpeg",
"caption": "First Floor Bathroom"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_14_0000.jpeg",
"caption": "Ground Floor Shower Room"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_49_0000.jpeg",
"caption": "Field back towards house"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_36_0000.jpeg",
"caption": "Rear Elevation"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_16_0000.jpeg",
"caption": "Driveway and Garages"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_17_0000.jpeg",
"caption": "Four Garages"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_18_0000.jpeg",
"caption": "Garden and View"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_20_0000.jpeg",
"caption": "Patio and view across field"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_21_0000.jpeg",
"caption": "View from Patio"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_23_0000.jpeg",
"caption": "Log Cabin"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_24_0000.jpeg",
"caption": "Log Cabin Exterior"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_25_0000.jpeg",
"caption": "Log Cabin Interior"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_26_0000.jpeg",
"caption": "Log Cabin Interior"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_37_0000.jpeg",
"caption": "View from the Cabin"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_38_0000.jpeg",
"caption": "Accompanying Field and View of Valley"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_30_0000.jpeg",
"caption": "Log Cabin Garden"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_31_0000.jpeg",
"caption": "Garden and Stone Archway"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_51_0000.jpeg",
"caption": "View across the field - looking north"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_47_0000.jpeg",
"caption": "View from the field - looking northeast"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_53_0000.jpeg",
"caption": "View from paddocks looking north"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_48_0000.jpeg",
"caption": "Tractor Store and Container"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_IMG_52_0000.jpeg",
"caption": "Raised decking"
}
],
"floorplans": [
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_FLP_00_0000.jpeg",
"caption": "Floorplan - Whole"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_FLP_03_0001.jpeg",
"caption": "Location Plan"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_FLP_01_0000.jpeg",
"caption": "Floorplan - Main"
},
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_FLP_02_0000.jpeg",
"caption": "Floor Plan - Log Cabin and Outbuildings"
}
],
"agency": {
"id": 168911,
"branch": "Tavistock",
"company": "Stags",
"address": "2 Market Street,\r\nTavistock,\r\nPL19 0DA",
"commercial": false,
"buildToRent": false,
"isNew": false
},
"industryAffiliations": [
"Royal Institute of Chartered Surveyors",
"Property Redress Scheme"
],
"nearest_airports": [],
"nearest_stations": [
{
"name": "Gunnislake Station",
"distance": 1.7462277108445943
},
{
"name": "Calstock Station",
"distance": 3.0283321381415
},
{
"name": "Bere Alston Station",
"distance": 3.9448043992784925
}
],
"sizings": [
{
"unit": "sqft",
"min": 2853,
"max": 2853
},
{
"unit": "sqm",
"min": 265,
"max": 265
},
{
"unit": "ha",
"min": 0.03,
"max": 0.03
},
{
"unit": "ac",
"min": 0.07,
"max": 0.07
}
],
"brochures": [
{
"url": "https://media.rightmove.co.uk/169k/168911/122875838/168911_31451179_DOC_00_0060.pdf",
"caption": "Latchley, Tamar Valley"
}
]
}
正如您所看到的，RightMove 提供了许多有价值的数据字段。让我们来看看如何刮掉它们吧！
项目设置
在本教程中，我们将使用 Python 和三个社区包：
- httpx – HTTP 客户端库，它将让我们与 RightMove.co.uk 服务器进行通信
- parsel – HTML 解析库，它将帮助我们解析网页爬取的 HTML 文件。
- jmespath – JSON 解析库。将帮助我们从大型 JSON 数据集中解析出重要的细节。
对于这个爬取工具，我们将主要使用 JSON 和隐藏的 Web 数据，因此我们将主要使用httpx和jmespath包。
所有这些软件包都可以通过以下pip install命令轻松安装：
$ pip install httpx parsel jmespath
或者，您可以随意更换httpx任何其他 HTTP 客户端包（例如requests），因为我们只需要基本的 HTTP 函数，这些函数在每个库中几乎都是相同的。至于，parsel另一个不错的选择是beautifulsoup包。
对于解析，如果您想了解更多信息，我们将使用在使用 JMESPath 解析 JSONjmespath教程中介绍的包。
爬取 Rightmove 属性数据
为了开始我们的爬取工具，让我们看一下爬取单个列表的属性数据。
RightMove 使用 JSON 支持的前端来呈现其数据。像这样的爬取页面被称为隐藏的 Web 数据爬取，因为数据不在 HTML 源代码中，而是在隐藏在 HTML 中的 javascript 变量中。
让我们看一下这个示例列表：rightmove.co.uk/properties/129828533
要查找隐藏数据，我们可以选择一个唯一的数据值（例如属性描述的一部分）并在 HTML 源代码中搜索它。例如，通过在此列表的 HTML 源代码中搜索“稀有机会”，我们可以看到数据在 HTML 变量中可用PAGE_MODEL：
因此，为了解决这个问题，我们将：
- 使用检索属性页的 HTML
httpx - 用于
parsel解析HTML，找到<script>带有PAGE_MODEL变量的元素 - 将
PAGE_MODELJSON 作为 python 字典加载并清理它。
让我们使用httpxandparsel包来爬取这些数据：
import asyncio
import json
from typing import List
from httpx import AsyncClient, Response
from parsel import Selector
# 1. establish HTTP client with browser-like headers to avoid being blocked
client = AsyncClient(
headers={
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9,lt;q=0.8,et;q=0.7,de;q=0.6",
},
follow_redirects=True,
http2=True, # enable http2 to reduce block chance
timeout=30,
)
# XXX: we'll fill this in later
def parse_property(data):
"""parse rightmove property data to only necessary fields"""
return data
# This function will find the PAGE_MODEL javascript variable and extract it
def extract_property(response: Response) -> dict:
"""extract property data from rightmove PAGE_MODEL javascript variable"""
selector = Selector(response.text)
data = selector.xpath("//script[contains(.,'PAGE_MODEL = ')]/text()").get()
if not data:
print(f"page {response.url} is not a property listing page")
return
data = data.split("PAGE_MODEL = ", 1)[1].strip()
data = json.loads(data)
return data["propertyData"]
# this is our main scraping function that takes urls and returns the data
async def scrape_properties(urls: List[str]) -> List[dict]:
"""Scrape Rightmove property listings for property data"""
to_scrape = [client.get(url) for url in urls]
properties = []
for response in asyncio.as_completed(to_scrape):
response = await response
properties.append(parse_property(extract_property(response)))
return properties
# Eexample run:
async def run():
data = await scrape_properties(["https://www.rightmove.co.uk/properties/129828533#/"])
print(json.dumps(data, indent=2))
if __name__ == "__main__":
asyncio.run(run())
示例输出
[
{
"id": "129828533",
"status": {
"published": true,
"archived": false
},
"text": {
"description": "A rare opportunity to purchase around 6,880 square feet (circa 639 square metres) of prime waterside real estate with mature gardens, private quay and foreshore, located on the water's edge in one of the South West's most exclusive and picturesque south coastal villages.",
"propertyPhrase": "10 bedroom detached house",
"disclaimer": "<b>Disclaimer</b> - Property reference 11794080. The information displayed about this property comprises a property advertisement. Rightmove.co.uk makes no warranty as to the accuracy or completeness of the advertisement or any linked or associated information, and Rightmove has no control over the content. This property advertisement does not constitute property particulars. The information is provided and maintained by <b>H Tiddy, St Mawes</b>. Please contact the selling agent or developer directly to obtain any information which may be available under the terms of The Energy Performance of Buildings (Certificates and Inspections) (England and Wales) Regulations 2007 or the Home Report if in relation to a residential property in Scotland.",
"auctionFeesDisclaimer": null,
"guidePriceDisclaimer": null,
"reservePriceDisclaimer": null,
"staticMapDisclaimerText": "Map data \u00a9<a href=\"http://www.openstreetmap.org/copyright\" rel=\"nofollow\" target=\"_blank\" title=\"Open Street Map\">OpenStreetMap</a> contributors.",
"newHomesBrochureDisclaimer": "Please note that Rightmove will send the above details to H Tiddy only. By submitting this form, you confirm that you agree to our website <a href=\"/this-site/terms-of-use.html\">terms of use</a>, our <a href=\"/this-site/privacy-policy.html\">privacy policy</a> and understand how we store <a href=\"/this-site/cookies.html\">cookies</a> on your device.",
"shareText": "Check out this 10 bedroom detached house for sale on Rightmove",
"shareDescription": "10 bedroom detached house for sale in St Mawes, Cornwall, TR2 for \u00a36,000,000. Marketed by H Tiddy, St Mawes",
"pageTitle": "10 bedroom detached house for sale in St Mawes, Cornwall, TR2",
"shortDescription": "10 bedroom detached house for sale"
},
"prices": {
"primaryPrice": "\u00a36,000,000",
"secondaryPrice": null,
"displayPriceQualifier": "Guide Price",
"pricePerSqFt": null,
"message": null,
"exchangeRate": null
},
"address": {
"displayAddress": "St Mawes, Cornwall",
"countryCode": "GB",
"deliveryPointId": null,
"ukCountry": "England",
"outcode": "TR2",
"incode": "5AZ"
},
"keyFeatures": [
"A Distinctive Detached Marine Residence comprising 6 Bedrooms, 4 Reception and 5 Bathrooms.",
"A Detached One Bedroom Ancillary Waterside Retreat with an Established Holiday Letting Income.",
"A Historic 4 Bedroom, 2 Reception, 4 Bathroom Detached Dwelling, Perched on the Waters Edge. ",
"Includes a Private Quay, Foreshore to Mean Low Water and a Mooring.",
"Magnificent Uninterrupted River, Creek and National Trust Countryside Views.",
"Triple Garage, Ample Boat and Car Parking plus good sized Mature Gardens.",
"Walking Distance of Beaches, Shops, Fine Dining Restaurants, Pubs and Cafe's. ",
"Excellent Sheltered Deep Water Sailing Facilities and Beautiful Creekside Walks on the Doorstep.",
"Exclusive Coastal Village Location.",
"Watch our Promotional Video to Explore this Rare Opportunity and it's Enchanting Location Further."
],
"images": [
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_00_0000.jpeg",
"caption": "Waterside Living",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_00_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_00_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_00_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_01_0000.jpeg",
"caption": "Pendragon",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_01_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_01_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_01_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_02_0000.jpeg",
"caption": "The Studio",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_02_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_02_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_02_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_03_0000.jpeg",
"caption": "Pendragon",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_03_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_03_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_03_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_04_0000.jpeg",
"caption": "Pendragon Lounge",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_04_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_04_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_04_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_05_0000.jpeg",
"caption": "Pendragon Kitchen",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_05_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_05_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_05_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_06_0000.jpeg",
"caption": "Pendragon Annexe",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_06_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_06_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_06_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_07_0000.jpeg",
"caption": "River Views",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_07_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_07_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_07_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_08_0000.jpeg",
"caption": "Garden and Views",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_08_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_08_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_08_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_09_0000.jpeg",
"caption": "Castaway",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_09_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_09_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_09_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_10_0000.jpeg",
"caption": "Castaway Living Room",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_10_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_10_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_10_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_11_0000.jpeg",
"caption": "The Studio",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_11_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_11_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_11_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_12_0000.jpeg",
"caption": "The Studio Living Room",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_12_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_12_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_12_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_13_0000.jpeg",
"caption": "Studio Balcony",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_13_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_13_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_13_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_14_0000.jpeg",
"caption": "Studio Kitchen",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_14_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_14_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_14_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_15_0000.jpeg",
"caption": "Studio Bathroom",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_15_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_15_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_15_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_16_0000.jpeg",
"caption": "The Percuil River",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_16_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_16_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_16_0000_max_656x437.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_17_0000.jpeg",
"caption": "Location",
"resizedImageUrls": {
"size135x100": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_17_0000_max_135x100.jpeg",
"size476x317": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_17_0000_max_476x317.jpeg",
"size656x437": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_IMG_17_0000_max_656x437.jpeg"
}
}
],
"floorplans": [
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_FLP_00_0000.jpeg",
"caption": "Pendragon and Castaway Floorplans",
"type": "IMAGE",
"resizedFloorplanUrls": {
"size296x197": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_FLP_00_0000_max_296x197.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_FLP_01_0000.jpeg",
"caption": "The Studio Floorplan",
"type": "IMAGE",
"resizedFloorplanUrls": {
"size296x197": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_FLP_01_0000_max_296x197.jpeg"
}
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_FLP_02_0000.jpeg",
"caption": "Boundary Pans",
"type": "IMAGE",
"resizedFloorplanUrls": {
"size296x197": "https://media.rightmove.co.uk/dir/9k/8240/129828533/8240_11794080_FLP_02_0000_max_296x197.jpeg"
}
}
],
"virtualTours": [
{
"url": "https://www.youtube.com/embed/i6FfzzV5yeU",
"caption": "Virtual Tour",
"videoId": "i6FfzzV5yeU",
"provider": "YOUTUBE"
}
],
"customer": {
"branchId": 8240,
"branchName": "St Mawes",
"branchDisplayName": "H Tiddy, St Mawes",
"companyName": "H Tiddy",
"companyTradingName": null,
"displayAddress": "The Square,\r\nSt. Mawes,\r\nTR2 5AG",
"logoPath": "https://media.rightmove.co.uk/company/clogo_2832_0000.gif",
"customerDescription": {
"truncatedDescriptionHTML": "<p>* We take the strain out of buying and selling.</p><p>* Independent estate agent established for over 100 years.</p><p>* Free, no obligation confidential valuations.</p><p>* Extensive international, national and local registered buyers database.</p><p>* Prominent office location in the exclusive coastal village of St Mawes.</p><p>* 24/7 presence on interactive touch screens in London\u2019s Park Lane displaying our openly marketed properties.</p><p>* Local knowledge essential to successful sales a</p>",
"isTruncated": true
},
"bannerAd": null,
"mpuAd": null,
"customerProfileUrl": "/estate-agents/agent/H-Tiddy/St-Mawes-8240.html",
"customerBannerAdProfileUrl": "/estate-agents/agent/H-Tiddy/St-Mawes-8240.html#ram_about_us",
"customerMpuAdProfileUrl": "/estate-agents/agent/H-Tiddy/St-Mawes-8240.html#ram_about_us",
"customerPropertiesUrl": "/property-for-sale/find/H-Tiddy/St-Mawes.html?locationIdentifier=BRANCH%5E8240&includeSSTC=true&_includeSSTC=on",
"isNewHomeDeveloper": false,
"spotlight": null,
"showBrochureLeadModal": false,
"developmentInfo": {
"sitePlanUri": null,
"micrositeFeatures": []
},
"buildToRent": false,
"commercial": false,
"buildToRentBenefits": []
},
"industryAffiliations": [
{
"name": "Association of Residential Letting Agents",
"imagePath": "https://media.rightmove.co.uk/dir/customer/industry-affiliation/association-of-residential-lettings-agent_max_135x100.png"
},
{
"name": "National Association of Estate Agents",
"imagePath": "https://media.rightmove.co.uk/dir/customer/industry-affiliation/national-association-of-estate-agents_max_135x100.png"
},
{
"name": "Ombudsman for REsale Estate Agents (no OFT logo)",
"imagePath": "https://media.rightmove.co.uk/dir/customer/industry-affiliation/ombudsman-for-resale-estate-agent_max_135x100.png"
}
],
"rooms": [
{
"name": "Accommodation Summary",
"description": "Pendragon Main House: Ground Floor: Entrance Hall, Family / Dining Room, Lounge, Study, Kitchen, Utility, Cloakroom. First Floor: Landing, Principal Bedroom with En Suite Bathroom and Dressing Area, Bedrooms 2 and 3, Shower Room, Bathroom, Study / Bedroom 4.<br /><br /><br />Inter-Connecting Annexe (or additional accommodation to the main house): Reception Hall, Living Room with open plan Kitchen, Bedroom 5 with En Suite Bathroom and Dressing Area, Bedroom 6, Shower Room.<br /><br /><br />Pendragon Outside: Triple Garage, Basement / Store / Plant Room, Mature Surrounding Gardens with Sun Terraces, Boat / Parking Facilities.<br /><br /><br />Castaway: Detached Ancillary Chalet (currently Holiday Let): Living Room, open plan Kitchen, Bedroom, Shower Room, Decking with Hot Tub.<br /><br /><br />The Studio Ground Floor: Reception Hall, Bedroom Three, Family Bathroom, Utility / Boot Room, Media Room / Bedroom Four. First Floor: Living Room with Balcony, Kitchen / Dining Room, Bedroom One with En-Suite Shower Room, Guest Bedroom Two with...",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Location Summary (distances and times are approximate)",
"description": "Coastal Path / Riverside Walk: On the doorstep. Summers Beach: 350 yards. Village Centre and harbour-side: 850 yards. King Harry Ferry: 5 miles. Truro: 10 miles via car ferry. Falmouth: 20 minutes by foot ferry or 15 miles by car ferry. Cornwall Airport Newquay: 29 miles (regular flights to London and other UK regional airports). St Austell: 15 miles (London Paddington 4.5 hours by rail).",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "St Mawes",
"description": "The exclusive Cornish coastal village of St Mawes was voted in 2020 by a Which? Customer Survey as the \u201cTop Seaside Town in the UK\u201d. This enchanting south facing harbour village, named \u201cBritain's St Tropez\u201d by the Daily Mail, is found on the eastern side of the Fal Estuary, in an Area of Outstanding Natural Beauty. Much of its surroundings are owned by the National Trust. The village is centred round its quaint harbour, its own beaches and Castle. There is an all-year-round pedestrian ferry service to and from Falmouth and a seasonal ferry runs to Place, providing access to the scenic walks on the National Trust owned St Anthony Headland.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Local Amenities",
"description": "St Mawes has a wide range of amenities, which are open all year, including butchers, two bakers, convenience store, post office/newsagent, doctors, hairdressers, dentist, pharmacy, village hall, social club, churches, delicatessen and clothing shops. The village also has two public houses, and an excellent range of cafes, restaurants, art galleries, gift and ice cream shops. Olga Polizzi's Hotel Tresanton and the chic Idle Rocks Hotel both lure the rich and famous. The village has superbly varied and accessible sailing waters and an active sailing club with a full programme to cater for all ages.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Cornwall",
"description": "Cornwall has been enjoying a renaissance over the last couple of years. The county recently leapfrogged London to be recently crowned by rightmove as the top area in the UK for the most online property searches over the last year. The Duchy offers a range of accessible attractions such as the Eden Project, the National Maritime Museum, the Lost Gardens of Heligan, and the Tate Gallery. The Cathedral City of Truro is the main financial and commercial centre of Cornwall. It has a fine range of stores, private schools, college and the main hospital in the county (RCH Treliske). Fine dining Michelin star and celebrity chef restaurants are in abundance, including Rick Stein (Padstow), Nathan Outlaw (Port Isaac), Paul Ainsworth (Padstow and Rock). Rising stars on The Roseland are Olly Pierrepont at the Driftwood, Rosevine and Dorian Janmaat at the Idle Rocks, St Mawes.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Historical Note (Pendragon and Castaway)",
"description": "Pendragon was originally built in 1929. During 2002 to 2004, under planning application CI/FP131/0494/02, a major reconstruction to the main house including the addition of a two bedroom annexe was built by a local highly regarded building firm. The re-modelling and refurbishment included a new roof, heating system and bathrooms as can be seen today.<br /><br />In 2012 to 2013, under planning permission PA12/05126, the old hut in the grounds was demolished and a new detached chalet was built as 'ancillary' accommodation to the main residence. The chalet is now named Castaway and currently is holiday let via Cornish Gems.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Historical Note (The Studio)",
"description": "The Studio was originally built and owned in 1936 by the Lancaster family, and was let as boathouses to local residents with the upstairs used as a sail loft. The garage was a blacksmith's forge, evidence of which can still be seen today. In 1939 the well-known marine artist Charles Pears with his wife took over the sail loft as their studio, whilst they lived on separate boats on the river. When the Second World War was declared, it was no longer possible to live on board a boat, so Charles Pears and his wife were given permission to live in the studio which remained their home for the rest of their life time. The building then transferred back to the Lancaster family, who sold the property to a local family in St Mawes where it was used for over 20 years as a holiday home for themselves and their family until it was sold in early March 2021.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Historical Note (World War Two)",
"description": "Just by the entrance to The Studio, an inscription is chiselled into the quay. It reads \u201c81st Sea Bees Construction Battalion\u201d which was engraved there by American soldiers who built the quay and the access road during the build-up to the D-Day landings in June 1944.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Introduction",
"description": "The location of Pendragon, Castaway and The Studio, in our opinion, is second to none. Offering the best of all worlds, from accessibility to the river, seclusion in the mature gardens, and breath-taking, tranquil water and countryside views, all situated within a relatively level walk past Summers Beach to shops, restaurants and a wide range of central village amenities within this exclusive Cornish coastal village.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Description",
"description": "Pendragon has been designed to capture the most of the spectacular views and cater for easily manageable accommodation for the retired couple as well as provide ample independent living and bedroom space for the large family. Castaway, the detached chalet, offers an excellent retreat for guests, family or au-pair and is currently holiday let providing Pendragon's owners a lucrative income. <br /><br />The Studio is not only a rare find in St Mawes, but throughout the South West of England. With a private quay and foreshore offering direct water access, this attractive and unique waterfront home has, over the last 12 months or so, undergone a modernisation programme, thus transforming the property as seen today. Internally, the building still retains its bohemian artistic flavour of days gone by.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "The Views",
"description": "The uninterrupted and panoramic views enjoyed from the individual properties are captivating and inspiring. Mere photography does not do them justice. These encompass an expanse of the deep waters of the Percuil River including Pelyn and Porth creeks and the National Trust owned lightly wooded St Anthony Headland, views of which should be there in perpetuity \u2013 a very significant attribute. The river is a hive of wildlife and marine activity most of the year.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "General Information ",
"description": null,
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Agents Note",
"description": "Two individual properties are available separately. For further information, Pendragon is on the market a guide price of \u00a33,500,000 and The Studio is for sale at a guide price of \u00a32,750,000.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Services",
"description": "Mains water, electricity and drainage (via pump to main sewer). Oil fired central heating to radiators at Pendragon and electric radiator heating for Castaway and The Studio. <br />Telephone (superfast broadband connected). Television aerials and Satellite.<br /> NB: The electrical circuit, appliances and heating system have not been tested by the agents.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": " Energy Performance Certificate Ratings: 'D' and 'E'. ",
"description": null,
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Council Tax Bands",
"description": "'G' for Pendragon, 'A' for Castaway, and 'D' for The Studio.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Tenure: Freehold",
"description": null,
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Mooring Availability ",
"description": "Upon application to Percuil River Moorings Ltd.",
"width": null,
"length": null,
"unit": null,
"dimension": null
},
{
"name": "Important Notice",
"description": "Every effort has been made with these details but accuracy is not guaranteed and they are not to form part of a contract. Representation or warranty is not given in relation to this property. An Energy Performance Certificate is available upon request. All negotiations must be with H Tiddy. Before proceeding to purchase, buyers should consider an independent check of all aspects of the property. General Data Protection Regulations: We treat all data confidentially and with the utmost care and respect. If you do not wish your personal details to be used by us for any specific purpose, then you can unsubscribe or change your communication preferences and contact methods at any time by informing us either by email or in writing at our office in St Mawes.",
"width": null,
"length": null,
"unit": null,
"dimension": null
}
],
"location": {
"latitude": 50.15871,
"longitude": -5.005751,
"circleRadiusOnMap": 0,
"zoomLevel": 15,
"pinType": "ACCURATE_POINT",
"showMap": true
},
"streetView": {
"heading": 0,
"pitch": 0,
"zoom": 0,
"latitude": 50.15871,
"longitude": -5.005751
},
"nearestAirports": [],
"nearestStations": [
{
"name": "Falmouth Docks Station",
"types": [
"NATIONAL_TRAIN"
],
"distance": 2.298093516329164,
"unit": "miles"
},
{
"name": "Falmouth Town Station",
"types": [
"NATIONAL_TRAIN"
],
"distance": 2.7436587311392673,
"unit": "miles"
},
{
"name": "Penmere Station",
"types": [
"NATIONAL_TRAIN"
],
"distance": 3.4787958948158195,
"unit": "miles"
}
],
"showSchoolInfo": true,
"countryGuide": null,
"channel": "RES_BUY",
"propertyUrls": {
"similarPropertiesUrl": "/property-for-sale/TR2-5AZ.html?radius=0.5&minPrice=5000000&maxPrice=7500000&minBedrooms=5",
"nearbySoldPropertiesUrl": "/house-prices/tr2-5az.html"
},
"sizings": [],
"brochures": [
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_DOC_00_0000.pdf",
"caption": "Property Brochure"
},
{
"url": "https://www.htiddy.co.uk/index.php?option=com_eapow&format=html&task=property.getProp&pid=cb12dc24-725e-483e-8967-4026f59b24d9&aid={85dcf889-5df7-4f92-a690-1b46a0234359}",
"caption": "Full Details"
}
],
"epcGraphs": [
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_EPC_00_0000.pdf",
"caption": "Energy Performance Certificate"
}
],
"bedrooms": 10,
"bathrooms": 9,
"transactionType": "BUY",
"tags": [
"ONLINE_VIEWING"
],
"misInfo": {
"branchId": 8240,
"offerAdvertStampTypeId": null,
"premiumDisplay": true,
"premiumDisplayStampId": 27,
"brandPlus": true,
"featuredProperty": false
},
"dfpAdInfo": {
"channel": "RES_BUY",
"targeting": [
{
"key": "SO",
"value": [
"FALSE"
]
},
{
"key": "R",
"value": [
"FALSE"
]
},
{
"key": "AUCP",
"value": [
"FALSE"
]
},
{
"key": "CCY",
"value": [
"GBP"
]
},
{
"key": "UTo_A",
"value": [
"TR"
]
},
{
"key": "UTo_C",
"value": [
"GB"
]
},
{
"key": "CT",
"value": [
"property-for-sale"
]
},
{
"key": "TT",
"value": [
"buy"
]
},
{
"key": "BD",
"value": [
"10"
]
},
{
"key": "PT",
"value": [
"houses"
]
},
{
"key": "PST",
"value": [
"detached"
]
},
{
"key": "RESALEPRICE",
"value": [
"6000000"
]
},
{
"key": "B",
"value": [
"Band12"
]
},
{
"key": "B_ID",
"value": [
"8240"
]
},
{
"key": "C_ID",
"value": [
"H Tiddy"
]
},
{
"key": "FP",
"value": [
"Y"
]
},
{
"key": "EPC",
"value": [
"Y"
]
},
{
"key": "P_ID",
"value": [
"129828533"
]
},
{
"key": "OV",
"value": [
"Y"
]
},
{
"key": "UTo_D",
"value": [
"TR2"
]
},
{
"key": "UTo_Inc",
"value": [
"5AZ"
]
},
{
"key": "F",
"value": [
"parking",
"garden",
"hottub"
]
},
{
"key": "UTo_UKZ",
"value": [
"England"
]
}
]
},
"staticMapImgUrls": {
"staticMapImgUrlMobile": "https://media.rightmove.co.uk/map/_generate?width=375&height=375&zoomLevel=15&latitude=50.15871&longitude=-5.005751&signature=gCN7Y1ET3ltCjZ1LqrkVX1nIsbw=",
"staticMapImgUrlTablet": "https://media.rightmove.co.uk/map/_generate?width=768&height=347&zoomLevel=15&latitude=50.15871&longitude=-5.005751&signature=jgoQdXfY73Fwd8tm3uZjIWO5FRA=",
"staticMapImgUrlDesktopSmall": "https://media.rightmove.co.uk/map/_generate?width=580&height=347&zoomLevel=15&latitude=50.15871&longitude=-5.005751&signature=NMq2cOYMH4oW4rweitefV_nS_18=",
"staticMapImgUrlDesktopLarge": "https://media.rightmove.co.uk/map/_generate?width=761&height=347&zoomLevel=15&latitude=50.15871&longitude=-5.005751&signature=1V-0yD2HnRskxvJ3BAniN1MlAps="
},
"listingHistory": {
"listingUpdateReason": "Reduced on 18/01/2023"
},
"feesApply": null,
"broadband": {
"disclaimer": "* This is the average speed from the provider with the fastest broadband package available on <a rel=\"sponsored\" href=\"https://www.comparethemarket.com\" target=\"_blank\">comparethemarket.com</a> at this postcode. The average speed displayed is based on the download speeds of at least 50% of customers at peak time (8pm to 10pm). Fibre/cable services at the postcode are subject to availability and may differ between properties within a postcode. Speeds can be affected by a range of technical and environmental factors. The speed at the property may be lower than that listed above. You can check the estimated speed and confirm availability to a property prior to purchasing on the broadband provider's website. Providers may increase charges. The information is provided and maintained by <a href=\"https://www.stickee.co.uk/\" target=\"_blank\">Stickee Technology Limited</a>.",
"broadbandCheckerUrl": "https://partnerships-broadband.comparethemarket.com/v1/broadband/rightmove/TR25AZ?apikey=c12fd0ac0c28b34758568c23ed503209bb27401b800cd1506bd81d459f71bba8"
},
"contactInfo": {
"contactMethod": "EMAIL",
"telephoneNumbers": {
"localNumber": "01326 331658",
"internationalNumber": null,
"disclaimerText": null,
"disclaimerTitle": null,
"disclaimerDescription": null
}
},
"lettings": null,
"infoReelItems": [
{
"title": "PROPERTY TYPE",
"type": "PROPERTY_TYPE",
"primaryText": "Detached",
"secondaryText": "",
"tooltipText": ""
},
{
"title": "BEDROOMS",
"type": "BEDROOMS",
"primaryText": "\u00d710",
"secondaryText": "",
"tooltipText": ""
},
{
"title": "BATHROOMS",
"type": "BATHROOMS",
"primaryText": "\u00d79",
"secondaryText": "",
"tooltipText": ""
}
],
"mortgageCalculator": {
"price": 6000000,
"propertyTypeAlias": "houses"
},
"tenure": {
"tenureType": "FREEHOLD",
"yearsRemainingOnLease": null,
"message": null
},
"soldPropertyType": "DETACHED",
"auctionProvider": null,
"propertySubType": "Detached",
"businessForSale": false,
"commercial": false,
"commercialUseClasses": [],
"affordableBuyingScheme": false,
"sharedOwnership": {
"sharedOwnership": false,
"ownershipPercentage": null,
"rentPrice": null,
"rentFrequency": "not specified"
},
"livingCosts": {
"councilTaxExempt": false,
"councilTaxIncluded": false,
"annualGroundRent": null,
"groundRentReviewPeriodInYears": null,
"groundRentPercentageIncrease": null,
"annualServiceCharge": null,
"councilTaxBand": "G",
"domesticRates": null
},
"termsOfUse": "This Application Programming Interface (API) is for the use of Rightmove Group Limited only. No other individual or business may link to, or make use of, this API without the express written permission of Rightmove's Legal Team. Any unauthorised use of this API will constitute a breach of Rightmove's Terms of Use, available at https://www.rightmove.co.uk/this-site/terms-of-use.html, and Rightmove will enforce its rights to the fullest extent permitted by law."
}
]
在上面的 Rightmove scraper 中，我们首先建立了httpx.AsyncClient类似浏览器的标头以避免阻塞。这个对象让我们能够异步检索 HTML 页面。我们在我们的scrape_properties函数中执行此操作，该函数用于asyncio.as_completed收集 HTTP 请求并同时发送它们。然后，我们使用简单的 XPath 解析隐藏 JSON 数据的响应，查找<script>包含变量名称的元素PAGE_MODEL。
JSON解析
输出数据集非常庞大，包含大量我们不一定想要的数据。让我们通过使用将其解析jmespath为更容易理解的内容来修剪它。为此，让我们填写parse_property之前定义的函数：
from typing import TypedDict
class PropertyResult(TypedDict):
"""this is what our result dataset will look like"""
id: str
available: bool
archived: bool
phone: str
bedrooms: int
bathrooms: int
type: str
property_type: str
tags: list
description: str
title: str
subtitle: str
price: str
price_sqft: str
address: dict
latitude: float
longitude: float
features: list
history: dict
photos: list
floorplans: list
agency: dict
industryAffiliations: list
nearest_airports: list
nearest_stations: list
sizings: list
brochures: list
def parse_property(data) -> PropertyResult:
"""parse rightmove cache data for proprety information"""
# here we define field name to JMESPath mapping
parse_map = {
"id": "id",
"available": "status.published",
"archived": "status.archived",
"phone": "contactInfo.telephoneNumbers.localNumber",
"bedrooms": "bedrooms",
"bathrooms": "bathrooms",
"type": "transactionType",
"property_type": "propertySubType",
"tags": "tags",
"description": "text.description",
"title": "text.pageTitle",
"subtitle": "text.propertyPhrase",
"price": "prices.primaryPrice",
"price_sqft": "prices.pricePerSqFt",
"address": "address",
"latitude": "location.latitude",
"longitude": "location.longitude",
"features": "keyFeatures",
"history": "listingHistory",
"photos": "images[*].{url: url, caption: caption}",
"floorplans": "floorplans[*].{url: url, caption: caption}",
"agency": """customer.{
id: branchId,
branch: branchName,
company: companyName,
address: displayAddress,
commercial: commercial,
buildToRent: buildToRent,
isNew: isNewHomeDeveloper
}""",
"industryAffiliations": "industryAffiliations[*].name",
"nearest_airports": "nearestAirports[*].{name: name, distance: distance}",
"nearest_stations": "nearestStations[*].{name: name, distance: distance}",
"sizings": "sizings[*].{unit: unit, min: minimumSize, max: maximumSize}",
"brochures": "brochures",
}
results = {}
for key, path in parse_map.items():
value = jmespath.search(path, data)
results[key] = value
return results
示例输出
{
"id": "129828533",
"available": true,
"archived": false,
"phone": "01326 331658",
"bedrooms": 10,
"bathrooms": 9,
"type": "BUY",
"property_type": "Detached",
"tags": [
"ONLINE_VIEWING"
],
"description": "A rare opportunity to purchase around 6,880 square feet (circa 639 square metres) of prime waterside real estate with mature gardens, private quay and foreshore, located on the water's edge in one of the South West's most exclusive and picturesque south coastal villages.",
"title": "10 bedroom detached house for sale in St Mawes, Cornwall, TR2",
"subtitle": "10 bedroom detached house",
"price": "\u00a36,000,000",
"price_sqft": null,
"address": {
"displayAddress": "St Mawes, Cornwall",
"countryCode": "GB",
"deliveryPointId": null,
"ukCountry": "England",
"outcode": "TR2",
"incode": "5AZ"
},
"latitude": 50.15871,
"longitude": -5.005751,
"features": [
"A Distinctive Detached Marine Residence comprising 6 Bedrooms, 4 Reception and 5 Bathrooms.",
"A Detached One Bedroom Ancillary Waterside Retreat with an Established Holiday Letting Income.",
"A Historic 4 Bedroom, 2 Reception, 4 Bathroom Detached Dwelling, Perched on the Waters Edge. ",
"Includes a Private Quay, Foreshore to Mean Low Water and a Mooring.",
"Magnificent Uninterrupted River, Creek and National Trust Countryside Views.",
"Triple Garage, Ample Boat and Car Parking plus good sized Mature Gardens.",
"Walking Distance of Beaches, Shops, Fine Dining Restaurants, Pubs and Cafe's. ",
"Excellent Sheltered Deep Water Sailing Facilities and Beautiful Creekside Walks on the Doorstep.",
"Exclusive Coastal Village Location.",
"Watch our Promotional Video to Explore this Rare Opportunity and it's Enchanting Location Further."
],
"history": {
"listingUpdateReason": "Reduced on 18/01/2023"
},
"photos": [
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_00_0000.jpeg",
"caption": "Waterside Living"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_01_0000.jpeg",
"caption": "Pendragon"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_02_0000.jpeg",
"caption": "The Studio"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_03_0000.jpeg",
"caption": "Pendragon"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_04_0000.jpeg",
"caption": "Pendragon Lounge"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_05_0000.jpeg",
"caption": "Pendragon Kitchen"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_06_0000.jpeg",
"caption": "Pendragon Annexe"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_07_0000.jpeg",
"caption": "River Views"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_08_0000.jpeg",
"caption": "Garden and Views"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_09_0000.jpeg",
"caption": "Castaway"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_10_0000.jpeg",
"caption": "Castaway Living Room"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_11_0000.jpeg",
"caption": "The Studio"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_12_0000.jpeg",
"caption": "The Studio Living Room"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_13_0000.jpeg",
"caption": "Studio Balcony"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_14_0000.jpeg",
"caption": "Studio Kitchen"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_15_0000.jpeg",
"caption": "Studio Bathroom"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_16_0000.jpeg",
"caption": "The Percuil River"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_IMG_17_0000.jpeg",
"caption": "Location"
}
],
"floorplans": [
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_FLP_00_0000.jpeg",
"caption": "Pendragon and Castaway Floorplans"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_FLP_01_0000.jpeg",
"caption": "The Studio Floorplan"
},
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_FLP_02_0000.jpeg",
"caption": "Boundary Pans"
}
],
"agency": {
"id": 8240,
"branch": "St Mawes",
"company": "H Tiddy",
"address": "The Square,\r\nSt. Mawes,\r\nTR2 5AG",
"commercial": false,
"buildToRent": false,
"isNew": false
},
"industryAffiliations": [
"Association of Residential Letting Agents",
"National Association of Estate Agents",
"Ombudsman for REsale Estate Agents (no OFT logo)"
],
"nearest_airports": [],
"nearest_stations": [
{
"name": "Falmouth Docks Station",
"distance": 2.298093516329164
},
{
"name": "Falmouth Town Station",
"distance": 2.7436587311392673
},
{
"name": "Penmere Station",
"distance": 3.4787958948158195
}
],
"sizings": [],
"brochures": [
{
"url": "https://media.rightmove.co.uk/9k/8240/129828533/8240_11794080_DOC_00_0000.pdf",
"caption": "Property Brochure"
},
{
"url": "https://www.htiddy.co.uk/index.php?option=com_eapow&format=html&task=property.getProp&pid=cb12dc24-725e-483e-8967-4026f59b24d9&aid={85dcf889-5df7-4f92-a690-1b46a0234359}",
"caption": "Full Details"
}
]
}
上面，使用 JMESPath，我们将隐藏的 Web 数据集减少为基本属性数据字段，使其更容易摄取到我们现有的数据管道中！
爬取 Rightmove 搜索
现在我们知道如何爬取 RightMove 属性，让我们看看如何使用 RightMove 的搜索功能找到它们。
如果我们打开开发人员工具并检查网络选项卡，我们可以看到 RightMove 使用 REST API 来获取搜索结果：
当我们点击下一页时，我们可以看到正在发出后端 API 请求
我们可以看到搜索结果是使用 GET 请求从以下 URL 获取的：
https://www.rightmove.co.uk/api/_search? locationIdentifier=REGION%5E61294& numberOfPropertiesPerPage=24& radius=0.0& sortType=6& index=24& includeSSTC=false& viewType=LIST& channel=BUY& areaSizeUnit=sqft& currencyCode=GBP& isFetching=false
但是，为了爬取此搜索，我们需要显示为数字的位置标识符（在本例中为REGION^61294）。我们如何获得这个标识符呢？
再次，我们可以检查一下搜索，我们可以看到另一个 API 端点用于将位置名称解析为标识符：
https://www.rightmove.co.uk/typeAhead/uknostreet/CO/RN/WA/LL/
此搜索端点接受两个字符字形并建议英国位置 ID。我们可以使用此端点将搜索查询解析为位置 ID，然后爬取搜索 API。
让我们使用这两个端点来开发我们的搜索爬取器：
import asyncio
import json
from typing import List, TypedDict
from urllib.parse import urlencode
from httpx import AsyncClient, Response
async def find_locations(query: str) -> List[str]:
"""use rightmove's typeahead api to find location IDs. Returns list of location IDs in most likely order"""
# rightmove uses two character long tokens so "cornwall" becomes "CO/RN/WA/LL"
tokenize_query = "".join(c + ("/" if i % 2 == 0 else "") for i, c in enumerate(query.upper(), start=1))
url = f"https://www.rightmove.co.uk/typeAhead/uknostreet/{tokenize_query.strip('/')}/"
response = await client.get(url)
data = json.loads(response.text)
return [prediction["locationIdentifier"] for prediction in data["typeAheadLocations"]]
async def scrape_search(location_id: str) -> dict:
RESULTS_PER_PAGE = 24
def make_url(offset: int) -> str:
url = "https://www.rightmove.co.uk/api/_search?"
params = {
"areaSizeUnit": "sqft",
"channel": "BUY", # BUY or RENT
"currencyCode": "GBP",
"includeSSTC": "false",
"index": offset, # page offset
"isFetching": "false",
"locationIdentifier": location_id, #e.g.: "REGION^61294",
"numberOfPropertiesPerPage": RESULTS_PER_PAGE,
"radius": "0.0",
"sortType": "6",
"viewType": "LIST",
}
return url + urlencode(params)
first_page = await client.get(make_url(0))
first_page_data = json.loads(first_page.content)
total_results = int(first_page_data['resultCount'].replace(',', ''))
results = first_page_data['properties']
other_pages = []
for offset in range(RESULTS_PER_PAGE, total_results, RESULTS_PER_PAGE):
other_pages.append(client.get(make_url(offset)))
for response in asyncio.as_completed(other_pages):
response = await response
data = json.loads(response.text)
results.extend(data['properties'])
return results
# Example run:
async def run():
cornwall_id = (await find_locations("cornwall"))[0]
print(cornwall_id)
cornwall_results = await scrape_search(cornwall_id)
print(json.dumps(cornwall_results, indent=2))
if __name__ == "__main__":
asyncio.run(run())
示例输出
[
{
"id": 130211456,
"bedrooms": 3,
"bathrooms": 2,
"numberOfImages": 18,
"numberOfFloorplans": 1,
"numberOfVirtualTours": 1,
"summary": "FOR SALE VIA ONLINE AUCTION - ENDING FRIDAY 3RD FEBRUARY AT 2PM! SEE BRADLEYS WEBSITE Imposing double fronted house set in an enviable position on Barbican Hill. The Haven is a fabulous home that enjoys sublime town and harbour views, the property also benefits from off road parking for...",
"displayAddress": "Barbican Hill, Looe, Cornwall",
"countryCode": "GB",
"location": {
"latitude": 50.354827,
"longitude": -4.452984
},
"propertyImages": {
"images": [
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_14_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_14_0000.jpeg",
"caption": "Picture No. 22"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_01_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_01_0000.jpeg",
"caption": "Picture No. 18"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_00_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_00_0000.jpeg",
"caption": "Picture No. 19"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_02_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_02_0000.jpeg",
"caption": "Picture No. 03"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_03_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_03_0000.jpeg",
"caption": "Picture No. 01"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_04_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_04_0000.jpeg",
"caption": "Picture No. 06"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_05_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_05_0000.jpeg",
"caption": "Picture No. 04"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_06_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_06_0000.jpeg",
"caption": "Picture No. 10"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_07_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_07_0000.jpeg",
"caption": "Picture No. 05"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_08_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_08_0000.jpeg",
"caption": "Picture No. 07"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_09_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_09_0000.jpeg",
"caption": "Picture No. 08"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_10_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_10_0000.jpeg",
"caption": "Picture No. 12"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_11_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_11_0000.jpeg",
"caption": "Picture No. 02"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_12_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_12_0000.jpeg",
"caption": "Picture No. 09"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_13_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_13_0000.jpeg",
"caption": "Picture No. 14"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_15_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_15_0000.jpeg",
"caption": "Picture No. 13"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_16_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_16_0000.jpeg",
"caption": "Picture No. 20"
},
{
"srcUrl": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_17_0000_max_476x317.jpeg",
"url": "43k/42980/130211456/42980_LOE220204_IMG_17_0000.jpeg",
"caption": "Picture No. 21"
}
],
"mainImageSrc": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_14_0000_max_476x317.jpeg",
"mainMapImageSrc": "https://media.rightmove.co.uk:443/dir/crop/10:9-16:9/43k/42980/130211456/42980_LOE220204_IMG_14_0000_max_296x197.jpeg"
},
"propertySubType": "Detached",
"listingUpdate": {
"listingUpdateReason": "price_reduced",
"listingUpdateDate": "2022-12-24T00:33:01Z"
},
"premiumListing": false,
"featuredProperty": true,
"price": {
"amount": 285000,
"frequency": "not specified",
"currencyCode": "GBP",
"displayPrices": [
{
"displayPrice": "\u00a3285,000",
"displayPriceQualifier": "Guide Price"
}
]
},
"customer": {
"branchId": 42980,
"brandPlusLogoURI": "/company/clogo_rmchoice_11501_0007.jpeg",
"contactTelephone": "01503 770698",
"branchDisplayName": "Bradleys, Looe",
"branchName": "Looe",
"brandTradingName": "Bradleys",
"branchLandingPageUrl": "/estate-agents/agent/Bradleys/Looe-42980.html",
"development": false,
"showReducedProperties": true,
"commercial": false,
"showOnMap": true,
"enhancedListing": false,
"developmentContent": null,
"buildToRent": false,
"buildToRentBenefits": [],
"brandPlusLogoUrl": "https://media.rightmove.co.uk:443/dir/company/clogo_rmchoice_11501_0007_max_100x50.jpeg"
},
"distance": null,
"transactionType": "buy",
"productLabel": {
"productLabelText": "",
"spotlightLabel": false
},
"commercial": false,
"development": false,
"residential": true,
"students": false,
"auction": true,
"feesApply": false,
"feesApplyText": null,
"displaySize": "",
"showOnMap": true,
"propertyUrl": "/properties/130211456#/?channel=RES_BUY",
"contactUrl": "/property-for-sale/contactBranch.html?propertyId=130211456",
"staticMapUrl": null,
"channel": "BUY",
"firstVisibleDate": "2022-12-24T00:27:13Z",
"keywords": [],
"keywordMatchType": "no_keyword",
"saved": false,
"hidden": false,
"onlineViewingsAvailable": true,
"lozengeModel": {
"matchingLozenges": [
{
"type": "ONLINE_VIEWINGS",
"priority": 6
}
]
},
"hasBrandPlus": true,
"displayStatus": "",
"enquiredTimestamp": null,
"enhancedListing": false,
"propertyTypeFullDescription": "3 bedroom detached house for sale",
"isRecent": false,
"heading": "Featured Property",
"addedOrReduced": "Reduced on 24/12/2022",
"formattedBranchName": " by Bradleys, Looe",
"formattedDistance": ""
},
...
]
通过爬取搜索，我们只需几个请求即可获得许多属性的结果。这比单独爬取每个属性要有效得多，但是单个属性数据更详细并且包含更多信息。
我们还可以使用此搜索爬取工具，只需按列出日期排序并不断检查结果的第一页即可获得新房产列表的通知。
常问问题
为了总结这个 RightMove scraper 教程，我们来看看一些常见问题：
爬取 RightMove 是否合法？
是的，在 RightMove.co.uk 上爬取公开列表是合法的，因为所有数据都是公开的。但是，某些数据（例如房地产图像）可能受版权保护，除非另有说明，否则不应直接爬取。
RightMove.co.uk 是否有公共 Web API？
不，没有可用于从 RightMove 检索公共财产数据的公共 Web API。然而，使用 Python 爬取网站是完全合法的，因此我们可以使用本文所述的网络爬取工具来检索数据。
是否可以爬取 RightMove？
是的，在本教程中我们介绍了受控网络爬取，但爬行也是一种可能。
例如，我们可以创建一个 RightMove 爬行程序，因为每个列表都有“与此属性相似”部分，我们可以按照该部分来爬行 RightMove 属性列表，而无需像本教程中介绍的那样探索搜索。
Rightmove 爬取总结
在本教程中，我们使用一些免费的社区包在 Python 中构建了RightMove爬取工具。我们用来httpx检索网页并parsel解析jmespath数据。
我们首先了解如何通过从 HTML 中可用的 javascript 变量中提取隐藏的 Web 缓存数据来爬取单个属性页。然后，为了解析属性数据，我们使用 JMESPath JSON 解析语言编写了一些简单的规则，将爬取的数据集减少到重要的属性数据字段。
为了查找房产列表并跟踪新的/更新的房产列表，我们使用 Chrome 开发人员工具套件探索了 RightMoves 的搜索系统和搜索 API。
