Fix scraping logic for all pages

Author: amaanjaved1

README.md

@@ -21,29 +21,75 @@ pip install 'ratemyprofessors-client[sentiment]'
 
 ## Quickstart
 
+**Professor by ID** (data from professor page HTML):
+
 ```python
 from rmp_client import RMPClient
 
-SCHOOL_ID = 1466  # example: Queen's University ID on RMP
+with RMPClient() as client:
+    professor = client.get_professor("2823076")  # legacy ID from URL
+    print(professor.name, professor.overall_rating, professor.num_ratings, professor.school.name)
+```
+
+**School by ID** (data from school page HTML):
+
+```python
+with RMPClient() as client:
+    school = client.get_school("1466")
+    print(school.name, school.location, school.overall_quality, school.num_ratings)
+```
+
+**Search professors or schools** (data from search page HTML):
+
+```python
+with RMPClient() as client:
+    profs = client.search_professors("test")
+    print(profs.total, profs.has_next_page)
+    for p in profs.professors[:5]:
+        print(p.name, p.school.name if p.school else "")
+
+    schools = client.search_schools("queens")
+    for s in schools.schools:
+        print(s.name, s.location)
+```
 
+**Compare two schools** (data from compare page HTML):
+
+```python
 with RMPClient() as client:
-    for prof in client.iter_professors_for_school(SCHOOL_ID, page_size=20):
-        print(prof.name, prof.overall_rating, prof.num_ratings)
+    result = client.get_compare_schools("1466", "1491")
+    print(result.school_1.name, result.school_2.name)
 ```
 
-Fetch details and iterate ratings incrementally:
+**Iterate professor ratings** (first page from HTML, further pages via GraphQL):
 
 ```python
 from datetime import date
 from rmp_client import RMPClient
 
 with RMPClient() as client:
-    professor = client.get_professor("PROFESSOR_ID")
-
-    for rating in client.iter_professor_ratings(professor.id, since=date(2024, 1, 1)):
+    for rating in client.iter_professor_ratings("2823076", since=date(2024, 1, 1)):
         print(rating.date, rating.quality, rating.comment)
 ```
 
+**Verify the client** (run the script to hit the live site and print sample data):
+
+```bash
+pip install -e .
+python scripts/verify_client.py              # up to 3 pages of ratings per section (default)
+python scripts/verify_client.py --max-pages 10 --page-size 20  # scrape more pages
+```
+
+**Scrape all ratings** for a professor or school: the client fetches the first page from HTML and subsequent pages via the site’s GraphQL API. Use the iterators to get every rating:
+
+```python
+with RMPClient() as client:
+    for rating in client.iter_professor_ratings("2823076"):
+        print(rating.date, rating.comment)
+    for rating in client.iter_school_ratings("1466"):
+        print(rating.date, rating.comment)
+```
+
 ## How it works
 
 ### Package architecture
@@ -68,43 +114,57 @@ flowchart TB
     end
 
     subgraph External
-        API["RMP GraphQL API\n(ratemyprofessors.com)"]
+        RMP["RMP pages\n(ratemyprofessors.com)"]
     end
 
     User --> Client
     Client --> Config
     Client --> HttpCtx
     HttpCtx --> Http
     Http --> Bucket
-    Http --> API
+    Http --> RMP
     Client --> Models
     Client --> Errors
 ```
 
 ### Request flow
 
+Professor, school, compare-schools, and search endpoints **fetch the relevant RMP page HTML** (GET), extract `window.__RELAY_STORE__` from the response, and parse it into `Professor`, `School`, `Rating`, or search result lists.
+
+**Ratings pagination (Relay):** The first page of professor or school ratings comes from the same HTML (Relay store). The store’s connection includes:
+
+- **`pageInfo.endCursor`** — opaque cursor for “start after this item”
+- **`pageInfo.hasNextPage`** — whether more ratings exist
+
+The client then requests the next page by POSTing to `/graphql` with the same query and variables:
+
+- `id` — Relay node id (base64 of `Teacher-{legacyId}` or `School-{legacyId}`)
+- `first` — page size (e.g. 20)
+- `after` — `pageInfo.endCursor` from the previous response
+
+Loop until `hasNextPage` is false. The cursor is typically base64 for an internal offset (e.g. `YXJyYXljb25uZWN0aW9uOjQ=` decodes to `arrayconnection:4`, meaning “after item 4”). RMP does not rotate or expire these cursors, so you can paginate with plain HTTP requests without a browser. This client sends the **full GraphQL query** in each request; if the site ever required persisted queries (e.g. `doc_id` only), you’d capture the real request from the browser and reuse that format.
+
 ```mermaid
 sequenceDiagram
     participant User
     participant RMPClient
     participant HttpClient
     participant TokenBucket
     participant httpx
-    participant RMP API
+    participant RMP
 
-    User->>RMPClient: e.g. get_professor(id) or iter_professors_for_school(school_id)
-    RMPClient->>RMPClient: Build GraphQL-style payload
-    RMPClient->>HttpClient: post_json(path, payload)
+    User->>RMPClient: e.g. get_professor(id), get_school(id), search_professors(q), get_compare_schools(id1, id2)
+    RMPClient->>HttpClient: get_html(url)
     HttpClient->>TokenBucket: consume()
     TokenBucket-->>HttpClient: (blocks until token available)
-    HttpClient->>httpx: POST base_url, json=payload
-    httpx->>RMP API: HTTPS request
-    RMP API-->>httpx: JSON response
+    HttpClient->>httpx: GET page URL
+    httpx->>RMP: HTTPS request
+    RMP-->>httpx: HTML (with __RELAY_STORE__)
     httpx-->>HttpClient: response
-    HttpClient->>HttpClient: Retry on 5xx / HTTP error
-    HttpClient-->>RMPClient: dict (parsed JSON)
-    RMPClient->>RMPClient: Parse into Professor / Rating / etc.
-    RMPClient-->>User: Professor, Rating, or list
+    HttpClient-->>RMPClient: HTML text
+    RMPClient->>RMPClient: Extract and parse __RELAY_STORE__, resolve refs
+    RMPClient->>RMPClient: Map to Professor / School / Rating / SearchResult
+    RMPClient-->>User: Professor, School, list, or CompareSchoolsResult
 ```
 
 ### Data models
@@ -117,9 +177,9 @@ erDiagram
     School {
         string id
         string name
-        string city
-        string state
-        string country
+        string location
+        float overall_quality
+        int num_ratings
     }
 
     Professor {
@@ -141,9 +201,21 @@ erDiagram
 
     ProfessorSearchResult {
         Professor[] professors
-        int page
-        int page_size
+        int total
         bool has_next_page
+        string next_cursor
+    }
+
+    SchoolSearchResult {
+        School[] schools
+        int total
+        bool has_next_page
+        string next_cursor
+    }
+
+    CompareSchoolsResult {
+        School school_1
+        School school_2
     }
 
     ProfessorRatingsPage {