From 1f22b1ad735141547ae67b55d6da637d63770997 Mon Sep 17 00:00:00 2001 From: SynthLuvr <131367121+SynthLuvr@users.noreply.github.com> Date: Sun, 21 Jun 2026 20:37:17 +0200 Subject: [PATCH 1/2] created nhs-crawler --- .github/workflows/nhs-crawler.yml | 35 +++ nhs-crawler/README.md | 164 ++++++++++++ nhs-crawler/data/acute-pancreatitis/index.md | 103 ++++++++ nhs-crawler/data/adhd-adults/index.md | 168 ++++++++++++ .../data/adhd-children-teenagers/index.md | 175 +++++++++++++ nhs-crawler/data/angina/index.md | 132 ++++++++++ nhs-crawler/data/appendicitis/index.md | 181 +++++++++++++ nhs-crawler/data/arthritis/index.md | 105 ++++++++ nhs-crawler/data/asthma/index.md | 202 +++++++++++++++ nhs-crawler/data/atopic-eczema/index.md | 178 +++++++++++++ nhs-crawler/data/atrial-fibrillation/index.md | 154 +++++++++++ nhs-crawler/data/autism/adult-life.md | 76 ++++++ nhs-crawler/data/autism/assessments.md | 85 ++++++ nhs-crawler/data/autism/help-and-support.md | 48 ++++ nhs-crawler/data/autism/signs-in-adults.md | 54 ++++ nhs-crawler/data/autism/signs-in-children.md | 96 +++++++ nhs-crawler/data/autism/supporting-a-child.md | 90 +++++++ nhs-crawler/data/autism/what-is-autism.md | 19 ++ nhs-crawler/data/back-pain/index.md | 103 ++++++++ nhs-crawler/data/bowel-cancer/causes.md | 36 +++ .../data/bowel-cancer/help-and-support.md | 46 ++++ nhs-crawler/data/bowel-cancer/symptoms.md | 68 +++++ .../data/bowel-cancer/tests-and-next-steps.md | 47 ++++ nhs-crawler/data/bowel-cancer/treatment.md | 82 ++++++ .../data/bowel-cancer/what-is-bowel-cancer.md | 11 + nhs-crawler/data/bronchiectasis/index.md | 159 ++++++++++++ nhs-crawler/data/bronchiolitis/index.md | 105 ++++++++ nhs-crawler/data/burns-and-scalds/index.md | 68 +++++ nhs-crawler/data/bursitis/index.md | 76 ++++++ nhs-crawler/data/cancer/index.md | 218 ++++++++++++++++ nhs-crawler/data/chest-infection/index.md | 108 ++++++++ nhs-crawler/data/chickenpox/index.md | 240 +++++++++++++++++ .../chronic-fatigue-syndrome-cfs/index.md | 61 +++++ .../index.md | 83 ++++++ nhs-crawler/data/coeliac-disease/index.md | 114 ++++++++ nhs-crawler/data/common-cold/index.md | 143 ++++++++++ .../covid-19-symptoms-and-what-to-do.md | 114 ++++++++ ...o-avoid-catching-and-spreading-covid-19.md | 53 ++++ .../data/covid-19/treatments-for-covid-19.md | 116 +++++++++ nhs-crawler/data/crohns-disease/index.md | 196 ++++++++++++++ .../data/deep-vein-thrombosis-dvt/index.md | 132 ++++++++++ nhs-crawler/data/dehydration/index.md | 133 ++++++++++ nhs-crawler/data/dementia/index.md | 33 +++ nhs-crawler/data/dental-abscess/index.md | 88 +++++++ .../data/diabetic-ketoacidosis/index.md | 106 ++++++++ nhs-crawler/data/discoid-eczema/index.md | 89 +++++++ .../index.md | 99 +++++++ .../data/dyslexia-in-children/index.md | 85 ++++++ nhs-crawler/data/ear-infections/index.md | 127 +++++++++ nhs-crawler/data/earwax-build-up/index.md | 80 ++++++ nhs-crawler/data/ectopic-pregnancy/index.md | 120 +++++++++ .../data/ehlers-danlos-syndromes/index.md | 178 +++++++++++++ nhs-crawler/data/encephalitis/index.md | 86 ++++++ nhs-crawler/data/endometriosis/index.md | 149 +++++++++++ nhs-crawler/data/epilepsy/index.md | 219 ++++++++++++++++ .../index.md | 120 +++++++++ nhs-crawler/data/febrile-seizures/index.md | 82 ++++++ .../female-genital-mutilation-fgm/index.md | 216 ++++++++++++++++ nhs-crawler/data/fibroids/index.md | 84 ++++++ nhs-crawler/data/fibromyalgia/index.md | 68 +++++ nhs-crawler/data/flu/index.md | 115 +++++++++ nhs-crawler/data/food-allergy/index.md | 117 +++++++++ nhs-crawler/data/food-intolerance/index.md | 87 +++++++ nhs-crawler/data/food-poisoning/index.md | 129 +++++++++ nhs-crawler/data/frozen-shoulder/index.md | 85 ++++++ .../data/fungal-nail-infection/index.md | 93 +++++++ nhs-crawler/data/gallstones/index.md | 108 ++++++++ nhs-crawler/data/gastritis/index.md | 115 +++++++++ nhs-crawler/data/genital-herpes/index.md | 189 ++++++++++++++ .../data/gestational-diabetes/index.md | 127 +++++++++ nhs-crawler/data/glandular-fever/index.md | 106 ++++++++ nhs-crawler/data/glaucoma/index.md | 146 +++++++++++ nhs-crawler/data/gonorrhoea/index.md | 154 +++++++++++ nhs-crawler/data/gout/index.md | 141 ++++++++++ nhs-crawler/data/gum-disease/index.md | 102 ++++++++ nhs-crawler/data/haemochromatosis/index.md | 87 +++++++ .../data/hand-foot-mouth-disease/index.md | 142 ++++++++++ .../data/head-injury-and-concussion/index.md | 70 +++++ nhs-crawler/data/heart-attack-old/index.md | 133 ++++++++++ nhs-crawler/data/heart-failure/index.md | 85 ++++++ .../data/heartburn-and-acid-reflux/index.md | 113 ++++++++ nhs-crawler/data/hiatus-hernia/index.md | 97 +++++++ nhs-crawler/data/high-blood-pressure/index.md | 139 ++++++++++ nhs-crawler/data/high-cholesterol/index.md | 15 ++ .../idiopathic-pulmonary-fibrosis/index.md | 97 +++++++ nhs-crawler/data/impetigo/index.md | 122 +++++++++ nhs-crawler/data/indigestion/index.md | 92 +++++++ nhs-crawler/data/infertility/index.md | 79 ++++++ .../data/inflammatory-bowel-disease/index.md | 61 +++++ nhs-crawler/data/ingrown-toenail/index.md | 91 +++++++ .../data/insect-bites-and-stings/index.md | 178 +++++++++++++ nhs-crawler/data/insomnia/index.md | 124 +++++++++ .../data/intracranial-hypertension/index.md | 103 ++++++++ .../data/iron-deficiency-anaemia/index.md | 124 +++++++++ .../diet-lifestyle-and-medicines.md | 73 ++++++ .../further-help-and-support.md | 52 ++++ .../getting-diagnosed.md | 38 +++ .../irritable-bowel-syndrome-ibs/symptoms.md | 54 ++++ .../what-is-ibs.md | 16 ++ .../data/japanese-encephalitis/index.md | 100 +++++++ nhs-crawler/data/jaundice-newborn/index.md | 95 +++++++ nhs-crawler/data/jaundice/index.md | 48 ++++ .../index.md | 99 +++++++ nhs-crawler/data/jet-lag/index.md | 65 +++++ .../joint-hypermobility-syndrome/index.md | 96 +++++++ nhs-crawler/data/kawasaki-disease/index.md | 81 ++++++ nhs-crawler/data/keloid-scars/index.md | 68 +++++ nhs-crawler/data/keratosis-pilaris/index.md | 78 ++++++ nhs-crawler/data/kidney-cancer/causes.md | 32 +++ .../data/kidney-cancer/help-and-support.md | 57 ++++ nhs-crawler/data/kidney-cancer/symptoms.md | 55 ++++ .../kidney-cancer/tests-and-next-steps.md | 56 ++++ nhs-crawler/data/kidney-cancer/treatment.md | 105 ++++++++ .../kidney-cancer/what-is-kidney-cancer.md | 10 + nhs-crawler/data/kidney-disease/index.md | 91 +++++++ nhs-crawler/data/kidney-infection/index.md | 109 ++++++++ nhs-crawler/data/kidney-stones/index.md | 81 ++++++ .../data/klinefelters-syndrome/index.md | 123 +++++++++ nhs-crawler/data/knock-knees/index.md | 57 ++++ nhs-crawler/data/kyphosis/index.md | 71 +++++ nhs-crawler/data/labyrinthitis/index.md | 84 ++++++ nhs-crawler/data/lactose-intolerance/index.md | 106 ++++++++ nhs-crawler/data/laryngitis/index.md | 113 ++++++++ nhs-crawler/data/leg-cramps/index.md | 77 ++++++ nhs-crawler/data/liver-disease/index.md | 79 ++++++ .../low-blood-pressure-hypotension/index.md | 89 +++++++ .../low-blood-sugar-hypoglycaemia/index.md | 119 +++++++++ .../data/low-white-blood-cell-count/index.md | 76 ++++++ nhs-crawler/data/lung-cancer/index.md | 55 ++++ nhs-crawler/data/lupus/index.md | 125 +++++++++ nhs-crawler/data/lyme-disease/index.md | 159 ++++++++++++ nhs-crawler/data/lymphoedema/index.md | 112 ++++++++ nhs-crawler/data/measles/index.md | 155 +++++++++++ nhs-crawler/data/menieres-disease/index.md | 97 +++++++ nhs-crawler/data/meningitis/index.md | 159 ++++++++++++ nhs-crawler/data/migraine/index.md | 156 +++++++++++ nhs-crawler/data/miscarriage/index.md | 173 +++++++++++++ .../data/molluscum-contagiosum/index.md | 75 ++++++ nhs-crawler/data/mouth-ulcers/index.md | 119 +++++++++ nhs-crawler/data/multiple-sclerosis/index.md | 168 ++++++++++++ nhs-crawler/data/narcolepsy/index.md | 90 +++++++ nhs-crawler/data/nasal-polyps/index.md | 80 ++++++ nhs-crawler/data/night-terrors/index.md | 86 ++++++ .../index.md | 115 +++++++++ .../data/non-allergic-rhinitis/index.md | 73 ++++++ .../data/non-hodgkin-lymphoma/causes.md | 17 ++ .../non-hodgkin-lymphoma/help-and-support.md | 56 ++++ .../data/non-hodgkin-lymphoma/symptoms.md | 38 +++ .../tests-and-next-steps.md | 52 ++++ .../data/non-hodgkin-lymphoma/treatment.md | 118 +++++++++ .../data/non-hodgkin-lymphoma/what-is.md | 15 ++ nhs-crawler/data/norovirus/index.md | 132 ++++++++++ nhs-crawler/data/nosebleed/index.md | 90 +++++++ nhs-crawler/data/obesity-old/index.md | 205 +++++++++++++++ nhs-crawler/data/oesophageal-cancer/causes.md | 38 +++ .../oesophageal-cancer/help-and-support.md | 57 ++++ .../data/oesophageal-cancer/symptoms.md | 63 +++++ .../tests-and-next-steps.md | 68 +++++ .../data/oesophageal-cancer/treatment.md | 86 ++++++ .../what-is-oesophageal-cancer.md | 9 + .../data/oral-thrush-mouth-thrush/index.md | 81 ++++++ nhs-crawler/data/osteoarthritis/index.md | 112 ++++++++ nhs-crawler/data/osteomyelitis/index.md | 87 +++++++ nhs-crawler/data/osteophyte/index.md | 41 +++ nhs-crawler/data/osteoporosis/index.md | 124 +++++++++ nhs-crawler/data/ovarian-cancer/index.md | 21 ++ nhs-crawler/data/ovarian-cyst/index.md | 108 ++++++++ .../index.md | 82 ++++++ nhs-crawler/data/parkinsons-disease/index.md | 103 ++++++++ .../data/peripheral-neuropathy/index.md | 111 ++++++++ nhs-crawler/data/piles-haemorrhoids/index.md | 135 ++++++++++ nhs-crawler/data/plantar-fasciitis/index.md | 96 +++++++ nhs-crawler/data/pneumonia/index.md | 105 ++++++++ .../polycystic-ovary-syndrome-pcos/index.md | 71 +++++ .../postural-tachycardia-syndrome/index.md | 121 +++++++++ nhs-crawler/data/pre-eclampsia/index.md | 145 +++++++++++ nhs-crawler/data/prostate-cancer/causes.md | 22 ++ .../data/prostate-cancer/help-and-support.md | 46 ++++ nhs-crawler/data/prostate-cancer/symptoms.md | 64 +++++ .../prostate-cancer/tests-and-next-steps.md | 40 +++ nhs-crawler/data/prostate-cancer/treatment.md | 106 ++++++++ .../what-is-prostate-cancer.md | 12 + nhs-crawler/data/q-fever/index.md | 97 +++++++ nhs-crawler/data/rabies/index.md | 97 +++++++ nhs-crawler/data/raynauds/index.md | 87 +++++++ nhs-crawler/data/reflux-in-babies/index.md | 79 ++++++ .../respiratory-syncytial-virus-rsv/index.md | 128 +++++++++ .../data/respiratory-tract-infection/index.md | 136 ++++++++++ .../data/restless-legs-syndrome/index.md | 84 ++++++ nhs-crawler/data/retinal-migraine/index.md | 97 +++++++ .../data/rheumatoid-arthritis/index.md | 75 ++++++ nhs-crawler/data/ringworm/index.md | 86 ++++++ nhs-crawler/data/rosacea/index.md | 111 ++++++++ nhs-crawler/data/scabies/index.md | 106 ++++++++ nhs-crawler/data/scarlet-fever/index.md | 130 ++++++++++ nhs-crawler/data/sciatica/index.md | 120 +++++++++ nhs-crawler/data/sepsis/index.md | 184 +++++++++++++ nhs-crawler/data/shingles/index.md | 165 ++++++++++++ .../data/sinusitis-sinus-infection/index.md | 116 +++++++++ .../data/slapped-cheek-syndrome/index.md | 95 +++++++ nhs-crawler/data/sleep-apnoea/index.md | 148 +++++++++++ nhs-crawler/data/stomach-ulcer/index.md | 124 +++++++++ nhs-crawler/data/stroke/causes.md | 48 ++++ nhs-crawler/data/stroke/diagnosis.md | 23 ++ nhs-crawler/data/stroke/help-and-support.md | 21 ++ nhs-crawler/data/stroke/recovery.md | 46 ++++ nhs-crawler/data/stroke/symptoms.md | 38 +++ nhs-crawler/data/stroke/treatment.md | 25 ++ .../temporomandibular-disorder-tmd/index.md | 89 +++++++ nhs-crawler/data/tendonitis/index.md | 96 +++++++ nhs-crawler/data/threadworms/index.md | 88 +++++++ .../data/thrush-in-men-and-women/index.md | 137 ++++++++++ nhs-crawler/data/tinnitus/index.md | 98 +++++++ nhs-crawler/data/tonsillitis/index.md | 139 ++++++++++ .../transient-ischaemic-attack-tia/index.md | 88 +++++++ .../data/trigeminal-neuralgia/index.md | 76 ++++++ .../data/type-1-diabetes/complications.md | 102 ++++++++ .../data/type-1-diabetes/living-with.md | 136 ++++++++++ nhs-crawler/data/type-1-diabetes/support.md | 50 ++++ nhs-crawler/data/type-1-diabetes/symptoms.md | 78 ++++++ nhs-crawler/data/type-1-diabetes/treatment.md | 100 +++++++ .../what-is-type-1-diabetes.md | 11 + .../data/type-2-diabetes/complications.md | 52 ++++ nhs-crawler/data/type-2-diabetes/support.md | 27 ++ nhs-crawler/data/type-2-diabetes/symptoms.md | 66 +++++ nhs-crawler/data/type-2-diabetes/treatment.md | 112 ++++++++ .../what-is-type-2-diabetes.md | 10 + nhs-crawler/data/ulcerative-colitis/index.md | 130 ++++++++++ .../index.md | 118 +++++++++ .../data/undescended-testicles/index.md | 47 ++++ nhs-crawler/data/urethritis/index.md | 85 ++++++ .../data/urinary-incontinence/index.md | 124 +++++++++ .../urinary-tract-infections-utis/index.md | 176 +++++++++++++ nhs-crawler/data/uveitis/index.md | 79 ++++++ nhs-crawler/data/vaginismus/index.md | 84 ++++++ nhs-crawler/data/vaginitis/index.md | 94 +++++++ nhs-crawler/data/varicose-eczema/index.md | 104 ++++++++ nhs-crawler/data/varicose-veins/index.md | 164 ++++++++++++ nhs-crawler/data/vascular-dementia/index.md | 127 +++++++++ nhs-crawler/data/vasculitis/index.md | 244 ++++++++++++++++++ nhs-crawler/data/vertigo/index.md | 107 ++++++++ .../index.md | 85 ++++++ nhs-crawler/data/vitiligo/index.md | 174 +++++++++++++ nhs-crawler/data/vulval-cancer/causes.md | 50 ++++ .../data/vulval-cancer/help-and-support.md | 49 ++++ nhs-crawler/data/vulval-cancer/symptoms.md | 58 +++++ .../vulval-cancer/tests-and-next-steps.md | 67 +++++ nhs-crawler/data/vulval-cancer/treatment.md | 84 ++++++ .../vulval-cancer/what-is-vulval-cancer.md | 12 + nhs-crawler/data/warts-and-verrucas/index.md | 98 +++++++ nhs-crawler/data/whiplash/index.md | 79 ++++++ nhs-crawler/data/whooping-cough/index.md | 115 +++++++++ .../wolff-parkinson-white-syndrome/index.md | 102 ++++++++ nhs-crawler/data/womb-cancer/index.md | 18 ++ nhs-crawler/data/yellow-fever/index.md | 97 +++++++ nhs-crawler/data/zika/index.md | 125 +++++++++ nhs-crawler/pyproject.toml | 44 ++++ nhs-crawler/src/__init__.py | 5 + nhs-crawler/src/cli.py | 68 +++++ nhs-crawler/src/constants.py | 23 ++ nhs-crawler/src/converter.py | 104 ++++++++ nhs-crawler/src/crawler.py | 184 +++++++++++++ nhs-crawler/src/http_client.py | 31 +++ nhs-crawler/src/logging_config.py | 35 +++ nhs-crawler/src/pipeline.py | 131 ++++++++++ nhs-crawler/src/utils.py | 74 ++++++ 266 files changed, 25415 insertions(+) create mode 100644 .github/workflows/nhs-crawler.yml create mode 100644 nhs-crawler/README.md create mode 100644 nhs-crawler/data/acute-pancreatitis/index.md create mode 100644 nhs-crawler/data/adhd-adults/index.md create mode 100644 nhs-crawler/data/adhd-children-teenagers/index.md create mode 100644 nhs-crawler/data/angina/index.md create mode 100644 nhs-crawler/data/appendicitis/index.md create mode 100644 nhs-crawler/data/arthritis/index.md create mode 100644 nhs-crawler/data/asthma/index.md create mode 100644 nhs-crawler/data/atopic-eczema/index.md create mode 100644 nhs-crawler/data/atrial-fibrillation/index.md create mode 100644 nhs-crawler/data/autism/adult-life.md create mode 100644 nhs-crawler/data/autism/assessments.md create mode 100644 nhs-crawler/data/autism/help-and-support.md create mode 100644 nhs-crawler/data/autism/signs-in-adults.md create mode 100644 nhs-crawler/data/autism/signs-in-children.md create mode 100644 nhs-crawler/data/autism/supporting-a-child.md create mode 100644 nhs-crawler/data/autism/what-is-autism.md create mode 100644 nhs-crawler/data/back-pain/index.md create mode 100644 nhs-crawler/data/bowel-cancer/causes.md create mode 100644 nhs-crawler/data/bowel-cancer/help-and-support.md create mode 100644 nhs-crawler/data/bowel-cancer/symptoms.md create mode 100644 nhs-crawler/data/bowel-cancer/tests-and-next-steps.md create mode 100644 nhs-crawler/data/bowel-cancer/treatment.md create mode 100644 nhs-crawler/data/bowel-cancer/what-is-bowel-cancer.md create mode 100644 nhs-crawler/data/bronchiectasis/index.md create mode 100644 nhs-crawler/data/bronchiolitis/index.md create mode 100644 nhs-crawler/data/burns-and-scalds/index.md create mode 100644 nhs-crawler/data/bursitis/index.md create mode 100644 nhs-crawler/data/cancer/index.md create mode 100644 nhs-crawler/data/chest-infection/index.md create mode 100644 nhs-crawler/data/chickenpox/index.md create mode 100644 nhs-crawler/data/chronic-fatigue-syndrome-cfs/index.md create mode 100644 nhs-crawler/data/chronic-obstructive-pulmonary-disease-copd/index.md create mode 100644 nhs-crawler/data/coeliac-disease/index.md create mode 100644 nhs-crawler/data/common-cold/index.md create mode 100644 nhs-crawler/data/covid-19/covid-19-symptoms-and-what-to-do.md create mode 100644 nhs-crawler/data/covid-19/how-to-avoid-catching-and-spreading-covid-19.md create mode 100644 nhs-crawler/data/covid-19/treatments-for-covid-19.md create mode 100644 nhs-crawler/data/crohns-disease/index.md create mode 100644 nhs-crawler/data/deep-vein-thrombosis-dvt/index.md create mode 100644 nhs-crawler/data/dehydration/index.md create mode 100644 nhs-crawler/data/dementia/index.md create mode 100644 nhs-crawler/data/dental-abscess/index.md create mode 100644 nhs-crawler/data/diabetic-ketoacidosis/index.md create mode 100644 nhs-crawler/data/discoid-eczema/index.md create mode 100644 nhs-crawler/data/diverticular-disease-and-diverticulitis/index.md create mode 100644 nhs-crawler/data/dyslexia-in-children/index.md create mode 100644 nhs-crawler/data/ear-infections/index.md create mode 100644 nhs-crawler/data/earwax-build-up/index.md create mode 100644 nhs-crawler/data/ectopic-pregnancy/index.md create mode 100644 nhs-crawler/data/ehlers-danlos-syndromes/index.md create mode 100644 nhs-crawler/data/encephalitis/index.md create mode 100644 nhs-crawler/data/endometriosis/index.md create mode 100644 nhs-crawler/data/epilepsy/index.md create mode 100644 nhs-crawler/data/erection-problems-erectile-dysfunction/index.md create mode 100644 nhs-crawler/data/febrile-seizures/index.md create mode 100644 nhs-crawler/data/female-genital-mutilation-fgm/index.md create mode 100644 nhs-crawler/data/fibroids/index.md create mode 100644 nhs-crawler/data/fibromyalgia/index.md create mode 100644 nhs-crawler/data/flu/index.md create mode 100644 nhs-crawler/data/food-allergy/index.md create mode 100644 nhs-crawler/data/food-intolerance/index.md create mode 100644 nhs-crawler/data/food-poisoning/index.md create mode 100644 nhs-crawler/data/frozen-shoulder/index.md create mode 100644 nhs-crawler/data/fungal-nail-infection/index.md create mode 100644 nhs-crawler/data/gallstones/index.md create mode 100644 nhs-crawler/data/gastritis/index.md create mode 100644 nhs-crawler/data/genital-herpes/index.md create mode 100644 nhs-crawler/data/gestational-diabetes/index.md create mode 100644 nhs-crawler/data/glandular-fever/index.md create mode 100644 nhs-crawler/data/glaucoma/index.md create mode 100644 nhs-crawler/data/gonorrhoea/index.md create mode 100644 nhs-crawler/data/gout/index.md create mode 100644 nhs-crawler/data/gum-disease/index.md create mode 100644 nhs-crawler/data/haemochromatosis/index.md create mode 100644 nhs-crawler/data/hand-foot-mouth-disease/index.md create mode 100644 nhs-crawler/data/head-injury-and-concussion/index.md create mode 100644 nhs-crawler/data/heart-attack-old/index.md create mode 100644 nhs-crawler/data/heart-failure/index.md create mode 100644 nhs-crawler/data/heartburn-and-acid-reflux/index.md create mode 100644 nhs-crawler/data/hiatus-hernia/index.md create mode 100644 nhs-crawler/data/high-blood-pressure/index.md create mode 100644 nhs-crawler/data/high-cholesterol/index.md create mode 100644 nhs-crawler/data/idiopathic-pulmonary-fibrosis/index.md create mode 100644 nhs-crawler/data/impetigo/index.md create mode 100644 nhs-crawler/data/indigestion/index.md create mode 100644 nhs-crawler/data/infertility/index.md create mode 100644 nhs-crawler/data/inflammatory-bowel-disease/index.md create mode 100644 nhs-crawler/data/ingrown-toenail/index.md create mode 100644 nhs-crawler/data/insect-bites-and-stings/index.md create mode 100644 nhs-crawler/data/insomnia/index.md create mode 100644 nhs-crawler/data/intracranial-hypertension/index.md create mode 100644 nhs-crawler/data/iron-deficiency-anaemia/index.md create mode 100644 nhs-crawler/data/irritable-bowel-syndrome-ibs/diet-lifestyle-and-medicines.md create mode 100644 nhs-crawler/data/irritable-bowel-syndrome-ibs/further-help-and-support.md create mode 100644 nhs-crawler/data/irritable-bowel-syndrome-ibs/getting-diagnosed.md create mode 100644 nhs-crawler/data/irritable-bowel-syndrome-ibs/symptoms.md create mode 100644 nhs-crawler/data/irritable-bowel-syndrome-ibs/what-is-ibs.md create mode 100644 nhs-crawler/data/japanese-encephalitis/index.md create mode 100644 nhs-crawler/data/jaundice-newborn/index.md create mode 100644 nhs-crawler/data/jaundice/index.md create mode 100644 nhs-crawler/data/jellyfish-and-other-sea-creature-stings/index.md create mode 100644 nhs-crawler/data/jet-lag/index.md create mode 100644 nhs-crawler/data/joint-hypermobility-syndrome/index.md create mode 100644 nhs-crawler/data/kawasaki-disease/index.md create mode 100644 nhs-crawler/data/keloid-scars/index.md create mode 100644 nhs-crawler/data/keratosis-pilaris/index.md create mode 100644 nhs-crawler/data/kidney-cancer/causes.md create mode 100644 nhs-crawler/data/kidney-cancer/help-and-support.md create mode 100644 nhs-crawler/data/kidney-cancer/symptoms.md create mode 100644 nhs-crawler/data/kidney-cancer/tests-and-next-steps.md create mode 100644 nhs-crawler/data/kidney-cancer/treatment.md create mode 100644 nhs-crawler/data/kidney-cancer/what-is-kidney-cancer.md create mode 100644 nhs-crawler/data/kidney-disease/index.md create mode 100644 nhs-crawler/data/kidney-infection/index.md create mode 100644 nhs-crawler/data/kidney-stones/index.md create mode 100644 nhs-crawler/data/klinefelters-syndrome/index.md create mode 100644 nhs-crawler/data/knock-knees/index.md create mode 100644 nhs-crawler/data/kyphosis/index.md create mode 100644 nhs-crawler/data/labyrinthitis/index.md create mode 100644 nhs-crawler/data/lactose-intolerance/index.md create mode 100644 nhs-crawler/data/laryngitis/index.md create mode 100644 nhs-crawler/data/leg-cramps/index.md create mode 100644 nhs-crawler/data/liver-disease/index.md create mode 100644 nhs-crawler/data/low-blood-pressure-hypotension/index.md create mode 100644 nhs-crawler/data/low-blood-sugar-hypoglycaemia/index.md create mode 100644 nhs-crawler/data/low-white-blood-cell-count/index.md create mode 100644 nhs-crawler/data/lung-cancer/index.md create mode 100644 nhs-crawler/data/lupus/index.md create mode 100644 nhs-crawler/data/lyme-disease/index.md create mode 100644 nhs-crawler/data/lymphoedema/index.md create mode 100644 nhs-crawler/data/measles/index.md create mode 100644 nhs-crawler/data/menieres-disease/index.md create mode 100644 nhs-crawler/data/meningitis/index.md create mode 100644 nhs-crawler/data/migraine/index.md create mode 100644 nhs-crawler/data/miscarriage/index.md create mode 100644 nhs-crawler/data/molluscum-contagiosum/index.md create mode 100644 nhs-crawler/data/mouth-ulcers/index.md create mode 100644 nhs-crawler/data/multiple-sclerosis/index.md create mode 100644 nhs-crawler/data/narcolepsy/index.md create mode 100644 nhs-crawler/data/nasal-polyps/index.md create mode 100644 nhs-crawler/data/night-terrors/index.md create mode 100644 nhs-crawler/data/non-alcoholic-fatty-liver-disease/index.md create mode 100644 nhs-crawler/data/non-allergic-rhinitis/index.md create mode 100644 nhs-crawler/data/non-hodgkin-lymphoma/causes.md create mode 100644 nhs-crawler/data/non-hodgkin-lymphoma/help-and-support.md create mode 100644 nhs-crawler/data/non-hodgkin-lymphoma/symptoms.md create mode 100644 nhs-crawler/data/non-hodgkin-lymphoma/tests-and-next-steps.md create mode 100644 nhs-crawler/data/non-hodgkin-lymphoma/treatment.md create mode 100644 nhs-crawler/data/non-hodgkin-lymphoma/what-is.md create mode 100644 nhs-crawler/data/norovirus/index.md create mode 100644 nhs-crawler/data/nosebleed/index.md create mode 100644 nhs-crawler/data/obesity-old/index.md create mode 100644 nhs-crawler/data/oesophageal-cancer/causes.md create mode 100644 nhs-crawler/data/oesophageal-cancer/help-and-support.md create mode 100644 nhs-crawler/data/oesophageal-cancer/symptoms.md create mode 100644 nhs-crawler/data/oesophageal-cancer/tests-and-next-steps.md create mode 100644 nhs-crawler/data/oesophageal-cancer/treatment.md create mode 100644 nhs-crawler/data/oesophageal-cancer/what-is-oesophageal-cancer.md create mode 100644 nhs-crawler/data/oral-thrush-mouth-thrush/index.md create mode 100644 nhs-crawler/data/osteoarthritis/index.md create mode 100644 nhs-crawler/data/osteomyelitis/index.md create mode 100644 nhs-crawler/data/osteophyte/index.md create mode 100644 nhs-crawler/data/osteoporosis/index.md create mode 100644 nhs-crawler/data/ovarian-cancer/index.md create mode 100644 nhs-crawler/data/ovarian-cyst/index.md create mode 100644 nhs-crawler/data/overactive-thyroid-hyperthyroidism/index.md create mode 100644 nhs-crawler/data/parkinsons-disease/index.md create mode 100644 nhs-crawler/data/peripheral-neuropathy/index.md create mode 100644 nhs-crawler/data/piles-haemorrhoids/index.md create mode 100644 nhs-crawler/data/plantar-fasciitis/index.md create mode 100644 nhs-crawler/data/pneumonia/index.md create mode 100644 nhs-crawler/data/polycystic-ovary-syndrome-pcos/index.md create mode 100644 nhs-crawler/data/postural-tachycardia-syndrome/index.md create mode 100644 nhs-crawler/data/pre-eclampsia/index.md create mode 100644 nhs-crawler/data/prostate-cancer/causes.md create mode 100644 nhs-crawler/data/prostate-cancer/help-and-support.md create mode 100644 nhs-crawler/data/prostate-cancer/symptoms.md create mode 100644 nhs-crawler/data/prostate-cancer/tests-and-next-steps.md create mode 100644 nhs-crawler/data/prostate-cancer/treatment.md create mode 100644 nhs-crawler/data/prostate-cancer/what-is-prostate-cancer.md create mode 100644 nhs-crawler/data/q-fever/index.md create mode 100644 nhs-crawler/data/rabies/index.md create mode 100644 nhs-crawler/data/raynauds/index.md create mode 100644 nhs-crawler/data/reflux-in-babies/index.md create mode 100644 nhs-crawler/data/respiratory-syncytial-virus-rsv/index.md create mode 100644 nhs-crawler/data/respiratory-tract-infection/index.md create mode 100644 nhs-crawler/data/restless-legs-syndrome/index.md create mode 100644 nhs-crawler/data/retinal-migraine/index.md create mode 100644 nhs-crawler/data/rheumatoid-arthritis/index.md create mode 100644 nhs-crawler/data/ringworm/index.md create mode 100644 nhs-crawler/data/rosacea/index.md create mode 100644 nhs-crawler/data/scabies/index.md create mode 100644 nhs-crawler/data/scarlet-fever/index.md create mode 100644 nhs-crawler/data/sciatica/index.md create mode 100644 nhs-crawler/data/sepsis/index.md create mode 100644 nhs-crawler/data/shingles/index.md create mode 100644 nhs-crawler/data/sinusitis-sinus-infection/index.md create mode 100644 nhs-crawler/data/slapped-cheek-syndrome/index.md create mode 100644 nhs-crawler/data/sleep-apnoea/index.md create mode 100644 nhs-crawler/data/stomach-ulcer/index.md create mode 100644 nhs-crawler/data/stroke/causes.md create mode 100644 nhs-crawler/data/stroke/diagnosis.md create mode 100644 nhs-crawler/data/stroke/help-and-support.md create mode 100644 nhs-crawler/data/stroke/recovery.md create mode 100644 nhs-crawler/data/stroke/symptoms.md create mode 100644 nhs-crawler/data/stroke/treatment.md create mode 100644 nhs-crawler/data/temporomandibular-disorder-tmd/index.md create mode 100644 nhs-crawler/data/tendonitis/index.md create mode 100644 nhs-crawler/data/threadworms/index.md create mode 100644 nhs-crawler/data/thrush-in-men-and-women/index.md create mode 100644 nhs-crawler/data/tinnitus/index.md create mode 100644 nhs-crawler/data/tonsillitis/index.md create mode 100644 nhs-crawler/data/transient-ischaemic-attack-tia/index.md create mode 100644 nhs-crawler/data/trigeminal-neuralgia/index.md create mode 100644 nhs-crawler/data/type-1-diabetes/complications.md create mode 100644 nhs-crawler/data/type-1-diabetes/living-with.md create mode 100644 nhs-crawler/data/type-1-diabetes/support.md create mode 100644 nhs-crawler/data/type-1-diabetes/symptoms.md create mode 100644 nhs-crawler/data/type-1-diabetes/treatment.md create mode 100644 nhs-crawler/data/type-1-diabetes/what-is-type-1-diabetes.md create mode 100644 nhs-crawler/data/type-2-diabetes/complications.md create mode 100644 nhs-crawler/data/type-2-diabetes/support.md create mode 100644 nhs-crawler/data/type-2-diabetes/symptoms.md create mode 100644 nhs-crawler/data/type-2-diabetes/treatment.md create mode 100644 nhs-crawler/data/type-2-diabetes/what-is-type-2-diabetes.md create mode 100644 nhs-crawler/data/ulcerative-colitis/index.md create mode 100644 nhs-crawler/data/underactive-thyroid-hypothyroidism/index.md create mode 100644 nhs-crawler/data/undescended-testicles/index.md create mode 100644 nhs-crawler/data/urethritis/index.md create mode 100644 nhs-crawler/data/urinary-incontinence/index.md create mode 100644 nhs-crawler/data/urinary-tract-infections-utis/index.md create mode 100644 nhs-crawler/data/uveitis/index.md create mode 100644 nhs-crawler/data/vaginismus/index.md create mode 100644 nhs-crawler/data/vaginitis/index.md create mode 100644 nhs-crawler/data/varicose-eczema/index.md create mode 100644 nhs-crawler/data/varicose-veins/index.md create mode 100644 nhs-crawler/data/vascular-dementia/index.md create mode 100644 nhs-crawler/data/vasculitis/index.md create mode 100644 nhs-crawler/data/vertigo/index.md create mode 100644 nhs-crawler/data/vitamin-b12-or-folate-deficiency-anaemia/index.md create mode 100644 nhs-crawler/data/vitiligo/index.md create mode 100644 nhs-crawler/data/vulval-cancer/causes.md create mode 100644 nhs-crawler/data/vulval-cancer/help-and-support.md create mode 100644 nhs-crawler/data/vulval-cancer/symptoms.md create mode 100644 nhs-crawler/data/vulval-cancer/tests-and-next-steps.md create mode 100644 nhs-crawler/data/vulval-cancer/treatment.md create mode 100644 nhs-crawler/data/vulval-cancer/what-is-vulval-cancer.md create mode 100644 nhs-crawler/data/warts-and-verrucas/index.md create mode 100644 nhs-crawler/data/whiplash/index.md create mode 100644 nhs-crawler/data/whooping-cough/index.md create mode 100644 nhs-crawler/data/wolff-parkinson-white-syndrome/index.md create mode 100644 nhs-crawler/data/womb-cancer/index.md create mode 100644 nhs-crawler/data/yellow-fever/index.md create mode 100644 nhs-crawler/data/zika/index.md create mode 100644 nhs-crawler/pyproject.toml create mode 100644 nhs-crawler/src/__init__.py create mode 100644 nhs-crawler/src/cli.py create mode 100644 nhs-crawler/src/constants.py create mode 100644 nhs-crawler/src/converter.py create mode 100644 nhs-crawler/src/crawler.py create mode 100644 nhs-crawler/src/http_client.py create mode 100644 nhs-crawler/src/logging_config.py create mode 100644 nhs-crawler/src/pipeline.py create mode 100644 nhs-crawler/src/utils.py diff --git a/.github/workflows/nhs-crawler.yml b/.github/workflows/nhs-crawler.yml new file mode 100644 index 0000000..2ba8468 --- /dev/null +++ b/.github/workflows/nhs-crawler.yml @@ -0,0 +1,35 @@ +name: nhs-crawler + +on: + pull_request: + paths: + - "nhs-crawler/**" + - ".github/workflows/nhs-crawler.yml" + +jobs: + nhs-crawler: + name: nhs-crawler + runs-on: ubuntu-latest + defaults: + run: + working-directory: nhs-crawler + steps: + - uses: actions/checkout@v7.0.0 + + - name: Install uv + uses: astral-sh/setup-uv@v8.2.0 + + - name: Set up Python + run: uv python install 3.11 + + - name: Install dependencies + run: uv sync --frozen --extra dev + + - name: Lint (ruff) + run: uv run ruff check . + + - name: Format check (ruff) + run: uv run ruff format --check . + + - name: Type check (pyright) + run: uv run pyright diff --git a/nhs-crawler/README.md b/nhs-crawler/README.md new file mode 100644 index 0000000..f44ecaf --- /dev/null +++ b/nhs-crawler/README.md @@ -0,0 +1,164 @@ +# NHS Conditions Scraper + +A Python scraper for all [NHS Health A to Z condition +pages](https://www.nhs.uk/health-a-to-z/conditions/). It discovers every +condition listed on the index (198 conditions), crawls all subpages for +each condition (254 pages total), and saves the content as clean +Markdown to `data/`. + +## Prerequisites + +- **[uv](https://docs.astral.sh/uv/)** — install with: + + ``` bash + curl -LsSf https://astral.sh/uv/install.sh | sh + ``` + +- **Python ≥ 3.11** — uv manages this automatically. + +## Quick Start + +``` bash +cd nhs-crawler +uv sync + +# Scrape ALL conditions (198 conditions, 254 pages) +uv run nhs-crawler + +# Scrape a single condition (discovers subpages automatically) +uv run nhs-crawler --url https://www.nhs.uk/conditions/type-2-diabetes/ + +# Scrape a single specific page +uv run nhs-crawler --page https://www.nhs.uk/conditions/type-2-diabetes/treatment/ + +# Show all options +uv run nhs-crawler --help +``` + +## Output + +Markdown files are saved to `data//.md`: + + data/ + ├── asthma/ + │ └── index.md # Inline hub page — no subpages + ├── type-2-diabetes/ + │ ├── what-is-type-2-diabetes.md + │ ├── symptoms.md + │ ├── treatment.md + │ ├── complications.md + │ └── support.md + ├── covid-19/ + │ ├── covid-19-symptoms-and-what-to-do.md + │ ├── how-to-avoid-catching-and-spreading-covid-19.md + │ └── treatments-for-covid-19.md + └── ... # 198 conditions, 254 Markdown files + +Each file starts with a title and source URL header: + +``` markdown +# Asthma + +> Source: https://www.nhs.uk/conditions/asthma/ + +Asthma is a common condition that affects your breathing... +``` + +### Reports + +After a full crawl, two files are written to `reports/`: + +- **`condition_urls.json`** — ordered list of all discovered condition + hub URLs. +- **`summary.json`** — aggregate statistics for the run: + +``` json +{ + "run_at": "2026-06-20T18:38:08.937007+00:00", + "index_url": "https://www.nhs.uk/health-a-to-z/conditions/", + "conditions_discovered": 198, + "pages_discovered": 254, + "pages_scraped": 254, + "error_count": 0, + "total_markdown_chars": 1136966, + "empty_pages": [], + "errors": [] +} +``` + +### Logs + +A detailed crawl log is appended to `logs/crawl.log` on every run. + +> **Note:** The `data/`, `logs/`, and `reports/` directories are created +> automatically at runtime by `src/constants.py`. They do not need to +> exist beforehand. Consider adding `logs/` and `reports/` to +> `.gitignore` if you don’t want to track generated output. + +## How It Works + +1. **Discover conditions** — fetches the A-to-Z index page and extracts + all `/conditions//` links using a regex filter. +2. **Discover subpages** — for each condition hub page, looks for a + `