Skip to content

Fix data exfil even with disabled Sentry logging - #319

Open
symbioquine wants to merge 1 commit into
openclimatefix:mainfrom
symbioquine:fix-sentry-optout-data-exfil
Open

Fix data exfil even with disabled Sentry logging#319
symbioquine wants to merge 1 commit into
openclimatefix:mainfrom
symbioquine:fix-sentry-optout-data-exfil

Conversation

@symbioquine

Copy link
Copy Markdown

Pull Request

Description

Motivation: The sentry_sdk package implements pretty aggressive instrumentation that hooks itself into various parts of the Python runtime which can result in data sent to Sentry that the user may not intend to share.

Ideally, the Sentry logging would be fixed to only do the explicit logging that the original commit intended, but for now this change just fixes the handling of the QUARTZ_SOLAR_FORECAST_LOGGING environment variable to at least fully disable the sentry_sdk integration.

Fixes #317

How Has This Been Tested?

Ran uv add --editable /path/to/repositories/git/open-source-quartz-solar-forecast from another package and tested my own app with these changes.

Also ran docker compose up -d and did some minimal smoke testing based on the example provided in the api/README.md file:

$ curl -X POST "http://localhost:8000/forecast/" -H "Content-Type: application/json" -d '{
  "site": {
    "latitude": 37.7749,
    "longitude": -122.4194,
    "capacity_kwp": 5.0,
    "tilt": 30,
    "orientation": 180
  },
  "timestamp": "2023-08-14T10:00:00Z"
}' | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6871  100  6696  100   175  99283   2594 --:--:-- --:--:-- --:--:--  100k
{
  "timestamp": "2023-08-14T10:00:00",
  "predictions": {
    "power_kw": {
      "2023-08-14T10:00:00": 0.0,
      "2023-08-14T10:15:00": 0.0,
      "2023-08-14T10:30:00": 0.0,
      "2023-08-14T10:45:00": 0.0,
      "2023-08-14T11:00:00": 0.0,
      "2023-08-14T11:15:00": 0.0,
      "2023-08-14T11:30:00": 0.0,
      "2023-08-14T11:45:00": 0.0,
      "2023-08-14T12:00:00": 0.0,
      "2023-08-14T12:15:00": 0.0,
      "2023-08-14T12:30:00": 0.0,
      "2023-08-14T12:45:00": 0.0,
      "2023-08-14T13:00:00": 0.0,
      "2023-08-14T13:15:00": 0.0,
      "2023-08-14T13:30:00": 0.001695776253860578,
      "2023-08-14T13:45:00": 0.010303744936803983,
      "2023-08-14T14:00:00": 0.02238687126776276,
      "2023-08-14T14:15:00": 0.04879000192265005,
      "2023-08-14T14:30:00": 0.17476006366761235,
      "2023-08-14T14:45:00": 0.24391340161264516,
      "2023-08-14T15:00:00": 0.3281125834421627,
      "2023-08-14T15:15:00": 0.39893134876767333,
      "2023-08-14T15:30:00": 0.4709161158521111,
      "2023-08-14T15:45:00": 0.5354829830232879,
      "2023-08-14T16:00:00": 0.6161486403075218,
      "2023-08-14T16:15:00": 0.7113728589441699,
      "2023-08-14T16:30:00": 0.9441712106213074,
      "2023-08-14T16:45:00": 1.0349439514299605,
      "2023-08-14T17:00:00": 1.1216623475320948,
      "2023-08-14T17:15:00": 1.2037465495886142,
      "2023-08-14T17:30:00": 1.2841781777060592,
      "2023-08-14T17:45:00": 1.3556940685888779,
      "2023-08-14T18:00:00": 1.3281501790919048,
      "2023-08-14T18:15:00": 1.0858910696463018,
      "2023-08-14T18:30:00": 1.840057038723507,
      "2023-08-14T18:45:00": 1.91600174406096,
      "2023-08-14T19:00:00": 1.9622414601521907,
      "2023-08-14T19:15:00": 1.829486042321728,
      "2023-08-14T19:30:00": 1.651197615669838,
      "2023-08-14T19:45:00": 1.6663322967495895,
      "2023-08-14T20:00:00": 1.6737511862753964,
      "2023-08-14T20:15:00": 1.673416205815065,
      "2023-08-14T20:30:00": 1.8523749442462805,
      "2023-08-14T20:45:00": 1.8348007606609407,
      "2023-08-14T21:00:00": 1.8087373226417423,
      "2023-08-14T21:15:00": 1.9291394277824478,
      "2023-08-14T21:30:00": 1.703908913894003,
      "2023-08-14T21:45:00": 1.633318657123137,
      "2023-08-14T22:00:00": 1.5796348278739039,
      "2023-08-14T22:15:00": 1.4965136296758799,
      "2023-08-14T22:30:00": 1.3533938317541132,
      "2023-08-14T22:45:00": 1.2811307525041669,
      "2023-08-14T23:00:00": 1.226332884039472,
      "2023-08-14T23:15:00": 1.1417175695640553,
      "2023-08-14T23:30:00": 0.9957101580384307,
      "2023-08-14T23:45:00": 0.9122502664428116,
      "2023-08-15T00:00:00": 0.790920233108484,
      "2023-08-15T00:15:00": 0.6972523127500065,
      "2023-08-15T00:30:00": 0.5405345890071961,
      "2023-08-15T00:45:00": 0.45721082941445423,
      "2023-08-15T01:00:00": 0.4367414102786118,
      "2023-08-15T01:15:00": 0.3368688494517868,
      "2023-08-15T01:30:00": 0.13808076838247285,
      "2023-08-15T01:45:00": 0.10014350510159636,
      "2023-08-15T02:00:00": 0.057876780647631534,
      "2023-08-15T02:15:00": 0.0255262598787441,
      "2023-08-15T02:30:00": 0.017929785498603495,
      "2023-08-15T02:45:00": 0.002248449097413385,
      "2023-08-15T03:00:00": 0.0,
      "2023-08-15T03:15:00": 0.0,
      "2023-08-15T03:30:00": 0.0,
      "2023-08-15T03:45:00": 0.0,
      "2023-08-15T04:00:00": 0.0,
      "2023-08-15T04:15:00": 0.0,
      "2023-08-15T04:30:00": 0.0,
      "2023-08-15T04:45:00": 0.0,
      "2023-08-15T05:00:00": 0.0,
      "2023-08-15T05:15:00": 0.0,
      "2023-08-15T05:30:00": 0.0,
      "2023-08-15T05:45:00": 0.0,
      "2023-08-15T06:00:00": 0.0,
      "2023-08-15T06:15:00": 0.0,
      "2023-08-15T06:30:00": 0.0,
      "2023-08-15T06:45:00": 0.0,
      "2023-08-15T07:00:00": 0.0,
      "2023-08-15T07:15:00": 0.0,
      "2023-08-15T07:30:00": 0.0,
      "2023-08-15T07:45:00": 0.0,
      "2023-08-15T08:00:00": 0.0,
      "2023-08-15T08:15:00": 0.0,
      "2023-08-15T08:30:00": 0.0,
      "2023-08-15T08:45:00": 0.0,
      "2023-08-15T09:00:00": 0.0,
      "2023-08-15T09:15:00": 0.0,
      "2023-08-15T09:30:00": 0.0,
      "2023-08-15T09:45:00": 0.0,
      "2023-08-15T10:00:00": 0.0,
      "2023-08-15T10:15:00": 0.0,
      "2023-08-15T10:30:00": 0.0,
      "2023-08-15T10:45:00": 0.0,
      "2023-08-15T11:00:00": 0.0,
      "2023-08-15T11:15:00": 0.0,
      "2023-08-15T11:30:00": 0.0,
      "2023-08-15T11:45:00": 0.0,
      "2023-08-15T12:00:00": 0.0,
      "2023-08-15T12:15:00": 0.0,
      "2023-08-15T12:30:00": 0.0,
      "2023-08-15T12:45:00": 0.0,
      "2023-08-15T13:00:00": 0.0,
      "2023-08-15T13:15:00": 0.0,
      "2023-08-15T13:30:00": 0.0025128175265366325,
      "2023-08-15T13:45:00": 0.017148414467565598,
      "2023-08-15T14:00:00": 0.03843319912164124,
      "2023-08-15T14:15:00": 0.084754522044518,
      "2023-08-15T14:30:00": 0.16055397142066136,
      "2023-08-15T14:45:00": 0.21253120376798798,
      "2023-08-15T15:00:00": 0.28821167672456965,
      "2023-08-15T15:15:00": 0.348771650752393,
      "2023-08-15T15:30:00": 0.4081525896950427,
      "2023-08-15T15:45:00": 0.48145537356741314,
      "2023-08-15T16:00:00": 0.5541972296817488,
      "2023-08-15T16:15:00": 0.6416309638890417,
      "2023-08-15T16:30:00": 0.9567600389995391,
      "2023-08-15T16:45:00": 1.0434312069804048,
      "2023-08-15T17:00:00": 1.2474234164231766,
      "2023-08-15T17:15:00": 1.33881734569758,
      "2023-08-15T17:30:00": 1.33963114793101,
      "2023-08-15T17:45:00": 1.41508121915256,
      "2023-08-15T18:00:00": 1.4834570140181373,
      "2023-08-15T18:15:00": 1.5422081032672592,
      "2023-08-15T18:30:00": 1.6352166031208557,
      "2023-08-15T18:45:00": 1.7050947371977911,
      "2023-08-15T19:00:00": 1.7462084184986193,
      "2023-08-15T19:15:00": 1.6092198354393017,
      "2023-08-15T19:30:00": 1.6064449451554075,
      "2023-08-15T19:45:00": 1.6210970923493466,
      "2023-08-15T20:00:00": 1.6282289935315006,
      "2023-08-15T20:15:00": 1.6278040756524816,
      "2023-08-15T20:30:00": 1.6529005060577775,
      "2023-08-15T20:45:00": 1.637089009439998,
      "2023-08-15T21:00:00": 1.6136889336827154,
      "2023-08-15T21:15:00": 1.7376372874919443,
      "2023-08-15T21:30:00": 1.879299784681192,
      "2023-08-15T21:45:00": 1.8033544752959578,
      "2023-08-15T22:00:00": 1.7435159881441396,
      "2023-08-15T22:15:00": 1.4974539087362817,
      "2023-08-15T22:30:00": 1.4211915952593248,
      "2023-08-15T22:45:00": 1.3449950492959661,
      "2023-08-15T23:00:00": 1.2859385796581027,
      "2023-08-15T23:15:00": 1.196820094723311,
      "2023-08-15T23:30:00": 0.9004358721295549,
      "2023-08-15T23:45:00": 0.8250285631529135,
      "2023-08-16T00:00:00": 0.7121624069277968,
      "2023-08-16T00:15:00": 0.6334084871059362,
      "2023-08-16T00:30:00": 0.44797904743038935,
      "2023-08-16T00:45:00": 0.37842789259228915,
      "2023-08-16T01:00:00": 0.3051069138244677,
      "2023-08-16T01:15:00": 0.23855829835170128,
      "2023-08-16T01:30:00": 0.17298440257878356,
      "2023-08-16T01:45:00": 0.12268927347473677,
      "2023-08-16T02:00:00": 0.07009507896546435,
      "2023-08-16T02:15:00": 0.030429956546061277,
      "2023-08-16T02:30:00": 0.015317088432629744,
      "2023-08-16T02:45:00": 0.0015886249835995962,
      "2023-08-16T03:00:00": 0.0,
      "2023-08-16T03:15:00": 0.0,
      "2023-08-16T03:30:00": 0.0,
      "2023-08-16T03:45:00": 0.0,
      "2023-08-16T04:00:00": 0.0,
      "2023-08-16T04:15:00": 0.0,
      "2023-08-16T04:30:00": 0.0,
      "2023-08-16T04:45:00": 0.0,
      "2023-08-16T05:00:00": 0.0,
      "2023-08-16T05:15:00": 0.0,
      "2023-08-16T05:30:00": 0.0,
      "2023-08-16T05:45:00": 0.0,
      "2023-08-16T06:00:00": 0.0,
      "2023-08-16T06:15:00": 0.0,
      "2023-08-16T06:30:00": 0.0,
      "2023-08-16T06:45:00": 0.0,
      "2023-08-16T07:00:00": 0.0,
      "2023-08-16T07:15:00": 0.0,
      "2023-08-16T07:30:00": 0.0,
      "2023-08-16T07:45:00": 0.0,
      "2023-08-16T08:00:00": 0.0,
      "2023-08-16T08:15:00": 0.0,
      "2023-08-16T08:30:00": 0.0,
      "2023-08-16T08:45:00": 0.0,
      "2023-08-16T09:00:00": 0.0,
      "2023-08-16T09:15:00": 0.0,
      "2023-08-16T09:30:00": 0.0,
      "2023-08-16T09:45:00": 0.0
    }
  }
}
  • Yes

If your changes affect data processing, have you plotted any changes? i.e. have you done a quick sanity check?

  • Yes N/A

Checklist:

  • My code follows OCF's coding style guidelines
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation N/A
  • I have added tests that prove my fix is effective or that my feature works N/A (I'm not convinced it's worthwhile to try and prove that the Sentry SDK is not imported/initialized in an automated test - though I'm happy to hear ideas about how such a test would work.)
  • I have checked my code and corrected any misspellings

Motivation: The `sentry_sdk` package implements pretty
aggressive instrumentation that hooks itself into various
parts of the Python runtime which can result in data sent
to Sentry that the user may not intend to share.

See openclimatefix/open-source-quartz-solar-forecast/317

Ideally, the Sentry logging would be fixed to only do the
explicit logging that the original commit intended, but for
now this change just fixes the handling of the
`QUARTZ_SOLAR_FORECAST_LOGGING` environment variable to at
least fully disable the `sentry_sdk` integration.
Comment thread quartz_solar_forecast/utils/sentry_logging.py

@peterdudfield peterdudfield left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks this looks great

@peterdudfield

Copy link
Copy Markdown
Contributor

you happy for this to get merged?

@symbioquine

Copy link
Copy Markdown
Author

you happy for this to get merged?

Yeah, sounds good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sentry Integration Security Issues

2 participants