Update nextbike feeds

[nbdh]

Context
Updated 64 nextbike systems.

What's Changed
This PR:

• Sorts existing systems.
• Adds nextbike_ap, nextbike_bi, nextbike_bk, nextbike_bp, nextbike_cx, nextbike_kr, nextbike_lr, nextbike_ma, nextbike_nl, nextbike_nu, nextbike_oa, nextbike_oe, nextbike_os, nextbike_pg, nextbike_rq.
• Changes nextbike_af, nextbike_ah, nextbike_am, nextbike_an, nextbike_bc, nextbike_br, nextbike_by, nextbike_cg, nextbike_cj, nextbike_cm, nextbike_cs, nextbike_dj, nextbike_dn, nextbike_ds, nextbike_em, nextbike_eq, nextbike_gd, nextbike_gk, nextbike_gp, nextbike_gs, nextbike_gt, nextbike_hb, nextbike_hf, nextbike_hr, nextbike_ig, nextbike_kc, nextbike_ml, nextbike_mr, nextbike_rt, nextbike_sb, nextbike_se, nextbike_sg, nextbike_sl, nextbike_vu, nextbike_wa, nextbike_zd.
• Removes nextbike_oc, nextbike_ti, nextbike_tb, nextbike_hg, nextbike_ff, nextbike_wo, nextbike_sv, nextbike_ey, nextbike_oy, nextbike_ku, nextbike_wh, nextbike_lp, nextbike_po.

[richfab]

Hi @nbdh,

Thank you for this update!

Your changes will be merged in #891 to update the sorting.

Indeed, your sorting method differs from the existing sort: it seems to ignore spaces in the systems names. Is this something you can change in future PRs?

Thanks again!
Fabien (MobilityData)

[nbdh]

Hi @richfab,
thanks for creating the follow up MR.

Indeed, your sorting method differs from the existing sort: it seems to ignore spaces in the systems names. Is this something you can change in future PRs?

Sure thing, back in 2018 when I build this automation, I tried to find any way of sorting the existing systems as closely as how the list was sorted back then, but since the exact sorting of the systems.csv file is not part of the spec (as far as I'm aware), it seems to have changed over time.

Are you aware of any tooling that sorts the list exactly as it is in the repository? Should it be considered to add sorting to the GitHub workflow triggered if the systems.csv is changed, so publishers can make their lives easier and simply append their systems to the list instead of patching them into the list?

Best regards
Daniel

[richfab]

Thanks @nbdh!

@davidgamez @fredericsimard Do you have any thoughts around Daniel's questions/suggestions?

[fredericsimard]

I think we would need to know what publishers prefer/require. I always imagine them processing the CSV into an array in a DB or a JSON, but it's only an assumption.

My general principle about sorting is based on whether it's humans or machines that need to read it. Humans expect a classic alphabetical order, machines expect a predictable order they can parse.

Not super helpful, but I don't know enough about how this CSV is used to give a straight answer, sorry!

[nbdh]

Is there some confusion about consumer/publisher here?

As a publisher, I must ensure our systems are listed in systems.csv as it is mandatory.

So what we currently do to update the file:

1. (sort the list and commit)
2. remove own systems from the list
3. append all our systems
4. sort again
5. commit.

If sorting was done in a GitHub workflow within this repo, publishers could omit step 4 (and we also 1.), guessing the exact way by which this list is sorted - either on each branch, or what is probably cleaner, just on push (merge) into master.

[fredericsimard]

Zero confusion. Fabien asked for my opinion, I don't know enough about the context to give a clear one.

If you need a list that's sorted a specific way we can run an automation each time the systems.csv file is updated to sort a copy of the file, something like systems-pub-sorted.csv and you could automate something on your end starting from it.

What do you all think?

[richfab]

Is there a way to setup a GitHub workflow that sorts systems.csv alphabetically every time a feed is added to systems.csv? If not, it's ok. I can sort the file manually every time there is a new contribution.

I found this example online but it does not show how to do the sorting.

Thanks!

[fredericsimard]

@richfab Once I know the logic for sorting, it's an easy addition to this repo. @nbdh Where can I find a reference on the sorting method?

[richfab]

The desired sorting method is: sort in ascending alphabetical order, based on the first four columns:

Country Code	Name	Location	System ID

[fredericsimard]

Alright, and should the sort happen on the existing systems.csv file or a copy like I mentioned earlier?

[richfab]

It should replace the existing systems.csv file if the sorting introduced a change in the file.

[fredericsimard]

@nbdh Would that sorting logic Fabien mentions just above be something you can work with?

[nbdh]

Thanks for following up on this.

I don't mind the way the systems are sorted in systems.csv. As a publisher, I'm usually not reading or parsing that file, and if it's sorted automatically, I also wouldn't need to sort the systems at all in our workflow when adding or updating them.

@fredericsimard I think the goal for the GitHub workflow would be to find a sorting that matches the existing order of the file as closely as possible. Even after several attempts, I couldn't come up with a solution that matches it exactly. In the end, though, I don't think the exact order matters all that much.

We've been using

(head -n 1; LC_ALL=en_US.UTF-8 gnusort --field-separator=',' --key=1,1 --key=2,2) < systems.csv

so far, which gets you close.

[fredericsimard]

Alright, that makes sense. @richfab What do you want to do? If we start sorting the systems.csv with this logic, would it have an impact that we didn't plan for? That's my only concern.

That's why I think we should have a sorted and unsorted variation of the file, for legacy compatibility reasons.

[richfab]

@fredericsimard I don't foresee any issue with changing the sorting in the original file. I've made manual changes to the sorting several times and never heard that it caused a compatibility issue.
Let's please proceed with sorting the original file.
Thank you!