Linter will now hint if the license_family fields in the about section doesn't exist

[scopatz]

Checklist

• Added a news entry

[isuruf]

See check 12a

[scopatz]

12a checks that it is non-empty. This checks that they exist. There are starting to be a lot of recipes without these fields because of grayskull.

[jakirkham]

Instead of making this a new lint, what about consolidating on 12a?

[jakirkham]

I'd prefer to wait on more changes to the linting here. We are already trying to get legal advice in some cases ( conda-forge/cfep#23 (comment) ). I know there was some disagreement between folks on licensing in a recipe ( conda-forge/staged-recipes#11010 ). Before making changes to the linter it would be nice to get some clarity on what we would like to do where and form consensus. To be clear this doesn't mean we shouldn't work on this PR, I just think it would be prudent to pause for a bit until this addressed. Perhaps we bring this up in the next core meeting. Thoughts? 🙂

cc @conda-forge/core

[scopatz]

@jakirkham - I think that is counter to the goals here. I am hoping to prevent the merger of bad recipes with missing metadata. It is really easy to miss that these fields don't exist, and I have reviewed many PRs in the past few days that don't have them because of grayskull.

[scopatz]

If anything this helps put PRs with nebulous metadata on hold until we can figure out what to do with them legally, rather than trying to identify and fix them later in a feedstock

[jakirkham]

@scopatz, it's not clear to me the issue is actually a technical one, in which case a technical fix may not help. There may be technical issues as well, but they need to be aligned with the goals of the larger group.

[scopatz]

Yeah, I think the questions are

• Should we require a license entry?
• Should we request a license_family entry?

I think the answer to both is yes. If we don't have a license declaration how can we know that we are able to distribute the code? If we do have a license declaration, I think it makes sense to request that people place these into categories, so that other tools can bundle according to the license restrictions their organization has more easily.

This PR just puts a stop gap on accidentally doing the wrong thing, which is why I feel a sense of urgency about it.

[scopatz]

OK, this has been updated and passes CI now!

[jakirkham]

The issues around licensing (at least recently) have not been technical AFAICT, but social. What are our values as a community? Which ones take precedent? etc. We can certainly make technical fixes, but that doesn't really answer these underlying questions. Does that make any sense? 🙂

[scopatz]

Yes, that makes sense. I understand they are social. I am trying to buy some time to figure those things out with this PR. I do a lot of reviews on staged-recipes and this PR would make my life a lot easier, by removing another three things I have to automatically check for, and prevent personal burnout. This PR in its current form doesn't change the way we are currently opperating. It just reduces the pain of operating that way. Does that make sense? Hopefully that helps clarify it

[scopatz]

This is ready for another look!

[marcelotrevisani]

I don't know... but I will give my personal opinion on that
I was doing some research regarding this and I just discovered that it is not widely used (actually, I just know that we are using it), and it does not make too much sense (from my point of view) to be very honest. Since that license variety too much even if it is considered to be the same "family".
What makes sense is to see if the license is compatible with others, but I think that is not our work here.

And on top of that, I think that is violating the DRY principle because we are duplicating information regarding the license if someone needs to know the license, the user can read it because the license is available with the license_file

Sorry if I am being a pedant here, but I still cannot understand the advantage of it, because it is not a legal thing (since we are specifying the license and the license file) and it may complicate to write recipes and all of that

For example
http://oss-watch.ac.uk/apps/licdiff/
https://www.opensource.org
https://spdx.org/licenses/
https://tldrlegal.com/
none of them use license family
I am still trying to understand why this is really necessary

[beckermr]

This PR recommends, not requires, license_family. As such I think it is fine. LGTM to merge!

[marcelotrevisani]

This PR recommends, not requires, license_family. As such I think it is fine. LGTM to merge!

Sorry, just to clarify, I am not against it. 😁

I just think that is not really necessary because of the points which I highlighted
I always got puzzled by this license_family
because someone can specify the license as Expat and the license family as BSD, for example
Not everybody knows that in fact expat is MIT, that is very likely to pass in staged recipes for example
I think we just duplicated our problems regarding the license with that 😁

[scopatz]

Pinging this again @conda-forge/core

[mbargull]

My point here is that this is a very minimal, 2 line change to help us catch issues potential issues, while we do figure out what we want to do with more formal things

I agree with that. But I'd like this PR to be changed to do the opposite, i.e., recommend to remove license_family if present.
Marcelo is right on his assessment and I never understood why this has been there in the first place.
My (very undiplomatic, sorry) stance is that it is

• useless
• not the job (i.e., the categorization of a license) of a packager
• redundant at best
• error-prone
• unnecessarily hard-coded in conda-build
• etc.

[scopatz]

@mbargull - I am happy with you stance, but someone should write a CFEP for it. We have thousands of packages with this entry right now, and it is still in the example recipe. This is a stop-gap measure to ensure consistency

[ocefpaf]

I agree with that. But I'd like this PR to be changed to do the opposite, i.e., recommend to remove license_family if present.

@mbargull here is some history on the license_family field and I'm hoping @msarahan can correct me if I'm wrong.

It was added by AnacondaInc to help costumers easily create package collections that would fit their license needs. At the time I was against enforcing that in conda-forge b/c we don't have those corporate needs but we did let AnacondaInc added them as they please just because that does not hurt our recipes in any way.

I still have the same opinion. I don't want to enforce this on the community, but if a corporation needs that field they can send PRs adding them.

[scopatz]

Also, I generally agree that the SPDX identifiers are a much better way of doing this kind of thing. However, we have yet to fully adopt those.

[beckermr]

I want to point out that @scopatz is doing the vast majority of the heavy lifting on staged-recipes at the moment. Unless some of the others on this thread want to start sharing the load more, I think we should all defer to him on changes that he would like to make his day-to-day easier. That is the original motivation for this change and is unaffected by any of these other, more significant issues. They should be discussed in a CFEP for SPDX identifiers, which I am happy to shepherd myself.

[ocefpaf]

I want to point out that @scopatz is doing the vast majority of the heavy lifting on staged-recipes at the moment. Unless some of the others on this thread want to start sharing the load more, I think we should all defer to him on changes that he would like to make his day-to-day easier. That is the original motivation for this change and is unaffected by any of these other, more significant issues. They should be discussed in a CFEP for SPDX identifiers, which I am happy to shepherd myself.

What is easier then not having a license_family field?

[beckermr]

Deferring to @scopatz on this, for now, is the point, not debating what is easiest for him.

[scopatz]

Having this hint would really help me in the short term. I would really appreciate it if this was merged. Otherwise the list of PRs that this issue affects will continue to grow and I will keep having to manually look for it.

[isuruf]

@scopatz, can you describe the issue please? Is it just inconsistency?

[mbargull]

IMO, just adding/changing a recommendation on such a non-critical part does not need a CFEP.
I don't agree with the need for such a "stop-gap measure to ensure consistency" because there is little benefit on this (and this is coming from someone who is very fond of consistencies) -- arguably, there are much more other inconsistencies in the "handwritten" recipes we have.
In my current (about a month old) clone of feedstocks, we have ~7000 recipes with license_family and ~2000 without, meaning to achieve consistency we'd have to change thousands of recipes anyway (I do acknowledge the w/o ones are fewer, but I'm more thinking of the future than current state of things).

@scopatz, would a hint to remove license_family not be equivalently helpful to your work?
Recommending not to add it means less work to be done in any case (i.e., no need to verify that field).

[scopatz]

Yes, it is inconsistency and we used to have a policy (before policies were as formal as they are now) where we required this field. So I see discussions of getting rid of license_family as a pretty major policy change.

It is also that if someone did want to filter on license type, right now license_family is the way to do that. So if we just start ignoring this field with out having a replacement (like SPXD), we actually lose a capability across our whole ecosystem. Part of the benefit of conda-forge is the curated nature of our packages. Dropping license_family without a replacement is akin to breaking an API without a deprecation cycle or a specific motivating reason. I think the onus is on not breaking APIs in most cases, including this one.

[beckermr]

Wow. 7k with vs 2k is de facto policy. Stopping adding it without replacement is not a good idea.

Again unless any of us plan to do staged recipes reviews at a similar pace as @scopatz, we need to defer to him, not second guess him.

[isuruf]

@beckermr, if your concern is just staged-recipes, we can make this a lint only for staged-recipes.
Adding a hint that license_family is recommended today and reverting that in a few weeks if the CFEP is accepted is going to be weird and confusing.

[beckermr]

We may not deprecate license_family in the cfep per my concerns that the depreciation would be unfriendly to the people that provide us with free services.

My concern here is only that defer to @scopatz and not bike-shed what is a defacto policy right now.

A staged recipes only hint would probably be fine but, again, let’s defer to @scopatz.

[ocefpaf]

my concerns that the depreciation would be unfriendly to the people that provide us with free services.

@forrestwaters I'd like to know your opinion here. Do you all still rely heavily on the license_family or would a strict SPDX license field suffice?

[isuruf]

#1220 is also ready.

[mbargull]

(semi off-topic)

Wow. 7k with vs 2k is de facto policy.

That only indicates prevalence. E.g.

$ grep -l '{% *set  *name' feedstocks/*/recipe/meta.yaml | wc -l 
5133

also only indicates prevalence, but not a policy on how much jinja2 templating one should use. There are historical (or other) reasons for such things, e.g., conda-skeleton-created recipes. We also have an indication of a correlation between license_family and artifacts known to be created by conda-skeleton:

 grep -l '^ *license_family:' feedstocks/*/recipe/meta.yaml | xargs grep -l 'name\[0\]\|version|replace' | wc -l
4764

to defer to him, not second guess him.

We should be critical to ourselves (to a reasonable extent ) and strive improvement ;).
(But I understand what you're getting at and agree on acknowledging who does the work.)

---

@scopatz, it's interesting to me that you think of the removal to be comparable to "breaking an API". Meaning, I hadn't expected such a strong reaction/classification on removing what I thought of as an "annoyance due to historical remnants".
(Also, if anyone uses license_family to filter packages, I would urge them to rethink their strategy due to the 2/9 recipes without that field, the OTHER-classified ones, and misclassified ones..)
Generally, I would advice you to promote the SPDX-expression use instead of license_family on staged-recipes. Not just because I think it is the better choice for us going forward, but I should also make your work easier, I would think/hope. I haven't yet really checked out Isuru's PR, but it should help a good bit on that.

[scopatz]

@mbargull @isuruf @jakirkham @marcelotrevisani - yeah, I agree with your sentiments here. They way things are done now, kinda sucks (relying on license_family) and there is a better way to do it (SPDX identifiers in license). I feel that since conda-forge is in the business of creating declarative descriptions for building packages, every field in the YAML file is part of our API. The responsible thing to do when changing APIs is to have a deprecation cycle, a plan, and a clear discussion about why the old API has to go (ie a CFEP).

And @ocefpaf @forrestwaters - Anaconda isn't the only company. There are untold masses that use our packages and their metadata, in large part because they are better than anything else out there in this space. We have to be cognizant of the fact that 90% of them will never communicate with us in any way, and only read the documentation we put out there.

For example, consider what would happen if conda-build dropped license_family support tomorrow and suddenly 77% of our feedstocks stopped building. It would kinda suck, and we would have to do something, even if that were just pinning to an older version of conda-build.

I think we are all in basic agreement here about what to do, and what the path forward is, so someone should just write the CFEP already 😉 🏆

[ocefpaf]

And @ocefpaf @forrestwaters - Anaconda isn't the only company.

Never said they are the only company. I asked @forrestwaters b/c they are a known company that supports us.

[scopatz]

OK, I have fixed the conflicts and this is ready for review again. CC @conda-forge/core

[CJ-Wright]

If desired the auto tick bot could help to fill in the license family (where it can inspect one reasonably) for packages that are lacking it.

[scopatz]

@mbargull - is your change request still valid?

[mbargull]

@scopatz, yes, I still think it makes more sense and that it would reduce the amount of review work etc. However, feel free to take that as an advice/recommendation from my part but not as a "decree" -- meaning, I don't insist on blocking this PR!

[isuruf]

I'll send a CFEP later today deprecating license_family and using SPDX for license field.

[beckermr]

@isuruf We need to wait to hear from @forrestwaters before we deprecate license_family

[isuruf]

Sure. Interested parties can weigh in on the CFEP.

[isuruf]

The linter now checks for SPDX on license field and the hotfixing code will add a license_family from the SPDX to packages without a license_family. So, the question right now is, do we need this information in the recipe or is having it in packages enough? If the latter, I guess we can close this.