Programming

29004 readers

189 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 3 years ago

MODERATORS

snowe@programming.dev

Ategon@programming.dev

UlrikHD@programming.dev

bugsmith@programming.dev

Spyro@programming.dev

411

Huggingface have scraped the entirety of Github. (huggingface.co)

submitted 3 months ago by e8d79@discuss.tchncs.de to c/programming@programming.dev

124 comments fedilink hide all child comments

Would be a terrible shame if lots of people opted out.

If you are a EU citizen you might also want to write a complaint to privacy@huggingface.co because they are collecting your personally identifiable information in machine-readable form which they are distributing to third parties.

(page 2) 50 comments

sorted by: hot top controversial new old

[–] Marija_@programming.dev 6 points 3 months ago

Scraping without consent steals developer personalIP.

[–] Swuden@lemmy.world 6 points 3 months ago* (last edited 3 months ago) (3 children)

They’re not able to scrape private repos, surely?

[–] qaz@lemmy.world 5 points 3 months ago* (last edited 3 months ago)

No, it seems to only be a subset of public repo's.
I have like 65 repo's and only 13 were scraped. I don't get why they specifically scraped those though. They don't have the most stars, they aren't the oldest or newest, not the ones with the most forks, nor do I see a pattern based on programming language.

load more comments (2 replies)

[–] jbrains@sh.itjust.works 6 points 3 months ago

Good thing I never finished a project!

[–] gnawmon@ttrpg.network 5 points 3 months ago* (last edited 3 months ago)

they can do whatever they want to do with my code.

as long as it's complaint to AGPL v3 :D

(doubt ai companies care about that)

[–] Kissaki@programming.dev 3 points 3 months ago

From the linked webpage readme:

2.1 Classify. Each file is labelled permissive (at least one permissive license detected, no conflicting non-permissive license), no_license (no licenses detected, or only non-license legal texts such as CLAs), or non_permissive. The permissive allowlist follows the Blue Oak Council list plus licenses categorized as Permissive or Public Domain by ScanCode. Files classified as non_permissive are excluded from both released datasets.

From https://www.bigcode-project.org/docs/about/the-stack/:

v1.1: The three copyleft licenses (MPL/EPL/LGPL) were excluded and the list of permissive licenses extended to 193 licenses in total. The list of programming languages was increased from 30 to 358 languages. Also opt-out request submitted by 15.11.2022 were excluded from this ersion of the dataset. The resulting near-deduplicated dataset is 6TB in size.

So MPL/EPL/LGPL are already not part of the dataset.

So… why were they in there? Was this added for v1.1?

"one permissive license" - So if my project includes a lib and I include the license file for that for the license notice…?

load more comments