The problem lies at scale
When a technologist designs a solution for a specific community, particularly one they live in, unintended consequences are more manageable. But when that same solution is massively scaled, we have to carefully consider three concepts:
- The technologies we create reflect what we value, but they don’t necessarily reflect the capabilities that other communities need.
As a result, even if our designs stem from our values, there might be a mismatch between our designs and another community's needs. For example: women are much more likely to be sexually harassed online, so if most tech developers are men, there is a natural gap in understanding the experiences of female users.
- Communities have differences in their culture, government, law, and economics.
These differences influence the effects of technology, regardless of the inventor's intentions. For example: Facebook designed text-based AI systems to detect harmful content, but there are developing markets, such as those in Cambodia, where challenges typing in the native language caused the predominant use of Facebook’s messenger to be via audio messaging, which bypasses text-based safety features.
- Technology subtly and systematically affects the human mind.
Scaling these effects globally is likely to yield outcomes you didn't consider or desire. For example: Creators of social media apps most likely could not have predicted that “Social Media Influencer” would be one of the most sought-after professions by our youth.
...
Learned about this in the 'Supporting Fairness & Justice' module in a foundational course over at the Center for Humane Technology.