Professional Ethics

MDSR Ch 6

Announcements

Announcements

• grading
  • Piazza posted (will update again)
  • GitHub posted (split 1st & 2nd halves of term)
• midterm exam
  • planning to return them & discuss on Wed (3/13)
  • Extra Credit survey (to boost avg slightly)
• MDSR Ch 6 (ethics): no programming assigned
• MDSR Ch 6 exercises–no code, but use GitHub, make commits, & submit R Notebook like always
• MDSR Ch 10 programming notebook

MDSR Ch 10 Errata / Tips

• Some sections don’t require programming, but please still include the headers for navigation purposes
• p. 234-235: Rename the result for 20,000 simulations as sim_results20k
  • the authors reuse the object name sim_results for the example with 20,000 simulations, but this overwrites the object expected for the plot at the top of p. 235.
    
  • by renaming the object resulting form 20,000 simulations, the ggplot call will still be able to access the intended object

In the news…

These Two Charts Prove A College Education Just Isn’t Worth The Money Anymore

• Q: Any thoughts?

source: https://www.businessinsider.com/these-two-charts-prove-a-college-education-just-isnt-worth-the-money-anymore-2012-6

Education vs Income Observations

• does not show comparison with non-college graduates
• is annual income at graduation the best metric?
  • (no doubt easier to access)
  • lifetime total income earnings?
  • maximum income achieved?
  • is income the only benefit?
  • others?
• Q: can you really trust me on this subject?
  • Q: why or why not?

source: https://io9.gizmodo.com/11-most-useless-and-misleading-infographics-on-the-inte-1688239674

Stand your ground

• Graphic once published by news source Reuters
  • Q: What can you learn from this graphic?
  • Q: Anything you would change about it?

Professional Ethics

• In the US, there are popular notions of professional ethics for some professions but not others
  • Medical Doctors–Hippocratic Oath
    • “(help or) Do no harm” is well-known, but actually comes from elsewhere
    • “I will not be ashamed to say ‘I know not,’ nor will I fail to call in my colleagues when the skills of another are needed for a patient’s recovery”
    • “I will respect the privacy of my patients, for their problems are not disclosed to me that the world may know”
    • “I will remember that I am a member of society, with special obligations to all my fellow human beings, those of sound mind and body as well as the infirm”
• Q: How do these relate to professional ethics in Statistics & Data Science

Professional Ethics

• Some behaviors are pretty universally viewed as unethical: lying, cheating, stealing
• Professional ethics additionally requires a more nuanced interpretation
• Reasonable people can disagree about what actions are best,
  • Ethics are not law
  • We are best to assess ethical action though use of consistent principles and guidelines

General principles

• You have skills that aren’t widely available; don’t use them to deceive others (intentionally or not)
• Do your work well by your own standards and by the standards of your profession
  • do work that would pass scrutiny by professional colleagues
  • know your limits (e.g. regarding use of methods you don’t understand)
• Recognize the parties (stakeholders) to whom you have a special professional obligation
  • employer & client
  • general public or participants in a study
  • research community or the profession itself
• Report results and methods honestly and respect your responsibility to identify and report flaws and shortcomings in your work.
  • don’t overstate your confidence
  • often, “flaws in your work” are unavoidable limitations or necessary compromise. These should still be known to the consumer of your analysis

American Statistical Association (2018 ethical guidelines)

• Professional integrity and accountability
• Integrity of data and methods
  • Note: both plots shown before violate this section of ASA ethical guidelines
• Responsibilities to Science/Public/Funder/Client
• Responsibilities to research subjects
• Responsibilities to research team colleagues
• Responsibilities to other statisticians and statistics practitioners
• Responsibilities regarding allegations of misconduct
• Responsibililies of [those] employing statistical practitioners

source: https://www.amstat.org/ASA/Your-Career/Ethical-Guidelines-for-Statistical-Practice.aspx

Data Science ethics

• Clearly a well-identified issue, but so far unclear who should be the competent authority
• Some examples I found easily
  • Penn State IST 402: Data Ethics
  • U Chicago Data Science for Social Good
  • U Michigan short course on Data Science Ethics
  • Coursera & edX courses on Data Science Ethics

Source: https://www.forbes.com/sites/kalevleetaru/2018/10/08/do-we-need-to-teach-ethics-and-empathy-to-data-scientists/#41540b512ee1

(PSU) Guidelines for decision making

1. Percieved problem or ethical dilemma?
2. What are the facts?
3. What stakeholders, values, and guidelines are involved?
4. What are the options (good & bad)?
5. Consider the options (outcomes, virtues, …)?
6. Which is the BEST (or “least bad”) option?
7. How might we prevent this issue in the future?

Some Examples

• Employment discrimination
• Scraping OkCupid data
• Reproducible spreadsheet analysis
• Legal negotiations

Applying our guidelines: Employment Discrimination

Applying our guidelines: Employment Discrimination

• Easy to criticize the method… quite likely that employers are inappropriately labeled as discriminators just by chance alone
• OFCCP didn’t create the method, they are actually required to implement this method…
  • US Office of Personnel Management (www.opm.gov)
  • “The Uniform Guidelines (http://uniformguidelines.com/) apply to all selection procedures used to make employment decisions” (according to OPM FAQ page)
• OFCCP enforces the law, and has a responsibility to the courts

Applying our guidelines: Data Scraping

Applying our guidelines: Data Scraping

• the researcher seems to use perfectly legitimate means to access the data
• Stakeholders: OkCupid users
  • Users did not consent to this use of their data… (& could not Opt out)
  • The data are personally identifiable and users could reasonably experience “damages” (embarassment or personal harm)
• Stakeholders: OkCupid
  • companies like this often specify terms of use that restrict how the data may be legitimately used.
  • terms of use cannot ethically be disregarded since they represent an explicit agreement between the service provider and (any) user of the service

Applying our guidelines: Reproducible spreadsheet analysis

Applying our guidelines: Reproducible spreadsheet analysis

• Researchers have an ethical obligation to be truthful in their reporting of research
  • honest reporting of results
  • permit results to be challenged or confirmed (Reinhart & Rogoff did this well)
• It is not unethical to be wrong so to speak
  • the ethical obligation is to take all reasonable steps to ensure that conclusions faithfully represent the data and the analysis framework
  • Reinhart & Rogoff may have done so to the best of their ability
• In statistics and data science, we have an ethical obligation to use tools that are reliable, verifiable, and conducive to reproducible data analysis
  • a reproducible workflow like that of R, RMarkdown, and Git accomplishes this purpose
  • tools should strive to avoid quiet (or silent) failure modes
  • Microsoft Excel does not
    • mixes the data with the analysis
    • difficult to program in a concise and readable way
    • commands are customized to a particular size and organization of data
    • validating against a known result is often impractical in Excel
    • click & drag operations are error-prone

Data reidentification and disclosure avoidance

• obligation to privacy protections
  • HIPAA protects privacy health information
  • FERPA protects privacy of student records
• there is an appropriate tension between disclosures for public good (healthcare costs and outcomes) and nondisclosures that protect personal privacy
• data scientists have skills that could circumvent these protections (even accidentally), and this carries with it a clear ethical responsibility

Data scraping and terms of use

• According to MDSR, (Slate.com)[https://slate.com/terms] recently stated in their terms of use explicitly prohibited users from scraping content or information (this appears to have changed)
• An application programming interface (API) is an approach to providing a program interface to allow consumers controlled access to services or applications owned by a company.
• If you want data from a public source
  • see if the company has a public API
  • see if someone has already written an R package
  • APIs vary, but lots of APIs allow you to pull data into R without the need to “scrape” it… R packages provide functions to make it easier
• If it’s not otherwise clear how to appropriate access the data… you need to ask permission.

Reproducibility

• Reproducible analysis is the practice of recording each and every step, no matter how trivial, in a data analysis
• Some elements of a reproducible analysis include (see MDSR for source)
  • Data: all source data in original form
  • Metadata: codebooks and other info needed to understand the data
  • Code: script needed to process data, conduct analysis, etc
  • Map: file that maps between output and results in the report
• Statisticians and data scientists should welcome code reviews and rigorous vetting, particularly when our work is expected to have significant impact or exposure.
• Q: How does our STAT 380 workflow compare to this standard?
  • what parts are we doing well?
  • what parts should we improve?

Multiple Testing

• One common way in which a statistician or data scientist can easily mislead others (and themselves in the process) is through multiple testing
• Imagine 100 research teams working to study efficacy of the same medical drug
  • with 5% significance level we expect 5 of those teams to observe statistically significant results by chance alone
  • 5 (false) positive reports are published; 95 are–of course–not published
  • Q: is the corroborating evidence of these 5 publications compelling evidence of an effect?

Multiple Testing

• we often consider dozens of variables… even hundreds (e.g., indicators for lots of categorical variables can add up)
• we are implicitly conducting parallel tests as part of model selection
• we are responsible to faithfully disclose such pertinent details of the analysis, and take reasonable steps to avoid misleading or overstated conclusions in those cases
• Q: What are some stretegies that have we discussed to avoid this issue?

More examples the ethical minefield we call Statistics & Data Science…

• Study design considerations
  • Sample size
    
  • Sampling methods
    
  • Assignment of subjects to treatment conditions
  • Data collection protocol
  • “re-randomizing”
  • Stopping rules (e.g. clinical trials)
  • Responsibility during double-blinding
• Data stewardship
  • Data cleaning (i.e. preparing raw data for analysis)
  • Data privacy / confidentiality
  • Outlier handling
  • Reproducibility of analysis
• Data analysis decisions
  • Analysis consistent with design / data collection
  • Misleading graphs
  • Checking assumptions
  • Analysis of data from unknown origin
  • Use of (deviation from) prescribed research protocol
  • Multiple testing
  • Fitting multiple statistical models (i.e. competing analyses)
  • Outlier handling
• Interpretations / Conclusions
  • Interpretation of results
  • Generalizeability of conclusions
  • Which is worse: bad data or no data?
  • Causal inference
  • Confirmation bias
    
  • post hoc conclusions
  • Role of statistician in polarizing contexts
    • Controversial issues (e.g. animal research)
      
    • Expert trial witness (e.g. litigation involving your employer)
      
    • Political policy (e.g. gerrymandering)