• Login
    View Item 
    •   SMARTech Home
    • Georgia Tech Interdisciplinary Research Centers (IRCs)
    • Machine Learning (ML@GT)
    • Machine Learning@Georgia Tech Seminars
    • View Item
    •   SMARTech Home
    • Georgia Tech Interdisciplinary Research Centers (IRCs)
    • Machine Learning (ML@GT)
    • Machine Learning@Georgia Tech Seminars
    • View Item
    JavaScript is disabled for your browser. Some features of this site may not work without it.

    You can lead a horse to water...: Representing vs. Using Features in Neural NLP

    Thumbnail
    View/Open
    pavlick.mp4 (215.4Mb)
    pavlick_videostream.html (1.323Kb)
    transcript.txt (53.08Kb)
    thumbnail.jpg (65.19Kb)
    Date
    2021-03-24
    Author
    Pavlick, Ellie
    Metadata
    Show full item record
    Abstract
    A wave of recent work has sought to understand how pretrained language models work. Such analyses have resulted in two seemingly contradictory sets of results. On one hand, work based on "probing classifiers" generally suggests that SOTA language models contain rich information about linguistic structure (e.g., parts of speech, syntax, semantic roles). On the other hand, work which measures performance on linguistic "challenge sets" shows that models consistently fail to use this information when making predictions. In this talk, I will present a series of results that attempt to bridge this gap. Our recent experiments suggest that the disconnect is not due to catastrophic forgetting nor is it (entirely) explained by insufficient training data. Rather, it is best explained in terms of how "accessible" features are to the model following pretraining, where "accessibility" can be quantified using an information-theoretic interpretation of probing classifiers.
    URI
    http://hdl.handle.net/1853/64417
    Collections
    • Machine Learning@Georgia Tech Seminars [52]

    Browse

    All of SMARTechCommunities & CollectionsDatesAuthorsTitlesSubjectsTypesThis CollectionDatesAuthorsTitlesSubjectsTypes

    My SMARTech

    Login

    Statistics

    View Usage StatisticsView Google Analytics Statistics
    facebook instagram twitter youtube
    • My Account
    • Contact us
    • Directory
    • Campus Map
    • Support/Give
    • Library Accessibility
      • About SMARTech
      • SMARTech Terms of Use
    Georgia Tech Library266 4th Street NW, Atlanta, GA 30332
    404.894.4500
    • Emergency Information
    • Legal and Privacy Information
    • Human Trafficking Notice
    • Accessibility
    • Accountability
    • Accreditation
    • Employment
    © 2020 Georgia Institute of Technology