Here’s what’s really going on inside an LLM’s neural network

Jump to latest Follow Reply
Status
You're currently viewing only pokrface's posts. Click here to go back to viewing the entire thread.
Muted (0)
Sort by date Sort by votes
pokrface

pokrface

Senior Technology Editor
21,513
Ars Staff
  • #5
    Concepts are encoded as tokens with coordinates in N-dimensional phase space. Concepts that share commonalities share locality. I think I actually grok this.
     
    Upvote
    102 (103 / -1)
    Quote
    Report
    Kyle Orland
    Kyle Orland
    Just to be clear on the terminology, tokens contain multiple "features."

    From the paper: "the average number of features active (i.e. with nonzero activations) on a given token was fewer than 300"

    edit LOL LOOK AT ME I HAVE ADMIN POWERS I CAN EDIT YOUR COMMENT TO MY COMMENT
    -lee <3 <3 <3
    Upvote
    102 (103 / -1)
    You must log in or register to reply here.
    Status
    You're currently viewing only pokrface's posts. Click here to go back to viewing the entire thread.
    Jump to latest Follow Reply
    Jump to latest Follow Reply