Wow. There is no shortage of critics on this site. First off, take Lloyd's advice. Lloyd gives great advice.
Second, seriously, Thursday, Friday and Saturday are prime time. Sunday will be a let down because folks are going to be breaking down everything starting shortly after breakfast. I used to come back to break down my pins after the early football game and they'd be among the last four on the floor.
That being said, you could certainly use Sunday to check out downtown, it's worth the effort.
Whatever you end up doing, say hi to folks and they'll likely be more friendly than you'd expect. Have fun.