what did you learn today? (part 2)

[PaveHawk-]

dredphul[/url]":25j1j1p7]that the Cisco UCS C240 M3S is advertised as a 2U product, but actually requires 3U of space as all the air vents are on top of the unit.

Found buried in the install guide:

Do not block the air vents on the top of the server's cover. Do not stack another server directly on top of the C240 server. Doing so blocks the proper airflow, which could result in overheating, higher fan speeds, and higher power consumption.

Glad this was discovered before the units were racked.

How can Dell manage to do it with the R720XD and not have those issues?

I'm also specifying those in a tender response right now, this will put a dent in my design. High level right now until we move to 2nd round, but still.

[edit]

also, I have to work with idiotic staff. One of the "seniors" is so fucking lazy, he's going to test king his way through the VCP as opposed to knowing the material. We had a huge latency issue with our VNX today, write latency in the order of 3s but read latency still sitting on 2ms. To me, that screams there's a configuration issue or some sort. His response "I'll log it with EMC later". I go poking around, notice a very very very very very slow SDRS move operation and cancel it, suddenly everything is fine.

Maybe learn storage you fucking dimwit. And Exchange. And Active Directory. And VMware.

When I was managing this infrastructure, we were capacity managed/planned - right now, there's no idea of wtf is going on. We're at >90% memory usage on all the hosts right now, never mind that they havent had 5.0U1 applied.

 

[sryan2k1]

We're at >90% memory usage on all the hosts right now, never mind that they havent had 5.0U1 applied.

You can put plenty of cooling in 2U. Our R810's are loaded pretty heavy and they stay cool, and more importantly, fucking blow the air out the back.

 

[DuRaNdAl^]

My god there are so many interesting looking sessions at TechEd. I have no idea where I should be, except everywhere.

 

[ferzerp]

sryan2k1[/url]":1wn0gk32]

We're at >90% memory usage on all the hosts right now, never mind that they havent had 5.0U1 applied.

You can put plenty of cooling in 2U. Our R810's are loaded pretty heavy and they stay cool, and more importantly, fucking blow the air out the back.

I'm really not sure how cisco has convinced anyone to buy their servers. Actually, after my organization was continually harassed, at all levels, I do know. They keep hammering away until you relent, or find another contact within your org to try to sell to (all the way up to execs). Or at least they try to. We never gave in because we don't buy based on how hard you try to market to us.

 

[gblansandrock]

ferzerp[/url]":m4fxwlom]

sryan2k1[/url]":m4fxwlom]

We're at >90% memory usage on all the hosts right now, never mind that they havent had 5.0U1 applied.

You can put plenty of cooling in 2U. Our R810's are loaded pretty heavy and they stay cool, and more importantly, fucking blow the air out the back.

I'm really not sure how cisco has convinced anyone to buy their servers. Actually, after my organization was continually harassed, at all levels, I do know. They keep hammering away until you relent, or find another contact within your org to try to sell to (all the way up to execs). Or at least they try to. We never gave in because we don't buy based on how hard you try to market to us.

Keep it up and they'll flat out give you UCS. That's what happened to us. After turning them down, they eventually gave us a couple chassis, dozen blades and dual interconnects, free of charge. Will make for one hell of a test environment I guess

 

[Danger Mouse]

gblansandrock[/url]":3v9w5gt6]

ferzerp[/url]":3v9w5gt6]

sryan2k1[/url]":3v9w5gt6]

We're at >90% memory usage on all the hosts right now, never mind that they havent had 5.0U1 applied.

You can put plenty of cooling in 2U. Our R810's are loaded pretty heavy and they stay cool, and more importantly, fucking blow the air out the back.

I'm really not sure how cisco has convinced anyone to buy their servers. Actually, after my organization was continually harassed, at all levels, I do know. They keep hammering away until you relent, or find another contact within your org to try to sell to (all the way up to execs). Or at least they try to. We never gave in because we don't buy based on how hard you try to market to us.

Keep it up and they'll flat out give you UCS. That's what happened to us. After turning them down, they eventually gave us a couple chassis, dozen blades and dual interconnects, free of charge. Will make for one hell of a test environment I guess

Wooooo, I need to keep this in mind :devious:

 

[ramases]

TIL today? Two things:
0) Certain conditions cause a 5.x XenServer to crash after a downed iSCSI path recovered. "Quick, make your bet. Will recovering the path crash the XenServer, yes or no?"
1) Good thing that this environment is soon to be decommissioned, because there are other mines just waiting to explode in that setup. Thin-provisioning at the SAN-level with an overcommit-ratio of 5:1, what could possibly go wrong?

 

[sryan2k1]

That would explain a lot. Getting rid of Xen was one of the first things I did here, and we saw random failures like that. It's all starting to make sense.

 

[afidel]

ramases[/url]":6zujtlsb]TIL today? Two things:
0) Certain conditions cause a 5.x XenServer to crash after a downed iSCSI path recovered. "Quick, make your bet. Will recovering the path crash the XenServer, yes or no?"
1) Good thing that this environment is soon to be decommissioned, because there are other mines just waiting to explode in that setup. Thin-provisioning at the SAN-level with an overcommit-ratio of 5:1, what could possibly go wrong?

The overcommit wouldn't bother me at all if your growth pattern was fairly predictable. I run up to 4.5:1 at the VMWare level. At the SAN level I'd want warnings at 10-15% depending on how long it took to procure and install hardware, but a 5:1 overcommit wouldn't bother my by itself.

 

[ferzerp]

afidel[/url]":3pwhjtxa]

ramases[/url]":3pwhjtxa]TIL today? Two things:
0) Certain conditions cause a 5.x XenServer to crash after a downed iSCSI path recovered. "Quick, make your bet. Will recovering the path crash the XenServer, yes or no?"
1) Good thing that this environment is soon to be decommissioned, because there are other mines just waiting to explode in that setup. Thin-provisioning at the SAN-level with an overcommit-ratio of 5:1, what could possibly go wrong?

The overcommit wouldn't bother me at all if your growth pattern was fairly predictable. I run up to 4.5:1 at the VMWare level. At the SAN level I'd want warnings at 10-15% depending on how long it took to procure and install hardware, but a 5:1 overcommit wouldn't bother my by itself.

Correct, as long as your scale is large enough that no small cluster of systems are statistically significant, you can run a really high overcommit ratio as long as you properly manage your storage. If you have 800 TB allocated on a 300 TB array, but only 150 TB used and a policy to expand when you only have 25% volume remaining for example, you're probably reasonably safe, as long as your org rigidly adheres to that policy.

The problem is when you have a small scale environment where a handful of systems can kill your thin provisioning design, or if you don't manage the storage properly and let it fill.

Overallocation is only bad if you ever let the array fill (or fill to the point it stops working well)

 

[M. Jones]

gblansandrock[/url]":k5h23vr5]

ferzerp[/url]":k5h23vr5]
I'm really not sure how cisco has convinced anyone to buy their servers. Actually, after my organization was continually harassed, at all levels, I do know. They keep hammering away until you relent, or find another contact within your org to try to sell to (all the way up to execs). Or at least they try to. We never gave in because we don't buy based on how hard you try to market to us.

Keep it up and they'll flat out give you UCS. That's what happened to us. After turning them down, they eventually gave us a couple chassis, dozen blades and dual interconnects, free of charge.

We don't use UCS, except to run Cisco CallManager/CUCM VMs, although we do control those from the normal vCenter. Now I wonder if we got those gratis -- probably not.

Our people who have gone to UCS training report back that the features are only going to work for you at scale. There's even more lock-in with UCS than with other blade servers.

 

[sryan2k1]

Optiplex 760's can take 8GB RAM (2 x 4GB) even though Dell/Crucial says it maxes out at 4GB. TAKE THAT!

In other news, I upgraded this from 4-->8GB and from a 1TB spinning drive to a 120GB Samsung 840.


Snappy.

 

[M. Jones]

CanSpice[/url]":1pi74fi7]

M. Jones[/url]":1pi74fi7]Cloudera's Hadoop seems not to let you relocate or expand existing HDFS. Who would go out of their way to buy Hadoop?!

Uh, what? What do you mean by "relocate or expand existing HDFS"?

We were forced to expand drives against preference because after adding drives, existing HDFS volumes couldn't be moved to the new volumes/mountpoints, according to the SA doing the work.

Apparently we're using the gratis version, out of the company's repos:

deb [arch=amd64] http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/ precise-cdh4 contrib
deb [arch=amd64] http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm/ precise-cm4.5.1 contrib

 

[ramases]

afidel[/url]":l2n71fb2]

ramases[/url]":l2n71fb2]TIL today? Two things:
0) Certain conditions cause a 5.x XenServer to crash after a downed iSCSI path recovered. "Quick, make your bet. Will recovering the path crash the XenServer, yes or no?"
1) Good thing that this environment is soon to be decommissioned, because there are other mines just waiting to explode in that setup. Thin-provisioning at the SAN-level with an overcommit-ratio of 5:1, what could possibly go wrong?

The overcommit wouldn't bother me at all if your growth pattern was fairly predictable. I run up to 4.5:1 at the VMWare level. At the SAN level I'd want warnings at 10-15% depending on how long it took to procure and install hardware, but a 5:1 overcommit wouldn't bother my by itself.

The main issue why I am uncomfortable with it is that this entire setup is one of those things that should have been gone for quite some time now. Consequently people pay it less attention than they perhaps should, and there is a chance that people let it slide further than what would be wise due to the "we don't want to invest in something already scheduled for decommissioning".
Its also a rather small (but nonetheless important for us) environment with an unfortunate ratio of average-vm-size to total-datastore-size, which means even a few VMs can wreck the system.

I really, really hope to have the entire mess gone in the next 6 weeks.

 

[ferzerp]

M. Jones[/url]":1dpnq44p]

gblansandrock[/url]":1dpnq44p]

ferzerp[/url]":1dpnq44p]
I'm really not sure how cisco has convinced anyone to buy their servers. Actually, after my organization was continually harassed, at all levels, I do know. They keep hammering away until you relent, or find another contact within your org to try to sell to (all the way up to execs). Or at least they try to. We never gave in because we don't buy based on how hard you try to market to us.

Keep it up and they'll flat out give you UCS. That's what happened to us. After turning them down, they eventually gave us a couple chassis, dozen blades and dual interconnects, free of charge.

We don't use UCS, except to run Cisco CallManager/CUCM VMs, although we do control those from the normal vCenter. Now I wonder if we got those gratis -- probably not.

Our people who have gone to UCS training report back that the features are only going to work for you at scale. There's even more lock-in with UCS than with other blade servers.

A couple sneaked in to our org via phone upgrades, but we caught wind of it, and all new ones run on our server standard.

 

[afidel]

M. Jones[/url]":3pfluave]

gblansandrock[/url]":3pfluave]

ferzerp[/url]":3pfluave]
I'm really not sure how cisco has convinced anyone to buy their servers. Actually, after my organization was continually harassed, at all levels, I do know. They keep hammering away until you relent, or find another contact within your org to try to sell to (all the way up to execs). Or at least they try to. We never gave in because we don't buy based on how hard you try to market to us.

Keep it up and they'll flat out give you UCS. That's what happened to us. After turning them down, they eventually gave us a couple chassis, dozen blades and dual interconnects, free of charge.

We don't use UCS, except to run Cisco CallManager/CUCM VMs, although we do control those from the normal vCenter. Now I wonder if we got those gratis -- probably not.

Our people who have gone to UCS training report back that the features are only going to work for you at scale. There's even more lock-in with UCS than with other blade servers.

That's been my assessment, if you have datacenters full of them it's a very sweet system, if you need 2-3 enclosures it's WAY too complicated to ever pay off the time investment. I keep telling the Cisco guys that I spend less than 5% of one FTE on managing hardware so it's impossible that they'll save me enough time, they still keep pitching UCS. The only part I really want from UCS (UCS P81E, aka Palo aka VIC) they don't sell without the servers =(

 

[smwht]

M. Jones[/url]":1f12e3u3]

CanSpice[/url]":1f12e3u3]

M. Jones[/url]":1f12e3u3]Cloudera's Hadoop seems not to let you relocate or expand existing HDFS. Who would go out of their way to buy Hadoop?!

Uh, what? What do you mean by "relocate or expand existing HDFS"?

We were forced to expand drives against preference because after adding drives, existing HDFS volumes couldn't be moved to the new volumes/mountpoints, according to the SA doing the work.

Apparently we're using the gratis version, out of the company's repos:

deb [arch=amd64] http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/ precise-cdh4 contrib
deb [arch=amd64] http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm/ precise-cm4.5.1 contrib

This is technically possible, but not pretty - you can either decommission/re-add/rebalance nodes with the additional drives, or add the data directories and move blocks around on each node to balance the nodes out (balance scripts work on the node level, not the drive level, unfortunately).

Generally, though, the process for hadoop is add/replace nodes rather than upgrade existing nodes.

 

[ronelson]

Or at least it was supposed to be...two serious replies made me second guess myself.

It's CheckPoint, so I wouldn't put it past them to re-order the MAC address storage in their 64 bit client, or to make that part of the client handshake It still wouldn't make sense, though.

 

[sryan2k1]

There is something massively wrong with the phone network in the US. Our AT&T SIP services are offline (except for 800 numbers), I can't call from my cell in or our either to another VZW customer or off network.


We have a ticket open with AT&T but they said something major happened and they don't know what.

 

[Digitlman]

TIL the new APC 350 UPS' electrical sockets are "catholic school virgin tight" out of the box. So tight, that when plugging something in, I feel like I am going to break it, and I have seen a touch of spark every time.

Is this a new "feature"?

 

[CanSpice]

smwht[/url]"49fn7km]

M. Jones[/url]"49fn7km]

CanSpice[/url]"49fn7km]

M. Jones[/url]"49fn7km]Cloudera's Hadoop seems not to let you relocate or expand existing HDFS. Who would go out of their way to buy Hadoop?!

Uh, what? What do you mean by "relocate or expand existing HDFS"?

We were forced to expand drives against preference because after adding drives, existing HDFS volumes couldn't be moved to the new volumes/mountpoints, according to the SA doing the work.

Apparently we're using the gratis version, out of the company's repos:

deb [arch=amd64] http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/ precise-cdh4 contrib
deb [arch=amd64] http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm/ precise-cm4.5.1 contrib

This is technically possible, but not pretty - you can either decommission/re-add/rebalance nodes with the additional drives, or add the data directories and move blocks around on each node to balance the nodes out (balance scripts work on the node level, not the drive level, unfortunately).

Generally, though, the process for hadoop is add/replace nodes rather than upgrade existing nodes.

Yeah, we've had to manually balance disks at the drive level before, but only after a drive drops out of a node and gets replaced. Our nodes have (generally) four drives in them, and if one of them goes kaput, the other three continue to chug along. Then when the fourth drive gets replaced, it's 0% full while the others are at, say, 75% full, and they'll all fill up at the same rate (which is kind of stupid).

I think we've generally "solved" this by just decommissioning the entire node, then re-adding it to the cluster, then doing a cluster rebalance to get data back onto the node. Once we tried manually moving blocks around but that's more trouble than it's worth.

The SA was wrong and probably just said that to spare the pain of all of the manual re-jiggery. You can add a new drive to a node just fine, it just requires time and patience.

 

[dredphul]

smwht[/url]":2w9dffam]

M. Jones[/url]":2w9dffam]

CanSpice[/url]":2w9dffam]

M. Jones[/url]":2w9dffam]Cloudera's Hadoop seems not to let you relocate or expand existing HDFS. Who would go out of their way to buy Hadoop?!

Uh, what? What do you mean by "relocate or expand existing HDFS"?

We were forced to expand drives against preference because after adding drives, existing HDFS volumes couldn't be moved to the new volumes/mountpoints, according to the SA doing the work.

Apparently we're using the gratis version, out of the company's repos:

deb [arch=amd64] http://archive.cloudera.com/cdh4/ubuntu/precise/amd64/cdh/ precise-cdh4 contrib
deb [arch=amd64] http://archive.cloudera.com/cm4/ubuntu/precise/amd64/cm/ precise-cm4.5.1 contrib

This is technically possible, but not pretty - you can either decommission/re-add/rebalance nodes with the additional drives, or add the data directories and move blocks around on each node to balance the nodes out (balance scripts work on the node level, not the drive level, unfortunately).

Generally, though, the process for hadoop is add/replace nodes rather than upgrade existing nodes.

You're actually using Cloudera's package which isn't the free Apache version. The free Apache version is at http://hadoop.apache.org. You'll need to click around before you find the link to the download page. Remember that 1.0.x is the release version, 1.1.x is the next beta, 2.0.x is the next alpha, and 0.23 is because fuck you.

I thought the approved way to expand storage is to:
1. Chose a node and remove it (so the replicated data gets rebalanced)
2. Add new storage and nuke and reinstall
3. Add the 'new' node to the cluster
4. Rebalance
5. Go to step 1 and repeat until you've expanded all the necessary nodes

 

[ronelson]

YIL that when you announce, "Everyone should pay attention to and fix the devices that the monitoring system rejects, specifically because they don't respond to SNMP and can't be discovered," someone will actually ask, "Can we ticket on that?"

...

 

[M. Jones]

ronelson[/url]":91pd6z9i]YIL that when you announce, "Everyone should pay attention to and fix the devices that the monitoring system rejects, specifically because they don't respond to SNMP and can't be discovered," someone will actually ask, "Can we ticket on that?"

...

So...can they?

When you measure by metrics, people are going to work to metrics, because that's what you're asking them to do.

 

[Rick25]

That a certain load balancer company expects you to pay for the tax/duties when their equipment fails. Aside from my opinion on all that, it's also requires you to be a customs expect and have a business number that's registered for import/export.....so much for the expedited replacement. Silly thing is that neither of the first two times we had one die did any of this foolishness happen.

 

[M. Jones]

CanSpice[/url]":1arq7yev]
Yeah, we've had to manually balance disks at the drive level before, but only after a drive drops out of a node and gets replaced. Our nodes have (generally) four drives in them, and if one of them goes kaput, the other three continue to chug along. Then when the fourth drive gets replaced, it's 0% full while the others are at, say, 75% full, and they'll all fill up at the same rate (which is kind of stupid).

I think we've generally "solved" this by just decommissioning the entire node, then re-adding it to the cluster, then doing a cluster rebalance to get data back onto the node. Once we tried manually moving blocks around but that's more trouble than it's worth.

The SA was wrong and probably just said that to spare the pain of all of the manual re-jiggery. You can add a new drive to a node just fine, it just requires time and patience.

Good to know. These nodes were in small and very small dev and QA environments where removing, rebuilding, and re-adding nodes wouldn't have been an option. I'm unclear on the practicality of reloading the HDFS from scratch, though.

 

[ronelson]

So...can they?

Can the ticketing system ticket on things it can't ticket on?

 

[ferzerp]

M. Jones[/url]":1rc169hz]


When you measure by metrics, people are going to work to metrics, because that's what you're asking them to do.

Few things have been said with more truth than that. Sadly, increasingly, that is all people seem to understand. Making one number go up, and/or another go down has become more important that what those numbers purport to measure.

 

[M. Jones]

ronelson[/url]":3gazqrql]

So...can they?

Can the ticketing system ticket on things it can't ticket on?

Ah, so you meant 'can the ticketing system automation create a ticket based on a negative configure-time trigger?'. I haven't previously encountered the verbing of 'ticket' to mean 'generate a ticket from automation based on triggers'. Two of the possible interpretations of your statement, and those I considered most likely, were asking permission to manually create tickets to rectify said problems and asking you, the speaker, to manually create tickets to rectify said problem.

 

[afidel]

So speaking of VIC/Palo, it looks like the Qlogic 8200 cards with nPAR can do basically the same thing, though they only have 4 buckets. Is anyone using these to assure vmotion and storage bandwidth?

 

[ronelson]

Ah, so you meant 'can the ticketing system automation create a ticket based on a negative configure-time trigger?'. I haven't previously encountered the verbing of 'ticket' to mean 'generate a ticket from automation based on triggers'. Two of the possible interpretations of your statement, and those I considered most likely, were asking permission to manually create tickets to rectify said problems and asking you, the speaker, to manually create tickets to rectify said problem.

Withing being too specific, 'discovery' in this application means "ability to add the target to the devices being monitored and ticketed."

We then had a 10 minute discussion on why no, we will not write automation around the failed automation. Especially since there's this nice command you can run to show you all the errors, and nearly every one involves manual fixes (bad snmp config, firewall in the way, poor timing of discovery during an outage, etc - none of which you can determine programatically BECAUSE YOU CAN'T TALK TO IT!). At the end I think there were still people who didn't get it.

 

[speedye1]

TIL Office 365 support is terrible. We moved my our first ten mailboxes and just happened to setup a new machine for myself. Load fresh and updated Win 8 and Office 2013, took 30 minutes to load my mailbox, and every time I open outlook, takes 5 minutes of trying to connect to get connected to exchange. I see over 5000ms avg resp time in connection status of outlook. I call 365 support and the guy with too many vowels in his name spends 3 hours and tells me it is my windows profile. I politely tell him bs as I just loaded windows last night and nothing else. He wants to transfer me to the outlook group and sends me over. The "outlook group" guy directs me to the support.microsoft.com site to open a case, he isn't technical. I hang up pissed off. Emailed our partner who opens a partner support case and 2 days later they look at it and I get, this might be part of a widespread unannounced issue, thanks for wasting my time......

 

[Duckie Dooh]

speedye1[/url]":1ab9eowv]TIL Office 365 support is terrible. We moved my our first ten mailboxes and just happened to setup a new machine for myself. Load fresh and updated Win 8 and Office 2013, took 30 minutes to load my mailbox, and every time I open outlook, takes 5 minutes of trying to connect to get connected to exchange. I see over 5000ms avg resp time in connection status of outlook. I call 365 support and the guy with too many vowels in his name spends 3 hours and tells me it is my windows profile. I politely tell him bs as I just loaded windows last night and nothing else. He wants to transfer me to the outlook group and sends me over. The "outlook group" guy directs me to the support.microsoft.com site to open a case, he isn't technical. I hang up pissed off. Emailed our partner who opens a partner support case and 2 days later they look at it and I get, this might be part of a widespread unannounced issue, thanks for wasting my time......

When they say that they are going to "Transfer this to their back end team" they really mean that they are going to put the ticket on hold until you talk to someone on another shift. I have had more luck with the team that handles support at 6AM Eastern than with the teams that work during the day or in the evening.

 

[Xon]

speedye1[/url]":1ry6fdo3]TIL Office 365 support is terrible.

At least you aren't in Australia, where the monopolistic Telco(Telstra) who owns all the copper access network and a majority of all backhaul fiber and via contract with Microsoft, is the sole provider of Office 365 in Australia. Any resellers need to go through Telstra, and not Microsoft.

Telstra has been known at being very aggressive at aquiring clients, even if they were clients of resellers of Telstra's Office 365 service.

 

[speedye1]

What a fun week, so expanding on my 365 woes, our first 10 users, including myself have had outlook connectivity issues where outlook would just stop updating. A close and open of outlook resolves the issue. So I find how to turn outlook advanced logging on to get an idea of when and how often connectivity may be dropping. I find the .etl log file and what a surprise, only Microsoft support has the tools to open these gems.

 

[gradster]

speedye1[/url]":qketqdn1]TIL Office 365 support is terrible.

Yes it is. Even when I was doing a trial of Office 365 for a possible sale I couldn't get any helpful support at all. In fact, the support was so terrible I almost would have rather had no support at all. They did nothing but waste my time.

 

[yak8998]

speedye1[/url]":cjdcjuy3]TIL Office 365 support is terrible

Fixed

 

[mdporter]

speedye1[/url]":1dwjxzhx]What a fun week, so expanding on my 365 woes, our first 10 users, including myself have had outlook connectivity issues where outlook would just stop updating. A close and open of outlook resolves the issue. So I find how to turn outlook advanced logging on to get an idea of when and how often connectivity may be dropping. I find the .etl log file and what a surprise, only Microsoft support has the tools to open these gems.

This is not good to hear, we are doing an eval of office 365 now and are looking at moving to it. Buy in from senior execs already done. 2,000+ mailboxes. Our exchange 2007 infrastructure has been flawless for years.

 

[SandyTech]

Today I learned the downside to our straight to offsite backup system... the restore time for a major restore *sucks*. For one or two files, even a gig or two worth of files it's ok, but for anything more (al la the 1.2TB directory that I'm now having to restore) it's shite.

 

[jshiplett]

Allow me to speak for the rest of the Server Room when I say, "Duh."