OpenSolaris, illumos, OpenIndiana, NexentaOS, SmartOS and OmniOS

[M. Jones]

OpenSolaris was Sun's open-sourcing of most of the System V (plus much else) Unix based Solaris operating system. It's now the third major family of open-source Unix, after Linux and BSD.

When Oracle closed Solaris source in 2010, illumos and OpenIndiana forked from OpenSolaris.

Illumos is considered the canonical open-source upstream. All derived systems have ZFS filesystem, DTrace instrumentation, Zones lightweight virtualization, Crossbow networking virtualization, and added recently, KVM hypervisor functionality.

OpenIndiana is a general purpose distribution with X and desktop environment. If you intend to run illumos on the desktop, you want OpenIndiana.

NexentaOS is a storage-oriented distribution with illumos kernel and GNU/Ubuntu userland, from a vendor of storage appliances.

SmartOS, created by Joyent, is a specialised distribution designed to be a minimal hypervisor host. It's designed to be either PXE/iPXE netbooted or booted from a discrete USB-based flash and to run from compressed RAM. It's comparable to ESXi or CoreOS, and it forms the foundation of Joyent's Smart Data Center (SDC) cloud platform. SmartOS is patched or upgraded by booting to a newer release system image. NetBSD's packaging system is native and can be used to install applications. SmartOS runs Zones virtualization and KVM virtualization; live-migration of KVM guests is not supported, in large part because all storage is local -- SDC doesn't depend on shared storage, and is mostly intended to run cloud (i.e., resilient application-level high availability) instances. Because SmartOS can boot from USB and runs from RAM, it might also be a good choice for a ZFS-based NAS appliance.

OmniOS is a server-only minimal self-hosting distribution of illumos maintained by OmniTI. It gets releases twice a year like Ubuntu Linux, and it gets a patch release every week of the year. OmniOS uses IPS, the Solaris 10 and OpenSolaris packaging mechanism. It doesn't have X or a desktop, but OmniOS is probably the best illumos distribution for running apps on metal and general server functionality (ZFS, CIFS/SMB and NFS serving, webserving).

 

[whm1974]

I was under the impression that Solaris, open or otherwise is pretty much dead or at least on life support.

 

[Deleted member 1068]

There is a vast global Solaris install base.
It's not going anywhere soon.
It's seen strong competition from Linux with respect to new installs, but huge financial institutions still like keeping Mr. Ellison in yachts.

 

[zelmak]

Fotan[/url]":ldh1i442]There is a vast global Solaris install base.
It's not going anywhere soon.
It's seen strong competition from Linux with respect to new installs, but huge financial institutions still like keeping Mr. Ellison in yachts.

And the US Gov't.

 

[Kydaria]

Something to note is that because Oracle unceremoniously closed the source code for Solaris. They have denied themselves of the features that Illumos has added since the split. Thus no KVM or Feature Flags in Solaris 11, conversely there is no encryption built into ZFS on Illumos.

 

[fandingo]

Kydaria[/url]":2cpaghd9]Something to note is that because Oracle unceremoniously closed the source code for Solaris. They have denied themselves of the features that Illumos has added since the split. Thus no KVM or Feature Flags in Solaris 11, conversely there is no encryption built into ZFS on Illumos.

Only by choice. There's no license restrictions that prohibit Oracle from merging Illumos code, portions of which are available under a mix of MIT, BSD, and CDDL licenses.

 

[M. Jones]

whm1974[/url]":2oojasp3]I was under the impression that Solaris, open or otherwise is pretty much dead or at least on life support.

At this point, illumos (nee OpenSolaris) and Solaris are two different things. illumos can't use new proprietary, closed Solaris source code, and Solaris can't use CDDL licensed illumos contributions unless they re-open source OpenSolaris.

The real reason why Solaris was and is in decline is because of the high costs of SPARC hardware today, compared to x86. SPARC is technically an open architecture, but I don't think there's been much of a competitive supplier base since the early 2000s or perhaps even the late 1990s.

It's a vicious cycle. Anyone who's still on Solaris/UltraSPARC is assumed to be in that position because they've locked themselves in to that architecture, and if they're still a profitable enterprise, are regarded as foolish but lucrative as a customer. Just like CA's modus operandi was always to buy software in decline, and to make money by riding down the demand curve while making minimal new investments, Oracle's new role is to charge decommodified prices to anyone who still buys Oracle or SPARC hardware.

But illumos isn't Solaris, it is the third family of open-source Unix operating systems. A typical OmniTI client is running PostgreSQL (not Oracle, and BSD-license open-source) on OmniOS (not Solaris, and CDDL license open-source) on commodity Intel AMD64 hardware (not UltraSPARC).

 

[stmiller]

Fotan[/url]":1dp4uqb5]There is a vast global Solaris install base.
It's not going anywhere soon.

Until the vendor-locked hardware becomes obsolete...

 

[fandingo]

M. Jones[/url]":373zuino]

whm1974[/url]":373zuino]I was under the impression that Solaris, open or otherwise is pretty much dead or at least on life support.

At this point, illumos (nee OpenSolaris) and Solaris are two different things. illumos can't use new proprietary, closed Solaris source code, and Solaris can't use CDDL licensed illumos contributions unless they re-open source OpenSolaris.

That's not true. While the CDDL is copyleft, it doesn't "infect" other code like the GPL does.

The real reason why Solaris was and is in decline is because of the high costs of SPARC hardware today, compared to x86.

True, although SPARC was always expensive.

SPARC is technically an open architecture, but I don't think there's been much of a competitive supplier base since the early 2000s or perhaps even the late 1990s.

Not true. Intel has had credible competition for years; yet, prices remain low and improvements have kept pace. SPARC costs more mainly due to a limited market that has trouble defraying research and fixed manufacturing costs. Intel has always delivered great value because it operates the best fabrication plants and processes.

 

[whm1974]

Until the vendor-locked hardware becomes obsolete...

I thought SPARC hardware is already obsolete.

Not true. Intel has had credible competition for years; yet, prices remain low and improvements have kept pace. SPARC costs more mainly due to a limited market that has trouble defraying research and fixed manufacturing costs. Intel has always delivered great value because it operates the best fabrication plants and processes.

And don't forget, Intel also has the huge market share pay for all this.

 

[M. Jones]

fandingo[/url]":2yi1iemu]SPARC costs more mainly due to a limited market that has trouble defraying research and fixed manufacturing costs. Intel has always delivered great value because it operates the best fabrication plants and processes.

Intel has always been able to amortize the massive development costs of successive x86 implementations across an equally massive number of units, in part by combining the consumer space with the enterprise. Despite attrition in the RISC space, no design ever came to own all of the RISC marketshare -- Intel had a large hand in engineering this. I ran and owned and was a big fan of SPARC and Alpha, and those two had decent prospects of owning the space and MIPS even more so.

With the rising tide of enterprise x86 sales to enterprises, most observers were convinced that IA64 was the heir apparent and conceded the fight. All of the RISC vendors did a deal with the x86 devil, except Sun. The market ended up rejecting IA64 because it was Intel's attempt to decommodify, but by the time the RISC competitors realized that that they'd already surrendered.

 

[fandingo]

M. Jones[/url]":1l5qz399]

fandingo[/url]":1l5qz399]SPARC costs more mainly due to a limited market that has trouble defraying research and fixed manufacturing costs. Intel has always delivered great value because it operates the best fabrication plants and processes.

Intel has always been able to amortize the massive development costs of successive x86 implementations across an equally massive number of units, in part by combining the consumer space with the enterprise.

But mostly by being successful in all of their markets separately. Intel has never needed to leverage consumer or business against the other to succeed.

Despite attrition in the RISC space, no design ever came to own all of the RISC marketshare -- Intel had a large hand in engineering this. I ran and owned and was a big fan of SPARC and Alpha, and those two had decent prospects of owning the space and MIPS even more so.

With the rising tide of enterprise x86 sales to enterprises, most observers were convinced that IA64 was the heir apparent and conceded the fight. All of the RISC vendors did a deal with the x86 devil, except Sun. The market ended up rejecting IA64 because it was Intel's attempt to decommodify, but by the time the RISC competitors realized that that they'd already surrendered.

This is nostalgia from last century and nothing more. The RISC vendors were more like trolls in the sun. They never had a chance because they had no advantage.

 

[MilleniX]

You folks saying SPARC is dead might want to let Fujitsu know about that.

 

[whm1974]

You folks saying SPARC is dead might want to let Fujitsu know about that.

Is anyone releasing new SPARC hardware? And not to start a flame war here, but what can we do with SPARC that we can't do with x86-64?

 

[atacama]

whm1974[/url]":1teawt74]

You folks saying SPARC is dead might want to let Fujitsu know about that.

Is anyone releasing new SPARC hardware? And not to start a flame war here, but what can we do with SPARC that we can't do with x86-64?

The problem isn't only what you can do with SPARC that you can't do with x86-64, but also what can you buy, and what you can buy in small quantities.

You need SPARC (or PowerPC or MIPS) instead of x86-64 for the following scenarios in the data center:

• data center friendly servers for really high available 24/7 operation. These would work with x86-64, too; but you can't buy them.
  A data center friendly server does not have a keyboard/monitor interface. It does not even enable you to attach a keyboard or a monitor.
  A data center friendly server has as its primary console interface a serial console.
  The point is, that there should be no need to have a crash cart with a monitor in the data center. All consoles should be connected to a serial console switch.
  A data center friendly server should not have a BIOS, but something like openboot. The problem is the GUI interface aspect of the BIOS. The output of the console should be easily logged to a log server.
  
• high bandwidth servers. Again these could work with x86-64, too; but you can't buy them.
  Traditionally SPARC/RISC have had more bandwidth to RAM and I/O than x86. This raises the cost, but was nearly always needed to give them the edge to compete against x86.
  
• Requirement for fast context switches. On all x86 context switches are relatively expensive. Multithreaded CPUs relieve the problem a bit, but the problem is still there. One solution would be the introduction of context identifiers into the MMU and the cache of the x86 CPU. That would require a major overhaul of the architecture. Most data paths would need to be wider by the number of bits required for the context identifier. (8 parallel context turned out to small for single threaded SPARC in 1990.)
  
• Requirement for binary incompatibility to x86-64. Doesn't need to be SPARC. But PowerPC, Itanium and IBM Z-Series are even more expensive.
  Announcements and prototypes for ARM Servers have been around for several years, but you still can't buy them.
  
• Requirement for highly reliable servers. This is usually covered by HP/Tandem (Itanium) and IBM Z-Series, but has been done with SPARC.
  
• Requirement for more than four/eight CPU sockets in a system.
  
• Requirement for big endian hardware. This requirement is rare in the data center today.

And SPARC systems are usually less expensive than PowerPC, Itanium or IBM Z-Series.


But Solaris isn't bound to SPARC only, there are still some use cases for Solaris x86-64, even if Oracle priced a lot of them away.

 

[Gandalf007]

M. Jones[/url]":1fv5i3rd]
With the rising tide of enterprise x86 sales to enterprises, most observers were convinced that IA64 was the heir apparent and conceded the fight. All of the RISC vendors did a deal with the x86 devil, except Sun. The market ended up rejecting IA64 because it was Intel's attempt to decommodify, but by the time the RISC competitors realized that that they'd already surrendered.

Sun dithered on x86 Solaris support (I think it was in 8, but not in 9, then came back for 10, or something like that), but they've had an amd64 server line (with Intel CPUs in recent years) for close to 15 years. Bet they sell a lot more of those than Sparc hardware (albeit at somewhat lower margins). I disagree that IA64 (Itanium) was ever the heir apparent -- Intel and HP certainly wished it were, but it never got there. amd64 is the real heir to both x86 and most RISC arches, which was sealed when Intel started making their own amd64-compatible CPUs.

The main use case for Solaris x86 was ZFS on cheap hardware. With ZFS spreading to FreeBSD and Linux (kinda) and Illumos et al replacing OpenSolaris, there's no reason to give Larry any more yacht money.

 

[Neolithic]

Moving away from the hardware discussion...

You can use the pkgsrc repo for SmartOS with the other illumos distros (OI and Omni).

http://pkgsrc.joyent.com/installing.html

This gives you a larger and more likely to be up-to-date repo for OI and it gives you an actually repo for Omni.

 

[Deleted member 43669]

atacama[/url]":1szz642b]

• data center friendly servers for really high available 24/7 operation. These would work with x86-64, too; but you can't buy them.
  A data center friendly server does not have a keyboard/monitor interface. It does not even enable you to attach a keyboard or a monitor.
  A data center friendly server has as its primary console interface a serial console.
  The point is, that there should be no need to have a crash cart with a monitor in the data center. All consoles should be connected to a serial console switch.
  A data center friendly server should not have a BIOS, but something like openboot. The problem is the GUI interface aspect of the BIOS. The output of the console should be easily logged to a log server.

Errm. ILO/DRAC/whatever sort out this- maybe you prefer serial consoles, but kb/monitors are not required for decent modern x86 servers.

• Requirement for binary incompatibility to x86-64. Doesn't need to be SPARC. But PowerPC, Itanium and IBM Z-Series are even more expensive.
  Announcements and prototypes for ARM Servers have been around for several years, but you still can't buy them.

I'm sure someone in the world has that requirement- but I've never heard of it. Why would someone need that?

Some other requirements you mention relate to availability, which to some extent is more efficiently (availability/cost) by redundancy.

SPARC customers are a split of locked in people and people with very rare and specific requirements, in general.

 

[M. Jones]

Gandalf007[/url]":30b5zgej]
Sun dithered on x86 Solaris support (I think it was in 8, but not in 9, then came back for 10, or something like that), but they've had an amd64 server line (with Intel CPUs in recent years) for close to 15 years.

Not counting the Sun i386, I'm pretty sure it's 10 years, or 12 at the most. I was running Solaris x86 on the desktop (Compaq) around 15 years ago and around that time bought SPARCs for an ecommerce project where the same investment would have bought a lot more x86, given more margin to throw hardware at certain problems, and saved some grief. That project was using Oracle Application Server, so it would have been dependent on x86 architecture support from Oracle. In retrospect I should have thrown in the towel a long time before that and followed the advice from Karl Denninger and been using x86 for servers once PCI and ATX were introduced to that platform. Not long before running Solaris on x86, my work desktop was a uniprocessor SPARCstation running OpenBSD.

I don't think Sun shipped any non-amd64 'x86' hardware, and AMD didn't ship 64-bit until 2003.

 

[Deleted member 1068]

stmiller[/url]":22fmqdah]Until the vendor-locked hardware becomes obsolete...

There is Solaris for x86 you know?
Crusty old Solaris Ops hate x86, but they would prefer to run Solaris on that than switch to another OS.

 

[keltor]

Most Solaris Admins have become Solaris + Linux admins. In my personal world we just have Windows and Unix admins - the Unix admins deal with both AIX and Linux with of the guys more specialized in one of our application platforms than the other. 10 years ago we had a lot more Sun/Fujitsu stuff. We also have a mainframe which crazy enough doesn't have an admin - it's a weird non-standard beast of some strange Japanese mainframe where the admin died about 7 years ago and apparently we basically cannot find an admin and there's been no need to have an admin ...

 

[M. Jones]

keltor[/url]":339w9ixu]We also have a mainframe which crazy enough doesn't have an admin - it's a weird non-standard beast of some strange Japanese mainframe where the admin died about 7 years ago and apparently we basically cannot find an admin and there's been no need to have an admin ...

Hitachi? I think several Japanese firms cloned IBM 'frames, but I think Hitachi Skylines were the biggest. Last mainframe on which I had privs I gave the network hostname 'Gojira'.

 

[keltor]

This thing is an ancient beast from the 1970s and I believe it's a Mitsubishi something, I'm actually quite surprised it wasn't something from Toshiba (Toshiba is part of the Mitsui Keiretsu like Sony.)

Thankfully my software only has to be deployed to a set of RHEL5 servers.

 

[atacama]

koala[/url]":1bi2q32b]

atacama[/url]":1bi2q32b]

• data center friendly servers for really high available 24/7 operation. These would work with x86-64, too; but you can't buy them.
  A data center friendly server does not have a keyboard/monitor interface. It does not even enable you to attach a keyboard or a monitor.
  A data center friendly server has as its primary console interface a serial console.
  The point is, that there should be no need to have a crash cart with a monitor in the data center. All consoles should be connected to a serial console switch.
  A data center friendly server should not have a BIOS, but something like openboot. The problem is the GUI interface aspect of the BIOS. The output of the console should be easily logged to a log server.

Errm. ILO/DRAC/whatever sort out this- maybe you prefer serial consoles, but kb/monitors are not required for decent modern x86 servers.

I like to have iDRAC on all new Dell servers. They allow remote handling of 90% to 95% of the problems, and do away with most drive ins to the data center during the night.

But they are not a 100% solution. The java applet only runs with some very specific versions of java, and there are enough limitations that the initial installation of a server is mostly done in the machine room (or in the lab with the server being carried into the lab). (One of the pain points is the initial configuration of the iDRAC and the BIOS.)

The suns with the serial console can be completely installed from the office.

koala[/url]":1bi2q32b]
SPARC customers are a split of locked in people and people with very rare and specific requirements, in general.

Yes.

 

[feistypenguin]

keltor[/url]":1d874vi6]Most Solaris Admins have become Solaris + Linux admins. In my personal world we just have Windows and Unix admins - the Unix admins deal with both AIX and Linux with of the guys more specialized in one of our application platforms than the other. 10 years ago we had a lot more Sun/Fujitsu stuff. We also have a mainframe which crazy enough doesn't have an admin - it's a weird non-standard beast of some strange Japanese mainframe where the admin died about 7 years ago and apparently we basically cannot find an admin and there's been no need to have an admin ...

+1 on this. My job title is "Unix systems administrator", which includes anything on the Unix family tree. I'm a newer Linux admin and had no exposure to Solaris before taking this job, though after a few years of working with it, I'm glad it is on the way out in our shop. You can do some cool things on it for sure (alternate boot environments, ZFS, Zones, etc) Liveupgrade is awesome when it works, and terrifying when it doesn't (I've been left adrift on the openboot prompt several times because of this.) For us though, it doesn't have any standout features that our business can't find in other products and at a lower cost.

Also, the more I have to deal with Oracle Support, the less I want to work with any Oracle products :/

 

[keltor]

And before anyone tells you about the "good old" days of Sun support, they pretty much behave the same. Actually Oracle and Sun were probably a super easy merger because they often seemed to behave the same way.

 

[Neolithic]

NexentaStor 4.0 has been released

The big deal for this thread with this release is the migration from the OpenSolaris codebase to the illumos codebase (with associated source, though that GitHub page has been available for a little while already). The feature I'm most looking forward to migrating upstream is SMB 2.1 support.

 

[M. Jones]

Neolithic[/url]":2xs9ojcb]The feature I'm most looking forward to migrating upstream is SMB 2.1 support.

Oh, I was just planning on looking into the latest version in support. Useful to know. Is Nexenta normally prompt about contributing to upstream?

 

[Neolithic]

M. Jones[/url]":394gxxqa]

Neolithic[/url]":394gxxqa]The feature I'm most looking forward to migrating upstream is SMB 2.1 support.

Oh, I was just planning on looking into the latest version in support. Useful to know. Is Nexenta normally prompt about contributing to upstream?

That's been a major gripe for some time. Since they were maintaining an old branch of OpenSolaris they weren't contributing much for quite some time. And now, with the recent push to get 4.0 out they may be too busy to upstream for a little while while dealing with all the x.0 support shenanigans. The code might be available but the bug database isn't so it might be hard for a third party to tease out all the SMB commits.

 

[Higgers]

Can anyone comment on any or all of ZFS/NFS/CIFS/iSCSI performance between the various distributions? I used to run Solaris Express then moved to OI and am now running ZoL on Ubuntu 13.10, all as VMs under ESXi. Performance seems ok but I'm seeing latency spikes on the NFS datastore in ESXi. I've built an OmniOS VM to play with to see if it provides better performance.

 

[Mr. Green Hat]

M. Jones[/url]":dw1nb8h1]
I don't think Sun shipped any non-amd64 'x86' hardware, and AMD didn't ship 64-bit until 2003.

Well Solaris x86 was definitely out before then. I've got an article containing screenshots of the installation process on my very own website for the 12/02 Intel Platform Edition of Solaris 9. Though if you look at the second page, the very last screenshot contains a post from Leftover Noodles @ Ars made in February of 2003 so that article couldn't have been written before then (my website engine didn't track article dates until later).