This blog talks about the how the amount of data generated is increasing day by data (data-explosion) and also the pace of data generation, which is explained as in 3 V’s nomenclature as volume, velocity and variety of data is changing dramatically. The data management and storage domain is getting matured and new market opportunities with innovative technology are getting created. In the following description, we will briefly see the data exposition and Big-data future trends and in detail how he basic data storage evolution happened till today.
Data explosion, Big-data management and market opportunity:
According to research, we send 294 billion emails, post more than 230 million tweets, and conduct over a billion Google searches every day, while trillions of sensors populate the Internet of Things with even more real-time data. Data storage means big business these days and is expected to account for 7.5% of semiconductor revenue by 2020. In the enterprise space, big data and analytics, cloud services, data protection, and legal retention requirements all add to the need for higher capacity and faster, more cost-efficient storage.
Big data1 is proving its value to organizations of all types and sizes, and in a wide range of industries. Enterprises that make advanced use of big data are realizing tangible business benefits, from improved efficiency in operations and increased visibility into rapidly changing environments to the optimization of products and services for customers.
1: The definition of big data is data that contains greater variety, arriving in increasing volumes and with more velocity. Put simply, big data is larger, more complex data sets, especially from new data sources.
Following picture depicts the trends in the Big data storage and processing.
By 2021, it’s estimated that heat-assisted magnetic recording will enable a desktop hard drive with a capacity of 60 TB – that’s 60,000,000 MB, capable of storing 12 million photos or 6,750 hours of high-definition (HD) video. The cloud storage market is expected to be worth US$74.94 billion by 2021, while the anticipated value of the big data storage market is US$ 61.44 billion by 2026 (a compounded annual growth rate of 20.4% over the 10 years from 2016).
Storage Device evolution:
Storage technology has evolved over a period of time at both individual and enterprise level. We have come a long way from the early days of computing. Here is the story of Storage device evolution.
1890’s Punch cards (0.08KB) were the first mechanical storage method. A punch card holds about 80 characters, so not even a fully fleshed out tweet could fit on one.
The punch card dates back to the 19th century, when it was used to program mechanical devices such as looms and player pianos. Punch cards were commonly used for computer programming through the 1980s. Although punch cards are now obsolete as a recording medium, we still use punch cards to store data today, mainly in standardized tests and voting.
Magnetic drums :
1932’s Magnetic drum the first magnetic drum held 48 KB, about five formatted .doc files.
Drum memory was originally invented by Gustav Tauschek in 1932, but magnetic drum memory wasn’t used in computing until US Navy codebreakers developed it during World War II. One drum was 16 inches long and held 40 tracks that spun at 12,500 revolutions per minute. Engineering Research Associates (ERA) continued development of the technology with their Atlas project. Drum memory consisted of a long metal cylinder coated in magnetic material, with rows of read-write heads situated on the axis of the drum. It was once used as a primary storage device and remained common in computing through the 50s and 60s, but is now used as an auxiliary storage device.
1951, A new standard. The first core memory used in a computer stored a little more than 2KB, roughly the size of a small PNG image file or 2,000 characters of text.
Picture showing A 32 x 32 core memory plane storing 1024 bits (or 128 bytes) of data. The small black rings at the intersections of the grid wires, organised in four squares, are the ferrite cores.
Invented in 1951, magnetic core memory was first used in the MIT Whirlwind computer. Core memory works by storing one bit of data on tiny magnetic rings, or cores. The more magnetic cores you pack into a core memory, the more data you can store on it. Core memory was the standard in computing from 1955 to 1975. As recently as 2004, a magnetic core memory system was found still in service in a telephony control system. It continues to capture the interest of modern enthusiasts today.
Hard Disk Drive:
1956 The spin doctor. With 3.75 MB of storage, the first HDDs had enough storage space to hold a whole mp3 file, 45 seconds of low-resolution video, or 5 million characters of text.
The HDD, first introduced by IBM in 1956, weighed over a ton and was the size of a refrigerator. The HDD stores data on one or more rapidly rotating magnetic metal platters, or disks. The HDD is still ubiquitous today, with portable models becoming smaller, with higher storage capacity, every year.
The solid state drive (SSD), created by Samsung, is only 2½ inches and can hold 16 terabytes of data. It is undoubtedly the highest capacity hard drive on the market today.
Floppy Disk Drive:
1967 the first 8-inch floppy disk developed held 80KB, enough to hold 8 formatted .doc files. The floppy disk was developed at IBM’s San Jose laboratory in 1967. Originally, floppy disks were uncovered magnetic disks, hence the “flop.” Later, plastic envelopes were added to protect from dirt and scratches and varying sizes of the disk emerged. By 1978, more than 10 manufacturers were producing 5¼ inch floppy drives, but you might remember the 3½ inch disk best. By the mid-70s, floppy disks were the most widely used form of portable data storage. Floppy disks have limited use today, but are shockingly still used in US nuclear bases
[1978-1990]: An early—if not the first—semiconductor storage device compatible with a hard drive interface (e.g. an SSD as defined) was the 1978 StorageTek STC 4305. The STC 4305 a plug-compatible replacement for the IBM 2305 fixed head disk drive, initially used charge-coupled devices (CCDs) for storage. In the late 1980s, Zitel offered a family of DRAM based SSD products, under the trade name “RAMDisk”, for use on systems by UNIVAC and Perkin-Elmer, among others.
The basis for flash-based SSDs, flash memory, was invented by Fujio Masuoka at Toshiba in 1980 and commercialized by Toshiba in 1987. SanDisk Corporation (then SanDisk) founders Eli Harari and Sanjay Mehrotra, along with Robert D. Norman, saw the potential of flash memory as an alternative to existing hard drives, and filed a patent for a flash-based SSD in 1989. The first commercial flash-based SSD was shipped by SanDisk in 1991. It was a 20 MB SSD in a PCMCIA configuration.
Most SSD manufacturers use non-volatile NAND flash memory in the construction of their SSDs because of the lower cost compared with DRAM and the ability to retain the data without a constant power supply, ensuring data persistence through sudden power outages. SSDs based on volatile memory such as DRAM are characterized by very fast data access, generally less than 10 microseconds, and are used primarily to accelerate applications that would otherwise be held back by the latency of flash SSDs or traditional HDDs.
In 2015, Intel and Micron announced 3D XPoint as a new non-volatile memory technology. Intel released the first 3D XPoint-based drive (branded as Intel® Optane™ SSD) in March 2017 starting with a data center product, Intel® Optane™ SSD DC P4800X Series, and following with the client version, Intel® Optane™ SSD 900P Series, in October 2017. Both products operate faster and with higher endurance than NAND-based SSDs, while the areal density is comparable at 128 gigabits per chip. SSD classification can be done based on memory type used for storage, host interface type like SCSI, SATA, PCIe M.2, U.2 , Fiber channel, USB Parallel ATA, Parallel SCSI
Compact Disk Drive (CD):
1982 The first highly portable optical storage. CDs had a capacity of 650 – 700 MB. That could hold 70,000 formatted .doc files, 140 minutes of low-resolution video, or, more appropriately, your favorite Radiohead album, OK Computer.
The Compact Disc was developed in 1982 by both Sony and Phillips. Although the CD was only 12 centi-meters in diameter, when first introduced, the CD could hold more data than a personal computer’s hard drive. CD drives read the data stored on discs by shining a focused laser beam at the surface of the disc. CDs revolutionized the music industry in the 1980s, eventually replacing the vinyl record and cassette tape. The sale of CDs has been eclipsed by digital music in recent years, but still sell by the tens of millions every year.
1992 The home of the floppy disk. The first Zip Drive could hold about 100 minutes of MP3 audio. That’s equal to 20 minutes of low-resolution video or 1/7 of a CD.
The original Zip Drive was a medium-to-high-capacity removable floppy disk storage system, introduced by Iomega in late 1994. It launched with capacities of 100 MB, but later versions improved capacity from 250 MB to 750 MB. Some organizations still use Zip Drives today.
Compact Disk (CD), Video Compact Disk (VCD), Digital Video Disk (DVD):
The CD’s invention started in late 1965 and however more popularity is seen in late 1995 when it becomes cost effective. CD stands for Compact Disk. It is a data storage format, which basically means that it is used to store data. It is a type of digital optical disc data storage format, which means that the data is stored on it via the use of an optic laser which burns pits and grooves on to the surface of the disk.
The VCD/DVD stands for Video Compact Disk, it is a type of CD format which is optimized to store Video formats.
The data storage capacity of CD, VCD and DVD are evolved over time and start with 680 Mbytes, 800Mbytes and 8.5Gbytes respectively. The first DVD had 1.46 GB of storage, big enough to hold a short movie or 2 CDs. Some manufacturers make dual-sided, single-layer discs that can hold 9.4 GB of data.
SD card (Solid-state Drive card):
1999: The first SD cards held around 64MB, enough to hold 50 photos or 13 minutes of low-resolution video which is around 1/11 of a CD. The highest capacity of an SD card today is 1 terabyte. The Secure Digital standard was a joint development by SanDisk, Panasonic, and Toshiba in 1999. This technology built on previous iterations, such as the Multimedia Card (MMC). SD cards use flash memory, which stores data in cells made of floating-gate transistors. The small size and thinness of the cards made them ideal for digital photography and videography. Later, mini and micro sizes of the cards were released
USB flash Drive:
1999: The drive of many names. The first flash drive developed held 8MB, so one or two eBooks, 90 seconds of low-resolution video or 800 .doc files. The world’s largest USB Flash Drive today has a 2 terabyte capacity.
M-Systems, an Israeli company, developed the USB Flash Drive in 1999. It is colloquially known as a thumb drive, pen drive, jump drive, disk key, disk on key, flash-drive, or a memory stick. Similar to SD cards, USB flash drives use flash memory. USB flash drives became popular as portable storage devices due to the convenience of plugging them into a computer’s USB port for data transfer.
Blue ray Disk:
2003: The disc. The myth. The legend. This high-definition disc supported and stored 25 GB of high definition video at 1080p, which is around 36 CDs. Sony has cranked up optical disc storage to 3.3 terabytes today. Intended to be the successor to the DVD, the Blu-ray optical disc was developed by a technology industry consortium. While older DVDs were only capable of 480p resolution, the Blu-ray swooped in with more than double the capacity. The name was derived from the relatively short wavelength blue laser capable of reading a higher density of data on the disc as opposed to the red laser used for reading
Cloud Storage: (Unlisted storage !!)
2006: Now, your storage capacity depends only on the plan you can pay for. The options are endless.
The first all web-based data storage system was PersonaLink Services, launched by AT&T in 1994. Amazon Web Services launched AWS S3 in 2006, in part starting the trend toward massive cloud data storage. With cloud storage, remote databases are used to store information, made accessible at any time via internet access. As cloud technologies improve, cloud storage will become less and less expensive.
The backend storage media will be composed of various types like storage disk and even to the solid state storage drives. The architecture of the storage system depends on the end application capacity needs and the performance.