What is Apache Cassandra?
* Apache Cassandra is an open source, distributed and decentralized/distributed storage system (database), for managing very large amounts of structured data spread out across the world.
* It provides highly available service with no single point of failure.
MongoDB:
* MongoDB is a cross-platform document-oriented database system that avoids using the traditional table-based relational database structure in favor of JSON-like documents with dynamic schemas making the integration of data in certain types of applications easier and faster.
Apache HBase :
* HBase is an open source, non-relational, distributed database modeled after Google’s BigTable and is written in Java.
* It is developed as a part of Apache Hadoop project and runs on top of HDFS, providing BigTable-like capabilities for Hadoop.
* HAL hardware abstraction layer is used by the operating system to interact with the hardware.
* It is simply a dll call HAL.dll.
* What does HAL.dll does is that it provides the necessary drivers for your operating system to interact with hardware.
* The problem with a HAl.dll is that when you restore your operating system you need to use identical hardware for successful restore this was huge drawback with HAL.dll
* In new your operating system such as Windows 10 this dependency on HAL.dll is removed.
* Now when you restore your operating system the BIOS will detect the existing hardware or the new hardware on which the operating system is being restored.
* To configure this dynamic HAL you need to use a tool call BCDedit.
Bcdedit /set {current} detecthal yes
Data Model
* Database is the outermost data structure/container in CouchDB.
* Each database is a collection of independent documents.
* Each document maintains its own data and self-contained schema.
* Document metadata contains revision information, which makes it possible to merge the differences occurred while the databases were disconnected.
* CouchDB implements multi version concurrency control, to avoid the need to lock the database field during writes.
NoSQL Databases
* A NoSQL database (sometimes called as Not Only SQL) is a database that provides a mechanism to store and retrieve data other than the tabular relations used in relational databases.
* These databases are schema-free, support easy replication, have simple API, eventually consistent, and can handle huge amounts of data (big data).
* The primary objective of a NoSQL database is to have the following −
1. Simplicity of design,
2.Horizontal scaling, and
3.Finer control over availability.
What is CouchDB?
* CouchDB is an open source database developed by Apache software foundation. The focus is on the ease of use, embracing the web. It is a NoSQL document store database.
* It uses JSON, to store data (documents), java script as its query language to transform the documents, http protocol for api to access the documents, query the indices with the web browser.
*Avro-Serialization is the process of translating data structures or objects state into binary or textual form to transport the data over network or to store on some persisten storage.
*Once the data is transported over network or retrieved from the persistent storage, it needs to be deserialized again. Serialization is termed as marshalling and deserialization is termed as unmarshalling.
* Apache Avro is a language-neutral data serialization system. It was developed by Doug Cutting, the father of Hadoop. Since Hadoop writable classes lack language portability, Avro becomes quite helpful, as it deals with data formats that can be processed by multiple languages. Avro is a preferred tool to serialize data in Hadoop.
* Avro has a schema-based system. A language-independent schema is associated with its read and write operations. Avro serializes the data which has a built-in schema. Avro serializes the data into a compact binary format, which can be deserialized by any application.
* Avro uses JSON format to declare the data structures. Presently, it supports languages such as Java, C, C++, C#, Python, and Ruby.
Sqoop:
Sqoop is an open source framework provided by Apache. It is a command-line interface application for efficiently transferring bulk data between Apache Hadoop and external datastores such as relational databases, enterprise data warehouses. It is used to import data from relational databases such as MySQL, Oracle to Hadoop HDFS, and export from Hadoop file system to relational databases.
Sqoop Import:
The import tool is used to imports individual tables from RDBMS to HDFS. Each row in a table is treated as a record in HDFS.
Sqoop Export:
The export tool exports a set of files from HDFS back to an RDBMS. The files given as input to Sqoop contain records, which are called as rows in table.
Why Sqoop is used?
For Hadoop developers, the work starts after data is loaded into HDFS. For this, the data residing in the relational database management systems need to be transferred to HDFS and might need to transfer back to relational database management systems. So developers can always write custom scripts to transfer data in and out of Hadoop, but Apache Sqoop provides an alternative.
Sqoop uses MapReduce framework to import and export the data, which provides parallel mechanism as well as fault tolerance. Sqoop makes developers life easy by providing command line interface. Developers just need to provide basic information like source, destination and database authentication details in the sqoop command.
Sqoop Connectors:
All the existing Database Management Systems are designed with SQL standard in mind. However, each DBMS differs with respect to dialect to some extent. So, this difference poses challenges when it comes to data transfers across the systems. Sqoop Connectors are components which help overcome these challenges. Data transfer between Sqoop and external storage system is made possible with the help of Sqoop's connectors. Sqoop has connectors for working with a range of popular relational databases, including MySQL, PostgreSQL, Oracle, SQL Server, and DB2. Each of these connectors knows how to interact with its associated DBMS. There is also a generic JDBC connector for connecting to any database that supports Java's JDBC protocol. In addition, Sqoop provides optimized MySQL and PostgreSQL connectors that use database-specific APIs to perform bulk transfers efficiently.
Hadoop is not a database: Hadoop is an distributed file system and not a database. It simply uses the filesystem provided by Linux to store data
What is Hadoop?
Apache Hadoop is an open-source java framework that is used to store,analyze and process Big data in a distributed environment across clusters of computers. It is used by Google, Facebook, Yahoo, Youtube, Twitter, LinkedIn and many more. Hadoop is top level apache project developed by Doug Cutting. It works on distributed system. It is inspired by google mapreduce algorithm for running distributed application. Hadoop could be good if you have a lot of data you do not know what to do and you do not want to loose it.
Hadoop is initiated and led by YAHOO. hadoop is written in java programming language with some code in C and some commands in shell-scripts. Companies that offer hadoop services are - IBM, Amazon Web Services, Cloudera, Microsoft Azure, Dell.
