Friday, May 5, 2017
HDFS Command
Wednesday, April 26, 2017
Share folder in Window(O/s) & access in Hadoop(VM)
As we are focusing more on practical aspect we need big files to test/learn big data technology. In order to work on that we can share big files in our local machine (window) & can be access in hadoop using below action.
1. Create Folder & share in our Windows System (ShareWindow)
Hadoop Configuration File
Please keep this configuration(.xml) file handy in order to work in Hadoop Ecosystem.
- core-site.xml: It contains the configuration settings for Hadoop Core such as I/O settings that are common to HDFS and MapReduce.
- hdfs-site.xml: All the configuration settings for HDFS daemons(background process), the namenode, the secondary namenode and the data nodes are specified or can be specified in this file.
- mapred-site.xml: Configuration settings related to MapReduce daemons : the job-tracker and the task-trackers can be done here.
Monday, September 26, 2016
My first Pig Script to count number of occurances of IP address from the log file
I'm glad to inform you all that today i have successfully wrote a PIG script to generate count of IP address from the 26 Lakhs of records.
I got chance to handle big data issue in my current company & i used PIG script to execute the task,
Here is my script to count the occurances of IP addresss from log file
Ldata = LOAD '/user/cloudera/Pigdata/totalIPcount.txt' AS (line:chararray);
IP = FOREACH Ldata GENERATE FLATTEN(TOKENIZE(line)) as IPaddress;
grouped = GROUP IP BY IPaddress;
IPCount= FOREACH grouped GENERATE group, COUNT(IP);
DUMP IPCount; OR STORE IPCount INTO '/Pigdata'
Thanks
Pradeep
Friday, June 26, 2015
Difference between Hadoop1.0 & Hadoop 2.0
- Ability to run Non MapReduce Application on Hadoop 2.0
- Improved Resource Utilization
- a global Resource Manager and
- Per-application Application Master.
- Native Windows Support
- Beyond Batch Oriented application: Hadoop goes beyond Batch oriented nature in its version 2.0 and now can run interactive, streaming application also.
- HDFS Federation
- HDFS- Multiple Storage
- Faster access to data—Data Node caching
- HDFS Snapshots
- Protection against user errors: An admin can set up a process to take snapshots periodically. If a user accidentally deletes files, these can be restored from the snapshot that contains the files.
- Backup: If an admin wants to back up the entire file system or a subtree in the file system, the admin takes a snapshot and uses it as the starting point of a full backup. Incremental backups are then taken by copying the difference between two snapshots.
- Disaster recovery: Snapshots can be used for copying consistent point-in-time images over to a remote site for disaster recovery.
Sunday, March 8, 2015
Difference Between Sql server 2005/2008 & Sql server 2008/2012
Here i am with most asked & common interview question in SQL.
What is the difference between SQL server 2005 & SQL server 2008.
Here, most of us[not all] used same functionality which was present in earlier version because out project scope. But in order to make interviewer impress here is the differnce
Difference between SQL Server 2008 & SQL Server 2012
Tuesday, December 4, 2012
Getting duplicate records with count
where kcode='4' group by type
having count() > 1 order by COUNT() DESC
Monday, December 3, 2012
latest / Min /Max Value Group wise
from (
select Kcode,max(creat
from tblAgentTran
group by Kcode
) as x inner join tblAgentTran as f on f.Kcode = x.Kcode and f.createddateti
Friday, November 2, 2012
From Recent Search , Vinay find good & quick solution for inserting record from one old table to new table, with specific columns
SELECT columns....
FROM initial_table where [condition]
using this query we can insert some records form old table to new table with given condition.
thanks vinay for your solution.
Recently @ time of checking records in Database table with case sensitive.
We are firing query like
select * from Tablename where strPassword = 'Vinay343@'
at the same time if we fire query like
select * from Tablename where strPassword = 'VINAY343@'
we are getting same result so technically me & vinay think that it is not good to get result if user send case sensitive input
after searching we found the solution that if we want to check exact match use query like this
select * from Tablename where strPassword COLLATE Latin1_General_
here We get exact match as per the input
Thanks Vinay for sharing this information
Cheers!!!!
Friday, August 24, 2012
Calculate Age from Sql function
As i m not too strong in SQL server , i have started from .Net code for calculating Age from DOB.
but not success ed in that. after searching in books & discussing with friends[Vinay Shah]
come to conclusion that we can get Age from Sql Function DATEDIFF.
in SQL Server 2005 u can use Datediff function
Sample Query = select datediff (year, '1988-05-17 00:00:00.000' , getDate())
from above query you will get the Age in year OUTPUT = 24
instead of 1988-05-17 00:00:00.000 you can enter your DOB & get your Age.
in Oracle
select ROUND((SysDate - DOB)/365) as Age from cust_table
here you we will get age of the customers where DOB should have Date DataType
Using Datediff function we can calculate not only age in years but also following
| Unit of time | Query | Result |
| NANOSECOND | SELECT DATEDIFF(NANOSECOND,'2011-09-23 17:15:22.5500000','2011-09-23 17:15:22.55432133') | 4321300 |
| MICROSECOND | SELECT DATEDIFF(MICROSECOND,'2011-09-23 17:15:22.5500000','2011-09-23 17:15:22.55432133') | 4321 |
| MILLISECOND | SELECT DATEDIFF(MILLISECOND,'2011-09-23 17:15:22.004','2011-09-23 17:15:22.548') | 544 |
| SECOND | SELECT DATEDIFF(SECOND,'2011-09-23 17:15:30','2011-09-23 17:16:23') | 53 |
| MINUTE | SELECT DATEDIFF(MINUTE,'2011-09-23 18:03:23','2011-09-23 17:15:30') | -48 |
| HOUR | SELECT DATEDIFF(HH,'2011-09-23 18:03:23','2011-09-23 20:15:30') | 2 |
| WEEK | SELECT DATEDIFF(WK,'09/23/2011 15:00:00','12/11/2011 14:00:00') | 12 |
| DAY | SELECT DATEDIFF(DD,'09/23/2011 15:00:00','08/02/2011 14:00:00') | -52 |
| DAYOFYEAR | SELECT DATEDIFF(DY,'01/01/2011 15:00:00','08/02/2011 14:00:00') | 213 |
| MONTH | SELECT DATEDIFF(MM,'11/02/2011 15:00:00','01/01/2011 14:00:00') | -10 |
| QUARTER | SELECT DATEDIFF(QQ,'01/02/2011 15:00:00','08/01/2011 14:00:00') | 2 |
| YEAR | SELECT DATEDIFF(YY,'01/02/2011 15:00:00','01/01/2016 14:00:00') | 5 |
Friday, July 13, 2012
Tuesday, February 7, 2012
Delete Duplicate Rows
/* Create Table with 7 entries - 3 are duplicate entries */
CREATE TABLE DuplicateRcordTable (Col1 INT, Col2 INT)
INSERT INTO DuplicateRcordTable
SELECT 1, 1
UNION ALL
SELECT 1, 1 --duplicate
UNION ALL
SELECT 1, 1 --duplicate
UNION ALL
SELECT 1, 2
UNION ALL
SELECT 1, 2 --duplicate
UNION ALL
SELECT 1, 3
UNION ALL
SELECT 1, 4
GO
The above table has total 7 records, out of which 3 are duplicate records. Once the duplicates are removed we will have only 4 records left.
/* It should give you 7 rows */
SELECT *
FROM DuplicateRcordTable
GO
The most interesting part of this is yet to come. We will use CTE that will re-generate the same table with additional column, which is row number. In our case, we have Col1 and Col2 and both the columns qualify as duplicate rows. It may be a different set of rows for each different query like this. Another point to note here is that once CTE is created DELETE statement can be run on it. We will put a condition here – when we receive more than one rows of record, we will remove the row which is not the first one. When DELETE command is executed over CTE it in fact deletes from the base table used in CTE.
/* Delete Duplicate records */
WITH CTE (COl1,Col2, DuplicateCount)
AS
(
SELECT COl1,Col2,
ROW_NUMBER() OVER(PARTITION BY COl1,Col2 ORDER BY Col1) AS DuplicateCount
FROM DuplicateRcordTable
)
DELETE
FROM CTE
WHERE DuplicateCount > 1
GO
It is apparent that after delete command has been run, we will have only 4 records, which is almost the same result which we would have got with DISTINCT, with this resultset. If we had more than 2 columns and we had to run unique on only two columns, our distinct might have not worked here . In this case, we would have to use above the mentioned method.
/* It should give you Distinct 4 records */
SELECT *
FROM DuplicateRcordTable
GO
Friday, November 11, 2011
INDEXING IN SQL SERVER
Using the Right Indexes for Optimal Performance Query optimization is a complex game with its own rules.
Let’s look at three examples to discover when SQL Server Query Optimizer uses clustered indexes and non-clustered indexes to retrieve data and when to use the primary key (PK) to influence performance.
Example 1: Default Index Usage
Let’s look first at Query Optimizer’s default use of indexes.
Query 1’s query cost (relative to the batch) is much lower than Query 2’s query cost (relative to the batch).
In this example, no query hint is specified, so Query Optimizer can use any index it wants to use, which results in optimal performance.
Notice that even though ContactID (the primary key [PK] of the Contact table) is retrieved in Query 1, Query Optimizer does not use a primary key clustered index; instead, it uses a non-clustered index. In Query 2, on the other hand, Query Optimizer uses a clustered index on PK where all columns (*) are retrieved.
This may be surprising. It is a common belief that when PK columns are used and no other condition or joins are used, Query Optimizer will use a PK clustered index to return the results of a SELECT statement. However, this is not always true.
Example 2 : Forcing a Primary Key (PK) Clustered Index
Let’s see how performance changes when a primary key clustered index is used to retrieve data.
Query 1’s query cost (relative to the batch) is equals to Query 2’s query cost (also relative to the batch).
In this example, we are using a primary key clustered index to retrieve data. The same execution plan is created whether we retrieve only one column or all the columns.
Example 3: Forcing Non-Clustered Index
Now consider two queries in which a non-clustered index is used to retrieve data.
Query 1’s query cost (relative to the batch) is much lower than Query 2’s query cost (also relative to the batch).
In this example, we are using a non-clustered index to retrieve data. From the execution plan, it is very clear that retrieving only one column is much faster than retrieving all the columns from the Contact table. When all the columns from a table are selected, a PK clustered index is clearly the best option.
Monday, July 11, 2011
TRIGGERS IN SQL SERVER
BACKGROUND
This article gives a brief introduction about Triggers in Sql Server 2000/2005.
What is a Trigger
A trigger is a special kind of a store procedure that executes in response to certain action on the table like insertion, deletion or updation of data. It is a database object which is bound to a table and is executed automatically. You can’t explicitly invoke triggers. The only way to do this is by performing the required action no the table that they are assigned to.
Types Of Triggers
There are three action query types that you use in SQL which are INSERT, UPDATE and DELETE. So, there are three types of triggers and hybrids that come from mixing and matching the events and timings that fire them.
Basically, triggers are classified into two main types:-
(i) After Triggers (For Triggers)
(ii) Instead Of Triggers
(i) After Triggers
These triggers run after an insert, update or delete on a table. They are not supported for views.
AFTER TRIGGERS can be classified further into three types as:
(a) AFTER INSERT Trigger.
(b) AFTER UPDATE Trigger.
(c) AFTER DELETE Trigger.
Let’s create After triggers. First of all, let’s create a table and insert some sample data. Then, on this table, I will be attaching several triggers.
Collapse
CREATE TABLE Employee_Test
(
Emp_ID INT Identity,
Emp_name Varchar(100),
Emp_Sal Decimal (10,2)
)
INSERT INTO Employee_Test VALUES ('Anees',1000);
INSERT INTO Employee_Test VALUES ('Rick',1200);
INSERT INTO Employee_Test VALUES ('John',1100);
INSERT INTO Employee_Test VALUES ('Stephen',1300);
INSERT INTO Employee_Test VALUES ('Maria',1400);
I will be creating an AFTER INSERT TRIGGER which will insert the rows inserted into the table into another audit table. The main purpose of this audit table is to record the changes in the main table. This can be thought of as a generic audit trigger.
Now, create the audit table as:-
Collapse
CREATE TABLE Employee_Test_Audit
(
Emp_ID int,
Emp_name varchar(100),
Emp_Sal decimal (10,2),
Audit_Action varchar(100),
Audit_Timestamp datetime
)
(a) AFTRE INSERT Trigger
This trigger is fired after an INSERT on the table. Let’s create the trigger as:-
Collapse
CREATE TRIGGER trgAfterInsert ON [dbo].[Employee_Test]
FOR INSERT
AS
declare @empid int;
declare @empname varchar(100);
declare @empsal decimal(10,2);
declare @audit_action varchar(100);
select @empid=i.Emp_ID from inserted i;
select @empname=i.Emp_Name from inserted i;
select @empsal=i.Emp_Sal from inserted i;
set @audit_action='Inserted Record -- After Insert Trigger.';
insert into Employee_Test_Audit
(Emp_ID,Emp_Name,Emp_Sal,Audit_Action,Audit_Timestamp)
values(@empid,@empname,@empsal,@audit_action,getdate());
PRINT 'AFTER INSERT trigger fired.'
GO
The CREATE TRIGGER statement is used to create the trigger. THE ON clause specifies the table name on which the trigger is to be attached. The FOR INSERT specifies that this is an AFTER INSERT trigger. In place of FOR INSERT, AFTER INSERT can be used. Both of them mean the same.
In the trigger body, table named inserted has been used. This table is a logical table and contains the row that has been inserted. I have selected the fields from the logical inserted table from the row that has been inserted into different variables, and finally inserted those values into the Audit table.
To see the newly created trigger in action, lets insert a row into the main table as :
Collapse
insert into Employee_Test values('Chris',1500);
Now, a record has been inserted into the Employee_Test table. The AFTER INSERT trigger attached to this table has inserted the record into the Employee_Test_Audit as:-
Collapse
6 Chris 1500.00 Inserted Record -- After Insert Trigger. 2008-04-26 12:00:55.700
(b) AFTER UPDATE Trigger
This trigger is fired after an update on the table. Let’s create the trigger as:-
Collapse
CREATE TRIGGER trgAfterUpdate ON [dbo].[Employee_Test]
FOR UPDATE
AS
declare @empid int;
declare @empname varchar(100);
declare @empsal decimal(10,2);
declare @audit_action varchar(100);
select @empid=i.Emp_ID from inserted i;
select @empname=i.Emp_Name from inserted i;
select @empsal=i.Emp_Sal from inserted i;
if update(Emp_Name)
set @audit_action='Updated Record -- After Update Trigger.';
if update(Emp_Sal)
set @audit_action='Updated Record -- After Update Trigger.';
insert into Employee_Test_Audit(Emp_ID,Emp_Name,Emp_Sal,Audit_Action,Audit_Timestamp)
values(@empid,@empname,@empsal,@audit_action,getdate());
PRINT 'AFTER UPDATE Trigger fired.'
GO
The AFTER UPDATE Trigger is created in which the updated record is inserted into the audit table. There is no logical table updated like the logical table inserted. We can obtain the updated value of a field from the update(column_name) function. In our trigger, we have used, if update(Emp_Name) to check if the column Emp_Name has been updated. We have similarly checked the column Emp_Sal for an update.
Let’s update a record column and see what happens.
Collapse
update Employee_Test set Emp_Sal=1550 where Emp_ID=6
This inserts the row into the audit table as:-
Collapse
6 Chris 1550.00 Updated Record -- After Update Trigger. 2008-04-26 12:38:11.843
(c) AFTER DELETE Trigger
This trigger is fired after a delete on the table. Let’s create the trigger as:-
Collapse
CREATE TRIGGER trgAfterDelete ON [dbo].[Employee_Test]
AFTER DELETE
AS
declare @empid int;
declare @empname varchar(100);
declare @empsal decimal(10,2);
declare @audit_action varchar(100);
select @empid=d.Emp_ID from deleted d;
select @empname=d.Emp_Name from deleted d;
select @empsal=d.Emp_Sal from deleted d;
set @audit_action='Deleted -- After Delete Trigger.';
insert into Employee_Test_Audit
(Emp_ID,Emp_Name,Emp_Sal,Audit_Action,Audit_Timestamp)
values(@empid,@empname,@empsal,@audit_action,getdate());
PRINT 'AFTER DELETE TRIGGER fired.'
GO
In this trigger, the deleted record’s data is picked from the logical deleted table and inserted into the audit table.
Let’s fire a delete on the main table.
A record has been inserted into the audit table as:-
Collapse
6 Chris 1550.00 Deleted -- After Delete Trigger. 2008-04-26 12:52:13.867
All the triggers can be enabled/disabled on the table using the statement
Collapse
ALTER TABLE Employee_Test {ENABLE|DISBALE} TRIGGER ALL
Specific Triggers can be enabled or disabled as :-
Collapse
ALTER TABLE Employee_Test DISABLE TRIGGER trgAfterDelete
This disables the After Delete Trigger named trgAfterDelete on the specified table.
(ii) Instead Of Triggers
These can be used as an interceptor for anything that anyonr tried to do on our table or view. If you define an Instead Of trigger on a table for the Delete operation, they try to delete rows, and they will not actually get deleted (unless you issue another delete instruction from within the trigger)
INSTEAD OF TRIGGERS can be classified further into three types as:-
(a) INSTEAD OF INSERT Trigger.
(b) INSTEAD OF UPDATE Trigger.
(c) INSTEAD OF DELETE Trigger.
(a) Let’s create an Instead Of Delete Trigger as:-
Collapse
CREATE TRIGGER trgInsteadOfDelete ON [dbo].[Employee_Test]
INSTEAD OF DELETE
AS
declare @emp_id int;
declare @emp_name varchar(100);
declare @emp_sal int;
select @emp_id=d.Emp_ID from deleted d;
select @emp_name=d.Emp_Name from deleted d;
select @emp_sal=d.Emp_Sal from deleted d;
BEGIN
if(@emp_sal>1200)
begin
RAISERROR('Cannot delete where salary > 1200',16,1);
ROLLBACK;
end
else
begin
delete from Employee_Test where Emp_ID=@emp_id;
COMMIT;
insert into Employee_Test_Audit(Emp_ID,Emp_Name,Emp_Sal,Audit_Action,Audit_Timestamp)
values(@emp_id,@emp_name,@emp_sal,'Deleted -- Instead Of Delete Trigger.',getdate());
PRINT 'Record Deleted -- Instead Of Delete Trigger.'
end
END
GO
This trigger will prevent the deletion of records from the table where Emp_Sal > 1200. If such a record is deleted, the Instead Of Trigger will rollback the transaction, otherwise the transaction will be committed.
Now, let’s try to delete a record with the Emp_Sal >1200 as:-
Collapse
delete from Employee_Test where Emp_ID=4
This will print an error message as defined in the RAISE ERROR statement as:-
Collapse
Server: Msg 50000, Level 16, State 1, Procedure trgInsteadOfDelete, Line 15
Cannot delete where salary > 1200
And this record will not be deleted.
In a similar way, you can code Instead of Insert and Instead Of Update triggers on your tables.
Friday, June 3, 2011
We find one validation expression where you can use it by your own,
Ex: 1 ValidationExpression="^[A-Z ]+"
in aboce exaple i want only character With Upper Case so i can enter only upper case character
Ex: 1 ValidationExpression="^[0-9 ]+"
in aboce exaple i want only Numbers so i can enter only Numbers using aboce statement
example where special character not allowed
ValidationExpression="^[a-zA-Z0-9 ]+"
Hope it will helop youi all!!!!!!!!!!!!!!!!!!!!!!!!
Friday, May 6, 2011
select records with comma
select * from msttest WHERE ',' + testfield1 + ',' LIKE '%,11,%' or testfield1 in('11','12')
Tuesday, April 26, 2011
Access denied in XML files
few days ago , i was so frustrated because i am getting "Access denied Error in Asp .net Project,
after doing lots of R&D i got sucess, whenever you do XML Operation always put
here
accountname = server name like adminisrator
password = your server password which u used for remote login
Tuesday, March 22, 2011
SQL SERVER - Create Script to Copy full Database including SP, Trigger,Views ,function & all
here is the solution
Step 1 : Start
Step 2 : Welcome Screen
Step 3 : Select One or Multiple Database
If Script all objects in the selected database checkbox is not selected it will give options to selected individual objects on respective screen. (e.g. Stored Procedure, Triggers and all other object will have their own screen where they can be selected)
Step 4 : Select database options
Step 5 : Select output option
Step 6 : Review Summary
Step 7 : Observe script generation process
Step 8 : Database object script generation completed in new query window
Saturday, March 5, 2011
Casting Solution
Recently My friend Manoj sharma faced a issue in casting, so here are few question Answers which help you related to casting
Few of the questions I receive very frequently. I have collect them in spreadsheet and try to answer them frequently.
How to convert text to integer in SQL?
If table column is VARCHAR and has all the numeric values in it, it can be retrieved as Integer using CAST or CONVERT function.
How to use CAST or CONVERT?
SELECT CAST(YourVarcharCol AS INT) FROM Table
SELECT CONVERT(INT, YourVarcharCol) FROM Table
Will CAST or CONVERT thrown an error when column values converted from alpha-numeric characters to numeric?
YES.
Will CAST or CONVERT retrieve only numbers when column values converted from alpha-numeric characters to numeric?
NO.
